mental-models.md

2016 PyData SF

Mental Models to Use and Avoid as a Data Scientist

• Jonathan Whitmore, PhD
• Data Scientist, SVDS
• @jbwhitmore

Watch the talk on YouTube here:

P-hacking

• https://xkcd.com/882/

Blinding

• Blinding statsmodels output: https://gist.github.com/dmargala/be3ef4c7cd676eac07e3dc8add4603d2
• http://pdg.lbl.gov/2005/reviews/historyrpp.pdf
• http://news.berkeley.edu/2015/10/08/blind-analysis-could-reduce-bias-in-social-science-research/
• https://www.slac.stanford.edu/econf/C030908/papers/TUIT001.pdf
• BLIND ANALYSIS IN NUCLEAR AND PARTICLE PHYSICS Joshua R. Klein and Aaron Roodman Annual Review of Nuclear and Particle Science, Vol. 55: 141 -163 (Volume publication date December 2005)

Gender-Bias Example

• https://peerj.com/preprints/1733v1/
• Terrell J, Kofink A, Middleton J, Rainear C, Murphy-Hill E, Parnin C. (2016) Gender bias in open source: Pull request acceptance of women versus men. PeerJ PrePrints 4:e1733v1 https://doi.org/10.7287/peerj.preprints.1733v1
• https://peerj.com/preprints/1733/
• https://peerj.com/questions/2002-do-you-have-data-on-the-gender-of-the-users-that/#annotation-2002-replies
• Some Criticism
• http://slatestarcodex.com/2016/02/12/before-you-get-too-excited-about-that-github-study/

Redcard Example

• Silberzahn, R. & Uhlmann, E. L. Psychol. Sci. 24, 2437–2444 (2013).
• Silberzahn, Raphael et al. “Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results.” Open Science Framework, 20 Aug. 2015. Web.
• Final Manuscript: https://osf.io/j5v8f/
• Data: https://osf.io/47tnc/
• http://fivethirtyeight.com/features/science-isnt-broken/#part2

Thresholdout

• http://science.sciencemag.org/content/349/6248/636

• Source: The reusable holdout: Preserving validity in adaptive data analysis, Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A, Science, 07 Aug 2015: 636-638.

• http://blog.mrtz.org/2015/03/09/competition.html

• https://research.googleblog.com/2015/08/the-reusable-holdout-preserving.html

• https://www.oreilly.com/ideas/3-ideas-to-add-to-your-data-science-toolkit

Nature blog posts

• http://www.nature.com/news/how-scientists-fool-themselves-and-how-they-can-stop-1.18517
• http://www.nature.com/news/blind-analysis-hide-results-to-seek-the-truth-1.18510
• http://www.nature.com/news/let-s-think-about-cognitive-bias-1.18520
• http://www.nature.com/news/crowdsourced-research-many-hands-make-tight-work-1.18508

Other sources

• Gelman, Andrew, and Eric Loken. "The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time." Department of Statistics, Columbia University (2013).

• Considering the opposite: A corrective strategy for social judgment. Lord, Charles G.; Lepper, Mark R.; Preston, Elizabeth Journal of Personality and Social Psychology, Vol 47(6), Dec 1984, 1231-1243.

• Kerr, Norbert L. "HARKing: Hypothesizing after the results are known." Personality and Social Psychology Review 2.3 (1998): 196-217.

• The reusable holdout: Preserving validity in adaptive data analysis, Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A, Science, 07 Aug 2015: 636-638.