Statistical Significance and the Replication Crisis in the Social Sciences

• The second most viewed TED talk with more than 48 million views as of September 2018 is about power posing. The TED talk is based on a study where 42 men and women were reported to have been randomized to hold either a high-power or a low-power position for a couple of minutes (Carney, Cuddy,XX1Yap, 2010). The results are stunning with large effects from a small intervention. High-power positions supposedly lead to higher testosterone levels, lower cortisol levels, increased financial risk-taking, and increased feelings of power. The study was published in the top scientific journal Psychological Science in 2010. Five years later the same journal published a replication attempt with 200 participants (Ranehill et al., 2015). The replication study differed from the original study in some aspects; one was that the replication was experimenter blind. In the larger sample, Ranehill et al. failed to find support for power posing having any effects on hormones or behaviour. A number of other failed replications have been published since then, and the first author has written an open letter describing some of the reasons why the original paper reports p values that are basically meaningless in terms of understanding false positive risk (Carney, 2016). The power posing paper is by no means alone in making big claims that turn out to not replicate; there are reasons to believe that in many fields in the social sciences a substantial share of published results are false-positives. This article describes the many reasons for why this may be the case. There is also a discussion of the various replication projects attempting to gauge the share of false-positive results, including the Reproducibility Project: Psychology (RPP; Open Science Collaboration, 2015), the Experimental Economics Replication Project (EERP; Camerer et al., 2016), and the Social Science Replication Project (SSRP; Camerer et al., 2018) and the lessons learned from these projects. In particular, the RPP led to a scientific debate about a potential replication crisis (Anderson et al., 2016; Gilbert, King, Pettigrew,XX1Wilson, 2016) and many subsequent replication projects (Cova et al., 2018; Ebersole et al., 2016; Klein et al., 2014; Schweinsberg et al., 2016). The many different solutions to the replication problem are also relevant. While some of the content in this article is also discussed in a recent book chapter (Camerer, Dreber,XX1Johannesson, 2019), the most significant difference between this article and the chapter is the focus here on the results from SSRP, which is not part of the book chapter. This article also summarizes the current pooled results on peer beliefs as reproducibility indicators and elaborates more on p values and decision markets.

Ämnesord

SAMHÄLLSVETENSKAP  -- Psykologi (hsv//swe)

SOCIAL SCIENCES  -- Psychology (hsv//eng)

Publikations- och innehållstyp

kap (ämneskategori)

vet (ämneskategori)