| 1. |
- Delios, A., et al.
(författare)
-
Examining the generalizability of research findings from archival data
- 2022
-
Ingår i: Proceedings of the National Academy of Sciences of the United States of America. - : Proceedings of the National Academy of Sciences. - 0027-8424 .- 1091-6490. ; 119:30
-
Tidskriftsartikel (refereegranskat)abstract
- This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
|
|
| 2. |
- Tierney, W., et al.
(författare)
-
A creative destruction approach to replication : Implicit work and sex morality across cultures
- 2021
-
Ingår i: Journal of Experimental Social Psychology. - : Elsevier BV. - 0022-1031 .- 1096-0465. ; 93
-
Tidskriftsartikel (refereegranskat)abstract
- How can we maximize what is learned from a replication study? In the creative destruction approach to replication, the original hypothesis is compared not only to the null hypothesis, but also to predictions derived from multiple alternative theoretical accounts of the phenomenon. To this end, new populations and measures are included in the design in addition to the original ones, to help determine which theory best accounts for the results across multiple key outcomes and contexts. The present pre-registered empirical project compared the Implicit Puritanism account of intuitive work and sex morality to theories positing regional, religious, and social class differences; explicit rather than implicit cultural differences in values; self-expression vs. survival values as a key cultural fault line; the general moralization of work; and false positive effects. Contradicting Implicit Puritanism's core theoretical claim of a distinct American work morality, a number of targeted findings replicated across multiple comparison cultures, whereas several failed to replicate in all samples and were identified as likely false positives. No support emerged for theories predicting regional variability and specific individual-differences moderators (religious affiliation, religiosity, and education level). Overall, the results provide evidence that work is intuitively moralized across cultures.
|
|
| 3. |
- Hudson, Lawrence N, et al.
(författare)
-
The database of the PREDICTS (Projecting Responses of Ecological Diversity In Changing Terrestrial Systems) project
- 2017
-
Ingår i: Ecology and Evolution. - : John Wiley & Sons. - 2045-7758. ; 7:1, s. 145-188
-
Tidskriftsartikel (refereegranskat)abstract
- The PREDICTS project-Projecting Responses of Ecological Diversity In Changing Terrestrial Systems (www.predicts.org.uk)-has collated from published studies a large, reasonably representative database of comparable samples of biodiversity from multiple sites that differ in the nature or intensity of human impacts relating to land use. We have used this evidence base to develop global and regional statistical models of how local biodiversity responds to these measures. We describe and make freely available this 2016 release of the database, containing more than 3.2 million records sampled at over 26,000 locations and representing over 47,000 species. We outline how the database can help in answering a range of questions in ecology and conservation biology. To our knowledge, this is the largest and most geographically and taxonomically representative database of spatial comparisons of biodiversity that has been collated to date; it will be useful to researchers and international efforts wishing to model and understand the global status of biodiversity.
|
|
| 4. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 5. |
- Ebersole, Charles R., et al.
(författare)
-
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
- 2020
-
Ingår i: Advances in Methods and Practices in Psychological Science. - : Sage. - 2515-2467 .- 2515-2459. ; 3:3, s. 309-331
-
Tidskriftsartikel (refereegranskat)abstract
- Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).
|
|
| 6. |
- Aczel, Balazs, et al.
(författare)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
Ingår i: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 7. |
|
|
| 8. |
- Benjamin, Daniel J., et al.
(författare)
-
Redefine statistical significance
- 2018
-
Ingår i: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 9. |
- Camerer, C. F., et al.
(författare)
-
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
- 2018
-
Ingår i: Nature Human Behaviour. - : Springer Science and Business Media LLC. - 2397-3374. ; 2:9, s. 637-644
-
Tidskriftsartikel (refereegranskat)abstract
- Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated truepositive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
|
|
| 10. |
- Ranehill, Eva, et al.
(författare)
-
Hormonal Contraceptives Do Not Impact Economic Preferences: Evidence from a Randomized Trial
- 2018
-
Ingår i: Management science. - : Institute for Operations Research and the Management Sciences (INFORMS). - 0025-1909 .- 1526-5501. ; 64:10, s. 4471-4965
-
Tidskriftsartikel (refereegranskat)abstract
- A growing body of correlational studies suggests that sex hormones such as those contained in, or affected by, oral contraceptives (OCs) may impact economic behavior. However, despite widespread use of OCs among women in Western countries, little is known about their potential behavioral effects. The present study investigates whether OCs causally influence economic preferences. We randomly allocate 340 women aged 18–35 to three months of a widely used OC or placebo treatment. At the end of treatment, we conduct an economic experiment measuring altruism, financial risk taking, and willingness to compete. The statistical power is 80% to detect an effect size equal to a Cohen’s d of 0.30 at the 5% level. We find no significant effects of OCs on any of the measured preferences, indicating that this widely used OC treatment, commonly used throughout the world, does not significantly affect the measured economic preferences. Further, we find no relation between menstrual cycle phase and economic preferences in the placebo group.
|
|
