| 1. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 2. |
- Altmejd, Adam, et al.
(författare)
-
Predicting the replicability of social science lab experiments
- 2019
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Tidskriftsartikel (refereegranskat)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 3. |
- Benjamin, Daniel J., et al.
(författare)
-
Redefine statistical significance
- 2018
-
Ingår i: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 4. |
- Camerer, Colin, et al.
(författare)
-
Correspondence : Are Cognitive Functions Localizable?
- 2013
-
Ingår i: Journal of Economic Perspectives. - : American Economic Association. - 0895-3309 .- 1944-7965. ; 27:2, s. 247-250
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 5. |
- Camerer, Colin F., et al.
(författare)
-
Testing Game Theory in the Field : Swedish LUPI Lottery Games
- 2011
-
Ingår i: American Economic Journal. - : American Economic Association. - 1945-7669 .- 1945-7685. ; 3:3, s. 1-33
-
Tidskriftsartikel (refereegranskat)abstract
- Game theory is usually difficult to test in the field because predictions typically depend sensitively on features that are not controlled or observed. We conduct one such test using both laboratory and field data from the Swedish lowest unique positive integer (LUPI) game. In this game, players pick positive integers and whoever chooses the lowest unique number wins. Equilibrium predictions are derived assuming Poisson distributed population uncertainty. The field and lab data show similar patterns. Despite various deviations from equilibrium, there is a surprising degree of convergence toward equilibrium. Some deviations can be rationalized by a cognitive hierarchy model. (JEL C70, C93, D44, H27)
|
|
| 6. |
- Camerer, Colin, et al.
(författare)
-
Replication and other practices for improving scientific quality in experimental economics
- 2019
-
Ingår i: Handbook of Research Methods and Applications in Experimental Economics. - : Edward Elgar Publishing. - 9781788110563 - 9781788110556 ; , s. 83-102
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)abstract
- Replications are key to the advancement of science. The recent “replication crisis” in the social sciences has highlighted the need for high-powered replications to confirm original results before strong conclusions can be drawn. Here we discuss two big replication projects in psychology and economics respectively and assess their results in terms of the various replication indicators that have been proposed. We further discuss how we have used prediction markets and surveys to predict replication outcomes, and end with a discussion of general quality control in research, and our opinions on improved practices.
|
|
| 7. |
|
|
| 8. |
- Poldrack, Russell Alan, et al.
(författare)
-
fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study
- 2019
-
Ingår i: Scientific Data. - : Nature Research (part of Springer Nature): Fully open access journals / Nature Publishing Group. - 2052-4463 .- 2052-4463. ; 6:1
-
Tidskriftsartikel (refereegranskat)abstract
- There is an ongoing debate about the replicability of neuroimaging research. It was suggested that one of the main reasons for the high rate of false positive results is the many degrees of freedom researchers have during data analysis. In the Neuroimaging Analysis Replication and Prediction Study (NARPS), we aim to provide the first scientific evidence on the variability of results across analysis teams in neuroscience. We collected fMRI data from 108 participants during two versions of the mixed gambles task, which is often used to study decision-making under risk. For each participant, the dataset includes an anatomical (T1 weighted) scan and fMRI as well as behavioral data from four runs of the task. The dataset is shared through OpenNeuro and is formatted according to the Brain Imaging Data Structure (BIDS) standard. Data pre-processed with fMRIprep and quality control reports are also publicly shared. This dataset can be used to study decision-making under risk and to test replicability and interpretability of previous results in the field.
|
|
