| 1. |
- Altmejd, Adam, et al.
(författare)
-
Predicting the replicability of social science lab experiments
- 2019
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Tidskriftsartikel (refereegranskat)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 2. |
- Engström, Per, 1974-, et al.
(författare)
-
Increasing the take-up of the housing allowance among Swedish pensioners : a field experiment
- 2019
-
Ingår i: International Tax and Public Finance. - : SPRINGER. - 0927-5940 .- 1573-6970. ; 26:6, s. 1353-1382
-
Tidskriftsartikel (refereegranskat)abstract
- Using a randomized field experiment in the Swedish pension system, we investigate whether receiving an information letter affects the take-up rate of the housing allowance for pensioners. We also investigate whether the framing of the information letter affects take-up. The results show that simple information letters had a significant effect on the application rate and subsequent take-up rate: The baseline application rate in the targeted control population was only 1.4%, while the corresponding rates in the different treatment groups were between 9.9 and 12.1%. However, while the applications in the control group were accepted in almost 3 out of 4 cases, up to 50% of the applications in the treatment group were declined. The lower conditional acceptance rate in the treatment group seems to be largely driven by wealth, which the Pensions Agency cannot observe prior to submission. Information campaigns aimed at increasing benefit take-up therefore need careful design in situations with imperfect targeting.
|
|
| 3. |
- Forsell, Eskil, et al.
(författare)
-
Predicting replication outcomes in the Many Labs 2 study
- 2019
-
Ingår i: Journal of Economic Psychology. - : Elsevier. - 1872-7719 .- 0167-4870. ; 75:Part A SI
-
Tidskriftsartikel (refereegranskat)abstract
- Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicited peer beliefs in prediction markets and surveys about two replication success metrics: the probability that the replication yields a statistically significant effect in the original direction (p < 0.001), and the relative effect size of the replication. The prediction markets correctly predicted 75% of the replication outcomes, and were highly correlated with the replication outcomes. Survey beliefs were also significantly correlated with replication outcomes, but had larger prediction errors. The prediction markets for relative effect sizes attracted little trading and thus did not work well. The survey beliefs about relative effect sizes performed better and were significantly correlated with observed relative effect sizes. The results suggest that replication outcomes can be predicted and that the elicitation of peer beliefs can increase our knowledge about scientific reproducibility and the dynamics of hypothesis testing.
|
|
