| 1. |
- Altmejd, Adam, et al.
(författare)
-
Predicting the replicability of social science lab experiments
- 2019
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Tidskriftsartikel (refereegranskat)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 2. |
|
|
| 3. |
- Engstrom, Per, et al.
(författare)
-
Increasing the Take-Up of the Housing Allowance Among Swedish Pensioners : A Field Experiment
- 2018
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- Using a randomized field experiment in the Swedish pension system, we investigate whether receiving an information letter affects the take-up rate of the housing allowance for pensioners. We also investigate whether the framing of the information letter affects take-up. The results show that simple information letters had a dramatic effect on the application rate and subsequent take-up rate: the baseline application rate in the targeted control population was only 1.4 percent while the corresponding rates in the different treatment groups were between 9.9 and 12.1 percent. The letter that addressed common misconceptions about the benefit caused significantly higher submission and acceptance rates. The letters had a substantial economic effect on the applicants. We estimate that the applicants, induced by the treatment, increased their monthly income by around 10 percent.
|
|
| 4. |
- Engström, Per, 1974-, et al.
(författare)
-
Bostadstillägg för pensionärer : ett randomiserat informationsexperiment
- 2018
-
Ingår i: Ekonomisk Debatt. - : Nationalekonomiska Föreningen. - 0345-2646. ; 46:5, s. 29-37
-
Tidskriftsartikel (populärvet., debatt m.m.)abstract
- Många äldre med låga inkomster ansöker inte om bostadstillägg för pensionärer trots att de kan ha rätt till det. En viktig fråga är därför hur man kan få fler berättigade att ansöka. Vi har tillsammans med Pensionsmyndigheten genomfört ett randomiserat informationsexperiment riktat till populationen av potentiellt berättigade pensionärer. Ungefär var tionde pensionär som fick ett brev (behandlingsgruppen) ansökte om bostadstillägg inom fyra månader jämfört med drygt en av hundra som inte fick ett brev (kontrollgruppen). Andelen avslag i behandlingsgruppen var dock något högre.
|
|
| 5. |
- Engström, Per, et al.
(författare)
-
Demand effects of consumers' stated and revealed preferences
- 2013
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- Knowledge of how consumers react to different quality signals is fundamental for understanding how markets work. We study the online marketplace for Android apps where we compare the causal effects on demand from two quality related signals; other consumers' stated and revealed preferences toward an app. Our main result is that consumers are much more responsive to other consumers' revealed preferences, compared to others' stated preferences. A 10 percentile increase in displayed average rating only increases downloads by about 3 percent, while a 10 percentile increase in displayed number of downloads increases downloads by about 20 percent.
|
|
| 6. |
- Engström, Per, 1974-, et al.
(författare)
-
Demand effects of consumers' stated and revealed preferences
- 2018
-
Ingår i: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 150, s. 43-61
-
Tidskriftsartikel (refereegranskat)abstract
- Knowledge of how consumers react to different signals is fundamental to understanding how markets work. The modern electronic marketplace has revolutionized the possibilities for consumers to gather detailed information about products and services before purchase. Specifically, a consumer can easily – through a host of online forums and evaluation sites – estimate a product’s quality based on either (i) what other users say about the product (stated preferences) or (ii) how many other users that have bought the product (revealed preferences). In this paper, we compare the causal effects on demand from these two signals based on data from the biggest marketplace for Android apps, Google play. This data consists of daily information, for 42 consecutive days, of more than 500,000 apps from the US version of Google play. Our main result is that consumers are much more responsive to other consumers’ revealed preferences, compared to others’ stated preferences. A 10 percentile increase in displayed average rating only increases downloads by about 3%, while a 10 percentile increase in displayed number of downloads increases downloads by about 25%.
|
|
| 7. |
- Engström, Per, 1974-, et al.
(författare)
-
Increasing the take-up of the housing allowance among Swedish pensioners : a field experiment
- 2019
-
Ingår i: International Tax and Public Finance. - : SPRINGER. - 0927-5940 .- 1573-6970. ; 26:6, s. 1353-1382
-
Tidskriftsartikel (refereegranskat)abstract
- Using a randomized field experiment in the Swedish pension system, we investigate whether receiving an information letter affects the take-up rate of the housing allowance for pensioners. We also investigate whether the framing of the information letter affects take-up. The results show that simple information letters had a significant effect on the application rate and subsequent take-up rate: The baseline application rate in the targeted control population was only 1.4%, while the corresponding rates in the different treatment groups were between 9.9 and 12.1%. However, while the applications in the control group were accepted in almost 3 out of 4 cases, up to 50% of the applications in the treatment group were declined. The lower conditional acceptance rate in the treatment group seems to be largely driven by wealth, which the Pensions Agency cannot observe prior to submission. Information campaigns aimed at increasing benefit take-up therefore need careful design in situations with imperfect targeting.
|
|
| 8. |
- Forsell, Eskil
(författare)
-
Experimental testing of old and new hypotheses in Economics
- 2016
-
Doktorsavhandling (övrigt vetenskapligt/konstnärligt)abstract
- Chapter summaries Evaluating Replicability of Laboratory Experiments in Economics The reproducibility of scientific findings has been called into question. To contribute data about reproducibility in economics, we replicate 18 studies published in the American Economic Review and the Quarterly Journal of Economics in 2011-2014. All replications follow predefined analysis plans publicly posted prior to the replications, and have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We find a signifi- cant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs. Trading performance in prediction markets with different structures This paper presents preliminary evidence on how researchers in the field of psychology judge the replicability of the 28 effects replicated in the Many Labs 2 project. We use individual surveys in combination with prediction markets to elicit beliefs about two replication success metrics — whether the estimated effect in the replication study is statistically significant, and what the ratio between the original and replicated effect size is. We find that survey answers and final market prices are very highly correlated for the binary measure suggesting that the prediction markets provide little additional value, but that the correlation is lower for the effect size measure. x EXPERIMENTAL TESTING IN ECONOMICS The impact of decision rules on the predictive accuracy of decision markets An appealing prospect of prediction markets is that their estimates of how likely future events are to occur can be used as inputs when making a decision. As prediction markets used in this way help guide decisions, they are called decision markets. These have stricter requirements on how scoring rules (payment schemes) must be specified to guarantee that traders are incentivized to trade according to their beliefs. They also require that decision rules (the link between market outcomes and what decision is taken) is specified in certain ways. We let participants trade on hypothetical markets using three different combinations of the rules to explore how the predictive accuracy of the markets is affected. Our main finding is that the decision markets perform worse than traditional prediction markets — likely due to their increased complexity — but that there is little impact of the specific rules used. Gamelab: An online game-theory laboratory The Gamelab platform offers a novel and easy way to perform experiments in game theory. Its options are flexible enough to allow for a wide range of experiments. It is particularly well designed for play against anonymous and randomly drawn opponents. Thanks to its responsive design it can be used on almost any device with internet access. We here report the implementation of experiments in two different settings. In both settings, the subjects were given data about past aggregate play of the same game, thus giving them the possibility for social learning how to play. This platform thus provides a tool to test non-cooperative solution concepts. Demand effects of consumers’ stated and revealed preferences Knowledge of how consumers react to different signals is fundamental to understanding how markets work. The modern electronic marketplace has revolutionized the possibilities for consumers to gather detailed information about products and services before purchase. Specifically, a consumer can easily – through a host of online forums and evaluation sites – estimate a product’s quality based on either i) what other users say about the product (stated preferences) or ii) how many other users that have bought the product (revealed pref- xi erences). In this paper we compare the causal effects on demand from these two signals based on data from the biggest marketplace for Android apps, Google play. This data consists of daily information, for 42 consecutive days, of more than 500 000 apps from the US version of Google play. Our main result is that consumers are much more responsive to other consumers’ revealed preferences, compared to others’ stated preferences. A 10 percentile increase in displayed average rating only increases downloads by about 3 percent, while a 10 percentile increase in displayed number of downloads increases downloads by about 25 percent.
|
|
| 9. |
- Forsell, Eskil, et al.
(författare)
-
Predicting replication outcomes in the Many Labs 2 study
- 2019
-
Ingår i: Journal of Economic Psychology. - : Elsevier. - 1872-7719 .- 0167-4870. ; 75:Part A SI
-
Tidskriftsartikel (refereegranskat)abstract
- Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicited peer beliefs in prediction markets and surveys about two replication success metrics: the probability that the replication yields a statistically significant effect in the original direction (p < 0.001), and the relative effect size of the replication. The prediction markets correctly predicted 75% of the replication outcomes, and were highly correlated with the replication outcomes. Survey beliefs were also significantly correlated with replication outcomes, but had larger prediction errors. The prediction markets for relative effect sizes attracted little trading and thus did not work well. The survey beliefs about relative effect sizes performed better and were significantly correlated with observed relative effect sizes. The results suggest that replication outcomes can be predicted and that the elicitation of peer beliefs can increase our knowledge about scientific reproducibility and the dynamics of hypothesis testing.
|
|
