| 1. |
- Menkveld, Albert J., et al.
(författare)
-
Nonstandard Errors
- 2024
-
Ingår i: JOURNAL OF FINANCE. - : Wiley-Blackwell. - 0022-1082 .- 1540-6261. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty-nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 2. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 3. |
- Benjamin, Daniel J., et al.
(författare)
-
Redefine statistical significance
- 2018
-
Ingår i: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 4. |
- Menkveld, Albert J., et al.
(författare)
-
Non-Standard Errors
- 2021
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- In statistics, samples are drawn from a population in a data generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.
|
|
| 5. |
- Menkveld, Albert J., et al.
(författare)
-
Non-Standard Errors
- 2024
-
Ingår i: Journal of Finance. - 0022-1082. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 6. |
- Menkveld, Albert J., et al.
(författare)
-
Nonstandard Errors
- 2024
-
Ingår i: Journal of Finance. - : Wiley-Blackwell. - 1540-6261 .- 0022-1082. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 7. |
- Pérignon, Christophe, et al.
(författare)
-
Reproducibility of Empirical Results : Evidence from 1,000 Tests in Finance
- 2022
-
Ingår i: SSRN Electronic Journal. - Paris : HEC Paris. - 1556-5068.
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Surprisingly, neither researcher seniority, nor the quality of the research paper seem related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research and underestimate the difficulty faced by their peers when attempting to reproduce their results. We further find that reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions and more complex code
|
|
| 8. |
- Aczel, Balazs, et al.
(författare)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
Ingår i: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 9. |
- Altmejd, Adam, et al.
(författare)
-
Predicting the replicability of social science lab experiments
- 2019
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Tidskriftsartikel (refereegranskat)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 10. |
- Brütt, Katharina, et al.
(författare)
-
Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs
- 2023
-
Ingår i: Proceedings of the National Academy of Sciences - PNAS. - : National Academy of Sciences. - 1091-6490 .- 0027-8424. ; 120:23
-
Tidskriftsartikel (refereegranskat)abstract
- Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity-variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity-estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs-indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.
|
|
| 11. |
- Camerer, C. F., et al.
(författare)
-
Evaluating replicability of laboratory experiments in economics
- 2016
-
Ingår i: Science. - : American Association for the Advancement of Science (AAAS). - 0036-8075 .- 1095-9203. ; 351:6280, s. 1433-1436
-
Tidskriftsartikel (refereegranskat)abstract
- The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that weremade publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.
|
|
| 12. |
- Camerer, C. F., et al.
(författare)
-
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
- 2018
-
Ingår i: Nature Human Behaviour. - : Springer Science and Business Media LLC. - 2397-3374. ; 2:9, s. 637-644
-
Tidskriftsartikel (refereegranskat)abstract
- Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated truepositive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
|
|
| 13. |
- Dijk, Oege, et al.
(författare)
-
Rank matters-The impact of social competition on portfolio choice
- 2014
-
Ingår i: European Economic Review. - : Elsevier BV. - 0014-2921 .- 1873-572X. ; 66, s. 97-110
-
Tidskriftsartikel (refereegranskat)abstract
- Tournament incentives' schemes have been criticized for inducing excessive risk-taking among financial market participants. In this paper we investigate how relative performance-based incentive schemes and status concerns for higher rank influence portfolio choice in laboratory experiments. We find that both underperformers and over-performers adapt their portfolios to their current relative performance, preferring either positively or negatively skewed assets, respectively. Most importantly, these results hold both when relative performance is instrumental for higher payoffs in a tournament and when it is only intrinsically motivating and not payout-relevant. We find no effects when no relative performance information is given. © 2013 Elsevier B.V.
|
|
| 14. |
|
|
| 15. |
- Fang, Dawei, 1983, et al.
(författare)
-
How tournament incentives affect asset markets: A comparison between winner-take-all tournaments and elimination contests
- 2017
-
Ingår i: Journal of Economic Dynamics and Control. - : Elsevier BV. - 0165-1889. ; 75, s. 1-27
-
Tidskriftsartikel (refereegranskat)abstract
- We investigate the impact of investment managers׳ tournament incentives on investment strategies and market efficiency, distinguishing between winner-take-all tournaments (WTA), where a minority wins, and elimination contests (EC), where a majority wins. Theoretically, we show that investment managers play heterogeneous strategies in WTA and homogeneous strategies in EC, and markets are more prone to mispricing in WTA than in EC. Experimentally, we find that investment managers play more heterogeneous strategies in WTA than in EC, but this does not trigger significant differences in prices. Moreover, prices in WTA and EC do not differ significantly from markets composed of linearly incentivized subjects.
|
|
| 16. |
- Farago, Adam, 1984, et al.
(författare)
-
Cognitive Skills and Economic Preferences in the Fund Industry*
- 2022
-
Ingår i: Economic Journal. - : Oxford University Press (OUP). - 0013-0133 .- 1468-0297. ; 132:645, s. 1737-1764
-
Tidskriftsartikel (refereegranskat)abstract
- We investigate the impact of cognitive skills and economic preferences on fund managers' professional decisions by running a battery of experiments with them. First, we find that fund managers' risk tolerance positively correlates with fund risk when accounting for fund benchmark, fund category and other controls. Second, we show that fund managers' ambiguity tolerance positively correlates with the funds' tracking error from the benchmark. Finally, we report that cognitive skills do not explain fund performance in terms of excess returns. However, we do find that fund managers with high cognitive reflection abilities compose funds at lower risk.
|
|
| 17. |
- Hanke, M., et al.
(författare)
-
Football championships and jersey sponsors' stock prices: an empirical investigation
- 2013
-
Ingår i: European Journal of Finance. - : Informa UK Limited. - 1351-847X .- 1466-4364. ; 19:3, s. 228-241
-
Tidskriftsartikel (refereegranskat)abstract
- Corporate sports sponsorship is an important part of many companies' corporate communication strategy. In this paper, we take the example of major football tournaments to show that sponsorship indeed affects the sponsor's (stock) market value. We find a statistically significant impact of football results (at an individual match level) of the seven most important football nations at European and World Championships on the stock prices of jersey sponsors. In general, the more important a match and the less expected its result, the higher its impact. In addition, we find a form of mere-exposure' effect which is difficult to reconcile with the efficient markets hypothesis.
|
|
| 18. |
|
|
| 19. |
- Holmén, Martin, 1966, et al.
(författare)
-
Do Option-like Incentives Induce Overvaluation? Evidence from Experimental Asset Markets
- 2012
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- One potential reason for bubbles evolving prior to the financial crisis was excessive risk taking stemming from option-like incentive schemes in financial institutions. By running laboratory asset markets, we investigate the impact of option-like incentives on price formation and trading behavior. We observe (i) that option-like incentives induce significantly higher market prices than linear incentives. We further find that (ii) option-like incentives provoke subjects to behave differently and to take more risk than subjects with linear incentives. We finally show that (iii) trading at inflated prices is rational for subjects with option-like incentives since it increases their expected payout.
|
|
| 20. |
- Holmén, Martin, 1966, et al.
(författare)
-
Do option-like incentives induce overvaluation? Evidence from experimental asset markets
- 2014
-
Ingår i: Journal of Economic Dynamics and Control. - : Elsevier BV. - 0165-1889. ; 40, s. 179-194
-
Tidskriftsartikel (refereegranskat)abstract
- One potential reason for bubbles evolving prior to the financial crisis was excessive risk taking stemming from option-like incentive schemes in financial institutions. By running laboratory asset markets, we investigate the impact of option-like incentives on price formation and trading behavior. The main results are that (i) we observe significantly higher market prices with option-like incentives than linear incentives. (ii) We further find that option-like incentives provoke subjects to behave differently and to take more risk than subjects with linear incentives. (iii) We finally show that trading at inflated prices is rational for subjects with option-like incentives since it increases their expected payout. © 2014 Elsevier B.V.
|
|
| 21. |
- Holmen, Martin, 1976, et al.
(författare)
-
Economic Preferences and Personality Traits Among Finance Professionals and the General Population
- 2023
-
Ingår i: Economic Journal. - 0013-0133 .- 1468-0297. ; 133:656
-
Tidskriftsartikel (refereegranskat)abstract
- Based on artefactual field experiments, we investigate whether finance professionals differ from a sample of the working population in terms of industry-relevant preferences and personality traits. When adjusting for socioeconomic characteristics, we find only few and less marked differences: finance professionals are less risk averse, less trustworthy, show higher levels of psychopathy and are more competitive than participants from the general population. In an additional survey, experts with hiring experience consider industry selection, self-selection and imprinting by industry norms as explanatory for the observed subject pool differences.
|
|
| 22. |
- Holzmeister, Felix, et al.
(författare)
-
Delegation Decisions in Finance
- 2020
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We run an online experiment with finance professionals and subjects from the general population (clients) to examine drivers and implications of clients' delegation decisions. We find that clients favor delegation to investment algorithms, followed by delegation to finance professionals with aligned incentives and lastly to those with fixed incentives. We also show that trust in investment algorithms or money managers (finance professionals), respectively, and clients' propensity to shift blame on others increases the likelihood of delegation, whereas own decision-making quality is associated with a decrease. In measuring the implications of clients' delegation decisions, we report high variability among finance professionals' perceptions of clients' preferred risk levels. We show that this results in overlaps in portfolio risk across risk-levels of clients, indicating problems of risk communication between clients and their money managers.
|
|
| 23. |
- Holzmeister, Felix, et al.
(författare)
-
Delegation Decisions in Finance
- 2023
-
Ingår i: Management Science. - : Institute for Operations Research and the Management Sciences (INFORMS). - 0025-1909 .- 1526-5501. ; 69:8, s. 4828-4844
-
Tidskriftsartikel (refereegranskat)abstract
- Based on an online experimentwith a sample of finance professionals and participants from the general population (acting as clients), we examine drivers and motives of clients' choices to delegate investment decisions to agents.We find that clients favor delegation to investment algorithms, followed by delegation to finance professionals compensated with an aligned incentive scheme, and lastly to finance professionals receiving a fixed payment for investing on behalf of others. We show that trust in investment algorithms or finance professionals, and clients' propensity to shift blame on others increase the likelihood of delegation, whereas clients' own decision-making quality is associated with a decrease in delegation frequency.
|
|
| 24. |
- Holzmeister, Felix, et al.
(författare)
-
Heterogeneity in effect size estimates : Empirical evidence and practical implications
- 2023
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confidence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confidence intervals. We illustrate that heterogeneity of this magnitude — unless properly accounted for —has severe implications for statistical inference with strongly increased rates of false scientific claims.
|
|
| 25. |
- Huber, Jürgen, et al.
(författare)
-
Experimental asset markets with endogenous choice of costly asymmetric information
- 2011
-
Ingår i: Experimental Economics. - 1386-4157. ; 14:2, s. 223-240
-
Tidskriftsartikel (refereegranskat)abstract
- Asymmetric distribution of information, while omnipresent in real markets, is rarely considered in experimental financial markets. We present results from experiments where subjects endogenously choose between five information levels (four of them costly). We find that (i) uninformed traders earn the highest net returns, while average informed traders always perform worst even when information costs are not considered; (ii) over time traders learn to pick the most advantageous information levels (full information or no information); and (iii) market efficiency decreases with higher information costs. These results are mostly in line with the theoretical predictions of Grossman and Stiglitz (Am. Econ. Rev. 70:393–408, 1980) and provide additional insights that studies with only two information levels cannot deliver.
|
|
