| 1. |
- Menkveld, Albert J., et al.
(författare)
-
Nonstandard Errors
- 2024
-
Ingår i: JOURNAL OF FINANCE. - : Wiley-Blackwell. - 0022-1082 .- 1540-6261. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty-nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 2. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 3. |
- Menkveld, Albert J., et al.
(författare)
-
Non-Standard Errors
- 2021
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- In statistics, samples are drawn from a population in a data generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.
|
|
| 4. |
- Menkveld, Albert J., et al.
(författare)
-
Non-Standard Errors
- 2024
-
Ingår i: Journal of Finance. - 0022-1082. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 5. |
- Menkveld, Albert J., et al.
(författare)
-
Nonstandard Errors
- 2024
-
Ingår i: Journal of Finance. - : Wiley-Blackwell. - 1540-6261 .- 0022-1082. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty—nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 6. |
- Perignon, Christophe, et al.
(författare)
-
Computational Reproducibility in Finance : Evidence from 1,000 Tests
- 2024
-
Ingår i: Review of Financial Studies. - : Oxford Univ Press. - 1465-7368 .- 0893-9454.
-
Tidskriftsartikel (refereegranskat)abstract
- We analyze the computational reproducibility of more than 1,000 empirical answers to 6 research questions in finance provided by 168 research teams. Running the researchers' code on the same raw data regenerates exactly the same results only 52% of the time. Reproducibility is higher for researchers with better coding skills and those exerting more effort. It is lower for more technical research questions, more complex code, and results lying in the tails of the distribution. Researchers exhibit overconfidence when assessing the reproducibility of their own research. We provide guidelines for finance researchers and discuss implementable reproducibility policies for academic journals.
|
|
| 7. |
- Pérignon, Christophe, et al.
(författare)
-
Reproducibility of Empirical Results : Evidence from 1,000 Tests in Finance
- 2022
-
Ingår i: SSRN Electronic Journal. - Paris : HEC Paris. - 1556-5068.
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We analyze the computational reproducibility of more than 1,000 empirical answers to six research questions in finance provided by 168 international research teams. Surprisingly, neither researcher seniority, nor the quality of the research paper seem related to the level of reproducibility. Moreover, researchers exhibit strong overconfidence when assessing the reproducibility of their own research and underestimate the difficulty faced by their peers when attempting to reproduce their results. We further find that reproducibility is higher for researchers with better coding skills and for those exerting more effort. It is lower for more technical research questions and more complex code
|
|
| 8. |
- Aczel, Balazs, et al.
(författare)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
Ingår i: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 9. |
- Brütt, Katharina, et al.
(författare)
-
Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs
- 2023
-
Ingår i: Proceedings of the National Academy of Sciences - PNAS. - : National Academy of Sciences. - 1091-6490 .- 0027-8424. ; 120:23
-
Tidskriftsartikel (refereegranskat)abstract
- Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity-variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity-estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs-indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.
|
|
| 10. |
- Farago, Adam, 1984, et al.
(författare)
-
Cognitive Skills and Economic Preferences in the Fund Industry*
- 2022
-
Ingår i: Economic Journal. - : Oxford University Press (OUP). - 0013-0133 .- 1468-0297. ; 132:645, s. 1737-1764
-
Tidskriftsartikel (refereegranskat)abstract
- We investigate the impact of cognitive skills and economic preferences on fund managers' professional decisions by running a battery of experiments with them. First, we find that fund managers' risk tolerance positively correlates with fund risk when accounting for fund benchmark, fund category and other controls. Second, we show that fund managers' ambiguity tolerance positively correlates with the funds' tracking error from the benchmark. Finally, we report that cognitive skills do not explain fund performance in terms of excess returns. However, we do find that fund managers with high cognitive reflection abilities compose funds at lower risk.
|
|
| 11. |
- Holmen, Martin, 1976, et al.
(författare)
-
Economic Preferences and Personality Traits Among Finance Professionals and the General Population
- 2023
-
Ingår i: Economic Journal. - 0013-0133 .- 1468-0297. ; 133:656
-
Tidskriftsartikel (refereegranskat)abstract
- Based on artefactual field experiments, we investigate whether finance professionals differ from a sample of the working population in terms of industry-relevant preferences and personality traits. When adjusting for socioeconomic characteristics, we find only few and less marked differences: finance professionals are less risk averse, less trustworthy, show higher levels of psychopathy and are more competitive than participants from the general population. In an additional survey, experts with hiring experience consider industry selection, self-selection and imprinting by industry norms as explanatory for the observed subject pool differences.
|
|
| 12. |
- Holzmeister, Felix, et al.
(författare)
-
Delegation Decisions in Finance
- 2020
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We run an online experiment with finance professionals and subjects from the general population (clients) to examine drivers and implications of clients' delegation decisions. We find that clients favor delegation to investment algorithms, followed by delegation to finance professionals with aligned incentives and lastly to those with fixed incentives. We also show that trust in investment algorithms or money managers (finance professionals), respectively, and clients' propensity to shift blame on others increases the likelihood of delegation, whereas own decision-making quality is associated with a decrease. In measuring the implications of clients' delegation decisions, we report high variability among finance professionals' perceptions of clients' preferred risk levels. We show that this results in overlaps in portfolio risk across risk-levels of clients, indicating problems of risk communication between clients and their money managers.
|
|
| 13. |
- Holzmeister, Felix, et al.
(författare)
-
Delegation Decisions in Finance
- 2023
-
Ingår i: Management Science. - : Institute for Operations Research and the Management Sciences (INFORMS). - 0025-1909 .- 1526-5501. ; 69:8, s. 4828-4844
-
Tidskriftsartikel (refereegranskat)abstract
- Based on an online experimentwith a sample of finance professionals and participants from the general population (acting as clients), we examine drivers and motives of clients' choices to delegate investment decisions to agents.We find that clients favor delegation to investment algorithms, followed by delegation to finance professionals compensated with an aligned incentive scheme, and lastly to finance professionals receiving a fixed payment for investing on behalf of others. We show that trust in investment algorithms or finance professionals, and clients' propensity to shift blame on others increase the likelihood of delegation, whereas clients' own decision-making quality is associated with a decrease in delegation frequency.
|
|
| 14. |
- Holzmeister, Felix, et al.
(författare)
-
Heterogeneity in effect size estimates : Empirical evidence and practical implications
- 2023
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty not accounted for in reported standard errors and confidence intervals. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population heterogeneity, design heterogeneity, and analytical heterogeneity. We estimate each type's heterogeneity from multi-lab replication studies, prospective meta-analyses of studies varying experimental designs, and multi-analyst studies. Our results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the estimates suggests that incorporating the uncertainty due to heterogeneity would approximately double sample standard errors and confidence intervals. We illustrate that heterogeneity of this magnitude — unless properly accounted for —has severe implications for statistical inference with strongly increased rates of false scientific claims.
|
|
| 15. |
- Huntington-Klein, Nick, et al.
(författare)
-
Subjective evidence evaluation survey for many-analysts studies
- 2024
-
Ingår i: Royal Society Open Science. - : The Royal Society. - 2054-5703 .- 2054-5703. ; 11:7
-
Tidskriftsartikel (refereegranskat)abstract
- Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
|
|
| 16. |
- Johannesson, Magnus, et al.
(författare)
-
Heterogeneity in effect size estimates
- 2024
-
Ingår i: Proceedings of the National Academy of Sciences - PNAS. - : The National Academy of Sciences. - 1091-6490 .- 0027-8424. ; 121:32
-
Tidskriftsartikel (refereegranskat)abstract
- A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
|
|
| 17. |
- Kirchler, Michael, 1977, et al.
(författare)
-
Delegated investment decisions and rankings
- 2020
-
Ingår i: Journal of Banking & Finance. - : Elsevier BV. - 0378-4266. ; 120
-
Tidskriftsartikel (refereegranskat)abstract
- Two aspects of social context are central to the finance industry. First, financial professionals usually make investment decisions on behalf of third parties. Second, social competition, in the form of performance rankings, is pervasive. Therefore, we investigate professionals' risk taking behavior under social competition when investing for others. We run online and lab-in-the-field experiments with 805 financial professionals and show that professionals increase their risk taking for others when they lag behind. Additional survey evidence from 1349 respondents reveals that professionals' preferences for high rankings are significantly stronger than those of the general population. (C) 2020 The Authors. Published by Elsevier B.V.
|
|
| 18. |
- Razen, M., et al.
(författare)
-
Domain-specific risk-taking among finance professionals
- 2020
-
Ingår i: Journal of Behavioral and Experimental Finance. - : Elsevier BV. - 2214-6350. ; 27
-
Tidskriftsartikel (refereegranskat)abstract
- Risk-assessment and risk-taking in various forms are among the most important tasks financial professionals face in their daily work. A large body of experimental studies has shown a substantial effect of the decision domain (gain vs loss domain) on risk-taking, predominantly among students. In a series of experiments set in different contextual frameworks, we investigate whether this domain effect is also present among experienced employees in the finance industry and compare their decisions with people from the general population. Our results show that employees in the finance industry are equally prone to the domain effect in risk-taking than the general population. Interestingly, for domain-specific risk-taking in a finance context, we find that professionals are even more reluctant to sell loser stocks than non-professionals. © 2020 The Authors
|
|
| 19. |
- Schwaiger, R., et al.
(författare)
-
Determinants of investor expectations and satisfaction. A study with financial professionals
- 2020
-
Ingår i: Journal of Economic Dynamics and Control. - : Elsevier BV. - 0165-1889. ; 110
-
Tidskriftsartikel (refereegranskat)abstract
- We investigate determinants of price expectations and satisfaction levels of financial professionals and students. In experiments with 150 professionals and 576 students, we systematically vary price paths according to the final return (positive or negative) and the way in which the final return is achieved (upswing followed by downswing or vice versa). Professionals show the most optimistic price expectations and are most satisfied if assets fall in price first and then recover. In addition, professionals’ price expectations are highest after positive returns. Among students, qualitatively similar patterns emerge, but professionals’ price expectations are less prone to framing effects. © 2019 The Author(s)
|
|
| 20. |
- Sutter, M., et al.
(författare)
-
Where to look for the morals in markets?
- 2020
-
Ingår i: Experimental Economics. - : Springer Science and Business Media LLC. - 1386-4157 .- 1573-6938. ; 23:1, s. 30-52
-
Tidskriftsartikel (refereegranskat)abstract
- There is a heated debate on whether markets erode social responsibility and moral behavior. However, it is a challenging task to identify and measure moral behavior in markets. Based on a theoretical model, we examine in an experiment the relation between trading volume, prices and moral behavior by setting up markets that either impose a negative externality on third parties or not. We find that moral behavior reveals itself in lower trading volume in markets with a negative externality, while prices mostly depend on the market structure. We further investigate individual characteristics that explain trading behavior in markets with negative externalities.
|
|
| 21. |
- Uhlmann, Eric, L., et al.
(författare)
-
Subjective Evidence Evaluation Survey For Multi-Analyst Studies
- 2024
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Multi-analyst studies explore how well an empirical claim withstands plausible alternative analyses of the same data set by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g., effect size) provided by each analysis team. Although informative about the range of plausible effects in a data set, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item Subjective Evidence Evaluation Survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous multi-analyst study.
|
|
