| 1. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 2. |
- Bishop, Michael, et al.
(författare)
-
Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme
- 2020
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 7:7
-
Tidskriftsartikel (refereegranskat)abstract
- The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
|
|
| 3. |
- Holzmeister, Felix, et al.
(författare)
-
Delegation Decisions in Finance
- 2020
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We run an online experiment with finance professionals and subjects from the general population (clients) to examine drivers and implications of clients' delegation decisions. We find that clients favor delegation to investment algorithms, followed by delegation to finance professionals with aligned incentives and lastly to those with fixed incentives. We also show that trust in investment algorithms or money managers (finance professionals), respectively, and clients' propensity to shift blame on others increases the likelihood of delegation, whereas own decision-making quality is associated with a decrease. In measuring the implications of clients' delegation decisions, we report high variability among finance professionals' perceptions of clients' preferred risk levels. We show that this results in overlaps in portfolio risk across risk-levels of clients, indicating problems of risk communication between clients and their money managers.
|
|
