| 1. |
- Silberzahn, Raphael, et al.
(författare)
-
Many analysts, one dataset : Making transparent how variations in analytical choices affect results
- 2018
-
Ingår i: Advances in Methods and Practices in Psychological Science. - : Sage Publications. - 2515-2459 .- 2515-2467. ; 1:3, s. 337-356
-
Tidskriftsartikel (refereegranskat)abstract
- Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a non-significant relationship. Overall 29 differentanalyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.
|
|
| 2. |
- Uhlmann, Eric, L., et al.
(författare)
-
Subjective Evidence Evaluation Survey For Multi-Analyst Studies
- 2024
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Multi-analyst studies explore how well an empirical claim withstands plausible alternative analyses of the same data set by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g., effect size) provided by each analysis team. Although informative about the range of plausible effects in a data set, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item Subjective Evidence Evaluation Survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous multi-analyst study.
|
|
| 3. |
- Aczel, Balazs, et al.
(författare)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
Ingår i: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 4. |
- Graham, Jesse R., et al.
(författare)
-
The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline
- 2016
-
Ingår i: Journal of Experimental Social Psychology. - : Elsevier. - 1096-0465 .- 0022-1031. ; 66, s. 55-67
-
Tidskriftsartikel (refereegranskat)abstract
- This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.
|
|
| 5. |
- Washburn, Anthony N., et al.
(författare)
-
Data from a pre-publication independent replication initiative examining ten moral judgement effects
- 2016
-
Ingår i: Scientific Data. - : Nature Research (part of Springer Nature): Fully open access journals / Nature Publishing Group. - 2052-4463. ; 3
-
Tidskriftsartikel (refereegranskat)abstract
- We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory's research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires.
|
|
| 6. |
- Benjamin, Daniel J., et al.
(författare)
-
Redefine statistical significance
- 2018
-
Ingår i: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 7. |
- Nilsson, Håkan, et al.
(författare)
-
Hierarchical Bayesian parameter estimation for cumulative prospect theory
- 2011
-
Ingår i: Journal of mathematical psychology (Print). - : Elsevier BV. - 0022-2496 .- 1096-0880. ; 55:1, s. 84-93
-
Tidskriftsartikel (refereegranskat)abstract
- Cumulative prospect theory (CPT Tversky & Kahneman, 1992) has provided one of the most influential accounts of how people make decisions under risk. CPT is a formal model with parameters that quantify psychological processes such as loss aversion, subjective values of gains and losses, and subjective probabilities. In practical applications of CPT, the model's parameters are usually estimated using a single-participant maximum likelihood approach. The present study shows the advantages of an alternative, hierarchical Bayesian parameter estimation procedure. Performance of the procedure is illustrated with a parameter recovery study and application to a real data set. The work reveals that without particular constraints on the parameter space, CPT can produce loss aversion without the parameter that has traditionally been associated with loss aversion. In general, the results illustrate that inferences about people's decision processes can crucially depend on the method used to estimate model parameters.
|
|
| 8. |
- Nilsson, Håkan, 1976-, et al.
(författare)
-
Hierarchical Bayesian parameter estimation for models of decision under uncertainty.
- 2011
-
Konferensbidrag (refereegranskat)abstract
- Cognitive modeling has long been the key tool in research on decision under uncertainty. Cognitive modeling has the benefits of generating exact model-predictions and providing the opportunity of rigorous model comparison tests. Cognitive modeling is not, however, without problems. This paper focuses on one of these, namely the problem of how to generate reliable estimates of a model´s free parameters.Traditionally, models are fitted to data at the individual level and parameter estimates are retrieved using maximum likelihood (ML) methods. A downside of this procedure is that it, due to the fact that it is susceptible to noise, tends to exaggerate individual differences. An alternative approach is to use hierarchical Bayesian (HB) parameter estimation. The approach is hierarchical because it uses models with several levels. In the simplest case (the case studied in this paper), the value of parameter x for individual i (individual level) is assumed to be sampled from a normally distributed hyper distribution with parameters m and s (the hyper level). The individual x-values and parameters m and s are estimated from individual data simultaneously. Importantly, the HB approach is less susceptible to noise because individual x-values are constrained by the hyper distribution.The present paper: The aim was to explore the benefits of the HB approach. The strategy was to estimate the parameters of cumulative prospect theory (CPT) with both the ML and the HB approach and compare the results. For clarity, CPT has five free parameters. The three that are most important here are α (quantifies the curvature of the value function for gains), β (quantifies the curvature of the value function for losses) and λ (quantifies the amount of loss aversion).Study 1: Study 1 was a parameter recovery study. Synthetic data was created as follows. A deterministic version of CPT, equipped with the parameter estimates from Tversky and Kahneman (1992; Journal of Risk and Uncertainty), generated choice-predictions for the 180 gamble-pairs used in Rieskamp (2008; JEP:LMC). Data sets with choices from 30 synthetic participants were created by adding noise to these predictions. The goal of Study 1 was to explore if the two approaches would be equally good at recovering the underlying parameters. Study 1 provided three key findings. (1) Overall, the medians for the ML-estimates corresponded well with the medians of the HB-estimates. (2) The HB approach was superior at filtering out noise. (3) Regardless of fitting approach, loss aversion was mainly captured by separating α and β so that α < β. As a result, λ was systematically underestimated. When CPT was constrained so that α = β, the underestimation was strongly reduced for ML-estimates and neutralized for HB-estimates.Study 2. CPT was fitted to the behavioral data from Rieskamp (2008). Study 2 replicated the main findings from Study 1. In addition, study 2 provided two finding regarding λ. (1) Neither the median ML-estimate nor the median HB-estimate indicated systematic loss aversion. (2) Extreme ML-estimates for λ can be caused by two very different factors, systematic loss aversion and noise. This was shown by the fact the 6 participants that received the most extreme ML-estimates all received HB-estimates very close to the mean of the hyper distribution for λ.Summary. Parameter λ is the weak-spot of CPT. In particular, it is systematically underestimated if α and β are allowed to vary freely and noisy data tend to attract extreme ML-estimates. On the general level, as it generates both more information and more stable parameter estimates, we conclude that HB parameter estimation has the potential of vitalizing the literature on decisions under uncertainty.
|
|
| 9. |
|
|
