| 1. |
- Ebersole, Charles R., et al.
(author)
-
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
- 2020
-
In: Advances in Methods and Practices in Psychological Science. - : Sage. - 2515-2467 .- 2515-2459. ; 3:3, s. 309-331
-
Journal article (peer-reviewed)abstract
- Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).
|
|
| 2. |
|
|
| 3. |
- Bonnier, Evelina, et al.
(author)
-
Exposure to half-dressed women and economic behavior
- 2019
-
In: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 168, s. 393-418
-
Journal article (peer-reviewed)abstract
- Images of half-dressed women are ubiquitous in advertising and popular culture. Yet little is known about the potential impacts of such images on economic decision making. We randomize 648 participants of both genders to advertising images including either women in bikini or underwear, fully dressed women, or no women, and examine the effects on risk taking, willingness to compete and math performance in a lab experiment. We find no treatment effects on any outcome measure for women. For men, our results indicate that men take more risk after having been exposed to images of half-dressed women compared to no women.
|
|
| 4. |
- Botvinik-Nezer, Rotem, et al.
(author)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
In: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Journal article (peer-reviewed)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 5. |
- Menkveld, Albert J., et al.
(author)
-
Nonstandard Errors
- 2024
-
In: JOURNAL OF FINANCE. - : Wiley-Blackwell. - 0022-1082 .- 1540-6261. ; 79:3, s. 2339-2390
-
Journal article (peer-reviewed)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty-nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 6. |
- Aczel, Balazs, et al.
(author)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
In: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Journal article (peer-reviewed)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 7. |
- Almenberg, Johan, et al.
(author)
-
Do More Expensive Wines Taste Better? Evidence from a Large Sample of Blind Tastings
- 2018
-
In: World Scientific Reference on Handbook of the Economics of Wine, Volume 1: Prices, Finance, and Expert Opinion. - : WORLD SCIENTIFIC. - 2010-1732. - 9789813232730 ; , s. 535-545
-
Book chapter (other academic/artistic)abstract
- Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. In a sample of more than 6,000 blind tastings, we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less. For individuals with wine training, however, we find indications of a non-negative relationship between price and enjoyment. Our results are robust to the inclusion of individual fixed effects, and are not driven by outliers: when omitting the top and bottom deciles of the price distribution, our qualitative results are strengthened, and the statistical significance is improved further. These findings suggest that non-expert wine consumers should not anticipate greater enjoyment of the intrinsic qualities of a wine simply because it is expensive or is appreciated by experts.
|
|
| 8. |
|
|
| 9. |
|
|
| 10. |
- Altmejd, Adam, et al.
(author)
-
Predicting the replicability of social science lab experiments
- 2019
-
In: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Journal article (peer-reviewed)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 11. |
- Anderson, Anders, et al.
(author)
-
Risk taking, behavioral biases and genes : Results from 149 active investors
- 2015
-
In: Journal of Behavioral and Experimental Finance. - : Elsevier BV. - 2214-6350 .- 2214-6369. ; 6, s. 93-100
-
Journal article (peer-reviewed)abstract
- Recent evidence suggests that there is genetic basis for economic behaviors, including preferences for risk taking. We correlate variation in risk taking and behavioral biases with two genetic polymorphisms related to the uptake of dopamine and serotonin (7R+ DRD4 and s/s 5-HTTLPR), hypothesizing that they are positively (negatively) related to risk taking. We use a small but detailed sample of active investors where we combine survey data with DNA samples and data from Swedish tax records that give us objective information about actual economic choices. We find a positive (negative) relationship between the dopamine (serotonin) gene and life expectancy bias, but no other significant correlations between the two genes and behaviors, including risk taking and measures of equity holdings. We acknowledge that our tests suffer from low power originating from the small sample size, which warrants some caution when interpreting these results.
|
|
| 12. |
- Ankel-Peters, Jörg, et al.
(author)
-
A Protocol for Structured Robustness Reproductions and Replicability Assessments
- 2024
-
Other publication (other academic/artistic)abstract
- Robustness reproductions and replicability discussions are on the rise in response to concerns about a potential credibility crisis in economics. This paper proposes a protocol to structure reproducibility and replicability assessments, with a focus on robustness. Starting with a computational reproduction upon data availability, the protocol encourages replicators to prespecify robustness tests, prior to implementing them. The protocol contains three different reporting tools to streamline the presentation of results. Beyond reproductions, our protocol assesses adherence to the pre-analysis plans in the replicated papers as well as external and construct validity. Our ambition is to put often controversial debates between replicators and replicated authors on a solid basis and contribute to an improved replication culture in economics.
|
|
| 13. |
- Apicella, Coren L, et al.
(author)
-
Salivary testosterone change following monetary wins and losses predicts future financial risk-taking
- 2014
-
In: Psychoneuroendocrinology. - : Elsevier BV. - 0306-4530. ; 39:1, s. 58-64
-
Journal article (peer-reviewed)abstract
- While baseline testosterone has recently been implicated in risk-taking in men, less is known about the effects of changing levels of testosterone on financial risk. Here we attempt to influence testosterone in men by having them win or lose money in a chance-based competition against another male opponent. We employ two treatments where we vary the amount of money at stake so that we can directly compare winners to losers who earn the same amount, thereby abstracting from income effects. We find that men who experience a greater increase in bioactive testosterone take on more risk, an association that remains when controlling for whether the participant won the competition. In fact, whether subjects won the competition did not predict future risk. These results suggest that testosterone change, and thus individual differences in testosterone reactivity, rather than the act of winning or losing, influence financial risk-taking. © 2013.
|
|
| 14. |
- Apicella, Coren L, et al.
(author)
-
Sex Differences in Competitiveness: Hunter-Gatherer Women and Girls Compete Less in Gender-Neutral and Male-Centric Tasks
- 2015
-
In: Adaptive Human Behavior and Physiology. - : Springer (part of Springer Nature): Springer Open Choice Hybrid Journals. - 2198-7335 .- 2198-7335. ; 1:3, s. 247-269
-
Journal article (peer-reviewed)abstract
- Despite numerous attempts to increase workplace equality, the near universal gender wage gap and underrepresentation of women in high status jobs persists in societies around the world. This persistence has led some researchers to speculate that psychological sex differences may be partly to blame. In particular, economists have begun to focus on sex differences in competitiveness as a possible cause. Here we test whether sex differences in competitiveness exist in a relatively isolated and evolutionarily relevant population of hunter-gatherers in Tanzania. In study 1 we examine sex differences in willingness to compete in a gender-neutral task in Hadza adults and children (N = 191). We find that when choosing between an individualistic payment scheme and a competitive payment scheme, boys and men are significantly more likely to compete than girls and women. We find no evidence that this sex difference varies with age. In study 2 (N = 88 and N = 70) we use both a female-centric and a male-centric task to explore sex differences in competitiveness in adults. While we find no sex difference in willingness to compete in the female-centric task, we find that men are more likely to compete in the male-centric task. While further work is needed, this study lends some support to the idea of a sex difference in willingness to compete among hunter-gatherers, but it also highlights the importance of the task type. The observation that a sizable proportion of male Hadza choose to compete in each of the tasks is discussed in light of the fact that hunter-gatherers are largely egalitarian and non-hierarchical.
|
|
| 15. |
- Apicella, Coren L, et al.
(author)
-
Testosterone and Economic Risk Taking: A Review
- 2015
-
In: Adaptive Human Behavior and Physiology. - : Springer (part of Springer Nature): Springer Open Choice Hybrid Journals. - 2198-7335 .- 2198-7335. ; 1:3, s. 358-385
-
Journal article (peer-reviewed)abstract
- Since precise forecasting of the future is not possible, most of life’s decisions are made with uncertain outcomes. One important facet of uncertainty that is of particular interest to decision scientists is risk—the choice between an option that is less rewarding but more certain and an option that is less certain, but potentially more rewarding. Recent developments in both neuroscience and behavioral endocrinology have helped to reveal the biological mechanisms that support decision-making involving economic risk, and consequently, potential factors associated with individual differences in risk taking. This review is dedicated to surveying recent developments that link the hormone testosterone to economic risk taking. Like neuroeconomics, endocrinological approaches may provide a potentially powerful framework from which to understand decision-making and may help to make sense of a number of well-documented behavioral anomalies involving economic risk. Specifically, we suggest that testosterone functions to modulate risky behaviors in ways that appear to be adaptive. Still, more work is needed to understand the nature of the relationship between testosterone and risk in both sexes.
|
|
| 16. |
|
|
| 17. |
- Arechar, Antonio A., et al.
(author)
-
“I'm just a soul whose intentions are good”: The role of communication in noisy repeated games
- 2017
-
In: Games and Economic Behavior. - : Elsevier. - 1090-2473 .- 0899-8256. ; 104, s. 726-743
-
Journal article (peer-reviewed)abstract
- We let participants indicate their intended action in a repeated game experiment where actions are implemented with errors. Even though communication is cheap talk, we find that the majority of messages were honest (although the majority of participants lied at least occasionally). As a result, communication has a positive effect on cooperation when the payoff matrix makes the returns to cooperation high; when the payoff matrix gives a lower return to cooperation, communication reduces overall cooperation. These results suggest that cheap talk communication can promote cooperation in repeated games, but only when there is already a self-interested motivation to cooperate.
|
|
| 18. |
- Benjamin, Daniel J., et al.
(author)
-
Redefine statistical significance
- 2018
-
In: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Journal article (other academic/artistic)
|
|
| 19. |
- Bernard, Mark, et al.
(author)
-
The subgroup problem : When can binding voting on extractions from a common pool resource overcome the tragedy of the commons?
- 2013
-
In: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 91, s. 122-130
-
Journal article (peer-reviewed)abstract
- Using a common pool resource game protocol with voting we examine experimentally how cooperation varies with the level at which (binding) votes are aggregated. Our results are broadly in line with theoretical predictions. When players can vote on the behavior of the whole group or when leaders from each group can vote for the group as a whole, extraction levels from the common resource pool are close to the social optimum. When players extract resources individually, there is substantial overextraction. When players vote in subgroups, there is initially less overextraction but it increases over time. This suggests that in order for binding voting to overcome the tragedy of the commons in social dilemmas, it should ideally affect the group as a whole.
|
|
| 20. |
- Bernard, Mark, et al.
(author)
-
The subgroup problem: When can binding voting on extractions from a common resource pool overcome the tragedy of the commons?
- 2013
-
In: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 91, s. 122-130
-
Journal article (peer-reviewed)abstract
- Using a common pool resource game protocol with voting we examine experimentally how cooperation varies with the level at which (binding) votes are aggregated. Our results are broadly in line with theoretical predictions. When players can vote on the behavior of the whole group or when leaders from each group can vote for the group as a whole, extraction levels from the common resource pool are close to the social optimum. When players extract resources individually, there is substantial overextraction. When players vote in subgroups, there is initially less overextraction but it increases over time. This suggests that in order for binding voting to overcome the tragedy of the commons in social dilemmas, it should ideally affect the group as a whole.
|
|
| 21. |
- Bilén, David, et al.
(author)
-
Are women more generous than men? A meta-analysis
- 2021
-
In: Journal of the Economic Science Association-Jesa. - : Springer Science and Business Media LLC. - 2199-6776 .- 2199-6784. ; 7, s. 1-18
-
Journal article (peer-reviewed)abstract
- We perform a meta analysis of gender differences in the standard windfall gains dictator game (DG) by collecting raw data from 53 studies with 117 conditions, giving us 15,016 unique individual observations. We find that women on average give 4 percentage points more than men (Cohen's d = 0.16), and that this difference decreases to 3.1% points (Cohen's d = 0.13) if we exclude studies where dictators can only give all or nothing. The gender difference is larger if the recipient in the DG is a charity, compared to the standard DG with an anonymous individual as the recipient (a 10.9 versus a 2.3% points gender difference). These effect sizes imply that many individual studies on gender differences are underpowered; the median power in our sample of standard DG studies is only 9% to detect the meta-analytic gender difference at the 5% significance level. Moving forward on this topic, sample sizes should thus be substantially larger than what has been the norm in the past.
|
|
| 22. |
- Bishop, Michael, et al.
(author)
-
Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme
- 2020
-
In: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 7:7
-
Journal article (peer-reviewed)abstract
- The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
|
|
| 23. |
- Boschini, Anne, et al.
(author)
-
Gender and altruism in a random sample
- 2018
-
In: Journal of Behavioral and Experimental Economics. - : Elsevier BV. - 2214-8043 .- 2214-8051. ; 77, s. 72-77
-
Journal article (peer-reviewed)abstract
- We study gender differences in altruism in a large random sample of the Swedish population using a standard dictator game. Beside a baseline treatment we implement a priming treatment where participants are reminded of their gender, and two treatments with known male and female counterpart respectively. We find suggestive evidence that women are more altruistic than men only in the priming treatment. A post-hoc analysis using data on interviewer gender to explore gender context effects indicates that priming affects behavior only in mixedgender contexts.
|
|
| 24. |
- Boschini, Anne, et al.
(author)
-
Gender, risk preference and willingness to compete in a random sample of the Swedish population
-
Other publication (other academic/artistic)abstract
- Working paper.Experimental results from student or other non-representative convenience samples often suggest that men, on average, are more risk-taking and competitive than women. Here we explore whether these gender preference gaps also exist in a simple random sample of the Swedish adult population. Our design comprises four different treatments to systematically explore how the experimental context may impact gender gaps; a baseline treatment, a treatment where participants are primed with their own gender, and a treatment where the participants know the gender of their counterpart (man or woman). We look at willingness to compete in two domains: a math task and a verbal task. We find no gender differences in risk preferences or in willingness to compete in the verbal task in this random sample. There is some support for men being more competitive than women in the math task, in particular in the pooled sample. The effect size is however considerably smaller than what is typically found. We further find no consistent impact of treatment on (the absence of) the gender gap in preferences.
|
|
| 25. |
- Boschini, Anne, et al.
(author)
-
Gender, risk preferences and willingness to compete in a random sample of the Swedish population
- 2018
-
Reports (other academic/artistic)abstract
- Experimental results from student or other non-representative convenience samples often suggest that men, on average, are more risk-taking and competitive than women. Here we explore whether these gender preference gaps also exist in a simple random sample of the Swedish adult population. Our design comprises four different treatments to systematically explore how the experimental context may impact gender gaps; a baseline treatment, a treatment where participants are primed with their own gender, and a treatment where the participants know the gender of their counterpart (man or woman). We look at willingness to compete in two domains: a math task and a verbal task. We find no gender differences in risk preferences or in willingness to compete in the verbal task in this random sample. There is some support for men being more competitive than women in the math task, in particular in the pooled sample. The effect size is however considerably smaller than what is typically found. We further find no consistent impact of treatment on (the absence of) the gender gap in preferences.
|
|
