| 1. |
- Ebersole, Charles R., et al.
(författare)
-
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
- 2020
-
Ingår i: Advances in Methods and Practices in Psychological Science. - : Sage. - 2515-2467 .- 2515-2459. ; 3:3, s. 309-331
-
Tidskriftsartikel (refereegranskat)abstract
- Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).
|
|
| 2. |
|
|
| 3. |
- Bonnier, Evelina, et al.
(författare)
-
Exposure to half-dressed women and economic behavior
- 2019
-
Ingår i: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 168, s. 393-418
-
Tidskriftsartikel (refereegranskat)abstract
- Images of half-dressed women are ubiquitous in advertising and popular culture. Yet little is known about the potential impacts of such images on economic decision making. We randomize 648 participants of both genders to advertising images including either women in bikini or underwear, fully dressed women, or no women, and examine the effects on risk taking, willingness to compete and math performance in a lab experiment. We find no treatment effects on any outcome measure for women. For men, our results indicate that men take more risk after having been exposed to images of half-dressed women compared to no women.
|
|
| 4. |
- Botvinik-Nezer, Rotem, et al.
(författare)
-
Variability in the analysis of a single neuroimaging dataset by many teams
- 2020
-
Ingår i: Nature. - : Springer Science and Business Media LLC. - 0028-0836 .- 1476-4687. ; 582, s. 84-88
-
Tidskriftsartikel (refereegranskat)abstract
- Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
|
|
| 5. |
- Menkveld, Albert J., et al.
(författare)
-
Nonstandard Errors
- 2024
-
Ingår i: JOURNAL OF FINANCE. - : Wiley-Blackwell. - 0022-1082 .- 1540-6261. ; 79:3, s. 2339-2390
-
Tidskriftsartikel (refereegranskat)abstract
- In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty-nonstandard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for more reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
|
|
| 6. |
- Aczel, Balazs, et al.
(författare)
-
Consensus-based guidance for conducting and reporting multi-analyst studies
- 2021
-
Ingår i: eLIFE. - : eLife Sciences Publications. - 2050-084X. ; 10
-
Tidskriftsartikel (refereegranskat)abstract
- Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
|
|
| 7. |
- Almenberg, Johan, et al.
(författare)
-
Do More Expensive Wines Taste Better? Evidence from a Large Sample of Blind Tastings
- 2018
-
Ingår i: World Scientific Reference on Handbook of the Economics of Wine, Volume 1: Prices, Finance, and Expert Opinion. - : WORLD SCIENTIFIC. - 2010-1732. - 9789813232730 ; , s. 535-545
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)abstract
- Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. In a sample of more than 6,000 blind tastings, we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less. For individuals with wine training, however, we find indications of a non-negative relationship between price and enjoyment. Our results are robust to the inclusion of individual fixed effects, and are not driven by outliers: when omitting the top and bottom deciles of the price distribution, our qualitative results are strengthened, and the statistical significance is improved further. These findings suggest that non-expert wine consumers should not anticipate greater enjoyment of the intrinsic qualities of a wine simply because it is expensive or is appreciated by experts.
|
|
| 8. |
|
|
| 9. |
|
|
| 10. |
- Altmejd, Adam, et al.
(författare)
-
Predicting the replicability of social science lab experiments
- 2019
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 14:12
-
Tidskriftsartikel (refereegranskat)abstract
- We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
|
|
| 11. |
- Anderson, Anders, et al.
(författare)
-
Risk taking, behavioral biases and genes : Results from 149 active investors
- 2015
-
Ingår i: Journal of Behavioral and Experimental Finance. - : Elsevier BV. - 2214-6350 .- 2214-6369. ; 6, s. 93-100
-
Tidskriftsartikel (refereegranskat)abstract
- Recent evidence suggests that there is genetic basis for economic behaviors, including preferences for risk taking. We correlate variation in risk taking and behavioral biases with two genetic polymorphisms related to the uptake of dopamine and serotonin (7R+ DRD4 and s/s 5-HTTLPR), hypothesizing that they are positively (negatively) related to risk taking. We use a small but detailed sample of active investors where we combine survey data with DNA samples and data from Swedish tax records that give us objective information about actual economic choices. We find a positive (negative) relationship between the dopamine (serotonin) gene and life expectancy bias, but no other significant correlations between the two genes and behaviors, including risk taking and measures of equity holdings. We acknowledge that our tests suffer from low power originating from the small sample size, which warrants some caution when interpreting these results.
|
|
| 12. |
- Apicella, Coren L, et al.
(författare)
-
Salivary testosterone change following monetary wins and losses predicts future financial risk-taking
- 2014
-
Ingår i: Psychoneuroendocrinology. - : Elsevier BV. - 0306-4530. ; 39:1, s. 58-64
-
Tidskriftsartikel (refereegranskat)abstract
- While baseline testosterone has recently been implicated in risk-taking in men, less is known about the effects of changing levels of testosterone on financial risk. Here we attempt to influence testosterone in men by having them win or lose money in a chance-based competition against another male opponent. We employ two treatments where we vary the amount of money at stake so that we can directly compare winners to losers who earn the same amount, thereby abstracting from income effects. We find that men who experience a greater increase in bioactive testosterone take on more risk, an association that remains when controlling for whether the participant won the competition. In fact, whether subjects won the competition did not predict future risk. These results suggest that testosterone change, and thus individual differences in testosterone reactivity, rather than the act of winning or losing, influence financial risk-taking. © 2013.
|
|
| 13. |
- Apicella, Coren L, et al.
(författare)
-
Sex Differences in Competitiveness: Hunter-Gatherer Women and Girls Compete Less in Gender-Neutral and Male-Centric Tasks
- 2015
-
Ingår i: Adaptive Human Behavior and Physiology. - : Springer (part of Springer Nature): Springer Open Choice Hybrid Journals. - 2198-7335 .- 2198-7335. ; 1:3, s. 247-269
-
Tidskriftsartikel (refereegranskat)abstract
- Despite numerous attempts to increase workplace equality, the near universal gender wage gap and underrepresentation of women in high status jobs persists in societies around the world. This persistence has led some researchers to speculate that psychological sex differences may be partly to blame. In particular, economists have begun to focus on sex differences in competitiveness as a possible cause. Here we test whether sex differences in competitiveness exist in a relatively isolated and evolutionarily relevant population of hunter-gatherers in Tanzania. In study 1 we examine sex differences in willingness to compete in a gender-neutral task in Hadza adults and children (N = 191). We find that when choosing between an individualistic payment scheme and a competitive payment scheme, boys and men are significantly more likely to compete than girls and women. We find no evidence that this sex difference varies with age. In study 2 (N = 88 and N = 70) we use both a female-centric and a male-centric task to explore sex differences in competitiveness in adults. While we find no sex difference in willingness to compete in the female-centric task, we find that men are more likely to compete in the male-centric task. While further work is needed, this study lends some support to the idea of a sex difference in willingness to compete among hunter-gatherers, but it also highlights the importance of the task type. The observation that a sizable proportion of male Hadza choose to compete in each of the tasks is discussed in light of the fact that hunter-gatherers are largely egalitarian and non-hierarchical.
|
|
| 14. |
- Apicella, Coren L, et al.
(författare)
-
Testosterone and Economic Risk Taking: A Review
- 2015
-
Ingår i: Adaptive Human Behavior and Physiology. - : Springer (part of Springer Nature): Springer Open Choice Hybrid Journals. - 2198-7335 .- 2198-7335. ; 1:3, s. 358-385
-
Tidskriftsartikel (refereegranskat)abstract
- Since precise forecasting of the future is not possible, most of life’s decisions are made with uncertain outcomes. One important facet of uncertainty that is of particular interest to decision scientists is risk—the choice between an option that is less rewarding but more certain and an option that is less certain, but potentially more rewarding. Recent developments in both neuroscience and behavioral endocrinology have helped to reveal the biological mechanisms that support decision-making involving economic risk, and consequently, potential factors associated with individual differences in risk taking. This review is dedicated to surveying recent developments that link the hormone testosterone to economic risk taking. Like neuroeconomics, endocrinological approaches may provide a potentially powerful framework from which to understand decision-making and may help to make sense of a number of well-documented behavioral anomalies involving economic risk. Specifically, we suggest that testosterone functions to modulate risky behaviors in ways that appear to be adaptive. Still, more work is needed to understand the nature of the relationship between testosterone and risk in both sexes.
|
|
| 15. |
|
|
| 16. |
- Arechar, Antonio A., et al.
(författare)
-
“I'm just a soul whose intentions are good”: The role of communication in noisy repeated games
- 2017
-
Ingår i: Games and Economic Behavior. - : Elsevier. - 1090-2473 .- 0899-8256. ; 104, s. 726-743
-
Tidskriftsartikel (refereegranskat)abstract
- We let participants indicate their intended action in a repeated game experiment where actions are implemented with errors. Even though communication is cheap talk, we find that the majority of messages were honest (although the majority of participants lied at least occasionally). As a result, communication has a positive effect on cooperation when the payoff matrix makes the returns to cooperation high; when the payoff matrix gives a lower return to cooperation, communication reduces overall cooperation. These results suggest that cheap talk communication can promote cooperation in repeated games, but only when there is already a self-interested motivation to cooperate.
|
|
| 17. |
- Benjamin, Daniel J., et al.
(författare)
-
Redefine statistical significance
- 2018
-
Ingår i: Nature Human Behaviour. - : Nature Research (part of Springer Nature). - 2397-3374. ; 2:1, s. 6-10
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)
|
|
| 18. |
- Bernard, Mark, et al.
(författare)
-
The subgroup problem : When can binding voting on extractions from a common pool resource overcome the tragedy of the commons?
- 2013
-
Ingår i: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 91, s. 122-130
-
Tidskriftsartikel (refereegranskat)abstract
- Using a common pool resource game protocol with voting we examine experimentally how cooperation varies with the level at which (binding) votes are aggregated. Our results are broadly in line with theoretical predictions. When players can vote on the behavior of the whole group or when leaders from each group can vote for the group as a whole, extraction levels from the common resource pool are close to the social optimum. When players extract resources individually, there is substantial overextraction. When players vote in subgroups, there is initially less overextraction but it increases over time. This suggests that in order for binding voting to overcome the tragedy of the commons in social dilemmas, it should ideally affect the group as a whole.
|
|
| 19. |
- Bernard, Mark, et al.
(författare)
-
The subgroup problem: When can binding voting on extractions from a common resource pool overcome the tragedy of the commons?
- 2013
-
Ingår i: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 91, s. 122-130
-
Tidskriftsartikel (refereegranskat)abstract
- Using a common pool resource game protocol with voting we examine experimentally how cooperation varies with the level at which (binding) votes are aggregated. Our results are broadly in line with theoretical predictions. When players can vote on the behavior of the whole group or when leaders from each group can vote for the group as a whole, extraction levels from the common resource pool are close to the social optimum. When players extract resources individually, there is substantial overextraction. When players vote in subgroups, there is initially less overextraction but it increases over time. This suggests that in order for binding voting to overcome the tragedy of the commons in social dilemmas, it should ideally affect the group as a whole.
|
|
| 20. |
- Bilén, David, et al.
(författare)
-
Are women more generous than men? A meta-analysis
- 2021
-
Ingår i: Journal of the Economic Science Association-Jesa. - : Springer Science and Business Media LLC. - 2199-6776 .- 2199-6784. ; 7, s. 1-18
-
Tidskriftsartikel (refereegranskat)abstract
- We perform a meta analysis of gender differences in the standard windfall gains dictator game (DG) by collecting raw data from 53 studies with 117 conditions, giving us 15,016 unique individual observations. We find that women on average give 4 percentage points more than men (Cohen's d = 0.16), and that this difference decreases to 3.1% points (Cohen's d = 0.13) if we exclude studies where dictators can only give all or nothing. The gender difference is larger if the recipient in the DG is a charity, compared to the standard DG with an anonymous individual as the recipient (a 10.9 versus a 2.3% points gender difference). These effect sizes imply that many individual studies on gender differences are underpowered; the median power in our sample of standard DG studies is only 9% to detect the meta-analytic gender difference at the 5% significance level. Moving forward on this topic, sample sizes should thus be substantially larger than what has been the norm in the past.
|
|
| 21. |
- Bishop, Michael, et al.
(författare)
-
Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme
- 2020
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 7:7
-
Tidskriftsartikel (refereegranskat)abstract
- The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
|
|
| 22. |
- Boschini, Anne, et al.
(författare)
-
Gender and altruism in a random sample
- 2018
-
Ingår i: Journal of Behavioral and Experimental Economics. - : Elsevier BV. - 2214-8043 .- 2214-8051. ; 77, s. 72-77
-
Tidskriftsartikel (refereegranskat)abstract
- We study gender differences in altruism in a large random sample of the Swedish population using a standard dictator game. Beside a baseline treatment we implement a priming treatment where participants are reminded of their gender, and two treatments with known male and female counterpart respectively. We find suggestive evidence that women are more altruistic than men only in the priming treatment. A post-hoc analysis using data on interviewer gender to explore gender context effects indicates that priming affects behavior only in mixedgender contexts.
|
|
| 23. |
- Boschini, Anne, et al.
(författare)
-
Gender, risk preference and willingness to compete in a random sample of the Swedish population
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Working paper.Experimental results from student or other non-representative convenience samples often suggest that men, on average, are more risk-taking and competitive than women. Here we explore whether these gender preference gaps also exist in a simple random sample of the Swedish adult population. Our design comprises four different treatments to systematically explore how the experimental context may impact gender gaps; a baseline treatment, a treatment where participants are primed with their own gender, and a treatment where the participants know the gender of their counterpart (man or woman). We look at willingness to compete in two domains: a math task and a verbal task. We find no gender differences in risk preferences or in willingness to compete in the verbal task in this random sample. There is some support for men being more competitive than women in the math task, in particular in the pooled sample. The effect size is however considerably smaller than what is typically found. We further find no consistent impact of treatment on (the absence of) the gender gap in preferences.
|
|
| 24. |
- Boschini, Anne, et al.
(författare)
-
Gender, risk preferences and willingness to compete in a random sample of the Swedish population
- 2018
-
Rapport (övrigt vetenskapligt/konstnärligt)abstract
- Experimental results from student or other non-representative convenience samples often suggest that men, on average, are more risk-taking and competitive than women. Here we explore whether these gender preference gaps also exist in a simple random sample of the Swedish adult population. Our design comprises four different treatments to systematically explore how the experimental context may impact gender gaps; a baseline treatment, a treatment where participants are primed with their own gender, and a treatment where the participants know the gender of their counterpart (man or woman). We look at willingness to compete in two domains: a math task and a verbal task. We find no gender differences in risk preferences or in willingness to compete in the verbal task in this random sample. There is some support for men being more competitive than women in the math task, in particular in the pooled sample. The effect size is however considerably smaller than what is typically found. We further find no consistent impact of treatment on (the absence of) the gender gap in preferences.
|
|
| 25. |
- Boschini, Anne, et al.
(författare)
-
Gender, risk preferences and willingness to compete in a random sample of the Swedish population
- 2019
-
Ingår i: Journal of Behavioral and Experimental Economics. - : Elsevier BV. - 2214-8043 .- 2214-8051. ; 83
-
Tidskriftsartikel (refereegranskat)abstract
- Experimental results from student and other non-representative convenience samples often suggest that men, on average, are more risk taking and competitive than women. We explore whether these gender preference gaps also exist in incentivized tasks in a simple random sample of the Swedish adult population. Our design comprises four different conditions to systematically explore how the experimental context may impact gender gaps; a baseline condition, a condition where participants are primed with their own gender, and two conditions where the participants know the gender of their counterpart (man or woman). We further look at competitiveness in two domains: a math task and a verbal task. We find no gender gap in risk taking or competitiveness in the verbal task in this random sample. There is some support for men being more competitive than women in the math task in the pooled sample, but the effect size is small. We further find no consistent impact of the respective conditions on (the absence of) the gender gap in preferences.
|
|
| 26. |
- Brodeur, Abel, et al.
(författare)
-
Replication games: how to make reproducibility research more systematic
-
Ingår i: Nature. - 1476-4687 .- 0028-0836. ; 621:7980, s. 684-686
-
Tidskriftsartikel (övrigt vetenskapligt/konstnärligt)abstract
- In October last year, one of us (A.B.) decided to run an ad hoc workshop at a research centre in Oslo, to try to replicate papers from economics journals. Instead of the handful of locals who were expected to attend, 70 people from across Europe signed up. The message was clear: researchers want to replicate studies. Replication is sorely needed. In areas of the social sciences, such as economics, philosophy and psychology, some studies suggest that between 35% and 70% of published results cannot be replicated when tested with new data. Often, researchers cannot even reproduce results when using the same data and code as the original paper, because key information is missing.
|
|
| 27. |
- Brütt, Katharina, et al.
(författare)
-
Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs
- 2023
-
Ingår i: Proceedings of the National Academy of Sciences - PNAS. - : National Academy of Sciences. - 1091-6490 .- 0027-8424. ; 120:23
-
Tidskriftsartikel (refereegranskat)abstract
- Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity-variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity-estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs-indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.
|
|
| 28. |
- Buckles, Grant, et al.
(författare)
-
Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency’s next-generation social science programme
- 2021
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703 .- 2054-5703. ; 8:7, s. 181308-181308
-
Tidskriftsartikel (refereegranskat)abstract
- There is evidence that prediction markets are useful tools to aggregate information on researchers’ beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency’s (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen’s d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
|
|
| 29. |
- Bueno, Natália, S., et al.
(författare)
-
Promoting Reproducibility and Replicability in Political Science
- 2024
-
Ingår i: Research & Politics. - Essen, Germany : Institute for Replication (I4R). - 2053-1680 .- 2053-1680. ; 11:1
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- This article reviews and summarizes current reproduction and replication practices in political science. We first provide definitions for reproducibility and replicability. We then review data availability policies for 28 leading political science journals and present the results from a survey of editors about their willingness to publish comments and replications. We discuss new initiatives that seek to promote and generate highquality reproductions and replications. Finally, we make the case for standards and practices that may help increase data availability, reproducibility, and replicability in political science.
|
|
| 30. |
- Buser, Thomas, et al.
(författare)
-
The flipside of comparative payment schemes
- 2016
-
Ingår i: Management Science. - : INFORMS (Institute for Operations Research and Management Sciences). - 0025-1909 .- 1526-5501. ; 62:9, s. 2626-2638
-
Tidskriftsartikel (refereegranskat)abstract
- Comparative payment schemes and tournament-style promotion mechanisms are pervasive in the workplace. We test experimentally whether they have a negative impact on people's willingness to cooperate. Participants first perform in a simple task and then participate in a public goods game. The payment scheme for the task varies across treatment groups. Compared with a piece-rate scheme, individuals in a winner-takes-all competition are significantly less cooperative in the public goods game. A lottery treatment, where the winner is decided by luck, has the same effect. In a competition treatment with feedback, winners cooperate as little as participants in the other treatments, whereas losers cooperate even less. All three treatments lead to substantial losses in the realised social surplus from the public good while having no significant impact on performance. In a complementary experiment, we aim to shed light on the psychological mechanisms behind our results.
|
|
| 31. |
- Buser, Thomas, et al.
(författare)
-
The impact of stress on tournament entry
- 2016
-
Ingår i: Experimental Economics. - : Springer (part of Springer Nature): Springer Open Choice Hybrid Journals. - 1573-6938 .- 1386-4157. ; , s. 1-25
-
Tidskriftsartikel (refereegranskat)abstract
- Individual willingness to enter competitive environments predicts career choices and labor market outcomes. Meanwhile, many people experience competitive contexts as stressful. We use two laboratory experiments to investigate whether factors related to stress can help explain individual differences in tournament entry. Experiment 1 studies whether stress responses (measured as salivary cortisol) to taking part in a mandatory tournament predict individual willingness to participate in a voluntary tournament. We find that competing increases stress levels. This cortisol response does not predict tournament entry for men but is positively and significantly correlated with choosing to enter the tournament for women. In Experiment 2, we exogenously induce physiological stress using the cold-pressor task. We find a positive causal effect of stress on tournament entry for women but no effect for men. Finally, we show that although the effect of stress on tournament entry differs between the genders, stress reactions cannot explain the well-documented gender difference in willingness to compete.
|
|
| 32. |
- Camerer, C. F., et al.
(författare)
-
Evaluating replicability of laboratory experiments in economics
- 2016
-
Ingår i: Science. - : American Association for the Advancement of Science (AAAS). - 0036-8075 .- 1095-9203. ; 351:6280, s. 1433-1436
-
Tidskriftsartikel (refereegranskat)abstract
- The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that weremade publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.
|
|
| 33. |
- Camerer, C. F., et al.
(författare)
-
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015
- 2018
-
Ingår i: Nature Human Behaviour. - : Springer Science and Business Media LLC. - 2397-3374. ; 2:9, s. 637-644
-
Tidskriftsartikel (refereegranskat)abstract
- Being able to replicate scientific findings is crucial for scientific progress1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 201516-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated truepositive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.
|
|
| 34. |
- Camerer, Colin, et al.
(författare)
-
Replication and other practices for improving scientific quality in experimental economics
- 2019
-
Ingår i: Handbook of Research Methods and Applications in Experimental Economics. - : Edward Elgar Publishing. - 9781788110563 - 9781788110556 ; , s. 83-102
-
Bokkapitel (övrigt vetenskapligt/konstnärligt)abstract
- Replications are key to the advancement of science. The recent “replication crisis” in the social sciences has highlighted the need for high-powered replications to confirm original results before strong conclusions can be drawn. Here we discuss two big replication projects in psychology and economics respectively and assess their results in terms of the various replication indicators that have been proposed. We further discuss how we have used prediction markets and surveys to predict replication outcomes, and end with a discussion of general quality control in research, and our opinions on improved practices.
|
|
| 35. |
|
|
| 36. |
- Campbell, Benjamin, et al.
(författare)
-
The 7R polymorphism in the dopamine receptor D(4) gene (DRD4) is associated with financial risk taking in men
- 2009
-
Ingår i: Evolution and Human Behavior. - : Elsevier. - 1090-5138. ; 30:2, s. 85-92
-
Tidskriftsartikel (refereegranskat)abstract
- Individuals exhibit substantial heterogeneity in financial risk aversion. Recent work on twins demonstrated that some variation is influenced by individual heritable differences. Despite this, there has been no study investigating possible genetic loci associated with financial risk taking in healthy individuals. Here, we examined whether there is an association between financial risk preferences, elicited experimentally in a game with real monetary payoffs, and the presence of the 7-repeat allele (7R+) in the dopamine receptor D(4) gene as well as the presence of the Al allele (Al+) in the dopamine receptor D(2) gene in 94 young men. Although we found no association between the Al allele and risk preferences, we did find that 7R+ men are significantly more risk loving than 7R- men. This polymorphism accounts for roughly 20% of the heritable variation in financial risk taking. We suggest that selection for the 7R allele may be for a behavioral phenotype associated with risk taking. This is consistent with previous evolutionary explanations suggesting that selection for this allele was for behaviors associated with migration and male competition, both of which entail an element of risk. (C) 2009 Elsevier Inc. All rights reserved.
|
|
| 37. |
- Campbell, Douglas, et al.
(författare)
-
The Robustness Reproducibility of the American Economic Review
- 2024
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- We estimate the robustness reproducibility of key results from 17 non-experimental AER papers published in 2013 (8 papers) and 2022/23 (9 papers). We find that many of the results are not robust, with no improvement over time. The fraction of significant robustness tests (p﹤0.05) varies between 17% and 88% across the papers with a mean of 46%. The mean relative t/z-value of the robustness tests varies between 35% and 87% with a mean of 63%, suggesting selective reporting of analytical specifications that exaggerate statistical significance. A sample of economists (n=359) overestimates robustness reproducibility, but predictions are correlated with observed reproducibility.
|
|
| 38. |
- Cárdenas, J.-C, et al.
(författare)
-
Cooperativeness and competitiveness in children
- 2015
-
Ingår i: Journal of Behavioral and Experimental Economics. - : Elsevier: 24 months. - 2214-8043. ; 59, s. 32-41
-
Tidskriftsartikel (refereegranskat)abstract
- Cooperation and competition are both essential elements of economic life. Here we explore how cooperativeness in a prisoner's dilemma correlates with competitiveness in a sample of 9-12 years old children in Colombia and Sweden. Using two different measures and four different tasks for competitiveness, we find no consistent relationship between cooperativeness and competitiveness. However, we find evidence of a negative relationship between willingness to compete in a math task and cooperativeness in the overall sample. Competitiveness in math has previously been related to educational choices, and may therefore be the most economically relevant relationship.
|
|
| 39. |
|
|
| 40. |
- Cárdenas, Juan-Camilo, et al.
(författare)
-
Gender and cooperation in children : experiments in Colombia and Sweden.
- 2014
-
Ingår i: PLOS ONE. - : Public Library of Science (PLoS). - 1932-6203. ; 9:3
-
Tidskriftsartikel (refereegranskat)abstract
- In this article we compare cooperation among Colombian and Swedish children aged 9-12. We illustrate the dynamics of the prisoner's dilemma in a new task that is easily understood by children and performed during a physical education class. We find no robust evidence of a difference in cooperation between Colombia and Sweden overall. However, Colombian girls cooperate less than Swedish girls. We also find indications that girls in Colombia are less cooperative than boys. Finally, there is also a tendency for children to be more cooperative with boys than with girls on average.
|
|
| 41. |
- Cardenas, Juan-Camilo, et al.
(författare)
-
Gender differences in competitiveness and risk taking : comparing children in Colombia and Sweden
- 2012
-
Ingår i: Journal of Economic Behavior and Organization. - : Elsevier BV. - 0167-2681 .- 1879-1751. ; 83:1, s. 11-23
-
Tidskriftsartikel (refereegranskat)abstract
- We explore gender differences in preferences for competition and risk among children aged 9-12 in Colombia and Sweden, two countries differing in gender equality according to macro indices. We include four types of tasks that vary in gender stereotyping when looking at competitiveness: running, skipping rope, math and word search. We find that boys and girls are equally competitive in all tasks and all measures in Colombia. Unlike the consistent results in Colombia, the results in Sweden are mixed, with some indication of girls being more competitive than boys in some tasks in terms of performance change, whereas boys are more likely to choose to compete in general. Boys in both countries are more risk taking than girls, with a smaller gender gap in Sweden.
|
|
| 42. |
|
|
| 43. |
- Czeszumski, Artur, et al.
(författare)
-
#EEGManyLabs: Investigating the Replicability of Influential EEG Experiments
- 2024
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- There is growing awareness across the neuroscience community that the replicability of findings on the relationship between brain activity and cognitive phenomena can be improved by conducting studies with high statistical power that adhere to well-defined and standardized analysis pipelines. Inspired by efforts from the psychological sciences, and with the desire to examine some of the foundational findings using electroencephalography (EEG), we have launched #EEGManyLabs, a large-scale international collaborative replication effort. Since its discovery in the early 20th century, EEG has had a profound influence on our understanding of human cognition, but there is limited evidence on the replicability of some of the most highly cited discoveries. After a systematic search and selection process, we have identified 27 of the most influential and continually cited studies in the field. We plan to directly test the replicability of key findings from 20 of these studies in teams of at least three independent laboratories. The design and protocol of each replication effort will be submitted as a Registered Report and peer-reviewed prior to data collection. Prediction markets, open to all EEG researchers, will be used as a forecasting tool to examine which findings the community expects to replicate. This project will update our confidence in some of the most influential EEG findings and generate a large open access database that can be used to inform future research practices. Finally, through this international effort, we hope to create a cultural shift towards inclusive, high-powered multi-laboratory collaborations.
|
|
| 44. |
- Delios, A., et al.
(författare)
-
Examining the generalizability of research findings from archival data
- 2022
-
Ingår i: Proceedings of the National Academy of Sciences of the United States of America. - : Proceedings of the National Academy of Sciences. - 0027-8424 .- 1091-6490. ; 119:30
-
Tidskriftsartikel (refereegranskat)abstract
- This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
|
|
| 45. |
- Dimant, Eugen, et al.
(författare)
-
Politicizing mask-wearing: predicting the success of behavioral interventions among republicans and democrats in the US
- 2022
-
Ingår i: Scientific Reports. - : Springer Nature. - 2045-2322. ; 12:1
-
Tidskriftsartikel (refereegranskat)abstract
- Scientists and policymakers seek to choose effective interventions that promote preventative health measures. We evaluated whether academics, behavioral science practitioners, and laypeople (N=1034) were able to forecast the effectiveness of seven different messages compared to a baseline message for Republicans and Democrats separately. These messages were designed to nudge mask-wearing attitudes, intentions, and behaviors. When examining predictions across political parties, forecasters predicted larger effects than those observed for Democrats compared to Republicans and made more accurate predictions for Republicans compared to Democrats. These results are partly driven by a lack of nudge effects on Democrats, as reported in Gelfand et al. (J Exp Soc Psychol, 2021). Academics and practitioners made more accurate predictions compared to laypeople. Although forecasters' predictions were correlated with the nudge interventions, all groups overestimated the observed results. We discuss potential reasons for why the forecasts did not perform better and how more accurate forecasts of behavioral intervention outcomes could potentially provide insight that can help save resources and increase the efficacy of interventions.
|
|
| 46. |
- Drazen, Allan, et al.
(författare)
-
A Journal-Based Replication of "Being Chosen to Lead"
- 2019
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- Recent large-scale replications of social science experiments provide important information on the reliability of experimental research. Unfortunately, there exist no mechanisms to ensure replications are done. We propose such a mechanism: journal-based replication, in which the publishing journal contracts for a replication between acceptance and publication. We discuss what we learned from a proof-of-concept journal-based replication at the Journal of Public Economics. Our experience indicates that journal-based replication would be relatively straightforward to implement for laboratory experiments.
|
|
| 47. |
- Drazen, Allan, et al.
(författare)
-
Journal-based replication of experiments: An application to “Being Chosen to Lead”
- 2021
-
Ingår i: Journal of Public Economics. - : Elsevier: 24 months. - 1879-2316 .- 0047-2727. ; 202
-
Tidskriftsartikel (refereegranskat)abstract
- Recent large-scale replications of social science experiments provide important information on the reliability of experimental research. Unfortunately, there exist no mechanisms to ensure replications are done. We propose such a mechanism: journal-based replication, in which the publishing journal insists on a replication attempt between acceptance and publication. We discuss what we learned from a proof-of-concept journal-based replication at the Journal of Public Economics. Our experience indicates that journal-based replication would be relatively straightforward to implement for laboratory experiments. © 2021 The Authors
|
|
| 48. |
|
|
| 49. |
- Dreber, Anna, et al.
(författare)
-
Olika slutsatser från samma data : Den analytiska flexibiliteten i medicinsk forskning är ofta stor
- 2023
-
Ingår i: Läkartidningen. - : Läkartidningen Förlag AB. - 1652-7518 .- 0023-7205. ; 120
-
Tidskriftsartikel (refereegranskat)abstract
- Analysis of research data entails many choices. As a result, a space of different analytical strategies is open to researchers. Different justifiable analyses may not give similar results. The method of multiple analysts is a way to study the analytical flexibility and behaviour of researchers under naturalistic conditions, as part of the field known as metascience. Analytical flexibility and risks of bias can be counteracted by open data sharing, pre-registration of analysis plans, and registration of clinical trials in trial registers. These measures are particularly important for retrospective studies where analytical flexibility can be greatest, although pre-registration is less useful in this context. Synthetic datasets can be an alternative to pre-registration when used to decide what analyses should be conducted on real datasets by independent parties. All these strategies help build trustworthiness in scientific reports, and improve the reliability of research findings.
|
|
| 50. |
|
|
