| 1. |
- Bishop, Michael, et al.
(författare)
-
Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme
- 2020
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 7:7
-
Tidskriftsartikel (refereegranskat)abstract
- The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
|
|
| 2. |
- Buckles, Grant, et al.
(författare)
-
Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency’s next-generation social science programme
- 2021
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703 .- 2054-5703. ; 8:7, s. 181308-181308
-
Tidskriftsartikel (refereegranskat)abstract
- There is evidence that prediction markets are useful tools to aggregate information on researchers’ beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency’s (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen’s d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
|
|
| 3. |
- Dreber Almenberg, Anna, et al.
(författare)
-
Using prediction markets to forecast research evaluations
- 2015
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 2:10
-
Tidskriftsartikel (refereegranskat)abstract
- The 2014 Research Excellence Framework (REF2014) was conducted to assess the quality of research carried out at higher education institutions in the UK over a 6 year period. However, the process was criticized for being expensive and bureaucratic, and it was argued that similar information could be obtained more simply from various existing metrics. We were interested in whether a prediction market on the outcome of REF2014 for 33 chemistry departments in the UK would provide information similar to that obtained during the REF2014 process. Prediction markets have become increasingly popular as a means of capturing what is colloquially known as the 'wisdom of crowds', and enable individuals to trade 'bets' on whether a specific outcome will occur or not. These have been shown to be successful at predicting various outcomes in a number of domains (e.g. sport, entertainment and politics), but have rarely been tested against outcomes based on expert judgements such as those that formed the basis of REF2014.
|
|
| 4. |
- Ebersole, Charles R., et al.
(författare)
-
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
- 2020
-
Ingår i: Advances in Methods and Practices in Psychological Science. - : Sage. - 2515-2467 .- 2515-2459. ; 3:3, s. 309-331
-
Tidskriftsartikel (refereegranskat)abstract
- Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).
|
|
| 5. |
- Forsell, Eskil, et al.
(författare)
-
Predicting replication outcomes in the Many Labs 2 study
- 2019
-
Ingår i: Journal of Economic Psychology. - : Elsevier. - 1872-7719 .- 0167-4870. ; 75:Part A SI
-
Tidskriftsartikel (refereegranskat)abstract
- Understanding and improving reproducibility is crucial for scientific progress. Prediction markets and related methods of eliciting peer beliefs are promising tools to predict replication outcomes. We invited researchers in the field of psychology to judge the replicability of 24 studies replicated in the large scale Many Labs 2 project. We elicited peer beliefs in prediction markets and surveys about two replication success metrics: the probability that the replication yields a statistically significant effect in the original direction (p < 0.001), and the relative effect size of the replication. The prediction markets correctly predicted 75% of the replication outcomes, and were highly correlated with the replication outcomes. Survey beliefs were also significantly correlated with replication outcomes, but had larger prediction errors. The prediction markets for relative effect sizes attracted little trading and thus did not work well. The survey beliefs about relative effect sizes performed better and were significantly correlated with observed relative effect sizes. The results suggest that replication outcomes can be predicted and that the elicitation of peer beliefs can increase our knowledge about scientific reproducibility and the dynamics of hypothesis testing.
|
|
| 6. |
- Gordon, Michael, et al.
(författare)
-
Forecasting the publication and citation outcomes of COVID-19 preprints
- 2022
-
Ingår i: Royal Society Open Science. - : Royal Society, The: Open Access / Royal Society. - 2054-5703. ; 9:9
-
Tidskriftsartikel (refereegranskat)abstract
- Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.
|
|
| 7. |
- Isaksson, Siri, et al.
(författare)
-
Using prediction markets to estimate the reproducibility of scientific research
- 2015
-
Ingår i: Proceedings of the National Academy of Sciences. - : National Academy of Sciences. - 0027-8424 .- 1091-6490. ; 112:50, s. 15343-15347
-
Tidskriftsartikel (refereegranskat)abstract
- Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.
|
|
| 8. |
- Johannesson, Magnus, et al.
(författare)
-
Can laypeople predict the replicability of social science studies without expert intervention : An exploratory study
-
Annan publikation (övrigt vetenskapligt/konstnärligt)abstract
- The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.
|
|
| 9. |
- Meng, Qianwen, et al.
(författare)
-
MHCCL : Masked Hierarchical Cluster-Wise Contrastive Learning for Multivariate Time Series
- 2023
-
Ingår i: Proceedings of the 37th AAAI Conference on Artificial Intelligence. - 9781577358800 ; 37, s. 9153-9161
-
Konferensbidrag (refereegranskat)abstract
- Learning semantic-rich representations from raw unlabeled time series data is critical for downstream tasks such as classification and forecasting. Contrastive learning has recently shown its promising representation learning capability in the absence of expert annotations. However, existing contrastive approaches generally treat each instance independently, which leads to false negative pairs that share the same semantics. To tackle this problem, we propose MHCCL, a Masked Hierarchical Cluster-wise Contrastive Learning model, which exploits semantic information obtained from the hierarchical structure consisting of multiple latent partitions for multivariate time series. Motivated by the observation that fine-grained clustering preserves higher purity while coarse-grained one reflects higher-level semantics, we propose a novel downward masking strategy to filter out fake negatives and supplement positives by incorporating the multi-granularity information from the clustering hierarchy. In addition, a novel upward masking strategy is designed in MHCCL to remove outliers of clusters at each partition to refine prototypes, which helps speed up the hierarchical clustering process and improves the clustering quality. We conduct experimental evaluations on seven widely-used multivariate time series datasets. The results demonstrate the superiority of MHCCL over the state-of-the-art approaches for unsupervised time series representation learning.
|
|
| 10. |
- Xiang, Ze, et al.
(författare)
-
Comprehensive Evaluation of Anti-PD-1, Anti-PD-L1, Anti-CTLA-4 and Their Combined Immunotherapy in Clinical Trials: A Systematic Review and Meta-analysis
- 2022
-
Ingår i: Frontiers in Pharmacology. - : FRONTIERS MEDIA SA. - 1663-9812. ; 13
-
Forskningsöversikt (refereegranskat)abstract
- Immunotherapy with immune checkpoint inhibitor (ICI) drugs is gradually becoming a hot topic in cancer treatment. To comprehensively evaluate the safety and efficacy of ICI drugs, we employed the Bayesian model and conducted a network meta-analysis in terms of progression-free survival (PFS), overall survival (OS) and severe adverse events (AEs). Our study found that treatment with ipilimumab was significantly worse than standard therapies in terms of PFS, whereas treatment with cemiplimab significantly improved PFS. The results also indicated that cemiplimab was the best choice for PFS. Treatment with nivolumab, pembrolizumab and nivolumab plus ipilimumab significantly improved OS compared to standard therapies. In terms of OS, cemiplimab was found to be the best choice, whereas avelumab was the worst. In terms of severe AEs, atezolizumab, avelumab, durvalumab, nivolumab, and pembrolizumab all significantly reduced the risk of grade 3 or higher AEs compared to standard therapy. The least likely to be associated with severe AEs were as follows: cemiplimab, avelumab, nivolumab, atezolizumab, and camrelizumab, with nivolumab plus ipilimumab to be the worst. Therefore, different ICI drug therapies may pose different risks in terms of PFS, OS and severe AEs. Our study may provide new insights and strategies for the clinical practice of ICI drugs.
|
|
