| 1. |
- Ebersole, Charles R., et al.
(author)
-
Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability
- 2020
-
In: Advances in Methods and Practices in Psychological Science. - : Sage. - 2515-2467 .- 2515-2459. ; 3:3, s. 309-331
-
Journal article (peer-reviewed)abstract
- Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3-9; median total sample = 1,279.5, range = 276-3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Delta r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00-.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19-.50).
|
|
| 2. |
- Schweinsberg, Martin, et al.
(author)
-
Same data, different conclusions : Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
- 2021
-
In: Organizational Behavior and Human Decision Processes. - : Elsevier BV. - 0749-5978 .- 1095-9920. ; 165, s. 228-249
-
Journal article (peer-reviewed)abstract
- In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists' gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for orga-nizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.
|
|
| 3. |
- Lee, Eun-Young, et al.
(author)
-
Play, Learn, and Teach Outdoors—Network (PLaTO-Net) : terminology, taxonomy, and ontology
- 2022
-
In: International Journal of Behavioral Nutrition and Physical Activity. - : BioMed Central (BMC). - 1479-5868. ; 19:1
-
Journal article (peer-reviewed)abstract
- Background: A recent dialogue in the field of play, learn, and teach outdoors (referred to as “PLaTO” hereafter) demonstrated the need for developing harmonized and consensus-based terminology, taxonomy, and ontology for PLaTO. This is important as the field evolves and diversifies in its approaches, contents, and contexts over time and in different countries, cultures, and settings. Within this paper, we report the systematic and iterative processes undertaken to achieve this objective, which has built on the creation of the global PLaTO-Network (PLaTO-Net). Methods: This project comprised of four major methodological phases. First, a systematic scoping review was conducted to identify common terms and definitions used pertaining to PLaTO. Second, based on the results of the scoping review, a draft set of key terms, taxonomy, and ontology were developed, and shared with PLaTO members, who provided feedback via four rounds of consultation. Third, PLaTO terminology, taxonomy, and ontology were then finalized based on the feedback received from 50 international PLaTO member participants who responded to ≥ 3 rounds of the consultation survey and dialogue. Finally, efforts to share and disseminate project outcomes were made through different online platforms. Results: This paper presents the final definitions and taxonomy of 31 PLaTO terms along with the PLaTO-Net ontology model. The model incorporates other relevant concepts in recognition that all the aspects of the model are interrelated and interconnected. The final terminology, taxonomy, and ontology are intended to be applicable to, and relevant for, all people encompassing various identities (e.g., age, gender, culture, ethnicity, ability). Conclusions: This project contributes to advancing PLaTO-based research and facilitating intersectoral and interdisciplinary collaboration, with the long-term goal of fostering and strengthening PLaTO’s synergistic linkages with healthy living, environmental stewardship, climate action, and planetary health agendas. Notably, PLaTO terminology, taxonomy and ontology will continue to evolve, and PLaTO-Net is committed to advancing and periodically updating harmonized knowledge and understanding in the vast and interrelated areas of PLaTO.
|
|
