| 1. |
- Garousi, Vahid, et al.
(författare)
-
Benefitting from the Grey Literature in Software Engineering Research
- 2020
-
Ingår i: Contemporary Empirical Methods in Software Engineering. - Cham : Springer Nature. - 9783030324889 ; , s. 385-413
-
Bokkapitel (refereegranskat)abstract
- Researchers generally place the most trust in peer-reviewed, published information, such as journals and conference papers. By contrast, software engineering (SE) practitioners typically do not have the time, access, or expertise to review and benefit from such publications. As a result, practitioners are more likely to turn to other sources of information that they trust, e.g., trade magazines, online blog posts, survey results, or technical reports, collectively referred to as grey literature (GL). Furthermore, practitioners also share their ideas and experiences as GL, which can serve as a valuable data source for research. While GL itself is not a new topic in SE, using, benefitting, and synthesizing knowledge from the GL in SE is a contemporary topic in empirical SE research and we are seeing that researchers are increasingly benefitting from the knowledge available within GL. The goal of this chapter is to provide an overview of GL in SE, together with insights on how SE researchers can effectively use and benefit from the knowledge and evidence available in the vast amount of GL.
|
|
| 2. |
- Garousi, Vahid, et al.
(författare)
-
Introduction to the Special Issue on : Grey Literature and Multivocal Literature Reviews (MLRs) in software engineering
- 2022
-
Ingår i: Information and Software Technology. - : Elsevier B.V.. - 0950-5849 .- 1873-6025. ; 141
-
Tidskriftsartikel (refereegranskat)abstract
- In parallel to academic (peer-reviewed) literature (e.g., journal and conference papers), an enormous extent of grey literature (GL) has accumulated since the inception of software engineering (SE). GL is often defined as “literature that is not formally published in sources such as books or journal articles”, e.g., in the form of trade magazines, online blog-posts, technical reports, and online videos such as tutorial and presentation videos. GL is typically produced by SE practitioners. We have observed that researchers are increasingly using and benefitting from the knowledge available within GL. Related to the notion of GL is the notion of Multivocal Literature Reviews (MLRs) in SE, i.e., a MLR is a form of a Systematic Literature Review (SLR) which includes knowledge and/or evidence from the GL in addition to the peer-reviewed literature. MLRs are useful for both researchers and practitioners because they provide summaries of both the state-of-the-art and -practice in a given area. MLRs are popular in other fields and have started to appear in SE community. It is timely then for a Special Issue (SI) focusing on GL and MLRs in SE. From the pool of 13 submitted papers, and after following a rigorous peer review process, seven papers were accepted for this SI. In this introduction we provide a brief overview of GL and MLRs in SE, and then a brief summary of the seven papers published in this SI. © 2021
|
|
| 3. |
- Rainer, Austen, et al.
(författare)
-
Case study identification : A trivial indicator outperforms human classifiers
- 2023
-
Ingår i: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 161
-
Tidskriftsartikel (refereegranskat)abstract
- Context: The definition and term “case study” are not being applied consistently by software engineering researchers. We previously developed a trivial “smell indicator” to help detect the misclassification of primary studies as case studies. Objective: To evaluate the performance of the indicator. Methods: We compare the performance of the indicator against human classifiers for three datasets, two datasets comprising classifications by both authors of systematic literature studies and primary studies, and one dataset comprising only primary-study author classifications. Results: The indicator outperforms the human classifiers for all datasets. Conclusions: The indicator is successful because human classifiers “fail” to properly classify their own, and others’, primary studies. Consequently, reviewers of primary studies and authors of systematic literature studies could use the classifier as a “sanity” check for primary studies. Moreover, authors might use the indicator to double-check how they classified a study, as part of their analysis, and prior to submitting their manuscript for publication. We challenge the research community to both beat the indicator, and to improve its ability to identify true case studies. © 2023 The Author(s)
|
|
| 4. |
- Rainer, Austen, et al.
(författare)
-
Recruiting credible participants for field studies in software engineering research
- 2022
-
Ingår i: Information and Software Technology. - : ELSEVIER. - 0950-5849 .- 1873-6025. ; 151
-
Tidskriftsartikel (refereegranskat)abstract
- Context: Software practitioners are a primary provider of information for field studies in software engineering. Research typically recruits practitioners through some kind of sampling. But sampling may not in itself recruit the "right" participants. Objective: To assess existing guidance on participant recruitment, and to propose and illustrate a framework for recruiting professional practitioners as credible participants in field studies of software engineering. Methods: We review existing guidelines, checklists and other advisory sources on recruiting participants for field studies. We develop a framework, partly based on our prior research and on the research of others. We search for and select three exemplar studies (a case study, an interview study and a survey study) and use those to illustrate the framework. Results: Whilst existing guidance recognises the importance of recruiting participants, there is limited guidance on how to recruit the "right" participants. The framework suggests the conceptualisation of participants as "research instruments" or, alternatively, as a sampling frame for items of interest. The exemplars suggest that at least some members of the research community are aware of the need to carefully recruit the "right" participants. Conclusions: The framework is intended to encourage researchers to think differently about the involvement of practitioners in field studies of software engineering. Also, the framework identifies a number of characteristics not explicitly addressed by existing guidelines.
|
|
| 5. |
- Rainer, Austen, et al.
(författare)
-
Reporting case studies in systematic literature studies—An evidential problem
- 2024
-
Ingår i: Information and Software Technology. - : Elsevier. - 0950-5849 .- 1873-6025. ; 174
-
Tidskriftsartikel (refereegranskat)abstract
- Context: The term and label, “case study”, is not used consistently by authors of primary studies in software engineering research. It is not clear whether this problem also occurs for systematic literature studies (SLSs).Objective: To investigate the extent to which SLSs in/correctly use the term and label, “case study”, when classifying primary studies.Methods: We systematically collect two sub-samples (2010–2021 & 2022) comprising a total of eleven SLSs and 79 primary studies. We examine the designs of these SLSs, and then analyse whether the SLS authors and the primary-study authors correctly label the respective primary study as a “case study”.Results: 76% of the 79 primary studies are misclassified by SLSs (with the two sub-samples having 60% and 81% misclassification, respectively). For 39% of the 79 studies, the SLSs propagate a mislabelling by the original authors, whilst for 37%, the SLSs introduce a new mislabel, thus making the problem worse. SLSs rarely present explicit definitions for “case study” and when they do, the definition is not consistent with established definitions.Conclusions: SLSs are both propagating and exacerbating the problem of the mislabelling of primary studies as “case studies”, rather than – as we should expect of SLSs – correcting the labelling of primary studies, and thus improving the body of credible evidence. Propagating and exacerbating mislabelling undermines the credibility of evidence in terms of its quantity, quality and relevance to both practice and research. © 2024 The Author(s)
|
|
| 6. |
- Rainer, Austen W., et al.
(författare)
-
Retrieving and mining professional experience of software practice from grey literature : An exploratory review
- 2020
-
Ingår i: IET Software. - : John Wiley & Sons. - 1751-8806 .- 1751-8814. ; 14:6, s. 665-676
-
Forskningsöversikt (refereegranskat)abstract
- Retrieving and mining practitioners' self-reports of their professional experience of software practice could provide valuable evidence for research. The authors are, however, unaware of any existing reviews of research conducted in this area. The authors reviewed and classified previous research, and identified insights into the challenges research confronts when retrieving and mining practitioners' self-reports of their experience of software practice. They conducted an exploratory review to identify and classify 42 studies. They analysed a selection of those studies for insights on challenges to mining professional experience. They identified only one directly relevant study. Even then this study concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. They discussed the challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. No directly relevant prior research appears to have been conducted in this area. They discussed the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. They formulated a set of guiding questions for further research in this area. © 2020 Institution of Engineering and Technology. All rights reserved.
|
|
| 7. |
|
|
| 8. |
- Wohlin, Claes, et al.
(författare)
-
Challenges and recommendations to publishing and using credible evidence in software engineering
- 2021
-
Ingår i: Information and Software Technology. - : Elsevier B.V.. - 0950-5849 .- 1873-6025. ; 134
-
Tidskriftsartikel (refereegranskat)abstract
- Context: An evidence-based scientific discipline should produce, consume and disseminate credible evidence. Unfortunately, mistakes are sometimes made, resulting in the production, consumption and dissemination of invalid or otherwise questionable evidence. In the worst cases, such questionable evidence achieves the status of accepted knowledge. There is, therefore, the need to ensure that producers and consumers seek to identify and rectify such situations. Objectives: To raise awareness of the negative impact of misinterpreting evidence and of propagating that misinterpreted evidence, and to provide guidance on how to improve on the type of issues identified. Methods: We use a case-based approach to present and analyse the production, consumption and dissemination of evidence. The cases are based on the literature and our professional experience. These cases illustrate a range of challenges confronting evidence-based researchers as well as the consequences to research when invalid evidence is not corrected in a timely way. Results: We use the cases and the challenges to formulate a framework and a set of recommendations to help the community in producing and consuming credible evidence. Conclusions: We encourage the community to collectively remain alert to the emergence and dissemination of invalid, or otherwise questionable, evidence, and to proactively seek to identify and rectify it. © 2021 The Authors
|
|
| 9. |
- Wohlin, Claes, et al.
(författare)
-
Is it a case study?—A critical analysis and guidance
- 2022
-
Ingår i: Journal of Systems and Software. - : Elsevier Inc.. - 0164-1212 .- 1873-1228. ; 192
-
Tidskriftsartikel (refereegranskat)abstract
- The term “case study” is not used consistently when describing studies and, most importantly, is not used according to the established definitions. Given the misuse of the term “case study”, we critically analyse articles that cite case study guidelines and report case studies. We find that only about 50% of the studies labelled “case study” are correctly labelled, and about 40% of studies labelled “case study” are actually better understood as “small-scale evaluations”. Based on our experiences conducting the analysis, we formulate support for ensuring and assuring the correct labelling of case studies. We develop a checklist and a self-assessment scheme. The checklist is intended to complement existing definitions and to encourage researchers to use the term “case study” correctly. The self-assessment scheme is intended to help the researcher identify when their empirical study is a “small-scale evaluation” and, again, encourages researchers to label their studies correctly. Finally, we develop and evaluate a smell indicator to automatically suggest when a reported case study may not actually be a case study. These three instruments have been developed to help ensure and assure that only those studies that are actually case studies are labelled as “case study”. © 2022 The Author(s)
|
|
