| 1. |
- Mowbray, Carol T, et al.
(författare)
-
Fidelity Criteria: Development, Measurement, and Validation
- 2003
-
Ingår i: American Journal of Evaluation. - : Elsevier. - 1098-2140 .- 1557-0878. ; 24:3, s. 315-340
-
Tidskriftsartikel (refereegranskat)abstract
- Fidelity may be defined as the extent to which delivery of an intervention adheres to the protocol or program model originally developed. Fidelity measurement has increasing significance for evaluation, treatment effectiveness research, and service administration. Yet few published studies using fidelity criteria provide details on the construction of a valid fidelity index. The purpose of this review article is to outline steps in the development, measurement, and validation of fidelity criteria, providing examples from health and education literatures. We further identify important issues in conducting each step. Finally, we raise questions about the dynamic nature of fidelity criteria, appropriate validation and statistical analysis methods, the inclusion of structure and process criteria in fidelity assessment, and the role of program theory in deciding on the balance between adaptation versus exact replication of model programs. Further attention to the use and refinement of fidelity criteria is important to evaluation practice.
|
|
| 2. |
- Dahler-Larsen, Peter, et al.
(författare)
-
Evaluation, language, and untranslatables
- 2017
-
Ingår i: American Journal of Evaluation. - : Sage Publications. - 1098-2140 .- 1557-0878. ; 38:1, s. 114-125
-
Tidskriftsartikel (refereegranskat)abstract
- The issue of translatability is pressing in international evaluation, in global transfer of evaluative instruments, in comparative performance management, and in culturally responsive evaluation. Terms that are never fully understood, digested, or accepted may continue to influence issues, problems, and social interactions in and around and after evaluations. Their meanings can be imposed or reinvented. Untranslatable terms are not just “lost in translation” but may produce overflows that do not go away. The purpose of this article is to increase attention to the issue of translatability in evaluation by means of specific exemplars. We provide a short dictionary of such exemplars delivered by evaluators, consultants, and teachers who work across a variety of contexts. We conclude with a few recommendations: highlight frictions in translatability by deliberately circulating and discussing words of relevance that appear to be "foreign"; increase the language skills of evaluators; and make research on frictions in translation an articulate part of the agenda for research on evaluation.
|
|
| 3. |
- Flygare, Erik, 1960-, et al.
(författare)
-
Lessons From a Concurrent Evaluation of Eight Antibullying Programs Used in Sweden
- 2013
-
Ingår i: American Journal of Evaluation. - : SAGE Publications. - 1098-2140 .- 1557-0878. ; 34:2, s. 170-189
-
Tidskriftsartikel (refereegranskat)abstract
- Sweden has a low prevalence of bullying and Swedish schools are legally obliged to have anti-bullying policies. Many commercial programs are available. A mixed methods, quasi-experimental, concurrent evaluation of 8 programs, chosen from a pool of 21 widely used anti-bullying programs, was planned. Preliminary data, based on 835 stakeholder interviews, both individual and group, revealed extensive program-contamination in "treatment" schools (n = 31), and extensive program use in nominally designated control/comparison schools (n = 8). An evolved evaluation strategy focused on arrays of program components, use of which was assessed through detailed qualitative data gathered from principals, teaching and non-teaching staff, anti-bullying teams, and pupils. Bullying victimization, Grades 4-9, in 39 schools, was assessed by questionnaire, administered in three waves. Responserates were 76.1% (with N = 10,919) at the first wave, 74.3% (of 10,660) at the second wave, and70.6% (of 10,523) at the final wave. Longitudinal aggregate and individual outcomes (3,487 pupils answered all three questionnaires) were assessed. Prevalence of victimization was 8.1% at first wave and (7.4% at last). Based on individual data, persistent victimization (bullied at both waves) was estimated at 1.5% with 4.7% experiencing a worsening situation (not bullied at the first wave but bullied at last) and 4.8% with an improved situation. Hierarchical cluster analysis was used to designate intervention types (shared arrays of strong, weak, or absent component use) and schools with similar approaches. A quasi-experimental, longitudinal design with individual data and an aggregate cross-sectional follow-up indicated differences in effective, ineffective, and iatrogenic components, component clusters (intervention types), and successful schools.
|
|
| 4. |
- Groth Andersson, Signe, et al.
(författare)
-
Data recording in performance management : trouble with the logics
- 2017
-
Ingår i: American Journal of Evaluation. - : Sage Publications. - 1098-2140 .- 1557-0878. ; 38:2, s. 190-204
-
Tidskriftsartikel (refereegranskat)abstract
- In recent years, performance management (PM) has become a buzzword in public sector organizations.Well-functioning PM systems rely on valid performance data, but critics point out thatconflicting rationale or logic among professional staff in recording information can undermine thequality of the data. Based on a case study of social service staff members, the authors explore threerecording logics. The findings reveal a complexity of recording behavior and show how frontline staffshift between recording logics according to the situation. The actual data recordings depend notonly on the overall logic but also on factors such as attitudes, assumptions, and motives. The authorssuggest that shifting recording logics weaken the validity of performance data. These shifts underminethe idea of PM as a trustworthy strategy to bridge the gap between professional and managerialstaff, as well as the possibility of a well-informed management.
|
|
| 5. |
- Hanberger, Anders, 1953-
(författare)
-
Policy and program evaluation, civil society, and democracy
- 2001
-
Ingår i: American Journal of Evaluation. - : American Evaluation Association. - 1098-2140 .- 1557-0878. ; 22:2, s. 211-228
-
Tidskriftsartikel (refereegranskat)abstract
- The overall aim of this article is to develop conceptual tools and practical guidelines for evaluating the effects of public policy and program on civil society and democracy, and to analyze three broad ways of taking evaluation to the people. The terms “civil society” and “democracy” are ambiguous, referring to different notions in various discourses. The article demonstrates how different notions (ideal-types) of civil society and democracy can be useful in assessing the effects of public policies and programs. Finally, three broad approaches to evaluation are analyzed as tools for taking evaluation to the people. I argue that, in light of the legitimacy crisis, the “mediating evaluation approach” is the most promising approach today. However, the role of the evaluator and the notion of both civil society and democracy differ among the three approaches.
|
|
| 6. |
- Hansen, Morten Balle, et al.
(författare)
-
Theory-Based Stakeholder Evaluation
- 2010
-
Ingår i: American Journal of Evaluation. - : SAGE Publications. - 1098-2140 .- 1557-0878. ; 31:3, s. 295-313
-
Tidskriftsartikel (refereegranskat)abstract
- This article introduces a new approach to program theory evaluation called theory-based stakeholder evaluation or the TSE model for short. Most theory-based approaches are program theory driven and some are stakeholder oriented as well. Practically, all of the latter fuse the program perceptions of the various stakeholder groups into one unitary program theory. The TSE model keeps the program theories of the diverse stakeholder groups apart from each other and from the program theory embedded in the institutionalized intervention itself. This represents, the authors argue, an important clarification and extension of the standard theory-based evaluation. The TSE model is elaborated to enhance theory-based evaluation of interventions characterized by conflicts and competing program theories. The authors argue that especially in evaluations of complex and complicated multilevel and multisite interventions, the presence of competing theories is likely and the TSE model may prove useful.
|
|
| 7. |
- Kokko, Suvi, et al.
(författare)
-
Using Zaltman Metaphor Elicitation Technique to Map Beneficiaries' Experiences and Values: A Case Example From the Sanitation Sector
- 2017
-
Ingår i: American Journal of Evaluation. - : SAGE Publications. - 1098-2140 .- 1557-0878. ; 38, s. 205-225
-
Tidskriftsartikel (refereegranskat)abstract
- Using a case example of an innovative sanitation solution in a slum setting, this study explores the usefulness of the Zaltman Metaphor Elicitation Technique in a program planning and evaluation context. Using a qualitative image-based method to map people's mental models of ill-structured problems such as sanitation can aid program planners and evaluators in understanding how a program can fit the reality of beneficiaries. The technique is a tool for investigating what beneficiaries think about specific problems a program is aimed at solving and their underlying beliefs. The results offer a comprehensive hierarchical value map of different types and levels of insights into parents' thoughts and feelings about school sanitation and their child's well-being, often expressed as desired values, goals, or end states. Based on the results, a discussion is provided about the usefulness of the technique in the given context.
|
|
| 8. |
- Matta, Corrado, 1978-, et al.
(författare)
-
The Mechanistic Rewards of Data and Theory Integration for Theory-Based Evaluation
- 2024
-
Ingår i: American Journal of Evaluation. - : Sage Publications. - 1098-2140 .- 1557-0878. ; 45:1, s. 110-132
-
Tidskriftsartikel (refereegranskat)abstract
- In this article, we discuss the methodological implications of data and theory integration for Theory-Based Evaluation (TBE). TBE is a family of approaches to program evaluation that use program theories as instruments to answer questions about whether, how, and why a program works. Some of the groundwork about TBE has expressed the idea that a proper program theory should specify the intervening mechanisms underlying the program outcome. In the present article, we discuss in what way data and theory integration can help evaluators in constructing and refining mechanistic program theories. The paper argues that a mechanism is both a network of entities and activities and a network of counterfactual relations. Furthermore, we argue that although data integration typically provides information about different parts of a program, it is the integration of theory that provides the most important mechanistic insights.
|
|
| 9. |
- Mowbray, Carol T, et al.
(författare)
-
Validation of a fidelity rating instrument for consumer-operated services
- 2006
-
Ingår i: American Journal of Evaluation. - : SAGE Publications. - 1098-2140 .- 1557-0878. ; 27:1, s. 9-27
-
Tidskriftsartikel (refereegranskat)abstract
- With the emphasis on the use of evidence-based practices has come a need to measure the fidelity of replications to the operations and principles of original models. Recent reviews have focused on methods to develop fidelity measures for evidence-based program models. However, the issue of how to validate such measures has been given scant attention. The research reported here attempted to validate a fidelity rating instrument (FRI) for consumer-operated drop-in centers on the basis of convergent validation methods. That is, fidelity ratings by trained observers were validated in relation to reports about similar concepts from the users of the centers. The authors describe their methods and present results indicating that all but 4 of 14 fidelity variables from the FRI were validated, as hypothesized. The authors discuss the success of this approach as well as its limitations and identify issues in the further development of fidelity measures.
|
|
| 10. |
- Nordesjö, Kettil, 1981-
(författare)
-
Evaluation Paradoxes : Responding to Tensions Between Stability and Change in Social Investment Evaluation
- 2023
-
Ingår i: American Journal of Evaluation. - : Sage Publications. - 1098-2140 .- 1557-0878.
-
Tidskriftsartikel (refereegranskat)abstract
- The relationship between stability and change is a central paradox of administration that pervades all forms of organizing. Evaluation is not unfamiliar with paradoxical objectives and roles, which can result in tensions for evaluators and stakeholders. In this article, paradoxes between stability and change in the implementation of evaluation, and responses to them, are investigated through the case of social investment funds in Swedish local government. From interviews with staff, managers, and evaluators, findings show how responses to four main paradoxes give priority to top-down summative evaluation that produces instrumental knowledge on outcomes and costs for decision makers. The responses show that the concept of social investment fund evaluation is elastic to contain paradoxes and address different audiences. Also, paradoxes within the structure of the organization develop into paradoxes concerning the roles and goals of evaluation, raising the question of whether individual actors can deal with paradoxes.
|
|
