Flashcards in Health-Related Evaluation Methods Deck (21)
What is the formative realist case evaluation example
Involved an evaluation of a difficult to describe large scale "transformation" of health services and its related report.
What are the overall questions considered in all realist evaluations?
What works, for whom, under what circumstances?
What are transferable lessons about effective change?O
Theory or logic theory whose aim is to explain how an intervention is understood to work. It used mixed data sources and collection methods of ethnographic observation, semi-structured interviews and document analysis
Realist context-intervention-mechanism theory
Summarize the "lessons" that were reported related to Mechanism 1 from the formative realist case evaluation example
1. Relationships between organizations are trusting
2. Approaches to integration are imaginative, locally responsive, negotiable and supported by technology rather than rigid and driven by tech
3. External incentives are designed to reward collaborative performance and do not make organizations compete
4. The strategy for integration includes both "soft" and "hard" approaches
5. Solutions are participatory rather than developed by one party and imposed on others
The generalizability of the findings in this type of formative realist case evaluation example can be increased by ___________________.
Providing details of the change actions and of the context
The description of the mechanisms or change principles and how they were triggered by the intervention in the setting enables others to try to ________ the mechanism in their different settings, possibly using different ___________.
How does this type of formative realist case differ from many other forms of evaluation, particularly experimental evaluations?
Gives a narrative understanding of the process of change in an environment rather than showing a few measured outcomes attributed to a change
What difficulties are involved with formative realist case evaluations?
1. Distinguishing between context and mechanism
2. Is there an end point?
3. When to "close"/
5. Can realist evaluation studies be replicated?
6. How does realist evaluation differ from traditional mediator/moderator analysis?
Describe the reasons to evaluate a reform before it starts, during its implementation and after its implementation
Assess whether it is feasible, to predict problems and to help plan how to implement it
During a reform, an evaluation is to help make corrections _________ implementation
After the reform, an evaluation helps to explain the lessons for _______ changes or reforms and ________ to scientific knowledge that could help the reformers explain, understand or predict the effects of _____________ changes.
List and explain the usual steps for evaluation a reform
1. Describing the reform "instrument"
2. Gathering data about changes in health service performance and possibly also in health
3. Assessing whether these data really are outcomes of reform
The aim of the reform is often to do the following
1. Present evidence of the results of the reform
2. Describe the process of implementing the reform
3. Assess the strengths and weaknesses of the reform as implemented and judge the value of the results
4. Recommend improvements
In most reforms, the intention is to change the organization, procedures, financing methods and other "things".
Immediate subject or target
Reform that is often aimed to improve patients health or well-being
Long term aim
Intervention that does not work directly on the _________ _________ (patients or the population) but works through changing the immediate subject (health organization) in order to achieve the ultimate benefits or better health.
4 designs typically utilized to evaluate health reforms
1. Implementation designs
2. Comparing achievements with reform objectives
3. Before/after design
4. Comparing one site with one of more others implementing the reform
Designs that report on strengths and weaknesses of the process and make explicit the criteria used to assess the process. For example, one criterion might be whether all stakeholder groups were involved.
Compares the intended goals of the reform with the extent to which they are achieved
Comparing achievements with reform objectives
This design describes the reform implementation process but also gathers data before the reform and then after the reform has been proceeding for enough time to have an effect.
Common criteria are costs, efficiency, access, equity, quality and service comprehensiveness.