2 - Impact Evaluation Flashcards
4 different forms of evaluation:
- Ex-ante appraisal (potential?)
- Programmatic evaluation
- Comprehensive expenditure review
- Impact analysis (some are just glad to help out but investigate if the project actually worked out well and had good impacts are increasingly popular).
Why evaluate? Objectives:
- Lesson learning (has it done what it was supposed to?beneficiaries, program, organization, world).
- Accountability
- Result-based management (use results to improve, ex test on small scale and then scale up)
The logical framework/model of evaluation:
- Needs - ex too low literacy in rural India.
- Inputs - ex monitor teacher attendance and activity.
- Output - parents vist schools daily and report.
- Outcome - teachers attend school more regularly and better quality.
- Impact - hopefully higher rate of literacy.
- Long-term goal - improved educational outcomes and career opportunities.
Different levels of program evaluation:
- Needs Assessment (who is the pop and what do they need?)
- Program Assessment (how address the needs and what are the prerequisites and shortcomings?)
- Process evaluation (are the things delivered? built? don’t assess impact, just process).
- Impact evaluation (all this lecture is about –> lead to the Q: why and when do it work? Can we scale up?)
- Cost-benefit analysis
Theory of change:
ToC analyses how inputs lead to intended outcomes/impacts. Identify causal steps and which underlying assumptions need to hold, what data we need etc..
Different types of correlation:
- Causation: X–>Y.
- Reverse causality: Y–>X
- Simultaneity: Y–>X and X–>Y
- Spurious correlation/OV bias: Z–>X and Z–>Y a third variable affecting both.
Counterfactual:
Need a group of people telling us what would have been the case if we did NOT implement the program. This cannot be done 100% since we don’t have two identical worlds… But we do our best to find a good enough counterfactual so that w can measure the impact (difference between T and C). This helps us measure causality.
What is the basic formula for measuring impacts?
To take the difference between outcome for participants vs non-participants:
Yi(1) - Yi(0)
But, as we cannot observe same unit, we must take the average impact:
E(Yi(1)) - E(Yi(0)).
So this is the expected value for the T minus the expected value for the C group.
What is the bias of the impact measurement?
The bias is:
E(Y(0)|T) - E(Y(0)|C).
So it’s the difference between being in the treatment group but not receiving the treatment and being in the control group where you obviously not receive the treatment.
If we have a perfect counterfactual, this bias=0.
This B happens because we use an estimate of ATE.
3 techniques for impact evaluation:
- Experimental design with randomisation (RCT)
- Matching methods (PSM)
- Difference in difference
- Other in the book… see notes.
Random sampling and assignment:
When we randomly select a sample from a population and den randomly assign some of them in the sample to the T and the rest to the C.
RCT
Random control trial. When using random sample and assignment, we create a relevant comparison group.
There shouldn’t be any systematical differences between the groups, no bias. –> T and C have same outcome Y in absence of the program.
Is it ethical to randomise?
Not always. If the program involves large benefits for the treated ones, then why should my neighbour get those benefits but not me? Just by luck? If we had the chance to prove who needed it the most, maybe it would have been me. But self selection destroys the properties of a relevant counterfactual…
ATE=
Average treatment effect
Issues with RCT:
- External validity (specific context)
- Hawthorne effects - changed behavior for the observed ones.
- John Henry effect - changed behavior for the controlled, work harder)
- Contamination/spillover
- Dropout or attrition
- Partial eq - measuring short term effects.
