📈Applied Impact Evaluation Unit 1 – Impact Evaluation: An Introduction

What's Impact Evaluation?

• Systematic approach to measuring the causal effects of a program, policy, or intervention on specific outcomes of interest
• Compares outcomes between a treatment group that receives the intervention and a control group that does not
• Aims to establish a counterfactual scenario to determine what would have happened in the absence of the intervention
• Focuses on attributing observed changes in outcomes directly to the intervention rather than other factors
• Provides rigorous evidence on the effectiveness of development programs and policies
• Helps policymakers, funders, and implementers make informed decisions about resource allocation and program design
• Contributes to building a body of knowledge on what works in development and under what conditions

Why Do We Need It?

• Development programs and policies often have limited resources and need to be allocated effectively to achieve desired outcomes
• Without impact evaluation, it is difficult to know whether observed changes in outcomes are due to the intervention or other factors
• Impact evaluation helps to identify which programs and policies are most effective in achieving their intended outcomes
• Provides evidence on the magnitude and significance of program impacts, which can inform decisions about scaling up or replicating successful interventions
• Helps to identify unintended consequences or negative impacts of interventions, which can inform program design and implementation
• Contributes to accountability and transparency in development by providing rigorous evidence on program effectiveness
• Builds knowledge on what works in development, which can inform future program design and policy decisions

Key Concepts and Terms

• Treatment group: The group that receives the intervention being evaluated
• Control group: The group that does not receive the intervention and serves as a comparison to the treatment group
• Counterfactual: The scenario that would have occurred in the absence of the intervention
• Selection bias: Systematic differences between the treatment and control groups that can affect outcomes and bias impact estimates
• Randomization: Assigning units (individuals, households, communities) to treatment and control groups by chance to minimize selection bias
• Spillover effects: Indirect effects of the intervention on non-participants or neighboring communities
• Heterogeneous effects: Variation in program impacts across different subgroups or contexts
  • Example: A nutrition program may have larger impacts on children from low-income households compared to those from higher-income households
• Attrition: Loss of participants from the study sample over time, which can bias impact estimates if it differs between treatment and control groups

Types of Impact Evaluation

• Experimental designs (randomized controlled trials)
  • Considered the gold standard for impact evaluation
  • Randomly assign units to treatment and control groups to minimize selection bias
  • Example: Evaluating the impact of a new vaccine by randomly assigning individuals to receive the vaccine or a placebo
• Quasi-experimental designs
  • Use non-random methods to construct a comparison group that is similar to the treatment group
  • Include methods such as propensity score matching, regression discontinuity, and difference-in-differences
  • Example: Evaluating the impact of a job training program by comparing outcomes for participants to a matched group of non-participants with similar characteristics
• Non-experimental designs
  • Do not involve a comparison group and rely on observational data and statistical methods to estimate impacts
  • Include methods such as pre-post comparisons and cross-sectional regression analysis
  • Generally considered less rigorous than experimental and quasi-experimental designs

Steps in Conducting an Impact Evaluation

1. Identify the research question and hypotheses
   • Clearly define the intervention, target population, and outcomes of interest
   • Develop a theory of change that outlines how the intervention is expected to affect outcomes
2. Design the evaluation
   • Select an appropriate evaluation design based on the research question, context, and available resources
   • Determine the sample size and sampling strategy
   • Develop data collection instruments and protocols
3. Collect baseline data
   • Gather data on key outcome variables and potential confounding factors before the intervention begins
   • Ensures that treatment and control groups are balanced on observable characteristics at the start of the study
4. Implement the intervention and monitor implementation fidelity
   • Ensure that the intervention is delivered as intended and consistently across treatment units
   • Monitor and document any deviations from the intended intervention design
5. Collect follow-up data
   • Gather data on key outcome variables and potential confounding factors at one or more points after the intervention has been implemented
   • Timing of follow-up data collection should be based on the expected timeline for intervention impacts to materialize
6. Analyze the data and estimate impacts
   • Use appropriate statistical methods to estimate the causal impact of the intervention on outcomes of interest
   • Conduct subgroup analyses to explore heterogeneous impacts and identify potential mechanisms
7. Interpret and disseminate results
   • Summarize key findings and implications for policy and practice
   • Communicate results to relevant stakeholders, including policymakers, funders, and implementing partners
   • Contribute to the broader evidence base on what works in development

Common Methods and Approaches

• Difference-in-differences
  • Compares changes in outcomes over time between a treatment group and a comparison group
  • Assumes that in the absence of the intervention, the treatment and comparison groups would have experienced parallel trends in outcomes
  • Example: Evaluating the impact of a new minimum wage law by comparing changes in employment in states that implemented the law to states that did not
• Propensity score matching
  • Constructs a comparison group by matching treatment units to non-treatment units with similar predicted probabilities of receiving the intervention (propensity scores)
  • Assumes that selection into the treatment is based on observable characteristics that can be balanced through matching
  • Example: Evaluating the impact of a microcredit program by matching borrowers to non-borrowers with similar demographic and socioeconomic characteristics
• Regression discontinuity design
  • Exploits a cutoff or threshold that determines assignment to the treatment based on a continuous variable (running variable)
  • Compares outcomes for units just above and below the threshold, assuming that they are similar in all respects except for treatment status
  • Example: Evaluating the impact of a scholarship program by comparing outcomes for students just above and below the eligibility cutoff based on test scores
• Instrumental variables
  • Uses an exogenous source of variation (instrument) that affects participation in the treatment but does not directly affect outcomes to estimate impacts
  • Assumes that the instrument only affects outcomes through its effect on treatment participation (exclusion restriction)
  • Example: Evaluating the impact of a health insurance program by using distance to the nearest enrollment center as an instrument for insurance take-up

Challenges and Limitations

• Ethical considerations
  • Random assignment to treatment and control groups may not be feasible or appropriate in some contexts
  • Withholding potentially beneficial interventions from the control group can raise ethical concerns
• External validity
  • Impact estimates from a specific context or population may not generalize to other settings or groups
  • Replicating and scaling up successful interventions may require adapting to local contexts and implementation challenges
• Spillover effects and contamination
  • Treatment effects may spill over to the control group or neighboring communities, biasing impact estimates
  • Contamination can occur if the control group is exposed to the intervention or similar programs
• Measurement and reporting biases
  • Self-reported data may be subject to social desirability bias or recall errors
  • Implementers or evaluators may have incentives to overstate program impacts or underreport negative findings
• Cost and feasibility
  • Rigorous impact evaluations can be time-consuming and resource-intensive, particularly for experimental designs
  • Sufficient sample sizes and statistical power may be difficult to achieve for rare outcomes or small target populations
• Long-term impacts and sustainability
  • Impact evaluations often focus on short-term outcomes and may not capture longer-term impacts or the sustainability of program effects
  • Measuring impacts over longer time horizons can be challenging due to attrition, changes in context, and resource constraints

Real-World Applications

• Education
  • Evaluating the impact of teacher training programs on student learning outcomes
  • Assessing the effectiveness of school-based interventions to improve attendance and reduce dropout rates
• Health
  • Measuring the impact of community health worker programs on maternal and child health outcomes
  • Evaluating the effectiveness of interventions to improve access to and utilization of health services
• Agriculture
  • Assessing the impact of agricultural extension services on farmer productivity and income
  • Evaluating the effectiveness of interventions to promote the adoption of improved agricultural technologies and practices
• Social protection
  • Measuring the impact of cash transfer programs on poverty, food security, and human capital outcomes
  • Evaluating the effectiveness of job training and employment programs for vulnerable populations
• Governance
  • Assessing the impact of community-driven development programs on local governance and citizen participation
  • Evaluating the effectiveness of interventions to improve public service delivery and reduce corruption