EC 402 - Impact Evaluation / Causal Inference Methods
Introductory Lecture
Sugata Bag
Delhi School of Economics
January 29, 2024
1 / 28
Causal Idea: Is it really a cause?
- Think of correlation between
number of cats in a locality and
number of potholes in a
locality.
What an impact!!
2 / 28
Causal Idea: Is it really a cause?
- Think of correlation between
number of cats in a locality and
number of potholes in a
locality.
- Now think that the ducks have
become so powerful, they are
breaking iron gates!
Wow!! Even better!!
2 / 28
What is the course about?
- OK, then ... “Correlation does not imply causation.”
- Then what is causation?
- When can we infer causation from an estimated regn coefficient?
3 / 28
What is causation?
Some food for thought!!
4 / 28
When can we infer causation from an estimated regn coefficient?
- An Example from MX (ch.3), Yule (1899) interested in causes of poverty in England
Each of the variables is expressed as an annual growth rate. So, each regression coefficient has elasticity interpretations
- Thus public assistance Out-relief increased pauper growth rates
- What’s wrong here?? This Course is all about that
5 / 28
What is the course about?
- “Correlation does not imply causation.” Then what is causation?
- Addressing the question within the context of the analysis of various interventions or
policies.
- Understanding the implications of and need for causal inference, and its relationship with
data.
- Objective: Make better sense of data, be it big or small.
- Highlighting the role of counterfactual analysis in causal inference.
- Methodology overview & application demonstration with (India-based) examples.
- Critique/discussion of limitations of the methodology/application.
6 / 28
What is it not about?
- Not a substitute for a full-fledged econometric methods course.
- Far narrower scope.
- Will not teach Stata nor R. Main textbook by Cunningham has detailed code.
- Replication exercises may be assigned, but very little or no coding assistance provided.
7 / 28
What is it not about? ... For Coding resources
Codes will not be provided. However, you can have a look at the followings -
- Introduction to Base R:
https://cran.r-project.org/doc/manuals/r-release/R-intro.pdf
- Kieran Healy, Data Visualization: A Practical Introduction, https://socviz.co/
- Cameron, A. Colin and Pravin Trivedi (2005) / (2022). Microconometrics: Methods and
Applications. Excellent materials, including codes, for STATA users and beginners
8 / 28
Course Prerequisite, Assumed Background
- This course is targeted towards PhD researchers / MA (4th Sem) students interested
in empirical work
- The course assumes somewhat exposure to econometrics material already.
- Students completed EC401 will have better grip.
- I assume the first year sequence here at DSE
- This is not because the material is deeply technical, but because I want to be able to
assume some basic fluency in statistical concepts
9 / 28
Readings & Texts
- See syllabus for details
- There are no mandatory readings, but the papers listed in the syllabus are relevant to
the material we will cover in class. Good to go through them.
10 / 28
Readings & Texts
- See syllabus for details
- There are no mandatory readings, but the papers listed in the syllabus are relevant to
the material we will cover in class. Good to go through them.
- Textbooks: I recommend the following texts:
- Cunningham, 2021. Causal Inference: The Mixtape. [MX]
https://mixtape.scunning.com/
- Angrist and Pischke, 2009. Mostly Harmless Econometrics [MHE]
- Hansen, 2021. Econometrics [BH]
For everything related to theories and ideas
- Imbens and Rubin, 2015. Causal Inference for Statistics, Social, and Biomedical Sciences
[IR]
10 / 28
Assessment / Exams
- Midterm Exam or Assignments
- NOT SURE AT THIS MOMENT
- This is endogenous to class size
- Endterm Exam - Yes
Either 50 or 70 marks, depending upon university rules
- OPTIONS
- Two/ one Assignments (15 marks each, one every month, done in groups).
- Midterm (15 marks) OR term paper in groups of 2 (limited opt-ins).
- Volunteer presentations of term papers in class (20 minutes + 10 minutes discussion, no
marks but aids in learning).
- For PhD students There will be term-paper/replication work for internal assessments
11 / 28
Course Logistics
- Office hours: Course days, before/after lecture. Or through appointments.
- Lecture slides and other reading materials will be provided via Piazza page for the
enrolled students.
- No video recordings of lectures. (Please do not record)
12 / 28
Structure of the course
See Syllabus for detailed info
1. Causality, Statistics, and Economics
1.1 Potential Outcomes and Directed Acyclic Graphs
1.2 Research Design, Randomization, and Model- vs. Design-Based Inference
1.3 Matching (Propensity Scores)
1.4 Interference, Spillovers and Dynamics
2. Likelihood Methods
2.1 Binary Discrete Choice, GLM and Computational Methods
2.2 Multiple Discrete Choices
2.3 Duration Models
13 / 28
Structure of the course... contd
3. Canonical Research Designs
- Instrumental Variable
- Quantile regressions
- Regression Discontinuity
- Difference in Difference
- Synthetic Controls
- Partial Identifications (Bounds)
- Machine Learning (basic)
14 / 28
Important caveat
- In the end, this is an advanced course targeted at making you a good empirical
researcher
- My goal is to expose you to a wide range of empirical methods, and understand how
they connect to research question at hand.
- We will not drill down deeply into some material
- I am happy to discuss it more outside of class
- I will also emphasize how to communicate the econometrics underlying your research
ideas
- This includes good graphic design!
15 / 28
Interested in this course?
- Shall we begin?
16 / 28
Motivation and Terminology
Lec 0: Basic Understanding and Terminology
17 / 28
What is impact assessment/evaluation?
- Several definitions exist, but we can start with:
- Impact Assessment seeks to document whether “changes in well-being are indeed due
to program intervention ... and to what extent the measured effect can be attributed
to the program and not to some other causes.”
∼ (Khandker et al., 2010. p. 18. Handbook of IE, WBG).
- The focus is on the impact of interventions, whether small or large scale
18 / 28
Distinction: Impact Assessment vs Impact Evaluation?
- IE is a broader concept than IA.
- The OECD has a set of five criteria for evaluation of a development intervention:
relevance, effectiveness, efficiency, impact and sustainability.
(Chianca 2008, http://tinyurl.com/22pwhysu)
- One way to characterize impact evaluation is to examine
“The positive and negative changes produced by a development intervention, directly or
indirectly, intended or unintended. This involves ... effects ... on the local social, economic,
environmental and other development indicators.” (Chianca, 2008, p. 43).
- In reality the distinction is getting blurred, but in principle, and IA has a more tightly
focused set of questions that it seeks to answer.
19 / 28
Distinction: Impact Assessment vs Impact Evaluation?
- IE is a broader concept than IA.
- The OECD has a set of five criteria for evaluation of a development intervention:
relevance, effectiveness, efficiency, impact and sustainability.
(Chianca 2008, http://tinyurl.com/22pwhysu)
- One way to characterize impact evaluation is to examine
“The positive and negative changes produced by a development intervention, directly or
indirectly, intended or unintended. This involves ... effects ... on the local social, economic,
environmental and other development indicators.” (Chianca, 2008, p. 43).
- In reality the distinction is getting blurred, but in principle, and IA has a more tightly
focused set of questions that it seeks to answer.
19 / 28
Criteria for causation
- 1. Temporal sequence: cause precedes effect
- 2. Consistency: association is found repeatedly in different conditions
- 3. Gradient: if a dose/treatment response is present, causation is more plausible
- 4. Specificity of effect: occurs if the introduction of cause produces effect,
AND its removal implies absence of effect
- 5. Supporting evidence of plausible causal mechanism.
20 / 28
Counterfactuals and identification of causal impacts
- Measuring causal impact requires an identification strategy:
- THAT MEANS: Can impacts be attributed to the policy/treatment/intervention?
-
- Thus need to compare outcomes: for those exposed to the intervention (factual)
with those that would have obtained without the exposure (counterfactuals).
Comparisons must be like-to-like, and based on a valid counterfactuals.
- Counterfactual capture what would have happened to the same individual /household
/firm in the absence of the intervention.
It cannot be observed & must be constructed using an appropriate comparison group.
Ex: Effect of taking (or Not taking) Dispirin on severity of headache (for same person!!)
21 / 28
Counterfactuals and identification of causal impacts
- Measuring causal impact requires an identification strategy:
- THAT MEANS: Can impacts be attributed to the policy/treatment/intervention?
-
- Thus need to compare outcomes: for those exposed to the intervention (factual)
with those that would have obtained without the exposure (counterfactuals).
Comparisons must be like-to-like, and based on a valid counterfactuals.
- Counterfactual capture what would have happened to the same individual /household
/firm in the absence of the intervention.
It cannot be observed & must be constructed using an appropriate comparison group.
Ex: Effect of taking (or Not taking) Dispirin on severity of headache (for same person!!)
- Counterfactual interpretation: Mathematically, A → B ⇒∼ A →∼ B.
i.e. A causes B ⇒ if A had not occurred, then B would not have occurred either
- Counterfactual is not the same as a control group, but often used interchangeably.
- Unless constructed carefully, before-after or with-without studies do not form valid
counterfactuals
21 / 28
IE: Pre and Post Comparisons - Example
- Q. consider the PDS:
Did the PDS (or reform of the PDS)
result in higher intakes of food?
Two periods: Pre & Post
- Observed Outcomes:
A for Pre-regulation and
C for Post-regulation
22 / 28
IE: Pre and Post Comparisons - Example
- Q. consider the PDS:
Did the PDS (or reform of the PDS)
result in higher intakes of food?
Two periods: Pre & Post
- Observed Outcomes:
A for Pre-regulation and
C for Post-regulation
.
Another Q: Did agri-deregulation result in productivity growth?
If you simply compare states before and after they deregulated, will it work?
Is there any other benchmark? Say, states where reform did not take place at all
22 / 28
IE: Comparisons of With- and Without- treatment; Cross-sec data
- Q: Did ICDS participation help in
improving diets of children?
- The observed W & WO values in graph
would suggest the ICDS had a apparent
negative impact of (A-C) [and, not (C-A)].
But is that true measure of impact?
- Fact: average food intakes of children
participating in ICDS was/ might be
lower than the food intakes of those
who did not participate in the ICDS.
So, if those who participated were from poorer households who cannot afford adequate
diets, the relevant counterfactual is intakes of children absent the ICDS.
True impact is (A − B ), a positive effect.
23 / 28
Why Standard regression representation problematic?
- Suppose we estimated regressions of the “first-difference” form:
Yi = α0 + α1 Di + ϵi
- where Yi is the outcome of interest, Di is a dummy variable that takes value 1 if the
unit/individual receives the treatment/intervention and 0 if not.
- i.e. E[Yi ] = α0 if Di = 0 (non-treated), and
E[Yi ] = α0 + α1 if Di = 1 (treated)
- OLS α̂1 is the observed difference between those who received the intervention and
those who did not;
it is likely biased and therefore may not be vested with causal interpretation because:
- Omitted variables (observed) correlated with Di
- Omitted variables (unobserved) correlated with Di
- ...
24 / 28
IE: Internal validity
- An ideal impact evaluation has internal validity.
- There is internal validity if differences in outcomes between intervention and
control groups – used to compute impact – are based on valid counterfactuals.
- It answers the question: Does effect stem from cause (and not some other factors)?
Requires:
- Knowledge of underlying causal pathway
- Knowledge of assignment mechanism (e.g. RCT, ”as if” randomized, etc.)
- Ability to test a null hypothesis that can be falsified with data (is the sample size large
enough to detect impact)
25 / 28
IE: External validity
- An ideal impact evaluation also has external validity.
There is external validity if the evaluation sample can be extrapolated to the
population of individuals/units eligible to receive the intervention/treatment.
- It answers the question: To what extent can the causal inference be generalizable to
other situations?
- This includes geographical extrapolation.
- It encompasses sub-population based heterogeneity. How is external validity to be
achieved at a low cost?
26 / 28
Causal Inference
- Causal Inference is a broader term that attempts to ensure that a key parameter of
interest, say the ‘impact’ of ‘college education’ on wages, is identified.
- More example: Effect of early marriage on female labour force participation / child
bearing / IPV.
- (Note that there need not be any intervention).
- Much of this course focuses on impact assessment and causal inference.
27 / 28
Next Lecture
- Potential Outcomes Framework (Read: MX ch.3)
- Conditional Expection Theory (Read: MHE ch.3 sec 3.1.1 - 3.2.1)
28 / 28
