See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/296702538

What is impact?

Working Paper · February 2016

DOI: 10.13140/RG.2.1.4799.9766

CITATION READS

1 20,820

2 authors, including:

Simon Hearn
Overseas Development Institute
19 PUBLICATIONS 171 CITATIONS

SEE PROFILE

All content following this page was uploaded by Simon Hearn on 04 March 2016.

The user has requested enhancement of the downloaded file.

 February 2016

What is impact?
Simon Hearn and Anne L. Buffardi

• Impact is a multi-dimensional concept. Some definitions focus on very precise understandings of

Key impact, while others cast a much broader net.
messages • How impact is defined and used has a significant effect on the design, management and
evaluation of development programmes.
• Development programmes should hold explicit conversations with different stakeholders about
how impact is used and understood, in order to come to a shared understanding.
• There are six dimensions of impact that may help development programmes be clearer about
what they mean.

A Methods Lab publication

odi.org/methodslab

developmentprogress.org
The Methods Lab is an action-learning
collaboration between the Overseas Development
Institute (ODI), BetterEvaluation (BE) and the
Australian Department of Foreign Affairs and Trade
(DFAT). The Methods Lab seeks to develop, test,
and institutionalise flexible approaches to impact
evaluations. It focuses on interventions which are
harder to evaluate because of their diversity and
complexity or where traditional impact evaluation
approaches may not be feasible or appropriate,
with the broader aim of identifying lessons with
wider application potential.

Readers are encouraged to reproduce Methods Lab

material for their own publications, as long as they
are not being sold commercially. As copyright holder,
ODI requests due acknowledgement and a copy of the
publication. For online use, we ask readers to link to
the original resource on the ODI website. The views
presented in this paper are those of the author(s) and
do not necessarily represent the views of ODI, the
Australian Department of Foreign Affairs and Trade
(DFAT) and BetterEvaluation.

© Overseas Development Institute 2016. This work

is licensed under a Creative Commons Attribution-
NonCommercial Licence (CC BY-NC 4.0).

How to cite this working paper:

Hearn, S. and Buffardi, A.L. (2016) ‘What is impact?’.
A Methods Lab publication. London: Overseas
Development Institute.

Overseas Development Institute BetterEvaluation

203 Blackfriars Road E-mail: [email protected]
London SE1 8NJ www.betterevaluation.org
Tel +44 (0) 20 7922 0300
Fax +44 (0) 20 7922 0399
[email protected]
www.odi.org
Contents

Acknowledgements 4
Acronyms 4
1. Introduction 5
2. The importance of impact in international development 6
3. Who says what about impact? 7
4. How to make sense of the variations 9
4.1 What kinds of impact are there? 9
4.2 How is the term impact used in practice? 9
4.3 How do characteristics of impact vary? 12
5. Conclusion 14
References 15

What is impact? 3  

Acknowledgements
The authors thank staff involved in DFAT Methods Lab programmes for their insights about impact; Rick Davies, Nancy
MacPherson and participants at the ODI public event ‘Beyond Methods: Unpacking Evaluation Challenges’ and the panel
session at American Evaluation Association Conference ‘Measurement and meaning in a messy world’ for their thoughtful
comments and ideas; Irene Guijt, Tiina Pasanen, Nat Mason and Nina Hall for valuable suggestions on previous drafts; Andre
Ling and Ricardo Wilson-Grau for their own thoughts on impact which provided inspiration for this paper; and Louise Ball,
Hannah Caddick and Steven Dickie for excellent editorial assistance.

Acronyms
3ie International Initiative on Impact Evaluation
BRICS Brazil, Russia, India, China and South Africa
DFAT ODE Australian Department of Foreign Affairs & Trade Office of Development Effectiveness
DFID UK Department for International Development
EC European Commission
ECOWAS The Economic Community Of West African States
EU The European Union
G20 The Group of Twenty
GEF Global Environment Facility
IFAD International Fund for Agricultural Development
JPAL the Abdul Latif Jameel Poverty Action Lab
MDGs the Millennium Development Goals
OECD DAC Organisation for Economic Development, Development Assistance Committee
SDGs Sustainable Development Goals
UN United Nations
UNAIDS Joint United Nations Programme on HIV/AIDS
UNDG United Nations Development Group
USAID US Agency for International Development
WHO World Health Organisation

4 Methods Lab
1. Introduction
‘Impact: the action of one object coming forcibly The purpose of this discussion paper is not to
into contact with one another; a marked effect propose a single, universal definition of impact or to
or influence’ debate existing definitions. Instead, we examine how
Oxford Dictionary some definitions focus on very specific and precise
understandings of impact while others cast an extremely
It seems everyone is looking to achieve and demonstrate broad net. We then wade into the murky middle
impact. Private foundations talk of ‘impact investing’,1 between the two to explore ways in which ideas about
social change actors talk about ‘collective impact’2 and impact can be contextually grounded and its scope
‘social impact’,3 academics are being asked about their bounded to make measurement feasible. We aim to
‘research impact’.4 elevate the discussion about impact, moving beyond
The international development community is also the methodological debates that have dominated
increasingly preoccupied with impact. Since the early attention paid to impact so far, and present different
2000s, the terms ‘impact’ and ‘impact evaluation’ have perspectives and dimensions that can affect how impact
skyrocketed in use5 and have become common parlance could be framed and evaluated. Rather than arguing
among development practitioners and agencies. which definition is universally superior, we encourage
The premise of this paper is that the way in development programmes to structure an explicit
which impact is framed has a significant influence conversation about how different stakeholders conceive
on development processes and how programmes are of and are using the term impact in order to come to a
designed, managed and evaluated. For example, the shared understanding.
ways in which a programme is accountable for achieving The paper starts by looking at why impact is
intended impacts will affect the ambition of the design important in international development and how
and how its success is ultimately judged. Currently selected development agencies define the concept. Next,
there is too much ambiguity and confusion about what we present three ways to think about and approach
‘impact’ is, how it should be defined, how to measure it discussions about impact: the impact possibility
and what kind of measurement is sufficient. Evaluation continuum, how the term impact is used in practice, and
often serves as the process through which different dimensions along which impact can vary, which affect
definitions of impact surface, and sometimes very late in what is asked, measured, when and how frequently, and
the programme lifecycle. However, questions of who is how findings are interpreted.
defining impact and how development is being judged
are more fundamental matters that relate to, but are
larger than, a single programme evaluation.

1 www.thegiin.org/cgi-bin/iowa/home/index.html
2 www.ssireview.org/articles/entry/collective_impact
3 http://socialvalueint.org
4 www.ref.ac.uk
5 A number of specialised groups and initiatives emerged around this time: the Abdul Latif Jameel Poverty Action Lab (JPAL) in 2003, Center for
Global Development’s Evaluation Gap Working Group in 2004, International Initiative on Impact Evaluation (3ie) in 2008, Masters of Science
in Impact Evaluation for International Development at the University of East Anglia, and the Centre for Development Impact at the Institute of
Development Studies, among others. In 2012 DFID commissioned a study ‘Broadening the range of designs and methods for impact evaluations’
(Stern et al.). Cameron, Mishra and Brown (2015) document the dramatic increase of published impact evaluations as indexed and defined by 3ie
as ‘counterfactual-based programme evaluations that attempt to attribute specific outcomes to programmatic activities’ (p.1) (a subset of the total,
which would include other definitions of impact evaluation).

WHAT IS IMPACT? 5  

2. The importance of impact in
international development
Development practitioners and agencies have long impact can be used to clarify a vision through which to
sought to achieve impact with their work; they are often build cooperation and coordinate action; assessments
highly and intrinsically motivated to create change. of potential impact are used to identify possible risks or
Measuring and demonstrating impact, however, has adverse effects (i.e., environmental impact assessments);
not always been high on the agenda. With hindsight, ambitions of impact are used to make decisions about
this may seem odd but for decades, development which programmes to fund; they establish expectations
assessments were dominated by outputs – such as the of achievement by which success will be defined; and
number of trainings held or goods distributed. Showing these in turn are used to plan appropriate inputs and
you were doing what you promised you would was strategy. During or after a programme, measurements
sufficient for funders, and impact was more or less of impact are used to determine to what extent the
presumed to follow. intervention achieved what it set out to achieve;
The growing importance of demonstrating impact determine other effects, positive or negative; decide
in international development over the past 10-15 whether to stop, continue, scale up, replicate or adapt
years has been driven by a number of economic and the intervention; and to draw lessons for other similar
political factors: recently there has been a reduction interventions.
or retargeting of development budgets in many donor The way that impact is defined and understood,
agencies. There has been a drive among major donors therefore, has widespread implications. It affects how
for greater demonstration of ‘value for money’ and a programme is perceived and how people will want
wanting to get more ‘bang for their buck’. At the same to or are expected to be involved. It affects which
time, there is an increasing public perception that five programmes get funded and the level of risk a funder
decades of development assistance have not had effects is willing to tolerate. It affects how programmes are
as hoped. This perception has pressured donors to do a designed, the strategies they take, and how ambitious
better job of demonstrating clear, tangible results that they are. It affects the way in which programmes will be
can be understood by the general public. judged, who takes the credit and who takes the blame
The evidence-based policy movement, which has gained for particular outcomes. It affects what can be learned
momentum over the past few years, has led to more from one programme and applied to another. It affects
systematic examination of some of the main assumptions the view of the world in which a programme operates.
underlying development work. Together, these trends Given the implications of different conceptions of
have led to much greater attention among development impact, there is a strong imperative to be very clear
actors to measure and demonstrate what works more and about what we mean when we use this term and to
less well, and to use this knowledge to leverage greater use it carefully. But as we examine next, there is wide
effectiveness from development programmes. Impact has variation across the development sector in the definition
become the watch word for this shift. and use of it, which contributes to confusion and, in
Impact is a concept that is used for many purposes some cases, conflict.
and at all stages of development programming. In
planning a programme, discussion about intended

6 Methods Lab
3. Who says what about impact?
Despite heightened attention paid to, expectations Among researchers and evaluators in the international
around, and use of the term impact, the development development community, discussion about impact has
community does not have a shared definition of what predominantly been methodological. Most evaluation
constitutes impact. scholars assert that programme theory and evaluation
Box 1 illustrates the diversity in how different questions should drive the choice of methods and
development organisations define the concept. This list state the importance of considering a mixed methods
includes several large bilateral agencies, multilateral funds approach. They differ, however, in the methods they
and programmes in the environmental, agriculture and consider to be sufficiently robust to be able to claim
health sectors, selected to demonstrate a cross section of impact and in their relative emphasis of participatory,
agencies. The first two definitions from the Organisation quantitative and qualitative data collection and analyses
for Economic Co-operation and Development – (Chambers, Karlan, Ravallion and Rogers 2009, White
Development Assistance Committee (OECD-DAC) and 2009). There have been lively debates about the role
the World Bank are the most commonly cited and in of experimental and quasi-experimental methods to
many ways represent opposite ends of the spectrum. evaluate impact, alternative approaches to casual
The scope of the World Bank’s definition is tightly inference (many of which, it has been noted, have been
bounded, ‘the indicator of interest with and without the infrequently or not applied in a development context),
intervention: Y1 - Y0’. In contrast, OECD-DAC, echoed by what constitutes attribution and contribution, the
several other agencies, takes a much broader approach: relative weight of internal and external validity, and
‘positive and negative, primary and secondary long-term consideration of implementation and programme theory
effects produced by a development intervention, directly failures as well as methodological ones (White 2010,
or indirectly, intended or unintended’ – a seemingly Stame 2010, Stern et al. 2012, Befani, Ramalingam and
limitless definition. Stern 2015).
Many development programmes likely fall between The lack of a consistent definition and technical
these two ends of the spectrum, with multiple debates about methods have led to confusion among
intervention components and change pathways aimed the donors and implementation staff we have interacted
at having an impact on more than a single indicator with, and trepidation that their understanding of
of interest, and operating within financial and time impact is not the ‘correct’ interpretation. Moreover,
limitations that render measurement of all possible these debates have focused attention on technical,
options and indicators infeasible. The scope of the methodological issues and shifted discussion away
definition of impact, and subsequent evaluations from relational and political matters. It is these
determining the extent to which they were achieved and latter questions – who is defining impact and how is
why, must therefore be appropriately bounded. development being judged – that this paper aims to
make clear.

7 Methods Lab What is impact? 7  

Table 1: Definitions of impact according to different international development organisations

Organisation Definition
Organisation for Economic Co-operation ‘Positive and negative, primary and secondary long-term effects produced by a development intervention,
and Development – Development directly or indirectly, intended or unintended’.
Assistance Committee (OECD-DAC)6, also
used by the UK Department for International
Development (DFID)7
World Bank (as cited by White 2009) ‘The difference in the indicator of interest (Y) with the intervention (Y1) and without the intervention (Y0). That
is, impact = Y1 - Y0.’
International Initiative for Impact Evaluation ‘How an intervention alters the state of the world. Impact evaluations typically focus on the effect of the
(3ie)8 intervention on the outcome for the beneficiary population.’
Australian Department of Foreign Affairs & ‘Impacts are positive or negative changes produced by a development intervention–directly or indirectly,
Trade Office of Development Effectiveness intended or unintended–in the context of its environment, as it interacts with the multiple factors affecting
(DFAT ODE)9 development change.’
US Agency for International Development ‘A results [sic] or effect that is caused by or attributable to a project or program. Impact is often used to refer
(USAID)10 to higher level effects of a program that occur in the medium or long term, and can be intended or unintended
and positive or negative.’
European Commission (EC)11 ‘In an impact assessment process, the term impact describes all the changes which are expected to happen
due to the implementation and application of a given policy option/intervention. Such impacts may occur over
different timescales, affect different actors and be relevant at different scales (local, regional, national and EU).
In an evaluation context, impact refers to the changes associated with a particular intervention which occur
over the longer term.’
United Nations Development ‘Impact implies changes in people’s lives. This might include changes in knowledge, skill, behaviour, health or living
Group (UNDG)12 conditions for children, adults, families or communities. Such changes are positive or negative long term effects on
identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended.
These effects can be economic, socio-cultural, institutional, environmental, technological or of other types. Positive
impacts should have some relationship to the Millennium Development Goals (MDGs), internationally-agreed
development goals, national development goals (as well as human rights as enshrined in constitutions), and national
commitments to international conventions and treaties’.
Global Environment Facility (GEF)13 ‘A fundamental and durable change in the condition of people and their environment brought about by the project’
International Fund for Agricultural ‘The changes in the lives of rural people, as perceived by them and their partners at the time of evaluation, plus
Development (IFAD)14 sustainability-enhancing change in their environment to which the project has contributed. Changes can be positive
or negative, intended or unintended. In the logframe terminology these “perceived changes in the lives of the people”
may correspond either to the purpose level or to the goal level of a project intervention.’
World Health Organisation (WHO)15 ‘Improved health outcomes achieved. The overall impact of the Organization sits at the highest level of the results chain,
with eight impact goals. Outcomes can combine in different ways to contribute towards one or more impacts.’
Joint United Nations Programme on HIV/ ‘The long-term, cumulative effect of programs/interventions over time on what they ultimately aim to change, such
AIDS (UNAIDS)16 as a change in HIV infection, AIDS-related morbidity and mortality.
Note: Impacts at a population-level are rarely attributable to a single program/intervention, but a specific program/
intervention may, together with other programs/interventions, contribute to impacts on a population.’

6 OECD (2002). Glossary of Key Terms in Evaluation and Results Based Management. Available at: http://bit.ly/1KG9WUk
7 Independent Commission for Aid Impact (ICAI). DFID’s Approach to Delivering Impact: Terms of Reference. Available at: http://bit.ly/1noYc3e
8 3ie (2012). Impact Evaluation Glossary. Available at: http://bit.ly/1Urz3zU
9 AusAID Office of Development Effectiveness. 2012. Impact Evaluation: A Discussion Paper for AusAID Practitioners. Available at: http://bit.ly/1noYk2P
10 USAID (2009). Glossary of Evaluation Terms. Available at: http://1.usa.gov/1Tow8cN
11 European Commission. Glossary. Available at: http://bit.ly/2060g24
12 United Nations Development Group (2011). Results-Based Management Handbook. Available at: http://bit.ly/1nPVO65
13 Global Environment Facility (2009). The ROtI Handbook: Towards Enhancing The Impacts of Environmental Projects. Available at: http://bit.ly/1WOyXE1
14 International Fund for Agricultural Development. Glossary of M&E Concepts and Terms. Available at: http://bit.ly/1Towa4D
15 World Health Organisation. The results chain. Available at: http://bit.ly/1VsVAN4
16 Joint United Nations Programme on HIV/AIDS. Glossary: Monitoring and Evaluation Terms. Working draft. Available at: http://bit.ly/1ZTLy8M

8 Methods Lab
4. How to make sense of
the variations
Given the range and breadth of definitions, this paper Considering each of these possibilities can help to
suggests three ways to approach discussions of what break down an all-encompassing definition. For example,
constitutes impact. The first approach is to look across if a programme is implemented according to a pre-
the definitions at what they suggest as different types of planned design in a stable context following predictable
impact or possible places to look for it. The second is strategies then a case may be made to limit the evaluation
to examine the different ways in which the term impact to foreseen, intended impacts only (top-left). This may
is used in practice and the kinds of values that are not be the case, however, and it is likely that risks will
implied in each use. The third is to consider different be identified which will need to be monitored to asses if
dimensions that vary across these different definitions they were adequately managed (top-right). It is also likely,
and interventions, including the direction, subject and in many contexts, that unexpected things will happen
level of change, degrees of separation, timescale, rate outside control of the programme and the impact of these
and durability of change and homogeneity of benefits. will also need to be assessed (bottom right and bottom
middle). If a programme is likely to be adaptive and
evolve over time, then evaluating based on initial goals
4.1 What kinds of impact are there? will be insufficient and the evaluation will have to take
From the broadest definition, as taken from the OECD- into account emergent goals (bottom-left).
DAC, impact is seen to be intended or unintended,
positive or negative. We can also infer that specific impact
might have been foreseen ahead of time or unforeseen. 4.2 How is the term impact used
This represents the realm of possibilities of the kinds in practice?
of impact a programme might have. Table 2 shows the Beyond the official definitions used by development
impact possibility continuum, which, assuming intended agencies, it is also important to consider how the term
impacts are always positive,17 yields six kinds of impact: impact is used in practice by different stakeholders.
planned programme goals, emergent programme goals, Development practitioners may interpret medium-term
predictable (positive) spill-over effects, nice surprises, outcomes, long-term outcomes and impact in quite
predictable risks or side-effects and nasty surprises different ways. We identify five main uses,18 which overlap
(backlash, mishap or calamity). with the definitions above but which also include more
utilitarian functions.

Table 2: Impact possibility continuum (adapted from Ling 2014) with examples from a job skills programme to reduce
unemployment among young people in a rural district

Intended Positive unintended Negative unintended

Foreseen Planned programme goals, e.g., Predicted spill over effects, e.g., Predicted risks or side-effects, e.g.,
decrease in unemployment rates investment in local business increases students from the programme migrate to
find better jobs
Unforeseen Emergent programme goals, e.g., Nice surprise, e.g., students from the Calamity, mishap or backlash, e.g.,
during implementation, the programme programme start to mentor their siblings youth not participating in the programme
realises the importance of increasing the and peers ostracise participating students and
diversification of products and income vandalise local businesses
sources and adds this as a goal

17 This assumption might be contested given that good intentions do not always yield positive effects, however, it is fair to assume that in international
development, no programme sets out to achieve negative effects. Hence any negative effect, even if it is foreseen, is assumed to be unintended.
18 Thanks to Ricardo Wilson-Grau for inspiring this categorisation of definitions, shared through the Outcome Mapping Learning Community.

What is impact? 9  

1. Counterfactual use. For many development economists Figure 1: Impact of a hypothetical intervention on
(e.g., White 2010; Duflo, Glennerster and Kremer 2006;
agricultural yield
Ravallion 2008) impact is a technical term with a specific
Y1

Agricultural yeild (Y)

definition that requires comparison with a counterfactual: Intervention
that is, what would have happened in the absence of the
programme? (White 2010). For example, if an evaluation
demonstrates that there was a significant increase in
average agricultural yields in the intervention village when
Y1* Counterfactual

compared to a village with similar characteristics that did

not receive the intervention, the impact attributed to the
programme would be the difference between agricultural
Y0
yields in the intervention and non-intervention sites (see
Figure 1). This definition examines the extent to which
an intervention caused a particular effect. It also narrows
impact down to a measurable change in a pre-specified
variable. In the definitions presented in section 3 (Table t0 t1 Time (t)
1), USAID considers impact as results that are attributable
to a programme. UNAIDS, on the other hand, notes that Note: The impact is the difference between Y1 (intervention) and Y1*
impacts at a population level are rarely attributable to a (counterfactual) at time t1 (adapted from Woolcock 2009)
single programme or intervention.
2. Boundless use. As noted, the OECD-DAC definition is the ocean (which may be another country). Figure 2
by far the broadest: positive and negative, primary and demonstrates the diversity of effects. When taken to its
secondary, direct or indirect, intended or unintended. ultimate conclusion, this definition is unlimited in scope,
Using this definition to examine the impact of dam allowing any and all effects to be considered, including
construction, for example, expands the potential scope spill-over effects. This very broad definition is flexible to
of inquiry beyond agricultural yield to consider other accommodate all types of development programmes. In
primary effects such as energy generation and consistent practice, programmes will have a core set of objectives
water supply. Secondary effects may include decreased they are aiming to achieve and, in order to measure
carbon dioxide emissions and better health and security impact in a systematic way, an operational definition
for villages connected to electricity and water supplies. would have to be developed to bound the scope of
Negative effects may include displacement of people in the inquiry into a more manageable size. Rather than
nearby and downstream communities whose homes or identifying the cause of an effect, this definition considers
farmland have been flooded, and effects on the ecosystem all possible effects of a cause,19 that is, the multiple effects
in the surrounding area and along the river valley to of a programme or intervention.

Figure 2: Example of a network of hypothetical impacts from the construction of a dam

Increased Altered
household water Food
Decreased morphology security of
and electricity of river
dependency access communities
on energy channel increased
imports

Dam
Increased Access
Agricultural
clean energy to irrigation
generation is built increases
yield
increases

Decreased
Income
carbon
of farming
dioxide Reduced
Displacement household
emmissions Reduced biodiversity
of communities increases
in flooded risk of along river
areas flooding valley

19 We thank Rick Davies for suggesting the distinctions between cause of an effect and effects of a cause.

10 Methods Lab
3. Results-chain use. Many approaches to development 4. Environmental sustainability use. The advent of
planning, monitoring and evaluation use a results the Sustainable Development Goals has brought
chain to illustrate the progression of levels of results environmental sustainability more prominently into
from inputs to activities to outputs to outcomes and focus. If sustainable development is the ambition
finally to impact (UNDG 2011). Logical frameworks for all development interventions then the impact
– one of the most ubiquitous, yet hotly contested – of these programmes should be framed in terms
management tools in international development, is of how it is meeting this ambition. For example,
based on this idea (DFID 2011). Results chains like a major unexpected impact of global economic
this define impact implicitly in terms of its relation development has been the rise in carbon dioxide
to other kinds of results, namely outputs, which levels and the resulting climate change. According
are direct effects of the intervention, and outcomes, to the recent international declaration, the world
which are short and intermediate term changes. For leaders committed to balancing the three dimensions
example, the outputs of the dam project might be of sustainable development: the economic, social and
the completed dam. The outcomes you would expect environmental (UN 2015). In this view, impact is
from this would be that farmers will have increased defined as the contribution to these goals, and needs
access to water, will use more water and grow more to consider effects on the economy, the environment
crops. The ultimate impact of these changes may be and social wellbeing. Among the agency definitions
that farmers earn more money, and they and nearby in section 3, and with the exception of the UN
communities are healthier as a result of greater access Development Group, Global Environment Facility
to nutritious food (see for example figure 3). and International Fund for Agricultural Development,
There are two common variations of the results-chain all other current definitions do not explicitly consider
approach which are worth noting. Firstly, the theory of the environment.
change approach allows complicated networks of results 5. Colloquial use. Impact is often used in common
to be mapped visually, rather than reducing the logic to language when talking about development
a single chain (Anderson, undated). Impact is sometimes interventions to mean the general effect of an action,
used in this approach to mean the long term outcomes as in ‘the dam’s construction had a huge impact on our
at the end of the network, although in many cases the family’ or ‘our event had little impact on the audience’.
term impact is not used. This use might be described as colloquial since it
Secondly, the spheres of influence approach, used is rarely intended to be taken as an evidence-based
in outcome mapping and elsewhere, incorporates judgement, and the effects that it is used to describe
concepts from systems thinking and places results are extremely broad and varied. Included here is the
in one of three ‘spheres’: the sphere of control, the use of impact as a vision statement to describe, in
sphere of direct influence and the sphere of indirect narrative form, the ideal situation which a programme
influence (sometimes termed concern or interest). aspires to bring about: ‘all people will live free, healthy
Impact is defined in this framework as being the and prosperous lives’. In this form, impact is used
results that fall outside the sphere of direct influence interchangeably with other words such as result,
(Montague et al. 2011). Therefore, it is beyond the outcome, effect and difference.20
control and influence of a particular development
programme.

Figure 3: Example results chain from the construction of a dam

Inputs Activities Outputs Outcomes Impact

Money Planning and Completed dam Farmers use more Food security
construction water and grow more increases
Flooded river valley
crops
Hunger decreases
Water irrigation supply
Household income
increases

20 As in ‘the new approach has made a big difference’.

What is impact? 11  

4.3 How do characteristics of impact vary? individual beneficiaries. In contrast, institutional capacity-
The definitions and uses of impact presented in this building programmes have several degrees of separation
paper demonstrate that organisations and individuals between the intervention with staff and institutional
have different interpretations of the term and value policies and end users, such as patients attending health
various aspects differently. In this section, we identify clinics. In the former, impact is relatively linear and the
six dimensions of impact and pose guiding questions to pathway is direct. In the latter, impact is more systemic;
help stakeholders clarify their interpretation and come it confronts and converges with other factors (contextual
to a shared understanding for a particular programme or programmatic), which, like two waves meeting in the
and context. Table 3 summarises the six dimensions ocean, can either resonate and produce greater effects or
and suggests more specific variations of each and their they can disturb each other producing chaotic effects.
implications for measurement. •• How direct is the causal chain? How far from the
intervention do you expect to see impact?
1. Application. Impact is a concept that is applied both 5. Immediacy, rate and durability of change. Many agency
prospectively and retrospectively. For example, in an definitions refer to long-term change. But how long is
environmental impact assessment, impact refers to the long-term? Different ‘arenas of change’ will have more
potential effects of an intervention on the environment and or less rapid manifestations of impact. In some cases,
can be used to help decide whether or not to proceed with impact may be more immediate – an accident caused
a planned course of action. In programme planning stages, by malpractice in an infrastructure programme can
impact can refer to the intended or desirable effects. In an have very direct and immediate consequences for those
impact evaluation, impact refers to measured or observed involved. A conflict prevention programme, on the other
effects of an intervention, which could help decide whether hand, may take years or decades to have an appreciative
to stop, continue, scale up or adapt the intervention. and observable effect. In addition to the length of time,
•• Which application are you referring to when you use impact may not be static; assessments of impact may
the term impact? come back with different results at different times. A
2. Scope. Impact can be defined in terms of very specific vaccine may provide immunity for life. Information
changes or it can be broad and open. Specific impact and education cannot be subsequently unknown or
focuses on a fixed number of pre-defined variables, such unlearned. In unstable environments, however, hard-
as household income, disease status or air quality, and won successes can quickly unfold and the situation can
statements of impact discuss about impact according to change very quickly.
these variables. Broad impact is not limited to pre-defined •• How soon are changes likely to manifest? Are they
variables but considers as many changes as makes sense permanent or temporary? How variable is impact
to make a useful judgement (and are feasible to measure), likely to be over time?
including variables that may not be foreseen. 6. Homogeneity of benefits. Impact can be measured as an
•• Are you looking for impact on specific variables or average effect across a population. Or, it can consider
will you include unintended or unforeseen effects? positive and negative effects separately and disaggregate
3. Subject and level of change. Among the agency definitions among different population groups and contexts.
in section 3, the UN, Global Environment Facility, •• Among whom are you looking for impact? How will
International Fund for Agricultural Development, World the impact of the programme vary across subgroups?
Health Organisation, UNAIDS and 3ie refer to changes How will mixed results be judged?
in people’s lives. Yet development programmes are
increasingly focused on more mezzo or macro levels, Each of these six dimensions has specific implications
intervening with groups, institutions and policies, rather for what, when and how frequently change is measured.
than or in addition to individuals. As mentioned above, Moreover, these dimensions and measurement implications
several agencies also include the environment as a potential are related. Assessing impact across multiple subjects
subject of change. and levels of change, among whom benefits may vary by
•• Where are you looking for impact? subgroup, will require longer amounts of time to observe
4. Degrees of separation between intervention and impact, changes and disaggregation of results. Other evaluation
as illustrated through results chain or logic model, which scholars examine in greater depth the implications of
is related to the subject and level of change. Interventions different programme attributes for programme design
operate at different distances from individual beneficiaries. and measurement.21 Table 3 presents an overview of
For example, there is a direct, immediate link between the dimension with guiding questions to help facilitate
a programme providing bed nets or vaccinations and discussions about impact among different stakeholders.

12 Methods Lab
Table 3: Summary of dimensions of impact

Dimension Range of options and examples Which aspects of measurement

are affected?
Application Projection: forecasting change (environment impact assessment) • Type of design
Prospective: explicit statement of intended change(s) at outset guides strategy, • Questions asked
management, monitoring and evaluation
Retrospective: evaluation designed and conducted at the end of an intervention
Scope • Mode of inference:
Specific, pre-defined, knowable Generally use deductive methods
Undefined, unexpected, unknowable Generally use inductive methods
Subject and level Subject or unit Level • Unit of measurement
of change • Individual • Local • Extent of confounding factors
• Household • State/province • Sampling
• Community group (women’s • National
cooperative, natural resource • Regional (ECOWAS, Mekong
management committee) Valley)
• Network (i.e., Tax Justice Network) • Continental (African Union,
• Institution (civil society organisation, European Union)
government agency, business) • Global (SDGs, G20, BRICS, High
• Population (i.e., HIV incidence, national Level Fora on Aid Effectiveness)
economic growth)
• Sector (civil society, public sector,
private sector)
• Policy
• Environment
Degree of separation Direct and fairly certain, e.g. health screening and treatment will reduce disease • Number of measurements along
(number of steps in Direct but uncertain, e.g. advocacy with legislators will lead to policy change the change pathway
causal pathway from • Extent of confounding factors,
Moderate, e.g. improved access to markets will increase sales, which will
activity to impact) strength of causal claims and
increase income
need to account for alternative
Distant and uncertain, e.g. vulnerable groups will participate in community
explanations
groups, which will increase their influence in decision-making, which will improve
pro-poor policy-making, which will improve livelihoods
Immediacy, rate and Time frames Permanence • Timing of measurement
durability of change • Immediate- to short-term change, • Irreversible, e.g. education • Frequency of measurement
e.g. medication treating infection • Reversible/changeable, e.g. • Effect of measurement on the
• Seasonal, e.g. agricultural yields income, job status, agricultural measured
• Short- to medium-term, yields, bacterial infection
e.g. increased income • More durable but still reversible,
• Medium- to long-term, e.g. institutional changes and
e.g. sustained business norms, rights, improved natural
resource management
• Long-term, e.g. gender norms
Homogeneity Predominantly uniform benefits, e.g. vaccination, education and training • Extent of disaggregation
of benefits Variable benefits, e.g. outcomes that are relationship-dependent where certain • External validity/ generalisability
(How consistent are groups may have differential access or be treated differently (market linkages,
benefits across groups? community decision-making)
Do benefits for one
Rival benefits, e.g. access to markets (increased supply of the same products
group affect benefits of
could reduce prices for all farmers)
another?)

What is impact? 13  

5. Conclusion
This paper highlights the existence of different definitions interventions has also shifted over the past two decades,
and interpretations of impact, within a context where away from the delivery of goods and services and
attention to impact is increasingly prominent and more towards supporting systemic change, focusing on
instrumental in shaping the funding, design, management issues such as governance, political economy and policy
and evaluation of development programmes. First and dialogue (Gillies and Alvarado 2012). The goals of these
foremost, we call upon development practitioners and programmes are not solely about improving people’s lives
evaluators to be explicit in how they use the term. Formal but about ensuring that their lives and the contexts in
definitions are abundant but it is clear that in practice which they are living can continue to be improved, which
people have different understandings; it cannot be implies different kinds of impact.
assumed that others will know what is meant when the This paper presents three approaches that can serve as
term impact is used. Some might argue for an outright the basis for discussions among development stakeholders
ban on the word impact because it carries with it so much to clarify how they are conceiving of impact: the impact
baggage and misunderstanding. While this may not be possibility continuum, five ways the term is used in
possible, it should be feasible to enrich the use of the practice and six dimensions along which impacts can
term with more nuance about what we mean – using the vary. No one approach or definition will be universally
alternative terminology discussed in this paper. appropriate for all programmes and people, which is
Secondly, the paper has pointed to two issues in implicitly acknowledged in the OECD-DAC definition
particular that require more discussion with regards to by the breadth of its scope. However, in order to assess
impact: climate change and systems change. Climate impact, it must be more operationally defined and bounded
change is high on the agenda of most development to be feasible to measure in a systematic way. Explicitly
actors and yet environmental effects rarely form part discussing who means what by ‘impact’ is a critical first
of our definition of impact. The nature of development step in enabling impact assessment and judgement.

14 Methods Lab
References
Anderson, Andrea (undated) The Community Builder’s Approach to Theory of Change: A practical guide to theory
development. The Aspen Institute.
Befani, B., Ramalingam, B. and Stern, E. 2015. Introduction – Towards systemic approaches to evaluation and impact.
IDS Bulletin, 46(1): 1–6.
Cameron, D.B., Mishra, A., Brown, A.N. (2015) The growth of impact evaluation for international development: how
much have we learned? Journal of Development Effectiveness.
Chambers, R., Karlan, D., Ravallion, M. and Rogers, P. (2009) Designing impact evaluations: Different perspectives.
International Initiative for Impact Evaluation Working Paper 4. New Delhi: 3ie.
Duflo, E., Glennerster, R. and Kremer, M. (2006) Using Randomization in Development Economics Research: A Toolkit.
Gillies, J. and Alvarado, F. 2012. Country Systems Strengthening: Beyond Human and Organizational Capacity
Development. Background Paper for the USAID Experience Summit on Strengthening Country Systems. USAID.
Ling, A. (2014) Revisiting impact in the context of unintended consequences. Presentation to the 2014 African Evaluation
Association Conference, Yaoundé, Cameroon.
Montague, S., Porteous, N., and Sridharan, S. (2011) The Need to Build Reach into Results Logic and Performance
Frameworks. www.pmn.net/wp-content/uploads/Build-Reach-into-Results-Logic.pdf.
Ravallion, M. (2008) Evaluating Anti-Poverty Programs , T.P. Schultz and John Struass Handbook of Development
Economics Volume 4.
Rogers, P.J. (2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation,
14(1): 29-48.
Stame, N. (2010) What doesn’t work? Three failures, many answers. Evaluation, 16(4): 371-87.
Stern, E., Stame, E., Mayne, J., Forss, K., Davies, R. and Befani, B. (2012) Broadening the range of designs and methods
for impact evaluations. Report of a study commissioned by the Department for International Development. DFID
Working Paper 38.
UK Department for International Development (2011) Guidance on using the revised Logical Framework. A DFID
practice paper. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/253889/using-revised-
logical-framework-external.pdf.
United Nations Development Group (2011) Results-Based Management Handbook. https://undg.org/wp-content/
uploads/2014/06/UNDG-RBM-Handbook-2012.pdf.
United Nations (2015) Transforming our world. The 2030 agenda for sustainable development. A/RES/70/1.
https://sustainabledevelopment.un.org/content/documents/21252030%20Agenda%20for%20Sustainable%20
Development%20web.pdf.
White, H. (2009) Theory-based impact evaluation: Principles and Practice. International Initiative for Impact Evaluation
Working Paper 3. New Delhi: 3ie.
White, H. (2010) A contribution to current debates in impact evaluation. Evaluation, 16(2): 153-64.
Woolcock, M. (2009) Toward a plurality of methods in project evaluation: a contextualised approach to understanding
impact trajectories and efficacy. Journal of Development Effectiveness, 1(1): 1-14.

What is impact? 15  

ODI is the UK’s leading independent
think tank on international
development and humanitarian
issues.

Readers are encouraged to reproduce

Methods Lab material for their
own publications, as long as they
are not being sold commercially.
As copyright holder, ODI requests due
acknowledgement and a copy of
the publication. For online use,
we ask readers to link to the original
resource on the ODI website. The
views presented in this paper
are those of the author(s) and do
not necessarily represent the views
of ODI, the Australian Department of
Foreign Affairs and Trade (DFAT) and
BetterEvaluation.

© Overseas Development Institute

2016. This work is licensed under
a Creative Commons Attribution-
NonCommercial Licence
(CC BY-NC 4.0).

ISSN: 2052-7209
ISBN: 978-0-9941522-1-3

All ODI Reports are available

from www.odi.org

Overseas Development Institute

203 Blackfriars Road
London SE1 8NJ
Tel +44 (0)20 7922 0300
Fax +44 (0)20 7922 0399

www.odi.org

View publication stats