This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License.
Your use of this
material constitutes acceptance of that license and the conditions of use of materials on this site.
Copyright 2006, The Johns Hopkins University and Jane Bertrand. All rights reserved. Use of these materials
permitted only in accordance with license rights granted. Materials provided AS IS; no representations or
warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently
review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for
obtaining permissions for use from third parties as needed.
Fundamentals of Program Evaluation
Course 380.611
Developing a Conceptual Framework,
and Introduction to Formative
Research
Topics to cover today
Importance of a conceptual framework
Key points from Earp and Ennett article
Examples of conceptual frameworks
In-class group discussion on CFs
Discussion of Assignment #1
Introduction to formative research
Importance of a conceptual
framework (program theory)
Articulates the pathways by which an
intervention is expected to cause the
desired outcomes
Provides evaluator with specific
elements to assess
Other names:
Logic model, program model, outcome line,
cause map, action theory
Conceptual Framework of Family
Planning Demand and Program
Impact on Fertility
Societal and
Individual
factors
Value and
Demand
for Children
FP Demand
Spacing
Limiting
Other
Intermediate
Variables
Fertility
Wanted
Unwanted
Contraceptive
Practice
Development
Programs
Family Planning
Supply Factors
Service Outputs:
Access
Quality
Image/
Acceptability
Service
Utilization
Other Health
and Social
Improvements
Conceptual models:
Earp and Ennett (1991)
Definition of a conceptual model:
Diagram of proposed causal linkages among a set
of concepts believe to be related to a particular
public health problem.
Concepts = in boxes
Processes = shown by arrows
Can reflect factors at multi levels (macro to
micro)
Conceptual models
Often draw on:
One or more theories
Empirical evidence
Knowledge specific to the particular case
Serve to:
Summarize and integrate knowledge
Provide explanations for causal linkages
Generate hypotheses
Building a conceptual model
Start with the endpoint (dependent
variable, outcome, or target point for
intervention)
Identify potential correlates, based on
empirical or theoretical evidence
Show antecedent or mediating variables
by proximity to dependent variables
Conventions for drawing a
conceptual model
1)
2)
3)
4)
5)
Only include concepts that will be
operationally defined and measured
Present left-to-right or top-to-bottom
Use arrows to imply causality
Label concepts succinctly
Do not include operational definitions
or values of variables in the model
How to think through a
conceptual framework
Example: Tobacco Prevention and
Control
Interventions to Reduce Exposure to
Environmental Tobacco Smoke
http://www.thecommunityguide.org/tobacc
o/tobac.ppt
Tobacco use is the single largest cause of preventable premature mortality in the United States. It also
represents an enormous cost burden to the nation. The question is, what works to make tobacco use
prevention and control at the population or community level? The Guide to Community Preventive
Services addresses the effectiveness of community-based interventions for three strategies to promote
tobacco use prevention and control: 1) prevent tobacco product use initiation, 2) increase
cessation and 3) reduce exposure to environmental tobacco smoke (ETS). The findings
strengthen and complement existing guidelines (hyperlink table and text to existing guidelines) on
tobacco prevention and control.
Analytic Framework
For every intervention that we evaluate in the
Community Guide, we develop an analytic
framework, in which we postulate how we think the
intervention works and what outcomes we think are
important to capture information on. In many cases,
our analytic frameworks change over the course of
our reviews as we learn more about the intervention,
the potential outcomes, and the body of the evidence
in the literature.
Lets start with our intervention: Smoking bans
And our goal: A reduction in morbidity and
mortality.. Our analytic framework will connect these
two.
Analytic Framework: Smoking Bans
Smoking
Bans
Reduced
Morbidity
and
Mortality
Smoking Bans Might Result from
Community Education Efforts
Now its important to recognize that smoking bans
might be the result or outcome of interventions, such
as a community-wide education and/or political
campaign.
The experience from the state of California with a
broad smoking ban has been described quite well in
the literature, giving you a blow by blow account of
how that state managed to adopt and implement and
extend a statewide clean indoor air laws.
Its also important to note that in many states, preemption legislation precludes local governments from
strengthening clean indoor air laws. These efforts
have been described as a industry-sponsored effort
to obstruct efforts to extend protections from ETS to
workers.
Smoking Bans Might Result from Community
Education Efforts
Community
Education
Smoking
Bans
Reduced
Morbidity
and
Mortality
Bans Might Reduce ETS Exposure
Now back to how smoking bans work.
They work in one or three ways.
First, directly by reducing exposure to
ETS in the restricted environment. This
alone will have health effects.
Bans Might Reduce ETS Exposure
Reduced
Exposure to ETS
Smoking
Bans
Reduced
Morbidity
and
Mortality
Bans Might Increase Smoking
Cessation
Second, smoking bans might work by affecting the
tobacco use behaviors of smokers
In response to a smoking ban, they might think twice
about continuing their habit. They might reduce their
daily consumption of tobacco, and these two effects
might increase the number who attempt to quit.
Since smoking bans also reduce ques to smokers to
relapse, more smokers attempting to quit will be
successful. This will result in fewer tobacco users
and a reduction in adverse health outcomes.
Bans Might Increase Smoking Cessation
Reduced
Exposure to ETS
Fewer
Tobacco
Users
Smoking
Bans
Change
In
Attitudes
Reduced
Consumption
Increased
Quit
Attempts
Increased
Cessation
Reduced
Morbidity
and
Mortality
Bans Might Reduce Smoking Initiation
Finally, we acknowledge that smoking
bans might directly affect tobacco
consumption by youth or affect their
impressions of the social desirability of
smoking. These will reduce tobacco
use prevalence among adolescents and
contribute to fewer tobacco users.
Bans Might Reduce Smoking Initiation
Reduced
Exposure to ETS
Smoking
Bans
Change
In
Attitudes
Change
In
Attitudes
Reduced
Initiation
Fewer
Tobacco
Users
Reduced
Consumption
Increased
Quit
Attempts
Increased
Cessation
Reduced
Morbidity
and
Mortality
Bans Might Increase ETS in the Home
One unintended effect described in the literature, at
least initially, was a concern that smokers might
respond to workplace smoking restrictions by
compensating at home, smoking more and thus
increasing ETS exposures in the home.
We looked for evidence of this potential harm in our
review.
We also examined the evidence, not shown here,
that smoking bans in restaurants and hotels
adversely affects business revenue and tourism.
Bans Might Increase ETS in the Home
Reduced
Exposure to ETS
Smoking
Bans
Change
In
Attitudes
Change
In
Attitudes
Reduced
Consumption
Reduced
Initiation
Fewer
Tobacco
Users
Increased
Quit
Attempts
Increased
Cessation
Diverted
Consumption
Increased
Home Exposure
Reduced
Morbidity
and
Mortality
Body of Evidence: Bans and
Restrictions
We did a series of electronic database searches, and
screened titles and abstracts and ended up with the
following body of evidence:
56 studies were reviewed
17 studies measured differences or changes in ETS
exposure, of which 10 met our criteria for good or fair
quality
51 studies measured smoking habits of employees exposed
to bans or restrictions, of which only 9 met our criteria for
good or fair. In most cases, the excluded studies did not
included concurrent comparison groups.
Study Measurements by Outcome
If you take all of the qualifying studies, and
plotted their outcomes to slots on our analytic
framework this is what we found across this
body of evidence.
For example we have 12 measurements of
differences or changes in exposure to ETS, 6
studies of changes in tobacco use prevalence
among employees, 4 measurements of
cessation by smoking employees etc.
Study Measurements by Outcome
12
Reduced
Exposure to ETS
0*
Change
In
Attitudes
Smoking
Bans
0*
9
Change
In
Attitudes
Reduced
Consumption
Reduced
Initiation
6
Fewer
Tobacco
Users
Increased
Quit
Attempts
Increased
Cessation
Diverted
Consumption
Increased
Home Exposure
Reduced
Morbidity
and
Mortality
Examples of different conceptual
frameworks
Model of Program Impact
Socioeconomics
Status
Knowledge
Gender
Income
Education
Psychographic
Characteristics
Program
Exposure
Practice
Family
Characteristics
Interpersonal
Contacts
Attitude
Determinants of Domestic Violence
(no intervention)
Contextual and
Community Factors
Household and IndividualLevel Factors
Socioeconomic
development
Socioeconomic
status
Domestic violence
norms
Life cycle factors
Gender inequality
Intergenerational
exposure to violence
Crime levels
Risk behaviors
Domestic
violence
Womens status/
autonomy
Conceptual
Framework
Conceptual
Framework
Communication
to a Health-Competent
Pathways toPathways
a Health-Competent
SocietySociety
Domains for
Communication Interventions
Communication to
Strengthen the
Social Political
Environment
Communication
for Effective
Service
Delivery
Systems
Communication
to Create
Health Literate
Communities
and Individuals
Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions
CommunicationSocial Political
Environment
Communication
for Service
Delivery
System
Initial Outcomes
Environment
Service Systems
Community
Communication
for Community/
Individual
Individual
Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions
CommunicationSocial Political
Environment
Communication
for Service
Delivery
System
Initial Outcomes
Environment
Behavioral
Outcomes
Supportive
Environment
Service
Performance
Service Systems
Client
Behaviors:
Community
Community
Communication
for Community/
Individual
Individual
Individual
Conceptual Framework
Communication Pathways to a Health-Competent Society
Domains for
Communication Interventions
CommunicationSocial Political
Environment
Communication
for Service
Delivery
System
Initial Outcomes
Environment
Behavioral
Outcomes
Supportive
Environment
Service
Performance
USAID 5 SOs
Service Systems
Client
Behaviors:
Community
Community
Communication
for Community/
Individual
Sustainable
Health Outcomes
Individual
Individual
Conceptual Framework
Communication Pathways to a Health-Competent Society
Underlying
Conditions
Domains for
Communication Interventions
CommunicationSocial Political
Environment
Initial Outcomes
Environment
Behavioral
Outcomes
Sustainable
Health Outcomes
Supportive
Environment
Context
Communication
for Service
Delivery
System
Service
Performance
USAID 5 SOs
Service Systems
Client
Behaviors:
Community
Resources
Community
Communication
for Community/
Individual
Individual
Individual
Conceptual Framework
Communication Pathways to a Health-Competent Society
Environment
Social
CommunicationSocial Political
Environment
Initial Outcomes
Political will
Resource allocation
policy changes
Institutional capacity
building
National coalition
National communication
strategy
Service Systems
Context
Disease Burden
Domains for
Communication Interventions
Availability
technical competence
Information to client
Interpersonal
communication
Follow-up of clients
Integration of services
Community
Underlying
Conditions
Leadership
Participation equity
Information equity
Priority consensus
Network cohesion
Ownership
Social norms
Collective efficacy
Social capital
Economic
Communication
Technology
Political
Legal
Resources
Communication
for Service
Delivery
System
Human and
Financial
Resources
Strategic
Plan/Health
Priorities
Other Development
Programs
Policies
Communication
for Community/
Individual
Individual
Cultural
Message recall
Perceived social
support/stigma
Emotion and values
Beliefs and attitudes
Perceived risk
Self-efficacy
Health literacy
Behavioral
Outcomes
Sustainable
Health Outcomes
Supportive
Environment:
Multi-sectoral
partnerships
Public opinion
Institutional
performance
Resource acquisition
Media support
Activity level
Service
Performance:
Access
Quality
Client volume
Client satisfaction
Client
Behaviors:
Community
Sanitation
Hospice/PLWA
Other actions
Individual
Timely service use
Contraception
Abstinence/partner
reduction
Condom use
Safe delivery
BF/nutrition
Child care/immuniz.
Bednet use
USAID 5 SOs
Reduction in:
Reduction in:
Unintended/mistimed
Pregnancies
Morbidity/mortality
From pregnancy/
Childbirth
Infant/child
morbidity/mortality
HIV transmission
Threat of infectious
diseases
Requirement for
exercise #1
Present the diagram in terms of initial,
intermediate, and long-term outcomes
Note: this is NOT a standard
requirement of conceptual frameworks
but it is a useful way to look at program
effects.
Criteria for grading conceptual
framework on exercise #1
Diagram respects the 5 conventions for
drawing a conceptual model
The model presented is:
Conceptually clear (explains to the reader
how you expect the program to achieve its
objectives)
Visually pleasing
Concise but covers key factors (suggestion:
include 10-15 concepts in your model)
Rules relating to confounding and
modifying variables (Earp & Ennett)
See page 169 of the article
Technically fine, but not necessarily
used among all researchers
In exercise #1, dont feel bound by
these two rules.
Formative evaluation
Guides the design of a program
Different types:
Needs assessment (esp. in U.S.)
Diagnostic (formative) research
(Specific to media) Pretesting
Needs assessment in the
program cycle (McDavid)
Strategic Planning
Program Development
Stakeholder Input
Program Implementation
Program Evaluation
Environment Scanning
Stakeholder Input
Stakeholder assessments of services/outputs in
relation to needs (relevance)
Program Accountability
Steps in conducting a needs
assessment (McDavid & Hawthorn)
Become familiar with political context
Identify users and uses
Identify target pop. (geographic, socio-dem)
Inventory existing services (what gaps exist?)
Identify needs
Prepare document
Evidence, benchmarks, conclusions, recs
Communicate findings, implement
Use of benchmarks in needs
assessment
Compare current levels and types of
services to benchmarks (or reference
points)
Conceptions of human needs
Moral/ethical values (no child left behind)
Levels of service provided elsewhere
Service provider opinions/preference
Client (current, prospective) opinions
Sources of data: primary
(new) & secondary (existing)
Lit reviews
Similar studies
Demographic statistics
Government reports
Surveys (mail, phone, in person)
Focus groups
Interviews
Direct observation
Diagnostic research (very
similar to needs assessment)
Also called formative research or
formative evaluation
Learn more about all aspects of the
problem, population, and context
Diagnostic research uses both
quantitative & qualitative
Quantitative (demographic,
epidemiological):
To quantify the extent of the problem
To identify subgroups most affected
To identify explain determinants
Qualitative:
To understand problem from user
perspective, identify barriers
Great diversity in types of
formative research
Examples:
Formative research for Stop Aids Love Life
Louisiana study on teen smoking behavior
Investigation of places with high rates of
new partner acquisition (PLACE
methodology)
Publication of formative
research in peer-reviewed lit
Quite rare
Results often presented in a report
More likely in form of baseline findings
Ex: Stop AIDS Love Life
Louisiana adolescent smoking study
If value goes beyond study location
PLACE methodology in S. Africa
Key points from Louisiana
smoking study
National surveys of adolescent smoking
didnt provide adequate data on target
population
Survey of 4808 students provided data:
Smoking patterns by ethnic group, gender
Social relationships related to smoking
Friends, family; smoking and alcohol
Example of a baseline survey
as a two-fer (two for one)
Formative research in form of baseline
survey serves two purposes:
Establishing a baseline level against which
to evaluate program after intervention
Providing insights into the problem that
help to guide the design of the program
Findings from LA study useful
in developing intervention
LA rate higher than national rate for
adolescents
Who was most likely to smoke:
Among whites: no male/female differences
Among blacks: males more likely to smoke
Both black and whites:
Discretionary $$
Low academic achievers
Findings from LA study useful
in developing intervention
Strong relationship of smoking to:
Smoking of family & friends, alcohol use
Authors discuss challenges of designing
a program with these dynamics
Formative research doesnt give all the
answers to program design!
PLACE (priorities for local AIDS
control activities) methodology
MEASURE Evaluation Project (UNC)
Identifies where to access sexual
networks with individuals with high
rates of new partner acquisition
Provides information on availability of
preventive services (info, condoms)
Methods of PLACE: 3 phases
of data collection
Key informants: where do people meet
new sexual partners?
Visit to sites compiled from interviews
Community leaders, health care providers,
youth on street, taxi drivers, STD clients
Type of site, patrons, AIDS prevention?
Sites marked on aerial map
Interviews with people at these sites
Useful information for
designing an intervention
Key locations: taverns and shebeens
<2% of sites had on-site anti-AIDS info
<10% didnt have condoms onsite
Almost 60% of owners/managers would
be willing to have condoms onsite
Patrons at these locations frequent
visitors (regulars)
Indicators useful for
monitoring programs
# new sites identified sexual activity
% of sites with condoms (verified)
Mean rate of new partnership formation
at site in past 4 weeks, by gender
Portion of patrons who ever used a
condom
Portion used condom at last sex
Use of qualitative research to
inform quantitative
To learn vocabulary used by local
population to describe problem
Yoder study on diarrhea: 9 different words
To identify new concepts that
researchers hadnt considered
To generate hypotheses to be tested
through subsequent research
