2
6 Things We MUST Understand…1. Big “need” to which we are contributing
2. “What” vs. the “so what” of the program
3. Sequence of ST outcomes MT outcomes LT outcomes
4. “Accountable” outcome—the outcome that equals success
5. “Moderators”—cntextual factors that get in the way
6. “Outputs”—activities strong enough to get to the outcomes
33
A Simple “Program Roadmap” Helps
Activities Inputs Outputs
Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will change because of the program…
What the program needs…
Context and Assumptions
External factors that influence getting to outcomes
4
Payoff: For Strategy
1. The key outcomes we “own”
2. How to implement our activities in way that most assures our outcomes
3. How to mitigate/own outside factors that may keep us from getting outcomes
5
Payoff: For Measurement
1. Accountable outcome is key one to measure
2. What ST/MT drive it, IF we don’t think we’ll get there?
3. Outputs=Process measures for our activities
4. Moderators=Ability to explain (lack of) progress
6
Today1. Presentation:
1. Creating and using a roadmap.
2. Some examples
2. Small groups:1. Creating/refining a roadmap for antibiotic
resistance effort
2. Identifying main outcomes of interest
3. Defining main “pathways” to get there
4. Identifying big outside factors
3. Quick debrief from small groups
7
Thursday1. Identifying main outcomes for our
“dashboard”
2. Defining some measures for outcomes
3. Identifying potential sources and potential challenges
8
REMEMBER!1. Strawman only
2. First, not last, discussion
3. Many steps post-session to get to clarity and consensus
99
(Simple) Logic Models are Roadmaps for ANY Effort: Lab Safety Example
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
1010
(Simple) Logic Models are Roadmaps for ANY Effort
Activities
Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will change because of the program…
Big “need” to which program is contributing
“What” vs. “So what”
1111
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
Big “need”: Lab Safety Training EXAMPLE
“What” vs. “So what”
12
Case: Lead PoisoningLead poisoning is a widespread environmental hazard facing young children, especially in older inner-city areas. Lead poisoning effects can be ameliorated through medical interventions. But, ultimately, the source of lead in the environment must be contained/eliminated through renovation or removal of the lead-based paint by professionals. Short of that, families can reduce the bad effects on their children through intensive housekeeping practices and selected
nutritional interventions.
13
The ProgramCounty X, with a high number of lead-poisoned children, has received money from CDC to support its Childhood Lead Poisoning Prevention Program. The program aims to do outreach to identify screening sites, screen and identify those with elevated blood lead levels (EBLL), assess their environments for sources of lead, and case manage both their medical treatment and the correction of their environment. They will also train families of EBLL children in selected housekeeping and nutritional practices. While as a grantee they can assure medical treatment and reduction of lead in the home environment, the grant cannot directly pay for medical care or for renovation of homes.
14
Listing Activities and Outcomes: Lead Poisoning
Activities Outreach Screening Case management Referral for medical tx Identification of kids with
elevated lead (EBLL) Environmental assessment Referral for env clean-up Family training
Effects/Outcomes Lead source identified Families adopt in-home
techniques Providers treats EBLL kids Housing Authority
eliminates lead source EBLL reduced Developmental “slide”
stopped Q of L improved
15
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Activities Outcomes
16
Antibiotic Resistance: National Plan Goals1. Slow the Emergence of Resistant Bacteria and
Prevent the Spread of Resistant Infections.
2. Strengthen National One-Health Surveillance Efforts to Combat Resistance.
3. Advance Development and Use of Rapid and Innovative Diagnostic Tests…
4. Accelerate Basic and Applied Research and Development for New Antibiotics, Other Therapeutics, and Vaccines.
5. Improve International Collaboration and Capacities…
17
The “What” of Antibiotic Resistance Monitor sales, use, and management practices Mobilize, advocate, engage across sectors Strengthens infrastructure for surveillance and
reporting Establish regional PH lab network with standard
platform for resistance testing Provide R&D funding and support Form public-private research partnerships Deliver education programs
18
The Big “So Whats” Less drug resistance Preserve the usefulness of antibiotics and
effective treatment of infection More effective prevention, control, and
treatment of infection/disease in humans and animals
Prevention of infection and disease Reduced spread of antibiotic resistant
pathogens Fewer resistant infections; Less disease in
humans and animals
19
Some “So Whats” That Drive Them Improved use of antibiotics in healthcare
settings Improved use of antibiotics in food production Increased trust and reduced blame Rapid recognition of resistance in humans,
animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics/non-antibiotic
prevention methods and therapies New and innovative animal management and
food production innovations
20
1 Activities
2 ST Outcomes
3 MT Outcomes
4 MT Outcomes
5 LT Outcomes
[WHICH PLAYER(S)] Monitors sales, use, and management practices [WHICH PLAYER(S)] Mobilizes, advocates, engages across sectors [WHICH PLAYER(S)] strengthens infrastructure for surveillance and reporting [WHICH PLAYER(S)] establishes regional PH lab network with standard platform for resistance testing [WHICH PLAYER(S)] provides R&D funding and support [WHICH PLAYER(S)] forms public-private research partnerships [WHICH PLAYER(S)] delivers education programs
Strong antibiotic stewardship in humans/healthcare Strong antibiotic stewardship/vet oversight in food production More regional cross-sectoral cooperation Consensus on strong and supportive policy Stronger lab capacity One health surveillance; Integrated surveillance Development of new technologies and innovations Increased private sector investment Increased private sector interest
Improved use of antibiotics in healthcare settings ; Optimal treatment decisions Improved use of antibiotics in food production Real-time info sharing on resistance and infections Increased trust and reduced blame Rapid recognition of resistance in humans, animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics New non-antibiotic prevention methods and therapies New and innovative animal management and food production innovations
Less drug resistance Preserve the usefulness of antibiotics and effective treatment of infection More effective prevention, control, and treatment of infection/disease in humans and animals Prevention of infection and disease
Reduced spread of antibiotic resistant pathogens Fewer resistant infections; Less disease in humans and animals
S A F E A D E Q U A T E F O O D S U P P L Y
21
Small Group Task1. Review the draft/strawman roadmap
2. Refine as needed:1. BIG activities missing?
2. BIG outcomes missing?
3. Some outcomes really activities that we do?
4. Sequence of outcomes right?
“Networked” Interventions
Agency A
Program A-n
Program A-1
Agency B Program B-1
Agency C
Program C-n
Program C-1
Agency D
Program D-n
Program D-1
OUTPUTS
SHORT-TERMOUTCOMES
LONG-TERMOUTCOMES
SYSTEMOUTCOME
23
Small Group Task1. Review the draft/strawman roadmap
2. Refine as needed:1. BIG activities missing?
2. BIG outcomes missing?
3. Some outcomes really activities that we do?
4. Sequence of outcomes right?
3. Think about YOU/your role. Can you trace a pathway thru the roadmap?
2424
(Simple) Logic Models are Roadmaps for ANY Effort
Activities
Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will change because of the program…
“Accountable” outcome
25
Choosing Accountable Outcome— “Staking Our Claim”
Want to include outcome(s) that is (are):Relevant—sufficiently “downstream”
to matter to stakeholdersResponsive—sufficiently “upstream”
that program’s efforts can be expected to make a difference
2626
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
“Accountable” outcome: Lab Safety Training EXAMPLE
27
Lab Safety Accountable Outcome: (Selected) Behaviors Sustained on Job Proper use of PPE Proper use of BSC Conduct formal risk assessments Ability to recognize hazards in any
situation Complete appropriate documentation Proper waste management
28
Lead Poisoning: “Causal” Roadmap—ACCOUNTABLE OUTCOME
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
**ReducingEBLLs**
**ReducingEBLLs**
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Activities Outcomes
29
1 Activities
2 ST Outcomes
3 MT Outcomes
4 MT Outcomes
5 LT Outcomes
[WHICH PLAYER(S)] Monitors sales, use, and management practices [WHICH PLAYER(S)] Mobilizes, advocates, engages across sectors [WHICH PLAYER(S)] strengthens infrastructure for surveillance and reporting [WHICH PLAYER(S)] establishes regional PH lab network with standard platform for resistance testing [WHICH PLAYER(S)] provides R&D funding and support [WHICH PLAYER(S)] forms public-private research partnerships [WHICH PLAYER(S)] delivers education programs
Strong antibiotic stewardship in humans/healthcare Strong antibiotic stewardship/vet oversight in food production More regional cross-sectoral cooperation Consensus on strong and supportive policy Stronger lab capacity One health surveillance; Integrated surveillance Development of new technologies and innovations Increased private sector investment Increased private sector interest
Improved use of antibiotics in healthcare settings ; Optimal treatment decisions Improved use of antibiotics in food production Real-time info sharing on resistance and infections Increased trust and reduced blame Rapid recognition of resistance in humans, animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics New non-antibiotic prevention methods and therapies New and innovative animal management and food production innovations
Less drug resistance Preserve the usefulness of antibiotics and effective treatment of infection More effective prevention, control, and treatment of infection/disease in humans and animals Prevention of infection and disease
Reduced spread of antibiotic resistant pathogens Fewer resistant infections; Less disease in humans and animals
S A F E A D E Q U A T E F O O D S U P P L Y
30
Small Group Task1. What is/are the accountable outcome(s)
for the antibiotic resistance effort?
2. REMEMBER:1. Thinking about THE WHOLE EFFORT—all
the pathways working jointly.
2. Looking to balance relevance with responsiveness. May/may not be outcomes on the far right!
3. Not naming ALL outcomes, just the ones furthest “east”
“Networked” Interventions
Agency A
Program A-n
Program A-1
Agency B Program B-1
Agency C
Program C-n
Program C-1
Agency D
Program D-n
Program D-1
OUTPUTS
SHORT-TERMOUTCOMES
LONG-TERMOUTCOMES
SYSTEMOUTCOME
32
Small Group Task1. Looking at the pathways, what
earlier outcomes seem most important to get to the accountable one/s?
3333
Simple Logic Models as Roadmaps
Activities Inputs Outputs
Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will change because of the program…
What the program needs…
Context and Assumptions
External factors that influence getting to outcomes
5. What “strong” activities means
Killer assumptions: Outside factors that can
help or hobble
34
Lab Safety: (Selected) Moderating Factors, Even if Our Training is Great! Supervisors’ attitudes and expectations Lack of resources for quality and safety Lack of reinforcement (decay of KAB) Time/safety trade-off, particularly during
public health emergency (time constraint) Competing priorities
35
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Activities Outcomes
37
Moderators—Lead Poisoning
Political—“Hazard” politicsEconomic— Health insuranceTechnological— Availability of
hand-held technology
38
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
Assessment ID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Outcomes Inputs
Funds
Staff
R’ships
Legal
Authority
Funds
Staff
R’ships
Legal
Authority
Activities
technology
insurance climate
“hazard politics”
39
(Selected) Actions We Can Take to Mitigate Moderating Factors
Supervisors’ attitudes and expectationsAdd consistent messages in supervisor
trainingPush others to create SOPs or policies to
induce supervisor cooperation.
Lack of reinforcement (decay of KAB)Refresher training; written materials
40
(Selected) Actions We Can Take to Mitigate Moderating Factors (cont’d) Time/safety trade-off, particularly during
public health emergency (time constraint) Competing priorities
Highlight consequences of not doing things right
Lack of resources for quality and safetyArray solutions in class from
cheapexpensivePush others for more budget for these efforts
41
1 Activities
2 ST Outcomes
3 MT Outcomes
4 MT Outcomes
5 LT Outcomes
[WHICH PLAYER(S)] Monitors sales, use, and management practices [WHICH PLAYER(S)] Mobilizes, advocates, engages across sectors [WHICH PLAYER(S)] strengthens infrastructure for surveillance and reporting [WHICH PLAYER(S)] establishes regional PH lab network with standard platform for resistance testing [WHICH PLAYER(S)] provides R&D funding and support [WHICH PLAYER(S)] forms public-private research partnerships [WHICH PLAYER(S)] delivers education programs
Strong antibiotic stewardship in humans/healthcare Strong antibiotic stewardship/vet oversight in food production More regional cross-sectoral cooperation Consensus on strong and supportive policy Stronger lab capacity One health surveillance; Integrated surveillance Development of new technologies and innovations Increased private sector investment Increased private sector interest
Improved use of antibiotics in healthcare settings ; Optimal treatment decisions Improved use of antibiotics in food production Real-time info sharing on resistance and infections Increased trust and reduced blame Rapid recognition of resistance in humans, animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics New non-antibiotic prevention methods and therapies New and innovative animal management and food production innovations
Less drug resistance Preserve the usefulness of antibiotics and effective treatment of infection More effective prevention, control, and treatment of infection/disease in humans and animals Prevention of infection and disease
Reduced spread of antibiotic resistant pathogens Fewer resistant infections; Less disease in humans and animals
S A F E A D E Q U A T E F O O D S U P P L Y
42
Small Group Task1. What are some big outside factors that
might keep us from getting our outcomes?
2. Use the P-E-S-T taxonomy IF helpful
3. Consider at HIGH LEVEL: Which of the outside factors:
1. Are “laws of gravity” we must work around?
2. Can be influenced by adding activities to our roadmap or doing things differently.
4343
Simple Logic Models as Roadmaps
Activities Inputs Outputs
Intermediate Outcomes
Short-term Outcomes
Long-term Outcomes/
Impacts
What the program does…
Who or what will change because of the program…
What the program needs…
Outputs: What “strong” activities means
4444
(Simple) Logic Models are Roadmaps for ANY Effort
Lab Safety Training
3: Behaviors sustained on the job
1: Happy trainees
4: Org change—Culture of lab safety
What the program does… Who or what will change
because of the program…
2: KAB and skills
Outputs: What “strong” activities means: Lab
Safety Training EXAMPLE
45
Lead Poisoning: “Causal” Roadmap
ScreeningScreening
Do Environment
Assessment
Do Environment
AssessmentID Source and
Refer for clean-up
ID Source and
Refer for clean-up
MedicalManagement
MedicalManagement
Lead Source
Removed
Lead Source
Removed
ReducingEBLLs
ReducingEBLLs
ImprovedDevelopment
and Intelligence
ImprovedDevelopment
and Intelligence
MoreProductive
and/or QualityLives
MoreProductive
and/or QualityLives
Family performs
in-home techniques
Family performs
in-home techniques
ID kids with
EBLL
ID kids with
EBLL
OutreachOutreach
Train
Families
Train
Families
Refer for
Medical Treatment
Refer for
Medical Treatment
Case
Management
Case
Management
Activities Outcomes
4646
The Plot Thickens
Screening so good it will lead to…
Behavior Change
ID People with Condition
Training in Self-Management so good it will lead to…
Referral for Medical Treatment so good it will lead to…
Quality Medical Management
4747
“Upgraded” Outputs: More than Simple Counts Screening: Pool (#) of screened kids
(meeting likely risk profile) Training: Pool (#) of clients trained
(using culturally-competent curriculum and with appropriate supports)
Referrals: Pool(#) of referrals to (qualified or willing) medical treatment providers
48
Outputs: (Selected) Key Attributes of Activities to Ensure Outcomes are Reached Content of training should explain why it
should be done Focus on actual work practices/evidence-
based work practices Real-time experience during the training Include reinforcing materials Personalizing content to the situation of
the person as opposed to general regs
49
1 Activities
2 ST Outcomes
3 MT Outcomes
4 MT Outcomes
5 LT Outcomes
[WHICH PLAYER(S)] Monitors sales, use, and management practices [WHICH PLAYER(S)] Mobilizes, advocates, engages across sectors [WHICH PLAYER(S)] strengthens infrastructure for surveillance and reporting [WHICH PLAYER(S)] establishes regional PH lab network with standard platform for resistance testing [WHICH PLAYER(S)] provides R&D funding and support [WHICH PLAYER(S)] forms public-private research partnerships [WHICH PLAYER(S)] delivers education programs
Strong antibiotic stewardship in humans/healthcare Strong antibiotic stewardship/vet oversight in food production More regional cross-sectoral cooperation Consensus on strong and supportive policy Stronger lab capacity One health surveillance; Integrated surveillance Development of new technologies and innovations Increased private sector investment Increased private sector interest
Improved use of antibiotics in healthcare settings ; Optimal treatment decisions Improved use of antibiotics in food production Real-time info sharing on resistance and infections Increased trust and reduced blame Rapid recognition of resistance in humans, animals, and environment Rapid ID and characterization of infection New diagnostic tests New antibiotic therapeutics New non-antibiotic prevention methods and therapies New and innovative animal management and food production innovations
Less drug resistance Preserve the usefulness of antibiotics and effective treatment of infection More effective prevention, control, and treatment of infection/disease in humans and animals Prevention of infection and disease
Reduced spread of antibiotic resistant pathogens Fewer resistant infections; Less disease in humans and animals
S A F E A D E Q U A T E F O O D S U P P L Y
50
Small Group Task—TIME ALLOWING1. Pick (one or more) activity in column 1.
2. Follow its pathway—what outcomes is it to generate?
3. Knowing the outcomes it is to achieve:1. How must we do this activity so it reaches
that outcome?
2. What are some attributes of doing that activity “well”? (quantity, quality, intensity, reach, etc.)
51
WRAP UP: In Your Groups—TODAY
Groups pre-assigned. Facilitator assigned to guide discussion Scribe assigned to take notes HIGH-LEVEL Topics:
Refine simple roadmap Set accountable outcome and some major
outcomes on pathways to get there“Moderator” discussion