Upload
letitia-nicholson
View
217
Download
1
Tags:
Embed Size (px)
Citation preview
WELCOME
June 2012
Evaluation Planning and Management
John Wooten ~ Lynn Keeys
Session 8: Evaluation Team Debriefing and ReportObjectives:• Review some important considerations in debriefing
evaluation teams• Review requirements of core sections of the evaluation
report• Understand how to transition smoothly from team
debriefings to the draft and final evaluation reports
Evaluation Road Map :Evaluation Stage 5
2
Evaluation Team Debriefing and Report
• Don’t engage in detail meddling• Don’t disallow team’s structuring inputs• Do consider iterative debriefings• Do understand how report drafting is unfolding • Do be prepared to guide the team in developing an
acceptable product
Some DOs and DON’Ts to remember…
3
Evaluation Team Debriefing and ReportWhich comes first? Why?
Draft evaluation report?
Team debriefing?
4
Evaluation Team Debriefing and Report
Draft evaluation report?
Team debriefing?
Consider requesting an annotated report outline
before the debriefing
Which comes first? Why?
Tip:
5
Evaluation Team Debriefing
• Finalize/activate team debriefing plan• Iterative debriefings, successively higher level• Representatively participatory/transparent• Evaluation team inputs on structure• ~One hour (20 min. presentation)
• Organize around either :- Findings, conclusions, recommendations, lessons- Key evaluation questions - Project components
• Agree on modifications for next debriefing
Evaluation team debriefing
6
Evaluation Team Debriefing
• Provide team brief, written summary of decision points, guidance, OR
• Request annotated outline of draft report (highlight decision points, guidance from debriefings)
• Promptly ID in writing misunderstandings, internal conflicts, differences of opinion; resolve as indicated
• Preview all data to be submitted in agreed formats• Identify data-related issues in advance
Tip: Avoid making assumptions
Evaluation team debriefing … FINAL
7
Evaluation Team ReportAll conspiring to answer…
Whether /why intended results were/were not
achieved
Accurate description
Existing,
new info (evidence)
Analysis, interpretation of
facts
General conclusions
Set of expert recs.
8
Evaluation Team ReportDraft evaluation report…
AIMED AT GENERAL PROFESSIONAL
READER
IN TARGETED AUDIENCES
CLEAR
CONCISE
CREDIBLE
IMPACTFUL
PERSUASIVE
9
Evaluation Team Report
• Final evaluation SOW (illustrative outline)
• Evaluation work plan (finalized/approved)
• Cursory review field experience (final +)
• Judicious use of annotated outline (R.e.a.l.l.y. FINAL)
So when is final FINAL?
Draft evaluation report…Iterative outline
10
Evaluation Team Report
1. USAID’s Monitoring and Evaluation TIPS #17,
“Constructing and Evaluation Report” (Table 1, p. 8 guided outline)
2. Un-numbered TIPS (same title)
(Annotated report outline; Management Systems Int’l, 4/2006)
3. “Checklist for Assessing USAID Evaluation Reports”
(Hard copies @ Tab 7)
Draft evaluation report…References
11
Evaluation Team Report
RecommendationsProposed actions for management
ConclusionsInterpretations and judgments
based on the findings
FindingsEmpirical facts collected during the
evaluationArtfully constructed, to convey/reflect team’s clarity, skill, professional judgments,
experience and passion, contributing to decision-making processes that lie ahead
Organizing key elements of the evaluation report
Fig. 1, TIPS #17, p. 2
Attribution?
Counterfactual?
12
Evaluation Team Report
• Organize around either :- Findings, conclusions, recommendations, lessons- Key evaluation questions - Project components- Otherwise
• Share your own perspectives
• Share good, bad, ugly samples/experiences• Understand key sectional differences…
Draft evaluation report…Organization
13
Tips on Findings?
Evaluation Team Report
• Based on analysis: - Facts -- empirical evidence - Typically emerge from data analysis- Begin to answer key evaluation questions
Draft evaluation report…Findings
14
Evaluation Team Report
• Bundle findings that answer a macro-question• Examine how management transformed inputs-to-outputs• Examine how management “positioned” outputs to
influence (+,-) outcomes (causality)
• Relate findings to project design (theory of change, results framework, logical framework, critical assumptions)
• Include clear, hard-hitting summary of data analyses
Draft evaluation report…Findings Tips
15
Evaluation Team Report
• Based on judgments: - Involve deductive judgments (meanings)- Emerge when applying reasoning to interpret evaluation
findings
Draft evaluation report…Conclusions
Tips on Conclusions?
16
Evaluation Team Report
• Communicate importance, benefit, impact of empirical findings about project, context, problem, management framework
• Demonstrate understanding of context• Summarize team’s views on whether/why a project
succeeded or failed achieve various design aspects• Clearly answer “So what?” questions• Clearly/logically support empirical findings• Reflect skills, expertise of team members
Draft evaluation report…Conclusions Tips
17
Evaluation Team Report
• Based on reflective experiences: - Answer, “So what do we do about it?” - Are judgments on what needs adjusting to enhance likely
achievement of results in the current or future project or related programs
Draft evaluation report…Recommendations
Tips on Recommendations?
18
Evaluation Team Report
• Flow clearly/logically from findings/conclusions• Anticipate, respond to pending decisions cited in SOW• Are actionable (i.e., feasible, cite responsible agency or authority,
within their ‘manageable interests’)
Draft evaluation report…Recommendations Tips
But how to write the report?!? 19
…by report outline?
…by technical specialty?
Consider how to avoid a report that reads like…
…a committee product?!
> BONUS TIP <
But how to write the report?!?
Draft evaluation report…Writing
Evaluation Team Report
20
Evaluation Team Report
• Triangulate review of draft report based on:- Final evaluation SOW- Written agreements from debriefings - Evaluation report checklist
• Organize peer reviews (w/strict time limit)
• Allow ample time for stakeholders’ review
(cultural, language, availability constraints)
• Push to resolve ALL dissenting perspectives, and feedback conflicts (…and if not resolved?)
Draft evaluation report…Review
21
Evaluation Team ReportDraft evaluation report…Review
22
• Prepare timely, written, contradictions-free feedback (the “corporate” perspective ~ USAID et al)
• Follow-up w/confirmation phone call/meeting• Limit track changes to minor editorials only; provide major,
substantive comments separately• Re-confirm timeframe for receipt of final report• Reconfirm LOE/budget availability
Evaluation Team Report
• Review per evaluation checklist• Verify responsiveness to the “corporate” feedback• Negotiate required refinements• Review, confirm content/completeness of all quantitative
data in agreed formats • Verify all SOW terms are fully satisfied• Issue acceptance letter (per A&A Office guidance)
Final evaluation report…Review/Acceptance
Congrats on this significant milestone~
23
([email protected]) ([email protected])
24
Evaluation Planning and Management
John Wooten ~ Lynn Keeys
Thank you~