Embed Size (px)
Citation preview
11/14/2012
1
ARUNASELAM RASAPPANARUNASELAM RASAPPANSenior Advisor
Center for Development & Research in Evaluation (CEDRE) InternationalEmail: [email protected]
Internet: www.cedre.org.my
Session 4: Evaluation as Undertaken by Various Stakeholders in their Own Specific Contexts
M&E Network Philippines2nd M&E Network Forum
Manila: 7-8 November 2012
Internalised SelfInternalised Self--Evaluation ApproachEvaluation Approachforfor
Sustainable Program Performance Improvement: Sustainable Program Performance Improvement: Experiences from MalaysiaExperiences from Malaysia
PresentationPresentation
Internalised Self-Evaluation Approachfor Sustainable Program Performance Improvement: Experiences from
Malaysia
Evaluation Evaluation –– Friend or Foe?Friend or Foe? Back to Basics!Back to Basics! Budgeting & Evaluation in MalaysiaBudgeting & Evaluation in Malaysia Fundamentals of ISE ApproachFundamentals of ISE Approach ISE ISE –– the Good and Badthe Good and Bad Integrating Meaningful EvaluationIntegrating Meaningful Evaluation Next Steps for Developing Countries Next Steps for Developing Countries
11/14/2012
2
Evaluation: Friend or Foe!Evaluation: Friend or Foe!
ViewpointsExternal Government Donors Beneficiaries
Internal Service Providers Contractors
External Concerns on Evaluation…External Concerns on Evaluation…
Need for accountability
Reporting to funders
Compliance agenda
Independent & Unbiased
Performance?
Results?
Others?
11/14/2012
3
Internal Concerns on Evaluation…Internal Concerns on Evaluation…
Time consuming
Too technical & complicated
Additional burden
Perception of checking/ policing
Not part of official mandate
No financial budget
No personnel to do this!
Experience with Real-World Evaluations
Ex-Post
Data Challenges
Perception Based
Feel-Good Factor
Ownership?
Black-Box
Lessons Learned
Sustainability
11/14/2012
4
Evaluation: Back to Basics!Evaluation: Back to Basics!
Traditional
PPBS
MBS
OBBPrograms &
Performance
Accountability, flexibility & delegation
1969 - 1990
1990 - 2012
2013 >
Detailed Controls
& Discipline1957 - 1968
• Incremental line item budgeting
• Disaggregated Budget
• Budget Ceilings
• Program Agreements
• Evaluation• Exceptions
Reporting
• Integration• Alignment• M&E focus• Results/
Performance Reporting
Integrated Approach
• Program Performance
• Program-Activities
• Performance Indicators
• Evaluation
Malaysia’s Budgeting & Evaluation System in PerspectiveMalaysia’s Budgeting & Evaluation System in Perspective
Adapted from: 2012 Sakeri Kadir, MoF, Malaysia
11/14/2012
5
Integrated ResultsIntegrated Results--Based Management (IRBM) & ComponentsBased Management (IRBM) & Components
IRBMIRBM
21
Management Management Information Information
SystemSystem
Provides critical information to support informed decision making for program improvement and adjustments to strategies & policies
55
Integrated Integrated Development Development
PlanningPlanning11
Systematic & structured approach to development planning with full vertical-horizontal integration & focus on program outcomes & impact
Result Based Result Based BudgetingBudgeting 22
Provides an integrated Outcome-Based Program Budgeting system for planning & management of financial resources to turn policies into reality
Monitoring & Monitoring & EvaluationEvaluation
44 Mandates structured monitoring and evaluation to support systematic program performance management
Personnel Personnel PerformancePerformance
Ensures personnel performance at every level is linked systematically with substantive program performance
33
Source: 2009 Center for Development & Research in Evaluation, International, Malaysia
Evaluation Focus Under ISEEvaluation Focus Under ISE
AppropriatenessAppropriateness EffectivenessEffectiveness EfficiencyEfficiency EconomyEconomy
11/14/2012
6
© ARTD Malaysia
ISE: What is it?ISE: What is it?An evaluation of a Program,Conducted internally by the Program
personnel,As a means to improve program
planning and performance,Using the principle of continuous
improvement through improved internal performance management capacity where,Performance dimensions are well
understood and internalised by service providers for sustainable performance
© ARTD Malaysia
ISE: Some Basic AssumptionsISE: Some Basic Assumptions If a manager fully understands the Program being
managed, he/she will perform better? If Program personnel have good knowledge of program
management fundamentals, they can apply it
If Program personnel know how to analyse and diagnose their programs better, they can plan performance better
Sustainable improvement & results can be better achieved through internal capacitation & motivation
Program personnel are basically good and we need to empower them and Let Managers Manage!
ISE supports accountability and external evaluations through systematic data collection
11/14/2012
7
© ARTD Malaysia
ISE Implementation in MalaysiaISE Implementation in Malaysia
ISE model was designed and adapted in 1999*
Tested and implemented between 2000 and 2004 in 20 major Activities in select Ministries
(Education, Welfare, Information, Works, Defence, Health, Public Enterprises, Islamic Religious Department)
Technical support and training for Facilitators for ISE for several Ministries were provided by the external technical advisors
* Malaysian version of the ISE Model was designed by Dr Arunaselam Rasappan (Technical Advisor) and tested in collaboration with the MOF
ISE ISE –– The Gory DetailsThe Gory Details
The SelfThe Self--Evaluation ModelEvaluation Model
11/14/2012
8
ProLL™ Evaluation Flowchart © 1999/2002/2010 Arunaselam Rasappan/Jerry Winston
Public Sector Policy & Program EnvironmentPublic Sector Policy & Program Environment
Doing The Right ThingsThe Right Way
For The Right People
Financial AccountabilityManagement Accountability
Program Accountability
Performance Mng’t Framework:
Verify Client Needs & Objectives
Prioritize Evaluation Questions
Determine Evaluation Design
Information Needed for Evaluation
Determine Research Design
Design Adjustment
Data Entry
• Stakeholder Analysis of Report
• Stakeholder Feedback
Stakeholder Utilization
Evidence-basedDecision-making
Logical Linkages with Substantive Purpose of Program
Data Collection
Data Sources andInstrumentation
Preliminary Planning & Assessment Stage
Evaluation DesignStage
Data Collection & Analysis Stage
Reporting & Information
Utilization Stage
Pilot Testing• Data Collection• Data Tabulation & Analysis• Data Interpretation• Validity & Reliability
Testing
Resource ProgramEfficacy Improvement Compliance
Determine Evaluation Questions• Identify pre-determined evaluation
issues• Analyse PMF & Expected Results• Conduct S/H & Needs Analysis• Analyse Program Logic
Findings & Conclusions
Data Validation
Evaluation Reporting• Multiple reports• Multiple audiences• User-oriented reports• Utilization-focused
Reports
Data Analysis
Evaluability Assessment
11 22 33 44
EVALUATION COMMITTEE(MOF)
STEERING COMMITTEE(Line Agency)
EVALUATION TEAM(Drawn from the Activity
or Department)
ISE Institutional StructuresISE Institutional StructuresChair: Deputy Sec. Gen. to Treasury Economic Planning UnitBudget Div, MoFMAMPU, ICU, INTAN Tech. Advisor (Consultant)
Chair: Deputy Sec. Gen./Dep. DG Head of Activity being evaluated Related Program and Activity Heads Head of Finance Head of Policy& Planning Activity Budget Review Officer (MOF) Internal Audit
Chair: Activity HeadDeputy HeadsActivity’s Senior Officer Facilitator Technical Advisor
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
11/14/2012
9
Identify issues that need to be assessed/evaluated
Identify appropriate internal evaluation team members
Identify the methodology, design and scope
Obtain the relevant approvals from the Steering committee
Carry out evaluation with technical advice (external or internal)
Ensure evaluation findings are used for program improvement.
Role of Evaluation Team Role of Evaluation Team
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
ShortShort--TermTerm Improved program implementation, delivery
processes, and importantly, program results Becomes an integral part of program performance
management Stakeholders& managers have ownership over
evaluation process Empowers managers to make informed decisions A part of ongoing innovation in government and also
value of “continuous improvement” Supports formative evaluation agenda.
Benefits of ISE ImplementationBenefits of ISE Implementation
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
11/14/2012
10
Longer Term Improved communication and better understanding
between partners of shared program outcomes Clear understanding of the program logic, vertical and
horizontal linkages Able to establish causal linkages and key program drivers Build internal capacity in better program/policy
planning, implementation and M&E Inculcates new knowledge about effective practices
contributing to knowledge-based public sector ISE complements external evaluation through systematic
data collection and performance analysis.
Benefits of ISE Implementation (cont’d)Benefits of ISE Implementation (cont’d)
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
Less perceived credibility than with external evaluation
Objectivity may be questioned
Some possibility of role confusion
Workload may delay completion of evaluations
Key findings, if negative may not be reported
Potential for bias & gaming.
© ARTD Malaysia
Issues in ISE Approach Issues in ISE Approach
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
11/14/2012
11
Rigorous peer reviews during the planning and implementation stage of the evaluation (Steering Committee at the Line Agency) & Technical Committee to provide technical oversight
Internal audit reviews
External reviews (random third party reviews)
Meta evaluation (done by central agency)
Highly transparent process
Technical training and guidance by external technical advisor.
Balancing the BiasBalancing the Bias
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
Stakeholder/top management support and commitment
Positive and capable leadership to understand and internalize the evaluation function into organization life
Organizational culture and commitment to support continuous learning and critical program review to improve decision making
Capacity building and resourcing issues
Poorly structured M&E framework
Poor data management protocols
Changing mind-sets and behavior takes time.
Challenges in Implementing the ISEChallenges in Implementing the ISE
Adapted from: 2012 Sakeri Karim ,MoF, Malaysia
11/14/2012
12
Accountability is important but it has to be inbuilt as a value system within organizations
Value system is best inculcated through awareness, understanding, appreciation, and buy-in
Governments should consider adopting ISE as it promotes performance improvement
It should be a part of the other performance and management improvement agenda in government
Donors support and resources for building internal capacity
Accountability is best addressed through self-realization and internalised value systems.
Lessons for Developing Countries & DonorsLessons for Developing Countries & Donors
Internalised Self Evaluation forInternalised Self Evaluation forImproved Development ResultsImproved Development Results
ARUNASELAM RASAPPANARUNASELAM RASAPPANSenior Advisor
Center for Development & Research in Evaluation (CEDRE) InternationalEmail: [email protected]
Internet: www.cedre.org.my
