Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Towards Professionalization of Evaluation: Voluntary Evaluator Peer Review: Is this the Road to Travel?
Zurich 11th September 2014EvaluationConference
DeGEVAL/SEVAL
Professor Helen SimonsUniversity of Southampton
© Helen Simons 2014
Structure
Hallmarks of a profession – are we there? Competencies and Capabilities Scope of UKES Capabilities framework Voluntary Evaluator Peer Review (VEPR) Eval/Partners Initiative: survey & workshop Issues arising Reflections on which road to take.
© Helen Simons 2014
Hallmarks of a Profession 1
Purpose -public service to societySpecific body of knowledge Agreed activities as part of professionProfessional society - to further aims
and sanction bad practiceGuidelines, standards and ethical code
to uphold practice in the field
© Helen Simons 2014
Hallmarks of a Profession 2
Training to perform agreed activities -probationary period
Continuous professional development Registration to practice & potential to
deregisterEvaluation competence/ capability
© Helen Simons 2014
Observations on Progress Towards A Profession
Public service to society Specific body of knowledge Agreed activities Professional society Ethical code and principles Training Probationary period Registration to practice Potential to deregister if incompetence proven.
© Helen Simons 2014
Rise of Competency/Capability Frameworks
On The Road to Voluntary Evaluator Peer Review
© Helen Simons 2014
Reasons for Rise of Competency Frameworks
Zeitgeist of activity in evaluation societies and other organizations
Professional – extension of ethical guidelines standards; help recent evaluators develop capacity
Political - reaction to dominance of accountability; greater awareness of use and misuse
.
© Helen Simons 2014
Competencies and Capabilities -Purposes
Purposes vary: Professional development Evaluation Training Selection Professional designation Mentoring But all designed to enhance professionalism of
evaluation
© Helen Simons 2014
UKES Evaluation Capabilities Framework
Early issues Flexible and aspirational Not for certification at this time Track record as important a
skills/knowledge Response cautious Focused on risks of use
© Helen Simons 2014
UKES Evaluation Capabilities Framework
Later Issues –related to purpose Educative Quality assurance for ourselves Reassurance to funders and audiences of
quality of evaluators and evaluation Overall aim to promote professionalization
of evaluation as a social practice. Preference for Capability
© Helen Simons 2014
Why Capability
Broader conception – agency Much evaluation takes place in teamsSocio/political circumstances impactResponsibility not entirely with
individual Capabilities to conduct a quality and
credible evaluation
© Helen Simons 2014
Contexts of Use
Individuals - professional development Commissioners - skill mix for teams Organizations - build understanding
- capacity for self-evaluation - ensure quality
Framework for designing training Quality assurance process for evaluators,
stakeholders & audiences Voluntary Evaluator Peer Review
© Helen Simons 2014
UKES Evaluation Capabilities Framework
Evaluation Knowledge
Professional Practice
Qualities and Dispositions
© Helen Simons 2014
UKES Evaluation Capabilities Framework
1. Evaluation Knowledge 1.1 Understands the social and political role of
evaluation
1.2 Familiarity with evaluation designs and approaches
1.2.3 Identifies relevant evaluation questions
1.3 Comprehends & makes effective use of evaluation methodologies
© Helen Simons 2014
UKES Evaluation Capabilities Framework
2 Professional Practice 2.1 Demonstrates ability to manage and
deliver evaluations 2.1.3 Identifies data gathering instruments
appropriate to the task
2.2 Demonstrates Interpersonal skills 2.22 Shows ethical sensitivity in specific
socio/political contexts 2.26 Manages conflicts of interest & values fairly
© Helen Simons 2014
UKES Evaluation Capabilities Framework
3. Qualities and dispositions
3.2 Exercises sound, rigorous and fair judgment
3.4 Displays independence of mind ad integrity especially when evaluation challenged
3.5 Upholds democratic values in conducting & reporting evaluations
© Helen Simons 2014
Voluntary Evaluator Peer Review
© Helen Simons 2014
Voluntary Evaluator Peer Review
Different forms –paper/pencil, interview, professional conversation, online
CES - first - particular form Eval Partners Initiative UKES/ EES 2013/4 Two phases: survey and workshop Built on VEPR system from NZ Issues in designing a VEPR system
© Helen Simons 2014
Voluntary Evaluator Peer Review
Eval /Partners Initiative UKES/ EES 2013/4 Concept summary:
Members apply to undergo a structured professional practice review with two accredited peer reviewers
Applicant identifies practice areas for building their professional capacity
© Helen Simons 2014
Voluntary Evaluator Peer Review -Process
Set of questions for review related to capabilities framework
Review face-to face or via Skype or tele-conference
Reviewers recommend 1) list on VEPR index or 2) follow-up review
Lapses after 3 years unless renewed
© Helen Simons 2014
Voluntary Evaluator Peer Review - Issues and Choices
© Helen Simons 2014
Voluntary Evaluator Peer Review -Issues and Choices
Purpose Quality Motivation Context Whose role Voluntary or Mandatory
© Helen Simons 2014
Voluntary Evaluator Peer Review -Issues and Choices
Quality and Integrity of Process Levels Focus Choice of reviewers Criteria for selection of reviewers Timescale
© Helen Simons 2014
Voluntary Evaluator Peer Review -Issues and Choices
PaymentPrematerial InfrastructureGovernance
© Helen Simons 2014
Risks (1) Costs could exclude some evaluators
Lack of consistency across reviewers
Too bureaucratic for too little reward
May result in lowest common denominator
Danger of knowledge about evaluator affecting subsequent proposal chances
Danger of society seen to be endorsing evaluator as competent whereas only signifies been through pd process
© Helen Simons 2014
Risks (2)
Not used sufficiently to improve quality
Insufficient reviewers
Process takes too long
Disputes may take time to resolve
Clients could favour ‘approved evaluators’
Difficult of actually assessing evaluators’ capabilities beyond knowledge and skills
© Helen Simons 2014
Voluntary Evaluator Peer Review: Is this the road to travel ?
© Helen Simons 2014
Voluntary Evaluator Peer Review : Is this the Road to Travel ?
Benefits identified by participants Improve profile of discipline – give it a
professional stamp of approval Promote agreed standards for moving
towards accreditation and professionalization
Improve quality of evaluation
© Helen Simons 2014
Voluntary Evaluator Peer Review : Is this the Road to Travel ?
A cautionary note‘But in order to make the process work for evaluators it must be stressed that the gestation process requires extensive volunteer time, inclusive consultation that is not rushed and a willingness to ensure that the tested competencies are meaningful and adapted to the context (Mc Guire, 2012)
© Helen Simons 2014
Voluntary Evaluator Peer Review : Is this the Road to Travel ?
And a caution from me.
Quality of evaluation, as distinct from designation and/or regulation ?
It is here - in the field- that the credibility of our works stands or falls, whatever we claim and whatever credentials we hold.
© Helen Simons 2014
Voluntary Evaluator Peer Review : Is this the Road to Travel ?
End for Now
And... good luck if you follow this road
© Helen Simons 2014
REFERENCES Stake, R.E. & Schwandt, T.A. ( 2006) ‘ On Discerning Quality in Evaluation’,
in Shaw, I.F., Greene, J.C. & Mark, M.M. (2006) The Sage Handbook of Evaluation, London: Sage, pp 404-418
Picciotto, R. (2011) The logic of evaluation professionalism, Evaluation, 17(2) 165-80, Sage Publications
Simons, H (2006) in Shaw. I., Greene, J.C., & Mark, M (eds.) The Sage Handbook of Evaluation, London Sage, pp243-265 for an overview of the evolution of the move towards professional ethics, their foci and purposes
© Helen Simons 2014
Early Considerations (2009)
Flexible and aspirational
Starting point for commissioners/organizations to think about team skills required
Track record of doing evaluations as important as skills and knowledge
Guidelines for developing evaluation training Not for certification at present- needs infra
structure
© Helen Simons 2014
Choice in selection of reviewers
Nearly a third said yes – to assure that reviewers understood their area of interest /expertise
Half said no (except for right of appeal) to ‘ensure objectivity’
Others said a voice but not total selection
© Helen Simons 2014
Role of Evaluation Societies
Majority indicated society should take lead
External agencies less qualified to judge
Possibly work with other groups/societies
Too soon, need to incorporate existing courses
Depends on cost & capacity for setting up
Danger of lowest common denominator
© Helen Simons 2014
Should applicants have to pay?
Majority said yes but payment figure varied
Sliding scale suggested according to workplace & expertise, commensurate with cost of reviewers & admin.
Needed payment to give the process value
Maybe reduction in membership fee to encourage take-up
© Helen Simons 2014
Should peer reviewers be remunerated?
Over half agreed, but modestly
Somewhat dependent upon position & expertise of reviewers
Reason – doubtful that evaluators would take on task for no payment, need to attract calibre of reviewers
Minority said no; others dependent upon precise nature of scheme
© Helen Simons 2014
Administrative Infrastructure
Keep light and demand led
Panel of 20-30 reviewers; Part/full-time admin.
Procedures for recording results, handling submissions, recruiting reviewers, criteria, timeframe, dispute mechanism, marketing
Precise form dependent upon demand and society resources - needs to be self sustainable
Enormity of task if taken up widely - keeping track, review etc.,
© Helen Simons 2014
UKES Survey : Response
Small survey (10 questions) low response (after 3 extensions) related to concept summary
Majority responses, practicing evaluators from range of discipline backgrounds
Place of work, consultancy and public sector
View with caution but useful issues
© Helen Simons 2014
Voluntary Evaluator Peer Review -Process
Clear statement of process & advocacy to gain support
Society committee and infrastructure Appointment of co-ordinator Selection of reviewers Applicants apply & reviewers agreed Co-ordinator liaises review time & place Applicant submits form nominating
areas & two recent projects
© Helen Simons 2014
Potential Risks (2009)
Could become too complicated
Instrument of bureaucratic control – ‘tick box’, ‘too prescriptive’
Fail to promote diversity of evaluation contexts
Stifle innovative methods and practice
© Helen Simons 2014
Issues from Workshops (2010) Capabilities preferable term to
competencies
Purpose should be educative - to build capacity
Competencies should relate to evaluation task rather than evaluators
Balance between technical and practical
© Helen Simons 2014
Issues from Workshops (2010)
How to represent complexity of ‘lived experience’
Take account of dynamics of change in fluid policy environment
How to ensure does not exclude certain groups e.g. self- evaluation
How to ensure flexibility of use
© Helen Simons 2014
Purpose of UKES Capabilities Framework
Professionalization of evaluation as a social practice
Quality assurance for ourselves
Reassurance to funders and audiences of quality of evaluators and evaluation
© Helen Simons 2014
Further Potential Uses
Continuous professional development- Matrix of opportunities Mentoring Selection Designation of individual evaluators Register of team competencies –
procurement processes Accreditation of evaluation training
© Helen Simons 2014
Pros and Cons of VEPR
However...
cumbersome nature of system
how to accommodate different specialismsapproaches and wider capabilities
how to get permission to share evaluation conducted in team settings
consistency across reviewers
© Helen Simons 2014
Pros and Cons of VEPR
Majority said would endorse a VEPR system as first step in designation
‘potential to facilitate skills and knowledge transfer between peers & thus expand competencies, qualities & capacity of sector’
Should be voluntary – better able to respond to needs than if imposed
Good learning process for individuals
© Helen Simons 2014
Should applicants have to pay?
Majority said yes but payment figure varied
Sliding scale suggested according to workplace & expertise, commensurate with cost of reviewers & admin.
Needed payment to give the process value
Maybe reduction in membership fee to encourage take-up
© Helen Simons 2014
Reflections
…. I would hate to see any such set of competencies becoming used as a crude checklist by commissioners… to rule evaluators/trainers in or out of potential funding opportunities. I would prefer to see the society offer guidance to key competencies rather than develop an orthodoxy of ‘must haves’
( UKES Teacher/researcher/other)
© Helen Simons 2014
Reflections
The long term goal should be to engage evaluation buyers sufficiently that they start to use commitment to the framework as a discriminating factor in placing contracts.
(UKES, Consultancy)
© Helen Simons 2014
Reflections
A competency framework is useful in terms of professional development and training and…for those new to evaluation who need to commission work. However it should be seen as an indicative list of some of the most important competencies – it should not be interpreted too rigidly.
( Non UKES, Analyst/ Manager Public sector)
© Helen Simons 2014
Benefits Improve profile of discipline –’professional
stamp of approval’
Encourage practitioners/ commissioners to think about capabilities for evaluation
Assuring commissioners/audiences
Providing assurance to both sides in contracting for external evaluation
Agreed standards for moving towards accreditation & professionalization