Program Evaluation and Management

7/31/2019 Program Evaluation and Management

1/28

PROGRAM EVALUATION ANDMANAGEMENTJoining Theory and Practice

By: Jenny Katherine L. Henson, R.N.


2/28

Program Evaluation

Is intended to be a flexible andsituation specific means ofanswering questions, testinghypotheses, or describing programprocesses.

Can be formative or summative.

Its purposes clearly affect

relationships between evaluators andmanagers.


3/28

Program Evaluation

Generally, managers are more likelyto view formative evaluations as

friendly evaluations, and are morelikely to be willing to cooperate withthe evaluators.


4/28

Program Evaluation

From an evaluators standpoint, the

experience of conducting a formative

evaluation can be quite different fromconducting a summative evaluation.


5/28

Prospects for Building Culturesthat Support Evaluation

Instead of seeing evaluation as anactivity that challenges management,

they are encouraged to believe thatevaluators can work with managers todefine and execute evaluations thatcombine the best of what both parties

bring to that relationship.


6/28


Utilization focused evaluation, forexample is premised on producing

evaluations that managers and otherstakeholders will use and ensuringthat means developing a workingrelationship between evaluators and

managers.


7/28


Patton (1997) characterizes the role ofevaluator as the one who facilitates judgmentand decision making by intended users

rather than acting as a distant, independentjudge. Since no evaluation can be valuefree, utilization focused evaluation answers

the question of whose values will frame theevaluation by working with clearly identified,primary intended users who haveresponsibility to apply evaluation findings and

implement recommendations.


8/28

Love (1993) outlined six stages in thedevelopment of internal evaluation

capacity Ad hoc evaluations focused on single programs

Process focused regular evaluations

Goal setting, measurement of program outcomemonitoring, adjustment

Evaluations of program effectiveness, improving

organizational performance Evaluations of technical efficiency and cost

effectiveness

Cost benefit analyses


9/28

Learning Organizations asSelf Evaluating Organizations

Morgan (1997), in Images of Organizations,elaborates a organizational metaphor that

suggests that the organization can be seenas a brain. Within that broad metaphor, heelaborates a metaphor for learning

organizations.


10/28


Using the work of Senge (1990), Morgan suggests

that learning organizations must developcapacities to:

Scan and anticipate in the wider environment todetect significant variations

Develop an ability to question, challenge, andchange operating norms and assumptions

Allow an appropriate strategic direction andpattern of organization to emerge


11/28


Key to estabishing a learning organizationis what Morgan (1997) calls double - loop

learning; that is learning how to learn. Garvin (1993) has suggested five building

blocks for creating learning organizations.

By reviewing these steps, one can see akey role for evaluations.


12/28

Building Blocks for Creating Learning Organizations

1. Systematic problem solving

Tackling problems using a sequence of hypothesis generating hypothesis testingactions

Insisting on data rather than assumptions

Attention to details

2. Experimentation

Small, controlled modifications and tests of existing programs

Searching for and testing new knowledge

Managers must have both the incentives and skills to experiment

3. Learning from past experience

Systematically recording, displaying, and reviewing the evidence from pastperformance

Both this information and the skills to use and interpret it need to be widely distributedin the organization

4. Learning from others

Steal ideas shamelessly Find out who is the best learn why there are, and adapt their practices to your

organization

5. Transferring knowledge

Knowledge must be spread quickly and efficiently throughout the organization

Knowledge is treated as a resource


13/28


Fetterman (Fetterman, 2001; Fetterman,Kaftarian, & Wndersman, 1996) argues thatone way to contribute to the development of

a learning organization is through theprocess of empowerment evaluation.

Empowerment evaluation is defined as theuse of evaluation concepts, techniques, and

findings to help program managers and staffevaluate their own programs and thusimprove practice and foster selfdetermination in organizations.


14/28


Empowerment evaluation can only besuccessful in the right kind of organizational

environment, one which is guided by acommitment to truth and honesty.


15/28

Can Program Managers EvaluateTheir Own Programs?


16/28


Clearly, expecting managers to evaluatetheir own programs, given the incentivesalluded to above, can result in biased

program evaluations. Love (1993) envisions evaluators working

closely with program managers to produce

evaluations on issues that are of directrelevance to managers.


17/28


Patton (1997) stresses that among thefundamental premises of utilization focusedevaluation, the first is the commitment to

working with the intended users to ensure thatthe evaluation actually gets used.

Fetterman (2001) argues that the best data aresecured through close interaction andobservation with program managers and staff,because they are typically the mostknowledgeable about their program and its

strengths and weaknesses.


18/28


An important dissenting voice in the chorus thatadvocates evaluator manager contact and evencollaboration is Scrivens (1997) view that program

evaluators should keep distance from theorganizations and people with whom they work.

Getting too close to program managers amountsto compromising the objectivity of the evaluation

process, and undermines the key contribution thatevaluators can make: speaking the truth andoffering an unbiased view of a program.


19/28


Objectivity has been a criterion for highquality evaluations historically (Office of the

Comptroller General of Canada, 1981), andcontinues to have a scientific appeal to

practitioners and clients.


20/28

Striving for Objectivity in ProgramEvaluations

For Scriven (1997), objectivity is defined as withbasis and without bias and an important part of

being able to claim that an evaluation is objective isto maintain distance between the evaluator andwhat is being evaluated.

Objectivity has a certain cachet, and as apractitioner, it would be appealing to be able to

assert prospective clients that ones work will beobjective.


21/28

Criteria for Best Practices in ProgramEvaluation: Assuring Stakeholders that

Evaluations are High Quality

A review of several of these guideline documentsindicates that there are no specific mentions ofobjectivity among the criteria suggested for good

evaluations (AERA, 2000; American EvaluationAssociation, 1995; Australasian EvaluationSociety, 2002;Organization for Economic

Cooperation and Development, 1998).


22/28

It would seem that although some programevaluators and perhaps clients/stakeholders are

prepared to make objectivity a criterion for soundpractice, the evaluation profession as a whole isnot.

Professional evaluation organizations tend tomention the accuracy and credibility of evaluationinformation, the honesty and integrity of evaluators

and the evaluation process, the completeness andfairness of evaluation assessments and the validityand reliability of evaluation information.


23/28

In addition, professional guidelines emphasize theimportance of declaring and avoiding conflicts ofinterest and the importance of impartiality inreporting findings and conclusions. As well asguidelines tend to emphasize the importance ofcompetence in conducting evaluations and the

importance of upgrading evaluation skills.

Collectively, these guidelines cover many of thecharacteristics of evaluators and evaluations that

we might associate with objectivity it is a processthat involves corroboration of ones findings by

ones peers.


24/28

Ethics and Evaluation Practice

The evaluation guidelines, standards, andprinciples that have been developed for theevaluation profession all speak, in different ways,to ethical practice. Although evaluation practice isnot guided by a set of professional norms that areenforceable, ethical guidelines are an importantreference point for evaluators.


25/28

Ethics and Evaluation Practice

Newman and Brown (1996) have undertaken anextensive study of evaluation practice to establishethical principles that are important for evaluatorsin the roles they play. Underlying their work areethical principles, which they trace to Kitcherners

(1984, 1985) discussions of ethical norms.

Relationships Bet een the Ameri an E al ation Asso iation Prin iples and Ethi al Prin iples for E al ation


26/28

American Evaluation Association Guiding Principles

Systematic inquiry

Evaluators conduct systematic, data

basedinquiries about the subject of evaluation

Competence

Evaluators provide competent performance to

stakeholders

Integrity/honesty

Evaluators ensure the honesty and integrity of the

entire evaluation process

Respect for people

Evaluators respect the security, dignity, and self

worth of the respondents, program participants,clients, and other stakeholders with whom they

interact

Responsibilities for the general and public welfare

Evaluators articulate and take into account the

diversity of interests and values that may be related

to the general and public welfare

Ethical Principles for Evaluators

Maximizing Benefits

Minimizing harmsBalancing harms and benefits

Minimizing harms

Being honest

Keeping promises

No conflict of interest

Free and informal consent

Privacy and confidentiality

Respect for vulnerable persons

Procedural justice ethical reviews of projects

are fair, independent and transparent

Distributive justice persons are not

discriminated against, and there is respect for

vulnerable persons

Relationships Between the American Evaluation Association Principles and Ethical Principles for Evaluation


27/28

Reference:

Mc. David, Hawthorn Program Evaluation& Performance Measurement: An

Introduction to Practice, 2006, ThousandOaks, California


28/28

Documents

Program Evaluation and Management