E. D. Bell

Kapp & Anderson (2010) argue that the program evaluation process has eight steps that are fluid, dynamic, and interdependent, i.e., they are not linear – they influence and shape one another throughout the evaluation process.

1. Identify the evaluation question

2. Conceptualize the evaluation question

3. Operationalize the evaluation question

4. Pick and define the method

5. Select the sample

6. Collect the data7. Analyze and

manage the data8. Utilize the

information to improve program performance

Kapp & Anderson (2010) make this pointThe steps of the evaluation process need

to be framed with an appreciation of two fundamental notions:(1). The use of information for service

improvement is primary, and (2). The context for evaluation is an

agency where evaluation must compete with many enterprises for attention. (p. 29)

Given these notions, the agency needs to implement the following steps concurrently with the implementation of the program evaluation:

1. Define the program2. Support users3. Implement and report design of the evaluation (Kapp & Anderson, 2010)

“The process of conducting research and the evaluation of programs and the outcomes associated with these activities have the potential to be helpful or harmful to individuals, groups, organizations, and the broader society” (Kapp & Anderson, 2010, p. 39).

So, if you were a participant in a study, what would you want to know? If you were a program director, how would you want the evaluators to conduct themselves? If the evaluation of your program found a number of faults, how would you want these to be reported? (Kapp & Anderson, 2010, p. 59)

Guiding Principles for EvaluatorsSystematic inquiryCompetenceIntegrity/HonestyRespect for peopleResponsibilities for general public

welfare

“The process of program evaluation does not take place in a social and cultural vacuum devoid of gender, race, ethnicity, and other consideration” (Kapp & Anderson, 2010, p. 107).

Kapp & Anderson (2010) describe four attributes:1. Awareness and acceptance – that generates respectful treatment and no prejudgment of abilities, performance, and competency.2. Self-awareness – that helps you understand how your culture shapes what you see and provides the ability to examine, control, and change assumptions, false beliefs, and stereotypes

3. Dynamics of differences – historical and cultural differences influence how you see and define reality, e.g., see Emmett Till. This story that I saw in Jet magazine as a young person still influences how I see and understand reality. Click on the link “The Picture of Emmett Till”.

4. Knowledge of individual cultures – of the stakeholders who are involved with the program being evaluated.

5. Adaption of skills – The evaluator must be able to determine if her or his methods are culturally appropriate and respectful of the persons and programs that are being evaluated.

Review Kapp & Anderson (2010) pp. 111 – 131.

“With most evaluation projects, one of the first things to address is the way the program is supposed to operate” ( Kapp & Anderson, 2010, p. 133).

Develop the logic model of how the program is supposed to work (Program Development and Evaluation, University of Wisconsin – Extension for resource tools)

Interview StakeholdersBoard of DirectorsClientsCommunity PartnersFundersManagementStaff (Kapp & Anderson, 2010).

Use Logic Model Questionnaire (Kapp & Anderson, 2010, p. 139)

Given the first draft of the logic model for the agency what areas of the model require additional information, i.e., what are the potential evaluation questions.

Select one of the areas as the evaluation question for your evaluation plan