So in this lecture we’re going to cover an overview of evaluation and look at the topics of just that – what is evaluation - so the key elements of evaluations, what are the different types of evaluation and what are the purposes of evaluation?

1

So what is evaluation? In its simplest form it’s a way or a defined process of judging merits, so now we have to think about what do we mean when we say we’re going to judge something. We really need to think about two key elements: first is the object of that judgment, so that theme we’re going to make a judgment about – it can be a person, it can be a thing like a program or an idea, and then we also need some yardstick by which we’re going to measure that object. So you can think of it as a yardstick or some measure by which we’ll say this object meets the criteria that we set for merit.

2

So let’s think of a simple example to illustrate. I want to evaluate my route to work so I’m really asking myself the question what is the best route to take to work. So here the object of the evaluation is the route or routes that are available to me. Now what is the yardstick by which I’m going to make my judgment of the best route to take? There’s likely to be many different yardsticks that give different answers to that question. First, am I concerned about how quickly the route gets me to my office, or here in Baltimore is it the easiness of the route when it’s raining? The difference in those two yardsticks can mean different routes to take to work, or am I not concerned about speed but I’m really concerned about how scenic the route is? Well I actually have third option. So you can see depending on the yardstick I get a different answer to the question “What is the best route to take to work?”

3

A more formal definition of evaluation is the systematic assessment of the operation and or outcomes of a program or policy compared to a set of explicit or implicit standards of those yardsticks as a means of contributing to the improvement of a program or policy, and we’ll break these each down separately.

4

Let’s look at the key elements of that definition: first is systematic assessment. So we need to have a methodical, formal and rigorous way of judging worth and this is done according to accepted research cannons. The second is the focus of that object and in our case it’s going to be the operation and outcomes of social programs and policies. The third point is the standard of comparison of our yardstick. We have some set of explicit expectations by which we’re going to compare the evidence from our evaluation, it provides the element of judgment and we need to really think clearly about how those criteria are defined, so how do we choose those yardsticks, and finally we have a purpose and for evaluation it’s the improvement of the social programs and policies that we’re interested in.

5

Some of the standard questions we might explore with evaluation are how is the program doing? What is it actually doing? How is it being conducted? Is it following the guidelines that were set for the program? So in a sense does it have fidelity to its model? What kinds of outcomes is it producing? Are kids learning more as a result of the reading program? Is it worth the money it costs? That’s a very important question to ask of social programs. Often we want to know should we continue doing this, should we expand it, scale it up, make it bigger, or should we cut back or should we change the program in some way or should we just give up altogether, and does it work for everyone or are there different types of people where it has different effects? These are all vital questions and most of our questions will follow along these general questions.

6

Let’s take a little definitional break here and define some of the key terms that are going to continually come up during this course. So when we say a program we mean an organized plan of ongoing effort designed to address a problem or improve conditions. So examples of programs in education are Head Start, KIPP schools, TFAA and others like that. A project then is a local instance of a program, so KIPP in Baltimore, TFAA in Baltimore, those would be projects of a program. The third definition we need to consider is component, so within a program or a project of a program we have various elements, discrete elements that we can look at and call those components, and finally a policy which is an officially accepted statement of objectives tied to a set of activities that are intended to realize the objectives.

7

There are two general forms of evaluation: first is formative, and this type of evaluation provides information for developing a service or a program and it helps to ensure quality or to improve quality; it’s proactive in that its helping programs change what they’re doing, potentially. Usually it comes during a development phase of a program so you’re evaluating while you’re developing so you’re saying this is working, this is not working during the ongoing operation of that program and it’s really aimed at program providers and the developers of those programs.

8

The second form of evaluation is summative and this usually happens after a program has become established, it’s been in operation for a while, it has completed its work, and it’s a way of looking back and assessing and saying did the program meet its goals, did it meet the criteria that were set for it? It’s often used in terms of accountability – were we successful, were we a failure, and it’s usually aimed toward the sponsors or funders of programs; we want to know did it work and how well did it work and who are the consumers of those programs so that they can do their own assessment of is this the best program for me.

9

We can also think about evaluations in terms of two basic types; the first is process evaluation so when we do a process evaluation we’re studying what the program actually does or often we’re quite interested in what it doesn’t do, so we’re thinking about fidelity of implementation, satisfaction of consumers or elucidating underlying problems with the program, and often process evaluations are very important to helping us understand the second type which is outcome evaluations and the data that is generated from outcome evaluations

10

So a second basic type is outcome evaluations and here we’re interested in what are the end results of the program? What are the end results for the people who the program served, and we’ve got to be careful about our language a little bit. There are a lot of words that are used in kind of the same way but often they don’t mean the same thing, so we can talk about results, we can talk about outcomes, we can talk about effects or we can talk about impacts and at this point I really want you to consider that ‘impacts’ has a very particular definition and often that really means the long-term outcomes of a program; often they’re broader in scope, so impacts for Baltimore city kids, and it really means of everything else what would have happened in the absence of the program so that the thing that is the effect that is attributable to the program alone, and as we’ll see throughout this course the requirements for making an estimate of impact are quite high so we want to be careful with the word ‘impact’; it has a very specific meaning that we’ll go into in more detail. We also really need to consider… we often focus about the intended outcomes of programs, but often programs have unintended outcomes and we want to be aware of those in our evaluations of them.

11

Now we’re going to talk about some of the main tasks that we carry out when we do an evaluation, and the first is one that you should have been doing already as part of your program and that’s delineating, so this is focusing the evaluation on key questions, so identifying those key questions, identifying the audiences of the evaluation, thinking about those yardsticks, what are the measures of success, determining what are the requirements – what do we need to conduct this evaluation and thinking about cost, how much is it going to cost, how much time, money and effort, and then securing agreements with stakeholders so that we can conduct the evaluation.

12

The next main task is obtaining, so collecting, organizing data, analyzing that data, synthesizing various bits of information and pulling everything together, recording – so that’s writing up our reports and providing that back to the correct audience and then the fourth is applying. So we’ve conduct evaluations because we want them to be used and to be useful so that applying step is very important to an evaluation.

13

We’ve talked about it a bit already but what are the purposes of evaluation, and it’s for improvement; it’s for accountability, to generate new knowledge. These are the stated purposes of evaluation and for our particular individual evaluations they should be explicit; they should be right up front and we should know what is the purpose of this evaluation. We also need to think about broadly… often evaluations have hidden purposes so they’re conducted as a way to obfuscate outcomes or to provide some faulty evidence that a program is working so that people will continue to fund it and we really need to think about that. So we need to think about who is wanting the evaluation, who is engaging the evaluator to conduct the evaluation and depending on the person we get slightly different purposes, and really what do they want and why do they want it? So we need to think about all of those things as we’re building our evaluations.

14

How is evaluation different from other types of research – and it is different, slightly. So first is this utility aspect of evaluation. Evaluation is meant to be used by folks to improve programs and it pushes us to begin with the end in mind, so knowing why we’re conducting the evaluation and what we hope it to do for our program or policy. The questions come from programs themselves versus an academic type question that leads us to higher levels of knowledge. There is this judgmental quality that’s bound in the definition of evaluation and so we need to make objective judgments against some criteria that we determine and then it’s in an action setting – evaluation takes place among programs that are in operation. So the most important thing going on is not the evaluation, it’s the program itself, and as an evaluator we need to think about that and be aware of that as we’re building our evaluations.

15

Evaluation is also different from other research in that there can often arise role conflicts, so when evaluations are conducted by programs in the field we can get this tension between the role of evaluator and the role of practitioner and sometimes that leads to potential for faulty development or faulty inferences and we need to be aware of that. Another is publication, and evaluations are much more informally published as reports that are given back to the funder of the evaluation and often they don’t become public – an allegiance – so as an evaluator where is the responsibility? Is the responsibility to the funder, is the responsibility to improving the field under study in the evaluation, is the responsibility to develop a knowledge? We all need to think about where is responsibility; it’s to all of those.

16

Evaluation though is also very similar to other types of research in that both types attempt to describe a phenomenon of interest or a thing of interest. We want to both understand the relationship between variables; we want to be able to trace causal sequences from one variable to another – that’s common to all types of research. Methods are shared across evaluation in more academic types of research and all research should be formalized, systematic, empirical and rigorous, so we should have a very clear, laid out plan that is part of standard practices to go about making judgments about the effectiveness of a program, let’s say, and we need to do it in a way that it leads us the best answer possible.

17