Good morning, my name is Robert Brader and I’m the Performance & Data Analysist at Washington Service Corps. I’d like to welcome all of you who are attending this on-line training.

This presentation covers the basic steps to design a Pre and Post survey instruments to measure Knowledge gain.

WSC currently tracks knowledge gain in four of focus areas, economic opportunity, Environmental Stewardship, and Disaster preparedness.

By the end the training, you should understand:

Outcomes Knowledge Measuring Knowledge using a pre/post Examples of Knowledge Questions Designing a quality survey Pilot Testing Analyzing and Scoring your results

When thinking about outcomes it’s important to consider the underlying assumptions the community problem that your program is working on. Ultimately the relationship between the community problem, program activities and how these activities make improvements on the community problem; Collecting short-term data to measure the change you expect to see is important because it allows you to examine the improvements you are making.

The pre/post design is generally viewed as a more accurate method to measure a change in knowledge. Pre and post surveys should be administered at two or more points in time. (e.g., the beginning, and the end of activities).

The pre-test is a set of questions given to participants before the training begins which will give you a baseline of knowledge level at the beginning of the course.

After the completion of the course, participants are given a post-survey to answer the same set of questions, By comparing the difference in scores from pre to post enables you to see whether the training increased participant knowledge of the training content.

Performance measurement does not and cannot “prove” that your intervention caused an outcome. It does provide a snapshotof how a program is functioning and draws from the evidence found in your program’s theory of change.

WSC is currently tracking knowledge gain in four focus areas, Economic Opportunity, Environmental Stewardship, Healthy Futures and Disaster Preparedness. Knowledge is often defined as a belief that is true and justified. This definition has led to its measurement by methods that rely solely on the correctness of answers. A correct or incorrect answer is interpreted to mean simply that a person knows or does not know something. Generally, knowledge viewed as: Facts, Certainty, True or Factual, something that exists and is verifiable.

Perhaps the most important element that should be considered when choosing an evaluation method is to ask the question, “what are we trying to measure?” If the goal of an evaluation is to measure a change in KNOWLEDGE then a pre/post design is generally viewed as a more accurate measure of change between two-time intervals. The pre/post gives us a starting point, or the amount of pre-existing knowledge on the

course topic Pre and post surveys should be administered at two or more points in time. (e.g., the beginning, and the end of activities). To inform the instructor about topics that are/are not needed to cover in the course based on student’s previous knowledge or can be used to identify any instructional needs to improve the training course.

What are some of the most important aspects to consider in the survey design?

Learning objectives: These are a great starting point for question design. There’s certainly no sense in designing questions that don’t relate to your course curriculum and learning objectives. I often hear comments from sites that we don’t know our learning objectives yet, or our members are going to design our course curriculum. In many cases, the AmeriCorps member will be enhancing and expanding your program, but It’s our expectation that sites should have some basic understanding of the preliminary learning objectives or intent of the project.

Think about all the specific information (facts, figures, formulas, procedures, categories, etc.) that you want the participants in your program to know? Which content is the most important that participants remember?

True-False - easy to write and score; tends to promote memorization and encourage a high degree of guessing. I would limit true false questions and include all different types to gain a true picture of what the participants know.

Matching - easy to write and score; ideal for measuring associations between facts; tends to emphasize memorization.

When designing a matching question, Put descriptions, longer phrases, or statements in the first column and the shorter options, words or symbols in the second column. there should be at least three more options than descriptions – this makes it harder for respondents to eliminate options. In directions, specify whether the basis for matching and whether options can be used more than once.

Multiple Choice - considerable versatility in measuring outcomes; designed to make participants discriminate among options that vary in degree of correctness, thus a higher level of cognitive thinking than T/F questions; time consuming to write and must be carefully written.

Completion - relatively easy to write, but sometimes hard to score because there can be more than one defensible answer; restriction of an answer to a few words tends to measure the recall of specific facts, names, places, and events as opposed to more complex items. Some other things to consider:

The question should clearly state the problem and keep response options short. Be sure wrong choices are plausible, especially with multiple choice Use the option “none of the above” sparingly. Avoid using the option “all of the above.” Statements should require a single-word answer. Omit only key words: don’t eliminate so many elements that the statement doesn’t

make sense. Be sure that the answer is factually correct.

Question from a Disaster Preparedness CERT training

Multiple Choice - considerable versatility in measuring outcomes; designed to make participants discriminate among options that vary in degree of correctness, thus a higher level of cognitive thinking than T/F questions; time consuming to write and have to be carefully written.

Question from a Financial Litereacy class

True-False - easy to write and score; tends to promote memorization andencourage a high degree of guessing. I would limit true false questions and include all different types to gain a true picture of what the participants know.

!!!!Make sure that you Write statements that are definitely true or false. This is a common complaint of mine. For example. Please remove the I don’t know response choice.

When networking, you should ask your friends and family if they know anyone who is hiring once and then never bring it up again. True, False, I don’t know

Telling a clerk at the grocery store that you are unemployed and asking if they are hiring is an example of networking. True, False, I don’t know.

Question from an environmental stewardship project – Clark County Stream Team

Completion - relatively easy to write, but sometimes hard to score because there can be more than one defensible answer; restriction of an answer to a few words tends to measure the recall of specific facts, names, places, and events as opposed to more complex items.

Matching - easy to write and score; ideal for measuring associations between facts; tends to emphasize memorization.

Put descriptions, longer phrases, or statements in the first column and the shorter options, words or symbols in the second column. there should be at least three more options than descriptions – this makes it harder for respondents to eliminate options. In directions, specify whether the basis for matching and whether options can be used more than once.

Although demographic question can’t be used to measure knowledge, you can use Demographics questions…These are questions that describe the characteristics of your participants. Demographic questions add to the length of your survey, so keep this in mind. Only ask about the attributes you really think are important to know. Also be careful about privacy

The pilot test is a trial run of your instrument to see how it works. A pilot test helps you increase the accuracy and consistency of your instrument. Before pilot testing your instrument on an audience, it is always good to let another person who is not familiar with your program review the instrument. They may often find jargon and other wordsthat might not be understood by the audience. Make these changes and then you are ready to pilot test. During the pilot test you want to discover the answers to the following questions:

The instrument should be pilot tested on 10 to 15 people who are representative of the target audience for the program. These should not be people who may later be participants in the program.

If your pilot participants had difficulty understanding a lot of the questions,, or the format, you should consider redesigning the instrument and pilot again.

The next step in the pilot is called item analysis. After you have administered the pre survey to the pilot participants, you should conduct an item analysis to determine which questions you should keep or eliminate.

Once you have administered the pre test, you need to record all the scores on a piece of notebook or graph paper.

Assuming there were 15 pilot participants. In this example, questions 1, 4, 5, 9, and 10 would need to be eliminated. These questions need to be eliminate because if your participants already know the information, your program is not going to help them gain that knowledge. You should only use questions that your participants need to learn. If your participants already know this information then you might want to reconsider the or purpose of your project or you might want to adjust your learning objectives or re-think your training project.

Use clear and concise words. State the program name or name of training Include information about the instrument’s purpose or Include information about how

the data will be used. Include questions that ask respondents for relevant information about themselves and

their background (e.g., student’s name, grade, age). Include general directions on how to complete the instrument, itself (e.g., when, where,

and how to return the instrument). Use language that respondents understand Develop an instrument that it is pleasing to the eye (e.g., not “busy”) Arrange for the pilot test participants to be as close to the actual conditions as possible

(e.g., time of day, location, methods, respondents).

Data analysis is the process of aggregating your data and figuring out what it means.

Did your results answer the question, “Did we accomplish what we set out to do?” or “What changed for our service participants?

WSC defines “improvement” as any increase in the number of correct answers from pre-test to post-test.

Pre-Test scores (yellow), then record Post-Test scores (green) Be sure that each set of pre- and post- scores is for the same participants! In this spreadsheet I’ve designed function to calculate the difference between pre and post, the % of change and a group averages in bottom line. In this analysis, you’ll be reporting to WSC 6 out of 10 participants increased knowledge from pre to post. 10 particiants will be reported as your output and 6 will be reported as an outcome.