Collecting Evaluation Data: Surveys

G3658-10Program Development

and EvaluationUniversity of Wisconsin-Extension

Cooperative ExtensionMadison, Wisconsin

Collecting Evaluation Data:SurveysEllen Taylor-PowellCarol Hermann

May 2000

Acknowledgments

We would like to acknowledge all the many usefulresources available on surveys which we used aswe developed this manual. Many are noted in thereferences. Thanks go to Weiwu Zhang for his earlywork on this manual. A special thanks goes toJoan Palmer for her patience and skill in formattingand publishing the final product.

COLLECTING EVA L UATION DATA: SUR V E Y S ¢ ¢ ¢

TABLE OF CONTENTS

1

Up Front Planning or Think Before You Jump 2

What Is a Survey? 3

When Is a Survey Appropriate? 3

Choosing a Survey Method 4

Hand Out Surveys 4Mail Surveys 5Telephone Surveys 5Face-to-Face Surveys 6Electronic Surveys 6

Planning a Survey 7

Implementing a Survey or Getting It Done 9

Mail Surveys 9Telephone Surveys 13

Getting a Good Response to Your Survey 15

Interpreting Results When Your Response Rate Is Low 17

Summary 17

References 18

Appendix A. Sample Mail Survey Materials 19

Appendix B. Sample Telephone Survey Materials 21

Appendix C. Sample Press Releases 23

¢ ¢ ¢ P RO G R A M D E V E L O P M E N T A N D E VA L UATION

UP FRONT PLANNING OR THINK BEFORE YOU JUMP

Too often we jump into planning and conducting a survey before we’ve taken time to seeif a survey is warranted, appropriate for our evaluation questions and needs, likely to havebuy-in and provide useful information. Conducting a valid and meaningful survey takestime, energy and resources. Devote some time up front to consider its likelihood ofsuccess.

So, before you begin planning a survey consider the following questions. These are criticalto address before initiating any evaluation:

1. What is the purpose of the evaluation?Why is this evaluation being undertaken? What is it expected to achieve or toprovide insight about?

2. Who will use the information?Is this evaluation only for your own use? Are there others who are asking for thisevaluation? What are their expectations and agendas? Who else might be inter-ested in the process and results of the evaluation?

3. How will they use the information?What will the different users do with the evaluation results? What decisions oractions are likely to be influenced by the results?

4. What questions will the evaluation seek to answer?What specifically do you and others want to know?

5. When are the evaluation results needed? What’s your time line?

6. What are your own abilities and skills? Are there others who can help?

7. What resources do you have or can access, including time, money and people?

Deciding to undertake a survey depends not only upon the type of information you wantbut also upon a number of human, political and financial factors. For further assistance inplanning an evaluation, see Planning a Program Evaluation (UW-Extension publicationG3658-1)1 .

1 All UW-Extension publications referenced in this document can be downloaded via the Internet atwww.uwex.edu/ces/pdande/evaluat.html

2


Surveys are one of the most popular waysto collect evaluation information. Most ofus have conducted a survey of one type oranother. This manual is written to helpcommunity-based educators improve theirpractice. It is designed to provide basicinformation and practical tips to help youdesign and implement better quality sur-veys. It provides

¾ Guidelines for choosing a surveymethod that is appropriate to yourneeds

¾ Steps in survey planning to help ensure thoughtful, successful surveys

¾ Tips for implementing mail andtelephone surveys

¾ Ways to increase response rates¾ Suggestions for discussing results

when response is less than expected

This manual does not cover data processingand analysis or using your survey results.Likewise, questionnaire construction andsampling, important aspects of surveydesign, are covered in companion UW-Extension publications. Some of the manyexcellent resources on survey methods arelisted in the reference section for additionalguidance.

WHAT IS A SURVEY?

A survey is a way to collect informationdirectly from people in a systematic, stan-dardized way. Surveys use questionnairesthat ask the same question in the same wayto all respondents. Data collected this waycan then be used to make inferences aboutthe population of interest (e.g. farmers inyour county, parents who attend a parentingworkshop, etc.). Information can be col-lected about people’s opinions, knowledge,attitudes, beliefs, behaviors, plans andbackgrounds. Surveys are used in needsassessments and opinion polls, as well as to

evaluate the process, outcomes and impactsof programs and policies.

Surveys can be carried out in a number ofdifferent ways: through the mail, over thetelephone, in face-to-face interviews, ashandouts, electronically (e-mail or web-based surveys) or a combination of thesemethods. They may be conducted at a singlepoint in time, at repeated times or concur-rently with multiple samples. Surveys maybe used to collect information from arelatively small number of people or frommany thousands. They may be used tocollect information from all members of aparticular group or from a sample of thetarget population.

WHEN IS A SURVEYAPPROPRIATE?

A survey is most appropriate when yourevaluation questions and information needsare best answered by the people themselves.Often we want people’s own reports of theiropinions and behaviors. Often there arethings we cannot see directly, such asattitudes and beliefs. Or, we may not be ableto observe people’s actions, such as theirdaily consumption of fruits and vegetables.So, we have to ask them.

Not all information, however, is best col-lected through a survey. It may be moredirect and useful to use another method.A variety of alternatives exist such as:

n Observationsn Existing data, records, documentationn Tests of abilitiesn Case study

For example, it may be more useful toactually observe land management practicesthan to ask a participant if s/he is using the

3

¢ ¢ ¢ P RO G R A M D E V E L O P M E N T A N D E VA L UATION4

recommendation. Or, it may be difficult fora person to know how many times s/heattended meetings of the community col-laborative over the past year or the decisionsand actions taken by the group. An analysisof the attendance register and the minutes ofthe meetings would be better sources ofinformation.

It is most important to be clear about theinformation you want and need to collect.Then, you can decide if a survey is appro-priate. If your purpose is to find out thepercentage of individuals who think or dosomething, and the information is notavailable from existing sources, a survey ismost likely appropriate.

A survey is just one of various ways tocollect information to answer your evalua-tion questions. Ask yourself, “Is a surveythe right tool for this job?”

CHOOSING A SURVEYMETHOD

There are five main survey methods: mail,telephone, face-to-face, handout and elec-tronic. Each method has advantages anddisadvantages. No single method is superiorto another. Each needs to be assessed interms of the survey content, respondentcharacteristics, time line and availableresources.

Survey Content

¾ What types of questions will beasked? How complex or sensitiveare they?

¾ Would people be more likely tounderstand and respond to questionspresented in print or orally?Anonymously or in person?

Respondent Characteristics

¾ From whom do you want to collectinformation?

¾ What is the easiest way to reach them?¾ Do they have certain characteristics

that rule out one method overanother (e.g., literacy skills, notelephone, etc.)?

Time Line

¾ How quickly do you need results?

Available Resources

¾ Who will work on the survey?¾ Will you have help from outside

experts in planning and managingthe survey?

¾ What facilities will you use?¾ How much money do you have to

spend?

Hand out surveys are often used at the endof a program for immediate evaluativefeedback. Hand out surveys are a goodchoice when:

¾ You want to capitalize on havingrespondents available (e.g. alreadyconvened at a conference, attendinga fair, etc.)

¾ You need or want immediate feedback¾ You have limited resources¾ You may not be able to reach the

people again (e.g. mobile populations)

When using hand out surveys, you canpersonally convey the purpose and importanceof the survey to respondents. Most people arelikely to consent to completing a survey whenasked in person. You are also available toanswer questions respondents may have.

Hand Out Surveys


Mail Surveys

Mail surveys are most appropriate when youare working under the following conditions:

¾ You have a complete and accuratemailing list

¾ The people you are surveying havesome interest in the survey topicandare likely to respond

¾ Your targeted respondents haveadequate reading and writing skills

¾ You want to give respondents timeto consider their answers or otherinformation when completing thesurvey

¾ You want to give respondents pri-vacy when completing the survey

¾ You need to survey a relativelylarge number of respondents

¾ You have limited resources

Mail surveys present several challenges. Toensure that your survey reaches the intendedrespondents, you need a complete andaccurate mailing list. Mail surveys are alsovulnerable to low response rate. That is whymail surveys are more effective whenpotential respondents have some interest orstake in the survey topic and are likely tocomplete it. Finally, respondents fill outsurveys privately, without assistance,prompts or clarification. You have no wayof knowing who is actually filling out thesurvey. Respondents may skip parts of thesurvey or have someone else assist them incompleting it. Because of this, you have lesscontrol over mail surveys than hand out ortelephone surveys.

5

Telephone surveys

Telephone surveys are the most appropriatemethod to use when:

¾ You need results relatively quickly¾ Your target audience has telephones¾ Members of your target audience

would have difficulty completing awritten questionnaire (e.g. respon-dents have low literacy rates or pooreyesight)

¾ Your survey questions are clear andeasily comprehensible

¾ The survey is relatively short¾ Staff is available to do the calling

Approximately 95 percent of U.S. homeshave telephones. However, those householdswithout telephones tend to be occupied bypeople with lower income and less educationthan the general population. In addition,people living in rural areas, younger headsof households and ethnic minorities are lesslikely to own telephones than the generalpopulation (Lavrakas, 1998). Obtaininginformation from these groups may present achallenge if you use the telephone surveymethod.

Telephone surveys can be completed in arelatively short period of time and give youmore control over the process. Callers canprovide additional information and clarifica-tion (in a standardized manner for eachrespondent), probe for more informationfrom respondents and encourage them toanswer all questions. Callers can be moresure that the person completing the survey isthe one intended to complete it. It is usuallyeasier to get higher response rates fortelephone surveys than for mail surveys.


Face-to-Face Surveys

Face-to-face surveys often yield the bestresults but they tend to be time consumingand expensive to implement. They are wellsuited to situations where you do not havean address or phone list or when respon-dents are not likely to answer willingly oraccurately by phone or mail. Because a face-to-face survey is actually an interview, theperson conducting the survey needs to havesound interviewing skills. This may meaninvesting additional resources for trainingstaff in these skills.

Face-to-face surveys are appropriate when:

¾ There is no list for the populationbeing surveyed

¾ People do not have a telephone orcannot be reached by mail ortelephone

¾ There is concern that people will notrespond willingly or accurately

¾ The questions are complex and mayneed in-person explanation

¾ The budget is not an issue

While face-to-face surveys may be costly,they have certain strengths. You can be morecertain that the person who you want torespond actually is the one who answers thesurvey. You can personally explain theimportance of the survey and assure therespondent of confidentiality. Flash cards orother aids can be used to help peopleunderstand the questions and answerchoices. The personal approach of the face-to-face survey helps put the person at easeand may increase willingness to participate.

Electronic Surveys

Electronic surveys use computers and theInternet to distribute and collect informa-tion. For simple surveys, you can use e-mailto communicate with respondents. Web-based surveys use more advanced technol-ogy and allow you to create complexinstruments using multimedia and interac-tive features.

E-mail surveys are appropriate to use when:

¾ Members of your target audienceare reachable via e-mail

¾ Your survey is short and simple¾ You need results quickly

Web-based surveys are a better choice when:

¾ Members of your target audienceare regular Internet users

¾ Your survey is more complex (e.g.responses to each question deter-mine which questions are answerednext, etc.)

¾ You have access to the services of aconsultant who can help you withthe technological aspects of Websurvey design

COLLECTING EVA L UATION DATA: SUR V E Y S ¢ ¢ ¢ 7

PLANNING A SURVEY

Once you’ve decided to conduct a survey,take time to plan it well in order to savetime and expense later. There are a numberof things to consider as you plan yoursurvey. This is not an ordered step-by-stepprocess. Many of the following factors willbe considered simultaneously as you plan ameaningful survey.

1. Decide who should be involved inthe process

Involving intended users and key stakehold-ers in the survey leads to greater commit-ment and helps ensure the survey is relevantand is used. Key people might be involvedin the planning, questionnaire construction,data collection or analysis and interpreta-tion. They might serve in an advisory role,be ‘helpers’, be involved in or manage oneaspect of the survey or be fully involvedthroughout survey implementation.

Who has a vested interest in the survey?Who else might be interested in the processor the results? How might they best beengaged? As you lay out the tasks that needto be done, you may delegate certainresponsibilities.

2. Define survey content

Clarifying the specific content and topics ofthe survey is difficult but necessary; other-wise it is possible to include a lot of ques-tions that do not answer your needs. Defin-ing the survey content means setting bound-aries so you can write the correct questions.Begin by determining what you want toknow and defining any terms that may bevague or ambiguous. For instance, you maywant to know if youth who participate in aleadership development program actuallyimprove their leadership skills. Leadership

can be expressed in various ways. What doesleadership mean in this program? Whataspects of leadership are you interested in?What would improved leadership look likeamong this group of participants?

Second, focus on your priority informationneeds. Make a distinction between what itwould be nice to know and what you need toknow.

Third, ensure that the potential surveyrespondents can actually provide the infor-mation. Sometimes people are reluctant torespond to certain topics or are not able torespond. That is, they may have forgotten,may not know or don’t have enough infor-mation to respond accurately. In such cases,it is better to seek another source of informa-tion, remove the topic from the survey orwait until the respondents are able to answer.

Finally, don’t include topics in your surveythat you are not able to act upon or doanything about.

3. Identify your respondents

From whom will you collect information?With relatively small programs, you mayneed to include all participants in the surveyto have enough data. Or, there may be caseswhere political considerations demand thatall people in the district or city be included.Often, however, surveys collect informationfrom a sample, a subset of a larger popula-tion, to save time and money. If you use asample, you will need to determine a samplesize and sampling method. For help withthis, see another Extension publication,Sampling (UW-Extension publicationG3658-3).

4. Decide on the survey method

Choose a survey method that best suits yourtopic, respondent characteristics, time line,


experience and resource availability. Asindicated in the previous section, one surveymethod is not better than another. However,one method may be more suitable thanothers depending upon the situation.

5. Develop the questionnaire

All surveys use questionnaires. However,the number and type of questions, theformat of questions and the layout of thequestionnaire vary depending on the type ofsurvey you choose. Questionnaire develop-ment is more an art than a science and italways takes longer than you expect. Planon writing several draft questionnaires. Inyour planning, include adequate time todevelop a clear, comprehensible question-naire that elicits the information you need.Working with others often helps build abetter questionnaire.

When you are developing the questionnaire,review the specific content and topics of thesurvey in order to keep your questionsdirected to what you want and need toknow. Be sure that you cover the completecontent that you identified as important tothe survey. Often it is good to write morequestions than you will use and then discardthe extras.

There are many ways to write questions andformat a questionnaire. The quality of yourquestionnaire largely determines the useful-ness of your survey results. For moreinformation on questionnaire development,see Questionnaire Design: Asking Questionswith a Purpose (UW-Extension publicationG3658-2).

6. Pilot test the questionnaire andother materials

Pilot testing is another key step along thepath to ensuring a good quality survey. Plan

for when and with whom you will conductthe pilot test. The pilot test should be donewith a small number of respondents who areas similar as possible to your potentialrespondents. Pilot testing helps you answerthe following questions:

¾ Do respondents understand thequestions?

¾ Do the words you use mean thesame thing to all respondents?

¾ Do the questions elicit the type ofinformation you want?

¾ Do respondents seem to understandthe instructions?

¾ How long does it take to completethe survey?

Sometimes we need to pilot test severaltimes before arriving at a satisfactoryquestionnaire. Consequently, you may wishto plan a bit of extra time for the question-naire development and pilot testing steps.Remember that the pilot test data is notincluded with the survey data, and therespondents who participated in the pilottest do not complete the actual survey.

7. Think about analysis

Often it is useful to construct a hypotheticalreport before you even start to collect data.This helps you focus on what you need toknow, what you want to be able to say, andwhat analyses you will need in order to sayit. Your data analysis needs should beconsidered at the same time you are choos-ing your sample size and sampling method.

Do you only need frequencies and percent-ages? Will you have sufficient data tocompute anything more than averages? Doyou need to be able to talk about subgroupswithin the respondent population? Will youuse hand tabulation or is the data set so largethat a computer program is needed? Whatresources and capabilities do you have?

8


Often our analysis abilities and resourcesdetermine what our survey looks like.

8. Communicate about your surveyand its results

Plan for communications. When and howwill you communicate with stakeholders?Some may want regular updates as thesurvey is being planned and carried outwhile others may only want to know whenresults are available. Who will need andwant to know about the results? When willthe results be released? In what format? Towhom? Some up front planning withresponsibilities laid out will help ensure thatyour survey is used.

9. Develop a budget, time line andmanagement process

Undertaking a survey is not without costs.There are costs associated with staff time,equipment and materials (phone charges,envelopes, printing, computer, etc.) andpostage costs for mail surveys to name afew. Each survey method has a differentprice tag attached. While costs of surveysare hard to estimate and can differ substan-tially depending upon sample size, wagerates and the amount of volunteered timeand supplies, in general, mail surveys arethe least expensive and face-to-face surveysare the most expensive. In any event, it isimportant to consider your resources,identify all expected costs and develop abudget accordingly.

A management plan helps you structuretime, resources and responsibilities. Asimple work sheet, as seen in Figure 1, canbe used to list tasks, estimated time neededfor each task, who is responsible andresources needed to accomplish the task.

Often we have a deadline for evaluation

results before we even begin the planning. Ifthat is the case, you may work backwardsfrom the finish date, estimating the amountof time associated with each task andadjusting when possible to be able to meetthe deadline. In other cases, you workforward from the project beginning data andmay have more flexibility about time lines.Figure 2 shows an example of a Gantt chartuseful in planning and managing time linesin a survey project. It lists activities, whenthey are expected to begin and how longeach will continue. The time frames illus-trated in the chart are merely suggestivesince each survey differs depending upon itssize and scope.

As we plan a survey, trade-offs always exist.Expediency and resource constraints moreoften determine how a survey gets imple-mented than any grand plan. However, goodplanning can help us use our resources morethoughtfully and effectively.

IMPLEMENTING A SURVEYOR GETTING IT DONE

The following section list the steps involvedin completing mail and telephone surveys.Many of these tasks are similar to what youwould do for other types of surveys withminor adaptations.

Develop the questionnaire

In addition to coming up with specificquestions, you need to design the question-naire to “fit” the survey method. The impor-tant thing to remember when designing amail survey is that the questionnaire shouldbe visually appealing, easy to read andcomplete. This increases the likelihood that

Mail Surveys


Figure 1. Example Management Planning Work Sheet (small-scale mail survey)

Task Begin-end date Resources neededPerson

responsible

1. Talk with committee membersabout their involvement in thesurvey; purpose and expectationsof the survey; uses of the survey;time frame

2. Determine specific informa-tion needs; prioritize informationneeds; seek consensus among keystakeholders

3. Share knowledge about therespondents; seek additionalinformation if necessary;determine sample and procedures

4. Discuss pros/cons of varioussurvey methods – make decision

5. Develop questionnaire. Sharefinal questionnaire with commit-tee and key stakeholders

6. Pilot test. Make revisions.

7. Develop cover letter, materi-als, mailing lists

8. Administer the survey

9. Update committee and otherkey stakeholders on surveyprogress

10. Process and analyze the data

11. Report and communicate

May 15th mtg

May 16- May 30

May 16-May 30

May 16-May 30

1st draft: June 52nd draft: June 153rd draft: June 25Final draft: July 15

July 15- Aug 30

July 15- Aug 30

Sept – Oct

Oct 1

November

Report due:December 1

Sue, Chad

Sue, Chad, Ron,Mary

Sue, Chad, Ron,Mary

Sue, Chad, Ron,Mary

Sue. Everyonewill draft ques-tions; Sue will putit together,circulate drafts forreview & finalize

Mary to organizeand supervise

Sue

Sue

Sue, Chad

Chad

Ron. Chad, Sue,Mary: review

Demographicprofiles

Survey design info

Help with formattingquestionnaire;graphics support

Help with identify-ing individuals

Clerical support

Printing, postage,envelopes, clericalsupport

Computer software;clerical support

Printing; slidepreparation; clericalsupport


respondents fill it out. Keep the followingtips in mind:

� Order questions logically.� Use plenty of white space.� Give brief instructions on how to com-

plete the questionnaire exactly wherethey are needed.

� Use a type size large enough to be easilyreadable (at least 10 point).

� Separate different components of thequestionnaire by using different typestyles. For example, put questions in boldand response choices in regular typeface(Figure 3).

� Use arrows to show people where to gonext (Figure 3).

� Don’t include information for internaluse only, e.g. data entry instructions.

Figure 2. Example of a Gantt Chart

1. Develop questionnaire2. Pilot test questionnaire3. Develop other materials4. Assemble all needed materials5. Collect data6. Code and enter data7. Analyze and interpret data8. Write reports

9. Publicize findings

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Time (weeks)

Pilot test the questionnaire

When pilot-testing a mail survey, you canhave respondents complete the question-naires in person rather than receiving themthrough the mail. This way, you can seehow people fill out the questionnaire andwhere problems may exist. Ask the pilot testgroup to fill out the surveys; then “debrief”the participants to get their feedback.

Decide if responses are to be anony-mous or confidential 1

Respondents may feel more comfortableknowing their individual responses can’t beidentified (anonymous). However, thisincreases your survey costs. You have tosend follow-up mailings to all respondentssince you don’t know who has alreadyresponded.

Develop a cover letter to send withthe survey (Appendix A). Personalizeit as much as possible

� Include information about the purposeand importance of the survey.

� Identify the survey sponsor.

1 Anonymous means individuals’ responses can’t be identified. Confidential means individuals’responses won’t be identified.

Figure 3. Example of formatting

1. Do you own or rent the place youlive in?

a. Own � GO TO QUESTION 10 b. Rent �2. What is your monthly rent?


� Explain your involvement with theproject. Use letterhead, if appropriate.

� Explain why the respondent was selectedto participate.

� Explain how the respondent will benefitfrom completing the survey.

� Assure respondents of anonymity orconfidentiality.

� Explain how results will be used and giveinstructions for requesting a copy ofresults.

� Give instructions for returning the survey.� Supply contact information.� Hand-sign the letter.

Develop advance notice materials forrespondents

These may be in the form of a letter (Ap-pendix A), a postcard or a brief phone call.The advance notice alerts respondents thatthey will be receiving a survey and mayhave them “on the lookout” for the materialsin the mail.

Develop follow-up postcards toremind respondents to return thesurvey

Alternately, you may develop a brief phonescript to use if you choose to follow up witha phone call (Appendix A).

Prepare mailing materials

In addition to the survey, cover letter andfollow-up postcards, you also need mailinglabels, outgoing envelopes and returnenvelopes. The idea is to personalize thesematerials as much as possible.

· Mailing labels: print out enough sets ofmailing labels to correspond with thenumber of mailings you plan to do. If youplan to track responses to your survey, printan extra set to use as a master list andnumber each label on this set sequentially.

· Outgoing envelopes: experts recommendusing stamped envelopes rather than me-tered mail or indicia.

· Return envelopes: always include a selfaddressed, stamped envelope (SASE).Again, experts recommend using stampedenvelopes rather than business-reply envel-opes.

Do advance notice about one weekbefore you begin the survey

Send out advance notice letters or postcardsor make phone calls to your target audience.

Assemble mailing materials and sendfirst mailing

If you plan to track responses to your survey,number each one sequentially to correspondwith the mailing labels. Make sure to matchthe numbers on mailing labels with thenumbered surveys when you put the materi-als in the envelopes. (If you don’t plan totrack survey responses, this is not neces-sary.) The first mailing includes the coverletter, survey and SASE.

Track survey responses

As surveys are returned, remove mailinglabels from the master list so that duplicatemailings are not sent. (Again, if you areconducting an anonymous survey, this stepis not necessary.)

Send follow-up postcards

This postcard should remind respondentsthat they received the survey and emphasizehow much their input is valued. Send thisone week after the first mailing.


Send second complete set of materials

Do this about two weeks after you havemailed the reminder postcard. You may alsochange the tone of the cover letter to place

more emphasis on the importance of thenonrespondents’ contribution and to againensure confidentiality or anonymity.

Send final mailing

This can be either another complete set ofmaterials or a final reminder postcard.Establish a deadline for responding. Do thisabout two weeks after the last mailing.Figure 4 illustrates a time line for sendingmail survey materials.

Telephone surveys

Develop the questionnaire

Telephone surveys are designed to make iteasy for the caller to ask questions over thephone and for the respondent to understandand respond to those questions. Keep thefollowing recommendations in mind:

� Develop a brief introduction to use when firstcontacting the respondent (Appendix B).Include:

¾ The caller’s name¾ The organizational sponsor¾ A procedure for making sure the

correct respondent is on the telephone¾ The purpose and importance of the

survey

oComplete planning processoDevelop the questionnaireoPilot test the questionnaireoDecide if survey is anonymous or

confidentialoDevelop a cover letteroDevelop advance notice materialoDevelop follow-up postcardsoPrepare mailing materialsoDo advance noticeoSend 1st mailingoTrack survey responsesoSend follow-up postcardoSend 2nd mailingoSend final mailingoProcess dataoAnalyze and interpret dataoWrite reportsoCommunicate findings

Mail survey checklist

Figure 4. Time line for sending mail survey materials

Advancenotice

1st mailing:all materials

3rd mailing:all materials

2nd mailing:follow-uppostcard

4th mailing:postcard orall materials

2 weeks 2 weeks1 week1 week


¾ An assurance of confidentiality oranonymity

¾ The option for the respondent toparticipate now or to schedule amore convenient time

� Make the first question close-ended withonly a few response choices (e.g. yes orno) to get the respondent used to answer-ing questions over the telephone.

� Keep questions short and simple with aminimum of answer choices.

� Use a consistent format. For example, usebold for words to be read, italics forwords sometimes read (e.g. probes formore information) and all caps for callerinstructions (Salant and Dillman, 1994).

Anticipate questions respondentsmay have

Develop a list of answers to frequentlyasked questions so that you can provideconsistent responses.

Develop call logs

Call logs (Appendix B) for each telephonenumber to be contacted help keep track ofthe number of calls made and the outcomeof each (e.g. survey completed, no answer,reschedule interview, etc.).

Establish guidelines for handling “no”answers, hesitant participants, etc.

Establish a maximum number of times eachnumber is called before it is considered a“no response.” Set guidelines for handlingrespondents who are hesitant to participate.Should they be called back at another time?Should they be given additional information(such as emphasizing the brevity of thesurvey)?

Pilot test the questionnaire andprocedures

It is important to conduct the pilot test(s)exactly as you would the actual survey. Allrespondents should receive all the informa-tion as they would in the actual survey, andall procedures should be carried out exactlyas planned. Pilot testing allows you todetermine how easily both respondents andthe caller(s) can follow the procedures andquestionnaire.

Develop advance notice materials forrespondents

This may be in the form of a letter (Appen-dix A) or a postcard. The advance noticealerts respondents that they will be con-tacted and “paves the way” for the caller toestablish credibility and rapport with them.

Arrange for facilities and hire callersif necessary

Based on your pilot test of the telephonesurvey, you have an idea of the amount oftime it takes to complete an interview aswell as the number of interviews completedper hour. Use this information to estimatethe number of telephones and callers youneed to collect information by your targetdate. Arrange for telephones and hire callersif needed.

Assemble all needed materials

Write a telephone number for each respon-dent on a questionnaire and attach a call log.

Train the caller(s) if necessary

The purpose of training is to make sure thecaller(s) know how to administer the ques-tionnaire consistently to all respondents. It isimportant to read questions as written, toprovide consistent answers when respon-dents ask questions and to probe for more


information when necessary in a non-leading fashion. Do not skip asking aquestion even if you think you already knowthe respondent’s answer.

Do advance notice about a weekbefore you begin the survey

Send out the advance notice letter or postcard to members of the target audience.

Schedule interviews and collect data

Think about when the respondents are likelyto be available. Usually the best time toreach people at home is between 4 and 9P.M. weekdays and 10 A.M. to 4 P.M. onSaturdays.

oComplete planning processoDevelop the questionnaireoAnticipate respondent questionsoDevelop call logsoEstablish guidelines for no an-

swers, hesitant participants, etc.oPilot test the questionnaire &

proceduresoDevelop advance notice materialoArrange for facilities & callers if

necessaryoAssemble materialsoTrain callers if necessaryoDo advance noticeoSchedule interviewsoCollect dataoProcess dataoAnalyze and interpret dataoWrite reportsoCommunicate findings

Telephone surveychecklist

GETTING A GOODRESPONSE TO YOUR SURVEY

A high response rate promotes confidencein your results. Response rate is the propor-tion of people in the particular sample thatparticipate in the survey. It is calculated bydividing the number of returned surveys bythe total number of surveys distributed.There is no standard response rate. In somecases, 80 percent and above may be ex-pected. In other cases, 60 percent may beadequate. Lower response rates increase thelikelihood of biased results. How do weknow that our results represent everyone inthe survey or represent the response of aselect group?

It is useful to discuss and establish anacceptable response rate with your stake-holders before beginning the survey. Con-sider the target audience, level of responseyou might expect, past experience withsurveys of this type and any other relevantfactors. What is considered an acceptableresponse rate?

Low response rates introduces the problemof nonresponse error. According to Salantand Dillman (1994) anything under 60percent should be a warning. If this hap-pens, find out whether the people whodidn’t respond are different than those whodid in ways that matter to your study. Youmay be able to figure this out through yourown knowledge of the sample, from infor-mation on the mailing list or from compar-ing respondents to a larger set of data thatdescribes the population. Sometimes it ispossible to telephone nonrespondents andsee how they compare on a few key charac-teristics with the respondent group. If bothgroups appear to be the same on key at-tributes of relevance to your study (e.g.,farm size, years of residence, family type,etc.), then a low response rate may not be aproblem. However, if these groups differ, it


is important to indicate how they differwhen reporting results. Qualify your find-ings to state who is and who is not repre-sented in the findings.

There are various things you can do toincrease your survey response rate. Con-sider these specific tips:

Get positive publicity for your survey.Announce the survey at a gathering whererespondents are likely to be present. Con-sider press releases or other ways to pro-mote the survey (see Appendix C).

Oversample. If you know the number ofpeople you need responses from and the ex-pected response rate, increase the samplesize accordingly. For example, if you expect75 percent of the people to respond andwant at least 75 responses, use a sample sizeof 100. (Divide the number of completedsurveys you want by the expected responserate to calculate the required sample size.)

Help respondents see the value of partici-pating in the survey. Tell them “what’s in itfor them.”

Use a combination of different methods.Use the least costly method first to reach thegreatest number of respondents (e.g. mail ore-mail). Then follow up with phone calls tononrespondents. Be aware that responsesmay be influenced by the type of methodused to collect information (known as the“method effect”).

With mail and e-mail surveys, the greaternumber of follow-up contacts, the higherthe response rate. For mail surveys, a goodrule of thumb is that each contact will return50 percent of the response to the previousmailing (Mangione, 1995). With effort,initial response rates of 20 percent havebeen raised to 70 or even 80 percent (Fink,1995).

Use incentives. For mail surveys, evensmall amounts of money ($1 - $2) have beenshown to increase response rates by 5 – 8percent (James & Bollstein, 1990). Givingthe incentive up front is more effective thanrequiring the respondent to complete thesurvey first. With handout surveys, youmight hold a raffle and draw the winnerusing completed surveys to encourageparticipation.

With mail surveys, use 2-day prioritymail for the final mailing of the question-naire. This lets those who haven’t yetresponded know how important theirresponse is.

At conferences or workshops wherehandout surveys are used, include timewithin the program for participants tocomplete the survey. Don’t expect partici-pants to fill out questionnaires after theclosing.

With Internet-based surveys, make it aseasy as possible for respondents to par-ticipate. Send them an e-mail announcingthe survey and include the URL as ahyperlink directly to the survey.


INTERPRETING RESULTSWHEN YOUR RESPONSERATE IS LOW

Sometimes, despite careful planning andimplementation, you receive a poor re-sponse to your survey. In these cases, youmay still use your results, but interpret themwith caution.

¾ When analyzing and interpretingdata, use simple descriptive statis-tics (counts, percents, averages). Donot attempt more complex statisticalanalyses like correlations whenresponse rates are low.

¾ Use language in your reporting that issuggestive rather than decisive. Forexample, use phrases like “it seems,”“it appears,” “the data suggest” ratherthan “these data show,” “we canconclude that.”

¾ Do not generalize findings to theentire group. Report results in termsof “these respondents” rather thangeneral terms like “parents in thiscounty.”

¾ Clearly describe whom the resultsrepresent. Give as much specificin-formation about the demographicand other characteristics of respon-dents as you have available.

SUMMARY

A survey is a systematic, standardized way ofcollecting information from people usingquestionnaires. There are many differentways of conducting a survey. Use the methodthat best fits your purpose, survey topic,target audience and resources. Spend ad-equate time planning your survey project tomake sure all aspects are well thought out.Use the tips in this booklet to help you get agood response to your survey and collectuseful information that meets your needs.


gies in Evaluation: Social, Moral, Episte-mological and Practical Implications. NewDirections for Evaluation, No. 84. SanFrancisco: Jossey-Bass Publishers.

Worthen, Blair R. and James R. Sanders.1987. Educational Evaluation: AlternativeApproaches and Practical Guidelines. NewYork: Longman.

REFERENCES

Bourque, Linda B. and Eve P. Fielder. 1995.How to Conduct Self-Administered and MailSurveys. Thousand Oaks, CA: Sage Publica-tions, Inc.

Fink, Arlene. 1995. How to Analyze SurveyData. Thousand Oaks, CA: Sage Publica-tions, Inc.

Fink, Arlene. 1995. The Survey Handbook.Thousand Oaks, CA: Sage Publications, Inc.

Fink, Arlene and Jacqueline Kosecoff.1985. How to Conduct Surveys: A Step-byStep Guide. Beverly Hills, CA: SagePublications, Inc.

James, J. M. and R. Bollstein. 1990. “Theeffect of monetary incentives and follow-upmailings on the response rate and responsequality in mail surveys.” Public OpinionQuarterly, 54:346-361.

Lavrakas, Paul J. 1998. “Methods forsampling and interviewing in telephonesurveys.” In L. Bickman and D. J. Bog(eds.), Handbook of Applied Social Re-search Methods. Thousand Oaks, CA: SagePublications, Inc.

Mangione, Thomas W. 1995. Mail Surveys:Improving the Quality. Thousand Oaks, CA:Sage Publications, Inc.

Polland, Ronald J. 1998. Essentials ofSurvey Research and Analysis: A Workbookfor Community Researchers. Duval CountyHealth Department.

Salant, Priscilla and Don A. Dillman. 1994.How to Conduct Your Own Survey. NewYork: John Wiley & Sons, Inc.

Watt, James H. 1999. “Internet systems forevaluation research.” In G. Gay and T. L.Bennington (eds.), Information Technolo-


APPENDIX A. SAMPLE MAIL SURVEY MATERIALS

Sample Advance Notice Letter

Date

Dear _________________________:

Within the next few days, you will be receiving a questionnaire in the mail. You have beenselected as one of a small number of people asked to give their opinion about the needs,strengths, and direction of Badger County. Your input will be used to help county officialsplan for the future of our community.

We would greatly appreciate your time in filling out and returning the questionnaire. Thankyou in advance for your help.

Sincerely,

Badger County Office Department Head, Chairperson of Oversight Committee


Sample Cover Letter

Date

Dear _________________________:

As a resident of Badger County, you know about concerns and issues facing the people ofour county. Knowing how people view their county--its current needs and strengths, and howthey would like the county to be in the future--is vital to those who make decisions about theprograms and services that are available.

You are one of a small number of people who are being asked to give their opinion aboutthe current situation in Badger County and your vision of the future. Your name was drawnrandomly from a list of all registered voters. In order for the results of this survey to trulyrepresent the thinking of the people of Badger County, it is important that you fully completethe enclosed questionnaire and return it in the postage-paid envelope provided.

You may be assured of complete confidentiality. The questionnaire has an identificationnumber for mailing purposes only. This is so that we may check your name off the mailinglist when your questionnaire is returned. Your name will never be placed on the question-naire. No individual responses will be reported. A summary of the results will be mailed toyou when completed.

When you have completed your survey, please return it in the enclosed, postage paidenvelope. If you have any questions about this survey, please contact me at (phone number).Thank you very much for your help in this important endeavor.

Sincerely,



Sample Follow-Up Post Card

Last week, a questionnaire was mailed to you seeking your opinions about the concernsfacing people in our county and what you would like the county to be like in five years. Yourname was selected randomly from a list of registered voters.

If you have already completed and returned the questionnaire, thank you very much. If not,please do so today. We appreciate your help because we know how useful your ideas will bein setting direction for the county.

It you did not receive a questionnaire, or if it was misplaced, please call us at (phone number)and we will get another one in the mail to you immediately.

Sincerely,



Result codes:

CS = completed surveyCB = respondent contacted; call back at scheduled timeNA = no answerLM = left message; call backRF = refusal

22

Call date Time Result code Comments

APPENDIX B. SAMPLE TELEPHONE SURVEY MATERIALS

Sample Telephone Survey Introduction

Hello, my name is _____________________ and I’m calling from the University ofWisconsin-Cooperative Extension. We are assessing our work in parenting educationand I’d like to talk with you about a program you participated in last spring. Thissurvey is completely voluntary and confidential. The questions will take about 5 min-utes of your time. Do you have a few minutes now? (If yes, proceed with interview. If no,ask for date and time to call back. Record date and time on call log in comments section.)

Sample Call Log

Phone number: _____________________


APPENDIX C. SAMPLE PRESS RELEASES

News ReleaseBADGER COUNTY RESIDENTS ENGAGEDIN COUNTY SURVEY

Your Town, WI, September 23, 2001: Badger County residents have a chance to

voice their opinions about county needs, concerns and strengths. A mail survey

is being distributed Monday to about 200 households by the Badger County

Extension Office. Households were randomly selected to participate to represent

public opinion across the county.

Residents selected for the survey are urged to complete the brief survey within

one week and return it in the accompanying stamped envelope. This survey will

give direction to the county Extension office and other service agencies and

organizations who provide education and services in Badger County. This is an

opportunity for local citizens to provide input to identify issues to be addressed

through local and state resources. All responses to the survey are confidential.

The results will only be representative if there is a high rate of return. “Only a high

return will give an accurate view of our concerns,” says County Board Chairper-

son, Jane Chair. A steering committee of local residents helped design the

survey.

For more information on the survey and the Extension statewide planning effort,

contact Bob Agent at 456-7890.

Contact: Bob AgentBadger County ExtensionPhone 123-456-7890Fax 1 -823-456-789012345 Main StreetYour Town, WI 55555

23


Contact: Bob AgentBadger County ExtensionPhone 123-456-7890Fax 123-456-789012345 Main StreetYour Town, WI 55555

News ReleaseCOUNTY SURVEY ACHIEVES RECORDSETTING 90% RETURN

Your Town, WI, December 28, 2001: The Badger County Assessment of Con-

cerns and Strengths conducted by Badger County UW-Extension Office in

conjunction with the County Board of Supervisors has set a new record. Of the

200 households randomly sampled to participate in this survey, 90 percent

returned their surveys.

“We can feel confident that these results represent the opinions and feelings of

our county population,” exclaimed Pete Person, Chairperson of the Extension

Education Committee of the County Board of Supervisors. “This is a very real

statement about the commitment of our county citizens to be involved in the

future of Badger County.”

The results are being analyzed and will be shared at the next meeting of the

Extension Committee on _______ at ____________. This countywide assess-

ment was designed to provide direction for the programs and activities to be

delivered by the UW-County Extension Office and other agencies in the county

during the next four years.

24

Authors: Ellen Taylor-Powell is an evaluation specialist for Cooperative Extension, University of Wisconsin-Extension.Carol Hermann is an evaluation associate.

Issued in furtherance of Cooperative Extension work, Acts of May 8 and June 30, 1914, in cooperation with the U.S.Department of Agriculture, University of Wisconsin-Extension, Cooperative Extension. An EEO/Affirmative Actionemployer, University of Wisconsin-Extension provides equal opportunities in employment and programming, includingTitle IX and ADA requirements. If you need this information in an alternative format, contact the Office of EqualOpportunity and Diversity Programs or call Extension Publishing at (608) 262-2655.

© Copyright 2000 by the Board of Regents of the University of Wisconsin System doing business as the division ofCooperative Extension of the University of Wisconsin-Extension. Send inquiries about copyright permission to: Director,Cooperative Extension Publishing, 201 Hiram Smith Hall, 1545 Observatory Dr., Madison, WI 53706

You can obtain copies of this publication from your Wisconsin county Extension office or from Cooperative ExtensionPublications, 45 N. Charter St., Madison, WI 53715, (608) 262-3346 or 1-877-947-7827 (WISPUBS). Before publicizing,please check on this publication’s availability. This publication is available in PDF format on the Internet on theProgram Development and Evaluation web page at http://www.uwex.edu/ces/pdande/evaluat.htmlTo see more Cooperative Extension publication visit the web site: http://www.uwex.edu/ces/pubs/

Collecting Evaluation Data: Surveys by Ellen Taylor-Powell and Carol Hermann (G3658-10)

Cooperative ExtensionUniversity of Wisconsin-Extension

Program Development and Evaluation432 North Lake Street, Room 613Madison, WI 53706Phone: (608) 262-9940Fax: (608) 262-9166

Documents

Collecting Evaluation Data: Surveys