Sampling and Surveying Handbook

8/13/2019 Sampling and Surveying Handbook

1/72

Air UniversitySampling and Surveying

Handbook

Guidelines for planning, organizing,and conducting surveys

The views and opinions epressed represent the personal views of the author

and editor only and should not in any way be construed to reflect any

endorsement or confirmation by the !epartment of !efense, the !epartment

of the Air "orce, or any other agency of the United States Government#

$evised %&!"' (dition) *ay ++

&revious (ditions) -../, -..0, -.11, -.12

This handbook contains guidelines for planning, organizing, and conductingsurveys# 3t should be useful to anyone embarking on a pro4ect re5uiring the

gathering of data through the medium of the 5uestionnaire# The tet is designedto be easily readable, even for someone with a limited background on the sub4ect#

The book is the product of the efforts of several people# *a4or 6eith 7# $ossdid the ma4ority of the work in fulfillment of his research re5uirements while astudent at the Air 7ommand and Staff 7ollege, Air University, *awell Air "orce8ase, Alabama in -.99# :t 7ol :awrence!# 7lark was the research advisor for the pro4ect, and *a4or Thomas 7# &adgett didthe final editing and assembling on the original edition# !r# Thomas $# $enckly, Air


2/72

University 7urriculum 7oordinator, edited the -.11 reprint of the first edition, the-..0 second edition, and the -../ 3nternet edition, in addition to providingsupplemental information on bias in survey research %7hapter 2' and commonstatistical analysis errors %7hapter /'# 3f you have any 5uestions about the book,!r# $enckly can be reached at)

H; AU' .20=+.1. or!S? >.0=+.1.thomas#renckly@mawell#af#mil

To say that peoples opinions and attitudes are more important now thanever before is very nearly an understatement# *ore and more we are seeingindividuals and groups relying heavily on the opinions and attitudes of customers,constituents, concerned citizens, focus groups, etc#, to provide information fordecision making# 3t is also obvious even to the casual observer that surveys%including paper=based 5uestionnaires, personal interviews, and telephone polls'

play a crucial role in gathering these opinions and attitudes#

Surveys are also used as evaluation and control devices# They can be usedto measure the effectiveness of an ongoing pro4ect, such as an informationprogram, for eample# 8y surveying the participants in a program, theeffectiveness of the program can be determined# Also, management can usesurveying as an aid to control, by finding new problem areas and insuring that oldproblem areas have been corrected %which is, for instance, one of thefundamental premises of total 5uality management'#

The need for accurate information to fuel the decision=making process eists

at all levels of management# This has created a trend for surveys to be generatedat lower management levels by staff officers, many of whom are not eperiencedin survey development or administration# The growing necessity to survey andthe relative lack of knowledge on surveying methodology leads to a significantdemand for information on the sub4ect# The primary purpose of this guide is tosupply this information in simple, non=technical language#

An e5ually important purpose of this guide is to identify problems that mayarise during development of a survey and to provide techni5ues and guidance forsolving these problems# The procedures presented in this guide are designed tohelp you develop valid and useful surveys#

The steps in surveying are varied and comple# Therefore, this guide onlyhighlights the ma4or information, techni5ues, and procedures available to thesurveyor# $eferences offering more detailed treatments of these sub4ects areprovided in the bibliography and appendices# Since the surveyor fre5uently isunable to reach the entire group in which he is interested, this guide eplainssampling techni5ues# To use these techni5ues necessitates only a rudimentaryknowledge of statistics# "inally, although many of the techni5ues and procedurescovered here apply e5ually well to the personal or telephone interview survey, the


3/72

primary focus is on the self=administered and group=administered surveys#

Bne word of caution) 8ecause the steps in survey preparation are closelyinterrelated, you should study this entire guide before beginning an initial surveyeffort#

"B$(CB$! ####################################################################################################################### i

&$("A7(###########################################################################################################################ii

7HA&T($- #######################################################################################################################-

3?T$B!U7T3B? TB SU$D(E3?G######################################################################################## -Types ofSurveys########################################################################################################## -Advantages F !isadvantages of Two Types ofSurveys############################################# -To Survey or ?ot toSurvey ######################################################################################### 0GuideButline ###############################################################################################################

>

7HA&T($+ #######################################################################################################################9

TH( A3$ "B$7( &($SB??(: SU$D(E &$BG$A*############################################################## 9

7HA&T($0 #######################################################################################################################.

!(D(:B&3?G TH( &U$&BS(, HE&BTH(S3S, F SU$D(E &:A?############################################.The&urpose################################################################################################################## .The Hypothesis, Bb4ective, or $esearch;uestion##################################################### -The Survey&lan ######################################################################################################### --The !ata 7ollection


4/72

&lan ########################################################################################### --The !ata $eduction and $eformatting&lan ############################################################## -+Bpen= and 7losed=(nded;uestions ########################################################################## -+The Analysis

&lan ###################################################################################################### -0

7HA&T($> ##################################################################################################################### -2

SA*&:3?G T(7H?3;U(S A?! $(:AT(! STAT3ST37A: 7B?7(&TS ###################################-2Sampling*ethods###################################################################################################### -2Simple $andomSampling########################################################################################## -.

Stratified $andomSampling ###################################################################################### +SystematicSampling################################################################################################## ++udgmentSampling#################################################################################################### +0&urposiveSampling#################################################################################################### +0$epresentativeSampling############################################################################################ +>"actors 3nfluencing Sample

Size################################################################################ +>7onfidence :evel and&recision################################################################################# +2

!etermining the Size of theSample########################################################################### +2

7HA&T($2 ##################################################################################################################### +.

TH(;U(ST3B??A3$(######################################################################################################+.&ros and

7ons############################################################################################################# +.The

7ontents ##############################################################################################################0-

Types of;uestions##################################################################################################### 00


5/72

;uestionnaire7onstruction######################################################################################## 00

3ntensity ;uestions and the :ikertScale#################################################################### 09

8ias and How to 7ombat3t######################################################################################## 0.

8ias 3nvolving DolunteerSamples############################################################################# >-7onclusions Carranting *aimum 7onfidence#####################################################

>+7onclusions Carranting 7onsiderable 7onfidence############################################### >+7onclusions Carranting Some 7onfidence############################################################ >07onclusions Carranting *inimum

7onfidence #####################################################>>Summary ###########################################################################################################

######### >2

7HA&T($

/ ##################################################################################################################### >9

7B**B? STAT3ST37A: A?A:ES3S ($$B$S #################################################################### >9

A&&(?!3A################################################################################################################### 2-

ST(&S 3? SU$D(E3?G######################################################################################################2-

A&&(?!3

8 ################################################################################################################### 20

SU$D(E !(D(:B&*(?TT3*(TA8:(S ############################################################################ 20

A&&(?!37 ################################################################################################################### 22

SU$D(E !ATA SBU$7(S ################################################################################################22

A&&(?!3

!################################################################################################################### 29

HBC TB US( A $A?!B* ?U*8($TA8:(######################################################################29TA8:( B" $A?!B*

?U*8($S######################################################################################## 21

A&&(?!3( ################################################################################################################### 2.


6/72

TA8:( B" I DA:U(S ######################################################################################################2.

A&&(?!3"#################################################################################################################### /-

SA*&:(S ######################################################################################################################## /-7over:etter ###############################################################################################################/-&rivacy ActStatement################################################################################################ /+Survey3nstructions##################################################################################################### /0

A&&(?!3G################################################################################################################### /2

$AT3?G B$ 3?T(?S3TE S7A:(S#######################################################################################/2TA8:( D333=(=- ==A77(&TA83:3TE &H$AS(S ##############################################################/2TA8:( D333=(=+ ==!(G$((S B" (7(::(?7() "3$ST S(T ##########################################/9TA8:( D333=(=0 ==!(G$((S B" (7(::(?7() S(7B?! S(T ######################################/1TA8:( D333=(=> ==!(G$((S B" :36( A?! !3S:36( #####################################################/.TA8:( D333=(=2 ==!(G$((S B" GBB! A?! &BB$ #######################################################9TA8:( D333=(=/ ==!(G$((S B" GBB! A?! 8A! #########################################################9-TA8:( D333=(=9 ==!(G$((S B" AG$(( A?! !3SAG$((###############################################

9+TA8:( D333=(=1 ==!(G$((S B" *B$( A?! :(SS ########################################################90TA8:( D333=(=. ==!(G$((S B" A!(;UAT( A?! 3?A!(;UAT( ###################################9>TA8:( D333=(=- ==!(G$((S B" A77(&TA8:( A?! U?A77(&TA8:(##########################92TA8:( D333=(=-- == 7B*&A$3SB? &H$AS(S##################################################################9/TA8:( D333=(=-+ ==!(G$((S B" SAT3S"A7TB$E A?! U?SAT3S"A7TB$E ################### 99TA8:(D333=(=-0 ==!(G$((S B" U?SAT3S"A7TB$E#################################################### 99TA8:(D333=(=-> ==!(G$((S B" &:(ASA?T################################################################ 99TA8:(D333=(=-2 ==!(G$((S B" AG$((A8:( ############################################################# 91TA8:(D333=(=-/ ==!(G$((S B" !(S3$A8:( ############################################################## 91TA8:(D333=(=-9 ==!(G$((S B" ?37(######################################################################### 9.TA8:(D333=(=-1 ==!(G$((S B" A!(;UAT(############################################################### 9.TA8:(

D333=(=-. ==!(G$((S B" B$!3?A$E ############################################################### 9.TA8:(D333=(=+ ==!(G$((S B" AD($AG(################################################################# 1-TA8:(D333=(=+- ==!(G$((S B" H(S3TAT3B? ############################################################# 1-TA8:(D333=(=++ ==!(G$((S B" 3?"($3B$################################################################## 1+TA8:(D333=(=+0 ==!(G$((S B" &BB$######################################################################## 1+TA8:(D333=(=+> ==!(S7$3&T3D( &H$AS(S################################################################# 10

A&&(?!3H################################################################################################################### 10


7/72

SA*&:( S(TS B" $(S&B?S( A:T($?AT3D(S ################################################################## 10Table D333="=- ==&hrases at :east Bne Standard !eviation Apart with &arallelCording###################################################################################################################### 1> Table D333="=+ ==3ntervals 8etween &hrases Are as ?early (5ual as&ossible####### 1/ Table D333="=0 ==Scale Dalue 3ntervals 8etween &hrases Are as

?early (5ual as&ossible ###################################################################################################################### 19 Table D333="=> ==Sets Using Brder=of=*erit :ists of!escriptors ########################## 11

838:3BG$A&HE############################################################################################################# 1.

3ntroduction to Surveying

Cebster defines a survey as Jthe action of ascertaining factsregarding conditions or the condition of something to provide eactinformation especially to persons responsible or interestedK and as Jasystematic collection and analysis of data on some aspect of an area orgroup#K A survey, then, is much more than the mere compiling of data# Thedata must be analyzed, interpreted, and evaluated# Bnly after thisprocessing can data become information# The LeactnessL of the

information is determined by the surveyors methods# Unless he makes asystematic collection of data, followed by a careful analysis and evaluationwith predefined ob4ectives, his collection of data cannot become JeactKinformation#

TE&(S B" SU$D(ES

Surveys can be divided into two general categories on the basis oftheir etensiveness# A complete survey is called a Jcensus#K 3t involvescontacting the entire group you are interested in == the total population orJuniverse#K The other category is more commonM it is a sample survey# Asample is a representative part of a whole group %universe'# Thus a samplesurvey involves eamining only a portion of the total group in which you areinterested, and from it, inferring information about the group as a whole#

A!DA?TAG(S F !3SA!DA?TAG(S B" TH( TCBTE&(S B" SU$D(ES

Bne of the decisions to be made in surveying is whether or not to


8/72

sample# &arten %-.2, p -.' presents a list of advantages anddisadvantages of the sample survey# %These, in turn, imply the advantagesand disadvantages of the census#' The three most important considerationsfor the surveyor are) speed, low cost, and increased accuracy and analysisof the data#

8y sampling only a small portion of a large population, it is possible tocollect data in far less time than would be re5uired to survey the entire group# ?otonly is data collection 5uicker, but data processing and analysis also re5uire lesstime because fewer pieces of data need to be handled# $apidly changingconditions and the short turn=around time imposed in many surveys make theefficient use of time a critical variable# 3f an accurate snapshot of the attitudes ofa particular group is desired, currency is of paramount importance# &rofessionalpolitical pollsters make their living by providing 5uick snapshots of the Jpoliticalclimate#K $esults of such polls lose their accuracy very 5uickly %sometimes in aslittle as +> hours==particularly in the days preceding a ma4or election'# So, forthese pollsters, time is truly of the essence# 3ts probably a safe bet that those of

you reading this guide will not need that degree of speed# ?evertheless, speed isessential to ensure the data are Lfresh,L especially when it comes to assessingpublic opinion in a volatile or contentious area before they change appreciably#

The smaller amount of data gathered by sampling as opposed to surveyingan entire population can mean large cost savings# 8y limiting the group to besurveyed, less time, hence less cost, are involved in collecting, formatting, andanalyzing the data# 3n addition, if automated data processing %A!&' e5uipment isbeing used to analyze data, your overall time investment will be even less, as willbe the overall cost# Sampling allows you to do a credible 4ob for a smallerinvestment of time and money#

&arten %-.2' also notes that sampling enables the surveyor Jto give moreattention to each return received and to make certain that the data are asaccurate as possibleK %p --'# This attention may lead to more preciseinformation than would a less careful collection of data from the entire population#?othing more than a rudimentary 5uality control is possible for the great volumeof raw data gathered in a census# The more data collected, the greater thepotential for making JaccountingK errors#

The disadvantages of sampling are few, but important# The maindisadvantages stem from risk, lack of representativeness, and insufficient sample

size, each of which can cause errors# 3nattention to any of these potential flawswill invalidate survey results#

3t is important to realize that using a sample from a population to infersomething about the entire population involves a risk# The risk results fromdealing with partial information# 3f risk is not acceptable in seeking thesolution to a problem or the answer to a 5uestion, then a complete surveyor census, rather than a sample survey, must be conducted#

!etermining the representativeness of the sample is the surveyors


9/72

greatest problem when sampling# 8y definition, LsampleL means arepresentative part of an entire group# To avoid the charge of using Jbiaseddata,K it is necessary to obtain a sample that meets the re5uirement ofrepresentativeness, and this is not an easy task# Cithout a representativesample, a survey will, at best, produce results that are misleading andpotentially dangerous# &rocedures for minimizing the possibility of using a

nonrepresentative sample are covered in 7hapter >#

The final ma4or problem in sampling is to determine the size of thesample# The size of the sample you need for a valid survey depends onmany variables including the risk you are willing to accept and thecharacteristics of the population itself# The determination of sample size isdiscussed in 7hapter ># Here, it is sufficient to say that if sampling becomestoo complicated, or the re5uired sample size becomes too large, the easiestsolution may be to survey the entire population#

The decision as to whether to survey the entire population or only a

sample of it is not based on the above advantages and disadvantagesalone# 3t is affected by many other variables that are covered later in thisguide#

TB SU$D(E B$ ?BT TB SU$D(E

8efore attempting a survey, you should investigate some basic factsand answer some pertinent 5uestions# The result of this investigation willbe a greater realization of the work involved in producing a survey# &erhapsit will lead to a decision not to survey#

Surveys demand time%maybe more time than you have available# Theeact amount of time varies greatly from survey to survey depending on thenumber of people to be surveyed and the content of the survey# A survey ofa few 5uestions administered to the people in your office may take only aday or so, whereas a larger survey administered to a great number ofpeople located worldwide can take over three months from the time thesurvey is delivered to the printer %see Appendi 8'# And this does notinclude the time needed to design the survey and construct the5uestionnaire# *oreover, coordination with officials and the customers ofthe survey takes additional time# 3f your estimate of the time needed to

produce the survey eceeds your deadline date, you are likely to decide youdo not have the time to conduct a survey# A hurried survey wastes bothyour time and that of your respondents# The results of a hurried effort are5uestionable at best#

Surveys are epensive to produce# The solution to the problem or theanswer to the 5uestion may not be worth the cost to produce it# (ven if itwould be worth the price, you may not be able to obtain the needed funds,either from your own pocket or from your organizational budget# Although


10/72

no standard estimates of survey cost are available, some of the items ofepense can be eamined# The primary epense is in time and effortM thetime you spend producing the survey could be spent on other tasks# 3f otherpersonnel are needed, they will have to be paid# Access to typewriters,word processors, and calculating machines %or computer resources' is amust# 3f you epect to gather a great deal of data, the cost of renting A!&

time and of purchasing A!& scanner sheets should be eamined# Surveysof more than -2=+ respondents cannot feasibly be tabulated by hand#The same is true for groups of less than -2 respondents if the survey5uestionnaire is lengthy# The final cost involves supplies# At a minimum youwill need paper and envelopes# Eou may also have to pay either the cost ofprinting the survey 5uestionnaire or the postage or both# (ach of the abovecosts that applies to your survey should be estimated and the total costmeasured against the survey re5uirement#

Since surveys are being used more and more, the information youwant may have already been gathered# A search of some of the survey data

sources listed in Appendi 7 might yield a solution to your problem or atleast provide eamples of how others have approached similar problems#So before you undertake a survey, first make sure the answer to yourproblem does not already eist# ?et, evaluate the time you will need anddetermine the cost involved to produce the survey results, and then weighthese findings against the importance of the survey# Undertake a surveyonly if it is worth the time, effort, and cost to make it a good one#

GU3!( BUT:3?(

The remaining chapters of this guide cover the various steps in surveying#7hapter + outlines the official policies and procedures within the Air "orce forconducting surveys# 7hapter 0 covers the determination of the purpose,hypotheses, and survey plan# 7hapter > deals with the design of a sample surveyand the techni5ue for determining the re5uired sample size# The conceptspresented in 7hapter > will not apply to you, of course, if you are conducting a fullcensus rather than a sample survey# 7hapter 2 outlines the construction of thesurvey 5uestionnaire# 7hapter / discusses some of the more common statisticalerrors committed by novice researchers and ways to avoid them# The variousappendices contain checklists and data sources useful in surveying# Thebibliography lists informative references on surveying#

The Air "orce &ersonnelSurvey &rogram


11/72

The purpose of the Air "orce &ersonnel Survey &rogram is to foster thedevelopment of compatible and effective surveys, and to minimize eposure of Air"orce personnel to repeated or unwarranted survey solicitations %Air "orce3nstruction %A"3' 0/=+/-, -..0M p -'# This instruction also describes the surveypolicy responsibilities and eplains how the survey program is conducted# Thischapter will highlight some of the important points covered in A"3 0/=+/-, but Air

"orce personnel who plan to conduct official surveys within the Air "orce shouldbecome familiar with the entire instruction#

The instruction designates the *ilitary &ersonnel Survey 8ranch, Air "orce*ilitary &ersonnel 7enter %A"*&7

government contractors, general public, etc#' are a special concern# These mustbe approved through the Bffice of *anagement and 8udget %B*8', specificallythe Bffice of the Administrative Assistant for the Secretary of the Air "orce,3nformation *anagement &olicy !ivision %SA"


12/72

introductory paragraphs of A"3 0/=+/- specify that all surveys sub4ect to theprovisions of A"3 09=-0+ %Air "orce &rivacy Act &rogram' must contain a &rivacyAct Statement# This re5uires all respondents be advised of)

N the "ederal statute or eecutive order that authorizes the solicitation of theinformation

N the principal purpose%s' for which the data are to be usedN the routine uses to be made of itN whether furnishing the information is mandatory or voluntaryN the effects %if any' on the individual of not providing all or part of the

re5uested information#

"inally, A"3 0/=+/-, paragraph ., specifies conditions under which releaseof survey results must be coordinated with H; A"*&7


13/72

N the surveyor understands eactly what the problem isN the stated problem is the real problem

The survey should be designed to answer only the stated problem#Adding additional interesting ob4ectives will lengthen and complicate thesurvey while clouding the real issue#

TH( HE&BTH(S3S, B8(7T3D(, B$ $(S(A$7H;U(ST3B?

Bnce the problem has been clearly stated, the net step is to formone or more hypotheses# The hypothesis is actually your educated guessabout the answer to the problem# 3t should not be a capricious guess,however# 3t ought to be based on your prior eperience related to theproblem, or perhaps any knowledge you may have of previous researchdone on the topic# Cithout such a framework in which to make an educatedguess, you really have no basis for making a guess at all# 3f you do not havea clear basis for formulating an hypothesis, you should instead develop one

or more ob4ectives or 5uestions to frame the scope of your 5uestionnaire#

"or eample, if a problem is identified on the base as declining use ofthe Bfficers 7lub, an immediately obvious 5uestion comes to mind) JAre theofficers on this base satisfied with the Bfficers 7lub facilitiesOK This wouldbe suitable as a research 5uestion# 3t is possible, though doubtful, if youcould come up with a supportable hypothesis, or educated guess, as to theanswer to the problem# Eou may, for instance, have gathered someanecdotal evidence %overhearing colleagues talking' of dissatisfaction withthe clubs facilities# 8ut, this may not be sufficient for making an educatedguess that this is the real reason for the decline in club use# The problem

could be seasonalM it could be related to a decline in the officer populationon the baseM or a number of other possibilities# The point is that withoutsome credible evidence to support an hypothesis, you should probably notformulate one#

3f you formulated an hypothesis for the current eample on the basisof the anecdotal evidence available to you, you would naturally construct a5uestionnaire to survey the opinions of officers regarding their use, or lackthereof, of the Bfficers 7lub and the reasons for it# Eou might never think togather data from the base military personnel office to see if the officerpopulation is lower now than usual or if there are seasonal %cyclic' trends in

the size of the officer population on the base# 3n other words, establishingthe hypothesis may blind you to collecting data on other possible causes ofthe problem# This is why all researchers are cautioned not to formulatehypotheses unless they have a solid base in theory or previously gatheredevidence that suggests the hypothesis is, in fact, probable#

Hypotheses must be carefully written# They should not contain moral4udgments or biased statements such as JAll pilots are good leaders#K Thereare many ideas on what constitutes a good leader and your idea may not bethe same as those of the people you will contact# Avoid words like should,


14/72

best, good, bad, and ought#

Hypotheses should be as specific as possible# Avoid words such asmost and some# 3f by most you mean a ma4ority, then say ma4ority# Asurvey can more easily be designed to test whether Jmore than 92 percentapproveK than whether Jmost approve#K

A well=formulated hypothesis, ob4ective, or research 5uestiontranslates the purpose into a statement that can be investigatedscientifically# The level of difficulty you will face in producing a valid surveywill increase dramatically if they are not well formulated# Take care in doingthis step, and it will save you much effort later in the survey developmentprocess#

TH( SU$D(E &:A?

The net step after determining the purpose and hypotheses isconstructing the survey plan# The purpose of the survey plan is to ensurethat the survey results will provide sufficient data to provide an answer%solution' to the problem you are investigating# The survey plan iscomprised of three different parts)

N data collection planN data reduction and reformatting planN analysis plan

?one of these plans stands on its own# !ecisions you make on how

you will analyze your data will affect your data collection plan# The type ofdata reduction you do will affect not only the types of analyses you can do,but also the amount and types of data you need to collect# 8ecause theseplans are closely interrelated, they should be developed concurrently#

TH( !ATA 7B::(7T3B? &:A?

The purpose of the data collection plan is to ensure that proper dataare collected in the right amounts# Eour hypothesis and your data analysisplan determine the appropriateness of the data# "or eample, if you plan to

analyze your results by age group to test a hypothesis, then you mustcollect data from each age group whose opinions you want to know# Theright amount applies to sample data# As pointed out earlier, the use ofsample data involves risk, and the amount of that risk is determined by thesize of your sample# The amount of risk you are willing or able to acceptshould be stated in your analysis plan# &roper and right come together whenyour analysis plan involves both sampling and analyzing data by groups# Eounot only have to collect data from some members of each group you plan toanalyze, but you also have to see that each group provides a response rate


15/72

that is high enough to ensure your meeting your minimum risk level# Theconcept of the proper sample size is covered in greater detail in 7hapter >#

TH( !ATA $(!U7T3B? A?! $("B$*ATT3?G&:A?

The purpose of the data reduction and reformatting plan is to identifyup front and to decrease as much as possible the amount of data handling%reduction and reformatting' you will have to do# This plan is highlydependent on the other two plans# As previously mentioned, if yourcollection plan calls for a great deal of data, you should plan to use acomputer to analyze the data# 3f A!& scanner sheets are to be used torecord respondents answers, include the sheets with the 5uestionnaire sothe respondent can fill out the scanner sheet# This will save a great deal oftime that you would have to spend if you transferred the survey data to thescanner sheets yourself# 3t also eliminates the possibility of your making

errors in transferring data# Eou should coordinate in advance with the A!&personnel to make sure they will be able to scan your answer sheets and, ifnecessary, analyze your data within your timeframe# A!& shops are busyplaces# The prudent surveyor will JbookK the scanning and analysis 4obs wellin advance with A!& personnel to ensure their resources are available whenneeded#

A strong potential for error and tedious corrective work lies in datareduction and reformatting# &roper care in developing this plan can save agreat deal of time later and preclude error#

B&(?= A?! 7:BS(!=(?!(! ;U(ST3B?SThe use of Automatic !ata &rocessing %A!&' necessitates the use of

closed=end 5uestions == a type of 5uestion you should consider even if youare hand=tabulating your data# A closed=end 5uestion lists possible answersfrom which the respondent picks the one he


16/72

list#

Then, later, all the computer will have to do is count the number ofanswers in each category# 8y having the survey respondent, not you,categorize the answer, you will collect data that is more valid, reliably, andaccurate than if you did the categorizing yourself# Additional information on

closed=end 5uestions is provided in 7hapter 2#

TH( A?A:ES3S &:A?

"inally, an analysis plan ensures that the information produced by theanalysis will ade5uately address the originally stated hypotheses,ob4ectives, or 5uestions# 3t also ensures an analysis that is compatible withthe data collected during the survey# 3n the analysis plan, you determinewhich statistics you will use and how much risk you can take in stating yourconclusions# (ach of these decisions will affect the amount and type of data

you collect and how you will reduce it# ?ovice researchers often misusestatistical analyses out of ignorance of the assumptions on which thestatistics are based# The most often committed error in statistical analysisby novices is using a statistical techni5ue with inappropriate data# Theresults of such analyses appear to be legitimate, but are actually impossibleto interpret correctly# Ce will discuss some of these common errors and howto avoid them in 7hapter /#

7B?7:U!3?G THBUGHTS

Bppenheim %-.//' suggests that to make sure all these parts of thesurvey plan are correctly interlocked, you can simply approach the natural

se5uence of survey operations in reverse order# "irst determine whatconclusions you are interested inM then decide what statistics and results willbe needed to draw these conclusions# "rom this, the type of 5uestionsneeded and the nature of the sample can be determined#

A conscientious survey plan will help you produce a well=designedsurvey# The proper data will be processed correctly and efficiently toproduce the information re5uired to shed light on, and hopefully provide asolution to, the original problem#

Sampling Techni5ues and$elated Statistical7oncepts


17/72

7hapter - identified some of the problems associated with sampling)

N acceptance of a risk of errorN choice of a representative sampleN determination of the size of the sample

This chapter outlines procedures for dealing with these challenges#"irst, different techni5ues designed to produce a representative samplefrom different types of populations are eplained# ?et, the relationshipbetween risk and sample size is investigated# "inally, techni5ues arediscussed for 5uantifying the amount of risk present in your results and fordetermining the sample size necessary to achieve the confidence andreliability specified in your analysis plan#

SA*&:3?G *(THB!S

Eour overarching goal in doing a survey is to determine what somegroup thinks or feels about some issue# 3f money, time, or other resourceswere not a concern, the most accurate data you could get would come fromsurveying the entire population of interest# Since limited resources are areality we all have to deal with, however, we are often forced to survey theviews of only a few members of the population# 8ut never lose sight of thefact that the real purpose is to discover the views of the entire population#Bbviously, then, we want to be able to say with as much confidence aspossible that the views of the group we surveyed represents the views ofthe entire population# Using a combination of powerful statistical tools,known as inferential statistics, and unbiased sampling techni5ues, any

surveyor can collect data that actually represent the views of the entirepopulation from which the sample was taken# Two things are absolutelynecessary, however, to ensure a high level of confidence that the samplerepresents the population)

N an unbiased sampleN a sufficiently large sample

8ias as a statistical term means error# To say that you want an unbiasedsample may sound like youre trying to get a sample that is error=free# Asappealing as this notion may be, it is impossible to achieveP (rror always occurs

== even when using the most unbiased sampling techni5ues# Bne source of error iscaused by the act of sampling itself# To understand it, consider the followingeample#

:ets say you have a bowl containing ten slips of paper# Bn each slip isprinted a number, one through ten# This is your Jpopulation#K ?ow you are goingto select a sample# Ce will use a random method for drawing the sample, whichcan be done easily by closing your eyes and reaching into the bowl and choosingone slip of paper# After choosing it, check the number on it and place it in the


18/72

sample pile#

?ow to determine if the sample is representative of the population, we mustknow what attribute%s' we wish to make representative# Since there are aninfinite number of human attributes, we must precisely determine the one%s' weare interested in before choosing the sample#

3n our eample, the attribute of interest will be the average numerical valueon the slips of paper# Since the JpopulationK contained ten slips numberedconsecutively from one to ten, the average numerical value in the population is)

-+ 0>2/91.-

QQQQQQQQQR22#

-

As you can see, no matter what slip of paper we draw as our first sampleselection, its value will be either lower or higher than the population average#:ets say the slip we choose first has a . on it# The difference between our sample%.' and the population %2#2' averages is Q0#2 %plus signifies the sample average islarger than the population average'# The difference between the sample averageand the population average is known as sampling error# That is, the sample mean%average' plus %or minus' the total amount of sampling error e5uals thepopulation mean#

Bn our second pick, we choose a slip that has a - on it# ?ow the average of

sample values is)

.-

Q

R 2#

+

The sampling error has shrunk from its previous value of Q 0#2 to its newvalue of = #2 %minus signifies the sample mean is now smaller than the population

mean'# (ach time we choose a slip from the population to include in the sample,one of three mutually eclusive things can occur == the sample mean will become)

N larger than the population meanN smaller than the population meanN e5ual to the population mean

Bn average, each sampling brings the sample mean a bit closer to thepopulation mean# Ultimately, if we sampled everyone from the population, the


19/72

sample mean and the population mean would be e5ual# This is why a completecensus is completely accurate = there is no sampling error# Eet, if we are forced touse only a sample from the population, the larger the sample the less samplingerror we will have, generally speaking#

(5ually important to the size of the sample is the determination of the type

of sampling to be done# 3n our eample, we randomly %blindly' chose from thepopulation# $andom sampling always produces the smallest possible samplingerror# 3n a very real sense, the size of the sampling error in a random sample isaffected only by random chance# The two most useful random samplingtechni5ues are simple random and stratified random sampling methods# Thesewill discussed shortly#

8ecause a random sample contains the least amount of sampling error, wemay say that it is an unbiased sample# ?ote that we are not saying the samplecontains no error, but rather the minimum possible amount of error#

?onrandom sampling techni5ues also eist, and are used more fre5uentlythan you might imagine# As you can probably guess from our previous discussion,nonrandom sampling techni5ues will always produce larger sampling errors %forthe same sample size' than random techni5ues# The reason for this is thatnonrandom techni5ues generate the epected random sampling error on eachselection plus additional error related to the nonrandom nature of the selectionprocess# To eplain this, lets etend our sampling eample from above#

:ets say we want to sample from a JpopulationK of - consecutivelynumbered slips of paper# 8ecause numbering these slips is time consuming, wehave - people each number - slips and place all - of them into our bowl

when they finish# :ets also say that the last person to finish has slips numberedfrom .- to -, and these are laid on top of all the other slips in the bowl# ?owwe are ready to select them#

3f we wanted to make this a truly random sampling process, we would haveto mi the slips in the bowl thoroughly before selecting# "urthermore, we wouldwant to reach into the bowl to different depths on subse5uent picks to make sureevery slip had a fair chance of being picked#

8ut, let us say in this eample that we forget to mi the slips in the bowl#:ets also say we only pick from the top layer of slips# 3t should be obvious what

will occur# 8ecause the top layer of slips is numbered .- through -, the meanof any sample %of - or less' we select will hover around .2#2 %the true mean ofthe numbers .- through -'# 7learly, this is not even close to the truepopulation mean %2#2 == the mean of the numbers from - to -'# Samplingerror amounts to the difference between the true population mean and thesample mean# 3n this eample, the sampling error can as large as >2 %.2#2 =2#2'#

This was a simple, and somewhat absurd, eample of nonrandom sampling#


20/72

8ut, it makes the point# ?onrandom sampling methods usually do not producesamples that are representative of the general population from which they aredrawn# The greatest error occurs when the surveyor attempts to generalize theresults of the survey obtained from the sample to the entire population# Such anerror is insidious because it is not at all obvious from merely looking at the data,or even from looking at the sample# The easiest way to recognize whether a

sample is representative or not is to determine if the sample was selectedrandomly# To be a random sampling method, two conditions must be met# 3f bothare met, the resulting sample is random# 3f not, it is a nonrandom samplingtechni5ue)

N every member in the population must have an e5ual opportunity of beingselected,

N the selection of any member of the population must have no influence onthe selection of any other member

All nonrandom sampling methods violate one or both of these criteria#The most commonly used nonrandom methods are)

N systematic sampling %selecting every nth person from a group'N cluster sampling %selecting groups of members rather than single members'N convenience or incidental sampling %selecting only readily available

members'N 4udgment or purposive sampling %selecting members who are 4udged to be

appropriate for the study'

S3*&:( $A?!B* SA*&:3?G

A simple random sample is one in which each member %person' in the

total population has an e5ual chance of being picked for the sample# 3naddition, the selection of one member should in no way influence theselection of another# Simple random sampling should be used with ahomogeneous population, that is, one composed of members who allpossess the same attribute you are interested in measuring# 3n identifyingthe population to be surveyed, homogeneity can be determined by askingthe 5uestion, JChat is %are' the common characteristic%s' that are ofinterestOK These may include such characteristics as age, se, rank


21/72

*any statistical tets or mathematical tables treat random numbergeneration# A less rigorous procedure for determining randomness is towrite the name of each member of the population on a separate card, andwith continuous miing, draw out cards until the sample size is reached#

The simple random sample re5uires less knowledge about the

population than other techni5ues, but it does have two ma4or drawbacks#Bne is if the population is large, a great deal of time must be spent listingand numbering the members# The other is the fact that a simple randomsample will not ade5uately represent many population attributes%characteristics' unless the sample is relatively large# That is, if you areinterested if choosing a sample to be representative of a population on thebasis of the distribution in the population of gender, age, and economicstatus, a simple random sample will need to be very large to ensure allthese distributions are e5uivalent to %or representative of' the population# Toobtain a representative sample across multiple population attributes, youshould use the techni5ue of stratified random sampling#

Ce made this point earlier in this chapter, but its such an importantconcept that it bears repeating# To determine if the sampling method youuse is random or not, remember that true random sampling methods mustmeet two criteria)

N (very member in the population must have an e5ual opportunity of beingchosen for the sample %e5uality'

N The selection of one member is not affected by the selection of previousmembers %independence'

8oth simple random and stratified random sampling methods meetthese two criteria# ?onrandom sampling methods lack one or both of thesecriteria# Ce discuss stratified random sampling net#

ST$AT3"3(! $A?!B* SA*&:3?G

This method is used when the population is heterogeneous ratherthan homogeneous %or as discussed above, when you want to obtain arepresentative sample across many population attributes'# A heterogeneouspopulation is composed of unlike elementsM such as, officers of differentranks, civilians and military personnel, or the patrons of a discount store

%differing by gender or age'#

A stratified random sample is defined as a combination ofindependent samples selected in proper proportions from homogeneousgroups within a heterogeneous population# The procedure calls forcategorizing the heterogeneous population into groups that arehomogeneous in themselves# 3f one group is proportionally larger thananother, its sample size should also be proportionally larger# The number ofgroups to be considered is determined by the characteristics of the


22/72

population# *any times the survey plan will determine some or all of thegroups# "or eample, if you are comparing enlisted and officer segments onyour base, each of these will be a separate group#

After dividing the population into groups, you then sample eachhomogeneous group# !ifferent sampling techni5ues can be used in each of the

different groups, but keep in mind that random techni5ues produce the minimumamount of sampling error# "inally, you should calculate the sample statistics foreach group to determine how many members you need from each subgroup#

Ce will discuss the calculations involved in determining the size of yoursample later in this chapter# These calculations are designed to determine thesize of a simple random sample# Since the stratified sampling techni5ue re5uiresyou to create simple, homogeneous subgroups from a large heterogeneous group,think of the calculations for a stratified sample as a series of simple randomsample size calculations for each homogeneous subgroup# The only otherinformation you must know is the proportion of the population possessing the

attribute contained in each homogeneous subgroup#

"or eample, lets say we want to draw a random sample from a populationof military personnel to assess their opinions on some issue# 3n addition, wewould like to determine if the opinions differ by officer=enlisted affiliation andgender of the individuals surveyed# Ce recognize that the population we want todraw our sample from is heterogeneous with respect to the two attributes ofinterest to us# So, we have to create homogeneous subgroups %four to be eact')

N (nlisted, maleN (nlisted, female

N Bfficer, maleN Bfficer, female

?ow, each group is homogeneous on both attributes# To ensure eachsubgroup in the sample will represent its counterpart subgroup in the population,we must ensure each subgroup is represented in the sample in the sameproportion to the other subgroups as they are in the population# :ets assume thatwe know %or can estimate' the population of Air "orce military personnel to bedistributed as follows) 9 percent male, 0 percent female and /2 percentenlisted, 02 percent officer# Cith that, we can determine the approimateproportions of our four homogeneous subgroups in the population)

N (nlisted, male #/2 #9 R #>22N (nlisted, female #/2 #0 R #-.2N Bfficer, male #02 #9 R #+>2N Bfficer, female #02 #0 R #-2

Thus, a representative sample of the Air "orce population %by race andenlisted=officer affiliation' would be composed of >2#2 percent enlistedmales, -.#2 percent enlisted females, +>#2 percent officer males, and -#2


23/72

percent officer females# (ach percentage should be multiplied by the totalsample size needed to arrive at that actual number of personnel re5uiredfrom each subgroup or stratum#

As this eample illustrates, stratified random sampling re5uires adetailed knowledge of the distribution of attributes or characteristics of

interest in the population to determine the homogeneous groups that liewithin it# A stratified random sample is superior to a simple random samplesince the population is divided into smaller homogeneous groups beforesampling, and this yields less variation within the sample# This makespossible the desired degree of accuracy with a smaller sample size# 8ut, ifyou cannot accurately identify the homogeneous groups, you are better offusing the simple random sample since improper stratification can lead toserious error#

SEST(*AT37 SA*&:3?G

Sometimes it is more epeditious to collect a sample of surveyparticipants systematically# This is fre5uently done, for instance, in eitpolling of voters or store customers# 3t is a nonrandom sampling techni5ue,but is used primarily for its ease and speed of identifying participants#

To use the systematic approach, simply choose every 6thmember inthe population where 6 is e5ual to the population size divided by there5uired sample size# 3f this 5uotient has a remainder, ignore it %rounddown'# "or eample, if you need - members in your sample and thepopulation consists of - people, you need to sample every -


24/72

nonrandom, the resulting sample will not necessarily be representative ofthe population from which it was drawn# This will affect your ability toconfidently generalize results of the survey since you may not be sure towhich segment of the population the results will apply# As a word of advice,unless you have eperience in systematic sampling techni5ues, and havefull knowledge of the population to be sampled, you should avoid using this

method#

U!G*(?T SA*&:3?G

udgment sampling involves asking an epert on an issue beinginvestigated to define the members that should comprise the sample# Therepresentativeness of the sample is determined solely by the 4udgment ofthe researcher# Since each member in the population does not have ane5ual chance of being chosen, a 4udgment sample is also a nonrandomsampling method# Since the sample does not meet the criterion of

randomness = the basis for many statistical sampling applications % a4udgment sample should never be used in a statistical evaluation effort#

&U$&BS3D( SA*&:3?GAs the name implies, purposive sampling involves selecting members

from the population to comprise a sample because they specifically meetsome prescribed purpose of possess specific attributes of interest thataddress the purpose of a particular research problem under investigation#&urposive sampling is used primarily in causal=comparative %e post facto'research where the researcher is interested in finding a possible cause=and=

effect link between two variables, one of which has already occurred# Theresearcher intentionally selects the samples in such a way that onepossesses the causal %independent' variable and one does not# The purposeof the research governs the selection of the sample and, thus, ecludesmembers of the population who do not contribute to that purpose#

$(&$(S(?TAT3D( SA*&:3?G

The types of sampling methods discussed above are only a few of themany available# Eou will find others in the references listed in the

bibliography# (ach type is designed to obtain the most representativesample possible from different kinds of populations# 8efore using anysampling method yourself, first think about the population to which youwant to generalize the results of your survey %which population do you wantto represent'# 3f generalizing results is not your aim, any sampling methodwill do# 3f generalizing results is important, use a sampling method that willensure your sample is representative of the population from which you drawit# $andom sampling methods typically ensure a high degree of confidencethat the results do, in fact, represent those of the whole population#


25/72

"A7TB$S 3?":U(?73?G SA*&:( S3I(

As pointed out in 7hapter -, when you sample you are dealing withonly partial information# And you must accept a risk of being wrong wheninferring something about the population on the basis of sampleinformation# 3n the analysis portion of your survey plan, you identify the

amount of risk you are willing %or allowed' to take# This amount of riskrelates directly to the size of your sample# Simply stated, the less risk youare willing to take, the larger your sample must be# 3f you cannot acceptany risk, you should survey the entire population %take a census' and youneed not study this chapter any further#

Chen determining your risk level, keep in mind the time and costinvolved in obtaining the sample size sufficient to achieve the risk level youcan accept# Eou may find it impossible to produce a sample large enough tomeet that risk level#

Another factor bearing on sample size is also obtained from your

analysis plan# 3t is the number of groups you are planning to eamine withinthe population# "or eample, if you are planning to compare two groups%enlisted and officer' on a base %your population', each of the groups mustbe sampled and each of the samples must be large enough to ensuresatisfying your risk level#

7B?"3!(?7( :(D(: A?! &$(73S3B?

$isk, as it relates to sample size determination, is specified bytwo interrelated factors)

N the confidence levelN the precision %or reliability' range#

To minimize risk, you should have a high confidence %say .2 percent'that the true value you seek %the actual value in the population' liessomewhere within a small interval %say Q or = 2 percent' around yoursample value %your precision'# Sawyer %-.9-M p >.' uses a baseball gameanalogy to eplain confidence level, precision range, and their relationship#A baseball pitcher may feel that he can get very few of his pitches %perhaps- percent' over the eact center %small precision range' of home plate#

8ut since home plate is -9 inches wide, he may feel that he can get .2percent of his pitches over the center of the plate with a precision of plus orminus 1 -


26/72

!(T($*3?3?G TH( S3I( B" TH( SA*&:(

Bnce you determine your desired degree of precision and yourconfidence level, there are several formulas you can use to determinesample size depending on how you plan to report the results of your study#Cell discuss three of them here# 3f you will be reporting results as

percentages %proportions' of the sample responding, use the followingformula)

3f you will report results as means %averages' of the sample responding, use thefollowing formula)

3f you plan to report results in a variety of ways, or if you have difficultyestimating percentage or standard deviation of the attribute of interest, thefollowing formula may be more suitable for use)

Ce illustrate this formula with the following eample# 3f the total population%?' is -,, and you wish a .2 confidence level and 2 percent precisionlevel %d R #2, I R -#./ from Appendi (', then)

So, a representative sample of 09 %0/.#.1 rounded up' would be sufficientto satisfy your risk level# 3nspection of the formula shows that the re5uiredsample size will increase most rapidly if)

N the confidence level %I factor' is increased, orN the precision level %d' is made smaller#

3f you have stratified your population into more than one group, the size of


27/72

each group will be its proportion %percentage' in the population times the totalsample size as computed above# To illustrate, recall our earlier eample of fourstratified groups# Using the n of 09 calculated above, each of these stratashould have the following sample sizes)

N (nlisted, male 09 #>22 R -/1#02 R -/1

N (nlisted, female 09 #-.2 R 9+#-2 R 9+N Bfficer, male 09 #+>2 R .#/2 R .-N Bfficer, female 09 #-2 R 01#12 R 0.

"inally, you should ad4ust the computed samplesize %n' by dividing n by the epected response rate#"or instance, if you epect 92 n

percent response rate, you should make your sample size e5ual # 3f

92

# you cant anticipate aresponse rate, assume a 2 percent response rate %i#e#, double the n value'# Thissort of ad4ustment should ensure you get a sufficient number of responsesregardless of return rate#

The ;uestionnaire

The final step in preparing the survey is developing the data collectioninstrument# The most common means of collecting data are the interviewand the self= or group=administered 5uestionnaire# 3n the past, the interviewhas been the most popular data=collecting instrument# $ecently, the5uestionnaire has surpassed the interview in popularity, especially in themilitary# !ue to this popularity, this chapter concentrates on thedevelopment of the 5uestionnaire#

TH( ;U(ST3B??A3$( &$BS A?! 7B?S

3t is important to understand the advantages and disadvantages ofthe 5uestionnaire as opposed to the personal interview# This knowledge willallow you to maimize the strengths of the 5uestionnaire while minimizingits weaknesses# The advantages of administering a 5uestionnaire instead ofconducting an interview are)

N lower costsN better samplesN standardization


28/72

N respondent privacy %anonymity'

The primary advantage is lower cost, in time as well as money# ?othaving to train interviewers eliminates a lengthy and epensive re5uirementof interviewing# The 5uestionnaire can be administered simultaneously tolarge groups whereas an interview re5uires each individual to be 5uestioned

separately# This allows the 5uestions to reach a given number ofrespondents more efficiently than is possible with the interview# "inally, thecost of postage should be less than that of travel or telephone epenses#$ecent developments in the science of surveying have led to incorporating

computers into the interview process, yielding what is commonly known ascomputer automated telephone interview %or 7AT3' surveys# Advances in usingthis survey techni5ue have dramatically reshaped our traditional views on thetime=intensive nature and inherent unreliability of the interview techni5ue# Eet,despite resurgence in the viability of survey interviews, instruction in thedevelopment and use of the 7AT3 techni5ue is well beyond the scope of thishandbook#

*any surveys are constrained by a limited budget# Since a typical5uestionnaire usually has a lower cost per respondent, it can reach more peoplewithin a given budget %or time' limit# This can enhance the conduct of a largerand more representative sample#

The 5uestionnaire provides a standardized data=gathering procedure# Usinga well=constructed 5uestionnaire can minimize the effects of potential humanerrors %for eample, altering the pattern of 5uestion asking, calling atinconvenient times, and biasing by JeplainingK# The use of a 5uestionnaire alsoeliminates any bias introduced by the feelings of the respondents towards the

interviewer %or vice versa'#

Although the point is debatable, most surveyors believe the respondent willanswer a 5uestionnaire more frankly than he would answer an interviewer,because of a greater feeling of anonymity# The respondent has no one to impresswith his


29/72

they might comprise the ma4ority of the nonreturns# This would introduce non=random %or systematic' bias into your survey results, especially if you found only asmall number of the returns were in favor of the policy# ?onreturns cannot beovercome entirely# Chat we can do is try to minimize them# Techni5ues toaccomplish this are covered later in this chapter#

*isinterpretation occurs when the respondent does not understandeither the survey instructions or the survey 5uestions# 3f respondentsbecome confused, they will either give up on the survey %becoming anonreturn' or answer 5uestions in terms of the way they understand it, butnot necessarily the way you meant it# Some view the latter problem as amore dangerous occurrence than merely nonresponding# The 5uestionnaireinstructions and 5uestions must be able to stand on their own and must useterms that have commonly understood meanings throughout the populationunder study# 3f novel terms must be used, be sure to define them so allrespondents understand your meaning#

The third disadvantage of using a 5uestionnaire is inability to check onthe validity of the answer# !id the person you wanted to survey give the5uestionnaire to a friend or complete it personallyO !id the individualrespond indiscriminatelyO !id the respondent deliberately choose answersto mislead the surveyorO Cithout observing the respondents reactions %aswould be the case with an interview' while completing the 5uestionnaire,you have no way of knowing the true answers to these 5uestions#

The secret in preparing a survey 5uestionnaire is to take advantage ofthe strengths of 5uestionnaires %lower costs, more representative samples,standardization, and privacy' while minimizing the number of nonreturns,

misinterpretations, and validity problems# This is not always as easy as itsounds# 8ut an inventive surveyor can very often find legitimate ways ofovercoming the disadvantages# Ce provide some suggestions below tohelp#

TH( 7B?T(?TS

The key to minimizing the disadvantages of the survey 5uestionnairelies in the construction of the 5uestionnaire itself# A poorly developed5uestionnaire contains the seeds of its own destruction# (ach of the three

portions of the 5uestionnaire = the cover letter, the instructions, and the5uestions = must work together to have a positive impact on the success ofthe survey#The cover letter should eplain to the respondent the purpose of the survey

and motivate him to reply truthfully and 5uickly# 3f possible, it should eplain whythe survey is important to him, how he was chosen to participate, and who issponsoring the survey %the higher the level of sponsorship the better'# Also theconfidentiality of the results should be strongly stressed# A well written coverletter can help minimize both nonreturn and validity problems# An eample is


30/72

given in Appendi "# 3n support of the statement above regarding level ofsponsorship, the signature block on the letter should be as high level as you canget commensurate with the topic being investigated# "or instance, a survey aboutAir "orce medical issues or policy should be signed by the Air "orce SurgeonGeneral or higher, a survey on religious issues by the Air "orce 7hief of 7haplains,etc# Another tip that seems to help improve response rate is to identify the survey

as official# (ven though the letter is on government stationery and is signed byan military official, it may help to mark the survey itself with an B""373A: stamp ofsome sort# 3n general, the more official the survey appears, the less likely it is tobe disregarded#

The cover letter should be followed by a clear set of instructions eplaininghow to complete the survey and where to return it# 3f the respondents do notunderstand the mechanical procedures necessary to respond to the 5uestions,their answers will be meaningless# The instructions substitute for your presence,so you must anticipate any 5uestions or problems that may arise and attempt toprevent them from occurring# 3f you are using A!& scanner sheets, eplain how

you want the respondent to fill it in = what portions to use and what portions toleave blank# $emember anonymityP 3f you do not want respondents to providetheir names or SSA?s, say so eplicitly in the instructions, and tell them to leavethe ?A*( and SSA? portions of the scan sheets blank#

3f you need respondents SSA? and


31/72

5uestion, a special form of the multiple=choice 5uestion, is used to measurethe intensity of the respondents feelings on a sub4ect# These 5uestionsprovide answers that cover a range of feelings#

The intensity 5uestion is covered in greater detail later in this chapter#The final type of 5uestion is the free response or open=end 5uestion# This

type re5uires respondents to answer the 5uestion in their own words %see7hapter 0'# 3t can be used to gather opinions or to measure the intensity offeelings# *ultiple=choice 5uestions are the most fre5uently used types of5uestions in surveying today# 3t is prudent, therefore, to concentrateprimarily on factors relating to their application#

;U(ST3B??A3$( 7B?ST$U7T3B?

*any researchers have investigated the comple art of 5uestionwriting# "rom their eperiences, they offer valuable advice# 8elow are some

helpful hints typical of those that appear most often in tets on 5uestionconstruction#

- 6eep the language simple# Analyze your audience and write on their level#&arten %-.2M p +-' suggests that writing at the sith=grade level may beappropriate# Avoid the use of technical terms or 4argon# An appropriatecorollary to *urphys :aw in this case would be) 3f someone canmisunderstand something, they will#

+ 6eep the 5uestions short# :ong 5uestions tend to become ambiguous andconfusing# A respondent, in trying to comprehend a long 5uestion, mayleave out a clause and thus change the meaning of the 5uestion#

0 6eep the number of 5uestions to a minimum# There is no commonly agreedon maimum number of 5uestions that should be asked, but researchsuggests higher return rates correlate highly with shorter surveys# Ask only5uestions that will contribute to your survey# Apply the JSo whatOK andJCho caresOK tests to each 5uestion# J?ice=to=knowK 5uestions only add tothe size of the 5uestionnaire# Having said this, keep in mind that you shouldnot leave out 5uestions that would yield necessary data simply because itwill shorten your survey# 3f the information is necessary, ask the 5uestion#Cith the availability of desktop publishing %!T&' software, it is often possibleto give the perception of a smaller survey %using smaller point


32/72

worded 5uestion gives no clue as to which answer you may believe to bethe correct one#

/ Use sub4ective terms such as good, fair, and bad sparingly, if at all# Theseterms mean different things to different people# Bne persons JfairK may beanother persons Jbad#K How much is JoftenK and how little is JseldomOK- 9# Allow for all possible answers# $espondents who cannot find their

answer among your list will be forced to give an invalid reply or, possibly,become frustrated and refuse to complete the survey# Cording the5uestion to reduce the number of possible answers is the first step# Avoiddichotomous %two=answer' 5uestions %ecept for obvious demographic5uestions such as gender'# 3f you cannot avoid them, add a third option,such as no opinion, dont know, or other# These may not

+ get the answers you need but they will minimize the number of invalidresponses# A great number of Jdont knowK answers to a 5uestion in afact=finding survey can be a useful piece of information# 8ut a ma4orityof JotherK answers may mean you have a poor 5uestion, and perhapsshould be cautious when analyzing the results#

9 Avoid emotional or morally charged 5uestions# The respondent may feelyour survey is getting a bit too personalP- .# Understand the should=would 5uestion# Selltiz, et al# %-./0, p +2-'

note that respondents answer JshouldK 5uestions+ from a social or moral point of view while answering JwouldK 5uestions in

terms of personal preference#1 "ormulate your 5uestions and answers to obtain eact information and to

minimize confusion# "or eample, does JHow old are youOK mean on yourlast or your nearest birthdayO !oes JChat is your %military' gradeOK meanpermanent or temporary gradeO As of what dateO 8y including instructionslike JAnswer all 5uestions as of %a certain date'K, you can alleviate many

such conflicts# %$efer to hint -0 below#'. 3nclude a few 5uestions that can serve as checks on the accuracy and

consistency of the answers as a whole# Have some 5uestions that areworded differently, but are soliciting the same information, in different partsof the 5uestionnaire# These 5uestions should be designed to identify therespondents who are 4ust marking answers randomly or who are trying togame the survey %giving answers they think you want to hear'# 3f you find arespondent who answers these 5uestions differently, you have reason todoubt the validity of their entire set of responses# "or this reason, you maydecide to eclude their response sheet%s' from the analysis#

-+# Brganize the pattern of the 5uestions)

N &lace demographic 5uestions at the end of the 5uestionnaire#N Have your opening 5uestions arouse interest#N Ask easier 5uestions first#N To minimize conditioning, have general 5uestions precede specific ones#N Group similar 5uestions together#N 3f you must use personal or emotional 5uestions, place them at the end of

the 5uestionnaire#


33/72

- &retest %pilot test' the 5uestionnaire# This is the most important step inpreparing your 5uestionnaire# The purpose of the pretest is to see 4ust howwell your cover letter motivates your respondents and how clear your

instructions, 5uestions, and answers are# Eou should choose a small groupof people %from three to ten should be sufficient' you feel are representativeof the group you plan to survey# After eplaining the purpose of the pretest,let them read and answer the 5uestions without interruption# Chen theyare through, ask them to criti5ue the cover letter, instructions, and each ofthe 5uestions and answers# !ont be satisfied with learning only whatconfused or alienated them# ;uestion them to make sure that what theythought something meant was really what you intended it to mean# Use theabove -+ hints as a checklist, and go through them with your pilot testgroup to get their reactions on how well the 5uestionnaire satisfies thesepoints# "inally, redo any parts of the 5uestionnaire that are weak#

+ Have your 5uestionnaire neatly produced on 5uality paper# A professionallooking product will increase your return rate# As mentioned earlier,desktop publishing software can be used to add a very professional touch toyour 5uestionnaire and improve the likelihood of its being completed# 8utalways remember the adage JEou cant make a silk purse out of a sowsear#K A poorly designed survey that contains poorly written 5uestions willyield useless data regardless of how JprettyK it looks#

-2# "inally, make your survey interestingP3?T(?S3TE ;U(ST3B?S A?! TH( :36($TS7A:(

As mentioned previously, the intensity 5uestion is used to measurethe strength of a respondents feeling or attitude on a particular topic# Such5uestions allow you to obtain more 5uantitative information about thesurvey sub4ect# 3nstead of a finding that 1 percent of the respondents favora particular proposal or issue, you can obtain results that show 2 percent ofthem are strongly in favor whereas 92 percent are mildly in favor# Thesefindings are similar, but the second type of response supplies more usefulinformation#

The most common and easily used intensity %or scaled' 5uestion

involves the use of the :ikert=type answer scale# 3t allows the respondent tochoose one of several %usually five' degrees of feeling about a statementfrom strong approval to strong disapproval# The J5uestionsK are in the formof statements that seem either definitely favorable or definitely unfavorabletoward the matter under consideration# The answers are given scores %orweights' ranging from one to the number of available answers, with thehighest weight going to the answer showing the most favorable attitudetoward the sub4ect of the survey# The following 5uestions from the*innesota Survey of Bpinions designed to measure the amount of Janti=US


34/72

lawK feelings illustrate this procedure)

- Almost anything can be fied up in the courts if you have enough money#+ Bn the whole, 4udges are honest#

The weights %shown by the numbers below the answers' are not shown onthe actual 5uestionnaire and, therefore, are not seen by the respondents# Aperson who feels that US laws are un4ust would score lower than one who feelsthat they are 4ust# The stronger the feeling, the higher %or lower' the score# Thescoring is consistent with the attitude being measured# Chether JagreeK orJdisagreeK gets the higher weight actually makes no difference# 8ut for ease ininterpreting the results of the 5uestionnaire, the weighting scheme should remainconsistent throughout the survey#

Bne procedure for constructing :ikert=type 5uestions is as follows %adaptedfrom Selltiz, et al#, -./0M pp 0/9=0/1')

- The investigator collects a large number of definitive statements relevant tothe attitude being investigated#

+ 7onduct and score a pretest of your survey# The most favorable response tothe attitude gets the highest score for each 5uestion# The respondents totalscore is the sum of the scores on all 5uestions#

0 3f you are investigating more than one attitude on your survey, intermi the5uestions for each attitude# 3n this manner, the respondent will be less able

to guess what you are doing and thus more likely to answer honestly#> $andomly select some 5uestions and flip=flop the Strongly Agree =

=Strongly !isagree scale to prevent the respondent from getting into apattern of answering %often called a response set'#

The intensity 5uestion, with its scaled answers and average scores, cansupply 5uantitative information about your respondents attitudes toward thesub4ect of your survey# The interested reader is encouraged to learn and use otherscales, such as the Thurstone, Guttman, and Semantic !ifferential scales, bystudying some of the references in the bibliography#

A number of studies have been conducted over the years attempting todetermine the limits of a persons ability to discriminate between words typicallyfound on rating or intensity scales# The results of this research can be ofconsiderable value when trying to decide on the right set of phrases to use in yourrating or intensity scale# Chen selecting phrases for a >=, 2=, 9=, or .=point :ikertscale, you should choose phrases that are far enough apart from one another tobe easily discriminated, while, at the same time, keeping them close enough thatyou dont lose potential information# Eou should also try to gauge whether the


35/72

phrases you are using are commonly understood so that different respondents willinterpret the meaning of the phrases in the same way# An obvious eample isshown with the following 0 phrases) Strongly Agree, ?eutral, Strongly !isagree

These are easily discriminated, but the gap between each choice isvery large# How would a person respond on this three=point scale if they

only agreed with the 5uestion being askedO There is no middle groundbetween Strongly Agree and ?eutral# The same thing is true for someonewho wants to respond with a mere disagree# Eour scales must have enoughchoices to allow respondents to epress a reasonable range of attitudes onthe topic in 5uestion, but there must not be so many choices that mostrespondents will be unable to consistently discriminate between them#Appendi H provides several tables containing lists of phrases commonlyused in opinion surveys with associated Jscale valuesK and standarddeviations %or inter=5uartile range values'# Also provided is a shortintroduction describing how these lists can be used in selecting responsealternatives for your opinion surveys# The information in that appendi is

derived from research done for the U#S# Army $esearch 3nstitute for the8ehavioral and Social Services at "ort Hood, Teas#

83AS A?! HBC TB 7B*8AT 3T

:ike any scientist or eperimenter, surveyors must be aware of waystheir surveys might become biased and of the available means forcombating bias# The main sources of bias in a 5uestionnaire are)

N a nonrepresentative sample

N leading 5uestionsN 5uestion misinterpretationN untruthful answers

Surveyors can epose themselves to possible nonrepresentativesample bias in two ways# The first is to actually choose a nonrepresentativesample# This bias can be eliminated by careful choice of the sample asdiscussed earlier in 7hapter ># The second way is to have a large number ofnonreturns#

The nonreturn bias %also called non=respondent bias' can affect both

the sample survey and the complete survey# The bias stems from the factthat the returned 5uestionnaires are not necessarily evenly distributedthroughout the sample# The opinions or attitudes epressed by those whoreturned the survey may or may not represent the attitudes or opinions ofthose who did not return the survey# 3t is impossible to determine which istrue since the non=respondents remain an unknown 5uantity# Say, foreample, a survey shows that / percent of those returning 5uestionnairesfavor a certain policy# 3f the survey had a 9 percent response rate %a fairlyhigh rate as voluntary surveys go', then the favorable replies are actually


36/72

only >+ percent of those 5uestioned %/ percent of the 9 percent whoreplied', which is less than 2 percentP a minority response in terms of thewhole sample#

Since little can be done to estimate the feelings of the nonreturnees,especially in a confidential survey, the only solution is to minimize the number of

nonreturns# *iller %-.9M p 1-' and Selltiz et al# %-./0M p +>-' offer the followingtechni5ues to get people to reply to surveys# Some of these have already beenmentioned in earlier sections of this chapter#

- Use follow=up letters# These letters are sent to the non=respondents after aperiod of a couple of weeks asking them again to fill out and return the5uestionnaire# The content of this letter is similar to that of the cover letter#3f you are conducting a volunteer survey, you should anticipate the need forfollowing up with non=respondents and code the survey in some unobtrusiveway to tell who has and who has not yet responded# 3f you dont do that, but

still need to get in touch with non=respondents, consider placing ads in localpapers or base bulletins, announcements at commanders call, or noticesposted in public places# 3f at all possible, provide a fresh copy of the surveywith the follow= up letter# This often increases return rate over simplysending out a letter alone#

+ Use high=level sponsorship# This hint was mentioned in an earlier section#&eople tend to reply to surveys sponsored by organizations they know orrespect# 3f you are running a military survey, obtain the highest=rankingsponsorship you can# (ffort spent in doing this will result in a higherpercentage of returns# 3f possible, use the letterhead of the sponsor on yourcover letter#

0 *ake your 5uestionnaire attractive, simple to fill out, and easy to read# Aprofessional product usually gets professional results#

> 6eep the 5uestionnaire as short as possible# Eou are asking for a personstime, so make your re5uest as small as possible#

2 Use your cover letter to motivate the person to return the 5uestionnaire#Bne form of motivation is the have the letter signed by an individual knownto be respected by the target audience for your 5uestionnaire# 3n addition,make sure the individual will be perceived by the audience as having avested interest in the information needed#

/ Use inducements to encourage a reply# These can range from a smallamount of money attached to the survey to an enclosed stamped envelope#

A promise to report the results to each respondent can be helpful# 3f you dopromise a report, be sure to send it#

&roper use of these techni5ues can lower the nonreturn rate toacceptable levels# 6eep in mind, though, that no matter what you do, therewill always be non=respondents to your surveys# *ake sure the effort andresources you spend are in proportion with the return you epect to get#

The second source of bias is misinterpretations of 5uestions# Ce have


37/72

seen that these can be limited by clear instructions, well=constructed5uestions, and through 4udicious pilot testing of the survey# 8iased5uestions can also be eliminated by constructing the 5uestions properly andby using a pilot test# "inally, internal checks and a good motivational coverletter can control bias introduced by untruthful answers# Although biascannot be eliminated totally, proper construction of the 5uestionnaire, a

well=chosen sample, follow= up letters, and inducements can help control it#

83AS 3? DB:U?T(($ SA*&:(S

This section illustrates the many diverse, and sometimes powerfulfactors that influence survey findings as a result of using volunteers in asurvey# The conclusions epressed here regarding volunteer samples areprovided to make the surveyor aware of the often profound effects of non=respondent bias on survey data#

The eclusive use of volunteers in survey research represents anotherma4or source of bias to the surveyor == especially the novice# Although itmay not be immediately evident, it is nonetheless empirically true thatvolunteers, as a group, possess characteristics 5uite different from thosewho do not generally volunteer# Unless the surveyor takes these differencesinto consideration before choosing to use an eclusively volunteer sample,the bias introduced into the data may be so great that the surveyor can nolonger confidently generalize the surveys findings to the population atlarge, which is usually the goal of the survey#

"ortunately, research findings eist which describe several uni5ue

characteristics of the volunteer sub4ect# 8y using these characteristicsappropriately, the surveyor may avoid inadvertent biases and pitfalls usuallyassociated with using and interpreting results from volunteer samples# Thefollowing list provides ++ conclusions about uni5ue characteristics of thevolunteer# The list is subdivided into categories representing the level ofconfidence to be placed in the findings# Cithin each category, the conclusions arelisted in order starting with those having the strongest evidence supporting them#%from $osenthall and $osnow, The Dolunteer Sub4ect, -.92M pp -.2=-./')

7onclusions Carranting *aimum 7onfidence

- Dolunteers tend to be better educated than nonvolunteers, especially whenpersonal contact between investigator and respondent is not re5uired#

+ Dolunteers tend to have higher social=class status than nonvolunteers,especially when social class is defined by respondents own status ratherthan by parental status#

0 Dolunteers tend to be more intelligent than nonvolunteers whenvolunteering is for research in general, but not when volunteering is forsomewhat less typical types of research such as hypnosis, sensory isolation,


38/72

se research, small=group and personality research#> Dolunteers tend to be higher in need for social approval than nonvolunteers#2 Dolunteers tend to be more sociable than nonvolunteers#

7onclusions Carranting 7onsiderable 7onfidence- Dolunteers tend to be more arousal seeking than nonvolunteers, especially

when volunteering is for studies of stress, sensory isolation, and hypnosis#+ Dolunteers tend to be more unconventional than nonvolunteers, especiallywhen volunteering is for studies

Documents

Sampling and Surveying Handbook