testing surveys

8/7/2019 testing surveys

http://slidepdf.com/reader/full/testing-surveys 1/4

Web material accompanying The International Handbook of Survey Methodology

Chapter 10

Testing Survey Questions

Pamela Campanelli Independent Consultant

Summary of the Advantages and Disadvantages of Different

Question Testing Methods

Advantages Disadvantages

T h e T r a d i t i o n a l F i e l d T e s t

• Can be an aid toidentify

— Troublesome

questions

— Difficult concepts

— Respondentreactions to new datacollection techniques

— Etc.

• Yet, many problems can go by unnoticed

• Respondents may

— Misunderstand questions

— Use inappropriate judgment strategies

— Provide socially desirable answers

— Etc.

without giving off any signals that these errorsources are occurring

continued on next page…



E x p e r t R e v i e w / S

y s t e m a t i c R e v i e w s o f Q u

e s t i o n n a i r e s

Advantages

• Quick

• Cost effective

• Can uncover a widerange of potentialproblems from typosand skip pattern logicerrors to problems inhow concepts havebeen operationalized,plus

— Covers cognitiveaspects for respondent — Can uncoverpossible difficulties forthe interviewer — Can uncoverpossible problems foranalysis

• Can generate

hypotheses for testingwith other methods

• If a specific

appraisal form is used,the method yieldsquantitative data

Disadvantages

• Depends on abilities of the experts

• No respondents involved, so less convincing




R e s p o n d e n t D e b r i e f i n g

Advantages

• Comments arereceived directly fromthe respondent• Question-specificcomments can beused to fix the surveyquestion

• Field setting

• Larger sample sizespermit greaterconfidence in the

results

• Can be used todiagnose problems incontinuous surveys.

• In continuoussurvey— Large N facilitatesstatistical analysis — Rare groups can be

debriefed

Disadvantages

• Potential main survey problems have to beidentified in advance

• Subject to its own sources of response error

• Difficult to write good debriefing questions

B e h

a v i o r C o d i n g

Advantages

• Direct observationof the question-answering process

• Quantitativeindicator

• Standard codes

enhancescomparability

• Replicable

• Flexible, codes canbe tailored to thespecific needs of the

study

Disadvantages

• Standard method is time consuming

• Coders must be well-trained and use thecodes consistently

• Gives no information about why problem

occurs

• Additional investigation is needed to follow upon those questions that receive many problemcodes




C o g n i t i v e I n t e r v i e w i n g

Advantages

• Studies have foundthat “many problemsthat were identified inthe first field pretestwere pinpointed in thelaboratory in lesstime, with fewerrespondents, with lessprofessional effort,and at lower cost. Thelaboratory setting canalso be used to gaingreater insight into

the source of respondent difficulties” (NCHS, 1989, p. 29)

Disadvantages

• How to generalize— Small sample size — Often a convenience sample

—Often non-field setting

• Need highly trained interviewers

• Full analysis can be very time consuming

F o c u s G r o u p s

• The approach is

flexible enough thatunexpectedinformation can beimmediately followedup on by themoderator

• Information isobtained directly fromthe types of individuals who willparticipate in thestudy eventually

• Speed and costsaving as compared to

one-on-one interview

• Small group dynamics must be appropriately

controlled or results will have limited value

• Preparing the data for analysis and analyzingthe results can be time-consuming— Qualitative review of the transcript

— Information is not as detailed or as

systematic as from a one-on-one interview

• Results from a small number of subjects mustbe interpreted with care

End summary table.

Documents

testing surveys