Testing and Documentation Part I

Copyright 2010, The World Bank Group. All Rights Reserved.

Testing and Documentation Part I


Testing the questionnaire

If the questionnaire doesn’t work at all or as intended:– considerable time and money will have been

wasted– it makes the agency look bad– future funding may be unobtainable– it may be more difficult to get cooperation in the

future– public trust may be lost


Testing can tell you…

if others have the same definition and ideas about a concept as you do.

– The same word may mean different things to different people.

• It is worth investigating whether respondents have preconceived ideas about terms you use.

• You may need to choose a different term or include wording in your questions to clarify your intent.

• Interviewers may need to be trained to recognize and solve misunderstandings.



if a topic or question is sensitive or offensive.– Respondents may not be comfortable talking about some

topics. • Examples may include earnings, paternity, religion, and

disability.• Respondents may refuse to answer sensitive questions, may

lie, or may quit the interview. These outcomes should be avoided.

• Careful wording of questions may make them more acceptable.• Placing sensitive topics toward the end of the survey, when

possible, minimizes data loss if a respondent ends the interview.



if everyone understands the question in the same way.– If people understand the question to mean different things,

they will essentially be answering different questions. • They will not be giving answers that are comparable.• The resulting data may clearly not make sense. In this case,

the results will not be used and analysis in that area will be absent.

• The resulting data may not be obviously wrong. In this case, they may be used without knowing there is a problem but the resulting analysis will be meaningless and incorrect.



if the question was understood as intended.– Perhaps people understand the question to mean the same

thing, but it is different than what the interviewer was trying to ask.

• This may happen when the same words are used differently in common conversation and in the survey world.

• The resulting data would likely not show signs of a problem and would be used.

• The data would be consistent across respondents, but analysis would be incorrect as the data would not mean what the question indicates they should.



if interviewers are able to read the question as written.– For consistency across

interviews, it is important that all respondents are asked the same questions.

• If a question is tricky to pronounce or awkward to read, interviewers will tend to alter it. Not all interviewers will alter it in the same way however, introducing inconsistency.



if respondents are able to understand the question as read.– Questions may be too long, too complicated, have too

many parts, or contain words the person does not know. • Respondents may give answers that don’t make sense, make

up answers to avoid embarrassment, ask questions to clarify what was asked, or not answer.

• Satisficing by respondents leads to nonsense data.• Respondents asking questions for clarification may produce

good data, but it extends the length of the interview.• Respondents are usually more likely to supply an answer than

seek to clarify the question when they don’t understand.• If respondents to not understand several questions, they may

become uncomfortable and choose to end the survey.



if the answer categories provided are appropriate.– Responses may cluster in a category if it contains a trigger

word. – Additional categories that should be included may be found.– Requests for clarification may indicate that the categories

are not understood or that the boundaries between them are not clear.

– Responses clustering at either end of a numerical range may indicate that it needs to be extended.

– Confusion on hearing a numerical range may indicate that the range is surprisingly high/low to the person and they are reevaluating their understanding of the question.



if respondents are able to recall the desired information.– If the recall period is long, respondents may not clearly

remember the needed details or how often the event happened.

– If an event isn’t particularly memorable, the details or frequency of occurrence may have been forgotten.



if the skip pattern is correct.– Broken skips can take respondents to questions that do not

make sense for them. • They may make up answers to the questions, resulting in bad

data.• They may quit the interview in frustration.• This leads to data missing in sections where it should exist and

bad data existing in sections that shouldn’t have been asked.

if interviewers can follow the skip pattern.– Even if the skips are correctly written, they may be too

complicated for interviewers to follow correctly. • When interviewers cannot follow the skips, the same problems

result as come from broken skips.



if translations of the survey into other languages work well.– Construct equivalence is more important than linguistic

equivalence. • A word for word translation might not have the same meaning in

the new language as in the original.• Focus groups with speakers of the target language are

desirable to determine if they think about any of the concepts differently. This can help choose between different possible terms to use in the translation.

• Cognitive testing is needed to ensure that people get the same meaning from the questions and answer them in the same way.

Documents

Testing and Documentation Part I