Testing and Documentation Part II

Copyright 2010, The World Bank Group. All Rights Reserved.

Testing and Documentation Part II


Types of testing

Focus groups– Gather several people to have a guided discussion

– Particularly useful before developing questions

– Give insight into how people think about issues and terms• May indicate what wording would be best understood• May reveal if there is sensitivity around certain issues or not• May identify differences by class, education, or other

characteristics


Types of testing

Vignettes– Participants are given hypothetical scenarios and asked to

answer questions about them.– Especially useful when finding enough subjects with the

desired traits would be difficult– Removes desire to give desirable responses – Focuses attention on particular questions, thought

processes, or answer categories


Types of testing

Cognitive interviews

– Are used during questionnaire development • Multiple rounds are often done to test changes made from previous feedback • A and B versions may be tested simultaneously

– Subjects can be asked to think out loud while answering the survey• Lets you understand their thought process• Clarifies how they are interpreting the question• May reveal issues with recall, saliency, or ability to calculate

– Subjects can be probed about the survey questions to target issues the investigator is interested in

– Subjects can be given vignettes rather than or in addition to, responding based on their experience


Types of testing

Behavior coding– Observe interviews and take note of respondent and/or

interviewer behaviors such as:• Pauses that indicate confusion, lack or understanding, or recall difficulty• Respondents asking for clarification• Respondents giving inappropriate or inconsistent answers• Interviewers rephrasing questions• Negotiations on which answer category is best when none really fit

– When done testing content, can ask follow-up questions on issues that were noted

– Can also be used when survey is in the field


Types of testing

Interviewer debriefing– Done individually or in a group– Interviewers could answer written questions about the

experience and note any problems, suggestions, and questions if not available in person

– Can ask about:• Flow of questionnaire• Questions that are awkward to read or difficult to pronounce• Places where respondents needed help• Answers that were difficult to code or for which a suitable

category did not seem to exist• Concepts, terms, and reference periods that need to better

defined


Types of testing

Expert reviews– Survey is given to people who are experts in survey design

and/or the subject matter – Experts point out potential problems based on their

experience and knowledge– Experts suggest question wording, answer categories, etc

that may work better– Does not replace need to test questionnaire on respondents


Revising the questionnaire

After a round of testing, revisions are made to try to fix identified problems.

Additional testing is needed to see if the fixes work and/or if they cause new problems– Subsequent rounds should focus on the changes but also make sure

the change didn’t negatively impact understanding of other questions and that no errors were accidentally introduced

Multiple rounds of testing will be needed, but it can’t go on indefinitely.

Good version control is very important.

Document the test results, what changes were made and why, and results of retesting. – This allows for justification of decisions and contributes to knowledge.

Avoids reinventing the wheel.


Tips and techniques for testing

Indicate the version in the file name and also on the print version of the document.

Not all types of testing need to be done, but it is good to do more than one type. Each reveals different kinds of issues.

Testing should be done using the same mode as the actual interviews. If the questions will be read by an interviewer, don’t test them by having the respondent fill a paper questionnaire and vice versa.

Different does not necessarily mean better. Changes need to be tested too.


Importance of documentationDetails what is included in the survey and why

– Explains concepts and how to make them– International standards and needed additions/variations to them– Shows possible paths through the survey– Results of testing

Lists changes to the survey over time– When, why, impact, comparability– Necessary for data users making historical comparisons

Furthers openness and transparency– Increases public confidence– Makes avoiding political pressure easier– Avoids having to repeatedly answer the same questions– Critics might not agree with method but can trust change in same

series over time


What should be documented?

Most everything– If you are thinking about including it in the documentation,

it is probably worth having.– It is much easier to document something while it is fresh in

the mind than having to remember the details later. Old documentation should be added to, not deleted and

replaced.– All documentation should clearly marked as to what

survey and time period it refers to. – Consistent titles should be given to the same type of

document produced for different years.



Definite items to be made publically available and internally archived:– Public use data set – Publications and tables– Questionnaire (including skips)– Concept dictionary with verbal definitions and instructions on variable

edits– Data dictionary including variables and their locations on the data file– Interviewer training materials– Changes to the survey over time– Response rates, sample design, weighting methods, and survey

metadata– Corrections to publications and released data– Contact information for data users with questions



Definite items to have for internal use:• Original and edited data sets with personally identifying

information that cannot be shared with the public• Data cleaning techniques and data edits stored on the file• Significance tables/confidence intervals• Procedures for going from raw data to publication• Where data files are stored• Results of testingMaterials made for internal use can generally be shared with

data users except when it would violate respondent confidentiality

13


Examples of documentation

• Uganda Concept Dictionary

• Uganda Interviewer Training Manual

• Uganda Survey Flowchart

Documents

Testing and Documentation Part II