DESIGNING GOOD SURVEY INSTRUMENTS

OUTLINE Questions as Measures: An OverviewoWhat is good Q?oCharacteristics of Qs and Answers That

Affect Measurement General Rules for Designing Good Survey

InstrumentsoWhat to ask AboutoWording QsoFormatting Survey instrumentsoTraining Respondents.

WHAT IS GOOD QUESTION? A good Question is one that produces

answers that are reliable and valid measures of something we want to describe.

ReliabilityThe answers are consistent (Nunally, 1978)Answers are correspond to some

hypothetical “true value” (Cronbach & Meehl, 1995)

DEFINITION Type of survey method (questionnaire vs. interview) [Brehob, 2001] defines a questionnaire to be "a form

that people fill out, used to obtain demographic information and views and interests of those questioned".

[Kirakowski, 1998] defines a questionnaire in a more structural way as "a method for the elicitation, and recording and collecting information".

CHARACTERISTIC OF QS AND ANSWERS THAT AFFECT MEASUREMENT Wording

The researcher's side of the question and answer process is fully scripted so that the questions as written fully prepare a respondent to answer questions.

The question means the same thing to every respondent.

The kinds of answers that constitute an appropriate response to the question are communicated consistently to all respondents.

Bad Better

5.1 Age? What was your age on your last birthday?

5.2 Reason last saw doctor?

What was the medical problem or reason for which you most recently went to a doctor?

Incomplete wording

Question: When did you move to Kota Bharu?Possible answers:In 1948When I was 10After I Left college

CHARACTERICTIC OF QS & AS THAT AFFECT MEASUREMENT

5 BASIC CHARACTERISTICS OF QS ANS AS FUNDAMENTAL TO A GOOD MEASUREMENT PROCESS Qs need to be consistently understood. Qs need to be consistently administered or

communicated to respondents. What constitutes an adequate answer should be

consistently communicated. Unless measuring knowledge is the goal of the Qs,

all respondents should have access to the information needed to answer the Qs.

Respondents must willing to provide the answers called for in the Q.

o Reliable, Invalido Unreliable, Invalido Reliable, Valid

QUESTION EVALUATION 2Types:

Aimed at evaluating how well Qs meet 5 standards.

Assessing the validity of answers that result.

QUESTIONS EVALUATION Reliability types

Repeated measurement

Internal consistency Interjudge

Validity typesContentCriterian

Predictive Concurrent

Construct

o Also known as precisiono Consistent resulto Capacity to give same answer

oAlso known as accuracyoGive true measure

AIMED AT EVALUATING HOW WELL QS MEET 5 STANDARDS Focus Group Discussions Cognitive interviews, in which people’s

comprehend of Qs and how go about answering Qs probed and evaluated.

Field pretests under realistic conditions

ASSESSING THE VALIDITY OF ANSWERS THAT RESULT Analysis of resulting data to evaluate the strength

of predictable relationship answers and with other characteristics of the respondents.

Comparisons of data from alternatively worded Qs asked of comparable samples.

Comparisons of answers against records Measuring the consistency of answers of the same

respondents at two points in time.

A good survey instrument must be custom made to address a specific set of research goals.

TARGET RESPONDENTS

Before anything else!

“think about the people who you expect to fill out the questionnaire”

o MEMORYo JUDGEMENTo RESPONSES ISSUES

PRINCIPLE 1….The strength of survey research is asking about their first experiences: what they have done, their current situations, their feelings and perceptions. Beware of asking information that only acquired

secondhand. Hypothetical Qs Asking about causality About solution to complex problems.

Be specific , e.g. exact time, period, place (Who,What,Where,When, How)

e.g. income - (monthly? Annual? Household? individual?) Ask short, clear and specific questions, relevant

to the study e.g. Did you get the pamphlet from the staff nurse this

morning or you take it from the registration counter?

Choose appropriate time frames( salient period of recall - Context specific!!) e.g. How many time you have episodes of flu in past 4

years? e.g. How many time you have episodes of flu in last 2

weeks?

PRINCIPLE 2….

Asking one question at a time.Avoid;

Asking 2 Q at onceQs that impose unwarranted assumptionsQs that include hidden contingencies

Ask one question/thing in a single question statement or Avoid “double barrel” question (force respondents to make two decisions in one question) e.g. How many times did you go to the government

hospital and private hospital for the last one year? e.g. What is your Quality of life score when you are

feeling happy and sad?

e.g., Would you like to be rich and famous?

e.g., Are you physically able to do things like run or swim without difficulty?

Comments:Both of these asking 2 Qs the answers to

which can be different. If both Qs are important, ask two Qs, one at

a time.

e.g. With the economy the way it is, do you think investing in the stock market is a good idea?

Comment:Sudman and Bradburn (1982) call this a

one-and-a half barreled Qs (as contrasted with doubled-barreled question). Only a single Q is asked. Admittedly the clause does not exactly specify the state economy. However the implication is negative.

e.g. Do you agree or disagree; Given the amount of crime these days, it makes sense not to walk alone at night.

Comments:The Q makes respondents assume there is a

lot of crime and that the assumed crime rate affect decisions about walking alone. If the respondent does not accept these assumptions, the Q is very difficult.

e.g. In the last one year, did you have problems with the teeth/mouth AND/OR did you go for a dental check-up?

Comments:

e.g. How many times in the last 1 year did you have problems with your teeth/mouth AND/OR had dental check-up??

Comments:

PRINCIPLE 3..

A survey question should be worded so that every respondent is answering the same questions. The word should be chosen-all respondents

understand their meaning & have the same sense. The terms used that have meanings that are not to be

shared, definitions should be provided to all respondents

Time period referred Q should be unambiguous If too complex in a single Q, ask multiple Q

PRINCIPLE 4..If a survey is to be interviewer administered, wording of the questions must constitute a complete and adequate script such that, when interviewers read the question as worded, respondents will fully prepared to answer the question. If the definitions are to be given, they should be given before

the Q itself is asked. A Q should end with the Q itself. If there are response

alternatives, they should constitute the final part of the Q.

PRINCIPLE 5..

Clearly communicate to all respondents the kind of answer that constitutes an adequate answer to a question. Specify the number of responses to be given to questions

for which more than one answer is possible.

Introduce each section with short statement e.g. background data, demographic item, etc.

Allow for all possible answer and Add option for Don’t know or Others e.g. closed ended questions

Give clear and full instructions e.g. please ( ), can tick more than one answer…

Avoid “Leading” questions e.g. “Do you breast feed?”

Avoid using technical terms or jargon (keep the language simple)

Avoid abbreviation or unfamiliar words e.g. what is your QOL ? (QOL=Quality of Life)

PRINCIPLE 6..

Design survey instruments to make the task of reading Qs, following instructions, recording answers as easy as possible for interviewers and respondents.

Design the questionnaire to be “user friendly” e.g. language, wording, font size, format etc

Give clear and full instructions e.g. please ( ), can tick more than one answer…

e.g. For the first episode, did you seek treatment/check-up or obtain medicine from anywhere?1. Yes2. No3. RA

Comments:

To episode 2

e.g. for this problem, how much did you pay for *out of your pocket?

(*not including claims which are reimbursed by other party)

No Subject RM NA DK RA

1 Registration + treatment charges

2 Purchase medicine

3 Others, specify

PRINCIPLE 7..

Measurement will be better to the extent that people answering Qs are oriented to the task in a consistent way

(Fowler F.J., 1995)

Avoid too many answer or level of choice/scale (four or five scale points is usually quite sufficient) e.g. Never, rarely, occasionally, fairly often, often,

very often, almost always, always e.g. Excellent, very good, good, fair, poor

Avoid complex questions (“Calculation” questions )

e.g. “ How many %? ” “what is the total.. Etc”

BETTER QUESTIONNAIRS Length Complexity and diversity Flow Layout

SizeSpace (for elaborate answers)Skips

Computer aided questionnairesSpeed of analysis Intrusion/suspicionReliability and back-ups

RESPONDENT DEBRIEFING

HOW TO DO RESPONDENT DEBRIEFING? (1)Respondent Debriefing Questions

Respondent Debriefing Questions are special follow-up questions use to determine respondents understanding of the original survey question.

HOW TO DO RESPONDENT DEBRIEFING? (2)

Respondent Debriefing Questions can be used to determineo R’s understanding of terms/phraseso If R’s definitions are consistent with official definitionso Etc.

IF motivated from cognitive paradigm can explore: MEMORY JUDGEMENT and RESPONSES ISSUES

CPS EXAMPLE – LAST WEEK Respondent debriefing question

At the start of the questions about work. I asked you what (name) was doing most LAST WEEK. When you answered that question, which days did you think LAST WEEK was supposed to cover?

Interpretations of LAST WEEK

Sunday-Saturday(CPS Definition)

17%

Monday-Friday 54%

Monday - Saturday 9%

Monday - Sunday 6%

Sunday - Sunday 4%

Other 10%

CPS EXAMPLE – MAIN JOB (1) Definition of “main job” is never

specified in the questionnaireOfficial definition:

The job at which the specified person worked the most hours” and several sub-definitions for special situations.

Debriefing Questions: “You mentioned earlier that you had more than

one job. How did you decide which job was your main job?”

CPS EXAMPLE – MAIN JOB (2) Responses:

Job work at the most hours 63%The one that pays the most 29%The one worked at the longest

20%

Other 28%

*Note multiples responses were permitted so percentages sum more than 100%

I wish to ask you about your last visit to a dental clinic. When was the last time you had a dental check-up or

treatment at a dental clinic? (excluding medical doctors, bomoh, sinseh etc)

RESPONSES1. Within 1 year ago2. Between 1 to 2 years ago3. In the last 2 years under school dental programme4. More than 2 years ago5. Never had dental check-up/treatment

THANKS FOR YOUR KIND ATTENTION…..

REFERENCES Brehob, K., et al. “Usability Glossary.” http://www.usabilityfirst.com

Provides methods for creating easy-to-use website and software design. “Charm Survey Methods.” http://www.otal.umd.edu/hci-rm/survey.html

This website is a good starting point and overview of conducting a survey. Cobanoglu, Cihan, Bill Warde, & Patrick J Moreo. “A comparison of mail, fax and web-based

survey methods,” International Journal of Market Research; Henley-on-Thames; Fourth Quarter 2001, Volume 43, Issue 4, p.441-452; This article compares and contrasts three methods for issuing surveys: mail, fax, and web-based. It is

significant because it contrasts older methods of surveying with those brought on by the Internet and new technology.

Kirakowski, J., “Questionnaires in Usability engineering.” http://www.ucc.ie/hfrg/resources/qfaq1.html Presents FAQ regarding questionnaires; in particular, this site looks at questionnaires that measure

usability. Newsted, P. R., W. Chin, O. Ngwenyama, and A. Lee, (1997). "Resolved: Surveys have Outlived

their Usefulness in IS Research," Panel presented at the 1996 International Conference on Information Systems, December 17, 1997 ,Cleveland, Ohio. Presentation that gives various perspectives regarding the usefulness of surveys in MIS research.

REFERENCES Pinsonneault, Alain & Kraemer, Kenneth L. (1993). “Survey Research Methods

Methodology in Management Information Systems: An Assessment.” Journal of Management Information Systems 10(2):75-105. The authors perform a study within the field of MIS to determine problems that are

commonly found in the survey design and methodology. Five critical problems are found within the MIS field along with suggestions for improvement of survey research.

Roberts, E. S. (1999). "In defence of the survey method: An illustration from a study of user information satisfaction." Accounting & Finance 39(1): 53-77. This paper provides arguments regarding why survey methods are a valid research

method. Stone, Eugene F. (1978). Research Methods in Organizational Behavior. Goodyear

Publishing Company, Inc. Santa Monica, CA. Provides an overview of various research methods that can be used by researchers.

“Survey Design, Questionnaire Design Tips,” Creative Research Systems, http://www.surveysystem.com/sdesign.htm This website is a relevant source referenced in the UMCP website. It covers scanning

methods for collecting data from surveys. Trochim, William M.K. “Survey Research.”

http://trochim.human.cornell.edu/kb/survey.htm Source cited as basis of UMCP’s website. Deals mostly with general survey methods,

but provides methodology for selecting a survey type.