Data Collection MethodsReliability & Validity
Research Methology WorkshopKerman University of Medical
Sciences
Data Collection Methods
⢠Primary: where the investigator is the first to collect the data. Sources include: medical examinations, interviews, observations, etc. Merits: less measurement error, suits objectives of the study better. Disadvantage: costly, may not be feasible.
⢠Secondary: where the data is collected by OTHERS, for other purposes that those of the current study. Sources include: individual records (medical / employment); group records (census data, vital statistics)
Data Collection Methods
1. Physiological/Biological Measurements
2. Observational Methods
3. Interviews and Questionnaires
4. Records/Available Data
Physiological/Biological Methods
⢠Requires specialized equipment & training
⢠Objective⢠Precise⢠Sensitive to pick up
subtle variations⢠Can be expensive,
intrusive
⢠Physicalâ Weight, temperature
⢠Chemicalâ Blood glucose, Hgb,
lead levels
⢠Microbiologicalâ Cultures
⢠Anatomicalâ Radiological exams
Observational methods
⢠Use standardized systematic plan for observing and recording data
⢠Observer may or may not be concealed â One-way mirror to watch children
⢠Intervention may or may not occurâ Handwashing demo, then observe
⢠Structured: checklist, rating scale⢠Unstructured: field notes
Records/Available Data
⢠Use already collected data â Hospital recordsâ Historical documentsâ Videotapes
⢠Data may/may not have been originally collected for research
⢠May need data abstraction tool
Interviews & Questionnaires
⢠Both ask subjects to report data for themselves
⢠Must be clearly written so subjects understand intent
Interviews
⢠Data collector questions subjects
⢠Open-ended or close-ended questions
⢠Provides more personal, in-depth information
⢠Types:â Face to faceâ Phone
است ŮŮŮŮ ŮŮŘ´Ůد٠است زŮŘŻÚŻŮ ŮŮŮŮ ŮŮŘ´Ůد٠زŮŘŻÚŻŮعسŮد٠٠زŮŘŻÚŻŮ ŘŞŘ´ŮŮ٠٠سا٠ÚŮŘŻ ÚŻŘ°Ř´ŘŞ از ٞس اŮŘŞŘŘľŮاŮŮ Ůاعغ از ÚŻŘąŮŮŮ
از ŮŮŮ ŘŻŮد٠ب٠ŮبŮŮ Ůعاع ءب٠ا؏ت٠اؚ٠٠Ůاع٠؎Ůب Ůا٠٠ŮŮŘšŮŘŞ ب٠. Ů ÚŻŮ٠ب٠زŮŘŻ ŘŽŮŮŮ Ůا آ٠؏٠ؚ٠بŘŘŤ ŘąŮŘŞŮŘŻ ŘŽŮŘŻ داŮشگا٠٠؏عب اساتŮŘŻ
Ř´ŘŻ ŮŘ´Ůد٠زŮŘŻÚŻŮ Ů Ůاع از Ůاش٠Ůا٠استعس از بعا٠.Ř´ŮاŮŘŞ استاداŮŮاؚ از تؚداد٠٠ŮŮŮŮ ŮŮع٠Ů٠با Ů ŘąŮŘŞ آشٞز؎اŮ٠ب٠٠ŮŮ٠اŮا٠از ٞذŮعاŮŮبع؎٠٠ساد٠بع؎٠ŮŮ ŮŘąŮستا٠٠ٞاŮستŮŮ٠سعا٠ŮŮŮŘ Ůا٠؎Ůع٠ŮŮŮŮ
. Ů ŮŮ٠اŮا٠از ٠گذاشت Ů Ůز ŘąŮ٠عا ŘłŮŮ٠بازگشت بŮŘŻŮŘŻ ŮŮŮ ŘŞ گعا٠. ŘąŮŘŽŘŞŮŘŻ ŮŮŮŮ ŘŽŮŘŻ بعا٠Ů٠٠آŮŮ٠از ٞس ŮŮŮŘŻ ٞذŮعاŮŮ ŘŽŮŘŻ از تا ŘŽŮاست
ÚŻŮŘŞ ŘŽŮع٠: استاد ŮŮŮŮ ŮŮ ÚŻŮ Ů٠اŮŘŻ Ř´ŘŻŮ Ů ŘŞŮŘŹŮ Řت٠ا باشŮŘŻ Ůعد٠دŮŘŞ اگعاŮŘŻ بŮŘŻŮ ŮŮŮ ŘŞ اعزا٠٠ساد٠Ů٠آŮŮا ٠اŮŘŻ بعداشت٠عا زŮبا Ů ŮŮŮ ŘŞ گعا٠ŮاŮ
. . است بدŮŮ٠٠ءبŮؚ٠ش٠ا بعا٠ا٠ع اŮ٠اŮبت٠اŮŘŻ ٠اŮد٠باŮŮ ŘłŮŮ٠دعاست ŮŮ ŮŮ Ů٠ش٠ا Ůا٠استعس Ů Ů Ř´ŮŮات Ů٠٠است سعÚŘ´Ů Ů ŮŮ ŮŮ Ů٠ش٠ا Ůا٠استعس Ů Ů Ř´ŮŮات Ů٠٠عا! سعÚŘ´Ů Ů Ůا بŮŘŞŘąŮŮ ŮŮء ش٠ا
ŘŽŮاŮŮŘŻ Ů Ů ŘŽŮŘŻ .بعاŮعا بŮŘŞŘą Ůا٠؎Ůع٠ŮŮŮ٠آگاŮاŮ٠ا٠ا بŮŘŻ ŮŮŮŮ ŮŮŘ´Ůد٠ش٠ا Ů٠٠اؾŮŮ ŮؾدتŮŘŹŮ ŮŮز داشتŮŘŻ بع٠٠دŮگعا٠Ú٠آ٠ب٠ŘŮ٠اŮŮ ŘŻŘą اŮبت٠٠ŮعدŮŘŻ اŮت؎اب
ا؏ت٠اؚŮ. Ů ŮŮŘšŮŘŞ ŮžŮŮŘ Ř´ŘşŮŘ Ř¨Ř§Ř´ŘŻŘ ŮŮŮ٠زŮدگ٠اگع تعتŮب اŮ٠ب٠داشتŮŘŻ . Ů ... ŘŮظ بعا٠ابزاع٠ŮŮء آŮŮا ŮستŮŘŻ ٠تؚدد Ůا٠؎Ůع٠ŮŮŮŮ Ů٠ا٠Ů
. آ٠گاŮŮŘ ŘŻŘ§Ř´ŘŞ ŮŘŽŮاŮŘŻ Ůع٠آŮŮا ŘŻŘą زŮŘŻÚŻŮ ŮŮŮŮŘŞ ا٠ا اŮŘŻŘ Ř˛ŮŘŻÚŻŮ ŮÚŻŮداعŮŮ ŮŘŹŮŘŻ ŮŮŮ٠٠ز٠٠ءؚ٠اؾا٠ŮŮ Ůاست ŘŽŮع٠ŮŮŮŮ Ů ŘŞŮ؏٠٠ا ŘŮاس ŮŘŻŘą
ŮŮŮ ŮŮ Ů٠٠عا آ٠.ŘŻŘą ... ببعŮŘŻ ŮŘ°ŘŞ ŘŽŮŘŻ ŮŮŮŮ ŮŮŘ´Ůد٠از آ٠؏ا٠ب٠ŮŘ´ŮŘŻ ٞعت Ůا ŮŮ؏ا٠ب٠ŘŮاستا٠٠ŮŘ ŘŻŮستا٠ٞس
Questionnaire Design
⢠As a rule of thumb, never to attempt to design a questionnaire!
⢠A questionnaire is very easy to design, but a good questionnaire is virtually impossible to design.
Four key cognitive stages in responsing
⢠First, the respondent comprehends the item
⢠Second, he retrieves from memory the information required to answer it.
⢠Third, he has to make a judgement about the information needed to answer the question.
⢠Finally, he communicates his response.
Constructing New Instruments
⢠Methodological Research⢠Occurs when no tool available or existing tools
are not appropriate⢠Requires multiple steps
â Formulating itemsâ Testing itemsâ Developing instructionsâ Pilot testing
⢠Tools must be valid & reliable
The key to a successful survey is to
ensure that your questions are concise and easy-to-understand and give you valid and reliable information.
Explain your survey
Use incentives where possible
⢠A six-year-old might cooperate for an hour in exchange for a treat or game.
⢠Academics and some business audiences are motivated by the promise to share results.
⢠In general, everyone is more motivated by personal rewards.
Ordering of Questions
⢠Initial questions should be non-threatening, easy to answer and of interest to respondent
⢠Group questions by topicâ respondents will answer 20 questions on
smoking if theyâre sequenced but will complain if you ask 10 questions on smoking, go on to another topic, then go back to smoking questions.
Interview/Questionnaire Design
â˘Group similar questions togetherâ˘Keep sensitive questions near the
endâ˘Donât put two questions into one⢠Offering âDonât knowâ as a response option
⢠Offering âCouldnât sayâ or âNot applicableâ
The first several questions should:
â be easy for respondents to understand
â be important to the study's purpose
â engage the attention and interest of your respondent
Get your respondent interested in the survey at the very beginning
⢠In cases where the topic is already of interest or importance to the respondent, start with general questions, then funnel to more specific ones.
⢠If the topic is of low importance to respondents, start with specific questions. This gives respondents a frame of reference; then ask broader, more general questions
⢠Group questions in sections, and position sections or questions in a logical order.
⢠Introduce new sections with a sentence or phrase so that participants have a chance to switch mental gears.
⢠Place questions about sensitive issues such as income, sexual habits or drug abuse toward the end of the document, or section.
position bias
⢠Consider lists of similar items carefully. For example, you might ask about product satisfaction using 10 different attributes (price, availability, delivery, color, fit and so on). The first items on a list such as this usually get treated differently by respondents than the last items do.
⢠There are several ways to minimize position bias in a self-administered survey. One good way is to print several versions of the questionnaire, each with a different ordering of the items on the list.
⢠You can also randomly or alphabetically order list items and indicate in the instructions how they are ordered. This reduces the likelihood that respondents will see the first items as most important.
⢠Put demographic questions at the end of the questionnaire, if possible. There are at least two reasons for this. First, some demographic questions such as age and income can be sensitive.
⢠Second, it is better to keep respondents' minds on the purpose of the survey at the beginning, while you have their attention.
Interview/Questionnaire Design
Determine your surveyâs length:1. A survey targeting children should take no longer than
5 to 7 minutes to complete2. For most populations, self-administered surveys should be no longer than four pages3. Web-based surveys should ideally enable participants
to answer each question in a single step 4. Phone interviews should run no longer than 10
minutes 5. Face-to-face interviews can continue for an hour or
longer
Interview/Questionnaire Design
⢠Keep questions short: less than 25 words
⢠Avoid âdouble-barreledâ questions
⢠Avoid âleadingâ and âloadedâ questions
⢠Be specific
⢠KISS Principle- Keep It Simple, Statistician
⢠Begin with interesting & easy questions
Avoid double-barreled questions
⢠Double-barreled questions ask for opinions about two subjects at the same time:
⢠âAre you satisfied with the amount and kind of information you receive from your benefits administrator?â
Avoid loaded questions
⢠Loaded questions discredit your results. like:
⢠âMany doctors recommend exercise as a way of maintaining good health. Do you agree?â
Donât lead with responses
⢠With interview surveys, avoid beginning questions with a phrase like:
⢠âDo you very often, frequently, seldom or never...â
⢠Try to minimize the number of times the respondent or the interviewer has to follow a skip rule. For example "If the answer to question 3 is 'None of the Above,' skip to question 6."
Interview/Questionnaire Design
⢠Distinguish questions from answers
⢠Make your survey look attractive
⢠Conduct a pilot study: Usually, no more than 75§ respondents are needed for a pilot study
Distinguish questions from answers
Aesthetic considerations
⢠The presentation of a written survey can either encourage or deter a person from responding. Make sure your survey is inviting by:
⢠Including plenty of white space so the document does not appear intimidating.
⢠Including color, if possible, but no more than two colors.
⢠Using no more than two typefaces.
Borrow from the pros
⢠You can borrow style from the
large national surveys
Sensitive questions
⢠Casual approach â âDid you by any chance murder your wife?â
⢠The âEverybodyâ approachâ âAs you know, many people have been killing their wives these
days. Do you happen to have killed yours?â
⢠The âOther Peopleâ approachâ âDo you know any people who have murdered their wives?â
Pause for replyâŚâand how about yourself?â
â˘Avoid questions that have a ârightâ Avoid questions that have a ârightâ answeranswerâ˘Avoid multiple or double-barreled Avoid multiple or double-barreled questionsquestionsâ˘Keep questions briefKeep questions briefâ˘Use simple termsUse simple termsâ˘Insure response options are Insure response options are mutually exclusivemutually exclusiveâ˘Insure that response options are Insure that response options are exhaustiveexhaustive
⢠Except in rare circumstances, include âDonât knowâ and âRefusedâ as response options.
⢠Scale response options to create equal distribution of responses
â always, nearly always, sometimes, seldom, never
⢠Always pilot test new questions
⢠If necessary, conduct second pilot⢠Most importantly, be polite and respectful
throughout the interview.
Wait about six weeks to start analysis
⢠Six weeks after the initial survey was sent, you should have received about as many completed surveys as you ever will.
Some tips⢠one would expect to be able to use adult questionnaires with adolescents aged 16 and
up with little difficulty.⢠âGood qualityâ means responses are both
validvalid (i.e. accurately reflect what the question intended to elicit) and
⢠reliablereliable (i.e. would be forthcoming if the same survey were conducted under similar conditions again). And
⢠contain few if any unanswered questions (âitem non-responseâ).
Some tips
⢠it is recommended that negative questions
⢠should be avoided in questionnaires for children; this is contrary to standard practice with adult questionnaires, where an equal balance of positively-and negatively phrased questions is thought to enhance data quality (Borgers et al., 2000).
⢠(e.g. âdo you find it difficult to finish your homework within the deadlines given?â).
Some tips
⢠Completely-labelled scales (i.e. those where every point has a label) have been shown to produce better-quality responses from children than partially-labelled ones (e.g. where only the two extremes are labelled).
⢠It has also been suggested that verbal labels are more easily understood than numeric ones (Borgers and Hox, 2000)
⢠As a general rule of thumb, it seems sensible not to offer an explicit âdonât knowâ, except at
questions where you consider it to be a valid response, e.g. where specific knowledge will be required to answer substantively.
⢠Similarly, where an absence of opinion is deemed to be a genuine possibility, there may be a case for offering this as an explicit answer option, in spite of the fact that it may decrease the amount of useable data collected overall (Borgers and Hox, 2000).
Some tips
⢠Cognitive pretest: The âthink aloudâ technique requires the respondent to articulate what he is thinking as he goes about answering the question. For example, he is encouraged to say what he thinks the question means; identify any words or concepts that seem unclear; talk through the process of retrieving and formatting the necessary information; and explain how he arrives at his chosen response. (De Leeuw et al., 2004).
Some tips
Reliability & Validity
Reliability & Validity
⢠Validityâ Does an instrument actually measure what it
is suppose to measure?
⢠Reliabilityâ Extent to which an instrument yields the
same results repeatedly
⢠Should be addressed in Methods section
Validity Types
⢠Based on Judgmentsâ Face Validity
â Content Validity
â Consensual Validity
Validity Types
⢠Based on Checks against Dataâ Criterion Validity
(Sensitivitu&specificity)
â Construct Validity