Embed Size (px)
Citation preview
This article was downloaded by: [Hyunjoo Park]On: 08 August 2013, At: 18:43Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
International Journal of SocialResearch MethodologyPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/tsrm20
Investigating validity and effectivenessof cognitive interviewing as apretesting method for non-Englishquestionnaires: Findings from Koreancognitive interviewsHyunjoo Park a , M. Mandy Sha b & Yuling Pan ca RTI International , 351 California St., Suite 500, San Francisco ,CA , 94104 , USAb RTI International , 230 W. Monroe, #2100, Chicago , IL , USAc U.S. Census Bureau , 4600 Silver Hill Road, Washington , DC ,USAPublished online: 08 Aug 2013.
To cite this article: International Journal of Social Research Methodology (2013): Investigatingvalidity and effectiveness of cognitive interviewing as a pretesting method for non-Englishquestionnaires: Findings from Korean cognitive interviews, International Journal of Social ResearchMethodology
To link to this article: http://dx.doi.org/10.1080/13645579.2013.823002
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
Investigating validity and effectiveness of cognitive interviewing asa pretesting method for non-English questionnaires: Findingsfrom Korean cognitive interviews
Hyunjoo Parka*, M. Mandy Shab and Yuling Panc
aRTI International, 351 California St., Suite 500, San Francisco, CA 94104, USA; bRTIInternational, 230 W. Monroe, #2100, Chicago, IL, USA; cU.S. Census Bureau, 4600 SilverHill Road, Washington, DC, USA
(Received 8 January 2013; final version received 1 July 2013)
This study aims to empirically investigate whether cognitive interviewing iseffective as a pretesting method for detecting problems in translated surveyquestionnaires as in the English language source questionnaire. The validity ofnon-English cognitive interviews is of concern among researchers because thecognitive interviewing technique was developed based on the communicativenorms of Western cultures where directness and openness in expressing one’sopinion is encouraged. However, different communicative norms are present inAsian languages and cultures. To date, little research has been conducted on thistopic. Using qualitative and quantitative analysis, we compared survey questionproblems identified through the cognitive interviews conducted in English andin Korean and found that Korean cognitive interviewing was indeed effectivefor detecting problematic survey questions.
Keywords: questionnaire design; cognitive interviews; pretesting; Korean;Asian
1. Introduction
Cognitive interviewing has been widely used as a tool for evaluating English surveyquestionnaires. With changes in the ethnic composition of the US population, moresurvey materials are being translated into non-English languages, and cognitiveinterviewing with such populations has become an accepted practice for assessingthe quality of translated questionnaires.
Previous research in the USA reported that respondents who speak little or noEnglish exhibited difficulties providing adequate answers to the interview probes innon-English language cognitive interviews. These respondents were usuallyunfamiliar with surveys in general or had limited exposure to the English languageand mainstream American culture (Coronado & Earle, 2002; Kissam, Heerrera, &Makamoto, 1993; Pan, 2004; Pasick, Stewart, Bird, & D’onofrio, 2001).
Beyond this difficulty, researchers expressed concerns about conducting cogni-tive interviews in languages that practice different communicative norms from theEnglish language and Western cultures. For example, Korean speakers are known to
*Corresponding author. Email: [email protected]
International Journal of Social Research Methodology, 2013http://dx.doi.org/10.1080/13645579.2013.823002
� 2013 Taylor & Francis
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
rely heavily on cultural context and background information rather than on theactual content of the message in communications. For example, an emphasis onpoliteness and social hierarchy in the Korean culture seems to lead to shortresponses in Korean language cognitive interviews (Pan, Landreth, Hinsdale, Park,& Schoua-Glusberg, 2010).
The cognitive interviewing methodology builds upon the participants’ open dis-cussion and articulation of their thoughts and feelings about the questionnaire beingtested. However, dramatic differences in communicative norms across languagesand the associated cultural contexts challenge the validity and effectiveness of usingthis methodology to pretest translated survey questionnaires. Despite the potentialimpact, little research has been conducted on this topic.
In this paper, we explore the validity and effectiveness of cognitive interviewingin ‘high-context’ languages by exploring the identified problematic survey questionsthrough cognitive interviewing conducted in English and Korean.
2. Literature review
Cognitive interviewing is a method used to identify problematic survey questionsby asking research participants to report what they are thinking either while answer-ing those survey questions or retrospectively. Because of its relatively short history,most cognitive interviewing research has focused on examining its effectiveness asa valid and objective pretesting tool (Bolton, 1993; Conrad & Blair, 1996, 2009),and researchers have tended to evaluate the cognitive interviewing method withother pretesting techniques to see whether problems detected in different pretestingtools are convergent (Presser & Blair, 1994; Rothegeb, Willis, & Forsyth, 2005;Willis, Schechter, & Whitaker, 1999). Despite the growing use of cognitive inter-viewing to pretest survey questions, there is a general dearth of research to informresearchers of how to implement cognitive interviews in a systematic way so thatfindings can be replicated. In addition, most cognitive interviewing research hasbeen focused on data collected in English.
Communication in English relies more on the facts of the message than on cul-tural contexts and background information, and is characterized as ‘low context.’The actual content of the message is more important than when, how, and by whomit is expressed. In contrast, speakers of ‘high-context’ languages, such as Asians,rely heavily on context, background information, and interpersonal cues (Hall &Hall, 1987). Furthermore, the perceptions of ‘self’ in Eastern cultures place highvalue on attending to others with an emphasis on maintaining harmonious relation-ships (Markus & Kitayama, 1991; Masuda & Nisbett, 2001). Silence and acquies-cence are also interpreted as ‘polite behaviour’ (Javeline, 1999) in the Easterncultures. These differences could be challenges for conducting cognitive interviewsin Asian cultures, since cognitive interviewing has been implemented in the com-municative norms of English and Western cultures where directness and opennessin expressing one’s opinion is a preferred communication style (Pan et al., 2010).Under the assumption that cognitive interviewing will work equally well in differentcultures, researchers have conducted cognitive interviews with non-English speakersto test translated questionnaires. They reported various difficulties that the respon-dents exhibited. For instance, in a study with bilingual, English-dominant, US-bornHispanics, and monolingual Spanish-speaking respondents, Kissam et al. (1993)reported that the difficulties respondents had often involved unfamiliarity with
2 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
self-administered questionnaires and a lack of literacy for understanding specificCensus concepts, lexical difficulties originating from the translation, or dialecticaldifferences among Hispanic subgroups of different national origins. While conduct-ing cognitive interviews with African-American, Chinese, Latino, and Vietnameserespondents, Pasick et al. (2001) reported respondent difficulty with think-aloudtasks, especially for those with low levels of education. Coronado and Earle (2002)also reported Spanish-speaking respondents’ difficulty with meaning-oriented probesand paraphrasing probes. The findings were echoed through Chinese cognitive inter-views by Pan (2004), who reported that many respondents had difficulty withthink-aloud tasks, process-oriented probes, and meaning-oriented probes. This wasalso true among those with advanced degrees. In addition, Yuan, Wake, Park, andNguyen (2009) observed from interviews with Asian monolingual respondents thatthey exhibited difficulty in understanding interview probes and articulating theirthoughts. Researchers think this difficulty originates from non-English-speakingrespondents’ unfamiliarity with the social context of surveys and the differentcultural assumptions and linguistic practice that they bring to the cognitiveinterview setting (Pan et al., 2010; Pasick et al., 2001; Willis et al., 2005).
Beyond these observed cross-cultural difficulties of providing adequate answersto the probes during interviews, what concerns researchers is that a considerablenumber of respondents expressed anger, disgust, or humiliation (Coronado & Earle2002; Kissam et al., 1993) as they tried to provide correct answers to probes andsought for interviewers’ confirmation that they performed well. This may indicatethat respondents perceive cognitive interviews as being more akin to a test than anopinion-based interview, despite the interviewers’ multiple assurances to the con-trary (Kissam et al., 1993; Yuan et al., 2009). Yuan et al. (2009) also reportedrespondents’ tendency to avoid giving negative feedback during the interviews.More recently, Chan (2010) found that Chinese-speaking respondents were morelikely to provide indirect and/or contrary-to-face value responses than Englishspeaking respondents. Considering that openness and directness of communicationare critical for detecting survey question problems in cognitive interviews, theseobservations are worth further investigation. None of these studies, however, wasspecifically designed to examine the methods of cognitive interviewing. They sim-ply reported the difficulties encountered in their interview experiences.
A few studies have attempted to investigate the cross-cultural function of surveyquestions using cognitive interview data. Willis and Zahnd (2007) reviewed ques-tion problems identified through cognitive interviews with Korean and Englishspeakers and classified the problems into three categories: (1) translation problems;(2) culture-related problems; and (3) general questionnaire problems that affect alllanguages. The fact that the Korean cognitive interviews identified general question-naire problems that affected both languages implies that Korean cognitive inter-views can be used for detecting some of the underlying problems common tosurvey questions in general. Similarly, Berrigan et al. (2010) identified survey ques-tion problems from cognitive interviews with Latino respondents, and they catego-rized the problems into ‘general’ survey problems and ‘socio-cultural’ problems.Kudela, Forsyth, Levin, Lawrence, and Willis (2006) explored the strengths andweaknesses of cognitive interviewing and behavior coding in Chinese, Korean, andVietnamese by comparing results from these two methods. In contrast to othercognitive interview studies with non-English-speaking respondents, they reportedthat the cognitive interviews went smoothly, and that cognitive interviewing and
International Journal of Social Research Methodology 3
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
behavior coding results were similar in that neither revealed many culture-specificissues. Furthermore, Warnecke et al. (1997) reported cross-cultural differences inthe survey answers obtained from cognitive interviews. Compared with non-His-panic white respondents, they found that Mexican Americans, Puerto Rican, andAfrican-Americans had a higher tendency of providing socially desirable answers.Although these studies have provided helpful information about non-Englishcognitive interviews, the findings could not be used to evaluate the validity fordetecting problematic survey questions. It is because these studies lack a referencepoint that can show how effective non-English cognitive interviews are in compari-son with English cognitive interviews. Recently, Miller et al. (2011) developed andrecommended the methodology and three levels of analysis for conducting compara-tive cognitive testing. Although their research sheds light on how to conductcognitive testing across cultures, this research starts from the assumption thatcognitive interviewing is a valid pretesting tool to collect information to identifyquestionnaire problems across different cultures.
The present research study seeks to fill the gap in the literature by investigatingdata from the Korean cognitive interviews. We selected the Korean languagebecause of the researchers’ expertise in Korean and also because Korean is a high-context language and contrasts with English as a low-context language. The specificresearch questions are as follows:
(1) Do Korean cognitive interviews identify a similar type and number ofquestionnaire problems as those conducted in English?
(2) Is there a difference in the pattern of detected survey problems betweenKorean-language cognitive interviews and English-language interviews?
3. Methodology
The present study uses data from a project conducted by the US Census Bureau toevaluate the translation of the 2010 Census questionnaire in multiple languages.Data are drawn from 23 Korean (12 interviews in Round 1 and 11 in Round 2) and16 English cognitive interviews (eight in each round). The Korean interviewrespondents spoke little or no English but were native Korean speakers. The inter-views took place in Chicago, Illinois, and the Greater Washington DC area betweenFebruary and April 2008. Bilingual and bicultural Korean language experts whowere also experienced cognitive interviewers interviewed respondents who haddiverse demographics to mirror the characteristics of the people who would likelyneed to use a translated 2010 Census form based on past statistics from theAmerican Community Survey (ACS) data collection efforts.
To systematically examine the detected survey problems in the cognitive inter-views, we adopted a coding scheme. It consists of five problem categories (Lexical,Inclusion/exclusion, Temporal, Logical, and Computational problems) originallydeveloped by Conrad and Blair (1996) and has an additional Translation problemcategory to explain problems introduced during the translation process. We chosethis particular coding scheme among others (Fitzgerald, Widdop, Gray, & Collins,2011; Presser & Blair, 1994; Rothgeb, Willis, & Forsyth, 2005; Willis et al., 1999)because of its practical simplicity while enabling us to consider a broad set ofcriteria that encompass most of the common survey problems. Below is a summaryof each problem category. See Conrad and Blair (1996) for more details.
4 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
(1) Lexical problems are related to not knowing the meaning of words or howto use them.
(2) Inclusion/exclusion problems arise in interpreting the scope of a term or con-cept. This type of problem also involves word meanings, but is more closelyrelated to determining whether certain concepts are considered within thescope of a word in the question.
(3) Temporal problems are related to the time period to which the questionapplies or the amount of time spent on an activity described in the surveyquestion.
(4) Logical problems involve false presuppositions of the question: the devicesused to connect concepts such as logical connectives like ‘and’ and ‘or,’ con-tradictions, and tautology.
(5) Computational problems include any problems related to respondents’ diffi-culty processing the given information in the survey question. When respon-dent problems do not fall into the other problem categories, they areclassified as computational problems in our study.
(6) Translation problems are related to inaccurate translation. Respondents maynot understand the intended meaning of survey questions because the transla-tion did not capture the anticipated meaning of English texts. In a strictsense, it is not a respondent’s problem but a problem embedded in the trans-lated questions.
Under this coding scheme, we extracted respondents’ answers to the 29 scriptedprobes (28 probes used in both rounds and one unique probe in Round 2) and theassociated spontaneous probes for both the Korean and English cognitive inter-views. Multiple codes were allowed in one response; when one response containedmultiple problems, they were coded into multiple corresponding problem categories.In addition, some probes were not asked either because the interviewers forgot toadminister the particular probes or because they intentionally omitted them becausethey knew that the respondent understood the survey questions from their earlierinterview responses.
Assuming that the interviewers asked all probes that were scripted in the inter-view protocols, we should have a total of 523 responses for the Korean interviews.They included 28 probes from 12 Round 1 interviews (336 responses) and 17probes from 11 Round 2 interviews (187 responses). For English interviews, thereshould be 360 responses consisting of 28 probes from eight Round 1 interviews(224 responses) and 17 probes from eight Round 2 interviews (136 responses). Dueto the lack of responses from the omitted probes, the Korean cognitive interviewdata-set consisted of 522 codes from 521 responses, and the English data-set con-sisted of 324 codes from 322 responses.
Coding was based on the interviewers’ written summary reports. This reportdetails the probing questions and the respondents’ answers to them from each cog-nitive interview. Each report was prepared by the interviewer who conducted thecognitive interview. Although summary reports may reflect each interviewer’s inter-pretation of the respondent’s answers, they included many actual quotes and conver-sations that were very close to the actual interview conversation.
However, because these reports were not transcripts, there was sometimes insuf-ficient detail for coding purposes, and this affected English and Korean interviewdata equally. When this happened, we followed several coding principles. The data
International Journal of Social Research Methodology 5
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
was coded as ‘no problem’ when the data did not provide clear evidence for us tocode it into one of the problem categories. For example, a respondent defined‘adopted son’ as ‘a son who is not biologically related to either parent.’ Althoughthis response did not discuss the legal process that defines an adoption, it was notindicative of comprehension problems because the respondent did discuss biologicalrelationship, one of the essential parts defining ‘adopted son.’ The response, how-ever, was probably too short and should have been probed further by the inter-viewer. Because the lack of information also could have been the result of thesummary reporting by the interviewer, we decided to code these responses as ‘noproblem.’
On the other hand, if the response had incomplete information but containedclear comprehension issues, we coded it as ‘problem.’ Another situation that arosewas when the respondents interpreted the wording as intended, but they did notmark the intended answer when they filled out the questionnaire. We also found afew examples where a respondent did not mark any response choices, but demon-strated no comprehension issues in the cognitive interview probes. This may indi-cate difficulty in filling out self-administered questionnaires. However, for ourresearch purpose, we coded them as ‘no problem’ because there was not enoughevidence to indicate the respondent had problems in understanding the survey ques-tions. This may be a topic for a separate usability study. The majority of the inter-view summaries had sufficient details, and we were able to code them according tothe six problem categories.
Three people who had completed advanced training in social sciences contrib-uted to the coding effort. The lead coder conducted interviews in Korean, and theother two coders participated in the Chinese and the English cognitive interviews.Therefore, the coders all had sufficient background knowledge about these cognitiveinterviews and the interview protocols. The coders held meetings and continuedemail communications to setup the coding criteria. After the first coding wascompleted by the lead coder, all coders met several times to resolve the unclearcases. Final coding values were assigned to each case from the results of thesediscussions.
4. Findings
To answer the research questions posed earlier, we first present the number of prob-lems detected through Korean cognitive interviews as a whole. Then, we examinethe detected problems by category from the Korean interviews along with thosefrom the English interviews. Lastly, we present some examples of the detected sur-vey questions to provide qualitative information to complement our quantitativefindings.
4.1. Quantitative findings
First, we identified a total of 125 codes (survey problems) from the 521 Koreaninterview responses (24.0%). This demonstrates that as a pretesting tool, cognitiveinterviewing in Korean achieves its function of identifying survey questionproblems. This answers our first research question concerning whether conductingKorean cognitive interviewing is effective in identifying questionnaire problems.
6 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
To examine if the detected problems appear in all six of the survey problemcategories, we reviewed the frequency of detected issues by survey problem cate-gory. Below is a summary of the frequency from the Korean cognitive interviews.The detected issues indeed appear in all of the categories. Specifically, Lexicalproblems were identified most often (49 problems), followed by Computation prob-lems (26 problems), Translation problems (24 problems), and Inclusion/exclusionproblems (24 problems). Only one problem each was detected under the Temporalproblem category and the Logical problem category. These findings may indicatethat Korean cognitive interviewing is only effective for identifying certain types ofsurvey problems, such as Lexical problems. However, this may have resultedbecause the questionnaire items being tested actually contained many Lexical prob-lems and few Temporal/Logical problems.
As a comparison, we examined the frequency of the detected issues by categoryfrom the English cognitive interview data. In the English cognitive interviews, weidentified a total of 62 problems (19.3%) by analyzing 322 responses, and theseappear in all of the survey problem categories (the Translation category is not appli-cable because the English questionnaire is not a translation). These findings indicatethat English cognitive interviews identified issues from all of the survey problemcategories. Similar to the Korean findings, most of the identified problems wereLexical (23 problems), followed by Computational (20 problems) and Inclusion/exclusion problems (12 problems). It is noteworthy that the English cognitive inter-views did not identify many Temporal problems or Logical problems either, asshown in Table 1. Only three problems were detected under the Temporal problemcategory, and four were detected under the Logical problem category. This findingindicates that Korean cognitive interviewing may be as effective as English cogni-tive interviewing for identifying problematic survey questions, and the few issuesidentified under some categories in this analysis might just mirror the actual distri-bution of the problems contained in the survey questionnaire.
In addition, we calculated the proportions of the problems identified. Afterexcluding Translation problems that can only occur in the Korean interviews, wefound very similar percentages of identified problems (18.6% of the Koreanresponses vs. 19.3% of the English responses). However, the overall frequencycomparison at the entire data level could be misleading if individual problemsdetected at each probe are significantly different. To address this issue, we took acloser look at the data-set by comparing the percentages of each probe beingdetected as problematic between the Korean and English cognitive data. Translationissues are regarded as no problems in the Korean data for fair comparison.
Table 1. Number of problems by category between English and Korean interviews⁄.
English interviews Korean interviews
No problems 262 81.4% 397 76.2%Problems 62 19.3% 125 24.0%1. Lexical 23 7.1% 49 9.4%2. Inclusion/exclusion 12 3.7% 24 4.6%3. Temporal 3 0.9% 1 0.2%4. Computational 20 6.2% 26 5.0%5. Logical 4 1.2% 1 0.2%6. Translation 0 N/A 24 4.6%
⁄Because of multiple codes given to one response, the total sum percentage is slightly over 100%.
International Journal of Social Research Methodology 7
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
Table 2 shows that 5 out of 29 different probes were identified as having noissues at all for both languages. We can also see the pattern showing that the per-centage of each probe being identified as problematic is quite similar between thetwo language data. As displayed in the last column of the table, all of the absolutedifferences are no greater than 0.5 except for Probe 4, meaning that if a certainprobe is detected as problematic from the English interview, it is also likely to bedetected as problematic from the Korean interview.
We continued to investigate the data-set to see if this similar pattern is stillfound in the distributions of the detected problem categories at each response level.
Table 2. Percentage of responses identified as problematic at each probe between Englishand Korean interviews and its absolute difference.
Probes English KoreanAbsolutedifference
1. Meaning of ‘living or staying’ 0.29 0.33 0.052. How did R choose that answer? Who did R include? 0.18 0.17 0.003. What does R think they (Q2) mean here? 0.50 0.17 0.334. Foster children? 0.29 1.00 0.715. What does R think they (Q3) mean here? 0.09 0.36 0.276. How did R choose R’s answer? 0.07 0.13 0.067. What does it mean by ‘include home equity loans’? 0.38 0.67 0.298. What does R think they mean by ‘free and clear’? 0.25 0.09 0.169. Are the first and second options (Q3) the same ordifferent?
0.19 0.13 0.06
10. Are the third and fourth options (Q3) the same ordifferent?
0.13 0.26 0.14
11. What does R think they (Q5) mean here? 0.40 0.33 0.0712. What does R think this instruction (Q5) is asking R todo?
0.50 0.22 0.28
13. How did R decide who is Person 1? 0.27 0.35 0.0814. Did R know what to write in the space marked ‘MI’? 0.00 0.00 0.0015. ‘Please report babies as age 0 when the child is lessthan 1 year old.’ If a person has a 4-month old babygirl, what age should R write here?
0.41 0.00 0.41
16. What did R think the instruction (before Q8) is askinghim/her to do?
0.00 0.36 0.36
17. For Question 8 [Hispanic origin], what is this questionasking in R’s own words?
0.27 0.39 0.12
18. Was R able to find a category that fit R (Q9)? 0.00 0.04 0.0419. What does R think this question [Question 10] isasking for?
0.07 0.04 0.02
20. College housing? 0.13 0.00 0.1321. At a seasonal or second residence? 0.00 0.00 0.0022. For child custody? 0.50 0.00 0.5023. In a nursing home? 0.00 0.00 0.0024. Housemate or roommate? 0.23 0.35 0.1225. Unmarried partner? 0.00 0.09 0.0926. Roomer or boarder? 0.38 0.04 0.3427. Stepson or stepdaughter? 0.00 0.00 0.0028. Adopted son or daughter? 0.00 0.00 0.0029. What do you think they mean by ‘related to person1’?
0.63 0.45 0.17
Average 0.16 0.13 0.03
8 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
To do this, we calculated the percentages of being classified into six different cate-gories. For example, there are a total of 12 responses from the Korean interviewsfor Probe 1. Among the 12 responses associated with Probe 1, eight responses werecoded as having no problems, three as Lexical problems, and one as an Inclusion/exclusion problem. In this case, the percentages calculated in each category are67% (=8/12), 25% (=3/12), 8% (=1/12), 0% (0/12), 0% (0/12), and 0% (0/12) inthe order of No problem, Lexical, Inclusion/exclusion, Temporal, Computational,and Logical problem. In this way, 174 data points (29 probes ⁄ 6 categories) werecalculated from each language interview data-set, and the 174 absolute differencesbetween the Korean and English interview data were also calculated.
Out of 174 data points, 102 data points are exactly the same, which explainsapproximately 60% of the data-set. Considering the small sample size forresponses to one probe, this level of match rate looks very positive for drawingthe conclusion that Korean cognitive interviewing is as effective as English cog-nitive interviewing. In line with the previous findings, the absolute differences ofproblem category distribution at each probe level were generally low, so thatover 80% of the data-set exhibited a very similar pattern with less than 0.1absolute difference between Korean and English interviews. This demonstratesthat the way the Korean cognitive interview and English cognitive interviewidentify and categorize problems is very similar and comparable. The absolutedifferences are summarized in Table 3.
4.2. Qualitative problem examples from Korean interviews
The quantitative analysis supports the idea that Korean cognitive interviewing is aneffective tool for identifying survey question problems. However, this numericalcomparison might be limited to providing substantial information about how Koreancognitive interviews actually work in order to revise and improve survey questions.To provide a more comprehensive picture of Korean cognitive interviewing, wepresent some examples of problems detected from the Korean cognitive interviewsbelow. This qualitative data does not offer new findings, but rather insights for thereader to better understand the quantitative findings and should be interpreted as anextension of the quantitative analysis.
4.2.1. Lexical problems
Lexical problems, which were identified most frequently in both the English andKorean interviews, occurred when respondents did not know the meaning of certainwords in the given contexts.
Table 3. Level of absolute difference distribution.
Level of absolute difference (AD) Frequency %
Exact match 102 59AD6 0.10 36 210.1 <AD6 0.3 25 140.3 <AD6 0.5 9 5AD>0.5 2 1
International Journal of Social Research Methodology 9
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
In Example 1, Question 3 intends to find out whether the respondent’s home isowned, rented, or occupied without payment of rent. However, the respondent didnot know the translated word for ‘household.’ Respondents who did not know themeaning of the words used in the Census questions or response choices most likelydid not comprehend the intended meaning of the question. Lexical knowledge ofphrases and words like ‘household’ played a critical role in comprehension.
Example 1
Probe question (Q): What do you think they mean here?
Response summary (R): The respondent said the question was very unclear and it wasdifficult to figure out what the question meant, especially the word ‘세대 (household).’The respondent kept asking the meaning of the word. When the interviewer explainedwhat was meant by a household, the respondent understood it but said the term shouldbe changed to another term such as ‘집 (house).’
4.2.2. Inclusion/exclusion problems
Inclusion/exclusion problems occur when a respondent includes or excludes thingsbecause he or she interprets a term in a survey question differently from how it isintended in the question.
The following text (Example 2) is located toward the end of the questionnaireand is intended for households with more than six household members. Earlier inthe questionnaire, the relationship between each household member and Person 1was asked (e.g. whether a certain household member is Person 1’s wife, daughter,parent-in-law, etc). However, this particular question (shown below) is only askingabout whether this seventh household member, Person 7, is related to the house-holder, Person 1, using an incomplete sentence for space saving at the end of thequestionnaire. The respondent in the example interpreted the term ‘related to’ toinclude relatives in general as opposed to first degree relatives that is intended.Thus, if Person 7 was the householder’s youngest sister, the respondent should haveanswered ‘yes.’ However, because he interpreted the term ‘related to’ in a different,narrowed way, this Inclusion/exclusion problem occurred.
Example 2
10 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
Q: What do you think they mean by ‘related to person 1?’ in this question?
R: The respondent said it was not clear to him what “친척관계 (translation of ‘relatedto’) meant. What he was not sure about was whether his immediate family memberscould be called ‘친척 (relative).’ He said he would mark on ‘No,’ if his youngest sis-ter happened to be Person 7 (when he put himself as person 1), because he regardedher as his own family, not ‘a relative.’
4.2.3. Temporal problems
Temporal problems occur when a respondent interprets the amount of time spent onan activity or certain period of time in the survey question in an unintended way.We identified only a few temporal problems in the English interviews and the Kor-ean interviews.
The cited question (Example 3) asks about the number of people living or stay-ing in the current residence on a certain day. The respondent ignored the informa-tion in the question text asking them to include the number of people living orstaying in the home who were present on 1 February 2008. Instead, he used ‘now,’(the time of the interview) as a reference period, which was past 1 February 2008.
Example 3
Q: How did you choose that answer? Who did you include?
R: The respondent wrote three, because this is the number of his family members,who are ‘living or staying at his house now.’ The respondent included himself (asPerson 1), his son (Person 2), and wife (Person 3). Later in the interview, it turned outthat his mother-in-law was staying with his family on 1 February 2008, but he hadexcluded her.
4.2.4. Computational problems
Computational problems involve respondent difficulty processing and manipulatinginformation included in a question. For example, a long and complicated question canmake it difficult for the respondent to parse the sentence and understand the meaningof the question. Problems involving mental arithmetic are also placed under this prob-lem category. Since Computational problems stem from their confusion regarding themeaning of the question, all of the respondents’ difficulties in understanding can becategorized as computational problems in a sense. In our analysis, we used Computa-tional problems as a residual category. That is, we tried to code responses to the spe-cific problem category first. Only problems that would not specifically fit into theother prominent categories were assigned as Computational problems.
The following question (Example 4) asks whether the respondent’s home isowned, rented, or occupied without payment of rent. Below, the respondent inter-
International Journal of Social Research Methodology 11
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
preted the question as asking home type. We believe this problem could be a resultof navigating the particular question format, which was different from the rest ofthe questions on the questionnaire. The lead-in text is designed to be completedwith one of the response choices, whereas in the other questions on the question-naire, the response choices do not require lead-in text and the question ends with aquestion mark as a complete sentence form.
Example 4
Q: What do you think they mean here?
R: The [respondent says that this] question asks what type of home that she currentlylives in (whether it is a house or an apartment). The respondent said the question shouldhave included ‘a condominium’ along with house, apartment, and mobile home.
4.2.5. Logical problems
Logical problems involve any logic embedded in the question. We identified a fewLogical problems from the English interviews and only one from the Korean inter-views.
The following question (Example 5) asks about additional people who are notincluded in the previous answer, which is asking for the number of people living orstaying at this residence on 1 February 2008. It intends to capture undercounting,for example, people who are missed when the number of household members isrecorded earlier on the questionnaire. When the term ‘foster children’ was probed,the respondent understood the meaning of foster children correctly. However, shethought the first response option was strange because she interpreted sons anddaughters (Korean translation for children) as biological sons and daughters only,and thought that foster children cannot be biological sons and daughters logically.
Example 5
12 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
Q: In your opinion, what are ‘foster children’ in Question 2?
R: While reading the first option, she commented this looked strange. The first optionwas currently translated to ‘신생아 또는 위탁아 등의 자녀 (son or daughter such asnew born babies or foster children.’ She said ‘위탁아는 자녀가 아닌데 (Fosterchildren can’t be daughters/sons …).’
4.2.6. Translation problems
Translation problems were identified only in the Korean interview data because theEnglish interview did not involve translation. Technically, Translation problems arenot equivalent to comprehension problems because respondents can clearly under-stand the translated terms literally as they are correctly presented in a question.However, poor translation prevents respondents from comprehending the intendedmeaning of the survey questions. Thus, we consider it as a Translation problemwhen the translation contains mistakes and does not convey the intended meaningof the source (English) language.
Most Korean respondents had problems understanding the intended meaning of‘foster children’ in the first response option of Question 2 (Example 5) above. Thisterm was translated to 위탁아, meaning ‘child who is consigned.’ When probedabout this term, many respondents interpreted it to mean a babysitting situation.The translation was inaccurate and too general to deliver the intended meaning.
Example 6
Q: What do you think they mean by “foster children”?
R: The respondent thought of cases when people asked their children to be taken careof during their vacation or absence.
5. Discussion
Differences in the ‘high-context’ and ‘low-context’ languages and the associatedcultures bring challenges to the cognitive interviews. More efforts to elicit enoughdetails and to clarify information may be necessary when conducting cognitiveinterviews in high-context languages, specifically Korean. Cultural expectations ofmaintaining harmonious relationships and using silence and acquiescence to indicatepoliteness may prevent Korean and other Asian respondents from providing directfeedback during the cognitive interviews. However, we found evidence supportingthe effectiveness of Korean cognitive interviews as a pretesting tool.
Our quantitative analysis demonstrates that the Korean cognitive interviewsidentified a significant number of questionnaire issues; 125 question problemsincluding 24 Translation problems. The percentages of identified problems betweenthe Korean and the English cognitive interviews were very similar (18.6% for Kor-ean vs. 19.3% for English). The comparison of the categorized survey problem dis-tribution between the Korean and the English cognitive interviews also exhibitedparallel findings; that is, the most often detected problems for both languages wereLexical problems, followed by Computational problems and Inclusion/exclusionproblems. Only a small number of problems were found under the Temporal prob-lem and Logical problem categories for both languages. This similarity in the gen-
International Journal of Social Research Methodology 13
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
eral problem distribution pattern re-emerged when the same analysis was conductedat each individual probe level. In other words, we found that a certain probedetected as problematic for English interviews tends to be detected as problematicfor Korean interviews as well. Our qualitative analysis of the Korean data alsoconfirmed that the information obtained during the interviews was adequate torevise the problematic survey questions.
These findings support the validity of conducting cognitive interviews in non-English languages in identifying survey question problems despite some challengesrooted at cross-cultural differences reported in literature. We believe usingexperienced cognitive interviewers who are bilingual and bicultural are crucial tothe success. Our paper also shows that the Korean cognitive interviews identified24 Translation problems, which make up about 19% of all the identified Koreansurvey problems. These Translation problems could not have been revealed if onlythe English cognitive interviews were conducted. This finding is particularly impor-tant because it assures the necessity of conducting cognitive interviews in a targetlanguage when a translated questionnaire is used for data collection.
There are several limitations to our findings. First, our analysis is based onsummary reports of the interviews, rather than on the transcripts. Thus, it was notfeasible to conduct a deeper analysis such as comparing the number of probes inboth languages that elicited appropriate answers. Although our research showed thatthe Korean cognitive interview results provided parallel findings to the Englishcognitive interviews, it is possible that interviewers of the Korean cognitive inter-views asked additional probes or even more direct probes to elicit qualifyinganswers to the original probes. Further research using transcripts would give usmore information for evaluating the effectiveness of cognitive interviewing inKorean and for conducting these interviews more effectively.
The second limitation is related to the Census questions that we investigated.These Census questions are not long or complex, and many do not involve complexresponse strategies. In fact, they could be regarded as relatively simple compared tosurvey questions in general. In addition, our protocol was geared toward translationbecause the primary purpose of the cognitive interviews was to discover potentialtranslation issues. As a result, our findings may be tilted toward Lexical problems,and we only identified one Logical problem and one Temporal problem from theKorean interviews. We are not sure if our findings would be applicable when pre-testing a set of long and complex questions.
Furthermore, the data we used for analysis were based on a convenience samplelimited to only several geographic areas in the USA, and we had to rely on volun-tary contacts from interested individuals. Because of these inherent limitations inthe data-set we used, it may be difficult to generalize the findings from these inter-views to the general public. However, this is a common limitation faced by all qual-itative studies. It is important to note that the English and Korean cognitiveinterviews on this study followed the same recruitment procedures, and we inter-viewed Korean respondents who demonstrate the characteristics of those who are inneed of a translated survey questionnaire according to the past ACS statistics. Thisis critical considering that our aim for conducting cognitive interviewing is toincrease survey participation by providing translated questionnaires to monolingual,non-English speakers.
14 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
6. Conclusion
This paper fills the gap of research in non-English language cognitive interviewingby showing empirical evidence to support the effectiveness of Korean cognitiveinterviews. The quantitative findings demonstrate that Korean cognitive interviewresults were comparable to those of the English cognitive interviews in terms of thenumber of survey problems identified and its distribution across the problem catego-ries. Qualitative findings suggest that the content of the Korean cognitive interviewsprovide enough details to revise the questionnaire.
Despite the challenges reported in prior research, our findings showed thatcognitive interviewing can be a valid tool for identifying problematic survey ques-tions in the Korean language and possibly in other high-context languages. Futureresearch examining other high-context languages, such as Chinese, would be usefulto support our findings and to further inform the cognitive interviewing methods fornon-English languages.
Notes on contributorsHyunjoo Park is a research survey methodologist in the Center for Survey Methodology atRTI International with interests in survey methods in cross-cultural contexts and pretesting.Her recent publications and presentations have focused on cognitive interviewing in non-English languages.
Mandy Sha is a survey methodologist in the Center for Survey Methodology at RTIInternational. Her areas of specialization include instrument design and evaluation, methodsto reach non-English speakers, and Census coverage issues.
Yuling Pan is a principal researcher/sociolinguist and group leader of Language andMeasurement Research Group at the Center for Survey Measurement of the U.S. CensusBureau. Her current research interests include survey translation, pretesting of translation inmultiple languages, linguistic politeness, and cross-cultural studies. She has publishedwidely, and has authored and co-authored four books and numerous journal articles in thefield of sociolinguistics, intercultural communication, and survey research.
ReferencesBerrigan, D., Forsyth, B., Helba, C., Levin, K., Norberg, A., & Willis, G. (2010). Cognitive
testing of physical activity and acculturation questions in recent and long-term Latinoimmigrants. BMC Public Health, 10, 481. Retrieved from http://www.biomedcentral.com/1471-2458/10/481
Bolton, R. N. (1993). Pretesting questionnaires: Content analyses of respondents’ concurrentverbal protocols. Marketing Science, 12, 280–303.
Chan, A. (2010). Analysis of Chinese speakers’ responses to survey intention questions. InProceedings of the Section on Survey Research Methods, Annual Meetings of theAmerican Statistical Association (pp. 1672–1686). Alexandria, VA: American StatisticalAssociation.
Conrad, F., & Blair, J. (1996). From impressions to data: Increasing the objectivity ofcognitive interviews. In Proceedings of the Section on Survey Research Methods, AnnualMeetings of the American Statistical Association (pp. 1–10). Alexandria, VA: AmericanStatistical Association.
Conrad, F., & Blair, J. (2009). Sources of error in cognitive interviews. Public OpinionQuarterly, 73, 32–55.
Coronado, I., & Earle, D. (2002). Effectiveness of the American Community survey of theUS Census in a borderlands colonial setting. Draft report submitted to the US CensusBureau.
International Journal of Social Research Methodology 15
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
Fitzgerald, R., Widdop, S., Gray, M., & Collins, D. (2011). Identifying sources of error incross-national questionnaires: Application of an error source typology to cognitiveinterview data. Journal of Official Statistics, 27, 569–599.
Hall, E. T., & Hall, M. R. (1987). Hidden differences: Doing business with the Japanese.New York, NY: Anchor Books/Doubleday.
Javeline, D. (1999). Response effects in polite cultures: A test of acquiescence inKazakhstan. Public Opinion Quarterly, 63, 1–28.
Kissam, E., Herrera, E., & Makamoto, J. (1993). Hispanic response to census enumerationforms and procedures. Contract report submitted to the US Census Bureau.
Kudela M. S., Forsyth B. H., Levin K., Lawrence D., & Willis G. (2006, May). Cognitiveinterviewing versus behavior coding. Paper presented at the annual conference ofAmerican Association for Public Opinion Research, Montreal, Canada.
Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition,emotion, and motivation. Psychological Review, 98, 224–253.
Masuda, T., & Nisbett, R. E. (2001). Attending holistically vs. analytically: Comparing thecontext sensitivity of Japanese and Americans. Journal of Personality and SocialPsychology, 81, 922–934.
Miller, K., Fitzgerald, R., Padilla, J. L., Willson, S., Widdop, S., Caspar, R., & Schoua-Glus-berg, A. (2011). Design and analysis of cognitive interviews for comparativemulti-national testing. Field Methods, 23, 379–396.
Pan, Y. (2004, May). Cognitive interviews in languages other than English: Methodologicaland research issues. Paper presented at the 59th annual conference of the AmericanAssociation for Public Opinion Research Conference, Phoenix, AZ.
Pan, Y., Landreth, L., Hinsdale, M., Park, H., & Schoua-Glusberg, A. (2010). Cognitiveinterviewing in non-English languages: A cross-cultural perspective. In J. A. Harkness,M. Braun, B. Edwards, T. P. Johnson, L. E. Lyberg, P. Mohler, B. Pennell, & T. W.Smith (Eds.), Survey methods in multinational, multiregional, and multicultural contexts(pp. 91–113). Berlin: Wiley Press.
Pasick, R. J., Stewart, S. L., Bird, J. A., & D’onofrio, C. N. (2001). Quality of data inmultiethnic health surveys. Public Health Reports, 116, 223–243.
Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce differentresults? Sociological Methodology, 24, 73–104.
Rothgeb, J., Willis, G., & Forsyth, B. (2005). Questionnaire pretesting methods: Do differenttechniques and different organizations produce similar results? Statistical research divi-sion research report series (Survey Methodology #2005-2). Washington, DC: US CensusBureau.
Warnecke, R. B., Johnson, T. P., Chavez, N., Sudman, S., O’Rourke, D. P., Lacey, L., &Horm, J. (1997). Improving question wording in surveys of culturally diverse popula-tions. Annual Epidemiology, 7, 334–342.
Willis, G., Lawrence, D., Thompson, F., Kudela, M., Levin, K., & Miller, K. (2005). Theuse of cognitive interviewing to evaluate translated survey questions: Lessons learned.Paper presented at the 2005 conference of the Federal Committee on Statistical Method-ology, Washington, DC.
Willis, G., Schechter, S., & Whitaker, K. (1999). A comparison of cognitive interviewing,expert review, and behavior coding: What do they tell us? In Proceedings of theSection on Survey Research Methods, Annual Meetings of the American StatisticalAssociation (pp. 28–37). Alexandria, VA: American Statistical Association.
Willis, G., & Zahnd, E. (2007). Questionnaire design from a cross cultural perspective: Anempirical investigation of Koreans and Non-Koreans. Journal of Health Care for thePoor and Underserved, 18, 197–217.
Yuan, M. Y., Wake, V., Park, H., & Nguyen, L. (2009, May). Conducting cognitiveinterviews with linguistically isolated Asian populations. Paper presented at InternationalField Directors and Technologies Conference, Delray Beach, FL.
16 H. Park et al.
Dow
nloa
ded
by [
Hyu
njoo
Par
k] a
t 18:
43 0
8 A
ugus
t 201
3
