Measuring Implicit Cognition in Qualtrics with iatgen: A Free, User-Friendly Tool for Building Survey-Based IATs

BackgroundThe Implicit Association Test (IAT; Greenwald et al., 1998) allows researchers to assess the strength of association between pairs of targets (e.g., Black vs. White) and attributes (e.g., Good vs. Bad). The IAT has seen widespread use, for example, to study brand preference, social attitudes, morality, self-esteem, and other social-cognitive constructs (e.g., Greenwald, Poehlman, Uhlmann, & Banaji, 2009). Although web-based research is gaining popularity (e.g., Qualtrics, MTurk, etc.; Paolacci & Chandler, 2014), there is no easy way to run IATs in web surveys. Commercial IAT software is expensive, cumbersome, and does not integrate well with mainstream survey software. In response, we created iatgen, a free, easy-to-use tool for building and analyzing IATs via common survey software. It provides full integration with Qualtrics, automates data cleaning and export, and is ideal for lab and online use.

iatgenIatgen consists of IAT building and analysis tools in both R package and web applet format (Figure1). These will be available for public use early 2017 (see our homepage www.iatgen.wordpress.com for updates).

How iatgen works: iatgen creates HTML and JavaScript files that are added into a special Qualtrics template to make a functional IAT. All stimuli are pre-loaded, so internet speed does not influence results. By default, iatgen counterbalances left/right starting configurations of targets and attributes and uses a 40-trial washout block to avoid order effects within the IAT (Nosek, Greenwald, & Banaji, 2005).

Building IATs: • In Shiny: The Shiny web applet automates IAT production (see Figure 1). Users configure the IAT via

the applet (image URLs can be included, but images must be hosted by the user, preferably in a Qualtrics account). Iatgen then provides a ready-to-run Qualtrics survey for download, import, and customization. The survey can then be customized like any other survey (e.g., allowing full use of survey flow). The shiny web applet offers all the features of the R package.

• In R: The R interface offers the same features but is configured via an R script. This generates the same fully functional survey as the web applet. However, it can be configured to generate custom HTML and JavaScript files which can be manually copy/pasted into a template (for multi-IAT designs).

Analyzing IATs: Iatgen processes the data via either the web applet or data-cleaning scripts in R, providing diagnostics such as reliability, error rates, drop counts, data reduction and D-scoring (Greenwald et al. 2003), and data export. The R package offers easy flexibility, as one can proceed to do all analyses in R.

Validation Study 1Overview: Study 1 tested iatgen in a large, online study of prejudice, demonstrating reliability and validity. Our study built upon Gervais, Shariff, and Norenzayan (2011), who observed a correlation between implicit distrust of Atheists and global religiosity in a small, in-person sample.

• Participants: Participants were 239 students at a public university in the Northeastern United States, participating in exchange for partial course credit.

• Procedure: Participants completed an Atheist (atheist, non-believer) / Theist (believer, religious) and Trust (truthful, credible, dependable, honest, upright, trustworthy) / Distrust (sneaky, lying, devious, dishonest, deceitful, shady) IAT. Participants completed a number of explicit attitude measures as well. (see below; contact for full measures).

• Reliability: The IAT demonstrated good internal reliability, .84 (De Houwer & De Bruycker, 2007 method).

• Validity: Implicit trust was associated with less negative explicit attitudes towards atheists (r = -.27, p < .001), more positive attitudes towards atheists (r = .37, p < .001), more frequent intergroup contact with atheists (r = .17, p = .01), and less intergroup anxiety towards atheists (r = .19, p = .003). Implicit trust was also associated with lower levels of religiousness (r = -.29 , p < .001) and religious fundamentalism (r = -.20, p = .002).

• Discussion: The Qualtrics IAT seemed both reliable and valid in an online student sample. Building upon Gervais et al. (2011), the Atheist/Trust IAT showed both reliability and validity, and atheist trust was associated with a range of social/attitudinal variables, consistent with levels observed elsewhere in the literature (Hofmann et al., 2005).

Thomas Carpenter1, Chris P. Pullig2,Ruth Pogacar3, Michal Kouril4,

Jordan LaBouff5 , Naomi Isenberg1, & Alek Chakroff6

1 Seattle Pacific University, Seattle, WA2 Baylor University, Waco, TX

3 University of Cincinnati, Cincinnati, OH4 Cincinnati Children’s Hospital Medical Center, Cincinnati, OH

5 University of Maine, Orono, ME6 Harvard University, Cambridge, MA

Validation Study 2 – MethodOverview: The goal of Study 2 was to validate iatgen for use on MTurk using a large, online sample. We also sought to show incremental validity, predicting a behavioral outcome above and beyond explicit measures. This is seldom found in IAT studies, often because of limited sample size. Given the ability to collect large samples online, we re-assessed this in the present study. Finally, we considered whether an iatgen IAT could detect implicit associations with the fidelity to observe moderation by choice speed (Frieseand colleagues have found that the IAT tends to predict better for rapid choices).

• Participants: Participants were 451 MTurk participants (245 male; ages 19-70; M = 34.86, SD = 10.20) from 48 states (including the District of Columbia). Participants were paid $1.50 for their time.

• Procedure: • Implicit measure: Participants completed a brand (Scope images; Listerine images) and preference

(good: good, excellent, superb, best, wonderful; bad: bad, terrible, awful, worst, horrible) IAT. The IAT had good reliability, .82.

• Explicit measure: Participants were asked to imagine shopping for mouthwash and to state their preference on a 7-point scale. In addition, participants were asked “In general, do you view Scope or Listerine more positively?” and “Overall, which do you think is the better mouthwash?” These were combined to create a composite measure of explicit preference, α = .95. Either this or the IAT were presented first (counterbalanced).

• Choice: at the end, participants were instructed to imagine they could take home a bottle of mouthwash. Participants were then shown two mouthwash bottles on the screen and instructed to click on the mouthwash they would choose.

Conclusions§ The Qualtrics IAT exhibited sound psychometrics (e.g., reliability) and correlations consistent with

effect sizes reported in meta-analyses (Hofmann et al., 2005). This suggests the Qualtrics IAT is both reliable and valid, even in online pools such as MTurk.

§ Iatgen offers advantages over other IAT tools:§ It is free, flexible, and provides full integration into Qualtrics.

§ It makes IATs easy to run with online subject pools (e.g., MTurk, university pools), making the rapid collection of large samples feasible.

§ Building custom IATs takes just a few minutes. Data analysis is largely automated and also takes only a few minutes.

§ The entire study (IAT, manipulations, explicit measures, etc.) runs within one survey, reducing attrition/error risk, and increases flexibility for randomization, multi-IAT designs, etc. This avoids the challenges that come with coordinating IAT and survey tools.

§ Researchers have total control over study design and implementation. The IAT can be positioned anywhere within the study (or placed randomly within the study), just as with any other survey element.

§ Note: iatgen is still in development (public release early 2017). Interested users should contact the first author at tcarpenter@spu.edu or check out www.iatgen.wordpress.com. Full study measures and details can be obtained by contacting tcarpenter@spu.edu.

Figure 1. Left: text and image IATs running in Qualtrics built using iatgen. Center: an screenshot of the user interface for building IATs. Right: Building an IAT using an R build script. IATs built using iatgen are highly customizable, can use words and/or images, have support for custom text colors, support multiple error-handling methods, and includes a full suite of analysis tools. See www.iatgen.wordpress.com.

Presented at the Society for Personality and Social Psychology (2017) Annual Meeting in San Antonio, Texas

Validation Study 2 – Key Results• The IAT strongly predicted brand choice: A logistic regression (note: for ease of interpretation, IAT

scores were z-scored prior to regression analyses) revealed the IAT was predictive of brand choice in a bivariate analysis, OR = 2.77, p < .001, pseudo R2 = .23. IAT scores also correlated with explicit preferences, r = .42, p < .001.

• The IAT uniquely predicted brand choice: even after controlling for explicitly stated preferences, the IAT continued to predict brand choice, OR = 1.55, p = .02. A comparison of models with / without the IAT indicated the IAT explained unique variance in choice, 𝚫R2 = .06.

• The IAT was highly uniquely predictive of fast choices but not very predictive of slow choices. The IAT interacted with the participants’ natural choice speed to predict choice (interaction p = .001, 𝚫R2 = .02). The interaction is visualized at left. The IAT is moderately more predictive of choices made at the mean choice speed (3.95 s) but is considerably more predictive of choices made 1 SD faster than the mean (i.e., 1.67 s) and considerably less predictive of choices made 1 SD slower than the mean (i.e., 6.22 s).

• Discussion: The IAT again exhibited reliability and convergent and incremental validity, even in an MTurk environment. The moderation findings suggest the IAT successfully captured impulsive components of participant preferences.