Methodological and Ethical Issues in Conducting Social ...psych.fullerton.edu/mbirnbaum/papers/Ch16birnbaum.pdf16 Methodological and Ethical Issues in Conducting Social Psychology

16Methodological and EthicalIssues in Conducting SocialPsychology Research via the Internet

MICHAEL H. BIRNBAUM

California State University, Fullerton

When I set out to recruit highlyeducated people with specializedtraining in decision making, I

anticipated that it would be a difficult project.The reason I wanted to study this group wasthat I had been obtaining some very startlingresults in decision-making experiments withundergraduates (Birnbaum & Navarrete,1998; Birnbaum, Patton, & Lott, 1999).Undergraduates were systematically violatingstochastic dominance, a principle that wasconsidered both rational and descriptive,according to cumulative prospect theory andrank dependent expected utility theory, themost widely accepted theories of decisionmaking at the time (Luce & Fishburn, 1991;Quiggin, 1993; Tversky & Kahneman,1992). These theories were recognized in the2002 Nobel Prize in Economics shared by

Daniel Kahneman, so these systematic viola-tions required substantial changes in thinkingabout decision making.

The results were not totally unexpected, forthey had been predicted by my configuralweight models of decision making (Birnbaum,1997). Nevertheless, I anticipated the chal-lenge that my results might apply only topeople who lack the education required tounderstand the task. I wanted to see if theresults I obtained with undergraduates wouldhold up with people highly trained in decisionmaking, who do not want to be caught behav-ing badly with respect to rational principles.

From my previous experiences in researchwith special populations, I was aware of thedifficulties of such research. In previouswork, my students and I had printed andmailed materials to targeted participants

C H A P T E R

359

AUTHOR’S NOTE: This research was supported by grants from the National Science Foundation to California StateUniversity, Fullerton, SBR-9410572, SES 99-86436, and BCS-0129453.

16-Sansone.qxd 6/14/03 2:04 PM Page 359

(Birnbaum & Hynan, 1986; Birnbaum &Stegner, 1981). Each packet contained a self-addressed and stamped envelope as well asthe printed materials and cover letter. Wethen sent, by mail, reminders with duplicatepackets containing additional postage outand back. As packets were returned, wecoded and entered data, and then verified thedata. All in all, the process was slow, labor-intensive, expensive, and difficult.

I had become aware of the (then) newmethod for collecting data via the Web(HTML forms), and I decided to try thatapproach, which I thought might be moreefficient than previous methods. I knew thatmy targeted population (university profes-sors in decision making) could be reached bye-mail and would be interested in the project.I thought that they might be willing to clicka link to visit the study and that it wouldbe convenient for them to do online. I wasoptimistic, but I was unprepared for howsuccessful the method would prove.

Within 2 days of announcing the study, Ihad more than 150 data records ready to ana-lyze, most from people in my targeted group. Afew days later, I was receiving data from grad-uate students of these professors, then fromundergraduates working in the same labs, fol-lowed by data from people all over the worldat all hours of the day and night. Before thestudy ended, I had data from more than 1,200people in 44 nations (Birnbaum, 1999b).Although the results did show that the rate ofviolation of stochastic dominance varied withgender and education, even the most highlyeducated participants had substantial rates ofviolation, and the same conclusions regardingpsychological processes were implied by eachstratum of the sample. That research led to aseries of new studies via the Web, to test newhypotheses and conjectures regarding pro-cesses of decision making (Birnbaum, 2000a,2001b; Birnbaum & Martin, in press).

My success with the method encouragedme to study how others were using the Web in

their research, and to explore systematicallythe applicability of Web research to a varietyof research questions (Birnbaum, 1999b,2000b, 2000c, 2001a, 2002; Birnbaum &Wakcher, 2002). The purpose of this chapteris to review some of the conclusions I havereached regarding methodological and ethicalissues in this new approach to psychologicalresearch.

The American Psychological Society pageof “Psychological Research on the Net,”maintained by John Krantz, is a good sourcefor review of Web experiments and surveys(http://psych.hanover.edu/research/exponnet.html). In 1998, this site listed 35 studies; by1999, the figure had grown to 65; as ofDecember 9, 2002, there were 144 listings,including 43 in social psychology. Althoughnot all Web studies are listed in this site,these figures serve to illustrate that use of themethod is still expanding rapidly.

The Internet is a new medium of commu-nication, and as such it may create new typesof social relationships, communication styles,and social behaviors. Social psychology maycontribute to understanding characteristicsand dynamics of Internet use. There are nowseveral reviews of the psychology of theInternet, a topic that will not be treated inthis chapter (see Joinson, 2002; McKenna &Bargh, 2000; Wallace, 2001). Instead, thischapter reviews the critical methodologicaland ethical issues in this new approach topsychological research.

MINIMUM REQUIREMENTS FORONLINE EXPERIMENTING

The most elementary type of Internet studywas the e-mail survey, in which investigatorssent text to a list of recipients and requestedthat participants fill in their answers andsend responses by return email. This type ofresearch saved paper and mailing costs;however, it did not allow easy coding and

EMERGING INTERDISCIPLINARY APPROACHES360


construction of data files, it did not allow thepossibility of anonymous participation, andit could not work for those without e-mail. Italso annoyed people by clogging their e-mailfolder with unsolicited (hence suspicious)material.

Since 1995, a superior technique,HyperText Markup Language (HTML)forms, has become available. This methoduses the World Wide Web (WWW) ratherthan the e-mail system. HTML forms allowone to post a Web page of HTML, whichautomatically codes the data and sends themto a CGI (Common Gateway Interface) script,which saves the data to the server. Thismethod avoids clogging up mailboxes withlong surveys, does not require e-mail, and canautomatically produce a coded data file readyto analyze.

Anything that can be done with paper andpencil and fixed media (graphics, pho-tographs, sound, or video) can be done instraight HTML. For such research, all oneneeds are a Web site to host the file, a way torecruit participants to that URL, and a way tosave the data.

To get started with the technique, it ispossible to use one of my programs, (e.g.,SurveyWiz or FactorWiz), which are availablefrom the following URL: http://psych.fullerton.edu/mbirnbaum/programs/

These free programs allow a person to cre-ate a Web page that controls a simple survey orfactorial study without really knowing HTML(Birnbaum, 2000c). Readers are welcome touse these programs to create practice studies,and even to collect pilot data with nonsensitivecontent. The default CGI included savesthe data to my server, from which you candownload your data.

To go beyond the minimal requirements—for example, to learn how to install serversoftware on your lab’s computer (Schmidt,Hoffman, & MacDonald, 1997), adddynamic functionality (present content thatdepends on the participant’s behavior), or

measure response times (Birnbaum, 2000b,2001a; McGraw, Tew, & Williams, 2000b)—see the resources at the following URLs: http://ati.fullerton.edu and http://psych.fullerton.edu/mbirnbaum/www/links.htm

POTENTIAL ADVANTAGES OFRESEARCH VIA THE WWW

Some of the advantages of the new methodsare that one can now recruit participants fromthe entire world and test them remotely. Onecan test people without requiring them toreveal their identities or to be tested in thepresence of others. Studies can be run withoutthe need for large laboratories, without expen-sive dedicated equipment, and without limita-tions of time and place. Online studies runaround the clock anywhere that a computer isconnected to the WWW. Studies can be run bymeans of the (now) familiar browser interface,and participants can respond by (now) famil-iar methods of pointing and clicking or fillingin text fields.

Once the programming is perfected, dataare automatically coded and saved by a server,sparing the experimenter (and his or her assis-tants) from much tedious labor. Scientificcommunication is facilitated by the fact thatother scientists can examine and replicateone’s experiments precisely. The greatestadvantages probably are the convenience andease with which special samples can berecruited, the low cost and ease of testing largenumbers of participants, and the ability to doin weeks what used to take months or years toaccomplish.

On the other hand, these new methodshave limitations. For example, it is not yetpossible to deliver stimuli that can be touched,smelled, or tasted. One cannot deliver drugsor shocks, nor can one yet measure GalvanicSkin Response, PET scans, or heart rates. It isnot really possible to control or manipulatethe physical presence of other people. One can

Conducting Social Psychology Research via the Internet 361


offer real or simulated people at the other endof a network, but such manipulations may bequite different from the effects of real personalpresence (Wallace, 2001).

When conducting research via the Web,certain issues of sampling, control, andprecision must be considered. Are peoplerecruited via the Web more or less “typical”than those recruited by other means? Howdo we know if their answers are honest andaccurate? How do we know what the preciseconditions were when the person was beingtested? Are there special ethical considera-tions in testing via the Web?

Some of these issues were addressed in“early” works by Batinic (1997), Birnbaum(1999a, 1999b), Buchanan and Smith (1999),Pettit (1999), Piper (1998), Reips (1997),Schmidt (1997a, 1997b), Schmidt, Hoffman,et al. (1997), Smith and Leigh (1997), Sternand Faber (1997), and Welch and Krantz(1996). Over the last few years, there has beena great expansion in the use of the WWW indata collection, and much has been learned orworked out. Summaries of more recent workare available in a number of recent books andreviews (Birnbaum, 2000b, 2001a; Janetzco,Meyer, & Hildebrand, 2002; Reips, 2001a,2001b, 2002). This chapter will summarizeand contribute to this growing body of work.

EXAMPLES OF RECRUITINGAND TESTING PARTICIPANTSVIA THE INTERNET

It is useful to distinguish the use of the WWWto recruit participants from its use to testparticipants. The cases of four hypotheticalresearchers will help to illustrate these dis-tinctions. One investigator might recruit par-ticipants via the Internet and then send themprinted or other test materials by mail,whereas another might recruit participantsfrom the local “subject pool” and test themvia the Internet. A third investigator might

both recruit and test via the Web, and afourth might use the WWW to recruit crite-rion groups in order to investigate individualdifferences.

In the first example, consider the situationof an experimenter who wants to study thesense of smell among people with a very rarecondition. It is not yet possible to deliverodors via the WWW, so this researcher plansto use the WWW strictly for recruitment ofpeople with the rare condition. Organizationsof people with rare conditions often maintainWeb sites, electronic newsletters, bulletinboards, and other means of communication.Such organizations might be asked to contactpeople around the world with the targeted,rare condition. If an organization considersresearch to be relevant and valuable to theconcerns of its members, the officers of theorganization may offer a great deal of help inrecruiting its members. Once recruited, partic-ipants might be brought to a lab for testing,visited by teams traveling in the field, ormailed test kits that the participants wouldadminister themselves or take to their localmedical labs.

The researcher who plans to recruit peoplewith special rare conditions can be contrastedwith the case of a professor who plans to testlarge numbers of undergraduates by means ofquestionnaires. Perhaps this experimenterpreviously collected such data by paper-and-pencil questionnaires administered to partici-pants in the lab. Questionnaires were typed,printed, and presented to participants by paidresearch assistants, who supervised testingsessions. Data were coded by hand, typedinto the computer, and verified.

In this second example, the professor plansto continue testing participants from the uni-versity’s “subject pool.” However, instead ofscheduling testing at particular times andplaces, this researcher would like to allow par-ticipation from home, dorm room, or wher-ever students have Web access. The advantagesof the new procedure are largely convenience



to both experimenter and participants, alongwith cost savings for paper, dedicated space,and assistants for in-lab testing, data coding,and data entry. This hypothetical professormight even conduct an experiment, randomlyassigning undergraduates to either lab or Web,to ascertain if these methods make a difference.

In the third example, which describes myinitial interest in the method, the experi-menter plans to compare data from under-graduates tested in the lab against specialpopulations, who could be recruited andtested via the Web. This experimenter estab-lishes two Web pages, one of which collectsdata from those tested in the lab. The secondWeb page contains the same content butreceives data from people tested via theWWW. The purpose of this research wouldbe to ascertain whether results observed withcollege students are also obtained in othergroups of people who can be reached via theWeb (Birnbaum, 1999b, 2000a, 2001a,2001b; Krantz & Dalal, 2000).

In the fourth example, a researcher com-pares criterion groups, in order to calibrate atest of individual differences or to examine ifa certain result interacts with individualdifferences (Buchanan, 2000). Separate Websites are established that contain the samematerials, but people are recruited to theseURLs by methods intended to reach membersof distinct criterion groups (Schillewaert,Langerak, & Duhamel, 1998). One variationof this type of research is cross-culturalresearch, in which people who participate fromdifferent countries are compared (Pagani &Lombardi, 2000).

Recruitment Method andSample Characteristics

Although the method of recruitment andthe method of testing in either Web or lab areindependent issues, most of the early Webstudies recruited their participants from theWeb and recruited their lab samples from use

the university’s “subject pool” of students.Therefore, many of the early studies comparedtwo ways of recruiting and testing participants.

Because the “subject pool” is finite, atsome universities there is competition for thelimited number of subject-hours available forresearch. In addition, the lab method usuallyis more costly and time-consuming persubject compared to Web studies, wherethere is no additional cost once the study ison the WWW. For these reasons, Webstudies usually have much larger numbers ofparticipants than do lab studies.

Participation in psychological research isconsidered an important experience forstudents of psychology. Web studies allowstudents to be able to participate in psycho-logical studies even if they attend universitiesthat do little research or if they engage in“distance” learning (taking courses via TV orthe Web).

Table 16.1 lists a number of characteris-tics that have been examined in comparisonsof participants recruited from subject poolsand via search engines, e-mail announce-ments, and links on the Web. I will compare“subject pool” against Web recruitment,with arguments why or where one methodmight have an advantage over the other.

Demographics

College students who elect to take psychol-ogy courses are not a random sample of thepopulation. Almost all have graduated highschool, and virtually none have collegedegrees. At many universities, the majority ofthis pool is female. At my university, morethan two thirds are female, most are between18 and 20 years of age, and none of them hasgraduated college. Within a given university,despite efforts to encourage diversity, thestudent body is often homogeneous in age,race, religion, and social class. Although manystudents work, most lower-division studentsare supported by their parents, and the



positions they hold typically are low-paying,part-time jobs.

Those who choose to take psychologyare not a random sample of college students;at my university, they are less likely to bemajors in engineering, chemistry, mathemat-ics, or other “hard” subjects than to be unde-clared or majoring in “soft” subjects. This isa very specialized group of people.

There are three arguments in favor of stick-ing with the “subject pool.” The first is thatthis pool is familiar to most psychologists, somost reviewers would not be suspicious of astudy for that reason. When one uses a samplethat is similar to those used by others, onehopes that different investigators at similaruniversities will find similar results. Second, itis often assumed that processes studied arecharacteristic of all people, not just students.Third, the university student pool consists of ahomogeneous group. Because the studentbody lacks diversity, one expects that the vari-ability of the data resulting from individualdifferences on these characteristics will be

small. This homogeneity should afford greaterpower compared with studies with equal-sizedsamples that are more diverse.

Web Participants DoNot Represent a Population

When one recruits from the Web, themethod of recruitment will affect the natureof the sample obtained. Even with methodsintended to target certain groups, one cannotcontrol completely the nature of the sampleobtained via the Web. For example, otherWebmasters may place Web links to a studythat attract people who are different from theones the experimenter wanted to recruit. Astudy designed to assess alcohol use in thegeneral population, for example, might beaffected by links to the study posted in self-help groups for alcoholics. Or links might beposted in an upscale wine tasting club, whichmight recruit a very different sample. Thereare techniques for checking what link on theWeb led each participant to the site, so one


Table 16.1 Characteristics of Web-Recruited Samples and College Student “Subject Pool”Participants

Recruitment Method

Characteristic Subject Pool Web

Sample sizes Small Larger

Cross-cultural studies Difficult Easier

Recruitment of rare populations Difficult Easier

Sample demographics Homogeneous Heterogeneous

Age 18-22 18-80

Nationality/culture Relatively homogeneous Heterogeneous

Education 12-14 years 8-20 years

Occupation Students, part-time work Various, more full-time

Religion Often homogeneous More varied

Race Depends on university More varied

Self-selection Psychology students Unknown factors; dependson recruitment method

Gender Mostly female More equal, depending onrecruitment method


can study how the people got there (Schmidt,1997a), but at the end of the day one mustconcede that the people recruited are not arandom sample of any particular population.

I think it would be a mistake to treat sam-ples recruited from the Web as if they repre-sented some stable population of “Webusers.” First, the populations of those whohave access to the Web and those who usethe Web are (both) rapidly changing. Thosewith access could potentially get on the Web;however, the population of people who usethe Web at any given time is a nonrandomsample of those with access. Second, thosewho receive the invitation to participate arenot a random sample of all Web users. Third,those who agree to complete a study are nota random sample of those who see the invi-tation. Fourth, there usually is no real con-trol of who will receive the invitation, once ithas been placed in a public file on the Web.

Researchers who have compared datafrom the Web against data from studentshave reported that participants recruited viathe Web are on average older, better edu-cated, more likely male (though females maystill be in the majority), and more likelyemployed in full-time jobs. Layered overthese average differences, most studies alsoreported that the variance, or diversity, of thesamples is much greater on the Web on allcharacteristics than one usually finds in thesubject pool. Thus, although one finds thatthe average years of education of a Websample is greater than the average number forcollege sophomores, one also finds adults viathe Web who have no high school diploma,which one does not find in college samples.

Effect of Diversityon Power and Generality

The theoretical problem of diversity is thatit may produce greater error variance andtherefore reduce power compared to a studyin which participants are homogeneous.

However, in practice, the greater diversity inWeb samples may actually be a benefit forthree reasons.

First, demographic characteristics rarelyhave shown large correlations with manydependent variables, so diversity of demo-graphics does not necessarily generate muchadded error variance. Second, sample sizes inWeb studies usually outweigh any added vari-ance resulting from heterogeneity, so it is usu-ally the Web studies that have greater power(Musch & Reips, 2000; Krantz & Dalal,2000). Third, with very large samples, onecan partition the data on various characteris-tics and conduct meaningful analyses withineach stratum of the sample (Birnbaum,1999b, 2001a). When similar results arefound with males and females, with youngand old, with rich and poor, with experts andnovices, and with participants of manynations, the confidence that the findings willgeneralize to other groups is increased.

In the case that systematically differentresults are obtained in different strata, thesecan be documented and an explanationsought. If results do correlate with measures ofindividual differences, these can be betterstudied in samples with greater variance. Websamples are likely better for studying suchcorrelations precisely because they do havegreater variation. For example, if a personwanted to study the correlates of education, a“subject pool” sample would not have enoughvariance on education to make the studyworthwhile, whereas a Web-recruited samplewould have great variance on this variable.

Cases in which demographic characteristicshave been shown to correlate with the depen-dent variable include surveys intended to pre-dict the proportion who will vote Republicanor Democratic in the next election. Here theinterest is not in examining correlations but inforecasting the election. Neither surveys ofcollege students nor large convenience samplesobtained from volunteers via the WWWwould likely yield accurate predictions of the



vote (Dillman & Bowker, 2001). It remains tobe seen how well one might do by statistically“correcting” Web data based on a theory ofthe demographic correlates. For example, ifresults depend on gender and education, onemight try to weight cases so that the Web sam-ple had the same gender and education profileas those who vote. It remains to be seen howwell one might do with this approach.

The failure of the famous 1936 LiteraryDigest Poll, which incorrectly predicted that AlfLandon would defeat Franklin D. Rooseveltfor president, is a classic example of how a self-selected sample (even with very large samplesize) can yield erroneous results (Huff, 1954).Bailey, Foote, and Throckmorton (2000)reviewed this topic in the area of sex surveys. Inaddition to sampling issues, these authors alsodiscussed the conjecture that people might beless biased when answering a questionnaire viathe Internet than they would when respondingin person.

Some survey researchers use the Web tocollect data from what they believe is a rep-resentative sample, even if it is not random.These researchers establish a sample withproportions of various characteristics (gender,age, education) that match those in thegeneral population, much like the NielsenFamilies for TV ratings, and then return tothis group again and again for differentresearch questions. See the URL http://www.nielsenmedia.com/whatratingsmean/

Baron and Siepmann (2000) described avariation of this approach, in which a fixedlist (of 600 selected volunteers) is used instudy after study. A similar approach hasbeen adopted by several commercial pollingorganizations that have established fixedsamples that they consider to be representa-tive. They sell polling services in their fixedsample for a price.

No method yet devised uses true randomsampling. Random digit dialing, which is stillpopular among survey researchers, has twoserious drawbacks: First, some people have

more phone numbers than others, so themethod is biased to obtain relatively more ofsuch people. Second, not everyone telephonedagrees to participate. I am so annoyed bycalls at home, day and night, from salespeoplewho pretend to be conducting surveys thatI no longer accept such calls. Even thoughthe dialing may be random, therefore, thesample is not.

Because no survey method in actual useemploys true random sampling, most ofthese arguments about sampling methods are“armchair” disputes that remain unresolvedby empirical evidence. Because experimentsin psychology do not employ randomsampling of either participants or situations,the basis for generalization from experimentsmust be based on proper substantive theory,rather than on the statistical theory ofrandom sampling.

EXPERIMENTAL CONTROL,MEASUREMENT, ANDOBSERVATION

Table 16.2 lists a number of characteristicsof experiments that may differ between laband Web studies. Some of these distinctionsare advantages or disadvantages of onemethod or another. The issues that seemmost important are experimental control ofconditions, precision of stimulus presenta-tion, observation of behavior, and the accu-racy of measurement.

Two Proceduresfor Holding an Exam

To understand the issue of control, con-sider the case of a professor who plans togive an exam intended to measure whatstudents learned in a course. The exam is tobe held with closed books and closed notes,and each student is to take the exam in afixed period of time, without the use of



computers, calculators, or help from others.The professor might give the exam in a class-room with a proctor present to make surethat these rules are followed, or the professormight give the exam via the Internet.

Via the Internet, one could ask thestudents if they used a computer or got helpfrom others, for example, but one cannot besure what the students actually did (or evenwho was taking the exam) with the samedegree of certainty as is possible in the class-room. This example should make clear thelack of control in studies done via the WWW.

Precision of Manipulationsand Measurements

In the lab, one can control the settings ona monitor, control the sound level on speak-ers or headphones, and control the noiselevel in the room. Via the Web, each user hasa different monitor, different settings, differ-ent speakers or earphones, and a differentbackground of noise and distraction.

In addition to better control of conditions,the lab also typically allows more precisemeasurement of the dependent variables aswell as affording actual observation of theparticipant.

The measurement devices in the labinclude apparatus not currently available viathe Web (e.g., EEG, fMRI, eye-trackers). Inaddition, measurement of dependent vari-ables such as response times via the WWWface a variety of problems resulting from thedifferent conditions experienced by differentparticipants. The participant via the WWW,after all, may be watching TV, may haveother programs running, may decide to dosome other task on the computer in the mid-dle of the session, or may be engaged in con-versation with a roommate. Such sources ofuncertainty concerning conditions can intro-duce both random and systematic error com-ponents compared to the situation in the lab.Krantz (2001) reviewed additional ways inwhich the presentation of stimuli via theWWW lacks the control and precision avail-able in the lab. Schmidt (2001) has reviewedthe accuracy of animation procedures.

At the user’s (i.e., participant’s) end, theremay be different types of computers, moni-tors, sound cards (including no sound), sys-tems, and browsers. It is important to checkthat the Web experiment works with themajor systems (e.g., Windows, Mac, Linux)and different browsers for those systems(Netscape Navigator, Internet Explorer, etc.).


Table 16.2 Comparisons of Typical Lab and WWW Experiments

Research Method

Comparison Lab Web

Control of conditions, Good control Less control; observationease of observation not possible

Measurement of behavior Precise Sometimes imprecise; pilot tests,lab vs. Web

Control of stimuli Precise Imprecise

Dropouts A serious problem A worse problem

Experimenter bias Can be serious Can be standardized

Motivation Complete an assignment Help out; interest; incentives

Multiple submission Not considered Not considered a problem


Different browsers may render the same pagewith different appearances, even on the samecomputer and system. Suppose the appearance(such as, for example, the spacing of a numer-ically coded series of radio buttons) is impor-tant and is displayed differently by differentbrowsers (Dillman & Bowker, 2001). If so, itmay be necessary to restrict which browsersparticipants can use, or at least keep track ofdata that are sent by different browsers.

The Need for PilotWork in the Lab

Because the ability to observe the partici-pant is limited in WWW research, one shoulddo pilot testing of each Web experiment inthe lab before launching it into cyberspace. Aspart of the pilot-testing phase, one shouldobserve people as they work through theonline materials and interview them after-ward to make sure that they understood thetask from the instructions in the Web page.

Pilot testing is part of the normal researchprocess in the lab, but it is even more impor-tant in Web research because of the difficultyof communicating with participants. Pilottesting is important to ensure that instruc-tions are clear and that the experiment worksproperly. In lab research, the participant canask questions and receive clarifications. Ifmany people ask the same question, the labassistant will become aware of the problem.In Web research, the experimenter needs toanticipate questions or problems that partic-ipants may have and to include instructionsor other methods for dealing with them.

I teach courses in which students learnhow to conduct their research via the Web.By watching participants in the lab workingon my students’ studies, I noticed that someparticipants click choices before viewing theinformation needed to make their decisions.Based on such observations, I advise mystudents to place response buttons below thematerial to be read rather than above it, so

that the participant at least has to scroll pastthe relevant material before responding. It isinteresting to rethink paper-and-pencil studies,for which the same problem may haveexisted but may have gone undetected.

A student of mine wanted to determinethe percentage of undergraduates who useillegal drugs. I recommended a variation ofthe random response method (Musch,Broeder, & Klauer, 2001), in order to protectparticipant anonymity. In the method I sug-gested, the participant was instructed toprivately toss two coins before answeringeach question. If both coins fell “heads,” theparticipant was to respond “yes” to the ques-tion; if both were “tails,” the participant wasto respond “no.” When the coins were mixed(one heads and one tails), the participant wasto respond with the truth. This procedureallows the experimenter to calculate the pro-portions in the population, without knowingfor any given participant whether the answerwas produced by chance or truth.

For example, one item asked, “Have youused marijuana in the last 24 hours?” Fromthe overall percentages, the experimentershould subtract 25% “yes” from the “yes”percentage (those who got two heads) andmultiply the difference by 2 (because only halfof the data are based on truth). For example,if the results showed that overall 30% said“yes,” then 30% – 25% = 5% (among the50% of those who had mixed coins andwho responded truthfully by saying “yes”); so,5% times 2 = 10%, the estimated percentagewho actually used marijuana during the last 24hours, assuming that the method works.

The elegance of this method is that no one(besides the participant) can ever know fromthe answers whether or not that person didor did not use marijuana, even if the partici-pant’s data were by identified by name!However, the method does require thatparticipants follow instructions.

I observed 15 pilot participants whocompleted my student’s questionnaire in the



laboratory. I noticed that only one participanttook out any coins during the session. Indeed,that one person asked, “It says here we aresupposed to toss coins; does that mean weshould toss coins?” Had this study simplybeen launched into cyberspace without pilottesting, we might never have realized thatmost people were not following instructions.

I suggested that my student add instruc-tions emphasizing the importance of follow-ing the procedure with the coins and that sheadd two questions to the end of the question-naire: “Did you actually toss the coins as youwere instructed?” and “If not, why not?”About half of the participants responded thatthey had not followed instructions, givingvarious reasons, including “lazy,” “I hadnothing to hide,” and “I don’t mind tellingthe truth.” Still, even among those who saidthey had followed instructions, one canworry that either by confusion or dissimula-tion, these students still had not followedinstructions. And those who say they fol-lowed instructions are certainly not a randomsample of all participants.

Clearly, in the lab, one could at least verifythat each student has coins, tosses the coins foreach question, and gives at least the superficialappearance of following instructions. In the lab,it might be possible to obtain blind urine sam-ples against which the results of the question-naire method could be compared. Questionsabout the procedure would be easy for the par-ticipant to ask and easy for the experimenter toanswer. Via the Web, we can instruct the par-ticipant and we can ask the participant if he orshe followed instructions, but we cannot reallyknow what the conditions were with the sameconfidence we have when we can observe andinteract with the participant.

The Need for Testingof HTML and Programming

One should also conduct tests of Webmaterials to make sure that all data coding

and recording are functioning properly.When teaching students about Web research,I find that students are eager to upload theirmaterials to the Web before they have reallytested them. It is important to conduct aseries of tests to make sure that each radiobutton functions correctly and that each pos-sible response is coded into the correct valuein the data file (see Birnbaum, 2001a,Chapters 11, 12, and 21).

My students tell me that their work hasbeen checked, yet when I check it, I detectmany errors that my students have not dis-covered on their own. It is possible to wastethe time of many people by putting uncheckedwork on the Web. In my opinion, such wasteof people’s time is unethical; it is a kind ofvandalism of scientific resources and a breachof trust with the participant. The participantsgive us their time and effort in the expectationthat we will do good research. They do notintend that their efforts will be wasted. Errorswill happen because people are human; know-ing this, we have a responsibility to checkthoroughly to ensure that the materials workproperly before launching a study.

Testing in Both Lab and Web

A number of studies have compared datacollected via the WWW with data collected inthe laboratory. Indeed, once an experiment hasbeen placed online, it is quite easy to collectdata in the laboratory, where the experi-menters can better control and observe theconditions in which the participants completedtheir tasks. The generalization from suchresearch is that despite some differences inresults, Web and lab research reach much thesame conclusions (Batinic, Reips, & Bosnjak,2002; Birnbaum, 1999b, 2000a, 2000b,2001a, 2002; Birnbaum & Wakcher, 2002;Krantz, Ballard, & Scher, 1997; Krantz &Dalal, 2000; McGraw, Tew, & Williams,2000a, 2000b). Indeed, in cognitive psychol-ogy, it is assumed that if one programs the



experiments correctly, Web and lab resultsshould reach the same conclusions (Francis,Neath, & Surprenant, 2000; McGrawet al., 2000a).

However, one should not expect Web andlab data to agree exactly, for a number of rea-sons. First, the typical Web and lab studiesdiffer from each other in many ways, andeach of these variables might make some dif-ference. Some of the differences are as follow.

Web versus lab studies often comparegroups of participants who differ in demo-graphic characteristics. If demographic char-acteristics affect the results, the comparisonof data will reflect these differences (e.g.,Birnbaum, 1999b, 2000a).

WWW participants may differ in motiva-tion (Birnbaum, 2001a; Reips, 2000). Thetypical college student usually participates asone option toward fulfilling an assignment ina lower-division psychology course. Studentslearn about research from their participation.Because at least one person is present, thestudent may feel some pressure to continueparticipation, even though the instructionssay that quitting is at any time permitted.Participants from the Web, however, oftensearch out the study on their own. They par-ticipate out of interest in the subject matter,or out of desire to contribute to scientificprogress. Reips (2000) argued that because ofthe self-selection and ease of withdrawingfrom an online study, the typical Web partic-ipant is more motivated than the typical labparticipant.

Analyses show that Web data can be ofhigher quality than lab data. For this reason,some consider Web studies to have an advan-tage over lab studies (Baron & Siepmann,2000; Birnbaum, 1999b; Reips, 2000). Onthe other hand, one might not be able to gen-eralize from the behavior of those who areinternally motivated to people who are lessmotivated.

When we compare Web and lab studies,there are often a number of other confounded

variables of procedure that might causesignificantly different results. Lab studies mayuse different procedures for displaying thestimuli and obtaining responses. Lab researchmay involve paper-and-pencil tasks, whereasWWW research uses a computer interface. Labstudies usually have at least one person present(the lab assistant), and perhaps many otherpeople (e.g., other participants). Lab researchmight use dedicated equipment, specializedcomputer methods, or manipulations, whereasWWW research typically uses the participant’sself-selected WWW browser as the interface.Doron Sonsino (personal communication,2002) is currently working on pure tests withrandom assignment of participants to condi-tions to examine the “pure” effect of the set ofWeb versus lab manipulations.

Dropouts andBetween-Subjects Designs

Missing data, produced by participantswho quit a study, can ruin an otherwise goodexperiment. During World War II, the Alliesexamined bullet holes in every bomber thatlanded in the United Kingdom after a bomb-ing raid. A probability distribution was con-structed, showing the distribution of bulletholes in these aircraft. The decision wasmade to add armor to those places wherethere were fewest bullet holes.

At first, the decision may seem in error,perhaps misguided by the gambler’s fallacy.To understand why the decision was correct,however, think of the missing data. Thedropouts in this research are the key to under-standing the analysis. Even though dropoutswere usually less than 7%, the dropouts arethe whole story. The missing data, of course,were the aircraft that were shot down. Theresearch was not intended to determine wherebullets hit aircraft, but rather to determinewhere bullet holes are in planes that return.Here, the correct decision was made to putextra armor around the pilot’s seat and to



strengthen the tail because few planes withdamage there made it home. This correctdecision was reached by having a theory of themissing data.

Between-subjects designs are trickyenough to interpret without having to worryabout dropouts. For example, Birnbaum(1999a) showed that in a between-subjectsdesign, with random assignment, the number9 is rated significantly “bigger” than thenumber 221. It is important to emphasizethat it was a between-subjects design, so nosubject judged both numbers. It can be mis-leading to compare data between subjectswithout a clear theory of the response scale.In this case, I used my knowledge of range-frequency theory (Parducci, 1995) to devisean experiment that would show a silly result.My purpose was to make people worry aboutall those other between-subjects studies thatused the same method to draw other dubiousconclusions.

In a between-subjects design, missing datacan easily lead to wrong conclusions. Evenwhen the dropout rate is the same in bothexperimental and control groups, and evenwhen the dependent variable is objective,missing data can cause the observed effect (ina true experiment with random assignmentto conditions) to show the opposite conclu-sion of the truth.

Birnbaum and Mellers (1989) illustratedone such case in which, even if a treatment isharmful, a plausible theory shows how onecan obtain equal dropouts and the harmfultreatment appears beneficial in the data. Allthat is needed is that the correlation betweendropping out and the dependent variable bemediated by some underlying construct. Forexample, suppose an SAT review course isharmful to all who take it; for example, sup-pose all students lose 10 points by taking thereview. This treatment will still look benefi-cial if the course includes giving each studenta sample SAT exam. Suppose those who dowell on the sample exam go on to take the

SAT and those who do poorly on the practicetest drop out. Even with equal dropout rates,the harmful SAT review will look beneficialbecause those who do complete the SAT willdo better in the treatment group than will thecontrol group.

In Web studies, people find it easy to dropout (Reips, 2000, 2001a, 2001b, 2002). Inwithin-subjects designs, the problem of attri-tion affects only external validity: Can theresults be generalized from those who finishto the sort of people who dropped out?

For between-subjects designs, however,attrition affects internal validity. When thereare dropouts in between-subjects designs, itis not possible to infer the true direction ofthe main effect from the observed effect.Think of the bullet holes case: Dropouts wereless than 7%, yet the true effect is oppositethe observed effect, because to protect peoplefrom bullets, places with the fewest bulletholes should get the most armor.

Because of the threat to internal validity ofmissing data, this topic has received someattention in the growing literature of Webexperimentation (Birnbaum, 2000b, 2001a;Frick, Bächtiger, & Reips, 2001; Reips, 2002;Reips & Bosnjak, 2001; Tuten, Urban, &Bosnjak, 2002). An idea showing somepromise is the use of the “high threshold”method to reduce dropouts (Reips, 2000,2002). The idea is to introduce manipulationsthat are likely to cause dropouts early in theexperimental sessions, before the randomassignment to conditions. For example, askpeople for their names and addresses first,then present them with a page that loadsslowly, then randomly assign those who areleft to the experimental conditions.

Experimenter Bias

A potential advantage of Web research isthe elimination of the research assistant.Besides the cost of paying assistants, assis-tants can bias the results. When assistants



understand the purpose of the study, theymight do things that bias the results in thedirection expected. There are many ways thata person can reinforce behaviors and interferewith objectivity, once that person knows theresearch hypothesis (Rosenthal, 1976, 1991).

A famous case of experimenter bias is thatof Gregor Mendel, the monk who publishedan obscure article on a genetic model of planthybrids that became a classic years after itwas published. After his paper was rediscov-ered, statisticians noticed that Mendel’s datafit his theory too well. I do not think thatMendel intentionally faked his data, but heprobably did see reasons why certain batchesthat deviated from the average were “spoiled”and should not be counted (see http://www.unb.ca/psychology/likely/evolution/mendel.htm). He might also have helped his theoryalong when he coded his data, countingmedium-sized plants as either “tall” or “short,”depending on the running count, to helpsupport his theory.

Such little biases as verbal and nonverbalcommunication, flexible procedures, datacoding, or data entry would be a big problemin certain areas of research, such as ESP or theevaluation of benefits of “talk” psychothera-pies, where motivation is great to find posi-tive effects, even if the effects are small. Thepotential advantage of Web experimentationis that the entry and coding of data are doneby the participant and computer, and noexperimenter is present to possibly bias thesubject or the data entry in a way that is notdocumented in the materials that control thestudy.

Multiple Submissions

One of the first questions any Webresearcher is asked is “How do you know thatsomeone has not sent thousands of copies ofthe same data to you?” This concern has beendiscussed in many papers (Birnbaum, 2001a;Reips, 2000; Schmidt, 1997a, 2000), and

the consensus of Web experimenters is thatmultiple submission of data has not been aserious problem (Musch & Reips, 2000).

When sample sizes are small, as they are inlab research, then if a student (perhaps moti-vated to get another hour of credit) partici-pated in an experiment twice, then thenumber of degrees of freedom for the errorterm is not really what the experimenterthinks it is. For example, if there were a dozenstudents in a lab study, and if one of themparticipated twice, then there were really only11 independent sources of error in the study.The consequence is that the statistical testsneed to be corrected for the multiple partici-pation by one person. (See Gonzalez & Griffin,Chapter 14, this volume.)

On the WWW, the sample sizes typicallyare so large that a statistical correctionwould be minuscule, unless someone partici-pated a very large number of times. Severalmethods have been proposed to deal withthis potential problem.

The first method to avoid multiple sub-mission is to analyze why people might par-ticipate more than once and then take stepsto remove those motivations. Perhaps theexperiment is interesting and people don’tknow they should participate only once. Inthat case, one can instruct participants thatthey should participate only once.

If the experiment is really enjoyable (e.g.,a video game), perhaps people will repeat thetask for fun. In that case, one could providean automatic link to a second Web site wherethose who have finished the experimentproper could visit and continue to play withthe materials as much as they like, withoutsending data to the real experiment.

If a monetary payment is to be given toeach participant, there might be a motive to bepaid again and again. Instructions might spec-ify that each person can be paid only once. Ifthe experiment offers a chance at a prize, theremight be a motive to participate more thanonce to give oneself more chances. Again,



instructions could specify that if a personparticipates more than once, only one chanceat the prize is given, or even that a person whosubmits multiple entries will be excluded fromany chance at a prize.

A second approach is to allow multipleparticipation but ask people how many timesthey have previously completed the study.The experimenter would then analyze thedata of those who have not previously par-ticipated separately from those who alreadyhave. This method also allows one to analyzehow experience in the task affects the results.

A third method is to detect and delete mul-tiple submissions. One technique is to exam-ine the Internet Protocol (IP) address of thecomputer that sent the data and deleterecords submitted from the same IP. (One caneasily sort by IP or use statistical software toconstruct a frequency distribution of IP.) TheIP does not uniquely identify a participantbecause most of the large Internet serviceproviders now use dynamic IP addresses andassign their clients one of their IPs as theybecome available. However, if one personsends multiple copies during one session onthe computer, the data would show up asrecords from the same IP. (Of course, whendata are collected in a lab, one expects thesame IP to show up again and again becausethat same lab computer is used repeatedly.)

A fourth method that is widely used is torequest identifying information from partici-pants. For example, in an experiment thatoffers payments or cash prizes, participantsare willing to supply identifying information(e.g., their names and addresses) that wouldbe used to mail payments or prizes. Otherexamples of identifying information are thelast four digits of a student’s nine-digit ID, e-mail address, a portion (e.g., last four digits)of the participant’s Social Security Number,or a password assigned to each person in aspecified list of participants.

A fifth method is to check for identicaldata records. If a study has a large number of

items, it is very unlikely that two records willhave exactly the same responses.

In my experience, multiple submissions areinfrequent and usually occur within a verybrief period of time. In most cases, the partic-ipant has apparently pushed the Submitbutton to send the data, read the thank youmessage or debriefing, and used the Back but-ton on the browser to go back to the study.Then, after visiting the study again, andperhaps completing the questionnaire, chang-ing a response, or adding a comment, theparticipant pushes the Submit button again,which sends another set of data. If theresponses change between submissions, theresearcher should have a clear policy of whatto do in such cases. In my decision-makingresearch, I want each person’s last, best, mostconsidered decision. It is not uncommon tosee two records that arrive within 2 minutesin which the first copy was incomplete andthe second copy is the same except that it hasresponses for previously omitted items.Therefore, I always take only the last submis-sion and delete any earlier ones from the sameperson. There might be other studies whereone would take only the first set of data anddelete any later ones sent after the person hasread the debriefing.

If this type of multiple submission is con-sidered a problem, it is possible to discourageit by an HTML or JavaScript routine thatcauses the Back button not to return to thestudy’s page. Cookies, in this case datastored on the participant’s computer indicat-ing that the study has already been com-pleted, could also be used for such a purpose.Other methods, using server-side program-ming, also are available (Schmidt, 2000);these can keep track of a participant andrefuse to accept multiple submissions.

I performed a careful analysis of 1,000 suc-cessive data records in decision making(Birnbaum, 2001b), where there were chancesat cash prizes and one can easily imagine amotive for multiple entries. Instructions stated



that only one entry per person was allowed. Ifound 5% were blank or incomplete (i.e., theyhad fewer than 15 of the 20 items completed)and 2% contained repeated e-mail addresses.In only one case did two submissions comefrom the same e-mail address more than min-utes apart. These came from one woman whoparticipated exactly twice, about a monthapart, and who interestingly agreed on 19 ofher 20 decisions. Less than 1% of remainingdata (excluding incomplete records and dupli-cate e-mail addresses) contained duplicate IPaddresses. In those cases, other identifiers indi-cated that these were from different peoplewho were assigned the same IPs, rather thanfrom the same person submitting twice. Reips(2000) found similar results, in an analysisdone in the days of mostly fixed IP addresses.

ETHICAL ISSUES IN WEB AND LAB

The institutional review of research shouldwork in much the same way for an onlinestudy as for an in-lab study. Research thatplaces people at more risk than the risks ofeveryday life should be reviewed to ensurethat adequate safeguards are provided to theparticipants. The purpose of the review is todetermine that the potential benefits of theresearch outweigh any risks to participants,and to ensure that participants will have beenclearly warned of any potential dangers andclearly accepted them. A comparison of ethi-cal issues as related to lab and Web is pre-sented in Table 16.3.

Risks of Psychological Experiments

For the most part, psychology experimentsare not dangerous. Undoubtedly, it is lessdangerous to participate in the typical 1-hourexperiment in psychology than to drive 1hour in traffic, spend 1 hour in a hospital,serve 1 hour in the armed forces, work 1 hourin a mine or factory, serve 1 hour on a jury,

or shop 1 hour at the mall. As safe aspsychology experiments are, those done viathe Internet must be even safer than labresearch because they can remove the greatestdangers of psychology experiments in the lab.

The most dangerous aspect of mostpsychology experiments is the trip to theexperiment. Travel, of course, is a danger ofeveryday life and is not really part of theexperiment; however, this danger shouldnot be underestimated. Every year, tens ofthousands are killed in the United States, andmillions are injured, in traffic accidents. Forexample, in 2001, the Department ofTransportation reported 42,116 fatalities inthe United States and more than 3 millioninjuries (see http://www.dot.gov/affairs/nhtsa5502.htm).

How many people are killed and injuredeach year in psychology experiments? I amnot aware of a single such case in the last 10years, nor am I aware of a single case in the10 years preceding the Institutional ReviewBoard (IRB) system of review. I suspect that ifthis number were large, professionals in theIRB industry would have made us all awareof them.

The second most dangerous aspect of psy-chology experiments is the fact that labsoften are in university buildings. Such build-ings are known to be more dangerous thanmost residences. For example, many build-ings of public universities in California werenot designed to satisfy building codesbecause the state of California previouslyexempted itself from its own building codesin order to save money.

An earthquake occurred at 4:31 a.m. onJanuary 17, 1994, in Northridge, California (seehttp: //www.eqe.com/publications/northridge/executiv.htm). Several structures of theCalifornia State University at Northridgefailed, including a fairly new building that col-lapsed (see http://www.eqe.com/publications/northridge/commerci.htm). Had the earthquakehappened during the day, there undoubtedly



would have been many deaths resulting fromthese failures (see http://geohazards.cr.usgs.gov/northridge/). Some of those killedmight have been participants in psychologyexperiments.

Certainly, the risks of such dangers astraffic accidents, earthquakes, communicablediseases, and terrorist attacks are not dangerscaused by psychology experiments. They arepart of everyday life. However, privatedwellings usually are safer from these dangersthan are public buildings, so by allowingpeople to serve in experiments from home,Web studies must be considered less risky thanlab studies.

Ease of Dropping OutFrom Online Research

Online experiments should be moreacceptable to IRBs than lab experiments foranother reason. It has been demonstrated thatif an experimenter simply tells a person to dosomething, most people will “follow orders,”even if the instruction is to give a potentiallylethal shock to another person (Milgram,

1974). Therefore, even though participantsare free to leave a lab study, the presence ofother people may cause people to continuewho might otherwise prefer to quit.However, Web studies do not usually haveother people present, so people tested via theWWW find it easy to drop out at any time,and many do.

Although most psychological research isinnocuous and therefore legally exempt fromreview, most psychological research isreviewed anyway, either by a campus-levelIRB or by a departmental IRB. It is interestingthat nations that do not conduct prior reviewof psychology experiments seem to have nomore deaths, injuries, property damage, orhurt feelings in their psychology studies thanwe have in the United States, where there isextensive review. Mueller and Furedy (2001)have questioned if the IRB system in theUnited States is doing more harm than good.The time and resources spent on the reviewprocess seems an expense that is unjustified byits meager benefits. We need a system thatquickly recognizes research that is exempt andidentifies it as such, in order to save valuable


Table 16.3 Ethical Considerations of Lab and Web Research

Experimental Setting

Ethical Issue Lab Web

Risk of death, injury, Trip to lab; unsafe public No trip to lab; can be doneor illness buildings; contagious disease from typically safer locations

Participant wants to quit Human presence may No human present; easy to quitinduce compliance

“Stealing” of ideas Review of submitted papers Greater exposure at earlier dateand grant applications

Deception Small numbers—concern is Large numbers—risk to sciencedamage to people and all society

Debriefing Can (almost) guarantee Cannot guarantee; free to leavebefore debriefing

Privacy and confidentiality Data on paper, presence of Insecure transmissionother people, burglars, data (e.g., e-mail); hackers,in proximity to participants burglars, increased distance


scientific resources. For more on IRB review,see Kimmel (Chapter 3, this volume).

Ethical IssuesPeculiar to the WWW

In addition to the usual ethical issues con-cerning the safety of participants, there areethical considerations in Web research con-nected with the impact of one’s actions onscience and other scientists. Everyone knowsa researcher who holds a grudge becausehe or she thinks that another person has“stolen” an idea revealed in a manuscript orgrant application during the review process.Very few people see manuscripts submittedto journals or granting agencies, certainlyfewer than would have access to a studyposted on the Web. Furthermore, a Webstudy shows what a person is working onwell ahead of the time that a manuscriptwould be under review. There undoubtedlywill be cases in which one researcher willaccuse another of stealing ideas from his orher Web site without giving proper credit.

There is a tradition of “sharing” on theWeb; people put information and resources onthe Web as a gift to the world. However, Webpages should be considered published, and oneshould acknowledge credit to Web sites in thesame way that one cites journal articles thathave contributed to one’s academic work.

Similarly, one should not interfere withanother scientist’s research. “Hacking” intoanother person’s online research would be ille-gal as well as improper, but even without hack-ing, a person can do many things that couldadversely affect another’s research program. Weshould proscribe interference of any kind withanother’s research, even if the person who takesthe action claims to act out of good motives.

Deception on the WWW

Deception concerning inducements toparticipate would be a serious breach of ethics

that not only would annoy participants butwould affect others’ research as well. Forexample, if a researcher promised to pay eachparticipant $200 for 4 hours and then failed topay the promised remuneration, such fraudnot only would be illegal but also would givea bad name to all psychological research.

Similarly, promising to keep sensitive dataconfidential and then revealing such personalinformation in the newspaper would be thekind of behavior we expect from a reporterin a nation with freedom of the press.Although we expect such behavior from areporter, and we protect it, we do not expectit from a scientist. Scientists should keep theirword, even when they act as reporters.

Other deceptions, such as were once pop-ular in social psychology, pose another trickyethical issue for Web-based research. If Webresearchers were to use deception, it is likelythat such deceptions would become thesource of Internet “flames,” public messagesexpressing anger and disapproval. Thedeception thus would become ineffective, butworse, people would soon come to doubtanything said by a psychologist. Such casescould easily give psychological research onthe Web a bad reputation. The potentialharm of deception to science (and thereforeto society as a whole) probably is greaterthan the potential harm of such deceptionto participants, who would be more likelyannoyed than harmed.

Deception in the lab may be discoveredand research might be compromised at asingle institution for a limited time among alimited number of people. However, deceptionon the Web could easily create a long-lastingbad reputation for all of psychology amongmillions of people. One of the problems ofdeceptive research is that the deception doesnot work—especially on the Web, it wouldbe easy to expose and publicize to a vastnumber of people. If scientists want truthfuldata from participants, it seems a poor ideato have a dishonest reputation.



Probably the best rule for psychologicalexperiments on the WWW is that false infor-mation should not be presented via the Web.

I do not consider it to be deception for anexperimenter to describe an experiment ingeneral layman’s terms without identifyingthe theoretical purpose or the independentvariables of a study. The rule against falseinformation does not require that a persongive complete information. For example, it isnot necessary to inform subjects in a between-subjects design that they have receivedCondition A rather than Condition B.

Privacy and Confidentiality

For most Web-based research, the researchis not sensitive, participants are not identified,and participation imposes fewer risks thanthose of “everyday life.” In such cases, IRBsshould treat the studies as exempt from review.

In other cases, however, data might beslightly sensitive and partially identifiable. Insuch cases, identifiers used to detect multiplesubmissions (e.g., IP addresses) can becleaned from data files once they have beenused to accomplish their purpose.

As is the case with sex surveys conductedvia personal interview or questionnaire, itusually is not necessary to keep personaldata or to keep identifiers associated withtheir data. However, there may be suchcases, for example, when behavioral data areto be analyzed with medical or educationaldata for the same people. In such cases, secu-rity of the data may require that data be sentor stored in encrypted form. Such datashould be stored on a server that is well pro-tected both from burglars who might stealthe computer (and its hard drive) and fromhackers who might try to steal the electronicinformation. If necessary, such researchmight rely on the https protocol, such as thatused by online shopping services to send andreceive credit card numbers, names, andaddresses.

As we know from the break-in of DanielEllsberg’s psychiatrist’s office, politicians andgovernment officials at times seek informa-tion to discredit their “rivals” or political“enemies.” For this reason, it is useful to takeadequate measures to ensure that informa-tion is stored in such a way that if the recordsfell into the wrong hands, they would be ofno use to those who stole them. Personalidentifiers, if any, should be removed fromdata files as soon as is practical.

A review of Web researchers (Musch &Reips, 2000) found that “hackers” had notyet been a serious problem in online research.Most of the early studies, however, dealt withtasks that were not sensitive or personal—nothing of interest to hackers. However,imagine the security problems of a Web sitethat contained records of bribes to public offi-cials, or lists of their sexual partners. Such asite would be of great interest to members ofthe tabloid press and political opponents.Indeed, the U.S. Congress set a bad examplein the Clinton-Lewinsky affair by posting tothe Web testimony in a grand jury hearingthat had not been cross-examined, eventhough such information is sealed by law. Inmost psychology research, with simple pre-cautions, it is probably more likely that infor-mation would be abused by an employee(e.g., an assistant) on the project than by a“hacker” or burglar.

The same precautions should be taken toprotect sensitive, personal data in both laband Web. Don’t leave your key to the labaround, and don’t leave or give out your pass-words. Avoid storing names, addresses, orother personal information (complete SocialSecurity Numbers) in identifiable form. Itmay help to store data on a different serverfrom the one used to host the Web site. Datafiles should be stored in folders to which onlylegitimate researchers have password access.The server that stores the data should be in alocked room, protected by strong doors andlocks, whose exact location is not public.



In most cases, such precautions are notneeded. Browsers automatically warn a per-son when a Web form is being sent unen-crypted, with a message to the effect that“the data are being sent by e-mail and can beviewed by third parties while in transit. Areyou sure you want to send them?” Onlywhen a person clicks a second time, assumingthe person has not turned this warning off,will the data be sent. A person makes aninformed decision to send such data, just as aperson accepts lack of privacy when using e-mail. The lack of security of e-mail is a riskmost people accept in their daily lives.Participants should be made aware of thisrisk but not bludgeoned with it.

I don’t believe it is necessary to insult ordemean participants with excessively lengthywarnings of implausible but imaginableharms in informed consent procedures. Suchdocuments have become so wordy and legal-istic that some people agree or refuse withoutreading. The purpose of such lengthy docu-ments seems to me (and to our participants)to be designed to protect the researcher andthe institution rather than the participantand society.

Good Manners on the Web

There are certain rules of etiquette on theWWW, known as “Netiquette.” Althoughthese, like everything else on the WWW, arein flux, there are some that you should adoptto avoid getting in trouble with vast numbersof people.

1. Avoid sending unsolicited e-mails to vastnumbers of people unless it is reasonableto expect that the vast majority want toreceive your e-mail. If you send such“spam,” you will spend more timeresponding to the “flames” (angry mes-sages) than you will spend analyzing yourdata. If you want to recruit in a specialpopulation, ask an organization to vouchfor you. Ask leaders of the organization to

send the message describing your studyand why it would be of benefit to themembers of the list to participate.

2. Do not send attachments of any kind in anye-mail addressed to people who don’t expectan attachment from you. Attachments cancarry viruses, and they clog up mailboxeswith junk that could have been better postedto the Web. If you promised to provide theresults of your study to your participants,you should post your paper on the Web andjust send your participants the URL, not thewhole paper. If the document is put on theWWW as HTML, the recipient can safelyclick to view it. If your computer crashedafter you opened an attachment from some-one, wouldn’t you suspect that attachmentto be the cause of your misery? If you sendattachments to lots of people, odds are thatsomeone will hold you responsible.

3. Do not use any method of recruiting thatresembles a “chain letter.”

4. Do not send blanket e-mails with readablelists of recipients. How would you feel ifyou got an e-mail asking you to participatein a study of pedophiles, and you saw yourname and address listed among a group ofregistered sex offenders?

5. If you must send e-mails, keep them short,to the point, and devoid of any fancy for-matting, pictures, graphics, or other mate-rial that belongs on the Web. Spare yourrecipient the delays of reading long mes-sages, and give them the choice of visitingyour materials.

CONCLUDING COMMENTS

Because of the advantages of Web-based stud-ies, I believe use of such methods will continueto increase exponentially for the next decade.I anticipate a period in which each area ofsocial psychology will evaluate Web methodsto decide whether they are suitable to thatarea’s paradigm. In some areas of research,such as social judgment and decision making,



I think that the method will be adoptedrapidly and soon taken for granted. As com-puters and software improve, investigatorswill find it easier to create, post, and advertisetheir studies on the WWW. Eventually,

investigators will regard Web-based researchas they now regard using a computer ratherthan a calculator for statistics, or using acomputer rather than a typewriter to preparea manuscript.


REFERENCES

Bailey, R. D., Foote, W. E., & Throckmorton, B. (2000). Human sexual behavior:A comparison of college and Internet surveys. In M. H. Birnbaum (Ed.),Psychological experiments on the Internet (pp. 141-168). San Diego: AcademicPress.

Baron, J., & Siepmann, M. (2000). Techniques for creating and using Web ques-tionnaires in research and teaching. In M. H. Birnbaum (Ed.), Psychologicalexperiments on the Internet (pp. 235-265). San Diego: Academic Press.

Batinic, B. (Ed.). (1997). Internet für Psychologen. Göttingen, Germany: Hogrefe.Batinic, B., Reips, U.-D., & Bosnjak, M. (Eds.). (2002). Online social sciences.

Seattle: Hogrefe & Huber.Birnbaum, M. H. (1997). Violations of monotonicity in judgment and decision

making. In A. A. J. Marley (Ed.), Choice, decision, and measurement: Essays inhonor of R. Duncan Luce (pp. 73-100). Mahwah, NJ: Lawrence Erlbaum.

Birnbaum, M. H. (1999a). How to show that 9 > 221: Collect judgments in abetween-subjects design. Psychological Methods, 4(3), 243-249.

Birnbaum, M. H. (1999b). Testing critical properties of decision making on theInternet. Psychological Science, 10, 399-407.

Birnbaum, M. H. (2000a). Decision making in the lab and on the Web. InM. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 3-34).San Diego: Academic Press.

Birnbaum, M. H. (Ed.). (2000b). Psychological experiments on the Internet. SanDiego: Academic Press.

Birnbaum, M. H. (2000c). SurveyWiz and FactorWiz: JavaScript Web pages thatmake HTML forms for research on the Internet. Behavior Research Methods,Instruments, & Computers, 32, 339-346.

Birnbaum, M. H. (2001a). Introduction to behavioral research on the Internet.Upper Saddle River, NJ: Prentice Hall.

Birnbaum, M. H. (2001b). A Web-based program of research on decision making.In U.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 23-55).Lengerich, Germany: Pabst Science Publishers.

Birnbaum, M. H. (2002). Wahrscheinlichkeitslernen. In D. Janetzko, M.Hildebrand, & H. A. Meyer (Eds.), Das Experimental-psychologischePraktikum im Labor und WWW (pp. 141-151). Göttingen, Germany: Hogrefe.

Birnbaum, M. H., & Hynan, L. G. (1986). Judgments of salary bias and test biasfrom statistical evidence. Organizational Behavior and Human DecisionProcesses, 37, 266-278.

Birnbaum, M. H., & Martin, T. (in press). Generalization across people, proce-dures, and predictions: Violations of stochastic dominance and coalescing. InS. L. Schneider & J. Shanteau (Eds.), Emerging perspectives on decisionresearch. New York: Cambridge University Press.


Birnbaum, M. H., & Mellers, B. A. (1989). Mediated models for the analysis ofconfounded variables and self-selected samples. Journal of Educational Statistics,14, 146-158.

Birnbaum, M. H., & Navarrete, J. B. (1998). Testing descriptive utility theories:Violations of stochastic dominance and cumulative independence. Journal ofRisk and Uncertainty, 17, 49-78.

Birnbaum, M. H., Patton, J. N., & Lott, M. K. (1999). Evidence against rank-dependent utility theories: Violations of cumulative independence, intervalindependence, stochastic dominance, and transitivity. Organizational Behaviorand Human Decision Processes, 77, 44-83.

Birnbaum, M. H., & Stegner, S. E. (1981). Measuring the importance of cues injudgment for individuals: Subjective theories of IQ as a function of heredity andenvironment. Journal of Experimental Social Psychology, 17, 159-182.

Birnbaum, M. H., & Wakcher, S. V. (2002). Web-based experiments controlled byJavaScript: An example from probability learning. Behavior Research Methods,Instruments, & Computers, 34, 189-199.

Buchanan, T. (2000). Potential of the Internet for personality research. InM. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 121-140).San Diego: Academic Press.

Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research:Personality testing on the World-Wide Web. British Journal of Psychology, 90,125-144.

Dillman, D. A., & Bowker, D. K. (2001). The Web questionnaire challenge to sur-vey methodologists. In U.-D. Reips & M. Bosnjak (Eds.), Dimensions ofInternet science (pp. 159-178). Lengerich, Germany: Pabst Science Publishers.

Francis, G., Neath, I., & Surprenant, A. M. (2000). The cognitive psychology onlinelaboratory. In M. H. Birnbaum (Ed.), Psychological experiments on theInternet (pp. 267-283). San Diego: Academic Press.

Frick, A., Bächtiger, M. T., & Reips, U.-D. (2001). Financial incentives, personalinformation, and drop-outs in online studies. In U.-D. Reips & M. Bosnjak(Eds.), Dimensions of Internet science (pp. 209-219). Lengerich: Pabst SciencePublishers.

Huff, D. (1954). How to lie with statistics. New York: Norton.Janetzko, D., Meyer, H. A., & Hildebrand, M. (Eds.). (2002). Das Experimental-

psychologische Praktikum im Labor und WWW [A practical course on psycholog-ical experimenting in the laboratory and WWW]. Göttingen, Germany: Hogrefe.

Joinson, A. (2002). Understanding the psychology of Internet behaviour: Virtualworlds, real lives. Basingstoke, Hampshire, UK: Palgrave Macmillan.

Krantz, J. H. (2001). Stimulus delivery on the Web: What can be presented whencalibration isn’t possible? In U.-D. Reips & M. Bosnjak (Eds.), Dimensions ofInternet Science (pp. 113-130). Lengerich, Germany: Pabst Science Publishers.

Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratoryand World-Wide Web samples on the determinants of female attractiveness.Behavior Research Methods, Instruments, & Computers, 29, 264-269.

Krantz, J. H., & Dalal, R. (2000). Validity of Web-based psychological research. InM. H. Birnbaum (Ed.), Psychological experiments on the Internet (pp. 35-60).San Diego: Academic Press.

Luce, R. D., & Fishburn, P. C. (1991). Rank- and sign-dependent linear utilitymodels for finite first order gambles. Journal of Risk and Uncertainty, 4, 29-59.

McGraw, K. O., Tew, M. D., & Williams, J. E. (2000a). The integrity of Web-basedexperiments: Can you trust the data? Psychological Science, 11, 502-506.

McGraw, K. O., Tew, M. D., & Williams, J. E. (2000b). PsychExps: An on-linepsychology laboratory. In M. H. Birnbaum (Ed.), Psychological experiments onthe Internet (pp. 219-233). San Diego: Academic Press.



McKenna, K.Y.A., & Bargh, J. A. (2000). Plan 9 from cyberspace: The implicationsof the Internet for personality and social psychology. Personality and SocialPsychology Review, 4(1), 57-75.

Milgram, S. (1974). Obedience to authority. New York: Harper & Row.Mueller, J., & Furedy, J. J. (2001). The IRB review system: How do we know it

works? American Psychological Society Observer, 14. Retrieved March 23,2003, from http://www.psychologicalscience.org/observer/0901/irb_reviewing.html

Musch, J., Broeder, A., & Klauer, K. C. (2001). Improving survey research on theWorld-Wide Web using the randomized response technique. In U.-D. Reips &M. Bosnjak (Eds.), Dimensions of Internet science (pp. 179-192). Lengerich,Germany: Pabst Science Publishers.

Musch, J., & Reips, U.-D. (2000). A brief history of Web experimenting. In M. H.Birnbaum (Ed.), Psychological experiments on the Internet (pp. 61-87). SanDiego: Academic Press.

Pagani, D., & Lombardi, L. (2000). An intercultural examination of facial featurescommunicating surprise. In M. H. Birnbaum (Ed.), Psychological experimentson the Internet (pp. 169-194). San Diego: Academic Press.

Parducci, A. (1995). Happiness, pleasure, and judgment. Mahwah, NJ: LawrenceErlbaum.

Pettit, F. A. (1999). Exploring the use of the World Wide Web as a psychology datacollection tool. Computers in Human Behavior, 15, 67-71.

Piper, A. I. (1998). Conducting social science laboratory experiments on the WorldWide Web. Library & Information Science Research, 20, 5-21.

Quiggin, J. (1993). Generalized expected utility theory: The rank-dependent model.Boston: Kluwer.

Reips, U.-D. (1997). Das psychologische Experimentieren im Internet. In B. Batinic(Eds.), Internet für Psychologen (pp. 245-265). Göttingen, Germany: Hogrefe.

Reips, U.-D. (2000). The Web experiment method: Advantages, disadvantages, andsolutions. In M. H. Birnbaum (Ed.), Psychological experiments on the Internet(pp. 89-117). San Diego: Academic Press.

Reips, U.-D. (2001a). Merging field and institution: Running a Web laboratory. InU.-D. Reips & M. Bosnjak (Eds.), Dimensions of Internet science (pp. 1-22).Lengerich, Germany: Pabst Science Publishers.

Reips, U.-D. (2001b). The Web experimental psychology lab: Five years of datacollection on the Internet. Behavior Research Methods, Instruments, &Computers, 33, 201-211.

Reips, U.-D. (2002). Standards for Internet experimenting. ExperimentalPsychology, 49(4), 243-256.

Reips, U.-D., & Bosnjak, M. (2001). Dimensions of Internet science. Lengerich,Germany: Pabst Science Publishers.

Rosenthal, R. (1976). Experimenter effects in behavioral research: Enlarged edition.New York: Irvington.

Rosenthal, R. (1991). Teacher expectancy effects: A brief update 25 years after thePygmalion experiment. Journal of Research in Education, 1, 3-12.

Schillewaert, N., Langerak, F., & Duhamel, T. (1998). Non-probability samplingfor WWW surveys: A comparison of methods. Journal of the Market ResearchSociety, 40(4), 307-322.

Schmidt, W. C. (1997a). World-Wide Web survey research: Benefits, potential prob-lems, and solutions. Behavioral Research Methods, Instruments, & Computers,29, 274-279.

Schmidt, W. C. (1997b). World-Wide Web survey research made easy with WWWSurvey Assistant. Behavior Research Methods, Instruments, & Computers, 29,303-304.



Schmidt, W. C. (2000). The server-side of psychology Web experiments. InM. H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. 285-310).San Diego: Academic Press.

Schmidt, W. C. (2001). Presentation accuracy of Web animation methods. BehaviorResearch Methods, Instruments & Computers, 33, 187-200.

Schmidt, W. C., Hoffman, R., & MacDonald, J. (1997). Operate your own World-Wide Web server. Behavior Research Methods, Instruments, & Computers, 29,189-193.

Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alter-native source of subjects and research environment. Behavior ResearchMethods, Instruments, & Computers, 29, 496-505.

Stern, S. E., & Faber, J. E. (1997). The lost e-mail method: Milgram’s lost-lettertechnique in the age of the Internet. Behavior Research Methods, Instruments,& Computers, 29, 260-263.

Tuten, T. L., Urban, D. J., & Bosnjak, M. (2002). Internet surveys and data qual-ity: A review. In B. Batinic, U.-D. Reips, & M. Bosnjak (Eds.), Online socialsciences (pp. 7-26). Seattle: Hogrefe & Huber.

Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulativerepresentation of uncertainty. Journal of Risk and Uncertainty, 5, 297-323.

Wallace, P. M. (2001). The psychology of the Internet. Cambridge, UK: CambridgeUniversity Press.

Welch, N., & Krantz, J. H. (1996). The World-Wide Web as a medium for psy-choacoustical demonstrations and experiments: Experience and results.Behavior Research Methods, Instruments, & Computers, 28, 192-196.



Documents

Methodological and Ethical Issues in Conducting Social ...psych.fullerton.edu/mbirnbaum/papers/Ch16birnbaum.pdf16 Methodological and Ethical Issues in Conducting Social Psychology