Data Collection Mode Effects Controlling for Sample Origins in a

Panel Survey: Telephone and Internet

Presented at the 2005 Annual Meeting fo the American Association of Public Opinion

Research

Mike Dennis, Knowledge Networks

Cindy Chatt, University of Nebraska-Lincoln

Alicia Motta-Stanko, Knowledge Networks

Paul Pulliam, RTI International

2

Acknowledgements

The survey designed by:

RTI International

The Odum Institute at the University of North Carolina

The authors wish to thank RTI International for permission to analyze the data used in this paper and disseminate the findings.

3

The Purpose of the Research

What matters more in accounting for variance in survey responses?

Mode of Data Collection (Telephone versus Internet)

or

Sample Origin (KN panel sample versus KN panel rejecters/non-cooperators)

This is a contest of data collection mode versus sample quality.

The main purpose of this research is to help further our understanding of the respective contributions of mode of data collection (phone vs. web) and sample origin (cooperators vs. non-cooperators) in explaining variance in survey response data.

4

Background of Study

The analysis is based on the Survey on Civic Attitudes and Behaviors After 9/11.

Study was created with a non-response follow-up study and implemented by Knowledge Networks from January to March 2002.

The study is important because: It is the first mode-effects study using KN data that

effectively controls for sample type (persons who joined the KN panel and participate in KN studies, versus those who refused to join the panel or who refused to participate in the panel study).

5

Questionnaire: Public Policy & Civic Attitudes & Behaviors

The survey questionnaire had several modules of interests: ratings of Bush and Gore (early 2002!), attitudes toward terrorism, the adequacy of governmental response to bioterrorism and terrorism more generally, attitudes toward what the government should be doing and would do in the event of a terrorist attack, civic participation and civic values questions, and background questions on religious faith and other aspects.

The survey was approximately 25 minutes in length.

6

Main Finding

The mode of data collection is more often a significant factor in accounting for variance in survey responses than is the origin of the sample. When controlling for panel experience and demographics, for every one question where sample origin was a significant factor, mode of data collection was significant for six questions.

MODESample

7

Sample Design for the Study

RDD Sample

Panel Acceptors

Panel Rejecters

Internet Sample

Telephone Sample (Controls)

Non-Response Follow-Up Sample(NRFUS)

Random sample for each of three groups.

8

Sample Composition, Mode, & Sample Size

Sample Group/ModeSample Size

FieldedCompleted Interviews

Completion Rate

KN Panel by Web 3627 2979 82%

KN Panel by Telephone (Controls)

477 300 63%

Nonresponse1 (NRFUS) by Telephone

2730 600 22%

1Nonresponse sample consists of ‘panel rejecters’ and those panelists that did not respond to the Internet survey.

9

Sample Representativeness: Demographic Comparison

Each interviewed group was compared to the January 2002 Current Population Survey (Census)

Main Findings: Mean Error (Unweighted):

– Panel by Web: 2.8 percentage points

– Panel by Phone: 4.1 percentage points

– NRFUS by Phone: 3.6 percentage points

10

Summary of Multivariate Analyses: Count of Statistically Significant Predictors of Answers by Mode and NRFUS (p < .05)

  N items

N significant for mode

N significant for NRFUS

Grade Bush's performance (attitudinal) 2   1

Worried about terrorism (attitudinal) 2 2

Information expected during bioterrorist event (attitudinal) 6 5

Sources from which anthrax information sought (behavioral) 6 6 2

Trusted source during bioterrorism event (attitudinal) 1  

Feeling thermometers for Bush and Gore (attitudinal) 2 2

Important issues, politics, current events (behavioral) 3 3

Neighborhood statements (combination - attitudinal and behavioral) 13 9 1

Self-perception statements (attitudinal) 5 5 2

Volunteerism and/or donating behavior (behavioral) 4 2

  44 34 6

100% 77% 14%

11

Results of Multivariate Analyses

Phone respondents more likely to: Disagree that bioterrorism is the most important problem Seek info on anthrax from …web, hotlines, national TV, own doctor, local

government, other Rate Pres. Bush and Al Gore higher on feeling thermometer Discuss politics Discuss community issues Help neighbors Be happy about their neighborhood Have pride in their neighborhood Have a sense of belonging in neighborhood Have a neighborhood that gives them pleasure Rely on neighbors Trust others Enjoy mixing socially with others

12

Freq Distributions by Group (Unweighted)

Question Panel by Internet

Panel by Telephone

NRFUS by Telephone

Rate George Bush for job done as President (% ‘A’ grade)

42 40 40

Grade Bush on dealing with terrorism

(% ‘A’ grade)

57 57 51

No large differences across the groups for grading political job performance.

13

Freq Distributions by Group (Unweighted)

Question: When a bioterrorism event happens, what info do you expect CDC to provide the public?

Panel by Internet

Panel by Telephone

NRFUS by Telephone

Type of GERM (% Yes) 93 94 96

Name CRIMINAL Suspects (% Yes) 38 60 60

No. of people dead, ill, etc. (% Yes) 71 85 88

Explanation of steps people should take to minimize risk (% Yes)

91 97 98

Recommendations for treating illness caused by germ (% Yes)

88 97 97

Travel advisories/restrictions (% Yes) 70 89 89

Telephone mode: Higher expectations for information from CDC.

14

Freq Distributions by Group (Unweighted)

Question:

Sought info on ANTHRAX or other SEVERE ILLNESSES from…

Panel by Internet

Panel by Telephone

NRFUS by Telephone

Local radio and TV 46 43 40

WEB health and news sites 12 26 17

Phone HOTLINES 1 5 6

Cable 24-hour News and Network News 26 46 43

Own Physician or Health Care Professional 4 10 12

Local or State Health Department 3 10 11

Telephone mode: Higher report of information-seeking behavior.

15

Freq Distributions by Group (Unweighted)

Question:

Within the past year, how often have you…

Panel by Internet

Panel by Telephone

NRFUS by Telephone

Borrowed, loaned, etc, small things with neighbors like sugar

(% Sometimes & Often)

39 43 46

Helped people in this neighborhood or they helped you with small favors

(% Sometimes & Often)

60 71 70

Telephone mode: Higher report of ‘good neighbor’ behavior.

16

Freq Distributions by Group (Unweighted)

Civic Self-Perception Statements

(% Completely Agree)

Panel by Internet

Panel by Telephone

NRFUS by Telephone

I am happy to live in this neighborhood 35 49 57

See myself as part of neighborhood 21 37 40

Feel sense of belonging to neighborhood

22 38 42

Being in neighborhood gives me a lot of pleasure

23 39 43

Would get my neighbors to fix problems

9 23 22

Telephone mode: Higher report of positive feelings toward own neighborhood.

17

Freq Distributions by Group (No weight)

Sociability Self-Perception Statements

(% Completely Agree)

Panel by Internet

Panel by Telephone

NRFUS by Telephone

I am trusting of others 9 23 22

I easily fit into groups 9 28 32

I like to mix with others 13 31 39

I tend to be a happy person 13 31 39

I enjoy helping others 23 41 46

Telephone mode: Higher report of positive sociability self-perceptions.

Discussion

19

Major Differences in Response Caused by Mode?

Major differences existing between Internet and telephone modes of data collection include:

Presence or absence of interviewer

Dependence on visual or aural communication

Interviewer or respondent control of interview pace or information sequence

Each difference can lead to multiple mode effects seen in the data.

(Dillman, Sangster, Tarnai, and Rockwood, 1996)

20

Presence or Absence of Interviewer

Can lead to: Tendency to answer on extreme ends of response

scales Non-differentiationSocial desirability / self-presentation bias

21

1. I am happy to live in this neighborhood.

2. I really see myself as a part of this neighborhood

3. I feel a sense of belonging to this neighborhood.

4. Being in this neighborhood gives me a lot of pleasure.

5. If there are things in my neighborhood that need to be fixed or improved, I would be able to get my neighbors to do something about it.

Presence or Absence of Interviewer:Neighborhood Statements

22

% Average Response by Sample Group:Neighborhood Questions

66.2%

79.6% 76.9%

16.9%

9.9% 10.6%16.5%

10.3% 11.6%

0.4% 0.2% 0.8%0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Internet Telephone NRFUS

Sam ple Group

% o

f A

vera

ge

Res

po

nse

Positive

Negative

Neither

Don't Know /Refused

Very similar results for self-perception items (e.g., “I am trusting.”)

23

Evidence of Non-Differentiation and Recency Effects on Phone (Neighborhood Statements)

Response: Completely Agree (+5)

Internet Telephone NRFUS

All same Answer 4.9% 16%* 13.8%*

4 out 5 Same 10.2% 14.8%* 17.3%*

Response: Neutral (0)

Internet Telephone NRFUS

All same Answer 2.1% 0.1% 1.4%

4 out 5 Same 2.5% 1.0% 2.4%

Response: Completely Disagree (-5)

Internet Telephone NRFUS

All same Answer 2.0% 1.1% 1.0%

4 out 5 Same 1.6% 0.4% 0.9%

*Significant differences at p<.05 between Internet and

Telephone and Internet and NRFUS for Completely Agree

24

Conclusions

Sample origins do not appear to be having much of an effect on results compared to mode of data collection

Differences in the data are consistent with inherent differences between Internet and telephone data collection, leading to patterns in the phone-collected data reflecting:

Social desirability bias Extreme positive responses on answer scales Answer distribution differences due to communication type differences Item Non-differentiation

Study limitation/Buyer Beware: Questionnaire potentially laden with more questions (compared to many surveys) susceptible to social desirability bias.

Thank you!mdennis@knowledgenetworks.com

(650) 289-2160