Survey Methods 2

Class 20

Estimating Sampling Error

Assume everything done correctly thus far:

Specified sampling frame

Stratified by area

Sampled to accommodate underrepresented groups

Is our survey error-proof? 100% reliable?

No: Surveyed only 10% of population—sample may misrepresent source population.

How accurate is our sample? Accuracy is a function of:

1) Sample size and 2) variability of thing measured

Sampling error = amount sample differs from total population

Standard Error of the Mean

1 SE = 67% of sample means

2 SE = 95% of sample means

2 SE = "Confidence interval"

Standard Error = SE = Variance/n

Standard Errors for Probability

SE for Probabilities p = Probability that the characteristic exists

1 - p = probability that the characteristic does not exist

p(1-p) = variance of proportion

p(1-p)/n = SE of a proportion Example

30 % of FIASCO respondents say that arguments are related to money.Sample size (n) = 1500

SE = p(1-p)/n = .30 X .70 /1500 = .21/1500 = .01 Interpretation

Confidence interval = 2 SE = 2 X .01 = .02True mean (proportion) = .30 .02; or 95% confident that true mean lies between .28 and .32

OOptimal Sample Size No Easy Answers—But a Possible Rule of Thumb

1. Samples of 150 fairly accurate: little gain in precision beyond n = 150

2. Consider the smallest sub-group you will sample—make sure

that that group is sampled at n =150, as are all other groups. Then, make sure to weight groups according to their

representativeness.

Response Rate

ComputationNumber Actually Sampled

Number in Targeted Sample

1400 = 78% Response Rate1800

Typical and Acceptable Ranges

Office of Management and Budget (OMB) Standard: 75%

Less than 30%: Self selected

Common response rate: Between 25% and 90%

Reasons for Non-Response Related to Bias

Survey Procedure

Respondent Interest Level Respondent Education Level

Respondent Availability

Respondent Accessibility

Respondent Ability to Participate Library Digest predicts Alf Landon for President, 1936

Correcting for Non-Response

Proxy Respondents

OK for factual items (how often do arguments occur?)

Not OK for subjective items (how do you feel about arguments?)

Statistical Adjustments

Re-weight sub-group with high level of non-response(But, what if non-respondents systematically differ from respondents?)

Survey Non-respondents

(But, what if “responding” non-respondents systematically differ from“non-responding” non-respondents?)

Developing the Survey Instrument

Families in America, Survey of Conflicts (FIASCO)

1. Sampling frame* Nuclear families* S. Orange, NJ

2. Sampling strategy

* Systematic stratified (strat. by quadrants)* Differential probability sampling (oversample new res.)

3. Contact strategy

* Random digit dialing (RDD)* Only 1 person per home, parents/kids = chance

4. Sample size

* n = 150 for smallest subgroup, based on est. of std. error 5. Non-response (NR) strategy

* Means to monitor NR* Preventive measures (e.g., ease of response, follow-up calls)* Corrective measures (e.g., stat. adjustments, survey NRs)

Survey Design

1. Define objectives 2. Draft and refine questions 3. Format questions 4. Field test instrument

Defining Objectives

1. Statement of Purpose 2. Listing of main variables 3. Analysis Plan

Statement of Purpose: FIASCO Survey

The purpose of this survey is to assess family conflicts in S. Orange, NJ. Toward this end, the survey measures the type, frequency, and causes of conflicts in S. Orange families. The survey also attempts to relate family characteristics, and family member's attitudes, towards the type, frequency, and resolution of conflicts.

Family characteristics -------> Conflicts Family attitudes -------> Conflicts

Listing of Variables: FIASCO StudyIndependent Variables

Demographics a. Family size b. Family composition c. Income level d. Education level e. ______________

Attitudes a. When should conflicts be aired? b. How should conflicts be settled?

Dependent Variables

Conflicts a. Type of conflict b. Frequency of conflicts c. Causes of conflicts

d. ______________

Analysis Plan 1. Sampling Frame (e.g., stratified by income) 2. Sampling Method (e.g., RDD phone survey) 3. Type of inquiry:

a. Associations (correlations)?b. Causes (experiment)?

* Pre-post?* Between groups?

Criteria for Good Survey Questions

1. Reliable

Respondents understand question meaning

Respondents understand what is expected of them

2. Valid

Question item addresses underlying concept.

Road Bumps in Human Communication

(Watzlawick, Beavin, & Jackson, 1967)

Watzlawick (Vaht-slahv-ick) applied for an assistantship with a psychiatric research institute. The following conversation took place with the receptionist.

WATZ: Good afternoon, I have an appointment with Dr. H. My name is Watzlawick

RECEPTIONIST: I didn't say it was.

WATZ.: (taken aback and somewhat annoyed). But I'm telling you it is.

RECEPTIONIST: (bewildered) Why then did you say it wasn't?

WATZ.: But I said it was.

Watz.’s INTERPRETATION: Certain that he was being made the object of some incomprehensible but disrespectful joke. RECEPTIONIST'S INTERPRETATION: First this guy says “my name is not Slavic”, then he says it is Slavic. He must be a new psychotic patient of Dr. H's.

Specific Questions Must Map on to Subjects' Reality

"Do people in your family employ a Socratic method of discourse when attempting to isolate the instigating cause of the discord under which you collectively toil?"

"To what degree do people in your household argue about the influence of the International Monetary Fund (IMF)?"

  Tell person to cheer up.

  Ask person how you can help. 

  Don't let person stew in bad mood. 

  Take charge of person's problem. 

  Listen without judging. 

  Let person express any feelings. 

 

Tell person to cheer up 

  Ask person how you can help.

Don't let person stew in bad mood. 

  Listen without judging.

Take charge of person's problem.   Let person express any feelings.

Directive  Non-Directive

???-Sorting

  Tell person to cheer up.

  Ask person how you can help. 

  Don't let person stew in bad mood. 

  Take charge of person's problem. 

  Listen without judging. 

  Let person express any feelings. 

 

Tell person to cheer up 

  Ask person how you can help.

Don't let person stew in bad mood. 

  Listen without judging.

Take charge of person's problem.   Let person express any feelings.

Directive  Non-Directive

Q-Sorting

??? Groups

Best done with 6-8 people Types of things participants can contribute:

I. Understanding of questions

a. Conceptsb. Terms

II. Insights related to topic

a. Perceptions b. Experiencesc. Feelings

Focus Groups

Best done with 6-8 people Types of things participants can contribute:

I. Understanding of questions

a. Conceptsb. Terms

II. Insights related to topic

a. Perceptions b. Experiencesc. Feelings

CONCEPTS AND WORDING

"When do conflicts most often occur in your family?"

"What do people in your family most often fight about?"

___ Finances ____ Chores ____ Noise ___ Messes ____ Other

INSIGHTS

What constitutes a conflict?

What are the biggest problems with conflicts?

How do conflicts get started?

How do conflicts get settled?

How do you feel about conflicts?

WHAT'S MISSING?

FIASCO Focus Group

Wording Problems

1. Incomplete wording

2. Inappropriate optional wording

3. Poor word choice

4. Poorly defined terms

5. Special cases

a. "Don't Know"b. Wording for special groups

Incomplete Wording

Better form "What was the main reason for the most recent conflict you personally had with someone in your family?"

Poor form

“Reason for last conflict?”

A 1. Yes

A 2. Because of competition for the TV remote.

A 1. Because of competition for the TV remote.

Unacceptable Optional Wording

"You said your last conflict was with your [NAME FAMILY MEMBER] How often did you get into conflicts with (HIM/HER) during the last month?"

"How intense was your last conflict" (Was there shouting, physical threats, acts of violence)?

Improper optional wording:

Proper optional wording:

Unclear Meanings

"On what calendared orientation point did the most recent conflagration transpire amongst those with whom you share primary genetic bonds?"

PROBLEM? Overly Complex

"Whom did you last see for professional advice regarding conflicts in your family?"

"Are you punished if you don't complete routine chores (like housecleaning) on time"?

PROBLEM? Ambiguous Terms

" When do arguments most often occur in your household?"

Response Format for Open-Ended Questions

A 1: When mom’s tiredA 2: On Sundays

"What things do you do to calm tensions between your siblings?"

A 1: A zillion things. 1…., 2.….., 3.…., 4.….., 5.……, 100000000…..

A 2: Take a deep breath

Specify times, quantities, frequenciesProblem?

Specify the number of points to be made.Problem?

“Why do you and your father argue about money?

A1: Because of rising tuition costs.

A2: Because he’s a jerk.

Problem? Focus the objective of questions.

Open Ended and Closed Ended Questions

Open Ended: Where respondent is free to provide an answer in his or her own words.

"What topics lead to arguments in your family?"

The things that lead to fracases in my family are usually about clothing. Mummy insists that we wear shirt and ties to the Sunday dinner table, but Natasha and Sterling just want to wear their day-glo capes and studded collars.

Closed Ended: Where the respondent is restricted to a pre-set range of response options.

"What topics lead to arguments in your family?"

___ Finances___ Noise_X_ Decorum (manners, clothing, make-up)___ Household chores___ Opinions about social issues___ Other

Closed Ended vs. Open Ended Questions

Closed Ended

Defines question more clearly

Defines response options more clearly

Does not require an interviewer

Easy for respondent

Easy for coding/analysis

Easy for data entry

Provides for unexpected responses

Less likely to bias responses

Enhances respondent good-will

Historical record

Open Ended

How would you rate the competence of your court-appointed family mediator?

Poor 

Fair Good Very Good Excellent

1 2 3 4 5

To what degree are family conflicts resolved in a way you find satisfactory?

Not at all 

Slightly Somewhat Quite a Bit To a Great Degree

1 2 3 4 5

Subjective Response Scales

Considerations Regarding Questionnaire Item Format

1. Limited interpretability of subjective response scales

2. Agree-disagree options

3. "Don't Know" option

4. Middle position choice options

5. Unbalanced vs. balanced format

6. Measures of attitude strength

How Do You Rate Robeson Cafeteria Food?

Good Fair Poor

How Do You Rate Robeson Cafeteria Food?

Superb Very Good Good Fair Poor

Agree/Disagree Format

1. I wish my family would be more courteous.

2. I am tired of having to watch what I say.

3. Venting pent up feelings is selfish.

4. My family needs to see a conflict mediator.

Agree Disagree

Problems with this format?

Problems With Agree/Disagree Format

Loose info when squeezing range of attitudes into a dichotomy. Can’t correlate responses w’ each other.

Works best for extreme attitudes. Can’t distinguish moderate or ambivalent att’s from extreme att’s.

Confusing to indicate a negative (or positive) attitude by responding in the affirmative (or negative).

Fowler discourages use of agree/disagree for these reasons.

What Difference Does It Make To Include / Exclude DK (“Don’t Know”) Response?

1. Does it change rate at which people say DK?

2. Does it change the rate at which substantive (non-DK) responses are endorsed, relative to one another?

3. Does DK option change relationships between variables?

Solution: Provide DK option.

Solution: Provide DK option.

“Floaters”: A Problem and a Mystery

Floaters: People who ??? when there is not a DK (“don’t

know”) filter, but who reply “???” when there is a DK filter. Relations between responses changes when DK filters are used.

* Floaters ??? true correlations* Floaters can create ??? correls, where none actually exist* Floaters sometimes ???

Who are floaters? ???

“Floaters”: A Problem and a Mystery

Floaters: People who give a response when there is not a DK (“don’t

know”) filter, but who reply “Don’t Know” when there is a DK filter. Relations between responses changes when DK filters are used.

* Floaters weaken true correlations* Floaters can create false correls, where none actually exist* Floaters sometimes make no difference at all

Who are floaters? Difficult to say. For obscure questions, floaters tend to

be among the less educated.