Primary Data and Measurement

Jeremy Kees, Ph.D.

Stages in theResearch Process

Determine Research Design

Analyze and Interpret the Data

Design Sample and Collect Data

Formulate Problem

Design Data Collection Method and Forms

Prepare the Research Report

Primary Data: Overview

• Types of Primary Data– Demographic / Socioeconomic Characteristics– Psychological / Lifestyle Characteristics– Attitudes / Opinions– Awareness / Knowledge– Intentions– Motivation– Behavior

• What, how much, where, when, how, who• Purchase behavior vs. use behavior

• E.g., --- basic hierarchy of effects models

• Qualities of Primary Data– Versatility

– Can be obtained quickly

– Quality• Objectivity

• Accuracy

Primary Data: Overview

• Two Classes of Primary Data– Observation

– Communication

Primary Data: Overview

Method ofAdministration

Degree ofStructure

Degree ofDisguise

Method ofAdministration

Degree ofStructure

Degree ofDisguise

Setting

Communication

Observation

StructuredUnstructured

StructuredUnstructured

DisguisedUndisguised

DisguisedUndisguised

Personal InterviewTelephone InterviewMail Questionnaire

NaturalContrived

HumanMechanical

Decisions, Decisions

Structure and Disguise Decisions

Motivation ResearchWord Association

Sentence CompletionStory Telling

InterviewsOpen-Ended Questions

Typical Questionnaire(very frequently used)

least used

Structured

Undisguised

Unstructured

Disguised

Trade-Offs Among Methods of Survey Administration

LIMITEDHIGHLYVERSATILE

phonewebemailmail

fax personal(e.g., mall, home)

HI COSTLOWCOST

phoneweb

mailpersonal fax

email

SLOWFASTTURN-AROUNDphone

webmailpersonal

faxemail

Mall Intercepts vs. E-Panels

mall internet paneltests tests members

household size 2.8 2.9 3.0average age 40.5 39.2 37.2employed 71% 72% 69%white 86% 88% 89%male 20% 21% 15%college 40% 43% 46%

Correlation between Responses:

mall vs. internetinternet test/retest reliability

purchase intent .86 .94frequency .94 .97liking .85 .91price / value .90 .99

Jeff Miller and Alan Hogg “Internet vs. Telephone Data Collection” Burke White Paper series 2 (4) (www.burke.com). Also see Ashok Ranchhod and Fan Zhou “Comparing Respondents of E-Mail and Mail Surveys,” Marketing Intelligence & Planning 19 (2001), 254.

Internet PhoneTime survey took to administer 12.5 19.4 minutes

Upon completion, would respondent participate in future studies? 35% yes 26% yes

More experienced Internet Users x

Used rating scale extreme “endpoints”more frequently x

E-Panels vs. Phone

• Open-ended question• Unprobed format: seeks no additional

information• Probed format: includes a response

probe instructing the interviewer to ask for additional information

Basic Question Formats

Basic Question Formats

• Close-ended question• Dichotomous: has only two response

options, such as “yes” or “no”• Multiple response: has more than two

options for the response• Scaled-response: utilizes a scale

developed by the researcher to measure the attributes of some construct under study• Labeled vs. Unlabeled

Other Considerations…

• The nature of the property being measured

• Previous research studies

• The data collection mode

• The ability of the respondent

• The scale level desired…

Types of Scales

• Nominal scales: those that use only labels

• Ordinal scales: those with which the researcher can rank-order the respondents or responses

• Interval scales: those in which the distance between each descriptor is equal

• Ratio scales: ones in which a true zero exists

15

IntervalPlease indicate how much you like each soft drink by checking the appropriate position on thescale: dislike like

a lot dislike like a lotCoke ____ ____ ____ ___Dr. Pepper ____ ____ ____ ___Mountain Dew ____ ____ ____ ___Pepsi ____ ____ ____ ___Seven Up ____ ____ ____ ___Sprite ____ ____ ____ ___

NominalWhich of the soft drinks in the following list do you like? (Check ALL that apply):

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite

OrdinalRank the soft drinks according to how much you like each (most preferred drink = 1, and least preferred drink = 6):

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite

RatioPlease divide 100 points among these soft drinksTo represent how much you like each:

___Coke___Dr. Pepper___Mountain Dew___Pepsi___Seven Up___Sprite100

Examples…

Thinking Ahead…What analysis technique is used when…

1. There are 2 nominally scaled variables2. There are 2 intervally (or ratio) scaled variables3. There is one DV that your trying to predict or explain

with multiple interval variables (the independent variables)

4. You are checking for differences between 2 groups (nominal) for an interval scaled variable

5. You are checking for differences between 4 stores on total $ sales across 4 categories?

Potential Answers --- t-test; multiple regression, oneway ANOVA, factorial ANOVA, correlation, crosstabs, multivariate analysis of variance

Survey (Questionnaire) Design

Procedure for Developing a Questionnaire

Step 1: Specify What Information Will be Sought

Step 2: Determine Type of Questionnaire and Method of Administration

Step 3: Determine Content ofIndividual Questions

Step 4: Determine Form of Response to Each Question

Procedure for Developing a Questionnaire

Step 9: PRE-TEST the Survey,Revise where Needed

Step 8: Re-examine Steps 1-7and Revise if Necessary

Step 5: Wording of Each Question

Step 6: Sequenceof Questions

Step 7: Physical Characteristicsof Questionnaire

Desirable Characteristics

• Brief

• Objective

• Specific

• Relevant

“Other” Important Issues

• Respondent qualification

• Frame of reference

• Question loading

QUESTION SEQUENCING - General Guidelines

• Use (more) simple, interesting opening questions

• Use the funnel approach, asking broad questions first, and follow with more specific questions

• Carefully design branching questions• Ask for classification information last• Place more difficult or sensitive

questions near the end

Question ordering #1

1 – EVALUATION OF FAT LEVEL OF A PRODUCT

2 – EVALUATION OF OVERALL PROD. NUTRITIOUSNESS

3 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE

Question ordering #2

1 – EVALUATION OF OVERALL PRODUCT ATTITUDE AND INTENTIONS TO PURCHASE

2 – EVALUATION OF LEVEL OF PROD. NUTRITIOUSNESS

3 – EVALUATION OF FAT LEVEL OF PRODUCT

QUESTION SEQUENCING - General Guidelines

QUESTION WORDING - General Guidelines

• Use simple words and questions• Avoid ambiguous words and

questions• Avoid leading questions---be

objective• Avoid implicit alternatives• Avoid generalizations and

estimates ---Be specific • Avoid double-barreled questions

1. What is your income?

$10,000 or less………………….1

$10,000 to $25,000……………..2

$25,000 to $50,000……………..3

$50,000 to $75,000………..…….4

$75,000 to $100,000..……..…….5

$100,000 or more…………..……6

What is the problem and how would you revise the question?

2. Do you agree with many experts that the United States will run out of oil by the year 2020?

Yes………………….1

No……………...……2

3. How important was price compared to service when your company decided to buy its current computer network?

Very important…………….4

Somewhat important……….3

Somewhat unimportant…….2

Very unimportant...…...……1

Please rate the importance of the following factors in making the choice among vendors.

Selling price:

Not Very Important ___ ___ ___ ___ ___ ___ ___ Important

Vendor’s reputation for service:Not Very Important ___ ___ ___ ___ ___ ___ ___ Important

4. Is the speed and efficiency of the drive-in teller services at your regular bank…..(READ CATEGORIES)

Very Satisfactory………………………4

Somewhat satisfactory…………………3

Somewhat unsatisfactory………………2

Very unsatisfactory……………….……1

PROBLEMS

a. Speed and efficiency (double-barreled)

b. Do they use drive-in teller service; Do R’s know what drive in tellers are? What about ATM drive-ins?

c. What are regular banks?

Itemized Rating Scales

• The respondents are provided with a scale that has a number or brief description associated with each category.

• The categories are ordered in terms of scale position, and the respondents are required to select the specified category that best describes the object being rated.

• The commonly used itemized rating scales are the Likert and semantic differential– Others include Thurstone and Guttman

Itemized Rating Scales

• Likert Scales– requires the respondents to indicate a degree of

agreement or disagreement with each of a series of statements about the stimulus objects

 Strongly Disagree Neither Agree Strongly

disagree agree nor agree

disagree

 

1. Wal-Mart sells high quality merchandise. 1 2X 3 4 5

 

2. Wal-Mart has poor in-store service.1 2X 3 4 5

 

3. I like to shop at Wal-Mart . 1 2 3X 4 5

4. Wal-Mart has low prices . 1 2 3X 4 5

Itemized Rating Scales

• Semantic Differential Scales– End points associated with bipolar labels that have

semantic meaning

SEARS IS:

Powerful --:--:--:--:-X-:--:--: Weak

Unreliable --:--:--:--:--:-X-:--: Reliable

Modern --:--:--:--:--:--:-X-: Old-fashioned

Key Decisions for Itemized Scales

• Number of categories• Although there is no single, optimal number, traditional guidelines

suggest that there should be between five and nine categories

• Balanced vs. unbalanced• In general, the scale should be balanced to obtain objective data

• Odd/even no. of categories• If a neutral or indifferent scale response is possible for at least some

respondents, an odd number of categories should be used

• Forced vs. non-forced• In situations where the respondents are expected to have no opinion,

the accuracy of the data may be improved by a non-forced scale( DK/NR)

• Verbal description• An argument can be made for labeling all or any scale categories. The

category descriptions should be located as close to the response categories as possible

Measurement Theory

• Construct Validity– Criterion (Face) Validity– Predictive Validity– Nominological Validity

• Reliability

Construct Validity

Construct Abstract Concept

“Unobservable”

“Latent”

“Psychological”We often use Multi-item measures

(scales) to assess unobservable constructs (WHY???)

Reliability

• True Score Test Theory– All measures have

1. “True” Score2. “Error” (Random and Systematic)

• Good measures minimize the systematic error component of the score

• Types of Reliability– Inter-Rater– Test-Retest– Internal Consistency (Cronbach’s Alpha)

Developing Sound Measures

Step 2:

Specify Domainof the Construct

Generate Sampleof Items

Collect Data

Purify Measure

Assess Validity

Step 1:

Step 3:

Step 4:

Step 5:

Tips for Maximizing Participation

• Personal communication• Asking a favor• Importance of the research project and its

purpose• Importance of the recipient, how recipient

was selected• How the recipient may benefit from this

research• Completing the questionnaire will take only a

short time• The questionnaire can be answered easily

Tips for Maximizing Participation

• A stamped reply envelope is enclosed• Answers are anonymous or

confidential• Offer to send report on results of

survey• Appreciation of sender• Importance of sender or sender’s

organization• Description and purpose of incentive• Style, format and appearance• Brevity

Tips for Maximizing ParticipationProfessional Mail Surveys Company

7432 East Court Avenue

Elveron, CA 90101

(213) 991-5550

Dear Mr. Smythe:

Will you do us a favor?

We are conducting a nationwide survey among executives and managers in the metalworking industry. The purpose of this research is to find out the opinion of experts like you on the advantages and disadvantages of using three new steel products. Your answers will enable steel manufacturers to be aware of the requirements of the users and the opinions of the nonusers of these products, and this in turn will help them design the products you need.

Your name appeared in a scientifically selected random sample. Your answers are very important to the accuracy of our research, whether or not your company is a user of one or more of the products described.

It will take only a short time to answer the simple questions on the enclosed questionnaire and to return it in the stamped reply envelope.

Of course all answers are confidential and will be used only in combination with those of other metalworking executives and managers.

If you are interested in receiving a report on the findings of this research, just write your name and address at the end of this questionnaire, or if you prefer, request the results of the Survey on Steel Products in a separate letter. We will be glad to send you a complimentary report when it is ready.

Please return the completed questionnaire at your earliest convenience. Thank you for your help!

Sincerely,

James B. Jones

James B. Jones

Director

P.S. The enclosed dollar bill is just a token of appreciation.

And finally, remember the golden rule….

Do unto your respondents as you would have them do unto you!!

Team Assignment #2

** Don’t start on this assignment until you’ve read Fowler (CH 6-7)

• Based on your research design1. Write a paragraph about what your

measurement instrument is supposed to accomplish

2. Make a list of what should be measured to accomplish the goals of the study

3. Develop your measurement instrument

Team Assignment #2

• Deliverables include:1. A very clean, polished version that you

could use to actually collect data• This means you will need to carefully think

through all of the issues we covered tonight (e.g., set-up, ordering, length, multi-item scales, etc.)

2. Intro paragraph and variable list (see previous slide)

(Note: Don’t worry about defining your sample--you’ll have your chance to do that next week)