Macro Consulting - Sawtooth Conference 2013 Image MD Presentation

MACRO Consulting, Inc.

A New Approach

Brand Imagery MEASUREMENT

Paul Richard McCullough

Sawtooth Software Conference 2013


3MACRO Consulting, Inc. w w w . m a c r o i n c . c o m

Ricky Odello, Survey Sampling International

Tom Eagle, Eagle Analytics of California

Kirill Zaitsev, MACRO Consulting, Inc.

Keith Chrzan, Sawtooth Software, Inc.

Christine Lafontaine, MACRO Consulting, Inc.YOUTHANK


AGENDA

MACRO Consulting, Inc. w w w . m a c r o i n c . c o m

Introduction New Approach Case Study

Brand Imagery Measurement

• Current Approach• Issues

• Research Objectives• Research Methodology• Summary of Findings


AGENDA


Introduction


• Current Approach• Issues


Current Approach

Introduction

• Rating Scales

• Each brand is rated independently on each statement in an image battery, eg, 10 point rating scale.


Current Approach Issues

Introduction

1

2

3

4

5

6

7

8

9

10

• Flat Responses Across Statements

• Flat Responses Across Brands

• Brand Halo

• Scale Usage Bias

Ratings Scales

Resulting data are typically non-discriminating and highly correlated.


AGENDA


New Approach



Brand-Anchored Max/Diff

New Approach

Brand-anchored Max/Diff removes brand halo, scale-usage bias and is more discriminating than rating scales.


Brand-Anchored Max/Diff with Dual Response

New Approach

Dual Response Max/Diff allows for a zero point in Max/Diff utilities, making comparisons across studies (and brands) feasible.


Modified Brand-Anchored Max/Diff

New Approach

Max/Diff takes longer than ratings scales. Modified brand-anchored Max/Diff hopes to decrease the interview time of the Max/Diff Tasks.


Animated Modified Brand-Anchored Max/Diff

New Approach

Animated Modified Brand-Anchored Max/Diff hopes to hold the respondent’s attention longer than traditional Max/Diff.


Direct Binary Response- Positive DBR

New Approach

Dual Response Max/Diff ALLOWS FOR A ZERO POINT in Max/Diff utilities, MAKING COMPARISONS ACROSS STUDIES feasible.

Direct Binary Response is a MORE TIME-EFFICIENT way to collect dual-response data.

However, Dual Response Max/Diff has been shown to RE-INTRODUCE SOME SCALE USAGE BIAS.


Dual Direct Binary Response- Negative DBR

New Approach

By ADDING A SECOND, NEGATIVE DIRECT BINARY RESPONSE QUESTION, we hope to REMOVE or MINIMIZE scale usage bias.

As a FURTHER ATTEMPT to minimize scale use bias, half of respondents will be required to SELECT AS MANY NEGATIVE ATTRIBUTES AS POSITIVE.

15MACRO Consulting, Inc.

A n i m a t e d M o d i f i e d B r a n d - a n c h o r e d M a x / D i f f S c a l i n g w i t h P o s i t i v e a n d N e g a t i v e D i r e c t B i n a r y R e s p o n s e

Summary of New Approach

A n i m a t e d M o d i f i e d B r a n d - a n c h o r e d M a x / D i f f S c a l i n g w i t h P o s i t i v e a n d N e g a t i v e D i r e c t B i n a r y R e s p o n s e ( A M B A M B R )


Analytics-Derived Parsimony

New Approach

Latent Class Choice Models

• With Large Sample• With Covariates

Hierarchical Bayes

• Covariates in upper model• Adjusted priors

2Alternatives

The goal of the above analytic approaches is to minimize the number of Max/Diff tasks each respondent must complete and still estimate disaggregate utilities with acceptable accuracy.


AGENDA


Case Study


• Research Objectives• Research Methodology• Summary of Findings


Research Objectives

• Compare two approaches to brand imagery measurement, ratings scales and max/diff, in terms of:- Inter-brand discrimination- Inter-item discrimination- Predictive validity

• Explore alternative methods of estimating max/diff utilities most accurately and most efficiently:- Standard HB - HB with positive Direct Binary Response- HB with positive DBR and unconstrained

negative DBR- HB with positive DBR and constrained

negative DBR- Latent Class Choice- Use of covariates- Tuned priors

MACRO Consulting, Inc. 18w w w . m a c r o i n c . c o m


Online Survey:

• Two cells- Rating Scales (n=436)- Max/Diff (n=2,605)

• Three brands

• 12 items• Questionnaire:

- Brand image measurement- Three dependent variables

* Item top 3 rank-order* Brand purchase likelihood* Brand forced-choice preference

- Demographics

Research Methodology

w w w . m a c r o i n c . c o m 19

MACRO Consulting, Inc. w w w . m a c r o i n c . c o m MACRO Consulting, Inc. 20

Summary of Findings

w w w . m a c r o i n c . c o m

In general, AMBAMBR is superior to ratings scales:

- Better inter-item discrimination- Better predictive validity- Fewer unacceptable respondents- Elimination of both brand halo and scale usage bias

Of the AMBAMBR methods tested, the two methods which included negative DBR were superior:

- Positive DBR reinserts brand halo into the data- Positive DBR has slightly weaker inter-item discrimination than either

Negative DBR

AMBAMBR takes longer to administer and has higher incompletion rates

Task set reduction could not be fully explored with these data

MACRO Consulting, Inc. w w w . m a c r o i n c . c o m21

Positive DBR Appears to Show Greater Inter-Item Discrimination Than Rating Scales

Rating Scales

Positive DBR

Item

1Ite

m2

Item

3Ite

m4

Item

5Ite

m6

Item

7Ite

m8

Item

9

Item

10

Item

11

Item

12

5.00

5.50

6.00

6.50

7.00

7.50

8.00

8.50

Brand#1

NewBrand

Brand#2

Item

1Ite

m2

Item

3Ite

m4

Item

5Ite

m6

Item

7Ite

m8

Item

9

Item

10

Item

11

Item

12

-3.00

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

Brand#1

NewBrand

Brand#2


22

Negative DBR Approaches Yield Similar Results

ConstrainedNegative DBR

UnconstrainedNegative DBR

Item

1Ite

m2

Item

3Ite

m4

Item

5Ite

m6

Item

7Ite

m8

Item

9

Item

10

Item

11

Item

12

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

Brand#1

NewBrand

Brand#2

Item

1Ite

m2

Item

3Ite

m4

Item

5Ite

m6

Item

7Ite

m8

Item

9

Item

10

Item

11

Item

12

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

Brand#1

NewBrand

Brand#2


Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

10

Item

11

Item

12

5.00

5.50

6.00

6.50

7.00

7.50

8.00

8.50

Brand#1

NewBrand

Brand#2

ConstrainedNegative DBR

UnconstrainedNegative DBR

Rating Scales

Positive DBR

Negative DBR Approaches Bring New Brand Closer

Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

10

Item

11

Item

12

-3.00

-2.50

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

Brand#1

NewBrand

Brand#2

Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

10

Item

11

Item

12

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

Brand#1

NewBrand

Brand#2

Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

10

Item

11

Item

12

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

2.50

Brand#1

NewBrand

Brand#2


Ratings No DBR Positive DBR Unconstrained Negative DBR

Constrained Negative DBR

BRAND#1 1.75 4.46 3.90 4.30 4.68

NEW BRAND 0 4.28 3.16 4.25 4.50

BRAND#2 1 4.69 3.78 4.48 4.70

Inter-Item Discrimination Greatest for Negative DBR

Average number of statistically significant differences across 12 items, within brand*

* 10 random draws of n=436 were pulled for all data sets except Ratings


Random Numbers Ratings No DBR Positive DBR Unconstrained

Negative DBRConstrained

Negative DBR

1 OF 1 8% 14% 27% 28% 27% 26%

(1 OR 2) OF 2 32% 30% 62% 64% 62% 65%

(1, 2 OR 3) OF 3 61% 51% 86% 87% 86% 88%

Predictive Validity of AMBAMBR Superior to Rating Scales

Hit Rates for Top 3 Items Ranking


AMBAMBR Yielded More Valid Completes

Invalid Completes

Max/Diff 4%Ratings 32% 32%


Brand Halo Was Measured Using Confirmatory Factor Analysis

If brand halo exists, halo latent will positively

influence scores on all items


Brand Halo Latent



Std Beta Prob Std Beta Prob Std Beta Prob Std Beta Prob Std Beta Prob

ITEM 1 0.85 *** -0.14 *** 0.90 *** 0.44 *** 0.27 ***

ITEM 2 0.84 *** -0.38 *** 0.78 *** -0.56 *** -0.72 ***

ITEM 3 0.90 *** -0.20 *** 0.95 *** 0.42 *** 0.32 ***

ITEM 4 0.86 *** 0.10 *** 0.90 *** 0.30 *** 0.16 ***

ITEM 5 0.77 *** -0.68 *** 0.88 *** 0.03 0.25 0.01 0.78

ITEM 6 0.85 *** -0.82 *** 0.87 *** -0.21 *** -0.24 ***

ITEM 7 0.83 *** 0.69 *** 0.83 *** 0.42 *** 0.20 ***

ITEM 8 0.82 *** 0.24 *** 0.75 *** 0.01 0.87 -0.23 ***

ITEM 9 0.88 *** 0.58 *** 0.90 *** 0.77 *** 0.62 ***

ITEM 10 0.87 *** 0.42 *** 0.94 *** 0.86 *** 0.90 ***

ITEM 11 0.77 *** -0.05 0.015 0.85 *** 0.07 0.02 -0.12 ***

ITEM 12 0.88 na 0.26 na 0.91 na 0.69 na 0.53 na

Ratings and Positive DBR Reflect Strong Brand Halos


Scale Usage Bias Was Measured Using Confirmatory Factor Analysis

Brand halo drives scores within brand. Scale

usage bias drives scores independent of brand.

If scale usage bias exists, the scale usage

latent should load positively on all items


Scale Usage Latent



NUMBER OF NEGATIVE LOADINGS

0 14 5 10 15

NUMBER OF STATISTICALLY SIGNIFICANT

LOADINGS

35 30 28 32 29

Only Ratings Reflect Strong Scale Usage Bias


AMBAMBR Superior But SlowerAMBAMBR Has Higher Dropout Rates

RATINGS AMBAMBR

TOTAL INTERVIEW LENGTH 9.7 MINUTES 15.8 MINUTES

BRAND IMAGE MEASUREMENT 1.7 MINUTES 6 MINUTES

RATINGS AMBAMBR

INCOMPLETION RATE 9% 31%?

Can We Reduce the Number of Max/Diff Tasks to Shorten Interview

Length and Decrease Dropout Rates?


U N C O N S T R A I N E D Negative DBR C O N S T R A I N E D Negative DBR

8 Tasks 4 Tasks 2 Tasks 8 Tasks 4 Tasks 2 TasksHB LC HB LC HB LC HB LC HB LC HB LC

1 OF 1 27% 19% 21% 20% 20% 19% 26% 21% 24% 21% 22% 22%

(1 OR 2) OF 2 62% 54% 59% 57% 58% 56% 65% 61% 61% 59% 59% 56%

(1, 2 OR 3) OF 3 86% 81% 85% 82% 82% 83% 88% 84% 86% 84% 85% 82%

HB Models May Perform Slightly Better Than LC With Full Task SetsBut All Perform Well


All HB and LC Models Perform Very Well -

• Hit rates seem relatively unaffected by:

− Number of tasks

− Number of latent classes

− Tuned priors

− Covariates

− Sample size

What’s Going On?

Perhaps Too Well


Perhaps These Data Have Little Heterogeneity

• Category is not emotionally engaging

• Brands are not differentiated− Commodity-like category

− No polarizing brands, eg, Microsoft, Apple or Donald Trump

− Brands with new technologies not yet established


Aggregate Model Works as Well as Disaggregate

U N C O N S T R A I N E D Negative DBR

RandomHB

8 TasksN=1,324

HB2 TasksN=105

HB2 Tasks

Constant utilsN=105

1 OF 1 8% 27% 22% 25%

(1 OR 2) OF 2 32% 62% 59% 61%

(1, 2 OR 3) OF 3 61% 86% 82% 82%

This seems to suggest

there is little

heterogeneity to

capture


Summary and Implications

• The forms of Max/Diff referred to here as AMBAMBR are superior to rating scales for measuring brand imagery:- Better inter-item discrimination- Better predictive validity- Elimination of brand halo- Elimination of scale usage bias- Fewer invalid completes

• Positive DBR alone reintroduces brand halo

• Positive DBR must be combined with some form of negative DBR

• For comparability across brands and time: - Raw utils must be used rather than rescaled utils- Some form of dual response must be used

• AMBAMBR takes longer to administer and has higher incompletion rates


Further Research

• Can we reduce the number of tasks when brands are heterogeneously perceived?

• How many brands and statements can realistically by accommodated?

• Is constrained or unconstrained negative DBR superior?

• How would traditional dual response format affect these results?

• Is there a better way to evaluate utility performance?


And The BIG Question:

• Is there a shorter name for this technique than:

Animated Modif ied Brand-anchored Max/Diff Scal ing

with Posit ive and Negative Direct Binary Response?

39

Thank you

Marketing

Macro Consulting - Sawtooth Conference 2013 Image MD Presentation