Issues in Improving Issues in Improving Responsible Retailing Responsible Retailing

PracticesPracticesJoel W. GrubeJoel W. Grube

Prevention Research CenterPrevention Research CenterPacific Institute for Research and Pacific Institute for Research and

EvaluationEvaluation

Complex Factors Affect Sales RatesComplex Factors Affect Sales Rates

ClerkStore

Characteristics

Corporate Characteristics

Context

Broader Legal and Social Environment

Probability of at Least One SaleProbability of at Least One SaleGiven True Compliance RateGiven True Compliance Rate

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Mystery Shopper Observations

0.5

0.6

0.7

0.8

0.9

0.95

0.975

0.99

ActualCompliance

Rate

Pro

babi

lity

of a

t le

ast

one

sale

Number of Observations Needed to Number of Observations Needed to Describe Prevalence of Sale for Describe Prevalence of Sale for

Individual StoreIndividual Store

0

50

100

150

200

250

300

350

400

0.50 0.60 0.70 0.80 0.90

Actual Compliance Rate

+/- 15%

+/- 10%

+/- 5%Ob

serv

atio

ns

95%Confidence

Interval

Why Does it Matter?Why Does it Matter?

The complexity and variability create uncertainty The complexity and variability create uncertainty when:when:

Evaluating program effects—How do we know Evaluating program effects—How do we know what works?what works?

Monitoring store performance—How do we Monitoring store performance—How do we know when a store has improved?know when a store has improved?

Comparing stores—How do we know one store Comparing stores—How do we know one store is doing better or worse?is doing better or worse?

Voluntary Compliance ProgramsVoluntary Compliance Programs

Acting under the authority of Consumer Acting under the authority of Consumer Protection laws, state Attorneys General have Protection laws, state Attorneys General have engaged national retail chains to intensify their engaged national retail chains to intensify their efforts to prevent underage sales of tobacco efforts to prevent underage sales of tobacco products products

National drug store, discount and convenience National drug store, discount and convenience store/gas station chains have executed store/gas station chains have executed Assurances of Voluntary Compliance (“AVC”) in Assurances of Voluntary Compliance (“AVC”) in which they stipulate to changes in hiring, which they stipulate to changes in hiring, training and point-of-sales protocoltraining and point-of-sales protocol

Voluntary Compliance ProgramsVoluntary Compliance Programs

Evaluation shows clerks and managers have:Evaluation shows clerks and managers have: High knowledge of laws and store policiesHigh knowledge of laws and store policies High perceived self-efficacyHigh perceived self-efficacy Strong personal support for the goal of Strong personal support for the goal of

preventing underage tobacco salespreventing underage tobacco sales

Nevertheless:Nevertheless: Monitoring shows that clerks fail to perform age-Monitoring shows that clerks fail to perform age-

verification protocols correctly in 20% - 30% of verification protocols correctly in 20% - 30% of attempted tobacco purchasesattempted tobacco purchases

Source: Krevor, Prottas & Fournier, 2006

Voluntary Compliance ProgramsVoluntary Compliance Programs

Compliance Rates by Quarter

2004 Quarter 1 83.7

Quarter 2 69.4

Quarter 3 69.5

Quarter 4 68.8

2005 Quarter 1 72.7

Quarter 2 77.8

Quarter 3 75.4

Mystery Shopper ReportsMystery Shopper Reports

Mystery Shopper (“MS”) reports are being Mystery Shopper (“MS”) reports are being considered by Attorneys General as an considered by Attorneys General as an additional stipulation in AVCsadditional stipulation in AVCs

Mystery Shopper Reports provide detailed Mystery Shopper Reports provide detailed feedback on observed age-verificationfeedback on observed age-verification

There is no evidence at present whether There is no evidence at present whether frequent MS reports improve age-verificationfrequent MS reports improve age-verification

Mystery Shopper Evaluation ProjectMystery Shopper Evaluation Project

BrandeisBrandeis

Brad KrevorBrad Krevor

Stephen Fournier Stephen Fournier

Jeffery Prottas Jeffery Prottas

PIREPIRE

Joel GrubeJoel Grube

William PonickiWilliam Ponicki

Mystery Shopper Evaluation ProjectMystery Shopper Evaluation Project

The primary aims of the study are:The primary aims of the study are: Do frequent (monthly) MS reports increase correct Do frequent (monthly) MS reports increase correct

age-verification conduct?age-verification conduct? Do the effects of intensive MS reporting persist?Do the effects of intensive MS reporting persist? How do managers use the reports?How do managers use the reports?

Identify store, purchaser, and clerk characteristics Identify store, purchaser, and clerk characteristics related to age-verification conduct.related to age-verification conduct.

Hypothesis: Hypothesis: Increased feedback will raise the salience of Increased feedback will raise the salience of

tobacco age-verification at the store and district tobacco age-verification at the store and district level leading to higher rates of age-verificationlevel leading to higher rates of age-verification

Mystery Shopper Evaluation ProjectMystery Shopper Evaluation Project Randomly assign 206 CORS to treatment and Randomly assign 206 CORS to treatment and

controlcontrol All CORs inspected by Mystery Shoppers twice All CORs inspected by Mystery Shoppers twice

a month for 13 months a month for 13 months Treatment conditionTreatment condition: In Months 3 –10 every 2nd : In Months 3 –10 every 2nd

inspection by the Mystery Shopper will be inspection by the Mystery Shopper will be reported reported

Control ConditionControl Condition: Every 6th inspection will be : Every 6th inspection will be reported reported

Telephone interviews with store and district Telephone interviews with store and district managers in months 6 and 9 managers in months 6 and 9

In months 11 - 13, the twice-monthly MS In months 11 - 13, the twice-monthly MS inspections will continue, but no MS reportsinspections will continue, but no MS reports

Mystery Shopper Evaluation ProjectMystery Shopper Evaluation Project

There are 206 CORS. For each COR there will be There are 206 CORS. For each COR there will be observations at 26 time pointsobservations at 26 time points

A total of 5,356 MS observations A total of 5,356 MS observations Sufficient observations for detecting a 5% Sufficient observations for detecting a 5%

treatment impact (75% to 80%) over the treatment impact (75% to 80%) over the treatment period and a 10% (75% to 85%) impact treatment period and a 10% (75% to 85%) impact within the shorter post-treatment period within the shorter post-treatment period

SummarySummary Accurately measuring individual store Accurately measuring individual store

performance is very difficultperformance is very difficult Measures are highly variableMeasures are highly variable Performance is affected by a wide range of Performance is affected by a wide range of

factorsfactors Adequately capturing individual store Adequately capturing individual store

performance requires a large number of performance requires a large number of measures at different times, days, and with measures at different times, days, and with different clerksdifferent clerks

Performance at the aggregate level (e.g., chain, Performance at the aggregate level (e.g., chain, community) more easily measuredcommunity) more easily measured

Despite difficulties, it is possible to measure Despite difficulties, it is possible to measure performance and evaluate programs (e.g., MS)performance and evaluate programs (e.g., MS)