1

ALWAYS BE TESTING

How eROI + Taco Bell Built a Testing Agenda(and What Happened Next)

Cher FullerSENIOR DIGITAL STRATEGIST

@cherfuller

Christine LiE-COMMERCE INNOVATION

@li_tacobeta

FULL BLEED ANCHORS

MARGINS

MARGINS

2

3

FULL BLEED ANCHORS

MARGINS

MARGINS

“Without data, you’re just another person with an OPINION.”

-W. Edwards Deming, data scientist

Here are some other people with opinions….

4

FULL BLEED ANCHORS

MARGINS

MARGINS

5

FULL BLEED ANCHORS

MARGINS

MARGINS

Product testing TOLEDO, OH.

6

FULL BLEED ANCHORS

MARGINS

MARGINS

Digitally engaged brandsswim in OCEANS OF DATA.

7

FULL BLEED ANCHORS

MARGINS

MARGINS

More data is being collected and analyzed today than at ANY OTHER TIME IN HISTORY.

8

FULL BLEED ANCHORS

MARGINS

MARGINS

2.5 billion gigabytesof data are collected

every day.

90% of globally stored data was collected in

the past two years.

IBM: What is Big Data? https://www-01.ibm.com/software/data/bigdata/what-is-big-data.html

9

A

B A

B

By A/B TESTING, right?

1 2 3 4 5

10

FULL BLEED ANCHORS

MARGINS

MARGINS

But A/B testing is step 5.

1 2 3 4 5

11

FULL BLEED ANCHORS

MARGINS

MARGINS

But A/B testing is step 5.

You need to start at step 1.

12

FULL BLEED ANCHORS

MARGINS

MARGINS

To swim this ocean of data,first TEST THE WATER.

13

Build a Learning Agenda

FULL BLEED ANCHORS

MARGINS

MARGINS

14

FULL BLEED ANCHORS

MARGINS

MARGINS

A LEARNING AGENDA asks specific questions in a specific order to set the framework for a testing strategy.

LEARNING AGENDA

FULL BLEED ANCHORS

MARGINS

MARGINS

15

HIERARCHYWHY

Information hierarchy strongly influences user engagement.

By testing how prominently we present messages, we can measure what our users really care about—and why.

BENEFITS CONTENT TYPESWHY

To understand consumer incentive, we must pinpoint the benefit to online ordering.

Without this incentive, online ordering loses its value and risks shifting to in-store purchases.

WHY

Test engagement within several modules against a control to learn if specific types of content are more engaging than

others.Once we understand the value of certain modules, we can

make strategic decision around when/where to place them in triggered emails.

Hierarchy

Benefits Content Types

Information hierarchy strongly influences user engagement.

By testing how prominently we present messages, we can measure what our users really care about—and why.

Create an outlineThe success of a deck relies on a solid outline, which should happen well before you create a new deck.Drafting a deck outline will not only prevent hours of unused work, but it will be the nucleus for team alignment and effective storytelling.

The deck outline can also serve as a helpful means to define who speaks to what points, and function as speaking notes!

To understand consumer incentive, we must pinpoint the benefit to online ordering.

Without this incentive, online ordering loses its value and risks shifting to in-store purchases.

Gangsta’s ParadiseMINUTE AFTER MINUTE, HOURI’m the kinda G that little homieswant to be like.They keep spending most their lives living in the gangsta’s paradise. They keep spending most their lives living in the gangsta’s paradise. They keep spending most their lives living...They keep spending most their lives living in the gangsta’s paradise.They keep spending most their lives living in the gangsta’s paradise.They keep spending most their lives living in the gangsta’s paradise.

TOO MUCH TELEVISION WATCHING GOT ME CHASING DREAMS.I’M EDUCATED FOOL WITH MONEY ON MY MIND.

Test engagement within several modules against a control to learn if specific types of content are more engaging than others.

Once we understand the value of certain modules, we can make strategic decision around when/where to place them in triggered emails.

16

Before starting any learning agenda, ask...

1. What are you testing?2. Why are you testing it?3. What do you want to learn?4. What will you do with your insights?

FULL BLEED ANCHORS

MARGINS

MARGINS

P

StHa

Before starting any LEARNING AGENDA, ask…

What do you want to learn?

What will you do

with your insights?

What are you

testing?

Why are you

testing it?

What are you testing?

What do you want to learn?

Why are you testing it?

What will you do with your insights?

1

2

3

4

17

SEE TEXT ON SLIDEFULL BLEED

ANCHORS

MARGINS

MARGINS

P

StHa Why does a brand care about the DATA STORY?

18

Where We Sometimes Misstep

FULL BLEED ANCHORS

MARGINS

MARGINS

19

FULL BLEED ANCHORS

MARGINS

MARGINS

NOT EVERYONE speaks data.We need to translate it.

20

WHAT WE WILL COVER TODAY

FULL BLEED ANCHORS

MARGINS

MARGINS

Make the complexSIMPLE. A LEARNING AGENDA

might test:

21

FULL BLEED ANCHORS

MARGINS

MARGINS

TELL STORIES,not reports.

22

So We RevisualizedThe Learning Agenda.

FULL BLEED ANCHORS

MARGINS

MARGINS

23

FULL BLEED ANCHORS

MARGINS

MARGINS

BEFORE AFTER

24

SEE TEXT ON PAGE

FULL BLEED ANCHORS

MARGINS

MARGINS

P

StHa This learning agendahelped Taco Bell’s brand teamTAKE TRULY DATA-DRIVEN ACTIONS.

25

We RevisualizedOur Performance Reports Too

FULL BLEED ANCHORS

MARGINS

MARGINS

26

FULL BLEED ANCHORS

MARGINS

MARGINS

Digital Strategists

Performance Analysts

27

FULL BLEED ANCHORS

MARGINS

MARGINS

P

StHa PERFORMANCE REPORTSinfluence brand decisions

PERFORMANCEReports

29

FULL BLEED ANCHORS

MARGINS

MARGINS

qualitative data branding operations quantitative data

performanceAudienceBrand positionMessage / toneVisual style directionContent strategy

30

Data Storytelling Opens the Door to Bigger (!) Creative Possibilities

FULL BLEED ANCHORS

MARGINS

MARGINS

TOP Emails 2016

31

FULL BLEED ANCHORS

MARGINS

MARGINS

headline

CTA

headline

CTA

copy

1

2

3

4

5

CYBER MONDAYDelivered 42% more clicksthan average

And a 24% higher click-to-open rate than average

FULL BLEED ANCHORS

MARGINS

MARGINS

32

FULL BLEED ANCHORS

MARGINS

MARGINS

NAKED CHICKEN CHALUPABy placing the secondary messagefirst, we gained +10% total clicksand 2x higher conversions.

34

To Recap

FULL BLEED ANCHORS

MARGINS

MARGINS

35

FULL BLEED ANCHORS

MARGINS

MARGINS

TO RECAPData is cool

36

FULL BLEED ANCHORS

MARGINS

MARGINS

TO RECAPFirst ask WHY, then WHAT

37

FULL BLEED ANCHORS

MARGINS

MARGINS

TO RECAPTell stories, not reports

38

FULL BLEED ANCHORS

MARGINS

MARGINS

TO RECAPCollaborate!

39

FULL BLEED ANCHORS

MARGINS

MARGINS

TO RECAPInformed creative > Pretty things

40

Any questions?

FULL BLEED ANCHORS

MARGINS

MARGINS