Opticon 2015- Benchmark Your Optimization Program

Benchmark Your Optimization Program Shana RusonisContent Marketing Manager, Optimizely

@srusonisshana@optimizely.com

#opticon2015

#opticon2015@srusonis

optimize.ly/optimizationbmrkSlides will be posted, and this session is also being recorded.

“What gets measured, gets managed.”

—Pete Drucker

Stories

Data

#opticon2015@srusonis

Overview of today’s session

• Debuting industry research from Optimizely

• Key findings: Benefits & budget

• Key findings: People behind optimization

• Key findings: Processes for optimization

• Implications & takeaways

#opticon2015@srusonis

Let’s collect some data.

How does optimization impact businesses?

#opticon2015@srusonis

Benchmark Methodologyn = 417

1/4 Large Enterprise

1/2 Mid-Size to Small Enterprise

1/4 Small Businesses

Company Size

1/4 Executives

1/2 Managers

1/4 Associate, Consultants

Level

Marketers

Product Managers

Engineers

Titles

#opticon2015@srusonis

94%Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers6%

Do you agree or disagree with the following statements about optimization?

93%Helps me to deliver relevant content to my customers

Increases my conversion and engagement rates7%

Do you agree or disagree with the following statements about optimization?

87%Allows me to take action on data quickly

Has created a culture of learning and experimentation at my company13%

Do you agree or disagree with the following statements about optimization?

87%Has created a culture of learning and experimentation at my company

Improves the creative output of my team13%

Do you agree or disagree with the following statements about optimization?

91%Increases my conversion and engagement rates

9%

Do you agree or disagree with the following statements about optimization?

88%Improves the ROI of my marketing campaigns

Allows me to take action on data quickly12%

Do you agree or disagree with the following statements about optimization?

80%Improves the creative output of my team

Improves my team’s speed to execute on ideas20%

Do you agree or disagree with the following statements about optimization?

98%

94%

93%

91%

88%

87%

87%

80%

76%

Is valuable to the success of my marketing campaigns and/or product strategy

Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers

Increases my conversion and engagement rates

Improves the ROI of my marketing campaigns

Allows me to take action on data quickly

Has created a culture of learning and experimentation at my company

Improves the creative output of my team

Improves my team’s speed to execute on ideas

6%

7%

9%

12%

13%

13%

20%

24%

Agree

Disagree

Do you agree or disagree with the following statements about optimization?

98% of optimizers agree that it’s valuable to their success

98%Is valuable to the success of my marketing campaigns and/or product strategy

Do you agree or disagree with the following statements about optimization?

#opticon2015@srusonis

… Along with many other qualitative and quantitative benefits

94%Improves my understanding of my customer’s needs and desires

Helps me to deliver relevant content to my customers6%

Do you agree or disagree with the following statements about optimization?

87%Has created a culture of learning and experimentation at my company

Improves the creative output of my team13%

Do you agree or disagree with the following statements about optimization?

91%Increases my conversion and engagement rates

9%

Do you agree or disagree with the following statements about optimization?

88%Improves the ROI of my marketing campaigns

Allows me to take action on data quickly12%

Do you agree or disagree with the following statements about optimization?

80%Improves the creative output of my team

Improves my team’s speed to execute on ideas20%

Do you agree or disagree with the following statements about optimization?

Quantitative Qualitative

#opticon2015@srusonisIncrease

57%61%

43%

How do you plan to change the amount of budget allocated to optimization in the next 12 months?

Optimization budgets are set to increase

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55% 53%

4%

43%

Increase

Decrease

No change

How do you plan to change the amount of budget allocated to optimization in the next 12 months?

#opticon2015@srusonisIncrease

57%61%

43%

How do you plan to change the amount of budget allocated to optimization in the next 12 months?Executives & Managers are more certain of budget increases

• 57% of Executives reported a planned increase

• 61% of Managers reported a planned increase

Executive

Manager

Individual

How do you plan to change the amount of budget allocated to optimization in the next 12 months?

“Optimization has become even more crucial in today's market,

so more budget is required to make a position in the market.”

– Anonymous Survey Respondent

#opticon2015@srusonis

The human capital behind optimization.

#opticon2015@srusonis

#opticon2015@srusonis

Who optimizers are planning to hire

Web Analyst UX DesignerMarketerFront-end Developer

Program Manager

#opticon2015@srusonis

0%

5%

10%

15%

20%

25%

30%

One, full-timeMultiple, full-time

One, part-timeMultiple, part-time

An agencyNone

23%

29%

17%

29%

2% 2%

How many members of your team are focused on optimization?

Multiple,full-time

Multiple,part-time

52% of optimization teams have 1 or more full-time hires

“I think the single best thing to do is to allocate headcount towards optimization.

If you can swing it, hire a dedicated person. If you can’t, get everyone on the team

(designers, engineers, product managers) evangelized. Let everyone run an experiment

so they can see how mission critical it is.”– Kyle Rush, Director of Frontend

Engineering and Optimization, Hillary for America

#opticon2015@srusonis

Optimization draws from a wide range of teams and skills

Marketing62%

Design53%

UI or Front-End Development40%

Product36%

Engineering35%

E-commerce31%

Sales30%

Customer Service or Support23%

Executive Staff20%

Which teams are involved with optimization at your company?

Marketing32%

E-commerce14%

UI or Front-End Development12%

Product12%

Engineering9%

Design8%

Executive Staff5%

Other3%

Customer Service or Support3%

Sales

Which team leads optimization at your company?

#opticon2015@srusonis

Move, Inc. Team Structure

#opticon2015@srusonis

The processes needed for optimization.

How do we scale testing to maximize our impact?

#opticon2015@srusonis

OPTIMIZATION LEAD

Build, QA, and launch

experiments

Conclude and analyze

experiments, share results

Prioritize backlog of

experiment ideas

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency

#opticon2015@srusonis

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency

OPTIMIZATION LEAD

“If you can’t state your reason for running a test simply, then you probably need to examine

why and what you are testing.”

.”

– Brian Schmitt, CROmetrics

#opticon2015@srusonis

Analytics are the top input for optimization

Analytics72%

Results from Previous Experiments57%

Customer Surveys51%

User Research Studies50%

A/B Testing Case Studies44%

Support Tickets or Live Chat Feedback38%

Heat Maps28%

Which of the following inform your optimization strategy?

Analytics72%

Which of the following inform your optimization strategy?

Results from Previous Experiments57%

Which of the following inform your optimization strategy?

Customer Surveys51%

Which of the following inform your optimization strategy?

User Research Studies50%

Which of the following inform your optimization strategy?

A/B Testing Case Studies44%

Which of the following inform your optimization strategy?

A/B Testing Case Studies44%

Which of the following inform your optimization strategy?

Support Tickets or Live Chat Feedback38%

Which of the following inform your optimization strategy?

Heat Maps28%

Which of the following inform your optimization strategy?

#opticon2015@srusonis

… Along with other qualitative inputs and sources of inspiration

Customer Surveys51%

Which of the following inform your optimization strategy?

User Research Studies50%

Which of the following inform your optimization strategy?

Support Tickets or Live Chat Feedback38%

Which of the following inform your optimization strategy?

Heat Maps28%

Which of the following inform your optimization strategy?

Results from Previous Experiments57%

Which of the following inform your optimization strategy?

A/B Testing Case Studies44%

Which of the following inform your optimization strategy?

Qualitative InputsInstitutional &

Industry Knowledge

#opticon2015@srusonis

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency

Prioritize backlog of

experiment ideas

OPTIMIZATION LEAD

#opticon2015@srusonis

Optimizers prioritize for revenue, customer satisfaction

Potential to increase the value of each customer interaction, or increase overall revenue

65%

The impact on ease of use and satisfaction for my customer

58%

Improvement to website or app stability or performance

48%

The inefficiencies that currently exist in my marketing and product funnels

33%

The cost to acquire customers on various platforms

31%

We don’t do this kind of prioritization

2%

How do you prioritize which digital interactions with your audience require optimization?

#opticon2015@srusonis

Optimizers care deeply about the customer experience, in addition to positive ROI.

The values we use to prioritize our tests are a

snapshot of why we choose to optimize in the first place.

“The way your customers use your product and the way they experience it … is everything.”

”

.”

—Lev Tatarov, Marktplaats, eBay

#opticon2015@srusonis

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency

Prioritize backlog of

experiment ideas

Build, QA, and launch

experimentsOPTIMIZATION LEAD

#opticon2015@srusonis

Optimizers are testing a variety of elements, each with different degrees of complexity

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%

Landing Pages61%

Messaging and Copy57%

Forms54%

Offers and Promotions42%

Recommended Content Functionality38%

Social Sharing Functionality37%

Payment Functionality37%

Other

What do you test when you are optimizing?

User Experience Components (Onboarding Patterns, Search Functionality, Modals, etc.)63%

Landing Pages61%

Messaging and Copy57%

Forms

What do you test when you are optimizing?

#opticon2015@srusonis

What you’re likely to test depends on which industry you’re in

• B2B more likely to test landing pages

E-Commerce B2B Media

What do you test when you are optimizing?

65%74%

67%

65%

Landing Pages

Messaging and Copy

What do you test when you are optimizing?

51%46%

53%

44%

Offers and Promotions

What do you test when you are optimizing?

What do you test when you are optimizing?

32%36%

42%

50%

Social Sharing Functionality

Payment Functionality

What do you test when you are optimizing?

• E-commerce more likely to test offers, payments

• Media more likely to test social sharing

#opticon2015@srusonisMultiple Times per Week

WeeklyMultiple Times per Month

MonthlyMultiple Times per Year

37%

21%

15%

12%

9%

Approximately how often do you run A/B or multivariate tests to opti-mize your website, mobile app, or other digital channels?

Multiple Timesper Week

Multiple Timesper Month

58% of optimizers are testing at least weekly

#opticon2015@srusonis

#opticon2015@srusonis

Research to create solid experiment

hypotheses

Processes help scale optimization efficiency

Prioritize backlog of

experiment ideas

Build, QA, and launch

experiments

Conclude and analyze experiments,

share results OPTIMIZATION LEAD

#opticon2015@srusonis

Measuring optimization success is multi-faceted

Conversion Improvements from Experiments

58%

Revenue Generated from Experiments

52%

ROI (Combination of Conversion Improvement and Revenue)

49%

Output or Velocity of Experiments

37%

Number of Team Members or Departments Involved

25%

Other

1%

0 5 10 15 20 25 30 35 40 45 50 55

How do you measure the success of your optimization program?

#opticon2015@srusonis

What does this mean for optimization as

an industry?

#opticon2015@srusonis

An emerging skills market for optimization

#opticon2015@srusonis

Widening gap between optimizing companies and those who have yet to start

#opticon2015@srusonis

How to leverage these findings to improve your optimization.

What can I do today?

#opticon2015@srusonis

• Evaluate the makeup of your team; focus on communication and removing bottlenecks

• Diversify your inputs for optimization beyond analytics

• Maintain a relentless focus on process to improve experiment quality and quantity

Conversation starters and next steps

#opticon2015@srusonis

Focus on building cross-functional teams

#opticon2015@srusonis

optimize.ly/optimizationbmrkDownload the full Benchmark Report

#opticon2015

Thank you!Shana RusonisContent Marketing Manager Optimizely

@srusonisshana@optimizely.com

#opticon2015@srusonis

Template Slides

Appendix

“The goal is to turn data into information, and

information into insight.” – Carly Fiorina, former CEO of HP

“As revenue and budget allowance increases, we plan

to put this money back into better optimization.”

– Anonymous Survey Respondent

#opticon2015@srusonis

Optimization is a cycle of continuous steps:

• Conduct research, uncover conversion points to optimize

• Create hypotheses, suggest variations and improvements

• Build, QA, and launch experiments

• Analyze experiment results

• Share findings and determine next steps

#opticon2015@srusonis

#opticon2015@srusonis

We all have challenges …

• Maximizing ROI

• Delivering custom content

• Making quick decisions

• Understanding, taking action on data

0 10 20 30 40 50 60 70 80 90 100

20%

24%

24%

27%

30%

31%

31%

Maximize the ROI of my resources

Deliver customized experiences to customer segments

Make quick decisions

Take action on company data

Embrace the creative talents of my team

Make data-driven decisions

Understand company data

37%

26%

35%

34%

35%

39%

35%

31%

26%

23%

31%

24%

21%

27%

12%

15%

15%

6%

10%

8%

6%

9%

Very Good Good OK Not Good Very Bad

#opticon2015@srusonis

Color palette

Primary

Secondary

AaAdelle Sans

AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz

0123456789