Introducing OptimizelyProgram Management

Claire Vo, VP of Product, OptimizelyHeather Miller, Senior Experimentation Analyst, BBC

SCALE YOUR PROGRAM AND 5X YOUR EXPERIMENT VELOCITY

3

Heather MillerSenior Experimentation Analyst

BBC

Claire VoVice President, Product

Optimizely

Agenda

● Introduction

● Optimizely Program Management

● Demo

● BBC’s Experimentation Program

● Q&A

“Our success is a function of how many experiments

we do per year, per month, per week, per day.”

“Instead of saying ‘I have an idea,’ what if you said

‘I have a new hypothesis, let’s go test it.’

“Our company culture encourages

experimentation and free flow of ideas.”

3

“One of the things I’m most proud of, and I think

what is the key to our success, is this testing

framework we’ve built.”

Experimentation is the Next Great Business Transformation

Jeff Bezos Larry Page

Mark Zuckerberg Satya Nadella

The Surprising Power of Online Experiments

Digital Experience Optimization:Digital Products, Commerce & Campaigns

Up to 5X Increase in Yield:Revenue, Share of wallet, Funnel conversion, Risk mitigation, Ops efficiency

Partner of Choice:Work with leading global enterprises & “digital disruptors” including 26 of F100

OUR COMPANY

Digital Experimentation Platform:Next gen “Test and Learn” System

Replaces Digital Guesswork: with Evidence-based optimization

Speeds Innovation & Optimization:Single platform for marketing & product teamsBest in class stats & machine learningConsumer grade usability Enterprise program management & prof. services

2

OUR SOLUTION

13

Global Companies Experimenting at Scale with Optimizely

Building an experimentation program can be challenging

PrioritizationIdeation can be siloed leading to inefficient prioritization and execution

Oversight & knowledge sharingLack of executive oversight can lead to serious problems in a global organization

CoordinationAs you scale, coordinating across additional teams and stakeholders becomes more and more difficult

ResourcesTime, people, and effort is wasted duplicating and reinventing different processes

Challenges of Building a Program

IDEATION & PRIORITIZATION

TESTING & IMPLEMENTATION

REQUIREMENTS GATHERING & DEVELOPMENT

REPORTING & ANALYSIS

DOCUMENTATION & IP PRESERVATION

Does this reporting match internal data?

What are we going to test next?

Can someone code this test for me?

Program Management

Scale your experimentation program across your entire enterprise

Manage your program across the entire experiment lifecycle

Teams Web & Full Stack Experimentation

Program Reporting

OPTIMIZELY PROGRAM MANAGEMENT

IDEATION REQUIREMENTS CREATIVE DEVELOPMENT TESTING ANALYSIS SHARING

5x Improved testing velocity through improved collaboration

UP TO

Democratize ideation and empower your entire team to participate

Make the experimentation process more accessible and transparent

Improve your program workflow with new tools for collaboration

TeamsCollaborate more effectively with an integrated hub for your team

Track the velocity of your program

Gain more oversight across projects and teams

Uncover new insights and iterate on your best ideas

Program ReportingGain powerful insights into your program with a new executive dashboard

Program Management at the BBC – with Optimizely

Heather MillerSenior Experimentation and Optimisation Analyst17 January 2018

•The BBC is the oldest and largest broadcaster in the world

•Founded in 1922, we employ 35,402 staff in total across the world

•Our primary audience is the UK - but some of our work reaches over 200 countries

•We deliver a wide range of services, from news, radio, video, music and online content

•The BBC is considered to be of ‘significant cultural importance’ to the UK

“To enrich people’s lives with programmes and services that inform, educate and entertain.”

- John Reith, Founder of the BBC

Experimentation at the BBC: our team•The Experimentation and Optimisation team sits within Data

Insights

•Our team operates as a Centre of Excellence

•There are five us in the Experimentation team, who leverage our expertise to advise and consult with teams across the BBC

•Our team assists with the entire end-to-end process - from strategy, to test design, to analysis, to final reports

•We want to upskill teams and make them self-sufficient in experimentation best practice

•Right now, 13 teams use Optimizely across the whole of the BBC

•Our ultimate goal is to have 40 teams using the tool

•On any given day dozens of people use Optimizely

•We strive to deliver continuous experimentation across web, mobile and connected devices

•Our team maintains a close working relationship with Optimizely to maximise productivity and deliver the best results

● The BBC reaches millions

● We strive to deliver digital excellence

● Experimentation plays a vital role

● Ultimately saves time, money and mitigates risk

● We have run around 200 experiments with

Optimizely

Why does testing matter to the BBC?

Program Management at the BBC

How we generate ideas and prioritise tests

● Test ideas are generated and prioritised at product level

● Qualitative & Quantitative: data insights, usability work, internal ideas and previous tests

● Tests are prioritised based on wider BBC Objectives, product roadmaps and the PIE Model

Why knowledge sharing is important to us● Sharing insights plays a key role in upscaling our

testing programme to new teams

● We deliver a monthly newsletter to 300 subscribers and counting

● Confluence knowledge hub with historical result reports helping teams make informed decisions

● Upskilling teams via our bespoke training supports our adoption of a consistent approach to experimenting

● Host regular internal showcases, engineering meet ups and workshops

The challenges we face

As we grow our programme across more teams and touch points we face some challenges

• Multiple tools are used to document ideas and information

• Different teams, different preferences leading to multiple places with similar information

• Ability to measure maturity, and track experiment end-to-end velocity as a BBC overview and product level

• A need for splitting out BBC tests by type to identify expansion into other forms of testing

• Demonstrating and visualising test scale across the BBC

How Program Management

will help the BBC

As we continue to increase our pace of innovation, we are excited about what Program Management has to offer.

• Consolidate multiple tools into a single slick environment

• One stop shop for past results which are easily found

• Improves collaboration and encourages transparency, allowing anyone within a product to add ideas

• Drives efficiency of our end-to-end workflow

• Offers progress reporting at BBC level and product level

• Supports our goal to scale Optimizely to more teams

Learn More

Visit optimizely.com/program-management