Strata New York 2012: Continuous Experimentation with Continuous Deployment

Continuous Experimentationwith Continuous

Deployment

Steve MardenfeldEtsy

Monday, November 5, 2012


World’s Handmade Market


Continuous Deployment




Metrics


Graph Factory


Oops


Oops


Good Enough?


What is Better?


Sometimes it’s Obvious


Most Times it’s Not



Cart Page Restyled



Etsy Mentioned on Oprah

Internet Goes Down in Brooklyn

Justin Beiber Sneezes

New Cat Meme

Cart Page Restyled


A/B Testing?Monday, November 5, 2012



Test Them Both


Statistics?


First Pass

HADOOP


First Pass

CASCADING


First Pass

R


First Pass

RHADOOP

CASCADING


Automate It

A/B Analyzer


A/B Analyzer

•Framework for looking at experiments

•Done for every experiment beforehand


A/B Analyzer


A/B Analyzer



3 Types of Metrics

•Co-occurence

•visits that did X (searched)

•average number of X (searches)

•Funnels

•Special Cases*


Co-Occurence

•Co-occurence of all actions with the variant

•i.e. - how many people saw this variant and then purchased

•i.e. - average number of purchases per variant


Funnels

BuySearch Listing Buy


Funnels

Buy


Search Listing BuyHome


Funnels

BuyBuy


Search Listing BuyHome


Special Cases

•Exiting pages

•Bounce rate

•Visit length


Pipeline

HDFS Events Visits MySql

A/B

PROCESSING

Client

Logs

DB

COLLECTION


Bucketing

Home Search Listing Buy

Search A/B Test


Bucketing

Home Search Listing Buy

Search A/B Test


Process

•Idea

•Hypothesis

•Deploy

•Wait

•Analysis


Process

•Idea

•Hypothesis

•Deploy

•Wait

•Analysis


Idea


Changing Search


Related Searches?


Process

•Idea

•Hypothesis

•Deploy

•Wait

•Analysis


Be Specific

•What does an improvement look like?

•More clicks?

•Less searches?

•Shorter visits?

•Longer visits?


Bet Driven Development


Is this even worth it?


Process

•Idea

•Hypothesis

•Deploy

•Wait

•Analysis


Code It Up


Code


Turn it On


Process

•Idea

•Hypothesis

•Deploy

•Wait

•Analysis


Realtime?


Wait


Process

•Idea

•Hypothesis

•Code

•Wait

•Analysis


Making Sense of ResultsMonday, November 5, 2012

Back To Related Searches


Back To Related Searches




Final Thoughts

•Bet driven development - be explicit about your changes

•Easy experimental setup and quick analysis makes for more experiments

•An experiment is successful if you learned something, not if you launched



http://www.slideshare.net/powerlexis/retro-slides-from-ibm




Technology

Strata New York 2012: Continuous Experimentation with Continuous Deployment