Funnel Analysis in Hadoop at Etsy

Funnel Analysis in Hadoop at Etsy

Steve Mardenfeld, Wil Stuckey, Matt Walker

Tuesday, March 12, 13


Handmade MarketPlace



Data Driven Development

• Use data to make informed decisions

• Use data to evaluate the efficacy of our products


Funnel AnalysisFunnel Analysis


Registration Funnel


Registration Funnel

5 0 , 0 0 0 1 0 , 0 0 0

1 0 0 % 2 0 %


Registration Funnel

5 0 , 0 0 0 1 0 , 0 0 0

1 0 0 % 6 0 %4 5 , 0 0 0

9 0 %

1 5 , 0 0 0

3 3 %1 0 0 % 9 0 % 2 9 % 2 0 %


Funnels ++

• Funnels are more than just an optimization tool

• Use them to understand different pathways throughout our site

• Partition and compare these pathways by attributes

• A/B tests, categories, queries, cohorts

• Attribution model


Attribution Models

• Tie conversions and successes to specific products and actions

• Use to gain understanding of our users’ interaction with Etsy

• Help us to measure gains in A/B testing

• Easily compare different varieties of the same product

• Attribution techniques for internal and external attribution


Attributions Illustrated



• last product interaction - browse




• co-occurrence - search and browse




• co-occurrence - search and browse

• direct funnel attribution - search


Indirect Attribution Illustrated



• direct funnel attribution - shop home



• direct funnel attribution - shop home

• indirect funnel attribution - search


Aggregate Results

Search Clicked Listing Purchased5 0 , 0 0 0 2 5 , 0 0 0 5 , 0 0 0E v e n t s

Direct


Aggregate Results

Search Clicked Listing Purchased

Search Clicked Listing Shop Clicked Listing

5 0 , 0 0 0

5 0 , 0 0 0

2 5 , 0 0 0

2 5 , 0 0 0

5 , 0 0 0

1 0 , 0 0 0 5 , 0 0 0

E v e n t s

E v e n t s

Direct

IndirectPurchased

3 , 0 0 0


Aggregate Results

Search Clicked Listing Purchased

Search Clicked Listing Shop Clicked Listing

5 0 , 0 0 0

5 0 , 0 0 0

2 5 , 0 0 0

2 5 , 0 0 0

5 , 0 0 0

1 0 , 0 0 0 5 , 0 0 0

E v e n t s

E v e n t s

Search Clicked Listing5 0 , 0 0 0 2 5 , 0 0 0E v e n t s

Direct

Indirect

Combined

Purchased3 , 0 0 0

Purchased8 , 0 0 0


Segmenting Within Funnels

Search Clicked Listing5 0 , 0 0 0 2 0 , 0 0 0C o u n t s

Old AlgorithmPurchased

1 5 , 0 0 0

Search Clicked Listing1 0 0 , 0 0 0 6 0 , 0 0 0C o u n t s

New HotnessPurchased

1 5 , 0 0 0


Segmenting Within Funnels

Search Clicked Listing1 0 0 % 5 0 %S t e p

1 0 0 % 5 0 %To t a l

Old AlgorithmPurchased

4 0 %2 0 %

Search Clicked Listing1 0 0 % 6 0 %S t e p

1 0 0 % 6 0 %To t a l

New HotnessPurchased

2 5 %1 5 %


Segmenting Across Funnels

* Clicked Listing1 0 0 % 5 0 %S e a r c h

1 0 0 % 4 0 %B r o w s e

Purchased4 0 %3 0 %

1 0 0 % 6 0 %H o m e 3 6 %

1 0 0 % 6 2 %A c t i v i t y F e e d

1 0 0 % 4 7 %Ta s t e Te s t2 8 %3 1 %

1 0 0 % 4 5 %S e a r c h A d s 3 8 %


Democratized FunnelsHow do we make this awesomeness available for everyone?



Awesome infrastructure but...

• Must be an engineer to write your own queries

• Engineering resources become the bottleneck

• Hard to scale as the company grows


So we want to:

• Allow the data engineers to focus on higher priority things.

• Allow people to answer their own questions.


How do we do this?


The Path of Funnel Tools

google spreadsheets internal tools

• Funnel Cake

• Feature Funnel


Funnel Cake


Version 1


Real Time


Lets do it in PHP!


Why?


It’s the “Etsy Way”

• Already have existing infrastructure

• Operationally stable

• The same tools everyone else is using means a better adoption rate across Engineering


• Event Stream

• Code runs on every page view

• Simple matching system

• (ab)Used memcached as a temporary storage

• Rolled up to DB every min (near real time)


What happened?


Broader adoption!Over 75 Funnels were created


Shiny Visualized Results


• Funnels had to be setup ahead of time (no backfills)

• Reconciliation is hard

• Limited to events in our web clickstream (ios/android would be excluded)

• Scaling and Operational issues

• Difficult to maintain multiple stacks


Turns out that we don’t make Product decisions in real time

http://mcfunley.com/whom-the-gods-would-destroy-they-first-give-real-time-analytics




Version 2



Getting back on the elephant

• Able to carry over the User Interface from v1

• Standardized event sessionization

• Operationally supported infrastructure

• Nightly batch process


Feature Funnel


Attribution for ALL


Built in metrics

• Click through rate

• Listing impressions

• Purchase rate

• Purchase from shop rate

• Favorite from shop rate

• More...

-


Built in metrics

• Click through rate

• Listing impressions

• Purchase rate

• Purchase from shop rate

• Favorite from shop rate

• More...

- Direct Attribution



- Indirect Attribution

- Indirect Attribution

-


Segmentation

• Primarily used for A/B test analysis

• Arbitrary segmentation


How do we get it?


How do we get it?

select (clicks/visits * 100.0) as "CTR" from feature_funnel where event_type = 'search' and ab_test = 'sitewide' and epoch_s = 1361318400 and group_name = 'ALL_GROUPINGS';


The Future?


Features

• Eliminate the need for an engineer to write the queries

• Robust segmentation

• Not be limited to visit sessions

• Run Ad Hoc queries


Build your own Funnels

show the builder ui here


Mechanics of Funnel Analysis


Input

• Event sequences

• Typically visits to website

• Sessionization scheme left to developer



Funnel

• Search

• Listing

• Cart







Query

• Only discussed event types

• Funnel steps require additional constraints:

• Listing referred by search

• Added that listing to cart


Searchquery “dinosaur”

listing_ids 119469855, 90583707, ...

loc “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

... ...


http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ




Listinglisting_id 119469855

ref “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

shop_id 7415158

... ...






Cartadded_listing_id 119469855

cart_listing_ids 119469855, ...

cart_type guest

... ...


Query

• Listing referred by search

• Added that listing to cart


Search

query “dinosaur”

listing_ids 119469855, 90583707, ...

loc “http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gallery&ship_to=ZZ”

... ...








shop_id 7415158

... ...








shop_id 7415158

... ...

listing.ref == search.loc








cart_type guest

... ...




cart_type guest

... ...

cart.added_listing_id == listing.listing_id


Query

• Search

• Listing & Referred

• Cart & AddedListing


Pattern Matching

• Apply query to event sequence

• Select out tuples of matching events


What About Incomplete Funnels?



null


nullnull


Funnel Analysis

• Replace events with 1

• Replace nulls with 0

• Sum


null

nullnull


null

nullnull

1 1 1

11

1


0

00

1 1 1

11

1


123


min(1, 1)min(2, 1)min(3, 1)


111


What Do We Keep?



Search


listing_ids 119469855, 90583707, ...

loc“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

... ...

Listing

listing_id 119469855

ref“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

shop_id 7415158

... ...

added_listing_id 119469855


cart_type guest

... ...

Cart















Segmenting Funnels



Search


listing_ids 119469855, 90583707, ...

loc“http://www.etsy.com/search/handmade/patterns?q=dinosaur&order=most_relevant&view_type=gal

lery&ship_to=ZZ”

... ...

Listing



lery&ship_to=ZZ”

shop_id 7415158

... ...



cart_type guest

... ...

Cart














Listing



lery&ship_to=ZZ”

shop_id 7415158

... ...



cart_type guest

... ...

Cart

query










cart_type guest

... ...

Cart

listing_idquery


query listing_id cart_type


Segmented Funnel Analysis

• Extract segmenting properties

• Compute indicators as before

• Group on segmenting properties and sum


MapReduce

• Work is done map-side

• Common first step in our jobs

• Expensive computation limited to first round mappers


Event Sequence Pattern Matching


Components

• Predicate: matches/rejects events

• Query: tuple of predicates

• Match: tuple of events


Match Predicates

• Select an event based on:

• Full event sequence

• Prior matched events

• Current candidate


Match Predicate DSL

• Combine predicates with logical operators

• val Query = Seq(Search, Listing & Referred, Cart & AddedListing)


Semantics

• Fixed number of events in match

• Arbitrary number of matches per sequence

• Collect and extend all partial matches



Search Listing Shop

Listing Cart Home

Search Listing Search


Search, Listing, Shop, Listing, Cart, Home, Search, Listing, Search


Root

Search

Query


Cart & AddedListing

Listing & Referred


Root

Search

Query


?

Cart & AddedListing

Listing & Referred


Root

SearchSearch

Query

Listing, Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred


Root

SearchSearch

Query

Listing, Shop, Listing, Cart, Home, Search, Listing, Search

?

?

Cart & AddedListing

Listing & Referred


Root

Search

Listing

Search

Query

Shop, Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred


Root

Search

Listing

Search

Query

Shop, Listing, Cart, Home, Search, Listing, Search

?

?

?

Cart & AddedListing

Listing & Referred


Root

Search

Listing

Search

Query

Listing, Cart, Home, Search, Listing, Search

Cart & AddedListing

Listing & Referred


Root

Search

Listing

Search

Query

Listing, Cart, Home, Search, Listing, Search

?

?

?

Cart & AddedListing

Listing & Referred


Root

Search

Listing

Search

Query

Cart, Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing


Root

Search

Listing

Search

Query

Cart, Home, Search, Listing, Search

Listing & Referred

?

?

?

Cart & AddedListing


Root

Search

Listing

Cart

Search

Query

Home, Search, Listing, Search

Listing & Referred

Cart & AddedListing


Root

Search

Listing

Cart

Search

Query

Home, Search, Listing, Search

Listing & Referred

?

?

?

Cart & AddedListing


Root

Search

Listing

Cart

Search

Query

Search, Listing, Search

Listing & Referred

Cart & AddedListing


Root

Search

Listing

Cart

Search

Query

Search, Listing, Search

Listing & Referred

?

?

?

Cart & AddedListing


Root

Search Search

Listing

Cart

Search

Query

Listing, Search

Listing & Referred

Cart & AddedListing


Root

Search Search

Listing

Cart

Search

Query

Listing, Search

Listing & Referred

?

? ?

?

Cart & AddedListing


Root

Search Search

Listing Listing

Cart

Search

Query

Search

Listing & Referred

Cart & AddedListing


Root

Search Search

Listing Listing

Cart

Search

Query

Search

Listing & Referred

?

? ?

? ?

Cart & AddedListing


Root

Search Search Search

Listing Listing

Cart

Search

Query

Listing & Referred

Cart & AddedListing


null

nullnull




Match Tree

• Purely functional data structure

• Holds matched events and indices

• Match prefixes shared


Match Tree Algorithm

• Fold over sequence accumulating tree

• May extend any non-terminal node

• Each level in tree corresponds to predicate


Practicality

• Explodes, but

• Queries are constant length

• Sequences are bounded (visits)

• Predicates constrain growth


Summary

• Why funnels are interesting

• What we’ve built with them at Etsy

• Our approach to funnel analysis


Questions?

• Steve Mardenfeld

• Wil Stuckey @quiiver

• Matt Walker @data_daddy


Technology

Funnel Analysis in Hadoop at Etsy