Krishna Gade

Why everything is an A/B test at Pinterest

Confidential

2

What is Pinterest?

A visual bookmarking tool

Discover an inspiring ideaSave it to a board

Go do it

Pinterest is a data product

Data Flow

App

events

Kafka

SecorSkyline

Pinball

Redshift

Pinalytics

Features

Qubole (Hadoop)

Singer

Confidential

Metrics

7

Confidential

User Lifecycle

New Users

Active Users

Dormant Users

Acquisition

Activation

Churn

Churn Resurrection

Confidential

Terminology

● MAU – Monthly Active User● WAU – Weekly Active User● DAU – Daily Active User● WAR – Weekly Active Repinner● WAC – Weekly Active Clickthrougher● WARC – Weekly Active Repinner or Clickthrougher● 1d7 – User who comes back at-least once in 7 days after

signup● 1rc7 - User who repins/clicks at-least once in 7 days

after signup

Confidential

MAUs

● Current progress to quarterly goal● MAU forecast● MAUs by app/gender/country● MAU Accounting

Confidential

MAU Accounting

Confidential

Acquisition Metrics

● Total Signups● Signups by App● Signups by Referrer● Invites Sent● Unique Invite Senders● Invite Signups

Confidential

Activation Metrics

Activation Rate = 1rc7/signups

● Activation Rate by App/Gender/Referrer● Overall signups to 1rc7● Signup engagement funnel● Signups to WAU 35 days after signup

Confidential

Engagement Metrics● *AU ratios

● # of Emails sent● # of Emails opened● # of Emails clicked● # of Notifications sent● # of Notifications clicked

Confidential

Resurrection Metrics

● # of Users who weredormant for 28+ daysbut came back, split byplatform, referrer etc

Confidential

Takeaways

● Emphasize on activation (process of converting a new user to MAU)

● Activation critical to long-term growth● Deeply understand how different

segments of users interact

Confidential

Automated Metrics

17

Confidential

Confidential

Chart creation

Confidential

Reuse existing query result

Confidential

Metrics creation

Confidential

A/B Experimentation

22

Confidential

Categories of Experiments

● A/B testing new features● Copy testing● SEO changes● Infrastructure updates

Confidential

A/B Testing Framework

Confidential

Confidential

Confidential

Confidential

A/B Testing Dashboard

Confidential

Confidential

p-value (via chi-squared test)

Statistical tests for validation

Confidential

Impact of Badging

Confidential

Impact of Badging

● 7% lift in DAU● 2.5% lift over long term● Effective at increasingengagement levels

Confidential

Copy Testing

Definition: Testing different variations of text we display to the user.

● A/B Testing copy was a tedious process● Even when we did this we were optimizing

only for English

Confidential

New Approach

● Ability to run a copy test w/o changing a line of code.

● Support for translating text dynamically● Use Multi-armed bandits to converge to showing

best performing text

Confidential

Multi Armed Bandits

Confidential

SEO

36

50+ Billion Pinscategorized by people

into more than

1 Billion Boards

Confidential

37

What makes SEO so challenging?

What is the Impact of the new SEO feature?

Will this new change affect SEO?

Uhhh...Dunno...

Uhhh...Ask Google.

Confidential

KEEPCALM

and

RUN ANSEO EXPERIMENT

Confidential

What is an SEO experiment?

39

Regular AB Experiment SEO Experiment

Metrics User behavior Search engine referred traffic

Unique id for hashing userid / sessionid page url

Confidential

What is an SEO experiment?

Launch

● Measure the gap ● Before vs After launch

Confidential

Dashboard

41

DIFF = enabled - control

Traffic Gain

Confidential

42

Decision Making 1 2

3

4

SHIP!

SHIP!

SHUT DOWN!VALIDATE

Confidential

43

Experiment Driven SEO

What is the Impact of the new SEO feature?

We got XX% of traffic growth from it! Can we use more resources to make bigger impact?

No, the experiment result is flat, so let’s ship it.

SEO experiment

SEO experiment

Is my UI change affecting SEO negatively?

Confidential

Why you should be A/B testing even your infrastructure

Confidential

Infrastructure Changes

● 2013: A new web framework● 2014: Pyapns● 2015: User service

Confidential

Experiment results for replacing PyAPNs

Confidential

Conclusions

● Figure out what success means● Define your metrics● Automate metric computation● A/B test everything!

Confidential

Thank You!