!

It Worked for UstreamProduct Development @ Ustream

Balázs Kereskényi product manager

API, discovery

Gergely Schmidt product manager,premium products

Making decisions by assumptions:“Let’s do this, because it worked for Spotify!” - you will never know why it worked for them.

Making decisions by experiments:“Let’s try this out, measure effectiveness,and see if it works!

What to test?

Icon & text: +12% conversion.

Myth #13: Icons enhance usability

What to test?

• CTA microcopies: “Get started”, “Try out for free”, “Sign up”, “Learn more” etc.

• Layout: arrangement of form fields, mixing elements on the whole page.

• “Meta parameters” of a product: trial period length, price.

• ...

How to measure effectiveness?

Conversion (basically any kind of your metrics):

• Increase in purchases,

• Engagement,

• Number of clicks,

• Number of signups,

• Form completions,

• ...

Always question assumptions

but be prepared that only 1 out of 8 A/B testswill drive you to significant changes.*

*based on the tests done by appsumo.com

FAMOUS!A/B!

TESTS

Marissa Mayer:41 shades

of blue

How Obama raised $60M by running a simple experiment

Optimizely, Dan Siroker

Who to blame for the slow loading up? Facebook or iOS?

How to set up a test?• Targeting (when the test should start, visitor characteristics &

segmentation, technical circumstances etc.),

• Experiment type (A/B, multivariate, multi page),

• Traffic allocation,

• Goals,

• Integration with other kind of analytics.

Technical background of a test• Frontend

• Quick and dirty methods

• Always having a usable fallback is anything goes wrong

• Backend for bigger and more complicated tests

• Measurement

• GA and Optimizely events are our best friends

Understanding your resultsChance to beat baseline.

…or you can simply use isvalid.org anytime.

USTREAM!CASE!

STUDIES

Changing directions: SaaS vs. content.

Started with concluding tiny preliminary experiments: less content, pro CTAs. Ended up with split testing the whole page.

editorializedpro purchases++

Ustream homepage test results

• Tested on approx. 700k unique visitors.

• No important general metrics harmed (pages / visit time on site & time on homepage).

• 1% increase in purchases starting from the home page.

• Showing 1/3 of content resulted only in halving the clicks.

A DIFFERENT APPROACH FOR VALIDATION:!GHETTO TESTING

http://bit.ly/ghettozynga

Ghetto test: Pay with PayPal

• Built by frontend engineers, only triggered from Optimizely

• Decided not to build it because of low conversion rate (~2%)

Pro Purchase Flow: 1 or 2 steps?

• Built by frontend engineers, only triggered from Optimizely

• Currently live, analysis is on the way

This is not a test

!

https://ustream.tv/platform/unlimited!

Thanks!

@bkereskenyi@sgergely