How Experiments drive product at Viki
Ishan Agrawal
Software Engineer, Data Viki
Why Experiment?
What is AB testing?
How to AB test?
What can you test?
Anything that affects user behavior
To build or not to build…
More control
Our own Data
… in walks Turing Our own Experimentation framework built in Golang
and Rails!
The Challenges
Flexibility Speed Scalability
User Partitioning Stickiness
Architecture
How to make it fast and furious!
• Api is in Golang
• Store everything in-memory hashes (timely syncs)
• No external calls (not even a redis cache)
~ 300 micro seconds
Advantages of using Go lang • Run as a binary anywhere!
• Static Type checking
• A good trade-off between performance and development speed
Stickiness and User partitions without a cache?
Points to note
• Server side determination
• Clients log the experiments tied to the current session.
• A user is under only 1 experiment at a time
So how do you create experiments?
Features and Variations on them
Feature and Variations Example
Feature: Homepage Variations: Default : { version : 0 }! Setting1 : { version : 1 }!conditions: { country : ‘us’}!
Experiment1 : {version : 2}!
!!conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!!
Experiment2 : {version : 0}!!conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!!
The magic! • Settings
o Power of customizations
o Different result for everyone • Country • Regions • User type • Time Ranges • Application versions
Data Collection and Analysis
Making it faster
Each release version is tagged with the enabled features.
Sometimes users define the product…
• Feature toggling
• Feature tweaking / improvement (with data feedback)
• Feature customization
• Slow feature roll-outs • In app messages
Some Experiments we ran…
Viki Pass Landing Page(1)
Viki Pass Landing Page(2)
Viki Pass Landing Page(3)
And the winner is…
Home page experiments
Home page experiments
20%
Conversion Rate
Ads optimization
Ads optimization
Ads optimization
We are still figuring it out..
Appendix
In-‐‑app messaging
AB testing Favorite CTA
AB testing like buYon CTA
Recommended
