Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)

Wrap up & Experimentation

CS147L Lecture 8Mike Krieger

Wednesday, November 25, 2009

Intro


Welcome back!


By the end of today...

- Questions from implementations

- A few implementation loose ends

- A/B testing primer

- Google Analytics


Questions?


Loose ends


Floaty bar


Canonical implementation

- Gmail's mobile web app


Gmail Demo


Fitts Thumb

- Though hand input not quite the same as mouse, general principle applies:

- Minimize thumb-moving distance

- Make targets even larger than you think they need to be (thumbs are clumsy)


Getting plugin

- Included with jQTouch under extensions/

- Copy jqt.floaty.js into your JS folder


Integrating & customizing


Initializing


Styling


Demo

- floaty.html


Bottom bar


UITabBarController on iPhone


This won't work...

- Traditional approach: position: fixed at bottom:0

- Or, div with overflow:hidden and bottom bar with absolute position and bottom:0


On the iPhone

- Scrolling scrolls entire page

- Floaty bar is probably the way to go...

- You could hack it up, but your users would have to learn to two-finger scroll for everything


Full screening


App mode

- Only engaged when users click home button (might want to prompt them, or do it before hand)


App mode


A/B Testing


Why A/B test?

- You can have the best designers & great PMs...

- But nothing beats seeing what on earth people actually do

- Differences in usability, virality, and revenue


Framework

- Selecting an experiment

- Choosing variations

- Selecting / sampling users

- Deploying & serving variations

- Measuring user behavior

- Analyzing results


What makes a good experiment?

- Measuring user funnels through a sale

- Click-through rates for links

- Time spent / time until an action is taken

- Performance questions

- Email newsletters


(continued)

- Minor tweaks to site design

- Flows with a clear goal


In sum—

- a measurable user behavior that you believe will be modulated by tweaks in design


What A/B testing won't tell you

- Is it aesthetically pleasing?

- Is it fun?

- Is it accessible?

- Is this even what my company should be doing?


As Buxton would say

- A/B testing will help you get the design right, but can't help you get the right design in the first place


Hillclimbing

controlled web experiments


But really...

controlled web experiments


Choosing Variations

- Think in terms of variables

- Spectrum of choices

- If time (and participant pool), look at interactions, too


Examples

- Twitter homepage call to action

- Join the Conversation

- or

- Get Started


Iteration


Google Homepage


...experiment?

- People actively want to join Google's A/B tests

- But can use interest/reactions as proxies for results in this case


Selecting/sampling users

- Two general approaches


The naive way

- Every time a user loads a page, they have a random chance of ending up in a bucket


Why doesn't this work?

- Order effects

- Confuse the users, who want a consistent experience

- Random functions not so random


Using a hashing function

- Suggested by Kohavi in his Web Experimentation paper

- How it's implemented at Meebo


General idea

- Be as consistent as possible per user

- If we can, use User ID (across computers)

- If we can't, use a cookie (at least consistent at one computer)

- At the very worst, assign randomly


MD5

- Hashing function; not great for encryption but fine for our purposes

- Problem: hashes will be long strings and we actually want a probability distribution


Solution

- Hash the unique identifier plus the experiment name

- Get the hexadecimal digest of the resulting hash

- Convert to a decimal and see where it falls along the range of 0 to the Max number in the distribution


In other words...


(continued)


Notes

- Will be evenly distributed from 0 to 1.0

- We can use this probability to bucket people

- Given the same input, will result in same number every time


Deploying & serving variations

- For prototypes, much can be hard-coded

- For real production use, infrastructure can make life easier in the long run


One easy way


In the long term

- Build out front-end to turn experiments on/off or config file


Overriding Javascript

- Problem: you already have most of your functions defined, but want your treatment to do something slightly different


Encapsulation


Overwriting


Monkeypatching

- Idea: we want to do mostly the same thing, but do something before/afterwards that's slightly different, or modify the input


How to


Measuring behavior

- Are people actually doing something different?

- Using log lines, or writing straight to DB


Normal Log lines10.32.109.7 - - [18/Nov/2009:22:36:32 -0800] "GET /courses/cs147/images/media.jpg HTTP/1.1" 200 191910 "http://hci.stanford.edu/courses/cs147/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"


http://hci.stanford.edu/courses/cs147/




Tracking Log Lines10.32.109.7 - - [18/Nov/2009:22:36:32 -0800] "GET /track?condition=bluebutton&event=click&timebeforeclick=500 HTTP/1.1" 200 191910 "http://hci.stanford.edu/courses/cs147/" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"




How to process?

- In the small: use Python

- In the large: use Hadoop and Pig


Pig, ultrabriefly

- (because I think this will be huge in a year or so)


Pig

- SQL-like language built on top of Hadoop

- Makes writing Map/Reduce tasks really quick

- In use at Yahoo!, Twitter, Meebo, etc


Pig sample code


And the best part...

- Will compile & run for you over as many machines as necessary


PHP Logging scripts

- Wouldn't work for production data

- Fine for any A/B tests or just logging / instrumentation you want to do


DB Schema


Logging events


Example


HTML


JS


Closing the loop

- Get the data out & aggregate

- Visualize!


Reading data from SQLite

- report.php


Basic code


Reformat as data series


Result[{"label":"blue","data":[[0,2]]},{"label":"red","data":[[1,12]]},{"label":"green","data":[[2,1]]}]


Analysis options

- Excel

- Tableau

- R

- Javascript or Flash graphing/visualization libraries


Briefly: flot

- jQuery plugin

- Super useful for basic graphing & charting needs

- Also handles time-series data well

- http://code.google.com/p/flot/


http://code.google.com/p/flot/

http://code.google.com/p/flot/

From report.php to flot


Demoflot.html


Even better: Protovis

- Stanford Graphics lab project

- http://vis.stanford.edu/protovis/


http://vis.stanford.edu/protovis/

http://vis.stanford.edu/protovis/

Significant change?

- Chi-Squared test


Chi-Squared

- Idea: measure whether a particular distribution of measures deviates significantly from expected


Null hypothesis

- Button color has no impact on click-through rate


Testing

Where:O(i) is the observed frequency,E(i) is the expected frequencydegrees of freedom = (number of categories) - 1


Sample data

Blue Red Green Total

267 267 266 800

Significant?


Sample data


270 250 260 800

Significant?


Sum the differences

Blue: (270-267)^2 / 267 = 0.03Red: (250-267)^2 / 267 = 1.08Green: (280-267)^2 / 267 = 0.63

0.03 + 1.08 + 0.63 = 1.74 = χ2


Look it up in table

- (or use R/SPSS/something fancier)

- http://www2.lv.psu.edu/jxm57/irp/chisquar.html


http://www2.lv.psu.edu/jxm57/irp/chisquar.html




Significant?

p ~ 0.4 (we want 0.05)not significant


Sample data


240 230 330 800

Significant?


Sum the differences

Blue: (240-267)^2 / 267 = 2.73Red: (230-267)^2 / 267 = 5.12Green: (330-267)^2 / 267 = 14.86

2.73 + 5.12 + 14.86 = 22 = χ2


Significant?

p < 0.001highly significant


Wrap-up

- Can be a bit of work

- But can lead to amazing insights

- Plus, data analysis & visualization is really fun


Google Analytics


Why?

- It's free!

- For a small company or project, much better than rolling out some of these analysis tools for yourself


Signing up

- http://analytics.google.com


http://analytics.google.com

http://analytics.google.com

Integrating JS

Insert before </body>


Tip: Tracking JS events

_trackevent function, takes(category, event, optional_key, optional_value)


Using Web Optimizer

- Attached to AdSense

- Provides A/B testing tools with integrated confidence interval and significant different calculations


Demo


Final notes


Where to go from here?

- Four fun things to explore


Building full web apps in Django

- Templating

- Wrapping Request/Response

- Interfacing with the database

- Forms


Some code from courseapp


and the view...


Trying out Google App Engine

- Great for weekend projects that could become something more

- Django templating built-in

- Can also use (most) of Django with app-engine-patch (http://code.google.com/p/app-engine-patch/)

- And, it'll scale if you need it to


http://code.google.com/p/app-engine-patch/




Adapting for native app

- PhoneGap lets you access native app features from JavaScript

- Can continue your class projects if you want to take them further

- Caveat: will have to get dev account


Using Mechanical Turk for app feedback

- And quick testing of ideas


Evaluating Scenarios









Reactions- “Personally I prefer the idea of storyboard one. This is

because the user freely walks around the museum as they would traditionally, yet automatically receive info about exhibits - a virtual guide without user input. Much more impressive.”

- “Personally I think the idea from storyboard one is more compelling. The reason for this is that I would be interested in finding out interesting information about a piece of artwork or a particular artist that I couldn't just get at the museum. The map of the museum is something that I can get at the museum on a piece of paper that doesn't require me to be pulling out my phone and wasting the battery to get to an exhibit.”


Reactions

- “Students who are visiting for a school assignment and have limited time to partake in all the exhibits would definitely find that option helpful.”

- “I liked storyboard 2 because of [the use of] cell phones in a physical space.”

- Self-reported as non-designers


Final plug

- We're hiring at Meebo!

- User Experience, Usability, UI...

- meebo.com/jobs or email me directly at [email protected]


mailto:[email protected]

mailto:[email protected]

Thanks!


Q’s?


Documents

Wrap up & Experimentation · falls along the range of 0 to the Max number in the distribution Wednesday, November 25, 2009. In other words... Wednesday, November 25, 2009 (continued)