7 Practical Tips on Lean Analytics

Janne Aukia — 17 Feb 2015

1. Lean Analytics requires a Lean project

Illustration from commons.wikimedia.org/wiki/File:Kepler-solar-system-1.png

Challenging for a consultancy?

Lean Analytics works well with Lean Startup, Lean UX and

Agile projects

Question designs, question authorities, ask difficult questions,

be prepared to kill features.

Requires freedom to experiment

2. Write down a success metric for every feature

Was this feature used more than Google+ login?

If feature isn’t successful, just kill it.

Measure only actionable metrics.

Is the feature for acquisition, retention or revenue?

3. You need lots of users for an A/B test

Typically at least 10k events, such as visits or people using a

feature.

Doesn’t work for small startups.

4. Make big changes

Illustration from http://quod.lib.umich.edu/m/moa/agb8710.0001.001/26

If you cannot see the changes from 2 meters away, most likely

your AB test won’t either.

Google A/B tested 41 shades of blue. Don’t do that — leads to

local optimum.

Illustration from http://www.doxsey.net/blog/too-practically-minded-to-be-of-any-theological-good/

5. Measure your tests correctly

Never terminate earlya running test

Use a split test calculator, for example:

http://getdatadriven.com/ab-significance-test

6. Push through radical changes with A/B

7. Use the right tools

Google Analytics can be used for tracking events but it is

painful.

Mixpanel more suited for the job. Consider Keen.io, Heap as

well.

1.Lean Analytics requires a Lean project

2.Specify success metric for every feature

3.You need lots of users for an A/B test

4.Make big changes

5.Measure your tests correctly

6.Push through radical changes with A/B

7.Use the right tools

Summary