How to tend a garden

Managing With MetricsNovember 17, 2005

In God we trust. Everyone else, bring data.

5. Data can tell how you are doing

4. It's objective

3. It keeps you honest

2. It will help you persuade people

1. It will make you more satisfied with your job

"16% of executives at companies that measure marketing said they were dissatisfied with their jobs. But at firms that don't measure marketing, 28% said they were dissatisfied."

Top 5 Reasons Why should spend an hour of your life listening to this?

Two different approaches

Explorer Farmer

Questions: Specific General

Frequency: One Time Daily

Distribution: Narrow Broad

Scrutiny: Minimal Substantial

We need both, but the real value is in Farming. Start with a garden.

3 Rules of Gardening

1.Choose good seeds and fertile soil

2.Tend to your garden every day

3.See what grows. Remove weeds

(Choose the right metrics and the right view)

(Understand and try to improve the metric. Look for patterns)

(Watch for and follow up on changes in the metric)

Choosing Fertile Soil and Good Seeds

• Choose one metric or one set of metrics– You can not chase two rabbits

• Choose a metric that matters– Don’t look at page views if it’s Leads that matter

• Choose a metric that you can control– Sales controls CPCs and SEM controls clicks

• Choose something actionable– Ask yourself what you would do with this data if you had it?– What would you do if it turns out to be higher? Lower? If you don’t have good

answers to these questions then pick a different metric

• Keep it small– You should be able to tend to your metrics in just a few minutes a day

• Avoid averages if you can– There is no such thing as average, particularly online

The 80/20 Rule

• Italian economist Vilfredo Pareto who noted that 80% of the wealth of Italy was owned by 20% of the people. “The Vital Few and the Trivial Many”

• In all things, there are a few vital contributors (20%) that contribute the vast majority of the results (80%)

– 20% of our partners drive 80% of our revenue

– 20% of our merchants get 80% of our leads

Remember that there is no such thing as average.

The 80/20 Rule for Partners

$15,895

$-

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

$70,000

$80,000

$90,000

Average Partner

The 80/20 Rule for Partners

$80,161

$15,895

$302.91$-

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

$70,000

$80,000

$90,000

Top 20% partner Average Partner Bottom 80% partner

Focus on what matters. Don’t sweat the small stuff.

Tending to your garden every day

• Just take note of the numbers every day

– Showing up is half the battle

• Try to improve them

– Take actions based on what you think the data is telling you

• Be sensitive to patterns. You will start to see some.

– Refine your view when you notice patterns.

– If Mondays are always better than Sundays, then compare them to the prior Monday.

• Notice the amount of natural variability in the data

– All data has noise

Finding patters and understanding noise

Visits to the Shopping.com network by day

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

August September

Visits

Average

+2

-2

Rob’s daily dashboard

2005

4

November October

Partner Type1Data 14 07 31 24 17 10 03

SEM Visits 1,362 1,132 900 998 925 931 894

Leads 822 668 463 506 503 514 489

Revenue Usd $280 $213 $164 $168 $170 $171 $175

Gross Margin Usd $66 $58 $52 $40 $52 $52 $59

Organic Visits 394 353 302 325 310 331 309

Leads 109 101 82 87 84 92 84

Revenue Usd $58 $53 $41 $41 $40 $44 $41

Gross Margin Usd $58 $53 $41 $41 $40 $44 $41

Algorithmic Visits 1,292 1,188 1,005 1,073 1,047 1,077 969

Leads 286 258 193 200 197 213 182

Revenue Usd $145 $127 $87 $89 $86 $90 $81

Gross Margin Usd$145 $127 $87 $89 $86 $90 $81

Other Visits 216 238 215 225 215 231 216

Leads 289 252 211 237 226 235 206

Revenue Usd $181 $161 $118 $119 $109 $113 $104

Gross Margin Usd $54 $51 $38 $37 $32 $35 $31

Total Visits 3,264 2,910 2,422 2,620 2,497 2,569 2,388

Total Leads 1,506 1,279 950 1,029 1,010 1,055 962

Total Revenue Usd $664 $555 $410 $417 $406 $419 $401

Total Gross Margin Usd $323 $288 $217 $206 $211 $221 $212

See What Grows and Remove Weeds

• Pay close attention to changes. – Sometimes you will find errors in the data– Be vigilant about escalating them and having them fixed– If you don’t keep your metrics clean, you won’t know what is happening

• Make it easy to spot changes– Color them in – Use Charts– We will help you look for them automatically

• If you see increases or decreases can you explain them?– Did you create the change? If not, now is the time to start exploring.– If it was caused by something someone else did, share the data.

• Follow up. Changes happen for a reason. Understand Why.

• If you have a hunch but can’t confirm it, consider a test.

2005

4

November October

Partner Type1 Data 14 07 31 24 17 10 03

SEM Visits 1,362 1,132 900 998 925 931 894

Leads 822 668 463 506 503 514 489

Revenue Usd $280 $213 $164 $168 $170 $171 $175

Gross Margin Usd $66 $58 $52 $40 $52 $52 $59

Organic Visits 394 353 302 325 310 331 309

Leads 109 101 82 87 84 92 84

Revenue Usd $58 $53 $41 $41 $40 $44 $41

Gross Margin Usd $58 $53 $41 $41 $40 $44 $41

Algorithmic Visits 1,292 1,188 1,005 1,073 1,047 1,077 969

Leads 286 258 193 200 197 213 182

Revenue Usd $145 $127 $87 $89 $86 $90 $81

Gross Margin Usd $145 $127 $87 $89 $86 $90 $81

Other Visits 216 238 215 225 215 231 216

Leads 289 252 211 237 226 235 206

Revenue Usd $181 $161 $118 $119 $109 $113 $104

Gross Margin Usd $54 $51 $38 $37 $32 $35 $31

Total Visits 3,264 2,910 2,422 2,620 2,497 2,569 2,388

Total Leads 1,506 1,279 950 1,029 1,010 1,055 962

Total Revenue Usd $664 $555 $410 $417 $406 $419 $401

Total Gross Margin Usd $323 $288 $217 $206 $211 $221 $212

Exception Reporting

Subplacement Alerts Report Date: 23-OCT-05

Total measures checked: 92,708Significant changes found: 14

$Impact:Site Name: Partner Type: Alert on data: Reasons:

-$2164 SDC - fr.shopping.omSearch Engine Marketing

COS/CLICKS_IN ( .546) 3.39 sigma less than average ( .590)

-$914 SDC - fr.shopping.omSearch Engine Marketing

COST ( 3,530) 5.10 sigma less than average ( 2,615)

-$390 SDC - fr.shopping.om Free Traffic CLICKS_IN ( 1,608) 3.23 sigma less than average ( 9,508)

$121 SDC - fr.shopping.om Free Traffic COS/CLICKS_IN ( .267) 15.86 sigma more than average ( .144)

$326DEALTIME - dealtime.co.uk

CoBrand and Other GROSS_MARGIN ( 2,204) 4.52 sigma more than average ( 1,877)

$338DEALTIME - dealtime.co.uk

CoBrand and Other PMCS/CLICKS_IN ( .929) 6.04 sigma more than average ( .720)

$340 SDC - fr.shopping.omSearch Engine Marketing

GROSS_MARGIN ( 1,703) 3.61 sigma more than average ( 1,363)

$449 SDC - fr.shopping.omSearch Engine Marketing

CLICKS_IN ( 65,677) 6.04 sigma more than average ( 49,411)

$637DEALTIME - dealtime.co.uk

CoBrand and Other PMCS ( 14,755) 4.96 sigma more than average ( 12,781)

$836 SDC - shopping.com Associate GROSS_MARGIN ( 3,565) 3.04 sigma more than average ( 2,728)

$1036 SDC - shopping.comSearch Engine Marketing

MC_REVENUE/PMCS ( .303) 3.98 sigma more than average ( .294)

$1254 SDC - fr.shopping.omSearch Engine Marketing

REVENUE ( 5,232) 5.82 sigma more than average ( 3,978)

$1711 SDC - shopping.com Associate PMCS ( 14,031) 3.15 sigma more than average ( 10,511)

$6740 SDC - fr.shopping.omSearch Engine Marketing

COS ( 35,866) 5.60 sigma more than average ( 29,126)

Gardening is a mindset

• It requires patience

• It requires diligence

• It is best done as a ritual

• It will give you perspective

• Eventually, you will find It relaxing

• Eventually, you will find it satisfying

• It will broaden your perspective

• It will make your life more satisfying

Thanks