CONFIDENTIAL | 1

CONFIDENTIAL | 2

Featured Project:

The Wroclaw West Bypass, Poland

Trusted by the world’s largest projects

Pivots for real: how we've hacked up some data to know we've hit the mark

Mark Smith @markjdsmith

CONFIDENTIAL | 3

Obligatory Who am I

CONFIDENTIAL | 4

BidContender

• Online Tender Management platform that has connected builders and subbies on $48 billion in projects this year

CONFIDENTIAL | 5

Stories -> Hypothesis

120 Points = done

http://www.agileproductdesign.com/

CONFIDENTIAL | 6

How to know what’s of value?

• Experiments, validated learning, testing things

Read it, loved it, but… when it came to it. How?

CONFIDENTIAL | 7

Torres format hypothesis

http://www.producttalk.org/2014/11/the-5-components-of-a-good-hypothesis/

CONFIDENTIAL | 8

Hypothesis Wall

CONFIDENTIAL | 9

CONFIDENTIAL | 10

CONFIDENTIAL | 11

Data Measures Outcomes

CONFIDENTIAL | 12

Share your data using online apps, not IT, and not Excel

GeckoboardDataHero

CONFIDENTIAL | 13

Then build until the number moves…

4 Weeks

CONFIDENTIAL | 14

or doesn’t.

2 weeks

CONFIDENTIAL | 15

Data can be sourced from anything…

• Don’t feel you need to build amazing infrastructure to get it

CONFIDENTIAL | 16

Fake it `til you make it.

CONFIDENTIAL | 17

TV’s are essential

CONFIDENTIAL | 18

The motivation is palpable

CONFIDENTIAL | 19

The journey from spec to outcome

CONFIDENTIAL | 20

An exampleHypothesis

2 X Transactions Fe

atu

res

Ou

tco

mes

$

Say, 20/ month 40/ month & growing

Many iterations of

price, duration

offer, app, marcomms

CONFIDENTIAL | 21Waterfront City, Australia | US $788M Value

Goals drive hypotheses

CONFIDENTIAL | 22

Goal Hypothesis* Measure

* Many of these, in a hierarchy

CONFIDENTIAL | 23

Discover the drivers for your goals (Impact Mapping)

The Goal

Who needs to change

Ways we can drive change

Down here –you’re just guessing

(hypothesis!)

http://www.impactmapping.org/

CONFIDENTIAL | 24

The ‘guesses’ down the tree can be stated as hypothesis

.

We believe that if we implement semi automated invites we will see the number of invited friends increasex% within 1 month

CONFIDENTIAL | 25

Goals are the outcome at the end of a funnel

Step 1

Step 2

Step 3

% that get to this step is the

goal

100 users

50 users

10 users (10%)

CONFIDENTIAL | 26

Funnel maps steps through a product workflow

Step 1

Step 2

Step 3

Step 1

Step 2

Step 3

CONFIDENTIAL | 27

We took a flow from our app and created hypothesis for change

STEPSRegistered

Profile Updated

Subscribed

Subbie connected to builder

Labelled by builder

Invited to quote

quoted

CONFIDENTIAL | 28

Hypothesis for changes to a step

STEPSRegistered

Profile Updated

Subscribed

Subbie connected to builder

Labelled by builder

Invited to quote

quoted

Adding a list of ‘builders in their state’ will increase

connections to builders for subbies by 50% within 1 month of introduction

Hypothesis

CONFIDENTIAL | 29

Hypothesis for changes to a step

STEPSRegistered

Profile Updated

Subscribed

Subbie connected to builder

Labelled by builder

Invited to quote

quoted

Adding a list of ‘builders in their state’ will increase

connections to builders for subbies by 50% within 1 month of introduction

Hypothesis

Search filter for builders in Network directory

Send URL to builders list in email

Static list (A-Z) of builders

Options to solve

CONFIDENTIAL | 30

Inference Tree

CONFIDENTIAL | 31

Inference Tree

CONFIDENTIAL | 32

Increasing Activation to 90% will increase retention by 20% for pro

subscribers within a quarter

Ensuring Subbies are being added to builders networks will increase

invitations to quote by 30% for subbies within 1 month of introduction

Adding a list of builders in their state will increase connections to builders for subbies by 50% within

1 month of introduction

If this holds

Then this may hold

And this will hold

(invitations create activation)

CONFIDENTIAL | 33

And then build it…

Adding a list of builders in their state will increase connections to builders for subbies by 50% within

1 month of introduction

As a subbie I want to see a list of

contractors in my state so that I can connect to them

As a builder I want an indication that subbies are trying to connect with me

so that I will go and complete the connection

As a product manager I want to know how many

subbies use the ‘find contractor feature’ so I can

measure the outcome

Etc…

CONFIDENTIAL | 34

There’s the measure:

CONFIDENTIAL | 35

Indira Gandhi International Airport, Terminal 3, India | US $2B Value

Data informs hypotheses.

Passing Hypotheses deliver your goals.

Failing Hypotheses don’t.

Find out which is which!

CONFIDENTIAL | 36

Further Reading

http://www.businessmodelgeneration.com/book

http://practicetrumpstheory.com/http://www.amazon.com/Value-Proposition-Design-Customers-Strategyzer/dp/1118968050

http://shop.oreilly.com/product/0636920033851.do

CONFIDENTIAL | 37

The Center to Grand Forks (CGF) Project, U.S.A. | US $350M Value

We’re Hiring!

http://Aconex.com/careers

CONFIDENTIAL | 38Seattle Tunnel State Route 99, U.S.A | US $1.4B Value

Thank You

CONFIDENTIAL | 39

Preparation (for 2nd Session – if you’re coming along!)

• Bring your laptop (or use one of the computers in the lab)

• Ensure you know your Google Analytics credentials.

• Have a google apps account so you can use google sheets

• Bring a list of 5 epics from your current backlog

CONFIDENTIAL | 40

CONFIDENTIAL | 41Dubai International Airport, U.A.E | US $4.5B Value

Session 2: Using Geckoboard and

DataHero to measure your hypothesis

CONFIDENTIAL | 42Yas Island, U.A.E. | US $40B Value

How do I turn my backlog of stories into hypotheses?

CONFIDENTIAL | 43

Some techniques

• Extract the assumptions/intention from the story

• As a <who>

• I want to <the change>

• So that <the measure>

As a player I want to have semi automated invites sent So that my list of friends increases

CONFIDENTIAL | 44

But Why?• The 5 why’s exercise on a stories ass:

– Q: Why do I want to get semi automated invites?

– A: so I increase the list of my friends

– Q: why do you want to increase the list of my friends?

– A: so that we get more players

– Q: Why do I want more players?

– A: so that we get to 1M players

– Q: why 1M players

– A: Cos then I get a Ferrari….

As a player I want to have semi automated invites sent So that my list of friends increases

CONFIDENTIAL | 45

Check the (HIPPO’s) bullshit

Bullshit! Will semi automated REALLY drive more friends?

• The 5 why’s exercise on a stories ass:

– Q: Why do I want to get semi automated invites?

– A: so I increase the list of my friends

– Q: why do you want to increase the list of my friends?

– A: so that we get more players

– Q: Why do I want more players?

– A: so that we get to 1M players

– Q: why 1M players

– A: Cos then I get a Ferrari….

Bullshit! Will more friends in the list really

get more players?

The implied hierarchy of the 5 Why’s and the bullshit detector drives a natural inference tree

CONFIDENTIAL | 46

Let’s try it…. Volunteer?

• Let’s take someone’s backlog and have a go at converting it to an inference tree:

– Volunteer – bring out your top 5 cards

– Let’s do it together.

CONFIDENTIAL | 47

Measures are easy in geckoboard

• Pick any piece of data you need for your inference tree

• Sign in to Geckoboard.

• Create a dashboard

• Put the data there.

CONFIDENTIAL | 48Dubai International Airport, U.A.E | US $4.5B Value

Demo: Geckoboard

CONFIDENTIAL | 49Dubai International Airport, U.A.E | US $4.5B Value

Demo: DataHero

CONFIDENTIAL | 50Dubai International Airport, U.A.E | US $4.5B Value

OK – your turn

CONFIDENTIAL | 51

Get going!

Your turn – in groups of 4:

– Take your backlog, and find the real goal

– Find the assumptions

– Write down the hypothesis as a tree of inference

– Work out the data you need

– Connect it to your geckoboard

CONFIDENTIAL | 52

Objections to Dashboards

• Data Objections:

– Don’t have access. (you don’t have permissions to that data)

– Can’t get that yet (integration problem)

– It’s in an email.

• Answer:

– Google Sheets.

• IT objection

– We’re not paying till you do a business case.

– OFFS – buy it on your credit card and then show the improvement, then get the corporate card on it…..

• Priority Objection

– We don’t have time to build measures, we need to build feature X

• (if nobody uses it, if it doesn’t convert, what’s the point!)

• A good product manager knows to kill features as well as make them.

CONFIDENTIAL | 53Seattle Tunnel State Route 99, U.S.A | US $1.4B Value

Visualisation

CONFIDENTIAL | 54

Static Numbers tell us NOTHING!That’s nice. Are

we better or worse than last

month/ pre change?

CONFIDENTIAL | 55

Stacked time series is better

Oh, up and to the right….

But why did it slow?

CONFIDENTIAL | 56

Sparklines show some history – puts the number in context!

CONFIDENTIAL | 57

Set a goal – so motivating

This goal is probably too far! Just to 5 would be great right

now!

CONFIDENTIAL | 58

The % change – when you want to see increments

CONFIDENTIAL | 59Seattle Tunnel State Route 99, U.S.A | US $1.4B Value

Never forget you are telling a story

CONFIDENTIAL | 60

Pro Activation Funnel has improved from our work….

• We’ve moved from 25% (28-May-2015) to 41% (3-Jun-2015) to 47% (17-Jun-

2015)

• We did these changes:

– Introduced auto-labelling

– Pending network connections show as icon on network directory

28-may-2015 3-Jun-2015 17-Jun-2015

CONFIDENTIAL | 61

And look at starter!

• This has been sitting

on 7% for some

weeks (haven’t

captured the

progression).

• Look at the numbers

on it now!

CONFIDENTIAL | 62

Still a problem, but not enough to finish?

• Drop off on connection

requests

– so we’re doing

reminder email tonight.)

• Drop off on invite to

quoted

– Will cover this as part

of builder onboarding

(they want to get

coverage).

CONFIDENTIAL | 63Seattle Tunnel State Route 99, U.S.A | US $1.4B Value

Thanks – questions?

CONFIDENTIAL | 64

The Center to Grand Forks (CGF) Project, U.S.A. | US $350M Value

We’re Hiring!

http://Aconex.com/careers