• Home

  • Software

  • You want it when? Probabilistic forecasting and decision making
42
@LMaccherone @TheAgileCraft Probabilist ic Forecasting When do you want it? Topic presented by: Larry Maccherone @LMaccherone ecision Making

You want it when? Probabilistic forecasting and decision making

Embed Size (px)

Citation preview

Page 1: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Probabilistic

Forecasting

When do you want it?

Topic presented by:Larry Maccherone@LMaccherone

Decision Making

Need to figure out how to animate these one at a time.-Larry Maccherone
Page 2: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Page 3: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Bias eats good decisions

for breakfast

By understanding probabilistic decision making, we learn to trust and overcome

bias

Page 4: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Larry Maccherone@LMaccherone

Need to figure out how to animate these one at a time.-Larry Maccherone
Page 5: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Every decision is aforecast!

Page 6: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

You are forecasting that your choice will

have better outcomes than the other alternatives

Page 7: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

So…quality of decision depends upon:

1. alternatives considered, and

2. models used to forecast the outcome of those alternatives. Probabilistic models are superior

Page 8: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Probabilistically, it’s better to go for it on 4th down more often than

coaches do

Page 9: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

For a given alternative, let: Pg = Probability of good thing happening Vg = “Value” of good thing happening

Then: Value of the alternative = Pg × Vg

Page 10: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

An lean/agile product

management example

Page 11: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

$8M

Best case (25%)

$1M

Likely case (50%)

$1M

Worst case (25%)

1$2M$2M$1M2

Which strategy is best… …for your company?

PW × VW = .25 × -$1.00M = -$0.25MPL × VL = .50 × $1.00M = $0.50M

PB × VB = .25 × $8.00M = $2.00M -----------$2.25M

…for your career?

PW × VW = .25 × $1.00M = $0.25MPL × VL = .50 × $2.00M = $1.00M

PB × VB = .25 × $2.00M = $0.50M -----------$1.75M

Page 12: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

If you get only 1 project then strategy 2 is better

75% of the time

If you get ∞ projects thenstrategy 1 is better100% of the time

How many projects do you need for strategy 1 to be better more often than not?

Page 13: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Page 14: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Play with it yourself at:http://jsfiddle.net/lmaccherone/j3wh61r7/

Page 15: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Page 16: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Emotion and bias plays a part

Page 17: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Did any of you get emotional about the $1M loss?

Did any of you want to question the $8M number?

We’ve totally… …eliminated fear from the

equation…changed the nature of the

conversation

Page 18: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Argument is about who is right.Decision making is about what is right.

Page 19: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Getting probabilit

y input you can

trust

Page 20: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Trained/Calibrated

Untrained/Uncalibrated

Statistical Error“Ideal” Confidence

30%

40%

50%

60%

70%

80%

90%

100%

50% 60% 80% 90% 100%

25

75 71 65 5821

17

68 15265

4521

70%Assessed Chance Of Being Correct

Per

cent

Cor

rect

99 # of Responses

We are inaccurate when assessing probabilities

Copyright HDR 2007 [email protected]

But, training can “calibrate” people so that of all the times they say they are X% confident, they will be right X% of the time

Page 21: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Equivalent Bet calibration

What year did Newton published the Universal Laws of Gravitation?

Pick year range that you are 90% certain it would fall within.

Win $1,000:1. It is within your range; or2. You spin this wheel and it lands green

Adjust your range until 1 and 2 seem equal.

Even pretending to bet money works.

90%

10%

Page 22: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Anagile delivery date forecast

example

Page 23: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Monte Carlo ForecastingWhat it looks likeLive demo: http://lumenize.com (use Chrome)

http://lumenize.com/
Page 24: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Seek tochange the nature of

the conversation

Page 25: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Getting even more sophisticated1. Only use slopes after it stabilizes. Discard the

first N.(Lumenize has v-optimal algorithm for finding this inflection point)

2. Weight later slopes more heavily.3. Markov chain pattern reproduction. Accomplishes

1 and 2 above automatically.4. Simulate the movement of each individual work

item through the system. Can find bottlenecks and help optimize your role balance.Troy Magennis has the expertise and tools for this.

Page 26: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Using measurement in an agile environment

Page 27: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Page 28: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

… but for those brave enough to journey into the dangerous world of

agile measurement there are great riches to be had.

The trick is to slay the dragons.

Page 29: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

The Dragons of Agile MeasurementIf you do metrics wrong, you will harm your agile transformation

1. Dragon: Measurement as a leverSlayer: Measurement as feedback

2. Dragon: Unbalanced metricsSlayer: 1 each for Do it fast/right/on-time, and Keep doing it

3. Dragon: Metrics can replace thinkingSlayer: Metrics compliment thinking

4. Dragon: Expensive metricsSlayer: 1st work with the data you are already passively gathering

5. Dragon: Using a convenient metricSlayer: Outcomes Decisions Insight Metric (ODIM)

6. Dragon: Bad analysisSlayer: Simple stats and simulation

7. Dragon: Single outcome forecasts Slayer: Forecasts w/ probability

Page 30: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Manipulating

Others

Dragon #1

Using metrics as a lever to drive someone else’s

behavior

Page 31: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Self Improveme

nt

Dragon slayer #1

Using metrics to reflect on your

own performance

Page 32: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Dragon #5Using a convenient metricaka “Lamp post metrics”

Page 33: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Page 34: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Good players?

Monta Ellis9th highest scorer (8th last season)

Carmelo Anthony (Melo)8th highest scorer(3rd last season)

Page 35: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Dragon slayer #5ODIM

OUTCOME

DECISION

INSIGHT

MEASURE

THINK

EFFECT

like Vic Basili’s Goal-Question-Metric

(GQM) but without

ISO/IEC 15939 baggage

Page 36: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

The Dragons of Agile MeasurementIf you do metrics wrong, you will harm your agile transformation

1. Dragon: Measurement as a leverSlayer: Measurement as feedback

2. Dragon: Unbalanced metricsSlayer: 1 each for Do it fast/right/on-time, and Keep doing it

3. Dragon: Metrics can replace thinkingSlayer: Metrics compliment thinking

4. Dragon: Expensive metricsSlayer: 1st work with the data you are already passively gathering

5. Dragon: Using a convenient metricSlayer: Outcomes Decisions Insight Metric (ODIM)

6. Dragon: Bad analysisSlayer: Simple stats and simulation

7. Dragon: Single outcome forecasts Slayer: Forecasts w/ probability

Page 37: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

What?So what?

NOW WHAT?

My 2nd talk3:45 Wednesday

National Harbor 6/7

Need to figure out how to animate these one at a time.-Larry Maccherone
Page 38: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Top 10 criteria for great visualization

1. Answers the question, "Compared with what?” (SO What?)

2. Shows causality, or is at least informed by it. (NOW WHAT?)

3. Tells a story with whatever it takes.

4. Is credible. 5. Has business value or impact in

its social context.

6. Shows differenceseasily.

7. Allows you to see the forest AND the trees.

8. Informs along multiple dimensions.

9. Leaves in the numbers where possible.

10. Leaves out glitter.

Credits:• Edward Tufte• Stephen Few• Gestalt

(School of Psychology)

Page 39: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Now what? Enter feedback in Sched App

Come to the AgileCraft booth:• Questions answered

• Demo of how AgileCraft surfaces probabilistic visualizations and is the best way to scale agile

Come to my talk on Wednesday at 3:45:• Using data to influence

• Top 10 criteria for great visualization

Page 40: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

“They” say…Nobody knows what’s gonna

happen next: not on a freeway, not in an airplane, not inside our own bodies and certainly

not on a racetrack with 40 other infantile egomaniacs.

– Days of Thunder

Trying to predict the future is like trying to drive down a

country road at night with no lights while looking out the

back window. – Peter Drucker

Never make predictions, especially about the future.

– Casey Stengel

Need to figure out how to animate these one at a time.-Larry Maccherone
Page 41: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

When you come to a fork in the road…

take it!

~Yogi Berra

Page 42: You want it when? Probabilistic forecasting and decision making

@LMaccherone @TheAgileCraft

Now what? Enter feedback in Sched App

Come to the AgileCraft booth:• Questions answered

• Demo of how AgileCraft surfaces probabilistic visualizations and is the best way to scale agile

Come to my talk on Wednesday at 3:45:• Using data to influence

• Top 10 criteria for great visualization


Probabilistic seasonal water supply forecasting in

Probabilistic seasonal water supply forecasting in

Documents
Probabilistic Mortality Forecasting with Varying Age ... · Probabilistic Mortality Forecasting with Varying Age-Specific Survival Improvements Christina Bohk and Roland Rau University

Probabilistic Mortality Forecasting with Varying Age ... · Probabilistic Mortality Forecasting with Varying Age-Specific Survival Improvements Christina Bohk and Roland Rau University

Documents
Probabilistic forecasting of peak electricity demand

Probabilistic forecasting of peak electricity demand

Data & Analytics
Probabilistic Wind Speed Forecasting using Ensembles - Statistics

Probabilistic Wind Speed Forecasting using Ensembles - Statistics

Documents
Probabilistic Forecasting

Probabilistic Forecasting

Documents
Ensembles and Probabilistic Forecasting. Probabilistic Prediction Because of forecast uncertainties, predictions must be provided in a probabilistic framework,

Ensembles and Probabilistic Forecasting. Probabilistic Prediction Because of forecast uncertainties, predictions must be provided in a probabilistic framework,

Documents
Probabilistic precipitation type forecasting based on GEFS

Probabilistic precipitation type forecasting based on GEFS

Documents
Probabilistic Forecasting for Intermittent Demand

Probabilistic Forecasting for Intermittent Demand

Documents
Comparing Probabilistic Forecasting Systems with the Brier ...empslocal.ex.ac.uk/people/staff/ferro/Publications/ferro...Comparing Probabilistic Forecasting Systems with the Brier

Comparing Probabilistic Forecasting Systems with the Brier ...empslocal.ex.ac.uk/people/staff/ferro/Publications/ferro...Comparing Probabilistic Forecasting Systems with the Brier

Documents
Probabilistic Weather Forecasting Using Bayesian Model Averaging

Probabilistic Weather Forecasting Using Bayesian Model Averaging

Documents
Probabilistic forecasting of long-term peak electricity demand

Probabilistic forecasting of long-term peak electricity demand

Data & Analytics
Towards a probabilistic hydrological forecasting and data …hydrocast.dhigroup.com/publications/Probabilistic... · 2013-06-10 · • Hydrological forecasting supports water management

Towards a probabilistic hydrological forecasting and data …hydrocast.dhigroup.com/publications/Probabilistic... · 2013-06-10 · • Hydrological forecasting supports water management

Documents
Probabilistic forecasting of earthquakes - Geophysical Journal

Probabilistic forecasting of earthquakes - Geophysical Journal

Documents
Probabilistic forecasting of shallow, rainfall-triggered landslides

Probabilistic forecasting of shallow, rainfall-triggered landslides

Documents
0528 Development of a Probabilistic Forecasting and ... of a Probabilistic...DEVELOPMENT OF A PROBABILISTIC FORECASTING AND HISTORY MATCHING ... to traditional reservoir simulation

0528 Development of a Probabilistic Forecasting and ... of a Probabilistic...DEVELOPMENT OF A PROBABILISTIC FORECASTING AND HISTORY MATCHING ... to traditional reservoir simulation

Documents
Probabilistic QPF & Impact- based Forecasting / Warning

Probabilistic QPF & Impact- based Forecasting / Warning

Documents
ensembleBMA: An R Package for Probabilistic Forecasting using

ensembleBMA: An R Package for Probabilistic Forecasting using

Documents
Probabilistic forecasting of maximum human lifespan by

Probabilistic forecasting of maximum human lifespan by

Documents
DeepAR: Probabilistic Forecasting with Autoregressive

DeepAR: Probabilistic Forecasting with Autoregressive

Documents
Probabilistic Forecasting with Temporal Convolutional ... · Probabilistic Forecasting with Temporal Convolutional Neural Network Yitian Chena, Yanfei Kangb,, Yixiong Chenc, Zizhuo

Probabilistic Forecasting with Temporal Convolutional ... · Probabilistic Forecasting with Temporal Convolutional Neural Network Yitian Chena, Yanfei Kangb,, Yixiong Chenc, Zizhuo

Documents
Probabilistic forecasting and optimization in CHP systems · Probabilistic forecasting and optimization in CHP systems Author: Maria Gr˝nnegaard Nielsen ... This impacts the operational

Probabilistic forecasting and optimization in CHP systems · Probabilistic forecasting and optimization in CHP systems Author: Maria Gr˝nnegaard Nielsen ... This impacts the operational

Documents
Probabilistic Energy Forecasting - KIT - ITI … 25.01.17 Institute forApplied Computer Science Agenda Forecasting challenges Renewable energy forecasting Electricity price forecasting

Probabilistic Energy Forecasting - KIT - ITI … 25.01.17 Institute forApplied Computer Science Agenda Forecasting challenges Renewable energy forecasting Electricity price forecasting

Documents
Probabilistic forecasting and optimization in CHP systems

Probabilistic forecasting and optimization in CHP systems

Documents
Spatio-temporal modelling and probabilistic forecasting of ...€¦ · Spatio-temporal modelling and probabilistic forecasting of infectious disease counts Sebastian Meyer Institute

Spatio-temporal modelling and probabilistic forecasting of ...€¦ · Spatio-temporal modelling and probabilistic forecasting of infectious disease counts Sebastian Meyer Institute

Documents
Feature selection for probabilistic load forecasting …...Feature selection for probabilistic load forecasting via sparse penalized quantile regression Yi WANG1, Dahua GAN1, Ning

Feature selection for probabilistic load forecasting …...Feature selection for probabilistic load forecasting via sparse penalized quantile regression Yi WANG1, Dahua GAN1, Ning

Documents
National Weather Service Probabilistic Forecasting in Practice J. Paul Dallavalle AMS Short Course on Probabilistic Forecasting San Diego, CA January 9,

National Weather Service Probabilistic Forecasting in Practice J. Paul Dallavalle AMS Short Course on Probabilistic Forecasting San Diego, CA January 9,

Documents
PROBABILISTIC FORECASTING FOR POWER …. Tong Probablistic...PROBABILISTIC FORECASTING FOR POWER SYSTEM OPERATIONS . ... Provide operator short-term forecast of LMP distribution,

PROBABILISTIC FORECASTING FOR POWER …. Tong Probablistic...PROBABILISTIC FORECASTING FOR POWER SYSTEM OPERATIONS . ... Provide operator short-term forecast of LMP distribution,

Documents
Earned Value Probabilistic Forecasting Using Monte Carlo Simulation

Earned Value Probabilistic Forecasting Using Monte Carlo Simulation

Business
Probabilistic Wind Power Forecasting in Electricity Market

Probabilistic Wind Power Forecasting in Electricity Market

Documents
Probabilistic Forecasting in Practice

Probabilistic Forecasting in Practice

Documents