1 Predicted-versus-Actual Studies: Why/how to do them and Lessons Learned Ken Cervenka Federal Transit Administration TRB Transportation Planning Applications

1

Predicted-versus-Actual Studies:Why/how to do them and Lessons Learned

Ken Cervenka

Federal Transit Administration

TRB Transportation Planning Applications Conference in Houston, Texas

May 20, 2009

TRB Transportation Planning Applications Conference in Houston 2May 2009

Topics

Why do them? How to do them? Lessons learned (so far)


Why do them?

Forecasts (should) matter If they don’t matter, what are we doing here? Supports informed decision-making

Poor track record for accurate predictions Even aggregate numbers are often way off Capital costs Weekday traffic volumes Weekday transit ridership (boardings)


Predicted-versus-Actual Traffic 104 Completed Toll Road Projects

Data Source: Standard & Poor’s, 2005 (Robert Bain)

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6

Actual/Forecast Traffic (Mean = 0.77)


Predicted-versus-Actual Ridership 18 Transit Projects Completed 2003-2007

Average = 74.5%

50th Percentile = 63.8%


So why do them?

Learn from past failures - and successes What went wrong and what went right More than just aggregate checks

Avoid hand-waving speculation Identify major drivers for errors (in either direction)

Insights for improved prediction tools Better understanding of uncertainties More informed decision-making Better use of limited funds


Predicted-versus-ActualSome Examples

New transportation project Roadway and/or transit Policy change

New developments Change over time (“validation”)

2009 forecast of 2005 base year 2000 forecast of 2005 base year (backcast)


So why else do them? Well…

Required for FTA discretionary funding of major transit projects (New Starts program)

Annual report to Congress Before-and-after comparisons Predicted-versus-actual (after) comparisons

See Session 11 from FTA’s March 2009 Travel Forecasting for New Starts workshop http://www.fta.dot.gov/planning/newstarts/planning_

environment_9547.html


How to do them

Start with the big picture comparisons What we thought would happen What actually happened

Gain insights by digging into the details Not just traffic volumes and transit passenger boardings Forensic analysis Reasons for big picture misses Confirmation of big picture successes

Prepare new post-implementation forecasts New model runs with corrected inputs Special runs to track down sources for other errors


How to do themExamples

Assess the accuracy of forecast inputs District-level demographics Roadway system Transit service levels and fares Auto-related costs, etc.

Check major transit rider travel patterns District-to-district flows Trip purpose and socio-economic class Access and egress modes


How to do themFTA’s Approach

Required: preservation of forecasts Project scope, capital cost, service levels, O&M

cost, and ridership (plus others as needed) For different project planning milestones Analysis/explanation of changes in forecasts

Ability to replicate the forecasts, e.g. for ridership: Input demographics/networks and outputs Scripts and application documentation

DVDs to FTA do not include proprietary software


How to do themFTA’s Approach

Approval of Before-and-After Study Plan Required for grant approval Data collection plan

E.g., before and after transit rider surveys Analytical approaches

Approval of Before-and-After Study Report Required to close out a grant Analysis/explanation of project impacts Analysis/explanation of prediction errors


Lessons LearnedRoadway and Transit Forecasts

Compounded optimism (optimism bias) The “high side” of feasible assumptions Vested interests, ethics, and objectivity

Travel model inputs matter Networks and demographics Parking costs, fares, etc. Value of time assumptions

Person trip tables (!)



Quality control checks matter Model testing

Data adequate for the task Caution on over-specification

Plausibility checks of behaviors Big picture insights for key markets



Recognition of uncertainties Local predictions of new behaviors

Local calibration not possible Toll roads in areas without toll roads Choice riders in areas with few/no choice riders

Park-and-ride service Premium service (e.g., light rail)


Lessons LearnedFTA Perspective

Useful information for the region and the profession Biggest missed opportunity of New Starts

program A compelling case for clear explanations

Early and often coordination Clearly defined responsibilities and budgets Preservation and analysis performed while

memories are still fresh Greater attention to opening-year forecasts

Documents

1 Predicted-versus-Actual Studies: Why/how to do them and Lessons Learned Ken Cervenka Federal Transit Administration TRB Transportation Planning Applications