A Modeller’s Dilemma: Overfitting or Underperforming?
Tracey Pershouse & Yun Bu (AECOM)
About this presentation
Requirements for transport modelling
Three case studies
o Trip generation
o Trip distribution
o Trip distribution and mode choice
Summary
Page 2
Requirements for Transport Modelling
Requirements for Transport Modelling
Replicating base year situations: In practice, it is used as key measurement of model quality and is often the focus of model development
Forecasting future year implications: In practice, it receives less attention although forecasting is the ultimate use of models
Case Study – Trip Generation
Case Study – Trip Generation
Alternative B: Introduce retail job density as an additional explanatory factor for the number of attractions of shopping trips
Presentation Title Page 6May 1, 2023
Modelling trip attraction to major shopping centres:
HBS trip attraction = a * number of retail jobs + b * Ln(retail job density, in jobs / hectare)
Alternative A: Flags a number of shopping centers as ‘special generators’ and apply adjustment factors derived from observed data
HBS trip attraction = [a * number of retail jobs] * adjustment factors
Validation Performance
Presentation Title Page 7May 1, 2023
The Alternative B produces considerably worse validation results for the largest centres (A and B) but better outcomes for relatively smaller shopping centres (C to H).
Forecasting Performance
Page 8
Alternative A: growth in jobs and growth in trip attractions are similar at all the shopping centres
Alternative B, faster growth rate for trip attractions at the shopping centres with relatively lower density, and slower growth rate at D where the density reaches a very high level
Case Study – Trip Distribution
Case Study – Trip Distribution
May 1, 2023
Calibrate gravity model for HBW trip distribution:
Alternative A:
- Use observed trip productions and attractions as inputs for model calibration, even though it is modelled trip ends are that input to the gravity model for model application
- Only include the OD movements with observed trips > 0, excluding the ODs with zero trips, even though some of the zeros may reflect travellers’ choice of not travelling
Case Study – Trip Distribution
Presentation Title Page 11May 1, 2023
Calibrating gravity model for HBW trip distribution:
Alternative B:
- Use modelled trip productions and attractions as inputs for calibrating the gravity model parameters
- Include all the o-d pairs including those with zero trips. This requires aggregating the movements to sector (SA2) level to remove the zeros that are due to the small sample size in HTS
Base Year Performance – for model calibration
Page 12May 1, 2023
Observed number of trip productions and attractions are used to produce the trip distribution curves
Base Year Performance – for model validation
Presentation Title Page 13May 1, 2023
Modelled number of trip productions and attractions are used to produce the trip distribution curves
Forecasting Performance
Presentation Title Page 14May 1, 2023
Modelled elasticity of changes in trip length in response to cost increase
Case Study – Trip Distribution and Mode Choice
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 16May 1, 2023
Alternative A: mode choice occurs after the trip distribution (or destination choice)
This ordering is computationally convenient: OD trip tables from distribution provide inputs to the mode choice
Arguably, this order also fits in a modelers’ experience of real life in so far as on many occasions travelers’ decide on their destinations first and then choose between transport modes
Develop destination and mode choice model for HBW trips:
Case Study – Trip Distribution and Mode Choice
Presentation Title Page 17May 1, 2023
Develop destination and mode choice model for HBW trips:
Alternative B: mode choice occurs before the trip distribution (or destination choice).
Less computationally straightforward: the generalized cost input for mode choice must be aggregated at trip production level, instead of at trip (origin-destination) level.
Due to the aggregation of cost inputs, the resultant base year validation outcomes tend to be less desirable than the destination choice first approach.
Validation Performance - Overall mode shares
Presentation Title Page 18May 1, 2023
Similar performance between the two alternatives
Validation Performance – Mode share for SA2 to SA2 movements
Presentation Title Page 19May 1, 2023
Alternative A:
Validation Performance – Mode share for SA2 to SA2 movements
Presentation Title Page 20May 1, 2023
Alternative B – similar performance to Alternative A
Forecasting performance
May 1, 2023 Page 21
Car GC = 30 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Base case assumptions:
1000 trips from origin to destination D1, D2, via Car or PT
Destination and Mode choice results in base case (for both alternative):
D1, Car = 478 trips
D1, PT = 22 trips
D2, Car = 478 trips
D2, PT = 22 trips
Forecasting performance
May 1, 2023 Page 22
Car GC = 40 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Scenario testing:
Car travel cost to D1 increases from 30 to 40 minutes
Destination and Mode choice results , Alternative A (double constrained)
D1, Car = 432 trips (decrease from 478 in Base)
D1, PT = 68 trips (increase from 22 trips in Base)
D2, Car = 478 trips (unchanged from Base)
D2, PT = 22 trips (unchanged from Base)
Forecasting performance
May 1, 2023 Page 23
Car GC = 40 mins
PT GC = 70 min
Car GC = 30 mins
PT GC = 70 min
D1
D2
Scenario testing:
Car travel cost to D1 increases from 30 to 40 minutes
Destination and Mode choice results , Alternative B
D1, Car = 394 trips (more significant decrease)
D1, PT = 106 trips (more significant increase)
D2, Car = 480 trips (nearly unchanged)
D2, PT = 20 trips (nearly unchanged)
Summary
Implications
Overfitting: Use synthetic measures for fitting base year outcomes to observed data, which may affect forecasting performance adversely.
Underperforming: Accommodate underperformance (compared with overfitting option) in base year validation, in exchange of improved forecasting performance
Another option:
Common practice?
Thank You