Hypothesis-Testing
Model-Complexity
Hypothesis Testing …..
Domain of groundwater model ...
…topographic contours ...
… a dam ...
… irrigated area ...
… channel system ...
… extraction bores ...
… native woodland ...
… observation bores
Inflow from uphill
Supplied “from outside”
Inflow from uphill
Groundwater interaction with rivers
Supplied “from outside”
Inflow from uphillGroundwater interaction with dam
Groundwater interaction with rivers
Supplied “from outside”
Inflow from uphillGroundwater interaction with dam
Groundwater interaction with rivers
Leakage from channels
Supplied “from outside”
Inflow from uphillGroundwater interaction with dam
Groundwater interaction with rivers
Leackage from channels
Aquifer extraction
Supplied “from outside”
Inflow from uphillGroundwater interaction with dam
Groundwater interaction with rivers
Leackage from channels
Groundwater recharge
Aquifer extraction
Supplied “from outside”
More often than not, a definitive model cannot be built.
Recognize this, focus on the question that is being asked and, if necessary, use the model for hypothesis testing.
Remember that model calibration is a form of data interpretation. The whole modelling process is simply advanced data processing.
Cattle Ck.
Cattle Creek Catchment
Soils and current land use
Model grid; fixed head and drainage cells shown coloured
Groundwater levels in June 1996
Groundwater levels in January 1991
Modelled and observed water levels after model calibration.
264
279
4
1000
1000 10001000 253
77
10
171000
56
9
1000
2
2
2 3
7
18
3
27
24
Calibrated transmissivities
Cattle Creek Catchment
CANE EXPANSION
New Development CURRENT
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
46R10P8
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
46R15P8
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
Zone 17 absent
48R14P8
Increased cane productionLeakage from balancing storage:
0.0 mm/d at calibration0.0 mm/d for prediction
46R3P7
Increased cane productionLeakage from balancing storage:
0.0 mm/d at calibration0.0 mm/d for prediction
46R4P7
Increased cane productionLeakage from balancing storage:
0.0 mm/d at calibration0.0 mm/d for prediction
Zone 17 absent
48R8P7
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
46R10P10
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
46R11P10
Increased cane productionLeakage from balancing storage:
2.5 mm/d at calibration2.5 mm/d for prediction
Zone 17 absent
48R14P10
P E
d
M
Ks
Simple ModelRunoff
P E
d
M
Ks
Runoff
Simple Model
•M Soil Moisture Capacity (mm/m depth)•d Effective Rooting Depth•Ki Initital loss•fcap Field Capacity•Ks Saturated Hydraulic Conductivity
MP E
d
M
Ks
Simple ModelRunoff
•M Soil Moisture Capacity (mm/m depth)•d Effective Rooting Depth•Ki Initital loss•fcap Field Capacity•Ks Saturated Hydraulic Conductivity
p1
p2
A probability contour:-
“Fixing” a parameter
p1
p2
This has the potential to introduce bias into key model predictions.
A probability contour:-
p1
p2
Also, what if this parameter is partly a surrogate for an unrepresented process?
A probability contour:-
p1
p2
“Fixing” a parameter
A probability contour:-
p1
p2
“Fixing” a parameter
A probability contour:-
• Not only does uncertainty arise from parameter nonuniqueness; it also arises from lack of certainty in model inputs/outputs and model boundary conditions.
• The model can be used as an instrument for data interpretation, allowing various hypotheses concerning inputs/outputs and boundary conditions to be tested.
• Where did the idea ever come from that there should be one calibrated model?
modeller
construction calibration prediction
“the deliverable”
prediction
“the deliverable”
prediction
“the deliverable”
modeller
construction calibration prediction
“Dual calibration”
Observation bore
Pumped bore
K = 5Sy = 0.1
K = 5Sy = 0.1
K = 25Sy = 0.3In
flow
= 2
750
Fix
ed h
ead
= 5
0
A River Valley
Recharge × 10-3
0 100 200 300
0
1
2
Recharge rate
0 100 200 300
0
1000
2000 Discharge
Discharge
0 100 200 300
0
1000
2000
Pumping rate
Water level
Water level
0 100 200 300
48
52
56
0 100 200 300
48
52
56
Borehole hydrographs
The finite-difference grid
The finite-difference grid
and parameter zonation
0 100 200 300
48
52
56
K=5; Sy=0.1
K=5; Sy=0.1
K=25; Sy=0.3
Calibrated parameters
Field dataModel-calculated
Field andmodel-generatedboreholehydrographs
0 100 200 300
48
52
56
K=10.2; Sy=0.21
K=10.2; Sy=0.21
K=18.8; Sy=0.21
Field dataModel-calculated
Calibrated parameters
Field andmodel-generatedboreholehydrographs
Simulation of Drought Conditions
• Decrease inflow from left from 2750 to 2200 m3/day.
• Increase pumping from left bore from (1500, 1000, 0, 1500)
to 2000 m3/day.
• Increase pumping from right bore from
(2000,1000,500,1500) to 3000 m3/day.
• Run model for 91 days.
• Same initial heads, ie. 54 m.
For “true parameters”, water level in right bore after this run is 43.9m.
Is it possible that the water level in the left bore will be as low as 42m?
Use PEST with “model” comprised of two MODFLOW runs, one under calibration conditions and one under predictive conditions.
In the latter case there is only one “observation”, viz water level in right pumped cell is 42m at end of run (weight is the sum of the weights used for all water levels over calibration period).
Methodology
Model
Input files
Output files
PEST
writes model input files
reads model output files
Modelcalibration conditions
Input files
Output files
PEST
Input files
Modelpredictive conditions
Output files
0 100 200 300
48
52
56
K=22; Sy=0.14
K=16; Sy=0.16
K=9.8; Sy=0.28
Field dataModel-calculated
Field andmodel-generatedboreholehydrographs overcalibration period.
Water level in right pumped bore at end ofdrought = 42m.
Calibrated parameters
Is it possible that the water level in the left bore will be as low as 40m?
0 100 200 300
48
52
56
K=22; Sy=0.14
K=16; Sy=0.16
K=9.8; Sy=0.28
Field dataModel-calculated
Water level in right pumped bore at end ofdrought = 40m.
Field andmodel-generatedboreholehydrographs overcalibration period.
Calibrated parameters
0 100 200 300
48
52
56
K=5; Sy=0.099
K=14; Sy=0.11
K=20; Sy=0.32
Field dataModel-calculated
Water level in right pumped bore at end ofdrought = 40m.
K=4.6; Sy=0.090
Calibrated parameters
Field andmodel-generatedboreholehydrographs overcalibration period.
Is it possible that the water level in the left bore will be as low as 36m?
0 100 200 300
48
52
56
K=8.8; Sy=0.13
K=15; Sy=0.14
K=18; Sy=0.29
Field dataModel-calculated
Water level in right pumped bore at end ofdrought = 36m.
K=2.7; Sy=0.19
Calibrated parameters
Field andmodel-generatedboreholehydrographs overcalibration period.
We are not calibrating a groundwater model.
We are calibrating our regularisation
methodology.
Some Lessons
• if possible, include in the calibration dataset measurements of the type that you need to predict
• intuition and knowledge of an area plays just an important part in modelling as does the model itself
• focus on what the model needs to predict when building the model…..
There should be no such thing as a model for an area, only for a specific problem.
So how should we model?
open cut mine
open cut mine
underground mine
underground mine
waterholes
A model area
extraction bores
open cut mine
open cut mine
underground mine
underground mine
waterholes
A model area
extraction bores
monitoring bores
guaging stations
A model area
A model area
A model area
A model area
A model area
A model area
A model area
A model area
A model area
A model area
Sources of Uncertainty Close to Waterholes
• conductance of bed (and heterogeneity thereof)
• change in bed conductance with wetted perimeter
• change in bed conductance with inflow/outflow and season
• relationship between area and level
• relationship between level and flow
• rate of evaporation
• hydraulic properties of rocks close to ponds
• behaviour during flood events
• change in hydraulic characteristics after flood events
• uncertainty in future flows
• inflow to ponds from neighbouring surface catchment
• lack of borehole data to define groundwater mounds
• uncertainties in streamflow
Let’s start again…..
Complexity leads to parameter uncertainty.
Parameter correlation can be enormous due to inadequate data.
Parameter uncertainty may lead to predictive uncertainty.
The more that the prediction depends on system “fine detail”, the more this is likely to occur.
Predictive uncertainty must be analysed.
Complexity must be “focussed” - dispense with non-essential complexity.
No model should be built independently of the prediction which it has to make.
A model area
A model area
A model area
A model area
A model area
A model area
open cut mine
open cut mine
underground mine
underground mine
waterholes
Sensitive area
open cut mine
open cut mine
underground mine
underground mine
waterholes
Sensitive area
open cut mine
open cut mine
underground mine
underground mine
waterholes
Sensitive area
A model is not a database! A model is a data processor.
Ubiquitous complexity in a “do-everything model”
Ubiquitous complexity in a “do-everything model”
Focussed complexity in a prediction-specific model
Focussed complexity in a prediction-specific model
Model Complexity
For reasons which we have already discussed, a complex model is really a simply model in disguise.
Complex models:-
More parameters Longer run times Greater likelihood of numerical
instability More costly Destroys user’s intuition
The level of complexity is set by system properties to which the prediction is most sensitive.
p1
p2
Objective functionminimum
Objective function contourslinear model
p1
p2
A probability contour:-
p1
p2
11
A probability contour:-
p1
p2
11
2
2
A probability contour:-
p1
p2
A probability contour:-
p1
p2
p1+p2
A probability contour:-
p1
p2
p1+p2 p1-p2
Ideally, simplification of a model should be done in such a way that only the parameters that “don’t matter” are dispensed with.
There are many cases where a specific prediction depends on at least one of the values of the individual parameters - the parameters that cannot be resolved by the parameter estimation process.
In fact, that is often why we are using a physically based model; if calibration alone sufficed for full parameterisation, then a black box would be all we need.
p1
p2
Over-simplified model design introduces bias, for we are effectively assuming values for unrepresented parameters.
p1
p2
A probability contour:-
“Fixing” a parameter
p1
p2
A probability contour:-
“Fixing” a parameter
p1
p2
A probability contour:-
“Fixing” a parameter
Increasing model complexity
pote
ntia
l er r
o r in
pre
dict
ion
complexity
bias
But we don’t know how much bias we are introducing.
?
Increasing model complexity
complexity
bias
predictive uncertainty
These levels are equalpo
tent
ial e
r ro r
in p
redi
ctio
n
Increasing model complexity
complexity
bias
predictive uncertainty
These levels are equalpo
tent
ial e
r ro r
in p
redi
ctio
n
The point where no further complexity is warranted, is the point where the uncertainty of a specific model prediction no longer rises.
Essential and non-essential complexity are prediction-dependent.
Complexity does not guarantee the “right answer” - it guarantees that the right answer will lie within the limits of predictive uncertainty.
Complexity without uncertainty analysis is a waste of time. A complex model can be just as biased as a simple model.
Use a simple model and add the “predictive noise” – far cheaper.
A complex model allows you to replace “predictive noise” with science. But if you don’t do it, what is the point of a complex model.
An Example….
NO RTH C AR O LINA
Neuse R iver basin
Contentnea C reekwatershed
N C County BoundariesSandy RunM iddle Sw am pLittle C ontentneaC ontentneaN euse
(77 km 2)
(140 km 2)
(470 km 2)
(2600 km 2)
(14500km 2)
1
10
100
1000
10000
1-Jan-83 1-Mar-83 1-May-83 1-Jul-83 1-Sep-83 1-Nov-83 1-Jan-84
Observed and modelled flows
0.E+00
1.E+09
2.E+09
3.E+09
4.E+09
5.E+09
6.E+09
7.E+09
8.E+09
1970 1972 1974 1976 1978 1980 1982 1984 1986
Observed and modelled monthly volumes
0
0.2
0.4
0.6
0.8
1
10 100 1000 10000
Flow (cu ft /sec)
Exc
ee
de
nce
fra
ctio
n
Observed and modelled exceedence fractions
ParameterLZSN 2.0UZSN 2.0INFILT 0.0526BASETP 0.200AGWETP 0.00108LZETP 0.50INTFW 10.0IRC 0.677AGWRC 0.983
1
10
100
1000
10000
1-Jan-83 1-Mar-83 1-May-83 1-Jul-83 1-Sep-83 1-Nov-83 1-Jan-84
Observed and modelled flows
0.E+00
1.E+09
2.E+09
3.E+09
4.E+09
5.E+09
6.E+09
7.E+09
8.E+09
1970 1972 1974 1976 1978 1980 1982 1984 1986
Observed and modelled monthly volumes
0
0.2
0.4
0.6
0.8
1
10 100 1000 10000
Flow (cu ft/sec)
Exc
eede
nce
frac
tion
Observed and modelled exceedence fractions
Parameter Set 1 Set 2 Set 3 Set 4 Set 5 Set 6LZSN 2.0 2.0 2.0 2.0 2.0 2.0UZSN 2.0 1.79 2.0 2.0 1.76 2.0INFILT 0.0526 0.0615 0.0783 0.0340 0.0678 0.0687BASETP 0.200 0.182 0.199 0.115 0.179 0.200AGWETP 0.00108 0.0186 0.0023 0.0124 0.0247 0.0407LZETP 0.50 0.50 0.20 0.72 0.50 0.50INTFW 10.0 3.076 1.00 4.48 4.78 2.73IRC 0.677 0.571 0.729 0.738 0.759 0.320AGWRC 0.983 0.981 0.972 0.986 0.981 0.966
1
10
100
1000
10000
1-Jan-93 1-Mar-93 1-May-93 1-Jul-93 1-Sep-93 1-Nov-93 1-Jan-94
Observed and modelled flows over validation period
0.E+00
1.E+09
2.E+09
3.E+09
4.E+09
5.E+09
6.E+09
7.E+09
8.E+09
9.E+09
1.E+10
1986 1987 1988 1989 1990 1991 1992 1993 1994 1995
Observed and modelled monthly volumes over validation period
0
0.2
0.4
0.6
0.8
1
10 100 1000 10000
Flow (cu ft/sec)
Exc
ee
de
nce
fra
ctio
n
Observed and modelled exceedence fractions over validation period
1
10
100
1000
10000
1-Jan-93 1-Mar-93 1-May-93 1-Jul-93 1-Sep-93 1-Nov-93 1-Jan-94
Observed and modelled flows over validation period
1
10
100
1000
10000
1-Jan-93 1-Mar-93 1-May-93 1-Jul-93 1-Sep-93 1-Nov-93 1-Jan-94
Observed and modelled flows over validation period
Parameterisation using PEST’s predictive analyser
1
10
100
1000
10000
1-Jan-83 1-Mar-83 1-May-83 1-Jul-83 1-Sep-83 1-Nov-83 1-Jan-84
Observed and modelled flows over calibration period
ParameterLZSNUZSNINFILTBASETPAGWETPLZETPINTFWIRCAGWRC
ParameterLZSNUZSNINFILTBASETPAGWETPLZETPINTFWIRCAGWRCDEEPFR
Observed and modelled flows over validation period
Parameterisation using PEST’s predictive analyser
1
10
100
1000
10000
1-Jan-93 1-Mar-93 1-May-93 1-Jul-93 1-Sep-93 1-Nov-93 1-Jan-94
1
10
100
1000
10000
1-Jan-83 1-Mar-83 1-May-83 1-Jul-83 1-Sep-83 1-Nov-83 1-Jan-84
Observed and modelled flows over calibration period