Cross Validation for Selection of Variogram Model

Cross Validation for Selection of Variogram Model andKriging Type: Application to IP Data from West Virginia

Bora Oz ([email protected]) and Clayton V. Deutsch ([email protected])Department of Civil & Environmental Engineering, University of Alberta

Abstract

A variogram model and kriging type must be chosen prior to the generation of geostatisticalmodels. Cross Validation gives us some ability to assess the impact of our many choices suchas the variogram models, kriging type, and search strategies. For estimation, one can plotthe estimated values versus true values and look for the scenario with the highest correlationcoecient. Other tools are available to check the results of simulation.

The aim of this paper is to review the methodology to choose parameters such as the can-didate variogram model and kriging type. An example study with initial potential data fromBarbour County, West Virginia is presented to clarify the application of this methodology.

KEY WORDS: simulation, accuracy, precision, uncertainty, cyclicity

Introduction

In practice, we would like to compare dierent geostatistical prediction scenarios and choosethe one that works best. These scenarios consist of dierent weighting procedures, searchstrategies, variogram models and so on. The comparison can be based on the dierencebetween true and estimated values. Cross Validation allows us to compare estimatedand true values using the information available in our sample data set. The sample valuesare temporarily discarded from the sample data set; the value is then estimated using theremaining samples. The estimates are then compared to the true values.

Since its intoduction in the geostatistical literature [2, 3], cross validation has been usedwidely for dierent purposes. Delner [3] used it to search for the generalized covariancefunction to be used in his estimation procedure among a nite number of candidates. Crossvalidation is used to nd the best model among the competitors. Clark [1] reviews thehistory of validation and its usefulness in geostatistics. She points out that this type ofcomparison was used to compare methods of estimation [2, 7] and to justify the use ofkriging as an estimation method [8]. Clark demonstrates that use of cross validation forselecting a semivariogram model may be acceptable, but may not be sensitive enough to bevery useful.

Notwithstanding some limitations, the results of cross validation provide practitionersan ability to examine and reformulate models to better utilize and conform to the data athand. Cross validation is primarily suited to exploratory purposes: it does not provide adenitive measure of the goodness of an estimation / simulation method.

1

We consider cross validation to help determine the variogram model and kriging type.An example application to initial potentials in Barbour County, West Virginia, is used toclarify the use of cross validation in estimation and simulation modes.

Methodology

Determine Scenarios to Test

Variograms are used to describe spatial variability. For a given eld or data set, one canconsider dierent variograms to describe the spatial distribution of the variables underconsideration; a number of dierent variogram models may appear to t the data set equallywell. It is for that reason that cross validation technique is used to test the calculatedvariograms

Values are estimated by kriging with the candidate variogram models. We can usedierent kriging types such as Simple Kriging or Ordinary Kriging. We can also proposedierent search parameters. A reasonable set of scenarios consisting of a variogram model,kriging type, and search strategy are proposed for testing with cross validation.

Cross Validation

The cross validation exercise is repeated for search scenario and then the results are com-pared. Each sample point is left out of the data set in turn and estimated with the remainingdata and appropriate scenario parameters.

In estimation, an error is calculated as the estimated value minus the true value. Theseerror values can be analyzed in dierent ways. The most common ones:

A scatterplot of the true values versus the estimated values. There should be fewoutliners and a high correlation coecient.

The histogram of errors should be symmetric, centered on a zero mean, with minimumstandard deviation.

The plot of the error versus the estimated value should be centered around the zero-error line, a property called conditional unbiasedness.

In simulation, we have a distribuition of simulated values instead of a single estimatedvalue; dierent techniques must be used to evaluate the results. Details can be found in thestudy of Deutsch [4]. In short, we check that the probabilistic model is both accurate andprecise while minimizing the unceratinity.

Application to Initial Potentials in Barbour County

The Data

The study area is located in the Barbour County, West Virginia. Hydrocarbon is beingproduced from Upper Devonian sandstones and siltstones. The data set, taken from Hohn[6], consists of 674 wells from a 22 km by 22 km area.

2

A location map of the eld is given in Figure 1. The potential values are higher in themiddle of the eld. There are large variations even between closely-spaced wells.

The histogram of the data is presented in Figure 2. It is clear from the histogram thatmost of that fall into the range of 30-1030 Mscfd. Values up to 16000 Mscfd are seen. Themean value is 1239.3 Mscfd.

Cell declustering was used to establish a more representative mean and sample his-togram. Figure 3 shows a cross plot of the declustered mean versus cell size. A Cell-size of3 km, corresponding to a value 1068 Mscfd, was chosen for declustering. These weights areused in simulation and for the simple kriging mean. The new histogram, with a mean of1068.0 Mscfd, is presented in Figure 4.

Candidate Variograms / Kriging Type

The experimental omminidirectional normal score semivariogram and the tted traditionalvariogram model (Candidate-1) are presented in Figure 5. A relatively high nugget value ofaround 0.45 is observed, which implies abrupt changes in the variable over small distances.The variogram rises too quickly for small values of lag distance which causes the range ofcorrelation appears at about 3 km.

The experimental variogram appears to oscillate. The general trend of the experimentalvariogram shows high and low values between successive lag distances, which is called holeeect or cyclicity. Therefore, our second candidate variogram model will be the one thatreects the eect of the cycylicity (Candidate-2). In Figure 6, the tted cyclic model andthe experimental variogram are presented.

Simple and Ordinary Kriging will be tested with both variogram models. We have fourdierent alternatives:

Simple Kriging with Traditional Variogram Model (Variogram Candidate-1) Simple Kriging with Cyclic Variogram Model (Variogram Candidate-2) Ordinary Kriging with Traditional Variogram Model Ordinary Kriging with Cyclic Variogram ModelFigure 7 shows the scatter plot of estimated values versus true values for the four al-

ternatives. It is clear ordinary kriging with cyclic variogram has the highest correlation.Another interesting conclusion is that cyclic variogram with simple kriging has higher corre-lation than the traditional variogram with ordinary kriging, which seems to place importanceon the cyclic variogram.

The mean of the error distribution is often referred to as the bias and a goal for the anyestimation method is to produce unbiased estimates. Ideally, one would like the mean to bezero which shows our overestimates and underestimates are in balance. Besides, we preferto have a more symmetric error distribution.

Figure 8 shows a histogram of the errors for the four alternatives. Although the meanof the four distributions are not equal to zero, when we compare them within the wholerange of error, they are small and assemble around zero. The Ordinary Kriging with cyclicvariogram model has the mean closest to zero. One clear observation is that all of themhave large underestimates reaching to a absolute error value of 14000.

3

Table 1: Summary of the mean squared errors, MSE, for the four alternatives

Another feature one want to have is that a small spread of error distribution. The MeanSquared Error, MSE, incorporates both the mean (or bias) and the spread of the errordistribution and can be calculated by,

MSE = E{[z z]2

}= errorvariance+ bias2 (1)

The results of Mean Squared Error, MSE, for the four alternatives are given in Table 1. Itis clear that while the three candidates have approximately same Mean Squared Error, theSimple Kriging with traditional variogram (Variogram Candidate-1) has the largest MSE,reaching twice as much as the alternatives.

Another tool to analyze the error data is to check for the conditional unbiasedness.Ideally, we would like to subdivide our estimates into many dierent groups and have anunbiased error distribution within each group (i.e. conditional unbiasedness). One wayof checking for conditional bias is to plot the errors as a function of the estimates values.Ideally, this scatter plot would result in; (1) no correlation, and (2) no increase in estimationvariance.

Error versus estimate plots for simple and ordinary kriging with cyclic variogram modelare given in Figure 9. It is clear that both of the models result in overall no correlation;however, for both of these models, estimation variances increase with the estimates. Thereason for this may be explained by the proportional eect or heteroscedasticity. Itis common to nd that the data values in some regions are more variable than in others,called heteroscedasticity.

The proportional eect is a positive correlation between the local means and the localvariances. After applying moving window averaging technique, by 2 km by 2 km windowsand ignoring any resultant window which contains less than 3 data; the scatter plot of thelocal means versus local variances is given in Figure 10 for the Ordinary Kriging with cyclicvariogram model (Variogram Candidate-2). It is clear that there is strong correlation be-tween the local means and the local variances. This explains the reason why the estimationvariance increases with the estimates. The data show (1) an increase in local variance withan increase in local mean, and (2) large variability between the data values.

Simulation mode: Accuracy, Precison and Uncertainity

We need to check the goodness of the probabilistic model itself. In other words, we haveto assess our model and answer the question of How good are our probabilities?. Two

4

Table 2: Summary of the results for simulation mode.

concepts, accuracy and precision help us answer this question .The accuracy and precision of the previous four alternatives are checked using the pro-

gram accplot from GSLIB [5]. The accuracy plots for the four alternatives are given inFigures 11.The accuracy plots show that until the probability interval of 0.3, all the meth-ods are accurate and precise, but for the other intervals, points are away from the 45 degreeline which makes the models inaccurate. That is the variance of the local distributions istoo high.

In Table 2, the results of the accuracy, precision and uncertainity are given respectivelyfor the four alternatives. It is again clear that cyclic variogram is more successful than theothers. When we look at the uncertainity values, cyclic variogram model with OrdinaryKriging has less uncertainity than the alternatives.

Discussion and Conclusion

Many of the statistics including the mean of the error distributions and the MSE values areambiguous and do not help for selection purposes with the Barbour County data. Never-theless, the highest correlation between estimate and true identies ordinary kriging withthe cyclic variogram model (Variogram Candidate-2) as the best alternative.

In simulation mode, lone again, ordinary kriging with the cyclic variogram model hasthe highest accuracy and precision values with the lowest uncertainity.

By looking at the results of estimation and simulation modes, the cyclic variogram withOrdinary Kriging appears as the best model among the others. Nevertheless, we shouldnote that best applies in a restricted sense. A variogram model is best, for instance, onlyunder the choice of discrepancy measure, data number, predictive function, and number ifmodels to be evaluated. If a change exists in any of these, a new best model may emerge.If one model performs better than another on a particular data set, it does not mean that,

5

in general, that the model will always perform better. The result does not guarantee thesuccess of the nal performance becauese our ability to produce good estimates at samplelocations may have little relevance to the nal estimation study.

The high correlation between the local means and local variances (or proportional eect)entails a correlation between the error variance and the estimate.

Cross validation is used best as an exploratory technique. It can lead the user toreexamine and reformulate models to better utilize and conform to the data at hand.

References

[1] I. Clark. The art of cross validation in geostatistical application. Proceeding of APCOM,pages 211-220, 1986.

[2] M. David. The practice of kriging,. Advance Geostatistics in Mining Industry, pages31-48, 1976.

[3] P. Delner. Linear estimation of nonstationary spatial phenomena,. Advance Geostatis-tics in Mining Industry, pages 49-68, 1976.

[4] C. V. Deutsch. Direct assessment of local accuracy and precision. In E. Y. Baa andN. A. Schoeld, editors, Geostatistics Wollongong 96, volume 1, pages 115125. KluwerAcademic Publishers, 1996.

[5] C. V. Deutsch and A. G. Journel. GSLIB: Geostatistical Software Library and UsersGuide. Oxford University Press, New York, 1992.

[6] Michael Edward Hohn. Geostatistics and Petroleum Geology, Second Edition. WestVirginia Geological Survey.

[7] A. G. Journel and Ch. J. Huijbregts. Mining Geostatistics. Academic Press, New York,1978.

[8] H. M. Parker, A. G. Journel, and W. C. Dixon. The use of conditional lognormalprobabiliy distribution fo the estimation of open-pit ore reserves in strata-bound ura-nium deposits-a case study. Proceeding of APCOM, Society of Mining Engineers, pages133-148, New York, 1979.

6

X, km

Y, km

569.0 574.0 579.0 584.0 589.04319.0

4324.0

4329.0

4334.0

4339.0

2004507009501200145017001950220024502700295032003450370039504200445047004950

Figure 1: Location map of the initial potentials.

Freq

uenc

y

Initial Potential, Mscfd

30. 1030. 2030. 3030. 4030. 5030. 6030. 7030. 8030..000

.040

.080

.120

Number of Data 674mean 1239.3

std. dev. 1595.4coef. of var 1.29

maximum 16021.0upper quartile 1399.0

median 789.5lower quartile 400.0

minimum 30.0

Figure 2: Histogram of the initial potentials.

7

Dec

lust

ered

Mea

n

Cell Size

.0 2.0 4.0 6.0 8.0 10.00.

400.

800.

1200.

Figure 3: Declustered mean versus cell size

Freq

uenc

y

Thickness, ft

30. 1030. 2030. 3030. 4030. 5030. 6030. 7030. 8030..000

.040

.080

.120

Number of Data 674mean 1098.0

std. dev. 1379.4coef. of var 1.26

maximum 16021.0upper quartile 1301.7

median 753.1lower quartile 381.0

minimum 30.0weights used

Figure 4: Histogram of the initial potentials (weight used).

8

Distance, km

Candidate Variogram Model#1

.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00.00

.20

.40

.60

.80

1.00

1.20

Figure 5: Experimental normal score semivariogram and the rst candidate model (omminidirec-tional)

Distance, km

Candidate Variogram Model#2

.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00.00

.20

.40

.60

.80

1.00

1.20

Figure 6: Experimental normal score semivariogram and the second candidate variogram model(cyclicity)

9

True

Estimate

SK- Candidate Variogram-1

0. 2000. 4000. 6000. 8000. 10000.200.

1200.

2200.

3200.

4200.

5200. Number of data 674Number plotted 670

X Variable: mean 1220.0std. dev. 816.3

Y Variable: mean 1239.3std. dev. 1595.4

correlation .453rank correlation .425

True

Estimate

SK- Candidate Variogram-2

0. 2000. 4000. 6000. 8000. 10000.200.

1200.

2200.

3200.

4200.

5200. Number of data 674Number plotted 670




True

Estimate

OK- Candidate Variogram-1

0. 2000. 4000. 6000. 8000. 10000.200.

1200.

2200.

3200.

4200.

5200.

Number of data 674Number plotted 670




True

Estimate

OK- Candidate Variogram-2

0. 2000. 4000. 6000. 8000. 10000.200.

1200.

2200.

3200.

4200.

5200.

Number of data 674Number plotted 670




Figure 7: Cross Validation results for the four alternative scenarios.

10

Figure 8: Histogram of the errors for the four alternatives.

11

Figure 9: Error versus estimate plots for simple and ordinary kriging with cyclic variogram model.

loca

l var

ianc

e

local mean

0. 1000. 2000. 3000. 4000.1000

10000

100000

1.0e+6

1.0e+7

1.0e+8Number of data 90Number plotted 90




Figure 10: Local mean versus local variance after moving average for ordinary kriging with cyclicvariogram model.

12

Wid

th o

f Loc

al D

ists

Number in This Width

SK- Accuracy plot for Variogram Candidate-1

.00 .20 .40 .60 .80 1.00.00

.20

.40

.60

.80

1.00

Wid

th o

f Loc

al D

ists


SK- Accuracy plot for Variogram Candidate-2

.00 .20 .40 .60 .80 1.00.00

.20

.40

.60

.80

1.00

Wid

th o

f Loc

al D

ists


OK- Accuracy plot for the Variogram Candidate-1

.00 .20 .40 .60 .80 1.00.00

.20

.40

.60

.80

1.00

Wid

th o

f Loc

al D

ists


OK- Accuracy plot for the Variogram Candidate-2

.00 .20 .40 .60 .80 1.00.00

.20

.40

.60

.80

1.00

Figure 11: Accuracy plots for the four alternatives.

13

Documents

Cross Validation for Selection of Variogram Model