Comparing exact and approximate spatial auto-regression model

Comparing Exact and Approximate Spatial Auto-Regression Model Solutions for Spatial Data Analysis

Baris M. Kazar1, Shashi Shekhar2, David J. Lilja1, Ranga R. Vatsavai2, and R. Kelley Pace3

1 Electrical and Computer Engineering Department, University of Minnesota, Twin-Cities MN 55455

{Kazar, Lilja}@ece.umn.edu 2 Computer Science and Engineering Department,

University of Minnesota, Twin-Cities MN 55455

{Shekhar, Vatsavai}@cs.umn.edu 3 LREC Endowed Chair of Real Estate 2164B CEBA, Department of Finance

E.J. Ourso College of Business Louisiana State University

Baton Rouge, LA 70803-6308 [email protected]

Abstract. The spatial auto-regression (SAR) model is a popular spatial data analysis technique, which has been used in many applications with geo-spatial datasets. However, exact solutions for estimating SAR parameters are computa-tionally expensive due to the need to compute all the eigenvalues of a very large matrix. Recently we developed a dense-exact parallel formulation of the SAR parameter estimation procedure using data parallelism and a hybrid pro-gramming technique. Though this parallel implementation showed scalability up to eight processors, the exact solution still suffers from high computational complexity and memory requirements. These limitations have led us to investi-gate approximate solutions for SAR model parameter estimation with the main objective of scaling the SAR model for large spatial data analysis problems. In this paper we present two candidate approximate-semi-sparse solutions of the SAR model based on Taylor series expansion and Chebyshev polynomials. Our initial experiments showed that these new techniques scale well for very large data sets, such as remote sensing images having millions of pixels. The results also show that the differences between exact and approximate SAR parameter estimates are within 0.7% and 8.2% for Chebyshev polynomials and Taylor se-ries expansion, respectively, and have no significant effect on the prediction accuracy.

1 Introduction

Explosive growth in the size of spatial databases has highlighted the need for spatial data analysis and spatial data mining techniques to mine the interesting but implicit spatial patterns within these large databases. Extracting useful and interesting patterns from massive geo-spatial datasets is important for many application domains, such as regional economics, ecology and environmental management, public safety, transpor-tation, public health, business, and travel and tourism [8,34,35]. Many classical data mining algorithms, such as linear regression, assume that the learning samples are independently and identically distributed (i.i.d). This assumption is violated in the case of spatial data due to spatial autocorrelation [2,34] and in such cases classical linear regression yields a weak model with not only low prediction accuracy [35] but also residual error exhibiting spatial dependence. Modeling spatial dependencies improves overall classification and prediction accuracies.

The spatial auto-regression model (SAR) [10,14,34] is a generalization of the linear regression model to account for spatial autocorrelation. It has been successfully used to analyze spatial datasets in regional economics and ecology [8,35]. The model yields better classification and prediction accuracy [8,35] for many spatial datasets exhibiting strong spatial autocorrelation. However, it is computationally expensive to estimate the parameters of SAR. For example, it can take an hour of computation for a spatial dataset with 10K observation points on a single IBM Regatta processor using a 1.3GHz pSeries 690 Power4 architecture with 3.2 GB memory. This has limited the use of SAR to small problems, despite its promise to improve classification and pre-diction accuracy for larger spatial datasets. For example, SAR was applied to accu-rately estimate crop parameters [37] using airborne spectral imagery; however, the study was limited to 74 pixels. A second study, reported in [21], was limited to 3888 observation points.

Table 1. Classification of algorithms solving the serial spatial auto-regression model

Exact Approximat e

Applying Direct Sparse M atrix Algorithms [25] M L based M atrix Exponential Specif icat ion [26]

Eigenvalue based 1-D Surface Part it ioning [16] Graph Theory Approach [32]

Taylor Series Approximat ion [23]

Chebyshev Polynomial Approximation M ethod [30]

Semiparametric Est imates [27]

Characterist ic Polynomial Approach [36]

Double Bounded Likelihood Est imator [31]

Upper and Lower Bounds via Divide & Conquer [28]

Spatial Autoregression Local Est imat ion [29]

Bayesian M atrix Exponent ial Specif icat ion [19]

M arkov Chain M onte Carlo (M CM C) [3,17]

M aximum Likelihoo d

B ayesian None

A number of researchers who have been attracted to SAR because of its high com-

putational complexities have proposed efficient methods of solving the model. These solutions, summarized in Table 1, can be classified into exact and approximate solu-tions, based on how they compute certain compute-intensive terms in the SAR solu-

tion procedure. Exact solutions suffer from high computational complexities and memory requirements. Approximate solutions are computationally feasible, but many of these formulations still suffer from large memory requirements. For example, a standard remote sensing image consisting of 3000 lines (rows) by 3000 pixels (col-umns) and six bands (dimensions) leads to a large neighborhood (W) matrix of size 9 million rows by 9 million columns. (The details for forming the neighborhood matrix W can be found in Sect. 2.) Thus, the exact implementations of SAR are simply not capable of processing such large images, and approximate solutions must be found. We choose Taylor and Chebyshev approximations for two reasons. First, the solu-tions are scalable for large problems and secondly these methods provide bounds on errors.

Major contributions of this study include scalable implementations of the SAR model for large geospatial data analysis, characterization of errors between exact and approximate solutions of the SAR model, and experimental comparison of the pro-posed solutions on real satellite remote sensing imagery having millions of pixels. Most importantly, our study shows that the SAR model can be efficiently imple-mented without loss of accuracy, so that large geospatial datasets which are spatially auto-correlated can be analyzed in a reasonable amount of time on general purpose computers with modest memory requirements. We are using an IBM Regatta in order to implement parallel versions of the software using open source ScaLAPACK [7] linear algebra libraries. However, the software can also be ported onto general-purpose computers after replacing ScaLAPACK routines with the serial equivalent open source LAPACK [1] routines. Please note that, even though we are using a parallel version of ScaLAPACK, the computational timings presented in the results section (Table 7) are based on serial execution of all SAR model solutions on a single processor. The remainder of the paper is organized as follows: Section 2 presents the problem statement, and Section 3 explains the exact algorithm for the SAR solution. Section 4 discusses approximate SAR model solutions using Taylor series expansion and Chebyshev polynomials respectively. The experimental design is provided in Section 5. Experimental results are discussed in Section 6. Finally, Section 7 summa-rizes and concludes the paper with a discussion of future work.

2 Problem Statement

We first present the problem statement and the notation in Table 2; and then explain the exact and approximate SAR solutions based on maximum-likelihood (ML) theory [12].

The problem studied in this paper is defined as follows: Given the exact solution procedure described in the Dense Matrix Approach [16] for one-dimensional geo-spatial datasets, we need to find a solution that scales well for large multi-dimensional geo-spatial datasets. The constraints are as follows: the spatial auto-regression pa-rameter ρ varies in the range [0,1); the error is normally distributed, that is, ε ∼N(0,σ2I) iid; the input spatial dataset is composed of normally distributed random

variables; and the size of the neighborhood matrix W is n. The objective is to imple-ment scalable and portable software for analyzing large geo-spatial datasets.

Table 2. The notation in this study

Variable Definition Variable Definition

ρ The spatial auto-regression (autocor-relation) parameter

I Identity matrix

y n-by-1 vector of observations on the dependent variable

λ Eigenvalue of a matrix

x n-by-k matrix of observations on the explanatory variable

tr(.) Trace of the “.” matrix

W n-by-n neighborhood matrix that accounts for the spatial relationships (dependencies) among the spatial data

π Pi constant which is equal to 3.14

k Number of features |.| Determinant of the “.” matrix

β k-by-1 vector of regression coeffi-cients

(.)-1 Inverse of the “.” matrix

n Problem size (also number of obser-vation points or pixels) iT (.) A Chebyshev polynomial of degree i.

“.” can be a matrix or a scalar number. p Row dimension of spatial frame-

work (image) k Index variable

q Column dimension of spatial framework (image) ∑(.) Summation operation on a matrix/vector

Index variable running on i of T (.) i

C n-by-n Binary neighborhood matrix ∏ Product operation on a matrix/vector

D n-by-n Diagonal matrix with elements , where is the row-sum of row i of C

is/1 isexp(.) Exponential operator i.e., e(.)

W~ n-by-n Symmetric equivalent of W matrix in terms of eigenvalues

(.)T Transpose of the “.” Matrix/vector

ε n-by-1 vector of unobservable error (.)ij ijth element of the “.” matrix 2σ The common variance of the error

ε ∑ n-by-n Diagonal variance matrix of error defined as I 2σ

ln(.) Natural logarithm operator O(.) “O” notation for complexity analysis of algorithms

q The highest degree of the Cheby-shev polynomials

N n-by-n Binary neighborhood matrix from Delaunay triangulation

Ψ Current pixel in the spatial frame-work (image) with “s” neighbors

cos(.) Cosinus trigonometric operation

2.1 Basic SAR Model

The spatial auto-regression model (SAR) [10], also known in the literature as a spa-tial lag model or mixed regressive model, is an extension of the linear regression model and is given in equation 1.

εxβWyy ++= ρ . (1)

Here the parameters are defined in Table 2. The main point to note here is that a spatial autocorrelation term Wyρ is added to the linear regression model in order to

model the strength of the spatial dependencies among the elements of the dependent variable, y.

2.2 Example Neighborhood Matrix (W)

The neighborhood matrices used by the spatial auto-regression model are the neighborhood relationships on a one-dimensional regular grid space with two neighbors and a two-dimensional grid space with “s” neighbors, where “s” is four, eight, sixteen, twenty-four and so on neighbors, as shown in Fig. 1. This structure is also known as regular square tessellation one-dimensional and two-dimensional pla-nar surface partitioning [12].

Ψ

1-D 2-neighbors

2-D 8-neighbors

2-D 16-neighbors

2-D 24-neighbors

2-D 4-neighbors

Fig. 1. The neighborhood structures of the pixel Ψ on one-dimensional and two-dimensional regular grid space

2.3 Illustration of the Neighborhood Matrix Formation on a 4-by-4 Regular Grid Space

As noted earlier, modeling spatial dependency (or context) improves the overall clas-sification (prediction) accuracy. Spatial dependency can be defined by the relation-ships among spatially adjacent pixels in a small neighborhood within a spatial frame-work that is a regular grid space. The following paragraph explains how W in the SAR model is formed. For the four-neighborhood case, the neighbors of the (i,j)th pixel of the regular grid are shown in Fig. 2.

≤≤≤≤−≤≤≤≤+≤≤≤≤+≤≤≤≤−

=

WEST21 )1,(SOUTH 111 j)1,(iEAST 111 )1,(

NORTH 12 ),1(

),(

qjp, ijiqj, p-iq-jp, i ji

qj p,iji

jineighbors

Fig. 2. The four neighbors of the (i,j)th pixel on the regular grid

The (i,j)th pixel of the surface will fill in the (p(i-1)+j)th row of the non-row-standardized neighborhood matrix, C. The following entries of C, i.e. {( p(i-1)+j),( p(i-2)+j)}, {( p(i-1)+j),( p(i-1)+j+1)}, {( p(i-1)+j),( p(i)+j)} and {( p(i-1)+j),( p(i-1)+j-1)} will be “1”s and the others all zeros. The row-standardized neighborhood matrix W is formed by first finding each row sum (i.e., there will be pq or n number of row-sums since W is pq-by-pq) and dividing each element in a row by its corre-sponding row-sum. In other words, where the elements of the diagonal

matrix C are defined as and . Fig. 3 illustrates the spatial frame-

work and the matrices. Thus, the rows of matrix W sum to 1, which means that W is row-standardized i.e., row-normalized or row-stochastic. A non-zero entry in the j

,CDW 1−=

ij 0=ijd∑==

n

iii cd1

th column of the ith row of matrix W indicates that the jth observation will be used to adjust the prediction of the ith row where i is not equal to j. We described forming a W matrix for a regular grid that is appropriate for satellite images; however, W can also be formed for irregular (or vector) datasets as discussed further in the Appendix [12].

1

0

0

0

010010000000000010100100000000000101001000000000010000100000001000010010000000010010100100000000100101001000000001001000010000000010000100100000000100101001000000001001010010000000010010000100000000100001000000000010010000000000001001010000000000010010

16151413121110987654321

16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

1 2 3 45 6 7 89 10 11 1213 14 15 16

111

0000000000021002

10

(a) (b) Fig. 3. (a) The spatial framework, which is p-by-q, where the pq-by-pq non-normalized neighborhood matrix C witnormalized version (i.e., W), which is also pq-by-pq. The psize

0 41

0 10

0 1

021002

100000000000

3103

100310000000000

03103

10031000000000

002100002

100000000

3100003003

10000000

041004

14

10041000000

004100404

1004100000

0003103

10000310000

0000310003

10031000

00000004104

1004100

0000041004

1041004

10

000000031003

1000031

000000002100002

100

00000000031003

10310

0000000000300303

(c)

p may or may not be equal to q, (b) h 4 nearest neighbors, and (c) the roduct pq is equal to n, the problem

3 Exact SAR Model Solution

The estimates for the parameters ρ and β in the SAR model (equation 1) can be found using either maximum likelihood theory or Bayesian statistics. In this paper we consider the maximum likelihood approach for estimating the parameters of the SAR model, whose mechanics are presented in Fig. 4.

Fig. 4. System diagram of the serial exact algorithm for the SAR model solution composed of three stages (A, B, and C )

Fig. 4 highlights the three stages of the exact algorithm for the SAR model solu-tion. It is based on maximum-likelihood (ML) theory, which requires computing the logarithm of the determinant (i.e., log-determinant) of the large )( WI ρ− matrix. The first term of the end-result of the derivation of the logarithm of the likelihood func-tion (equation 2) clearly shows why we need to compute the (natural) logarithm of the determinant of a large matrix. In equation 2 “I” denotes an n-by-n identity matrix, “T” denotes the transpose operator, “ln” denotes the logarithm operator, and is the common variance of the error.

2σ

222

)2ln(

2

)2ln(ln)ln(

σ

σπρ

SSEnnL −−−−= WI .

(2)

where { }))](1)([]1)([)(( yWIxxxxIxxxxIWIy ρρ −−−−−−= TTTTTTTSSE . Therefore, Fig. 4 can be viewed as an implementation of the ML theory. We now

describe each stage. Stage A is composed of three sub-stages: pre-processing, House-holder transformation [33], and QL transformation [9]. The pre-processing sub-stage not only forms the row-standardized neighborhood matrix W, but also converts it to its symmetric eigenvalue equivalent matrix W~ . The Householder transformation and QL transformation sub-stages are used to find all of the eigenvalues of the neighbor-hood matrix. The Householder transformation sub-stage takes W~ as input and forms the tri-diagonal matrix whose eigenvalues are computed by the QL transformation sub-stage. Computing all of the eigenvalues of the neighborhood matrix takes ap-proximately 99% of the total serial response time, as shown in Table 3.

Stage B computes the best estimates for the spatial auto-regression parameter ρ and the vector of regression coefficients β for the SAR model. While these estimates are being found, the logarithm of the determinant of )( WI ρ− needs to be computed at

Stage A Compute

Eigenvalues

n,,,~,,

εWWy

bestfitx

Eigenvalues of W

ρ 2ˆ,ˆ,ˆ σρ βρ of Range Stage C Compute

SSE

Stage B Golden Section

Search Calculate ML function

each step of the non-linear one-dimensional parameter optimization. This step uses the golden section search [9] and updates the auto-regression parameter at each step. There are three ways to compute the value of the logarithm of the likelihood function: (1) compute the eigenvalues of the large dense matrix W once; (2) compute the de-terminant of the large dense matrix )( WI ρ− at each step of the non-linear optimiza-tion; (3) approximate the log-determinant term. For small problem sizes, the first two methods work well; however, for large problem sizes approximate solutions are needed.

→ |lnlogarithm the

is

Equation 3 expresses the relationship between the eigenvalues of the W matrix and the logarithm of the determinant (i.e., log-determinant) of the ( )WI ρ− matrix. The optimization is of O(n) complexity.

∑=

−=−∏=

−=−n

iiρλρ

n

iiρλρ

1)ln(1|

1)(1||

taking

WIWI . (3)

D is defined as:

jid

cd

ij

n

1iijii

≠

=∑=

if 0=

and

C

CDW

ofversion )stochastic -row(or

normalized-row*= 1−

seigenvalue of in terms

of equivalent symmetric

**

**=~

2/12/1

2/12/1

W

DCD

DWDW−−

−

=

The eigenvalue algorithm applied in this study cannot find the eigenvalues of any dense matrix. The matrix W has to be converted to its symmetric version W~ , whose eigenvalues are the same as the original matrix W. The conversion is derived as shown in Fig. 5.

The Binary Neighborhood

matrix C

Fig. 5. Derivation of the W~ matrix, the symmetric eigenvalue equivalent of the W matrix

The matrix ) (i.e. ~ // 2121 CDDW −−

is/

is symmetric and has the same eigenvalues as W.

The row standardization can be expressed as , where D is a diagonal matrix with elements 1 , where is the row-sum of row i of C. The symmetriza-tion subroutine is the part of the code that does this job.

CDW 1−=

Finally, stage C computes the sum of the squared error, i.e., the SSE term, which is O(n2) complex. Table 3 shows our measurements of the serial response times of the stages of the exact SAR model solution based on ML theory. Each response time given in this study is the average of five runs. As can be seen, computing the eigen-values (stage A) takes a large fraction of the total time.

Now, we outline the derivation of the maximum likelihood function. This deriva-tion not only shows the link between the need for eigenvalue computation and the spatial auto-regression model parameter fitting but also explains how the spatial auto-regression model works and can be interpreted as an execution trace of the algorithm.

We begin the derivation by choosing a SAR model that is described by equation 1. Ordinary least squares are not appropriate to solve for the models described by

equation 1. One way to solve is to use the maximum likelihood procedure. In probability, there are essentially two classes of problems: the first is to generate a data sample given a probability distribution and the second is to estimate the parameters of a probability distribution given data. Obviously in our case, we are dealing with the latter problem. Table 3. Measured serial response times of stages of the exact SAR model solution for problem sizes of 2500, 6400 and 10K. Problem size denotes the number of observation points

Stage A Stage B Stage C

Computing Eigenvalues ML Function Least Squares

SGI Origin 78.10 0.41 0.06IBM SP 69.20 1.30 0.07IBM Regatta 46.90 0.58 0.06



Serial Execution Time (sec) Spent on

2500

6400

10000

Problem size (n ) Machine

It is assumed that is generated from a normal distribution, which has to be formally defined to go further in the derivation. The normal density function is given in equation 4.

ε

∑−∑≡ −−

εεε Tn

N21exp)2()( 2

12π ,

(4)

where and “I 2σ=∑ T” means transpose of a vector or matrix. It is worth noting again that we are assuming in our derivation that the error vector ε is governed by the standard normal distribution with zero mean and variance ∑ . The prediction of the spatial auto-regression model solution heavily depends on the quality of the normally distributed random numbers generated. The term ydεd needs to be calculated out in order to find the probability density function of the variable y, which is given by equation 5. The notation x denotes the determinant of matrix x. Hence, the probability density function of the observed dependent variable (y vector) is given by equation 6. When ε is replaced by ))(( xβyWI −− ρ in equation 6, the explicit form of the probability density function of y given by equation 7 is obtained. It should be noted that n22 σσ == I∑ , where n is the rank (i.e., row-size and column-

size) of identity matrix, I.

|||| WIyε ρ−=dd . (5)

||))(()( yεxβyWI ddNyL −−= ρ . (6)

{ } ])[(])[(2

1exp)2()( 222

−−−−−−=

−xβyWIxβyWIWI ρρ

σρπσ T

n

yL . (7)

L(y) will henceforth be referred to as the “likelihood function.” It is a probability distribution but now interpreted as a distribution of parameters which have to be calculated. Since the log-likelihood function is monotonic, we can then equivalently minimize the log-likelihood function, which has a simpler form and can handle large numbers. This is because the logarithm is advantageous, since

. After taking the natural logarithm of equation 7, we get equation 8, i.e., the log-likelihood function with the estimators for the variables and , which are represented by β and respectively in equations 9a and b.

)log()log()log()log( CBAABC ++=

β 2σ ˆ 2σ

{ } ])[(])[(2

12

)ln(2

)2ln(ln)ln( 2

2xβyWIxβyWIWI −−−−−−−−= ρρ

σσπρ TnnL .

(8)

β . )()(ˆ 1 yWIxxx ρ−= − TT

nTTTT /)]()([)(ˆ 12 yWIxxxxIWIy ρρσ −−−= − .

(9a)

(9b)

The term [( ]) xβyWI −− ρ is equivalent to after re-

placing given by equation 9a with β in equation 8. That leads us to equation 10 for the log-likelihood function (i.e., the logarithm of the maximum likelihood function) to be optimized for

yWIxxxxI )]()([ 1 ρ−− − TT

β

ρ .

{ }))]()([])([)((2

1 2

)ln(2

)2ln(ln)ln(

112

2

yWIxxxxIxxxxIWIy

WI

ρρσ

σπρ

−−−−

−−−−=

−− TTTTTTT

nnL

(10)

The first term of equation 10 (i.e., the log-determinant) is nothing but the logarithm of the sum of a collection of scalar values including all of the eigenvalues of the neighborhood matrix W. The first term transforms from a multiplication to a sum as shown by equation 3. That is why all of the eigenvalues of W matrix are needed.

{ } ))]()([])([)((2

1

1)1ln(MIN 11

21||yWIxxxxIxxxxIWIy ρρ

σρλ

ρ−−−−−∑

=− −−

<

TTTTTTTn

ii

. (11)

Therefore, the function is optimized using the golden section search to find the best estimate for ,ρ as shown in equation 11, which must be solved using nonlinear optimization techniques. Equation 9a could be estimated and then equation 8 could be optimized iteratively. Rather than computing that way, there is a faster and easier way such that equation 9a is substituted directly into equation 8 to get equation 10, which is a single expression in one unknown to be optimized. Once the estimate for ρ is

found, both β and can be computed. Finally, the simulated variable (y vectors or thematic classes) can be computed using equation 12.

ˆ 2σ

. )()( 1 εxβWIy +−= −ρ (12)

Equation 12 needs a matrix inversion algorithm in order to get the predicted ob-served dependent variable (y vector). For small problem sizes, one can use exact matrix inversion algorithms; however, for large problem sizes (e.g., > 10K) one can use geometric series expansion to compute the inverse matrix in equation 13. (For more details see lemma 2.3.3 [11].) In the next section we show how the complexity of these calculations can be reduced using approximate solutions.

4 Two approximate SAR model solutions

Since an exact SAR model solution is both memory and compute intensive, we need approximate solutions that do not sacrifice accuracy and can handle very large data-sets. We propose to use two different approximations for solving the SAR model solution, Taylor’s series expansion and Chebyshev polynomials. The purpose of these approximations is to calculate the logarithm of the determinant of )( WI ρ− .

4.1 Approximation by Taylor’s Series Expansion

Martin [23] suggests an approximation of the log-determinant by means of the traces of the powers of the neighborhood matrix, W. He basically finds the trace of the matrix logarithm, which is equal to the log-determinant. In this approach, the Taylor

series is used to approximate the function where ∑=−

n

i i1)1ln( ρλ iλ represents the ith

eigenvalue that lies in the interval [-1,+1] and ρ is the scalar parameter from the

interval (-1,+1). The term can be expanded as ∑=−−

n

i 11ln( i )ρλ ∑ =

ni

i1

k

k)(ρλ

provided that | 1|<iρλ , which will hold for all i if | . Equation 13, which states the approximation used for the logarithm of the determinant of the large matrix term

1| −< λρ

of maximum likelihood, is obtained using the relationship between eigenvalues and the trace of a matrix, i.e., . ∑=

=ni i tr

1kk )(Wλ

∑∞

=

−=−

1k

1({1ρW tr

n| n

)

kk }k/)|ln ρWI . (13)

The approximation comes into the picture when we sum up to a finite value, r, in-stead of infinity. Therefore, equation 13 is relatively much faster because it eliminates the need to calculate the compute-intensive eigenvalue estimation when computing the log-determinant. The overall solution is shown in Fig. 6.

One Dense Matrix (n-by-n) and Vector

(n-by-1) Multiplica-

tion

2 Dense Matrix (n-by-k) and

Vector (n-by-1)

Multiplica-tions

3 Vector (n-by-1)

Dot Products

Scalar

Operation

2ˆ,ˆ,ˆ σρ βW~ ρ

||ln WI ρ−

bestfitρρ

Similar to Stages A & B in Fig. 4

ML Function

Value

Taylor’s Series Expansion applied to

, W, , x, y

Golden Section search

Calculate ML

Function

SSE stage (Stage C in Fig. 4) Fig. 6. The system diagram for the Taylor’s Series expansion approximation for the SAR model solution. The inner structure of Taylor series expansion is similar to that of Chebyshev Polynomial except that there is one more vector sum operation, which is very cheap to compute

4.2 Approximation by Chebyshev Polynomials

This approach uses the symmetric equivalent of the neighborhood matrix W (i.e., W~ ) as discussed in Sect. 3. The eigenvalues of the symmetric W~ are the same as those of the neighborhood matrix W. The following lemma leads to a very efficient and accu-rate approximation to the first term on the right-hand side of the logarithm of the likelihood function shown in equation 2.

Lemma1. The Chebyshev solution tries to approximate the logarithm of the determinant of involving a symmetric neighborhood matrix as in equation 14. The first three terms are sufficient for approximating the log determinant term with an accuracy of 0.03%.

( WI ρ− W~

∑+

=− −≅−≡−

1q

111 )(

21))~(()(|~|ln||ln

jjj cTtrc ρρρρ WWIWI .

(14)

Proof. It is available in [33]. The value of “q” is 2, which is the highest degree of the Chebyshev polynomials.

Therefore, only )~( and )~( ),~( 210 WWW TTT have to be computed where:

)~()~(~2)~( ...; ;~2)~( ;~)~( ;)~( 1kk1k2

210 WWWWIWWWWIW −+ −=−=== TTTTTT The Chebyshev polynomial coefficients are given in equation 15. )(ρjc

)1q

)21k)(1(cos()]1q

)21k(cos(1ln[)1q

2()(1q

1k +−−

+−

−+

= ∑+

=

jc jππρρ .

(15)

In Fig. 7, the maximum likelihood function is computed by computing the maxi-mum of the sum of the logarithm of the likelihood function values and the SSE term. The spatial auto-regression parameter ρ that achieves this maximum value is the desired value that makes the classification most accurate. The parameter “q” is the highest degree of the Chebyshev polynomial which is used to approximate the term ln( )WI ρ− . The system diagram of the Chebyshev polynomial approximation is presented in Fig. 7. The following lemma reduces the computational complexity of the Chebyshev polynomial from O(n3) to approximately O(n2).

Lemma2. For regular grid-based nearest-neighbor symmetric neighborhood matrices, the relationship shown in equation 16 holds. This relationship saves a tremendous amount of execution time.

ijthn

i

n

j ij wjiwtrace ~ is ~ ofelement ),( where~)~(1 1

22 WW ∑ ∑= == . (16)

Proof. The equality property given in equation 16 follows from the symmetry prop-erty of the symmetrized neighborhood matrix. In other words, it is valid for all sym-metric matrices. The trace operator sums the diagonal elements of the square of the symmetric matrix W~ . This is the equivalent of saying that the trace operator first multiplies and adds the ith column with the ith row of the symmetric matrix, where the ith column and the ith row of the matrix are the same entries in a symmetric matrix. This results in squaring and summing the elements of the symmetric neighborhood matrix W~ . Equation 16 shows this shortcut for computing the trace of the square of the symmetric neighborhood matrix.

In Fig. 8, the powers of the W matrices, whose traces are to be computed, go up to 2. The parameter “q” is the highest degree of the Chebyshev polynomial which is used to approximate the term ln( )WI ρ− . The ML function is computed by calculat-ing the maximum of the likelihood functions (i.e. the logarithm determinant term plus the SSE term). The pseudo-code of the Chebyshev polynomial approximation ap-proach is presented in Fig. 8.

SSE stage (Stage C in Fig. 4)

2ˆ,ˆ,ˆ σρ β

|| WI ρ−

W~

W~ ρ

bestfitρ

ρ

)(ρjc

q-1 dense n-by-n

matrix-matrix multiplications

Chebyshev coefficients

Trace of n-by-n dense matrix

ML Function

Value

W, , ,x,y

Golden Section search

Calculate ML

Function

q

Chebyshev Polynomial Approximation

Similar to Stages A & B in Fig. 4 Chebyshev Polynomial applied to ln

Scalar

Operation

3 Vector (n-by-1)

Dot Products

2 Dense Matrix (n-by-k) and Vector

(n-by-1) Multiplica-

tions

One Dense Matrix (n-by-n) and Vector

(n-by-1) Multiplication

Fig. 7. System diagram of the approximate SAR model solution, where ln( )WI ρ− is ex-pressed as a Chebyshev polynomial. The term “q” is the degree of the Chebyshev Polynomial

[ ]

( )

[ ]

133)_(

33)_(

2RACE

OGDETTOONPPROXIMATIHEBYSHEV

* 11 * 10

*5.0 9 ))/)5.0(*).1(cos(

).1(log(sumnposs2 ],[ 8 1 7

)0.5)/-(*cos( 6 321 5

3 4 2] 0 1- 0; 1 0 ; 0 0 [1 3

)~(T 2 0 1

L--A-C||ln toestimate The :

)(q,ˆ,,~ :

××

××←

←−←

−−∗∗−←

←←

←←

←←

←

=

tdveccombotermet_approxcheby_logd_coeffscheby_polycposscomboterm

ntd2td1n tdvecnpossseq1npossj

xρjicpossnpossj

npossseq1npossx seq1npossnposs

_coeffscheby_poly td2 td1

ρOutputnpossnInput

veclength

npossveclength

T

k

k

T

ρ

ρ

π

π

ρ

dotofor

W

W-IW

Fig. 8. The pseudo code of the Chebyshev polynomial approximated )ln( WI ρ−

5 Experimental Design

We evaluate our solution models using satellite remote sensing data. We first present the system setup and then introduce our real dataset along with the comparison met-rics.

System setup. The control parameters for our experiments are summarized in Table 4. One of our objectives is to make spatial analysis tools available to the GIS user community. Notable solutions for the SAR model have been implemented in Matlab [18]. These approaches have two limitations. First, the user cannot operate without these packages and secondly these methods are not scalable to the application size. Our approach is to implement a general purpose package that works independently and scales well to the application size. All solutions described in this paper have been implemented using a general purpose programming language, f77, and use open

source matrix algebra packages (ScaLAPACK [7]). All the experiments were carried out using the same common experimental setup summarized in Table 4.

Table 4. The experimental design

Factor NameLanguage

Problem Size (n)Neighborhood Structure

Auto-regression Parameter

Data set Remote Sensing Imagery Data

Method

Hardware Platform IBM Regatta w/1.3 GHz Power4 architecture processor

Parameter Domain f77 2500,10K and 2.1M observation points2-D w/ 4-neighbors

[0,1)

Maximum Likelihood for Exact & Approximate SAR Model

Dataset. We used real data-sets from satellite remote-sensing image data in order to evaluate the approximations to SAR. The study site encompasses Carlton County, Minnesota, which is approximately 20 miles southwest of Duluth, Minnesota. The region is predominantly forested, composed mostly of upland hardwoods and lowland conifers. There is a scattering of agriculture throughout. The topography is relatively flat, with the exception of the eastern portion of the county containing the St. Louis River. Wetlands, both forested and non-forested, are common throughout the area. The largest city in the area is Cloquet, a town of about 10,000. For this study we used a spring Landsat 7 scene, taken May 31, 2000. This scene was clipped to the Carlton county boundaries, which resulted in an image of size 1343 lines by 2043 pixels and 6-bands. Out of this we took a subset image of 1200 by 1800 to eliminate boundary zero-valued pixels. This translates to a W matrix of size 2.1 million x 2.1 million (2.1M x 2.1M) points. The observed variable x is a matrix of size 2.1M by 6. We chose nine thematic classes for the classification. Comparison Metrics. We measured the performance of our implementation for accuracy, scalability (computational time), and memory usage. We first calculated the percentage error of the spatial auto-regression parameter ρ and the vector of regres-sion coefficients estimates from the approximate and exact SAR model solutions. Next, we calculated another accuracy metric using the standard root-mean-square (RMS) error. We computed the RMS error of the estimates of the observed dependent variable (y vectors or ) i.e. the thematic classes from the approximate and exact SAR model solutions. Scalability is reported in terms of computation (wall) time on an IBM Regetta 1.3GHz Power4 processor. Memory usage is determined by the total memory required by the program (which includes data and instruction space).

β

y

6 Results and Discussion

Since the main focus of this study is to find a scalable approximate method for the SAR model solution for very large problem sizes, the first evaluation is to compare

the estimates from the approximate methods for the spatial auto-regression parameter ρ and the vector of regression coefficients with the estimates obtained from the exact SAR model. Using the percentage error formula, Table 5 presents the compari-son of accuracies of

β

ρ and obtained from the exact and the approximate (Cheby-shev Polynomial and Taylor Series expansion based) SAR model solutions for the 2500 problem size. The estimates from the approximate methods are very close to the estimates obtained from the exact SAR model solution; there is an error of only 0.57% for the

β

ρ estimate obtained from the Chebyshev polynomial approximation case and an error of 7.27% for the ρ estimate from the Taylor series expansion ap-proximation. A similar situation exists for the estimates. The maximum error among the β estimates is 0.7% for the Chebyshev polynomial approximation case and 8.2% for the Taylor series expansion approximation. The magnitudes of the er-rors for the

β

ρ and estimates are on the same order across methods. β

Lemma 3. Taylor series approximation performs worse than Chebyshev polynomial approximation because Chebyshev polynomial approximation has a potential error canceling feature of the logarithm of the determinant (log-determinant) of a matrix. Taylor series expansion produces different error magnitudes for positive versus negative eigenvalue whereas the Chebyshev polynomials tend to produce error of more equal maximum magnitude [30].

iλ

Proof: The main reason behind this phenomenon is that Taylor series approximation does better than the Chebyshev polynomial approximation for values of ρ nearer to zero, bur far worse for extreme ρ (see Sect. 2.3 of [30]). Since the value of ρ is far greater than zero in our case, our experiments also verify this phenomenon, as shown in Table 5. Table 5. The comparison of accuracies of ,ρ the spatial auto-regression parameter, and β , the vector of regression coefficients, obtained from the exact and the approximate (Chebyshev Polynomial and Taylor Series expansion) SAR model solutions for the 2,500 problem size

β Problem

Size

ρ

1 2 3 4 5 6

Exact 0.4729 -2.473 -0.516 3.167 0.0368 -0.4541 3.428

Chebyshev 0.4702 -2.478 -0.520 3.176 0.0368 -0.456 3.440

50x50

(2500)

Taylor 0.4385 -2.527 -0.562 3.291 0.0374 -0.476 3.589

The second evaluation is to compute the RMS (root-mean-square) error of the es-

timates of the observed dependent variable (y vectors or ) i.e., the thematic classes. The RMS error is given in equation 17 to show how we use it in our formulation.

y

Table 6 presents the RMS values for all thematic classes. A representative RMS error value for the Taylor method is 2.0726 and for the Chebyshev method, 0.1686.

( )

( )∑

∑

−−

=

−

−=

2ˆˆ

2ˆˆ

2

2

nRMSerror

nRMSerror

eetsts

eecpcp

yy

yy

.

(17)

The values of the RMS error suggest that estimates for the observed dependent variable (y vector or thematic classes) from the Chebyshev polynomial approximated SAR model solution are better than those of the Taylor series expansion approxi-mated SAR model solution. This result agrees with the estimates for the spatial auto-regression parameter ρ and the vector of regression coefficientsβ shown in Table 5.

Table 6. RMS values for each thematic class of a dataset of problem size 2500

Training Thematic

Class

RMS error value for

Chebyshev

RMS error value for Taylor

Testing Thematic

Class

RMS error value for

Chebyshev

RMS error value for Taylor

y1 0.1686 2.0726 y1 0.1542 1.9077 y2 0.2945 2.0803 y2 0.2762 2.0282 y3 0.5138 3.3870 y3 0.5972 4.0806 y4 1.0476 6.9898 y4 1.4837 9.6921 y5 0.3934 2.4642 y5 0.6322 3.9616 y6 0.3677 2.3251 y6 0.4308 2.8299 y7 0.2282 1.5291 y7 0.2515 1.7863 y8 0.6311 4.3484 y8 0.5927 4.0524 y9 0.3866 3.8509 y9 0.4527 4.4866

Table 7. The execution time in seconds and the memory usage in mega-bytes (MB)

Time (Seconds) Memory (MB) Problem Size (n)

Exact Taylor Chebyshev Exact Taylor Chebyshev

50x50 (2500) 38 0.014 0.013 50 1.0 1.0

100x100 (10K) 5100 0.117 0.116 2400 4.5 4.5 1200x1800 (2.1M) Intractable 17.432 17.431 ~32*106 415 415

The predicted images (50 rows by 50 columns) using exact and approximate solu-

tions are shown in Fig. 9. Although the differences in the images predicted by the exact and approximate solutions is hard to notice, there is a huge difference between these methods in terms of computation and memory usage. As can be seen in Table 7, even for large problem sizes, the run-times are pretty small due to the fast log-determinant calculation offered by Chebyshev and Taylor’s series approximation. By contrast, with the exact approach, it is impossible to solve any problem having more than 10K observation points. Even if we used sparse matrix determinant computation, it is clear that approximate solutions will still be faster.

The approximate solutions also manage to provide close estimates and fast execu-tion times using very little memory. Such fast execution times make it possible to sale solutions for large problems consisting of billions of observation points. The memory usage is very low due to the sparse storage techniques applied to the neighborhood matrix W. Sparse techniques cause speedup since the computational complexity of linear algebra operations decrease because of the small number of non-zero elements within the W matrix. As seen from Figures 6 and 7, the most complex operation for Taylor series expansion and Chebyshev Polynomial approximated SAR model solu-tions is the trace of powers of the symmetric neighborhood matrix ,~W which re-quires matrix-matrix multiplications. These operations are reduced to around O(n2) complexity by Lemma 2 given in Sect. 4.2. All linear algebra matrix operations are efficiently implemented using the ScaLAPACK [7] libraries.

We fitted the SAR model for each observed dependent variable (y vector). For each pixel a thematic class label was assigned by taking the maximum of the pre-dicted values. Fig. 9 shows a set of labeled images for a problem size of 2500 pixels (50 rows x 50 columns). For a learning (i.e., training) dataset of problem size 2500, the prediction accuracies of the three methods were similar (59.4% for the exact SAR model, 59.6% for the Chebyshev polynomial approximated SAR model, and 60.0% for the Taylor series expansion approximated SAR model.) We also observed a simi-lar trend on another (testing) dataset of problem size 2500. The prediction accuracies were 48.32%, 48.4% and 50.4% for the exact solution, Chebyshev polynomial and Taylor series expansion approximation based SAR models respectively. This is an interesting result. Even though the estimates for the observed dependent variables (y vectors) or thematic classes are more accurate for the Chebyshev polynomial based approximate SAR model than for the Taylor series expansion approximated SAR model solution, the classification accuracy for the Taylor series expansion approxi-mated SAR model solution becomes better than the ones for not only the Chebyshev polynomial based approximate SAR model but also even the exact SAR model solu-tion. We think that the opposite trend will be observed for larger size images because SAR might need more samples to be trained better. Even though we do not suggest a new exact SAR model solution, further research and experimentation is needed to fully understand SAR model’s training needs and its impact on prediction accuracy with the solution methods discussed in this paper.

7 Conclusions and Future Work

Linear regression is one of the best-known classical data mining techniques. How-ever, it makes the assumption of independent identical distribution (i.i.d.) in learning data samples, which does not work well for geo-spatial data, which is often character-ized by spatial autocorrelation. In the spatial auto-regression (SAR) model, spatial dependencies within data are taken care of by the autocorrelation term, and the linear regression model thus becomes a spatial auto-regression model.

Fig. 9. The images (50x50) using exact and approxi-mate solutions

Incorporating the autocorrelation term enables better prediction accuracy. How-

ever, computational complexity increases due to the need for computing the logarithm of the determinant of a large matrix )( WI ρ− , which is computed by finding all of the

eigenvalues of the W~ matrix. This paper applies one exact and two approximate meth-ods to the SAR model solution using various sizes of remote sensing imagery data i.e., 2500, 10K and 2.1M observations. The approximate methods applied are Chebyshev Polynomial and Taylor series expansion. It is observed that the approxi-mate methods not only consume very little memory but they also execute very fast while providing very accurate results. Although the software is written using a paral-lel version of ScaLAPACK [7], SAR model solutions presented in this paper can be run either sequentially on a single processor of a node or in parallel on single or mul-tiple nodes. All the results presented in Sect. 6 (Table 7) are based on sequential runs on the same (single) node of an IBM Regetta machine. It should be noted that the software can be easily ported onto general purpose computers and workstations by replacing open source ScaLAPACK routines with the serial equivalent routines in the open source LAPACK [1,13] library. Currently, LAPACK libraries can be compiled on Windows 98/NT, VAX, and several variants of UNIX. In our future release of SAR software, we plan to provide both ScaLAPACK and LAPACK versions.

In this study we focused on the scalability of the SAR model for large geospatial data analysis using approximate solutions and compared the quality of exact and approximate solutions. Though in this study we focused only on quality of parameter estimates, we do recognize that training and prediction errors were also important for these methods to be widely applied in various geospatial application domains. To-wards this goal we are conducting several experiments on several geospatial data sets from diverse geographic settings. Our future studies will also focus on comparing SAR model predictions against competing models like Markov Random Fields. We are also developing algebraic cost models to further characterize performance and scalability issues.

8 Acknowledgments

This work was partially supported by the Army High Performance Computing Re-search Center (AHPCRC) under the auspices of the Department of the Army, Army Research Laboratory (ARL) under contract number DAAD19-01-2-0014. The con-tent of this work does not necessarily reflect the position or policy of the government and no official endorsement should be inferred. The authors would like to thank the University of Minnesota Digital Technology Center and Minnesota Supercomputing Institute for the use of their computing resources. The authors would also like to thank the members of the Spatial Database Group, ARCTiC Labs Group for valuable discussions. The authors thank Kim Koffolt for helping improve the readability of this paper and anonymous reviewers for their useful comments.

References

1. Anderson, E., Bai, Z., Bischof, C., Blackford, S., Demmel, J., Dongarra, J., Du Croz, J., Greenbaum, A., Hammarling, S., McKenney, A., Sorensen, D.: LAPACK User’s Guide, 3rd Edition, Society for Industrial and Applied Mathematics, Philadelphia, PA (1999)

2. Anselin, L.: Spatial Econometrics: Methods and Models, Kluwer Academic Publishers, Dorddrecht (1988)

3. Barry, R., Pace, R.: Monte Carlo Estimates of the log-determinant of large sparse matrices. Linear Algebra and its Applications, Vol. 289 (1999) 41-54

4. Bavaud, F.: Models for Spatial Weights: A Systematic Look, Geographical Analysis, Vol. 30 (1998) 153-171

5. Besag, J. E.: Spatial Interaction and the Statistical Analysis of Lattice Systems, Journal of the Royal Statistical Society, B, Vol. 36 (1974) 192-225

6. Besag, J. E.: Statistical Analysis of Nonlattice Data, The Statistician, Vol. 24 (1975) 179-195

7. Blackford, L. S., Choi, J., Cleary, A., D'Azevedo, E., Demmel, J., Dhillon, I., Dongarra, J., Hammarling, S., Henry, G., Petitet, A., Stanley, K., Walker, D., Whaley, R. C.: ScaLA-PACK User’s Guide, Society for Industrial and Applied Mathematics, Philadelphia, PA (1997)

8. Chawla, S., Shekhar, S., Wu, W., Ozesmi, U.: Modeling Spatial Dependencies for Mining Geospatial Data, Proc. of the 1st SIAM International Conference on Data Mining, Chicago, IL (2001)

9. Cheney, W., Kincaid, D.: Numerical Mathematics and Computing, 3rd ed. (1999) 10. Cressie, N. A.: Statistics for Spatial Data (Revised Edition). Wiley, New York (1993) 11. Golub, G. H., Van Loan, C. F.: Matrix Computations. Johns Hopkins University Press, 3rd

edn. (1996) 12. Griffith, D. A.: Advanced Spatial Statistics, Kluwer Academic Publishers (1988) 13. Information about Freely Available Eigenvalue-Solver Software:

http://www.netlib.org/utk/people/JackDongarra/la-sw.html 14. Kazar, B., Shekhar, S., Lilja, D.: Parallel Formulation of Spatial Auto-Regression,

AHPCRC Technical Report No: 2003-125 (August 2003) 15. Kazar, B. M., Shekhar, S., Lilja, D. J., Boley, D.: A Parallel Formulation of the Spatial

Auto-Regression Model for Mining Large Geo-Spatial Datasets, Proc. of 2004 SIAM Inter-national Conf. on Data Mining Workshop on High Performance and Distributed Mining (HPDM2004), Orlando, Fl. USA (2004)

16. Li, B.: Implementing Spatial Statistics on Parallel Computers, In: Arlinghaus S. (Ed.), ed. Practical Handbook of Spatial Statistics, CRC Press, Boca Raton, FL (1996) 107-148

17. LeSage, J.: Solving Large-Scale Spatial autoregressive models, presented at the Second Workshop on Mining Scientific Datasets, AHPCRC, University of Minnesota (July 2000)

18. LeSage, J. P.: Econometrics Toolbox for MATLAB. http://www.spatial-econometrics.com/ 19. LeSage, J., Pace, R. K.: Using Matrix Exponentials to Explore Spatial Structure in Regres-

sion Relationships (Bayesian MESS), Technical Report (October 2000) http://www.spatial-statistics.com

20. LeSage, J., Pace, R. K.: Spatial Dependence in Data Mining, in Data Mining for Scientific and Engineering Applications, R. L. Grossman, C. Kamath, P. Kegelmeyer, V. Kumar, and R. R. Namburu (eds.), Kluwer Academic Publishing (2001) 439-460

21. Long, D. S.: Spatial autoregression modeling of site-sepecific wheat yield. Geoderma, Vol. 85 (1998) 181-197

22. Marcus, M., Minc, H.: A Survey of Matrix Theory and Matrix Inequalities, New York: Dover (1992)

23. Martin, R. J.: Approximations to the determinant term in Gaussian maximum likelihood estimation of some spatial models, Communications in Statistical Theory Models, Vol. 22 Number 1 (1993) 189-205

24. Ord, J. K.: Estimation Methods for Models of Spatial Interaction, Journal of the American Statistical Association, Vol. 70 (1975) 120-126

25. Pace, R. K., Barry, R.: Quick Computation of Spatial Auto-regressive Estimators. Geo-graphical Analysis, Vol. 29, (1997) 232-246

26. Pace, R. K., LeSage, J.: Closed-form maximum likelihood estimates for spatial problems (MESS), Technical Report (September 2000) http://www.spatial-statistics.com

27. Pace, R. K., LeSage, J.: Semiparametric Maximum Likelihood Estimates of Spatial De-pendence, Geographical Analysis, Vol. 34, No.1 The Ohio State University Press (Jan 2002) 76-90

28. Pace, R. K., LeSage, J.: Simple bounds for difficult spatial likelihood problems, Technical Report (2003) http://www.spatial-statistics.com

29. Pace, R. K., LeSage, J.: Spatial Auto-regressive Local Estimation (SALE), Spatial Statistics and Spatial Econometrics, Edited by Art Getis, Palgrave (2003)

30. Pace, R. K., LeSage, J.: Chebyshev Approximation of Log-Determinant of Spatial Weight Matrices, Computational Statistics and Data Analysis, Technical Report, Forthcoming.

http://www.netlib.org/utk/people/JackDongarra/la-sw.html

http://www.spatial-econometrics.com/

31. Pace, R. K., LeSage, J.: Closed-form maximum likelihood estimates of spatial auto-regressive models: the double bounded likelihood estimator (DBLE), Geographical Analy-sis-Forthcoming

32. Pace, R. K., Zou, D.: Closed-Form Maximum Likelihood Estimates of Nearest Neighbor Spatial Dependence, Geographical Analysis, Vol. 32, Number 2, The Ohio State University Press (April 2000)

33. Press, W., Teukulsky, S. A., Vetterling, W. T., Flannery, B. P.: Numerical Recipes in For-tran 77, 2nd edn. Cambridge University Press (1992)

34. Shekhar, S., Chawla, S.: Spatial Databases: A Tour, Prentice Hall (2003) 35. Shekhar, S, Schrater, P., Raju, R., Wu, W.: Spatial Contextual Classification and Prediction

Models for Mining Geospatial Data, IEEE Transactions on Multimedia, Vol. 4, Number 2 (June 2002) 174-188

36. Smirnov, O., Anselin, L.: Fast Maximum Likelihood Estimation of Very Large Spatial Auto-regressive Models: A Characteristic Polynomial Approach, Computational Statistics & Data Analysis, Volume 35, Issue 3 (2001) 301-319

37. Timlin, J., Walthall, C. L., Pachepsky, Y., Dulaney, W. P., Daughtry, C. S. T.: Spatial Regression of Crop Parameters with Airborne Spectral Imagery. Proceedings of the 3rd Int. Conference on Geospatial Information in Agriculture and Forestry, Denver, CO; (November 2001)

Appendix: Constructing the Neighborhood Matrix W for Irregular Grid

Spatial statistics requires some means of specifying the spatial dependence among observations [12]. The neighborhood matrix i.e., spatial weight matrix fulfills this role for lattice models [5,6] and can be formed on both regular and irregular grid. This appendix shows a way to form the neighborhood matrix on the irregular grid which is based on Delaunay triangulation algorithm [28,29]. [30] describes another method of forming the neighborhood matrix on the irregular grid which is based on nearest neighbors.

One specification of the spatial weight matrix begins by forming the binary adja-cency matrix N where when observation j is a neighbor to observation i ( i ). The neighborhood can be defined using computationally very expensive Delaunay triangulation algorithm [18]. These elements may be further weighted to give closer neighbors higher weights and incorporate whatever spatial information the user desires. By itself, N is usually asymmetric. To insure symmetry, we can rely on the transformation

1=ijN

j≠

( ) 2/=

1/2

TNN +

D−

C . The rest of forming neighborhood matrix on irregular grid follows the same procedure discussed in Sect. 3 (see Fig. 5). Users often re-weight the adjacency matrix to create a row-normalized i.e., row-stochastic matrix or a matrix similar to a row-stochastic matrix. This can be accomplished in the following way. Let D represent a diagonal matrix whose ith diagonal entry is the row-sum of the ith row of matrix C. The matrix is row-stochastic (see Fig. 5) where is a diagonal matrix such that its i

CDCDDW 11/21/2 −−− ==th entry is the

inverse of the square root of the ith row of matrix C. Note that the eigenvalues of the matrix W do not exceed 1 in absolute value as noted in Sect. 4.1, and the maximum

eigenvalue equals 1 via the properties of row-stochastic matrices (see Sect. 5.13.3 in [22]). Despite the symmetry of C, the matrix W will be asymmetric in the irregular grid case as well. One can however invoke a similarity transformation as shown in equation 18 (see Sect. 3, Fig. 5 of this study).

( ) ( ) 1/21/21/21/2111/211/2 CDDWDDDWDW −−−−−−−− ==

=

~ . (18)

This results in W~ having eigenvalues i.e., λ equal to those of W [24]. That is why we call W~ the symmetric eigenvalue-equivalent matrix of W matrix. Note, the eigenval-ues of W do not exceed 1 in absolute value via the properties of row-stochastic matri-ces (5.13.3 of [22]) because W~ is similar to W due to the equivalent eigenvalues i.e.,

11~≤− W≤ iλ .

From a statistical perspective, one can view W as a spatial averaging operator. Given the vector y, the row-stochastic normalization i.e., Wy results in a form of local average or smoothing of y. In this context, one can view elements in the rows of W as the coefficients of a linear filter. From a numerical standpoint symmetry of W~ simplifies computing the logarithm of determinant and has theoretical advantages as well. (See [4,28,29,30] for more information on spatial weight matrices.)

Documents

Comparing exact and approximate spatial auto-regression model