regoStata module for decomposing goodness of fit according to Shapley and Owen values

Frank Huettner and Marco SunderDepartment of EconomicsUniversity of Leipzig, Germany

Presentation at the UK Stata Users Group meeting, London, 13/14 Sept. 2012

Motivation

• Challenges in interpreting regression results by the analyst (the quick and dirty way)- Sign econometrics- Confusion of statistical significance and economic significance- Interaction terms

• Challenges to readers- Units of measurement- “Omitted coefficients”

R2Decompose

• Imagine regressor variables as “players”

- Cooperative game theory: games with transferable utility

- Regressors may form coalition -- how to distribute the gains from cooperation?

- Shapley (1953) and Owen (1977) Values as means to decompose goodness of fit

- Axioms under which these values are unique solutions

• “hierarchical partitioning” (Lindeman et al. 1980, Chevan & Sutherland 1991)

• Previous implementations: Stata: shapley (Kolenikov 2000) R: relaimpo package (Grömping 2006)

Motivation

• Assume the “full model” is...

y = �0 + �1x1 + �2x2 + �3x3 + "

y = �0 + �1x1 + · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

1

log wage intelligence schooling experience

Illustrative calculationof the Shapley Value

0.24 0.12 0.18

0.72 0.00

0.54 0.48 0.30

Worths in 2k (sub-)models

∆ = 0.18

0.72 0.30 0.18 0.000.720.720.720.720.72

0.000.000.000.000.00

0.30 0.120.54 0.180.54 0.240.48 0.120.48 0.24

Marginal contributions (MC)remove first remove first remove first

0.12

0.12

0.12

0.24

0.18

0.18

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

1

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

1

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

1

Σ = 0.72

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permutations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

G = {G1

, ..., G`, ..., G�}

Gq 6=`

1

Axioms1) Efficiency

GOF of full model is decomposed among the regressors

2) MonotonicityIncrease in R2 must not decrease the value

3) Equal treatmentPerfect substitutes (in terms of GOF) receive the same value

The Shapley Value is the only value that satisfies these properties (Young 1985).

• Analyst may believe that some regressor variables belong together, e.g.: polynomial terms, region dummies

• Regressor variables are partitioned:

• When thinking about marginal contributions of variables from , variables of must be completely absent or present

• This limits the set of admissible model permutations☞ Owen Value

• E.g.: “nature”, “nurture”

A priori grouped regressors

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

G = {G1

, ..., G`, ..., G�}

1

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

G = {G1

, ..., G`, ..., G�}

Gq 6=`

1

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

G = {G1

, ..., G`, ..., G�}

Gq 6=`

1

0.72 0.30 0.18 0.000.720.720.720.720.72

0.000.000.000.000.00

0.30 0.120.54 0.180.54 0.240.48 0.120.48 0.24

4 permutations respect

y = �0

+ �1

x

1

+ �2

x

2

+ �3

x

3

+ "

y = �0

+ �1

x

1

+ · · ·+ �jxj + · · ·+ �kxk + "

MC(xj , ✓) := f (P (✓, xj) [ {xj})� f (P (✓, xj)) ,

Sh(xj) =1

k!

X

all permuations

MC

Sh( ) =1

6· (0.12 + 0.12 + 0.18 + 0.18 + 0.12 + 0.24) = 0.16

Sh( ) =1

6· (0.42 + 0.42 + 0.36 + 0.24 + 0.36 + 0.24) = 0.34

Sh( ) =1

6· (0.18 + 0.18 + 0.18 + 0.30 + 0.24 + 0.24) = 0.22

G = {G1

, ..., G`, ..., G�}

Gq 6=`

1

Ow( ) = 0.33 Ow( ) = 0.165 Ow( ) = 0.225

Axioms1) Efficiency*

2) Monotonicity*

3) Equal treatment of players

4) Equal treatment of groups

The Owen Value is the only value that satisfies these properties (Khmelnitskaya and Yanovskaya 2007).

Stata implementation

• With grouping, “large” models become possible

• User decides for which groups to calculate Owen values

• R2 calculated from covariance structure of the data

• Syntax uses “\” to indicate group boundaries in varlist

• Computation in Mata

• Bootstrapping option

Decomposing goodness of fit 1247

Table 1OLS regression results with decomposition of R2 (in %)

R2 decomposition (%)

Group Regressor Coef. Owen Group

1 SCT 0.789 ! 3.0 33.2SCT ! EDUC "0.048 ! 8.3EDUC 0.103 !!! 21.9

2 EXPER 0.025 !!! 7.0 11.0(EXPER)2/100 "0.041 !!! 4.0

3 TENURE 0.017 !!! 9.3 14.3(TENURE)2/100 "0.029 !! 5.0

4 MARRIED 0.084 !!! 5.0 5.0

5 Firm size (3 dummies) !!! 14.7

6 Industry (6 dummies) !!! 5.5

7 Region (14 dummies) !!! 16.2

Observations 850Full model R2 0.501

Remark: !/!!/!!! denotes statistical significance at the 10% / 5% / 1% levelfor individual variables (t-test) or groups of dummy variables (F-test), basedon the heteroscedasticity-robust covariance matrix.

R2), again with confidence intervals not overlapping (Figure 2). Looking at thecoe!cients, the model implies that up to 16 years of education, more cognitiveability is associated with higher earnings.

The polynomial terms of labor market experience and job tenure suggestpositive e"ects on earnings in the first years, with turning points after about30 years in both cases. Interestingly, our procedure assigns greater importancein terms of GOF to the tenure polynomial, although the coe!cients suggestthat the experience profile is the steeper one. However, both confidence inter-vals include the value of the respective other group, i.e., generalizations on thedi"erence in importance should not be drawn on the basis of our data (Figure 1).

In terms of ‘group importance’, firm size categories and the regional compo-sition reach a similar order of magnitude as the tenure polynomial. While ourfocus is not on these dummy variables, such information may nevertheless be ofinterest to the reader, e.g., against the backdrop of the long economic conver-gence process in East Germany after the fall of the Berlin wall. Group valuesmay thus provide the reader a space-conserving impression of the importanceof control variables that are usually omitted from regression tables.

Wage regression: German male employees

Wage regression: German male employees

1248 F. Huettner and M. Sunder

1 EDU / SCT

2 EXPER polynomial

3 TENURE polynomial

4 MARRIED

5 Firm size dummies

6 Industry dummies

7 Region dummies

0 .05 .1 .15 .2Absolute contribution to R!

EDU / SCT

EXPER polynomial

TENURE polynomial

MARRIED

Firm size dummies

Industry dummies

Region dummies

Fig 1. Decomposition results for groups, with 90% bootstrap confidence intervals, based on5000 bootstrap replications.

1-1 SCT01

1-2 SCT01_EDUC

1-3 EDUC

2-4 EXPER

2-5 EXPER2

3-6 TENURE

3-7 TENURE2

0 .02 .04 .06 .08 .1 .12 .14Absolute contribution to R!

SCT

SCT x EDUC

EDUC

EXPER

EXPER 2 / 100

TENURE

TENURE 2 / 100

Fig 2. Owen value decomposition results for ‘human capital’ variables, with 90% bootstrapconfidence intervals, based on 5000 bootstrap replications.

4. Concluding remarks

Decomposition of GOF provides an attractive diagnostic tool for identifyingimportant (groups of) explanatory variables in a given regression model. Wehave argued, on the grounds of its attractive properties, that the Shapley valueshould be used for this purpose.

The Shapley value and its axiomatic foundations can be generalized. TheOwen value constitutes such a generalization where an a priori grouping of theregressor variables is taken into account, which accommodates many empiricalanalyses in practice. A further generalization could allow for additional levels ofaggregation [23]. In our wage regression example, such a level structure designcould be implemented to assign the first three groups into a ‘human capital’cluster.

One can also imagine situations in which certain variables must always beincluded in all sub-models, e.g., time fixed e!ects in a panel data analysis, or

Wage regression: German male employees

1248 F. Huettner and M. Sunder

1 EDU / SCT

2 EXPER polynomial

3 TENURE polynomial

4 MARRIED

5 Firm size dummies

6 Industry dummies

7 Region dummies

0 .05 .1 .15 .2Absolute contribution to R!

EDU / SCT

EXPER polynomial

TENURE polynomial

MARRIED

Firm size dummies

Industry dummies

Region dummies

Fig 1. Decomposition results for groups, with 90% bootstrap confidence intervals, based on5000 bootstrap replications.

1-1 SCT01

1-2 SCT01_EDUC

1-3 EDUC

2-4 EXPER

2-5 EXPER2

3-6 TENURE

3-7 TENURE2

0 .02 .04 .06 .08 .1 .12 .14Absolute contribution to R!

SCT

SCT x EDUC

EDUC

EXPER

EXPER 2 / 100

TENURE

TENURE 2 / 100

Fig 2. Owen value decomposition results for ‘human capital’ variables, with 90% bootstrapconfidence intervals, based on 5000 bootstrap replications.

4. Concluding remarks

Decomposition of GOF provides an attractive diagnostic tool for identifyingimportant (groups of) explanatory variables in a given regression model. Wehave argued, on the grounds of its attractive properties, that the Shapley valueshould be used for this purpose.

The Shapley value and its axiomatic foundations can be generalized. TheOwen value constitutes such a generalization where an a priori grouping of theregressor variables is taken into account, which accommodates many empiricalanalyses in practice. A further generalization could allow for additional levels ofaggregation [23]. In our wage regression example, such a level structure designcould be implemented to assign the first three groups into a ‘human capital’cluster.

One can also imagine situations in which certain variables must always beincluded in all sub-models, e.g., time fixed e!ects in a panel data analysis, or

Limits• Only OLS with R2 decomposition at this time• Does not yet accept factor variables or weights• ...

Possible extensions?• Decomposition of other measures (e.g., AIC)• More levels of aggregation• Essential regressors / fixed effects model (econ)

Thank you!

www.uni-leipzig.de/~rego Huettner & Sunder (2012)Electronic Journal of Statistics