Accounting Research Center, Booth School of Business, University of Chicago

Discussion of the Experimental Design of Classification Models: An Application of RecursivePartitioning and Bootstrapping to Commercial Bank Loan ClassificationsAuthor(s): Chris OlsenSource: Journal of Accounting Research, Vol. 22, Studies on Current Econometric Issues inAccounting Research (1984), pp. 115-118Published by: Wiley on behalf of Accounting Research Center, Booth School of Business, Universityof ChicagoStable URL: http://www.jstor.org/stable/2490862 .

Accessed: 06/11/2014 01:11

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.

.

Wiley and Accounting Research Center, Booth School of Business, University of Chicago are collaboratingwith JSTOR to digitize, preserve and extend access to Journal of Accounting Research.

http://www.jstor.org

This content downloaded from 75.103.254.62 on Thu, 6 Nov 2014 01:11:28 AMAll use subject to JSTOR Terms and Conditions

Journal of Accounting Research Vol. 22 Supplement 1984

Printed in U.S.A.

Discussion of The Experimental Design of Classification Models:

An Application of Recursive Partitioning and Bootstrapping to Commercial Bank Loan

Classifications

CHRIS OLSEN*

Discussion of "The Experimental Design of Classification Models: An Application of Recursive Partitioning and Bootstrapping to Commercial Bank Loan Classifications" focused on two sets of issues. The first, labeled statistical issues, included questions and answers on the recursive partitioning and bootstrapping methods used in the paper. The second, labeled experimental design issues, consisted of suggestions for exten- sions of the research presented. Summaries of both issues appear below. Some personal observations follow the summaries.

Statistical Issues

Much of the dialogue during the presentation of the paper centered on statistical issues. One concern was the comparison of the recursive partitioning and polytomous probit approaches to model estimation. Misclassification loss rates were the (only) metric used in comparing these two estimation procedures. This approach raised two points. First, reduction of the expected loss rate was the stated objective of the classification exercise. Estimation of the relative contributions of indi- vidual variables to the classification was considered peripheral to this objective by the authors. However, both probit and recursive partitioning procedures could yield such estimates. Second, incorporation of an asym- metric loss function into the probit model entailed estimation of the

* University of Texas at Austin. The helpful comments of J. Richard Dietrich and James M. Patell are gratefully acknowledged.

115

Copyright (?, Institute of Professional Accounting 1985

This content downloaded from 75.103.254.62 on Thu, 6 Nov 2014 01:11:28 AMAll use subject to JSTOR Terms and Conditions

116 CURRENT ECONOMETRIC ISSUES IN ACCOUNTING: 1984

misclassification probabilities prior to computation of the losses. The recursive partitioning procedure estimated both simultaneously. This computational difference arose from the possibility that concurrent es- timation of misclassification probabilities and losses could bear little relation to a probit classification model.

A second group of statistical concerns centered on overfitting of the data. Discussion of the potential for statistical overfitting in the esti- mation of misclassification losses prompted one participant to question the concern, in the context of classification studies, noting that the issue rarely arose in other types of empirical research. The response empha- sized the tendency of classificatory research to rely on statistical tech- niques to specify the independent variables, rather than a priori theories of the phenomenon under investigation.

The remainder of the discussion on the estimation of misclassification losses considered four approaches designed to mitigate the effects of statistical overfitting. Selection of the bootstrap approach resulted from consideration of the problems associated with the other three, together with simulation results in the Efron studies that point to its superiority in small samples. In particular, concerns regarding intertemporal inde- pendence led to the rejection of a predictive ability test. Several partici- pants expressed concern that the bootstrapping procedure depended on distributional assumptions made by the researcher, and in that regard appeared to be a simulation technique. The resulting discussion empha- sized that the procedure provides a numerical approximation to an often incalculable analytical solution and is, hence, not a simulation procedure per se. The repetitive nature of the resampling process led one participant to inquire about the determination of the optimal number of bootstrap replications. The answer indicated that no analytical solution to the question exists, but that the procedure did provide an assessment of the standard error of the computed value, thus enabling the researcher to stop the process when the decrease in the standard error through an additional replication fell below some prespecified value.

Experimental Design Issues

The discussion considered four experimental design issues, in addition to the statistical issues discussed above. One topic that generated consid- erable comment from the participants was the focus of the study. As stated in the paper, the motivation for the research design derived from an "expert judgment" view of the problem, that is, an attempted repli- cation of the outcomes of a process, rather than the process itself. One conferee suggested that a comparison of the results obtained to those of a process-oriented protocol analysis would constitute an interesting extension of the study. Another proposed use of the data was to predict the number of loans that actually become uncollectible, rather than loan classifications. A major difficulty cited in conjunction with the latter

This content downloaded from 75.103.254.62 on Thu, 6 Nov 2014 01:11:28 AMAll use subject to JSTOR Terms and Conditions

EXPERIMENTAL DESIGN OF CLASSIFICATION MODELS 117

approach, however, was the endogeneity problem that results from dif- ferences in the subsequent reactions and collection efforts of loan officers to differentially classified loans.

A second research design issue centered on the decision to examine loan classifications at a single point in time. Loans outstanding for multiple periods would have generated multiple classifications (one for each period). However, these classifications need not be independent across time for a given loan, and any such dependence would alter the conditional probability of a particular loan's classification in the future. An alternative approach suggested by one participant would require examination of the changes in loans' classifications, as a function of changes in the various independent variables. One concern expressed with a transitions-based methodology, vis-A-vis its single-classification counterpart, was its inability to allow for individual firms' abilities to change banks. This fact, coupled with the addition of new firms and new loans each period, could drastically reduce the sample size and, corre- spondingly, the power of the test (particularly among the relatively small number of adversely classified loans).

Incorporation of an asymmetric loss function raised the question of the sensitivity of the results to the particular loss function given in table 3. The question assumed added significance in view of the loan officers' admitted difficulties in precisely quantifying the estimates. Consultation between the authors and the loan officers indicated that the cost of misclassifying a loan into a lower-risk category than its actual classifi- cation was three times the cost of the corresponding opposite misclassi- fication. The proposed sensitivity analysis entailed alteration of this ratio, rather than the individual numbers.

The fourth experimental design issue discussed was the absence of an independent variable that measured the extent of loan collateralization. The paper stated that this omission could be important in the interpre- tation of the results, particularly with respect to the loans made to privately held firms. One participant inquired about the difficulty of obtaining and incorporating information about the nature of the collat- eral that supported each loan. The response indicated that the bank studied did not maintain such information in a single place or standard format, thus rendering the information difficult, if not impossible, to obtain.

Conclusion

I would like to offer one last observation regarding the motivation and interpretation of the paper. The "expert judgment" approach adopted does not address the question of why banks adopt a loan classification scheme. The authors discuss this issue briefly, but some elaboration may be helpful. Within a single-person context, a classification scheme rep- resents an aggregation process that "throws away" information. Demand

This content downloaded from 75.103.254.62 on Thu, 6 Nov 2014 01:11:28 AMAll use subject to JSTOR Terms and Conditions

118 CHRIS OLSEN

for such a scheme, in this setting, would seem to reflect some disutility for the processing of this information. Within a multiperson context, the objective(s) of a classification scheme is (are) potentially quite subtle. Motivation and coordination of the bank's loan officers are obvious possibilities. Another is the potential to strategically misrepresent infor- mation to a federal regulator, with the objective of manipulating loan losses and loan loss reserves. Research that enables discrimination among these and other possible explanations could facilitate understanding of the expert judgment process and suggest improvements to statistical models of loan classification.

This content downloaded from 75.103.254.62 on Thu, 6 Nov 2014 01:11:28 AMAll use subject to JSTOR Terms and Conditions