Likely Impact of the GRE@ Writing Assessment on Graduate Admissions Decisions

Donald E. Powers Mary E. Fowles

GRE Board Report No. 97-06R

September 2000

This report presents the findings of a research project funded by and carried

out under the auspices of the Graduate Record Examinations Board

Educational Testing Service, Princeton, NJ 08541

********************

Researchers are encouraged to express freely their professional judgment. Therefore, points of view or opinions stated in Graduate

Record Examinations Board Reports do not necessarily represent official Graduate Record Examinations Board position or policy.

********************

The Graduate Record Examinations Board and Educational Testing Service are dedicated to the principle of equal opportunity, and their programs,

services, and employment policies are guided by that principle.

EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, the modernized ETS logo, GRADUATE RECORD EXAMINATIONS, and GRE are

registered trademarks of Educational Testing Service.

Educational Testing Service Princeton, NJ 08541

Copyright 0 2000 by Educational Testing Service. All rights reserved.

Acknowledgments

We are grateful to Doug Baldwin, Carol Dwyer, Nancy Glazer, and Cyndi Welsh for helpful

comments and suggestions; to Leona Aiken, Henry Braun, Jose Mestre, Phil Oltman, Larry Stricker, and

Art Young for providing formal reviews of a first draft; to Marisa Famum, Terry Santos, and Agnes

Yamada for providing advice on the kinds of essay errors to consider; to Liane Patsula for providing data

on the Graduate Management Admissions Test Writing Assessment; to Rob Durso for help in generating

our data; to Pankaja Narayanan for providing information about GRE score sending patterns; to Debra

Friedman for assistance with the development of study materials; to Laura Jerry for analyzing the data; to

Ruth Yoder for identifying relevant sources to guide the development of study materials, for devising

study forms, for overseeing the general administration of the study, and for numerous other tasks; to the

faculty members and graduate admissions staff who contributed to the study; and to the Graduate Record

Examinations Board and its Research Committee for supporting this research.

This “judgmental policy capturing” study investigated how the new Graduate Record

Examinations (GRE@) Writing Assessment might influence graduate admissions decisions. Of interest

was the likely impact of GRJZ Writing Assessment scores on graduate admissions decisions, as well as

the probable effects of sending actual examinee essays to graduate institutions along with test scores and

whether the presence of construct-irrelevant flaws in these essays might negatively influence admissions

decisions. To answer these questions, 23 graduate faculty -- who represent nine graduate psychology and

14 graduate history departments and who have at least some experience with the admissions process --

reviewed simulated admissions folders and made admissions decisions for a set of fictitious applicants.

The study examined the relationship between these admissions decisions and a number of variables,

including GRE Writing Assessment scores, the availability of examinee essays in admissions folders,

GRE General Test scores, undergraduate grades, and the quality of the applicant’s recommendation and

personal statement. The study results suggest that GRE writing scores will probably have some impact on

graduate admission decisions, but that overall, the availability of examinee essays will have substantially

less, if any, influence. The study did not detect any significant tendency for graduate faculty to attend

unduly to extraneous flaws in examinees’ essays. A substantial majority of the participants felt that

receiving applicants’ GRE essays would either probably or definitely be useful.

KEYWORDS: Graduate Record Examinations (GRE), writing assessment, graduate admissions, policy capturing, test use

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..*.........*....................................................................... 1

Policy Capturing . ..~................~..~~.......~.............~...~~~..........~~..............~.....................~...~.~...~.,...~....~... 2

Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ................................. 4

Overview of the Research Design ................................................................................................... 4

Study Design ................................................................................................................................... 5

Adding Errors to the Essays ............................................................................................................ I

Strength of Applicants’ Admission Credentials .............................................................................. 8

The Sample of Faculty Participants ............................................................................................... 12

Instruments/Procedures ................................................................................................................. 13

Analyses ........................................................................................................................................ 18

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ................................ 20

Admission Rates ............................................................................................................................ 20

Reliability/Consistency ................................................................................................................. 21

Effect of GRE Writing Assessment Scores and Presence of Essays in Folders ........................... 21

Effect of Construct-Irrelevant Flaws in Essays ............................................................................. 29

Influence of Other Pre-Admission Factors .................................................................................... 39

The Fidelity of Our Simulation ..................................................................................................... 46

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..**............................ 47

Limitations ..................................................................................................................................... 48

Implications ................................................................................................................................... 50

Further Considerations .................................................................................................................. 52

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..*......................................*....................................... 53

Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........................ 57

Appendix B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........................ 59

Appendix C . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..*........................................................................................ 61

Appendix D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..*......................................*................... 63

List of Tables

Table 1. Correlations Among Pre-Admission Variables ........................................................................... 12 Table 2a. Hierarchical Regression Analysis for All History Faculty Considering GRE Writing

Assessment Only ........................................................................................................................ 24 Table 2b. Hierarchical Regression Analysis for All Psychology Faculty Considering GRE Writing

Assessment Only ........................................................................................................................ 25 Table 3a. Hierarchical Regression Analysis for All History Faculty Considering All Information.. ....... .26 Table 3b. Hierarchical Regression Analysis for All Psychology Faculty Considering All

Information ................................................................................................................................. 27 Table 4a. Hierarchical Regression Analysis for All History Faculty for Abbreviated Applications ......... 28 Table 4b. Hierarchical Regression Analysis for All Psychology Faculty for Abbreviated

Applications ............................................................................................................................... 29 Table 5a. Hierarchical Regression Analysis for History Faculty Based on GRE Writing

Assessment Only (Essays Available in Folders) ........................................................................ 31 Table 5b. Hierarchical Regression Analysis for Psychology Faculty Based on GRE Writing

Assessment Only (Essays Available in Folders) ........................................................................ 32 Table 6a. Hierarchical Regression Analysis for History Faculty Based on All Information (Essays

Available in Folders) .................................................................................................................. 35 Table 6b. Hierarchical Regression Analysis for Psychology Faculty Based on All Information

(Essays Available in Folders) ..................................................................................................... 36 Table 7a. Hierarchical Regression Analysis for History Faculty Based on GRE Writing

Assessment Essays Only (Essays Available in Separate Packet) .............................................. 37 Table 7b. Hierarchical Regression Analysis for Psychology Faculty Based on GRE Writing

Assessment Essays Only (Essays Available in Separate Packet) .............................................. 38 Table 8a. Standardized Regression Weights for Prediction of Likelihood Estimates for History

Faculty Participants .................................................................................................................... 40 Table 8b. Standardized Regression Weights for Prediction of Likelihood Estimates for

Psychology Faculty Participants ................................................................................................ 41 Table 9a. Mean Faculty Ratings and Weightings of Importance of Admissions Factors .......................... 43 Table 9b. Mean Faculty Ratings of Importance of Information Available From Recommendations ....... .44

Table 9c. Mean Faculty Ratings of Importance of Information Available From Personal Statements .... .45 Table D 1 .GRE General Test Scores Interpretive Data ............................................................................... 65 Table D2.GRE Writing Assessment Interpretive Data ............................................................................... 65

List of Figures

Figure 1. Summary of study procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Figure 2. Mean likelihood of admission on the basis of the GRE Writing Assessment only, by

score level, availability of essays, and prevalence of errors (psychology faculty). . . . . . . . . . . . . . . . . . . . 33 Figure 3. Mean likelihood of admission on the basis of the GRE Writing Assessment only, by

score level, availability of essays, and prevalence of errors (history faculty) . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Figure 4. Mean likelihood of admission on the basis of the GRE Writing Assessment only

(essays evaluated separately), by score level and prevalence of errors (psychology faculty) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .............. 34

Figure 5. Mean likelihood of admission on the basis of the GRE Writing Assessment only (essays evaluated separately), by score level and prevalence of errors (history faculty). . . . . . . . . .34

Introduction

What, if any, additional supporting materials -- actual examinee essays, for example -- should

accompany Graduate Record Examinations (GRE’) Writing Assessment scores when they are sent to

institutional test-score users? This question, arising during the development of the GRE Writing

Assessment, was a central focus of the research we have reported here. Also of interest was a more

general question concerning the likely influence of the new GRE Writing Assessment on graduate

admissions. Within the context of the broader study through which we sought to answer these two

questions, we also considered, in a less rigorous fashion, the impact of traditional pre-admission

information -- such as GRE General Test scores, undergraduate grades, and the quality of the applicant’s

recommendation and personal statement -- on graduate admissions decisions.

The GRE Writing Assessment, which was introduced operationally in the 1999-2000 testing

year, requires GRE test takers to write two essays -- one that entails discussing an issue and another that

involves analyzing an argument’. On the basis of these two writing samples, a composite score is

reported that reflects each writer’s ability to analyze and discuss complex ideas in a clear, well-focused,

coherent, and effective manner. To help test-score users understand the meaning of scores derived from

the assessment, interpretive materials and sample essays are provided with the score reports that are sent

to graduate institutions. Currently, however, because of concerns about possible misuse, examinees’

essays do not accompany their test scores. This decision was based on the concern that, without proper

training, graduate admissions staff may misinterpret writing performance, perhaps by using idiosyncratic

criteria to re-evaluate essays or by focusing on extraneous features that are not considered integral to the

construct of writing ability as defined by the GRE program. (Trained essay readers are instructed to

disregard such features when scoring GRE essays.) Moreover, because irrelevant features may be

differentially prevalent by gender, ethnicity, or cultural background, there exists in this practice a

potential for unfairness to certain groups of test takers. Of course, there are also cost considerations

involved in sending (and receiving) essays.

The reluctance to release actual GRE examinee essays to graduate institutions is not felt by

everyone in the assessment community, however. In fact, sending the essays along with the score reports

’ For a full description of the GRE Writing Assessment, sample questions, and scoring criteria, see the GRE website (http://www.gre.org/twotasks.html) or the 1999-2000 GRE Guide to the use of scores (Educational Testing Service, 1999).

was endorsed by a majority of the experts in the teaching and assessment of writing who offered advice

during the development of the new assessment. This endorsement, it seems, was predicated on the belief

that releasing essays may actually increase access to graduate education for some applicants whose

writing scores might otherwise preclude their admission. Making essays available might also, it was

believed, improve the academic community’s understanding of the writing assessment, whereas

withholding the essays could exacerbate feelings of mistrust about the new measure. These sentiments

are perhaps best reflected in the minutes of one GRE Writing Advisory Committee meeting, held June

25-26, 1996, which cited the following reasons for releasing examinees’ essays:

l Many faculty are quite capable of interpreting GRE essays appropriately.

l Some testing programs -- such as the Graduate Management Admission Test (GMAT) -- have been releasing essays for some time, and there have been few, if any, reports to suggest that this information has been abused.2

l When considering candidates with marginal writing scores, departments may find that having essays is helpful for determining the seriousness of an applicant’s writing deficiencies. In addition, departments may be willing to accept applicants whose writing exhibits a strong “voice” and well-reasoned explanations, knowing that (for international students, for example) grammatical and other second-language problems could be remediated after admission.

In light of the GRE Writing Advisory Committee’s stance, the GRE Board left open the possibility of

revisiting its current policy of not releasing essays; hence the study reported here. Anticipating our main

findings, we note here that, of the 23 graduate faculty who participated in the study we describe below, a

substantial majority felt that receiving applicants’ GRE essays would either probably (n = 3) or definitely

(a = 16) be useful.

Policv Capturing

“Policy capturing” is a technique used to model human decision-making. More specifically, the

term refers to efforts to explicate the relative weights or rules for combining information of different

types or from several sources in order to arrive at summary judgments or decisions (see, for example,

’ As a prelude to the study, we conducted a small, informal survey of admissions personnel at 14 graduate schools of management, all of whom were addressed through their web sites. Half of those contacted offered responses to our query about ” . . . whether or not you find that receiving applicants’ GMAT essays, along with their test scores, is useful in your admissions process.” A variety of views were expressed, but the most frequent theme was that, if read, the essays were consulted selectively -- for example, for international students or when a question arose about the authenticity of an alternative writing sample submitted by an applicant.

Hammond, Mumpower, & Smith, 1977). As Schmidt, Johnson, and Gugel(l978) point out, policy-

capturing methodology has been used successfully to study virtually all kinds of decision-making

strategies. Applications have included:

modeling the diagnoses of clinical psychologists (Goldberg, 1970)

investigating the factors underlying judicial decisions (Roehling, 1993; Werner & Bolino, 1997)

studying teachers as decision makers (Shavelson, 1973; Shavelson Jz Atwood, 1977) and students as evaluators of teaching effectiveness (Harrison, Ryan, & Moore, 1996)

understanding the bases for performance appraisals (e.g., Hobson, Mendel, & Gibson, 1981) and employment interviews (Dougherty, Ebert, & Callendar, 1986)

learning about the factors underlying sexual behavior (Finkelstein & Brannick, 1997; Wiederman, 1999) and preference for mates (Wiederman 8z Dubois, 1998)

capturing the strategies that expert horse racing handicappers use to predict odds at post time (Ceci & Liker, 1986)

The studies most relevant to the effort reported here -- for example, research on employee

evaluation (Gaeth & Shanteau, 1984; Stumpf & London, 1981; and Zedeck & Kafry, 1977) -- have been

undertaken in organizational and industrial settings. Several efforts, however, have aimed to clarify the

nature of admissions decisions in academic settings. Perhaps best known is a study of admissions

decisions at the University of Oregon’s psychology department (Dawes, 1971), which found that GRE

General Test scores were more highly related to admission ratings than were undergraduate grade point

averages (GPA). Other studies include:

l Schmidt, Johnson, and Gugel’s 1978 study of applicants to psychology programs at Michigan State University, in which different subdisciplines of psychology weighted criteria very differently

l Wallace and Schwab’s 1976 study of decisions about applicants to an industrial relations program, in which GRE General Test scores and undergraduate grades were the most consistent predictors of admission decisions

l Gomey and Jaeger’s 1995 study of the likely impact of several new GRE tests on graduate admissions, in which tests received different weights by discipline

l Kline and Sulsky’s 1995 study of the acceptability of applicants to a graduate psychology program at the University of Calgary, in which professors used information relatively consistently, but sometimes applied complex rules when making judgments.

Besides contributing to a better understanding of graduate admissions, each of these investigations also

provided guidance for the study reported here. Gomey and Jaeger’s (1995) advice, for example, was to

use simulated applications that are as realistic and as representative of the typical applicant pool as

possible.

Method

Overview of the Research Design

The primary methodology employed in the study reported here has usually been referred to as

“judgmental policy capturing.” This technique was designed to model statistically, or “capture,” the

judgments of human decision makers. However, in contrast to most policy capturing studies, which

primarily center on the decisions of individuals, the focus here was on decision makers in the aggregate.

It should be stressed at the outset that, like most policy capturing studies, ours relies mainly on

correlational data. Thus, with the exception ‘of those variables that were experimentally manipulated and

were therefore uncorrelated with other variables, a large weight for a given variable does not necessarily

mean that the variable was actually used in decision making. Rather, it could signify instead only that

some highly related variable was used.

For this study, a small number of history (r! = 14) and psychology faculty (n = 9) involved in

graduate admissions each reviewed 27 simulated “admission folders” for a set of fictitious applicants and

made judgments of the admissibility of each one. Each folder contained an application for graduate

admission, which specified the applicant’s GRE General Test scores, GRE Writing Assessment score,

undergraduate GPAs, and other personal information that typically appears on applications to graduate

school. Each application also included a simulated recommendation from an undergraduate professor,

and a manufactured summary of information gleaned from a hypothetical personal statement. (The

rationale for our use of simulations instead of actual recommendations and personal statements will, we

hope, become clear later in this report.)

A key variable of interest to us was the amount of information provided about applicants’

performance on the GRE Writing Assessment. Within each graduate department in our study, half of the

faculty participants received applicants’ GRE writing scores, while the other half received applicants’

GRE writing scores plus copies of the essays on which the scores were based. All of these essays had

been written by participants in an earlier GRE research study and had been scored by ETS-trained

readers.

A second design factor involved the number of additional errors introduced by us into applicants’

GRE essays. To control this factor experimentally, we systematically inserted extraneous errors of

various sorts into two thirds of the essays, while leaving the remaining 18 essays intact. The errors were

the kind that trained GRE essay readers are instructed to keep in perspective when evaluating examinees’

essays. We hypothesized that such errors could negatively influence graduate faculty perceptions of

applicants’ writing skills3.

Analyses were undertaken to determine how the likelihood of the admission of our hypothetical

applicants relates to three variables: (1) the applicants’ GRE Writing Assessment scores, (2) whether or

not applicants’ essays accompanied their GRE writing scores, and (3) the extent to which applicants’

essays contained extraneous flaws. Secondarily, we also estimated the influence of several traditional

graduate school admissions criteria -- GRE General Test scores, undergraduate GPAs, and the quality of

the applicant’s recommendation and personal statement -- on these admission decisions.

Study Design

The specific factors varied in the study reported here were:

the presence or absence of actual essays in application folders

applicants’ GRE writing scores

the extent to which essays contained extraneous flaws (none, some, or many)

the strength of applicants’ other admissions credentials -- such as, the level of their GRE General Test scores and undergraduate GPAs

Inclusion of essays. As noted earlier, half of the participating faculty saw the scores from the

GRE Writing Assessment; the other half saw scores and actual essays. All participants received a set of

interpretive materials explaining the Writing Assessment. These materials included a description of the

3 A third feature of the essays, which was of possible interest to the GRE Research Committee, was the extent to which essays expressed controversial views. We decided, however, not to try to incorporate this feature into the current study because adequately evaluating the influence of controversial opinions would require additional information about faculty participants -- for example, what kinds of views they themselves hold. We felt that, while the issue of controversy is an important one, it was beyond the scope of the present effort.

5

assessment and the scoring guides, questions and answers about various aspects of measure, and sample

benchmark essays representing each point on the GRE writing score scale.

GZ?E Writing Assessment scores. Examinee essays were selected from some 4,000 essays that

were written by third- and fourth-year undergraduates and by first-year graduate students at 26

geographically diverse colleges and universities during two earlier research studies of the GRE Writing

Assessment (Powers, Fowles, & Welsh, 1999; Schaeffer, Briel, & Fowles, in press). From this sample,

we identified a total of 27 participants, each of whom had written both an issue essay and an argument

essay. The GRE writing scores for this small sample spanned the range of GRE Writing Assessment

scores: exactly three examinees were identified at each half-point score interval from 2.0 to 6.0. (Because

scores below 2.0 are rare in other holistically scored graduate-level writing tests, they were not included

in the study.) Each of the resulting 54 essays (that is, 27 issue essays and 27 argument essays) had

previously received exactly the same score by each of two trained readers (in operational scoring, a

difference of one point or less is considered to be an acceptable level of agreement.).

Insertion of minor-flaws. To control extraneous flaws or errors, we systematically modified some

essays by introducing irrelevant flaws. The work of Freedman (1979) provided a good model for

experimentally manipulating this design factor. For a third of the 27 applicants (one at each GRE Writing

Assessment half-score point interval), both the issue essay and the argument essay remained as originally

written, with no additional irrelevant characteristics added. For another third of the applicants (again, one

at each score level), both essays were modified by introducing “some” irrelevant features. For the

remaining third of the applicants, “many” such characteristics were inserted into essays at each score

level. Caution was needed here, however, as errors can differ significantly in their quality: Some are

considerably more serious than others because they render an argument less compelling or because they

interfere with meaning. The quantity of errors was also a concern: For example, a few diction errors may

matter little in an otherwise strong essay, but a large number of word-choice errors may have a greater

impact.

With respect to the quantity of errors, we defined “some” as any two of the following error

groupings, while “many” was defined as the presence of all three of the following error groupings:

0 a minor error of fact that did not affect the strength of the essay

l two errors of grammar or diction that did not seriously affect the coherence of the essay

0 three spelling errors

GRE Writing Assessment specialists judged all of these newly added errors to be contextually so

trivial as to have no effect on the overall quality of any essay. To verify this judgment, two additional

trained essay readers who had no knowledge of the original scores reevaluated each of the modified

essays. This step served to verify that scores had not changed as a result of our modifications.4

Other admissions credentials. To determine the impact of more traditional admissions

credentials, we specified the following for each hypothetical applicant:

l GRE General Test scores (verbal, quantitative, and analytical)

l undergraduate GPAs (overall and in major field)

0 the quality of the applicant’s recommendations (both a rating of the student’s overall promise, as suggested by the recommendation, and a summary of the recommendation)

0 the quality of the applicant’s personal statement (that is, a rating of the information that was, hypothetically, gleaned from the statement)

Using a data simulation program, these data were generated so as to meet specified means and variances

for each variable, and for the variables as a set, to meet specified correlations among them. Because the

number of constraints was relatively large compared with the number of cases, converging exactly on the

target statistics was not possible. Manual adjustments to some data values were needed in order to attain

a sufficiently close approximation to the targets. More detail about the specification of these variables is

provided below.

Adding Errors to the Essays

Our task was to decide which (and how many) errors to include in our “doctored” essays. Our

aim was to use errors that would be potentially distracting to untrained readers, but would not affect the

overall quality of essays. Toward this end, several sources guided our decisions about the kinds of errors

that we inserted into essays. First, we reviewed selected literature on “error gravity” for clues regarding

the sorts of errors that might be most noticeable or irritating to untrained readers (see Connors &

4 The new GRE Writing Assessment scores given to the 27 applicants corresponded very closely to the scores that were assigned originally (r = .92). The mean original score was 4.0, while the mean restore was 4.06. An inspection of original and subsequent restores for individual applicants did not suggest that modified essays were any more likely than unmodified essays to receive different scores upon restoring. Thus, the insertion of irrelevant errors did not appear to affect the scores assigned by trained essay readers.

7

Lunsford, 1988; Hairston, 1981; Janopoulos, 1992; Rifkin & Roberts, 1995; Santos, 1988; and Vann,

Meyer, & Lorenz, 1984, for example). Next, we examined essays that were written during the fall 1997

field testing of GRE writing prompts to glean this information. This was accomplished both

retrospectively (readers revisited previously scored essays to identify errors or features that were

potentially distracting), and concurrently (readers recorded errors and distracting features during their

initial scoring of GRE essays).* We also asked several experts6 to tell us the kinds of errors they feel are

prominent in GRE essays and that graduate faculty might find distracting. Finally, we reviewed the

manuals and materials used to train ETS readers in the holistic scoring of essays. Perusal of these

manuals was informative, both with respect to what is expected of readers and how the construct of

writing is defined.7

Strength of Annlicants’ Admission Credentials

Each simulated admissions folder contained three documents -- an application for graduate study

(which contained information about our hypothetical students’ test scores and grades), a recommendation

5 Among the potentially distracting features that readers noted were the following: (a) the accumulation of spelling errors, (b) “bull,” jargon, intrusive literary allusions, or other “inflated effects,” (c) ESL and other dialect markers that do not interfere with meaning, (d) factual errors, (e) highly charged emotional responses or diatribes, (f) incompleteness, (g) highly specialized, discipline-specific subject matter, (h) unusual or idiosyncratic approaches to the topic, (i) variation in the quality of various parts of the essay, (i) nonstandard forms, such as double negatives, (k) punctuation errors, (1) misuse of diction or poor word choice, (m) the absence of paragraphing, and (n) “mangled” cliches, such as “looking at the world through rose-colored eyes.”

6 Agnes Yamada, Professor of English and Department Chair at California State University at Dominguez Hills (personal communication, April 3, 1998), Terry Santos, Associate Professor at Humboldt State University (personal communication, April 5, 1998), and ETS staff member Marissa Farnum (personal communication, April 5, 1998). ETS staff members Nancy Glazer and Cyndi Welsh were especially helpful to us in our decision-making about which errors to insert into the essays.

7 These points are emphasized with regard to holistic scoring: Score holistically -- for the overall quality of thinking and writing, as described in the Scoring Guide. Don’t focus on minor errors or a single weakness. Three examples are not necessarily better than one; what matters is how well-chosen the examples are and how well they are discussed.

Read supportively; look for and reward what is done well in relation to the scoring criteria, rather than penalize what has been done poorly or omitted.

Read the entire response; the writing sometimes improves dramatically as the writer becomes more engaged in the task.

Acknowledge the limitations of time. Depending on the task, candidates have only 45 or 30 minutes to draft and revise a response. Judge the responses as first drafts and do not expect “perfect” or highly polished papers, even at the top score levels.

Do not penalize an unfinished but developed response for lacking a conclusion.

Do not judge a response by its length; some short responses are very good and some long ones deserve low scores.

form, and a summary of the applicant’s statement of purpose. The development of application materials

was based a small number of actual applications that GRE test takers supplied and permitted us to use as

models. The application form itself was a composite of actual forms currently used by graduate

institutions, as was the recommendation form.

For each of the two disciplines represented by our application folders -- history and psychology

-- we specified a target matrix of means, variances, and correlations among variables. For GRE General

Test scores, for example, target means and variances were based on statistics for actual examinees who

had majored in psychology and history (Educational Testing Service, 1998). Because the departments

that agreed to participate in the study were, with few exceptions, relatively selective ones, GRE General

Test score means supplied in the folders of our fictitious history and psychology applicants were actually

set somewhat higher (by about 25 points) than the means for all GRE test takers in each of the two

disciplines. Faculty comments confirmed the appropriateness of setting these higher test-score levels.

The intercorrelations reported for all GRE General Test takers (Educational Testing Service,

1998) served as the target for correlations among the three GRE General Test scores -- verbal,

quantitative, and analytical --that we provided. The target means and variances for undergraduate GPAs,

and their correlations with GRE General Test scores, were based on data provided by Schneider and

Brie1 (1990) and Wilson (1986). For GRE Writing Assessment scores, target statistics were based on

research data from pre-operational tryouts of the GRE Writing Assessment (Powers, Fowles, & Boyles,

1996) and on data from the first two years of the operational GMAT Analytical Writing Assessment

(Liane Patsula, personal communication, May 27, 1999). The use of data from the GMAT program

seemed justified, as the GMAT and GRE writing assessments are very similar to one another with respect

to both their content and scoring. Finally, overall undergraduate GPAs and GPAs in the major field of

study were specified so as to relate very strongly to one another, with the latter having a slightly higher

mean.

A review of several recommendation forms suggested a number of traits that recommendation

writers are often asked to rate or comment on. In order to identify other relevant traits, we also consulted

literature on the nature and use of recommendations (for example, Aamodt, Bryan, & Whitcomb, 1993;

Range, Menyhert, Walsh, Hardin, Ellis, & Craddick, 1991). In this regard, Keith-Spiegel’s (1991)

analysis of several hundred recommendations written in support of graduate applications, and an

accompanying discussion of some two dozen applicant qualities mentioned in these letters, proved very

helpful.

Taking McCauley’s (1991) advice, rather than using predominately narrative accounts of

applicants’ qualities, we chose to quantify the information in our hypothetical applicants’ letters of

recommendation and personal statements. For both recommendations and personal statements, we

specified applicant traits or qualities that might be manifested in narrative accounts, and then provided

numerical ratings of each trait, as if the narratives had been evaluated previously for evidence of each

trait. That is, for each applicant we provided a list of traits or personal qualities that had already been

rated (for recommendations) or coded (for personal statements).

Employing this strategy, instead of providing potentially ambiguous descriptive letters, was

intended to minimize the level of inference required of applications readers (Tommasi, Williams, dz

Nordstrom’ 1998). Although the resulting numerical evaluations may not have been entirely

characteristic of the information normally reviewed by graduate admissions committees, this strategy did

allow us to sidestep several potential complications associated with the use of actual narrative statements

and recommendations. In particular, we were able to exert some control over the numerous factors that

are known to moderate the influence of recommendations. Among these influences are length (Weins,

Jackson, Manaugh, dz Matarazzo, 1969), tone, content, and structure (Loher, Hazer, Tsai, Tilton, &

James, 1997), writer (Ralston & Yoder, 1989), and the presence of specific examples versus generalities

(Knouse, 1983).

With respect to summary recommendations, our fictitious applicants were divided evenly among

the following categories: “strongly recommended, ” “recommended,” and “recommended with

reservations.” None were placed in the lowest category (“not recommended”), since few graduate school

applicants are, we believe, so unfortunate as to secure entirely negative endorsements. We set the mean

rating of “overall promise” on the recommendation forms between “good” (4) and “very good” (5) on a

seven-point scale ranging from “below average” (1) to “truly exceptional” (7). Summary

recommendations and ratings of overall promise were specified so as to correlate strongly with one

another (r = .87), but only slightly with other variables. Each of the 16 individual traits (dependability,

intellectual ability, and so on) that appeared on the recommendation form were specified to correlate

moderately with one another (r = .31 to .77), and strongly with the overall rating of promise (r = .68 to

.79). Consistent with research by Zeleznik, Hojat, & Veloski (1983), recommendations were generated so

as to relate modestly to undergraduate GPAs (r = .21, .28), and lacking any other information, to correlate

only slightly to test scores and other variables (r = .03 to .34).

10

To devise personal statements, we followed a strategy similar to the one used for

recommendations. First, we reviewed a set of real personal statements that had been provided for

previous research on the value of personal statements as an indication of writing skill (Powers Jz Fowles,

1997). We also examined a number of personal statement forms that are currently used by graduate

institutions, and noted the kinds of information that applicants are asked to provide. Finally, we reviewed’

relevant literature on personal statements in order to ascertain further the kinds of evidence that these

statements typically contain (Ferguson, 1991; Hatch, Hill, & Hayes, 1993; Hawkins, 1993; Keith-

Spiegel, 1991; Murphy, 1991; Paley, 1994; Wickenden, 1982; Willingham, 1985; Willingham & Breland,

1982). On these bases, we created a checklist of important qualities and traits that might be inferred from

an applicant’s personal statement. We completed this checklist for each applicant and included it in each

application folder as “A Summary of Applicant’s Statement of Purpose.”

The average rating of overall strength of personal statements was set between “good” (4) and

“very good” (5) on the same seven-point scale used for recommendations, and the correlations of this

rating with other pre-admission measures was assumed to be slight. Correlations among the 11 qualities

or traits gleaned from the personal statements (motivation, likely commitment to the field, and so on)

were moderate to strong (r = .35 to .79), as was the correlation of each trait with the rating of the overall

strength of the statement (r = .67 to .79).

Descriptive statistics for, and correlations among, all pre-admission measures are shown in Table

1. The same statistics were used for history and psychology applicants, except for a different profile of

mean GRE General Test scores (for GRE verbal, quantitative, and analytical scores, respectively, m =

556, 545, and 589 for history applicants, and m = 486,525, and 569 for psychology applicants). These

between-discipline differences in test scores were specified on the basis of data available for all GRE

General Test takers by discipline.

11

Table 1. Correlations Among Pre-Admission Variables

Variable

Variable

M SD 1 2 3 4 5 6 7 8 9

1. Gm verbal score

2. GRE quantitative score

3. GRE analytical score

4. GRE writing score

5. Undergraduate GPA -- overall

6. Undergraduate GPA -- major

7. Summary recommendationb

8. Rating of overall promisec

9. Personal statementC

4861556” 86

5251545” 90 .47

56915 89” 85 .59 .61

4.0 1.32 .42 .13 .34

3.27 0.31 .30 .27 .23 .12

3.33 0.32 .20 .21 .29 .16 .71

2.04 0.81 .25 .03 .22 .05 .28 .21

4.26 1.46 .20 .12 .20 .07 .17 .23 .87

4.56 1.69 .03 .09 .08 .02 .04 .17 .18 .19

10. Selectivity/reputation of 2.0 0.83 .27 .31 .30 .05 .09 .07 .34 .32 .33 undergraduate schoold

Note. Correlations among variables were the same for psychology and history applicants, as were means and standard deviations for all variables except GRE General Test scores. a The first mean is for psychology applicants; the second is for history applicants. b On a three-point scale: 1 = recommend with reservations ,2 = recommend, 3 = strongly recommend. ’ On a seven point scale: 1 = below average, 2 = average, 3 = somewhat above average, 4 = good, 5 = very good, 6 = outstanding, 7 = truly exceptional. d On a three-point scale: 1 = less selective/good reputation, 2 = very selective/very good reputation, 3 = extremely selective/extremely good reputation.

The Sample of Faculty Participants

History and psychology departments were selected for the study because these two disciplines

typically require relatively extensive writing of students, and thus these departments seemed to us to be

reasonably likely prospects for adopting the GRE Writing Assessment. No attempt was made to identify

a representative sample of departments within these disciplines. Rather, for each of discipline, we

selected departments primarily on the basis of the high volume of GRE test scores that they currently

receive. Further, we identified clusters of departments that had, for a recent testing year, received GRE

General Test scores from at least some of the same test takers. That is, these departments appeared to

“compete” for the same or similar students, and could thus be expected to have somewhat similar

applicant pools. The aim here was to decrease the possibility that our simulated applicant pool would

deviate dramatically from the pool of applicants typically seen by the departments participating in our

12

study. This strategy obviated the need to tailor applications to each department, thus enabling us to use

the same set of admissions folders for every department within a single discipline.

Once departments were identified, letters of invitation were sent to department chairs. In order to

participate, each department was required to identify two faculty members who would be willing to

evaluate a package of fictionalized “admissions folders.” Each faculty participant received an honorarium

of $200. Our invitation intentionally solicited faculty who had recently been involved in graduate

admissions, and preferably, who currently served on a departmental admissions committee. A total of six

psychology departments and eight history departments agreed to participate. These departments were,

according to participants’ reports of admission rates, relatively selective. Selection ratios ranged from 7%

to 20% for psychology departments (median = 10%) and from 10% to 70% for participating history

departments (median = 33%).

From these participating departments, data were eventually received from nine psychology

faculty and 14 history faculty, all of whom had been involved in graduate admissions at some point in

their careers. The range of experience was from 3 to 25 years for psychology faculty (median = 20 years)

and from 1 to 25 years for history faculty (median = 5 years). A majority of the participants (6 of 9

psychology faculty and 13 of 14 history faculty) had been involved in graduate admissions during the

previous year.

Instruments/Procedures

Within each department, faculty members were randomly assigned to one of two conditions:

0 reading folders that contained both GRE Writing Assessment scores and examinee essays

0 reading folders that contained only GRE Writing Assessment scores, but no examinee essays

For each condition, participants received by mail a packet of materials, general information about the

study, and directions for completing the study tasks. Included in each packet were simulated admissions

folders for the 27 fictitious applicants, each applying to a doctoral program. Each folder contained the

following pieces of information:

13

an application’ for graduate study, which included GRE General Test scores, GRE Writing Assessment scores, undergraduate GPAs (overall and in major field), undergraduate institution, and proposed program of study (a sample application form is provided in Appendix A)

a recommendation from an undergraduate professor (a sample recommendation form is provided in Appendix B).

a summary of information gleaned from the applicant’s personal statement/statement of purpose (a sample personal statement form is provided in Appendix C)

For the score-plus-essays condition, each folder also contained two GRE Writing Assessment essays (one

issue essay and one argument essay) written by the applicant.

For the scores-only condition, application folders did not contain essays. However, an additional,

separate, sealed package containing applicants’ GRE essays was provided to participants who were

assigned to the scores-only condition. These participants were instructed to open this package and

evaluate the enclosed essays only after completing all other tasks. The aim here was to compare decisions

within the same group of faculty members according to whether or not they had access to GRE essays.

(For the score-plus-essays condition, this separate packet was missing, as applicants’ GRE essays were

already included in their application folders.) Application folders were arranged in a different random

order for each faculty participant.

Before completing study tasks, all participants were first asked to read material about the new

GRE Writing Assessment. This material contained background about the test, information about the

scoring criteria, and, for both the issue and the argument topics, examples of essays at each score level.

After becoming familiar with the Writing Assessment, participants in each condition were asked to judge

the admissibility of each of the hypothetical applicants included in the folders, indicating for each

applicant:

l the faculty member’s own recommendation for admission (deny or admit)

0 the faculty member’s estimate of the likelihood (0 to 100%) that their program/department would admit the applicant

8 Applications did not reveal the gender or the racial/ethnic background of participants, as investigating the effect of these factors was beyond the scope of the present study. Specifying these characteristics would, we believed, serve to confound the results, given the small number of applicants that were included. To conceal the gender of each applicant, gender-ambiguous first names -- such as Dana, Lee, and Robin -- were used.

14

We thought that these two different perspectives on the admissibility of applicants -- one referenced to

the individual faculty member and the other to an admissions committee -- might differ psychologically.

Furthermore, because of the nature of the resulting variables, either dichotomous or continuous, the

results might differ statistically also.

In making their judgments, participants were asked to take two passes through the applications,

first considering only information about the applicants’ writing skill, as reflected by GRE Writing

Assessment scores for the scores-only condition, or by GRE Writing Assessment scores and essays for

the scores-plus-essays conditiong. Then, on a second pass, they were to consider all the available

information for each applicant. After each pass, participants were asked to render the two admissions

decisions described above. Then, participants in the scores-only condition were asked to undertake a final

step -- to open the separate package of applicant essays and, on the basis of these essays, make the same

two judgments about each applicant’s admissibility again.

All participants were advised to evaluate the applications in a way that best matched their

department’s admissions procedures. For example, if applications are first sorted into broad categories

before making individual decisions, they should proceed accordingly for our study. The faculty were also

reminded that our “applicant pool” might differ in size and/or overall quality from their typical pool of

applicants. To minimize the potential effect of any such differences, they were asked to apply their usual

standards, so that students “admitted” from our applicant pool would approximate the general caliber of

their typical entering class.

In addition, recognizing that our study procedures could not faithfully simulate every aspect of

the graduate admissions process, we asked participants to accept a number of simplifying assumptions

and to make their best judgments in light of incomplete information. With regard to assumptions, for

instance, all of our hypothetical applicants had obtained undergraduate degrees in the same field to which

they were applying for graduate study, and all had graduated within six months of submitting their

applications. With respect to the recommendations we provided, study participants were asked to assume,

for the sake of standardization, that the unnamed recommendation writer was someone they knew and

respected, and who was active, but not necessarily well known, in their field. Because we supplied only

9 We asked participants to focus on the GRE Writing Assessment during their first pass to ensure that they would be at least somewhat oriented to this new measure. In retrospect, the alternative order -- that is, having participants attend first to all the available information in folders -- might have been even more appropriate, as this order might have minimized the likelihood of participants focusing unduly on the Writing Assessment in making their judgments.

15

one recommendation per applicant, participants were asked to assume further that other, missing

recommendations were consistent with the one provided.

With respect to missing information, we did not, for example, reveal a number of possibly

relevant characteristics of our fictitious applicants -- such as their gender, race/ethnicity, and age.

Furthermore, we omitted other important information, such as full undergraduate transcripts, which

departments undoubtedly consider when making real admissions decisions. Instead, we asked participants

to assume that each applicant had completed any undergraduate courses that were considered necessary

or prerequisite to their programs, and that performance in these courses was commensurate with

applicants’ overall undergraduate GPAs, as specified on the application. In addition, for each applicant

we identified only a fictitious undergraduate school, assumed to be regionally accredited, along with an

indication of that school’s selectivity/reputation as either (3) extremely selective/extremely good

reputation, (2) very selective/very good reputation, or (1) less selective/good reputation. Five to six

names of real undergraduate schools were given as examples of each category. Institutions were assigned

to these categories on the basis of ratings provided annually by such sources as Time Magazine, U.S.

News and World Report, Peterson’s Guides, and College Board listings. Only schools with

classifications that were consistent across these various sources were cited as examples.

As a next step, each participant made another round of admissions decisions for what they were

told was a second batch of 27 applicants; information for these applicants was given only in abbreviated

form. These “new” applicants were actually the same 27 applicants evaluated in the previous steps. For

this step, however, applicants (with names deleted) were presented in a different order, and information

was presented in a different format -- on a single, one-page form instead of in individual application

folders (A sample of this abbreviated application form is provided in Appendix D.). These abbreviated

applications were presented in order to obtain a crude estimate of the consistency of the decisions made

by each faculty participant. Because the abbreviated folders did not match the full folders exactly, this

procedure necessarily underestimates actual consistency.

Next, participants indicated, on a five-point scale (0 = “not considered at all” to 5 = “extremely

important”), the importance of each of several general factors (GRE General Test scores, undergraduate

grades, recommendations, and so on) in their admissions decisions. Using the same scale, they also

indicated the importance of each trait listed on our recommendation form, as well as each quality listed

on our summary of applicants’ personal statements. They were also asked to assign weights to each of

these admissions factors to indicate more precisely the relative importance they attached to each one.

16

This information allowed us to compare participants’ perceptions of the importance of various factors

with the weights that were computed for them when making their admissions decisions for our study.” It

also enabled us to assess the perceived importance of certain factors (for example, personal interviews)

that we were not able to consider in this study.

Finally, we asked participants to complete a brief background questionnaire about their

experience with graduate admissions. We also solicited their opinions as to how our admissions

procedures may have differed from the admissions procedures in their departments, and if so, how the

differences may have affected the admissions decisions they made for this study. Our objective here was

to identify any deviations that might have threatened the validity of our results.

When they had completed all of the tasks, participants returned all study materials to ETS. They

were promised that all information would be treated confidentially. Figure 1 summarr ‘zes the various

tasks undertaken by participants in each condition.

Before conducting the study proper, all instruments and procedures were pilot tested with

directors (and one assistant director) of graduate admissions at two graduate institutions in New Jersey.

Our simplifying assumptions were judged by these participants to be generally reasonable, and our

directions and procedures, understandable. The major concern they expressed was the omission of

personal statements or other nontest writing samples, which these directors thought might be useful for

obtaining a sense of applicants’ personalities.

lo We should be clear here about what this agreement (or lack thereof) may suggest. As one of our reviewers (Leona Aiken, personal communication, March 28,200O) pointed out, even if the actual importance given to pre-admissions variables in decision making is in complete agreement with participants’ perceptions of the importance of these variables, the computed regression weights can be expected to diverge from perceptions because, unlike perceptions, regression weights take into account the correlations among the pre-admissions variables. To put it simply, when two correlated predictors are both important, the stronger predictor will be “credited” with the variance explained by both predictors. The stronger predictor will then appear, according to its regression weight, to be even stronger, and the weaker predictor even weaker.

Study Condition

GRE Writing Assessment scores only GRE Writing Assessment scores plus essays

1. Read description of, and information about, the GRE Writing Assessment

2. Reviewed admissions folders, considering only GRE Writing Assessment scores

3. Made recommendations and estimated the likelihood of admission to department on the basis of GRE Writing Assessment scores only (Tables 2a, 2b)

4. Reviewed admissions folders again, this time considering all information in folders, including GRE Writing Assessment scores

5. Made recommendations and estimated likelihood of admission based on all information in folders (Tables 3a, 3b)

6. Read applicants’ GRE Writing Assessment essays that were provided separately, and made recommendations and estimates (Tables 7a, 7b)

2. Reviewed admissions folders, considering only GRE Writing Assessment scores and essays

3. Made recommendations and estimated the likelihood of admission to department on the basis of GRE Writing Assessment scores and essays only (Tables 2a, 2b, 5a, 5b)

4. Reviewed admissions folders again, this time considering all information in folders, including GRE Writing Assessment scores and essays

5. Made recommendations and estimated likelihood of admission based on all information in folders (Tables 3a, 3b, 6a, 6b)

7. Reviewed roster of abbreviated applications, made recommendations (Tables 4a, 4b)

8. Completed background questionnaire, provided reactions to the study procedures

9. Rated the importance of various pre-admissions factors (Tables 9a, 9b, 9c)

Figure 1. Summary of study procedures.

Analyses

To assess the degree to which participants made consistent decisions, we compared their

recommendations to admit or deny applicants based on the complete application folders with their

recommendations based on abbreviated applications for the same applicants. Finally, as is typical in most

policy capturing studies, the multiple R-square for each participant was taken as an indication of the

extent to which participants used the information that was available to them.

To determine the effects of GRE Writing Assessment scores, the presence of GRE Writing

Assessment essays in folders, and the prevalence of errors in these essays on admissions decisions, the

18

following analyses were conducted. For psychology faculty (and separately for history faculty),

hierarchical” regression analyses were performed in which (1) admit/deny recommendations and (2)

likelihood of admission estimates served, in turn, as dependent variables. At each stage, a significance

test was computed to determine the contribution of each additional independent variable to the multiple

correlation. Logistic regression was used to analyze the dichotomous admit/deny recommendations;

ordinary, least-squares linear regression was used for estimates of the likelihood of admission, to which a

logit transformation was first applied in order to better meet the assumptions underlying the analysis.

Independent variables were entered in the following sequence:

1. department to which the applicant applied -- that is, the institutional affiliation of faculty member (dummy codes were assigned to represent each department)

2. GRE General Test verbal scores, GRE General Test quantitative scores, GRE General Test analytical scores, overall undergraduate GPA, undergraduate GPA in major, summary faculty recommendation, overall rating of promise by recommendation writer, strength of personal statement, and selectivity of undergraduate school

3. GRE Writing Assessment scores

4. presence of GRE Writing Assessment essays in the admission folder

5. a product variable indicating the interaction between GRE Writing Assessment score and presence of GRE Writing Assessment essay in folder

The decision to enter GRE writing scores after other admissions variables reflects our interest in the

impact of the GRE Writing Assessment above and beyond the impact of traditional pre-admission

measures. It is possible, of course, that in actual admissions decisions, GRE writing scores could

supplant, not just supplement, other pre-admissions measures, thus making our analytic strategy a

conservative one with respect to estimating the impact of the GRE Writing Assessment.

Within each of the two disciplines, we treated each admissions recommendation (and each

likelihood estimate) as an independent observation (so that, for example, 27 applicants times 9

psychology faculty = 243 observations each for both admit/deny recommendations and likelihood

estimates). The same analyses were repeated for the data based on abbreviated applications (except that

variables 4 and 5 were not relevant to these).

l1 By “hierarchical,” we mean that explanatory variables, or sets of them, were entered cumulatively in a prespecified order as described by Cohen & Cohen ( 1983).

19

The same kind of analysis was repeated for the subset of participants who were given application

folders that included both GRE Writing Assessment scores and essays. This time, however, in order to

assess the effect of construct-irrelevant errors, we included two dummy-coded variables to reflect the

presence of these errors (none, some, or many) in the GRE Writing Assessment essays. As in the

previously described analyses, these variables were added in a final step to the regression equations

containing other independent variables. The increase in R-square that resulted from considering the

prevalence of errors in essays suggested the extent to which participants were influenced by the presence

of these irrelevant features. This analysis was repeated for faculty whose folders did not include GRE

essays, but who reviewed GRE essays as a final, separate step.

Finally, although the 27 applicants were few in relation to the number of predictors -- or

admissions variables -- we also computed regression equations for individual faculty participants. Only

likelihood estimates, not recommendations to admit or deny, were used here. To assess the lack of

stability due to small sample size, we conducted the analysis for estimates based on both complete and

abbreviated application folders. For each faculty participant, we calculated the correlation between the

regression weights based on their reviews of complete and abbreviated applications. Finally, we

compared the results of the policy capturing analysis with the perceptions of participants regarding the

importance of the various admissions criteria.

Results

Admission Rates

The admission rates for our hypothetical applicants were quite similar regardless of the particular

study condition under which they were judged. For applicants to psychology departments, acceptance

rates ranged from 29% (when all information was considered) to 37% (when abbreviated applications

were considered). For history applicants, the rates ranged from 30% (when only writing performance was

considered) to 35% (when abbreviated applications were considered). These rates are relevant from two

perspectives. First, they show that the quality of the hypothetical applicants in our study, and the

standards applied to their admission by our study participants, were reasonably similar on average to

those reported by study participants for their departments. Secondly, the proportion of decisions to admit

versus those to deny is germane to the regression analyses that we report below, as this split is known to

affect regression and correlation coefficients and their errors, as well as the estimation of agreement

rates. These results suggest proportions that are not so extreme as to affect these statistics dramatically.

20

Reliability/Consistency

For decisions to admit or deny, the degree of agreement based on complete applications versus

the degree of agreement based on abbreviated applications ranged from 70% to 96% for individual

psychology faculty (median = 85%) and from 74% to 93% for history faculty (median = 81%). Overall,

the kappa statistics were K = .66 for psychology faculty and K = 57 for history faculty. R-square values

were relatively large for each participant (66 to .90 for history and 55 to .92 for psychology), suggesting

that participants did make use of the information provided in folders. Thus, although variation was

evident, faculty participants were generally consistent in making their decisions and estimates, even

when different formats were used to present slightly different information.

To obtain another indication of consistency, we compared each faculty participant’s

recommendations with his or her estimates of the likelihood of admission by the department.” We

defined the following combinations as being inconsistent:

0

admit recommendations with likelihood of department admission below 50%

deny recommendations with likelihood of department admission above 50%

Of the 243 “admissions decisions” made by psychology faculty, only two were inconsistent according to

this definition. Similarly, of the 324 decisions made by history faculty, only 13 were inconsistent, with

five denials having a likelihood of departmental acceptance above 50% and eight admits having a

likelihood of departmental acceptance below 50%. Nearly all of these inconsistencies involved likelihood

estimates that were very close to 50%. The correlations between recommendations to admit or deny and

the corresponding departmental likelihood estimates ranged from .79 to .89 for the several kinds of

ratings made by study participants.

Effect of GRE Writing Assessment Scores and Presence of Essays in Folders

The most straightforward test of the effect of the presence of GRE Writing Assessment essays in

the application folders is based on data from a subset of study participants (four psychology faculty and

seven history faculty) whose admissions folders contained only GRE Writing Assessment scores, and

who were asked to review the actual essays separately in a subsequent, decision-making step. Thus, this

l2 This index may be only a rough surrogate for consistency, as a given faculty member may appear to be inconsistent by this criterion when in fact he or she is only exhibiting standards that differ from those of other faculty members.

21

sample made two sets of admissions decisions: the first set based on only GRE Writing Assessment

scores and the second set based on those scores plus the essays from which the scores were derived.

Overall, 84% of these 297 admissions decisions (27 applicants times 11 faculty) remained the same

regardless of whether or not essays were available. A total of 5% of the 297 decisions changed from

“deny” to “admit,” and 11% changed from “admit” to “deny.” More than a fourth of the total number of

changes from “admit” to “deny” came from one history faculty member, whose changes were all in the

same direction. Individually, faculty members’ decisions remained the same 67% to 96% of the time,

regardless of whether essays were available. The median correlations between estimates of the likelihood

of admission under the two conditions were 38 for psychology faculty and .77 for history faculty.

Finally, changes in admissions decisions were examined for each applicant to ascertain whether

some applicants either suffered or benefited when their essays were available. There was no detectable

tendency for changes to be more often in one direction than the other (e.g., from deny to admit) with any

consistency for any of the 27 hypothetical applicants.

The more formal regression analyses are also revealing with respect to the effect of GRE essays

and other factors. Table 2a and Table 2b show for history faculty and for psychology faculty,

respectively, the influence of various factors on admit/deny recommendations and on likelihood

estimates when they were asked first to consider only information from the GRE Writing Assessment --

not any other pre-admissions variables -- in making their judgments. (We have included other admissions

variables in this analysis mainly only to ascertain the extent to which participants did, as instructed,

disregard this information when making their first pass through the folders. Some contribution to the

multiple R would be expected from these variables, even if not considered by faculty, because of their

association with GRJ3 writing scores.)

Our first observation from the analyses presented in Table 2a through Table 7b is that, for each

discipline, the logistic regression analyses using admit/deny recommendations as the dependent variable

(shown in the top half of each table) and the linear regression analyses using likelihood estimates as the

dependent variable (shown in the bottom half of each table) produced quite similar results. For each

analysis, the institutional affiliation of participants accounted for a small portion of variance in both

admissions recommendations and likelihood estimates (from 5% to 20% for history and from 6% to 21%

for psychology). This suggests that the departments represented in the study are reasonably, though not

completely, homogeneous with respect to their overall admissions standards.

22

From Tables 2a and 2b we see that, for both kinds of decisions, the greatest influence (that is, in

terms of the portion of variance explained) was the applicants’ GRE Writing Assessment scores -- the

factor on which faculty were asked to focus for this analysis. The estimated portion for history applicants

was 40% and for psychology applicants, 53%. The next greatest portion was explained by applicants’

standing on the other pre-admission variables (because of their correlation with GRE writing scores).

The variable of primary interest -- whether or not GRE Writing Assessment essays were included

in admissions folders -- accounted for virtually none of the variance in either individual admit/deny

recommendations or in estimates of the likelihood of admission by the department. For psychology

participants, however, the 1% of variance that could be accounted for by the likelihood estimates was

statistically significant (E c .Ol). To put the significance of this influence in perspective, we note that the

availability of essays in the folders of psychology applicants resulted in an average likelihood of

admission that was about 7 percentage points lower than for applicants whose essays were not available.

In contrast, each additional point on the l-to-6 point GRE writing-score scale resulted in a nearly 25

percentage point increase in the probability of admission. The availability of GRE essays interacted

significantly @ < .05) with essay score level only for recommendations made by psychology faculty, but

accounted for only 1% of the variance.

A more realistic picture of the likely influence of the GRE Writing Assessment emerges from an

analysis of admit/deny recommendations and likelihood estimates made on the basis of aEZ information in

the admissions folders, not just information related to the GRE Writing Assessment. Tables 3a and 3b

show that, after explaining variation due to (a) differences among institutions and (b) the influence of

traditional admissions information included in folders, GRE writing scores explained a statistically

significant, but small, proportion of variance in the decisions made. Estimates were 3% to 4% for history

and 6% to 10% for psychology.

The presence of essays in folders was a significant influence in only one of the analyses

(accounting for 2% of the variance) -- for the estimates of likelihood provided by psychology faculty.

Again, this result translated, on average, to a 7-percentage-point lower likelihood of admission when

essays were available. (The extent to which this effect may be the result of our insertion of errors into

some essays is addressed below.) When considered with all other pre-admissions variables, the influence

of GRE writing scores decreased substantially, with a one-point increase in GRE writing scores raising

the average likelihood of admission by about 6 percentage points. Again, there was some indication for

23

psychology of an interaction between the availability of essays and GRE writing-score level. Again,

however, the effect was small, explaining only 1% of the variance @ < .OS).

Tables 4a and 4b provide the results when faculty were asked to make a second set of judgments

for a “new” set of applicants for whom only abbreviated admissions folders, but no GRE essays, were

provided. The influence of GRE writing scores above and beyond other admissions information was

again quite modest, accounting for about 1% to 3% of the variance in faculty decisions.

Table 2a. Hierarchical Regression Analysis for All History Faculty Considering GRE Writing Assessment Only

Explanatory variable(s) Cumulative !L2 (or ,R” )

Increase in BL’ (or ,R” )

x2 (or IJ for increase df

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

.15

.30

.70

.70

.70

.lO

.25

.65

.65

.65

Admit/deny recommendations

.15 72.6**

.15 74.g***

.40 192.6***

.OO 2.1

.OO 0.0

Likelihood of admission estimates

.lO 5 . 1***

.15 6 . 8***

.40 356.3***

.oo 0.7

.OO 0.2

7, 316

9, 307

1, 306

1, 305

1, 304

L&,2 is a logistic regression analog to B2 in ordinary least squares regression analysis. b2 Note. represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

** p < .Ol. *** p < .OOl.

24

Table 2b. Hierarchical Regression Analysis for All Psychology Faculty Considering GRE Writing Assessment Only

Explanatory variable(s) Cumulative h2 (or B2)

Increase in _RL2 (or B2)

x2 (or FJ for increase df

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

.06

.31

.84

.85

.86

.07

.25

.78

.79

.79

Admit/deny recommendations

.06 20.2* *

.25 84.2***

53 179.9***

.oo 1.6

.Ol 4.9*

Likelihood of admission estimates

.07 3.6**

.18 6 l*** .

.53 539.5***

.Ol 9.7**

.oo 1.6

5, 237

9, 228

1, 227

1, 226

1, 225

Note. h2 is a logistic regression analog tog2 in ordinary least squares regression analysis. a2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRJZ verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. ** p < .Ol. ***p < .OOl.

25

Table 3a. Hierarchical Regression Analysis for All History Faculty Considering All Information

Explanatory variable(s) Cumulative _RL~ (or ,R2 )

Increase in BL~ (or ,R2 )

x2 (or E) for increase

Department .18

Admissions variables’ .48

GRE writing .51

Availability of essays .51

GRE writing x availability .51

Department .13

Admissions variables’ .58

GREI writing .62

Availability of essays .62

GRE writing x availability .62

Admit/deny recommendations

-18 96.4***

.30 154.9***

.03 16.9***

.oo 0.0

.oo 0.4

Likelihood of admission estimates

.13 6 . 6***

.46 44.0***

.04 35.4***

.OO 0.3

.OO 0.0

8, 368

9, 359

1, 358

1, 357

1, 356

Note. l&2 is a logistic regression analog to R2 in ordinary least squares regression analysis. b2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

*** p c .OOl.

26

Table 3b. Hierarchical Regression Analysis for All Psychology Faculty Considering All Information

Explanatory variable(s) Cumulative h2 (or B2)

Increase in h2 (or _R2)

x2 (or IJ for increase

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

Department

Admissions variables’

GRE writing score

Availability of essays

GRE writing score x availability

.17

.57

.66

.66

.66

.09

.53

.59

.61

.61

Admit/deny recommendations

.I7 57.6***

.39 132.9***

JO 33.1***

.oo 0.1

.oo 0.0

Likelihood of admission estimates

.09 4 . 9***

.43 23.3***

.06 32.2***

.02 11.5***

.Ol 4.4*

5, 237

9, 228

1, 227

1, 226

1, 225

L2 Note. L2 is a logistic regression analog to B2 in ordinary least squares regression analysis. represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. *** p < .OOl.

27

Table 4a. Hierarchical Regression Analysis for All History Faculty for Abbreviated Applications

Explanatory variable(s) Cumulative Increase in h2 (or _R2) k2 (or _Rz)

x2 (or FJ for increase Df

Department .16

Admissions variables’ .53

GRE writing score .55

Admit/deny recommendations

.16 86.2***

.37 193.6***

.Ol 7.0**

Department .19

Admissions variables’ .55

GRE writing score .56

Likelihood of admission estimates

.19 10.8***

.36 32.2***

.Ol 7.2**

8, 369

9, 360

1, 359

h2 Note. l&2 is a logistic regression analog to R2 in ordinary least squares regression analysis. represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

** p c .Ol. *** p < .OOl.

28

Table 4b. Hierarchical Regression Analysis for All Psychology Faculty for Abbreviated Applications

Explanatory variable(s) Cumulative b2 (or _R2)

Increase in I&2 (or _R2)

II” (Or 1) for increase

Department .lO

Admissions variables’ .49

GRE writing score 53

Admit/deny recommendations

.lO 34.5***

.39 131.1***

.03 11.7***

Department .21

Admissions variables’ .62

GRE writing score .62

Likelihood of admission estimates

.21 12.3**” 5, 237

.41 27.5*** 9, 228

.Ol 3.3 1, 227

Note. L&2 is a logistic regression analog toR2 in ordinary least squares regression analysis. h2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

*** p < .OOl.

Effect of Construct-Irrelevant Flaws in Essays

In an effort to understand the influence of the flaws we introduced into the essays, we conducted

several analyses of data from faculty whose folders contained the applicants’ GRE essays. The main

focus of these analyses was the influence on admissions decisions of irrelevant flaws in GRE essays, and

whether these flaws, if influential, might play a greater role for strong essays than for weak ones (or vice

versa).

Tables 5a and 5b show that when judgments were based only on the GRE Writing Assessment,

the prevalence of errors explained none of the variance in the decisions of psychology faculty, and only

about 2% of the variance in the decisions of history faculty. The latter results were significant (p c .OS)

only for likelihood estimates. These tables also show that, in each analysis, the interaction of GRE

writing-score level with the prevalence of errors accounted for little if any variance. The effects for

29

estimates of the likelihood of departmental admission are depicted in Figure 2 and Figure 3. Also

included in these figures are the likelihood estimates that were made when essays were not available.

Similar results were found when faculty based their judgments on all information in admission

folders (see Table 6a and Table 6b) . The prevalence of errors in essays accounted for a small,

statistically significant @ < .OS) portion of the variation in admit/deny recommendations (about 3% for

history and 4% for psychology), but none of the variation in likelihood estimates. The effects are such

that the presence of extra errors decreases the likelihood of admission somewhat, regardless of the

quantity of added errors (some or many).

We repeated the analysis for data from faculty who initially did not see essays in their

admissions folders, but who instead evaluated the essays as a separate step, after they had already

reviewed the admissions folders. Table 7a and Table 7b show that the prevalence of errors under these

circumstances had no statistically significant impact on either admit/deny recommendations or likelihood

estimates. The lack of any large, consistent effect is illustrated in Figure 4 and Figure 5. (We note that in

the analysis of admissions recommendations made by psychology faculty, about 8% of the variance was

attributed to the interaction of GRE writing-score level and the presence of errors. Although statistically

significant, this estimate may not be very trustworthy, as the analysis did not seem to “behave” in a

reasonable manner with respect to how it converged on this estimate.)

30

Table 5a. Hierarchical Regression Analysis for History Faculty Based on GRE Writing Assessment Only (Essays Available in Folders)

Explanatory variable(s) Cumulative RJ~ (or _R2)

Increase in k2 (or _R2)

x2 (or E) for increase Lf

Admit/denv recommendations

Department .20 .20 43.9*** 5

Admissions variables’ .36 .16 36.1*** 9

GRE writing score 73 .37 83.5*** 1

Prevalence of errors .75 .02 5.2 2

GRE writing score x errors 75 .oo 0.0 2

Department .05

Admissions variables’ .21

GRE writing score .65

Likelihood of admission estimates

.05 1.5

.I7 3 . 5***

.43 178.5***

5, 156

9, 147

1, 146

Prevalence of errors .66 .02 3.3* 2, 144

GRJS writing score x errors .68 .Ol 3.1* 2, 142

Note. l&2 is a logistic regression analog to 8’ in ordinary least squares regression analysis. h2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. *** p c ,001.

31

Table 5b. Hierarchical Regression Analysis for Psychology Faculty Based on GRE Writing Assessment Only (Essays Available in Folders)

Explanatory variable(s) Cumulative I&2 (or B2)

Increase in h2 (or B2)

x2 (or FJ for increase df

Department

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

Department

Admissions variables 1

GRIZ writing score

Prevalence of errors

GRE writing score x errors

.08

.33

.95

.96

.96

.ll

.28

.82

.82

.82

Admit/deny recommendations

.08 14.5**

.25 47.4***

.62 116.6***

.OO 0.1

.oo 0.1

Likelihood of admission estimates

.ll 4.2**

.I7 3.2**

.54 364.8***

.OO 0.3

.OO 0.2

4, 130

9, 121

1, 120

2, 118

2, 116

Note. L&2 is a logistic regression analog toFJ2 in ordinary least squares regression analysis. L&2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p c .05. ** p < .Ol. *** p c.001.

32

100

I 80

60

Mean Likelihood 50

A.. AdmPsfsion 40

30

20

I 10

I 0 2 2.5 3 3.5 4 4.5 5 5.5 6

GRE Writing Assessment Score

Figure 2. Mean likelihood of admission on the basis of the GRE Writing Assessment only, by score level, availability of essays, and prevalence of errors (psychology faculty).

100

90

80

70

Mean Likelihood 60

of Admission 5O

40

30

20

10

0. .

2 2.5 3 3.5 4 4.5 5 5.5 6 GRE Writing Assessment Score

Figure 3. Mean likelihood of admission on the basis of the GRE Writing Assessment only, by score level, availability of essays, and prevalence of errors (history faculty).

33

100

90

80

70 Mean

Likelihood 60 of

Admission 50

40

30

20

10

0

2 2.5 3 3.5 4 4.5 5 5.5 6

GRE Writing Assessment Score

Figure 4. Mean likelihood of admission on the basis of the GRJZ Writing Assessment only (essays evaluated separately), by score level and prevalence of errors (psychology faculty).

100

90

80

70 Mean

Likelihood 60 of

Admission 50

40

30

20

10

0

2 2.5 3 3.5 4 4.5 5 5.5 6

GRE Writing Assessment Score

Figure 5. Mean likelihood of admission on the basis of the GRE Writing Assessment only (essays evaluated separately), by score level and prevalence of errors (history faculty).

34

Table 6a. Hierarchical Regression Analysis for History Faculty Based on All Information (Essays Available in Folders)

Explanatory variable(s) Cumulative &2 (or B2 )

Increase in I&2 (or _R2)

x2 (or IJ for increase Lf

Department

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

Department

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

.15

.40

.44

.47

.48

.07

51

56

.57

.57

Admit/deny recommendations

-15 39.5***

25 64.7***

.04 10.5**

.03 8.9”

.Ol 2.0

Likelihood of admission estimates

.07 2.4*

.43 16.7***

.06 22.2***

.OO 0.4

.Ol 1.6

6, 181

9, 172

1, 171

2, 169

2, 167

Note. k2 is a logistic regression analog toE2 in ordinary least squares regression analysis. F&2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. ** p < .Ol. *** p < .OOl.

35

Table 6b. Hierarchical Regression Analysis for Psychology Faculty Based on All Information (Essays Available in Folders)

Explanatory variable(s) Cumulative RJ~ (or B2)

Increase in l&2 (or _R2)

x2 (or 1) for increase Lf

Department

Admissions variables’

GRE writing score

Prevalence of errors

GREI writing score x errors

Department

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

.17

.65

.76

.79

.81

.06

.65

.74

.74

.75

Admit/denv recommendations

.17 32.3***

.48 89.6***

.ll 20.1***

.04 6.6*

.Ol 3.7

Likelihood of admission estimates

.06 2.2

58 22 . 4***

.09 42.9***

.OO 0.9

.oo 1.1

4, 130

9, 121

1, 120

2, 118

2, 116

L&2 is a logistic regression analog to _R2 in ordinary least squares regression analysis. I&2 Note. represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. *** p < .OOl.

36

Table 7a. Hierarchical Regression Analysis for History Faculty Based on GRE Writing Assessment Essays Only (Essays Available in Separate Packet)

Explanatory variable(s) Cumulative RJ~ (or _R2)

Increase in EL (or ,R” )

x2 (or IJ for increase ti

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

.30

.51

.53

Admit/deny recommendations

.30 77J3***

.22 56.4***

.02 5.3

not estimable

.15

.48

.48

.49

Likelihood of admission estimates

.15 3 . 4***

.33 114.5***

.OO 0.1

.oo 0.6

9, 178

1, 177

2, 175

2, 173

Note. &’ is a logistic regression analog toR2 in ordinary least squares regression analysis. k2 represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRJZ quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

*** p c ,001.

37

Table 7b. Hierarchical Regression Analysis for Psychology Faculty Based on GRE Writing Assessment Essays Only (Essays Available in Separate Packet)

Explanatory variable(s) Cumulative RJ~ (or _R)

Increase in I&2 (or g2)

x2 (or E) for increase lrlf

Admissions variables’ .28

GRE writing score .60

Prevalence of errors .62

GRE writing score x errors .70

Admissions variables’

GRE writing score

Prevalence of errors

GRE writing score x errors

.15

.61

.62

.62

Admit/deny recommendations

.28 42. I***

.32 47.2***

.02 3.5

.08 12.5**

Likelihood of admission estimates

.15 2.0*

45 112.3***

.Ol 1.0

.oo 0.6

9, 98

1, 97

2, 95

2, 93

k2 is a logistic regression analog to B2 in ordinary least squares regression analysis. L&2 Note. represents the proportional reduction in the value of the log-likelihood coefficient when alternative models are compared.

’ GRE verbal score, GRE quantitative score, GRE analytical score, overall undergraduate GPA, undergraduate GPA in major, summary recommendation, overall rating of promise, strength of personal statement, and selectivity of undergraduate school.

* p < .05. ** p < .Ol. ***p < ,001.

38

Influence of Other Pre-Admission Factors

As stated at the outset, a secondary purpose of the study was to ascertain, within the limits of our

data, the role of traditional pre-admission information in the graduate admissions process. For individual

faculty members and for aggregates of all history faculty and all psychology faculty, Table 8a and Table

8b provide the standardized regression weights for admissions judgments made on the basis of both

complete and abbreviated applications. Because of the relatively small sample of applicants, the weights

for individual faculty are not highly consistent across the two sets of applications. Median correlations

(over individual participants) between weights computed for pre-admission factors based on judgments

of complete and abbreviated applications were .52 for psychology faculty and .47 for history faculty.

Thus, individually, participants were at least modestly consistent in their decision making. (Part, and

perhaps most, of the apparent inconsistency is, of course, the result of our presenting slightly different

information and presenting it in a different format in the two types of applications.)

Although the study was not designed to determine the decision-making strategies of individual

faculty, the data do provide some sense of the influence of each admissions criterion on faculty

judgments about applicants. (The instability of the regression coefficients must be borne in mind,

however.) For example, with relative consistency, the decisions of history faculty participants 2,3,6, 8,

12, and 13 are relatively strongly related to GRE verbal scores. The decisions of history faculty member

4 are related to the quality of personal statements, and the decisions of participant 11 relate strongly to

faculty ratings of overall promise. Such differences are equally apparent among psychology faculty.

Thus, if our results are any indication, there is apparently considerable variation among individual faculty

__ often within the same department -- with respect to the importance they attach to various kinds of pre-

admission information.

In the aggregate, the judgments made by history faculty in our study correlated most strongly

with GRE verbal scores. Those made by psychology faculty correlated relatively strongly with GRE

verbal scores, overall undergraduate GPA, and ratings of overall promise. Thus, if our study procedures

bear any resemblance to actual admissions procedures at participating departments, then GRE verbal

scores in particular would seem to be an important factor in the admissions decisions at these

departments.

39

Table 8a. Standardized Regression Weights for Prediction of Likelihood Estimates for History Faculty

Variable

Indiyidual faculty participants Application

type 1 2 3 4 5 6 7 8 9 10 11

GRE verbal score

GRE quantitative score

GRE analytical score

Undergraduate GPA -- overall

Undergraduate GPA -- major

% Summary recommendation

Rating of overall promise

Personal statement

Selectivity of under- graduate school

GRJZ writing score

Multiple R

Complete 29 63’ Incomplete 38a 75c

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete 51 10 Incomplete 27 36

Complete 10 00 Incomplete 16 07

Complete Incomplete

Complete Incomplete

Complete 88 95 88 Incomplete 89 76 89

-11 10 -23 08

01 08 13 -26

22 10

42a 02

-00 -01 -10 -03

-30 -01 -19 -12

28 14 12 39a 11 28a

42b 24’ 17 20 -32 06

39a 46b

07 -09

07 14

21 41a

13 -03

23 -34

00 40

03 08

-07 79= 53’ 16 73’ 24 28 18 63’ 61’ 99c 5oc

16 -21 -13 33 -02 -04 -05 -16 10 -03 -16 04

-08 -00 04 14 -16 24 -13 29 02 -19 -06 07

31 -16 01 -03 -11 00 -08 19 01 04 -03 01

-35 17 19 04 28 -08 22 -06 17 08 02 -21

02 -21 -49 14 -18 17 -34 -34 07 -11 01 05

17 35 59a 04 32 -14 45 20 10 47a 10 22

61’ 15 25a 01 -06 12 76c 17 25a 2ga 17a 11

30 24a 08 14 39c 21 -09 24 15 3oa 11 16

-01 07 33b 39b -03 53c -13 4sb -15 -24a -06 41’

85 92 91 85 90 89 86 82 91 91 96 95

05 15

2oa -11

09 36

34c 28

01 -21

3oa 35

23 -22

00 14

11 22

2gc 27

97 81

-20 08

-14 -23

26 -01

31a 36b

-00 -12

-21 -23

7gc 75c

15 3sc

19 33’

34c 04

94 94

Note. Weights greater than .30 in absolute value appear in bold.

ap < .05; bp < .Ol; cp < .OOl

Table 8b. Standardized Regression Weights for Prediction of Likelihood Estimates for Psychology Faculty Participants

Variable

Individual faculty participants Application All

type 1 2 3 4 5 6 7 8 9 faculty

GRE verbal score Complete Incomplete

-08

01

GRE quantitative score Complete 05

Incomplete -24a

GRE analytical score

Undergraduate GPA -- overall

Undergraduate GPA -- major

Summary recommendation

Rating of overall promise

Personal statement

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete Incomplete

Complete

Incomplete

Complete Incomplete

37= 39c

21 17

-17 12

11 11

32

37a

Selectivity of under-

graduate school

GRE writing score 39c 02

Multiple R Complete 96

02 22

04 08

41b -09

11 08

-12 17

06 07

04 20

08 18

14 14

60’

52c

93 90

21

2sc

07 06

08 -04

7oc 77c

-01 -14

15 -28

-18

4sc

05

09

13

23’

16 13

95 97

14

13

08 23

43a -05

53b 9oc

-20 -36a

-22 -07

29 23

07

25a

14

-00

03 24 08 -00

87 91 Incomplete 96

Note. Weights greater than .30 in absolute value appear in bold.

ap < .05; bp < .Ol; cp < .OOl

3Sa

62c

51b 20

-14

-01

05 24

-09 -00

-18 -12

20 08

-07 20

17 12

86 90

-07

26a

33a 12

32a 32b

23

37b

-15 03

-20 -25

54a 35

-05

21a

11

13

33b 02

91 95

41b

6gc

42b

36’

-16

-13

19 05

01 11

14 -22

-09

49a

-03 05

12

-03

34b -16

92 93

06 73’

32a 79’

-01 19

26 33

28a -53a 05 -49b

58’ 25 32 -11

-18 -00 -34a 11

-05 -55 -26 -22

21 61 42 49

13 12

48’ -06

23a 06 -01 05

34c 04 25a 06

94 81 91 87

19c 34c

16b 14a

11 -02

3oc 27’

-10 -02

-08 -12

21a 32’

06

17c

14c 09

26’ 09a

74 81

As the final step of our study, we asked participants to rate, on a O-to-5 scale, the importance they

place on each type of pre-admission information contained in our admission folders. Table 9a shows the

participants’ ratings, as well as the relative weights that they assigned to each factor. (Table 9b and Table

9c display faculty ratings of the importance of the personal qualities that were listed in our fictitious

recommendations and personal statements, respectively.) Here, too, we see considerable variability

among participating faculty. However, these results differ somewhat from the results of our analysis of

the relationship between faculty recommendations and applicants’ standing on pre-admission variables.

That is, faculty reports of the importance of various admissions factors do not correspond precisely with

the actual weights that these factors receive in the admissions process. This difference may be the result,

in part, of both the correlations among the admissions factors included in the regression analyses, and the

incomplete correspondence between information included in admissions folders and the factors that

faculty were asked to rate. It may also suggest, however, that, to some degree, faculty perceptions of

importance do not fully reflect the actual weights that the factors receive. This, of course, is policy

capturing’s raison d ‘h-e.

42

Table 9a. Mean Faculty Ratings and Weightings of Importance of Admissions Factors

Factor Psychology History M SD M SD

GRE verbal score ,,,,,..,...,..~I....II....1..........I.. Rating Weight

GRE quantitative score .,...............I............. Rating Weight

GRE analytical score ,,,,...,.,...,,................... Rating Weight

GRE Subject Test score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rating Weight

Undergraduate GPA (overall) . . . . . . . . . . . . . . . . . . . . . Rating Weight

Undergraduate GPA (in major). . . . . . . . . . . . . . . . . . . Rating Weight

Undergraduate GPA (final two years) . . . . . . . . Rating Weight

Undergraduate course of study . . . . . . . . . . . . . . . . . . . Rating Weight

Quality of undergraduate institution . . . . . . . . . . . Rating Weight

Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rating Weight

Personal statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rating Weight

Interview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rating Weight

Writing samples . . . ..a.................................... Rating Weight

4.2 (O-6) 47 (22) 4.2 (0.6) 47 (22) 2.6 (1.6) 30 03) 2.6 (1.1) 30 (23)

4.1 (0.6) 48 (26)

4.0 (0.9) 48 (29)

4.2 (0.7) 59 (31)

3.3 (0.7) 32 (20)

3.3 (0.8) 40 (27)

3.9 (0.9) 49 (21)

3.1 (0.9) 31 (22)

3.1 (1.7) 29 (16)

2.7 (1.6) 29 (13)

4.4 (O-6) 61 (46)

2.5 (1.3) 25 (12)

3.5 (0.9) 40 (19)

0.4 (1.1) 10 (4)

3.3 (1.4) 51 (52)

3.8 (0.9) 53 (51)

3.8 (0.9) 52 (46)

3.9 (0.8) 52 (46)

3.7 (0.9) 55 (52)

4.1 (0.8) 53 (41)

4.6 (0.6) 76 (72)

0.3 (0.8) 13 (16)

4.0 (1.4) 78 (76)

Note. Ratings and weights were provided by 9 psychology faculty and by 13 history faculty. Ratings were provided on a O-to-5 scale with 5 = extremely important, 4 = very important, 3 = moderately important, 1 = of little importance, and 0 = not considered at all. Respondents were asked to give a weight of 10 to the least important factor. There was no limit at the upper end of the scale.

43

Table 9b. Mean Faculty Ratings of Importance of Information Available From Recommendations

Information Psychology History

Academic preparedness for proposed program... M SD

Intellectual ability/capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Aptitude (skill or potential) for research . . . . . . . . . . . . M SD

Teaching ability/potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Ability to communicate orally . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Ability to communicate in writing . . . . . . . . . . . . . . . . . . . . . . M SD

Creativity/originality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Independent thinking . . . . ..*.................................... M SD

Dependability/reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Initiative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Seriousness of purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Professional expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Character/integrity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Promise of productive scholarship . . . . . . . . . . . . . . . . . . . . . . M SD

Emotional maturity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Motivation for proposed program of study.. . . . . . . . M SD

3.8 (0.4)

4.6 (0.5)

4.3 (0.7)

2.8 (0.9)

3.7 (0.7)

4.2 (0.4)

3.7 (0.8)

4.2 (0.4)

3.8 (0.6)

4.0 (0.7)

4.0 (0.7)

2.7 (0.9)

4.1 (0.7)

3.9 (0.6)

3.6 (0.5)

4.2 (0.4)

3.9

(0.9)

4.9

(0.3)

4.6 (0.6)

3.5 (0.7)

3.9 (1.0)

4.8 (0.4)

4.1 (0.7)

4.4 (0.6)

3.9 (0.7)

3.9 (0.8)

4.2 (0.8)

3.0 (0.9)

2.9 (1.6)

4.2 (0.8)

3.0 (1.4)

4.1 (1.0)

Note. Ratings were provided by 9 psychology and 14 history faculty. Ratings were provided on a O-to-5 scale with 5 = extremely important, 4 = very important, 3 = moderately important, 1 = of little importance, and 0 = not considered at all. Respondents were asked to give a weight of 10 to the least important factor. There was no limit at the upper end of the scale.

44

Table 9c. Mean Faculty Ratings of Importance of Information Available From Personal Statements

Information Psychology History

Match of applicant’s interests with those of prospective advisors . . ..I...................... M SD

Match of applicant’s professional aspirations with program objectives . . . . . . . . . . . . . . . . . . M SD

Applicant’s rationale for graduate study . . . . . . . . . . . . . . . . . . . . . . . ..I........................................ M SD

Likely commitment to the field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..I................................... M SD

Evidence of motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Evidence of maturity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Evidence of other personal characteristics likely to facilitate degree completion . . . M SD

Degree to which applicant has overcome prior obstacles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Evidence of relevant experiences, special interests, or achievements . . . . . . . . . . . . . . . . . . . . . M SD

Depth of knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

Evidence of a “voice” (personality, style, etc.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . M SD

4.1

(0.6)

4.2 (0.6)

3.6 (1.0)

3.9 (0.6)

4.3 (0.7)

3.9 (0.9)

4.1 (0.6)

2.7 (1.1)

3.1 (1.4)

3.3 (0.7) 2.4

(1.1)

4.4 (0.7)

4.2 (0.9)

4.4 (0.6)

4.1 (0.9)

4.1 (0.7)

4.0 (0.8)

3.4 (1.0)

2.9 (0.7)

3.5 (0.8)

3.4 (0.6) 3.7

(0.7)

Note. Ratings were provided by 9 psychology and 14 history faculty. Ratings were provided on a O-to-5 scale with 5 = extremely important, 4 = very important, 3 = moderately important, 1 = of little importance, and 0 = not considered at all. Respondents were asked to give a weight of 10 to the least important factor. There was no limit at the upper end of the scale.

45

The Fidelitv of Our Simulation

Because it was impossible to faithfully simulate all aspects of actual graduate admissions

procedures, we asked participants to tell us how, if at all, our study procedures (admissions folders, etc.,

for instance) differed from the admissions procedures that they normally use, and how these differences,

if any, might have affected the “admissions decisions” they made for our study, relative to the decisions

they routinely make. In their responses, participants indicated that, when compared with actual

admissions procedures, our materials and methods were “similar,” “very similar,” “pretty close,” “had no

significant differences,“or”contained most of the information needed.”

At the same time, some participants identified ways in which our procedures differed from actual

graduate admissions. For example, one participant noted that the procedures used by his department were

“more straightforward” than our procedures, and another mentioned the need to adjust her thinking for

our “applicants” because none of them possessed masters degrees. The difference cited most often -- by

about a third of the participating faculty -- was the lack of actual transcripts or information about

performance in specific, critical undergraduate courses. About the same number of faculty observed at

least one of the following differences: the lack of written letters of recommendation, the lack of multiple

recommendations, or the lack of knowledge of the recommendation writer. Four or fewer participants

mentioned: (a) sources (such as interviews, writing samples, and actual personal statements), (b)

information (about computer skills, language proficiency, research experience, and ethnic identity, for

instance), or (c) opportunities (for example, to the opportunity to contact applicants or their advisors

personally) that our admissions process did not provide.

Several faculty commented on what they regarded as inconsistencies in our admissions folders --

for example, too many recommendations that were either too qualified or too candid (in contrast to actual

recommendations, which are, invariably, glowing), and applicants from highly selective schools who had

only average GRE scores. Several faculty also stated that their departments require all applicants to

possess a master’s degree before being considered for the doctoral program; none of our applicants did.

Finally, several also reported that their department’s admission process occurs in stages; typically, all

information is not considered simultaneously.

Despite these differences, most participants felt that, even if they had followed their

departments’ procedures instead of ours, their admissions decisions and estimates would have been about

the same. In response to our question about the likely effect of differences between their admissions

procedures and those used in our study, typical comments were:

Not a great deal.

My decisions weren’t aflected (except that I may have rejected some applicants whose advisors might have convinced me that my concerns were unfounded).

The deviation would only very occasionally a#ect my decisions.

Probably very little efect.

Made little dtrerence.

On the other hand, some expressed more serious concerns:

I had little confidence in my decisions without interview data.

Applicants with weak credentials might have been accepted if they had had strong clinical or research experience.

A4y responses were thrown ofsby the unrealistic character of the folders (they didn’t look “normal” to me).

Discussion

The overarching aim of this study was to ascertain the likely role of the GRE Writing

Assessment in graduate admissions decisions. A more specific objective was to determine the impact that

examinees’ GRE test essays would have on these decisions if the essays were made available to test-

score users during the admissions process. A secondary goal was to simultaneously assess the influence

of more traditional admissions criteria on the graduate-school application process.

Analyses revealed that scores from the GRE Writing Assessment accounted for a statistically

significant, but small, portion of the variation in faculty decisions above and beyond that explained by

traditional pre-admissions measures, and that faculty decisions were strongly related to applicants’

standing on traditional pre-admissions measures, such as GRE General Test scores, undergraduate

grades, faculty recommendations, and personal statements. Thus, if our study participants are at all

representative of potential users of the GRE Writing Assessment, it is likely that GRE writing scores will

play some role in graduate admissions decisions. However, while there is, of course, no guarantee that

every admissions decision will be unaffected by the impression that an applicant’s essay makes, our

results also suggest that the availability of applicants’ GRE essays will probably have little additional

influence on admissions decisions beyond that wielded by the writing scores themselves.

An additional analysis suggested that the presence of incidental writing errors in the actual test

essays sent to admissions departments are not likely to impress graduate faculty differently than similarly

scored essays that do not contain such errors. While noticeable, such errors are unlikely to hinder

communication (and are therefore accorded little weight in the operational scoring of GRE essays);

accordingly, this analysis revealed only a slight (and not entirely consistent) influence of these errors in

relation to the influence of applicants’ GRE Writing Assessment scores. There was a slight indication

that these errors may exert a slightly greater, negative influence on strong essays than on weaker ones.

We note, however, that in order to detect this effect, we had to insert either “some” or “many” errors into

the essays, thus creating an artifice for the purposes of this study. It has been our experience to date that,

in actual practice, GRE test takers who are skillful enough to earn high writing scores rarely write essays

with so many errors. Graduate applicants who are able to discuss complex ideas in an organized, well-

developed essay tend not to make the abundance of trivial errors that we inserted into essays for this

study. Thus, by using extreme examples of the kinds of essays that admissions staff might see if essays

were released, this study has perhaps presented a worst-case scenario.

Limitations

The first limitation of our study is characteristic of many research efforts. By providing material

that described the focus of our study -- that is, the GRE Writing Assessment -- we may have called extra

attention to the measure, thereby possibly elevating its importance in the hypothetical admissions

decisions made by study participants. This limitation was unavoidable, however, as it was necessary to

acquaint participants with the GRE Writing Assessment in order to allow its consideration in the

decision-making process.

Second, as with all simulations, the judgment process that we studied was necessarily somewhat

artificial. Although faculty participants reviewed admission folders that were developed from actual

application materials, they did not make real admissions decisions. The “admissions process” that we

employed was, undoubtedly, different in a variety of ways from the complex and varied processes that

graduate admissions committees typically employ. For instance, our applicant pool was not tailored

precisely to each department, and we did not necessarily provide study participants with all of the

information (or in the same form) that they typically review when making real admissions decisions.

48

Instead, we asked participants to make simplifying assumptions about both the admissions process and

the “applicants” they were asked to evaluate. For example, graduate admission decisions are usually the

work of committees, not of individual decision makers, as our study procedures suggest. (Along these

lines, one of our reviewers suggested that that the deliberations of admissions committees might in effect

serve as further training on the appropriate use of the GRE Writing Assessment, thereby resulting in even

fewer misreadings of applicants’ essays than we found here.) Moreover, we tacitly assumed a fully

decentralized model of graduate admissions, instead of a more centralized model in which baseline

standards and procedures are established on a university-wide basis (Council of Graduate Schools, 1993).

A further simplification, as is typical of many policy capturing studies, was the use of ordinary

linear regression to model the judgments of decision makers. It is well known, however (and confirmed

by study participants), that information is not always combined, and decisions not always made, in a

simple linear fashion. A rather extensive literature suggests, however, that despite the complexity of

human decision making, many kinds of decisions can be accurately represented by simple linear models

(see, for example, Dawes, 1979; Dawes & Corrigan, 1974; Dawes, Faust, & Meehl, 1989; and Wiggins,

1973, chapter 4).

The validity of our results also depends in part on how representative the “extraneous errors”

used in our study were of the errors that we can expect to find in actual GRE essays. Most of the errors

that we introduced into essays were relatively minor ones -- spelling or typing errors, minor grammatical

mistakes, or careless misstatements of fact. Introducing more serious errors in grammar or syntax might

have had a greater impact on “admissions decisions.” However, it is likely that essays containing these

more serious errors would also have received lower GRE scores. Thus, any devaluation by faculty of the

writing exhibited in these essays would probably have been appropriate.

In addition, besides errors we did not investigate other characteristics that might influence

perception of GRE essays. For example, we did not consider whether such factors as distinct cultural

dialects might sway (or bias) faculty perceptions. From one perspective, however, a concern about

dialects may be largely unwarranted. During our inspection of a relatively large number of GRE essays,

we found many essays that contained examples of international English (British spelling, non-American

phrasing, and so on) and references to foreign homelands, but we encountered virtually none that used

cultural or regional dialect. Most applicants to graduate school, it seems, choose to write their essays in

what might be called formal, academic English (standard American or International English.) Thus, we

have some evidence that GRE examinees do not frequently employ dialects in their test essays. We do

49

not, however, know how the apparently rare use of such dialects in GRE essays may affect graduate

admissions decisions.

Finally, our study was restricted to only two disciplines, which surely do not adequately

represent all possible graduate fields. As noted earlier, psychology and history were selected because

these two disciplines typically require relatively extensive writing of students, and because we were

advised that departments in both of these disciplines are interested in adding GRE Writing Assessment

scores to the information that they consider for graduate admissions. Because we do not know the extent

to which our study results apply more widely, it would be appropriate, we think, to extend the study to

other disciplines, since a multi-disciplinary advisory committee approved the measure as a means of

assessing writing skills that are important in “most” disciplines.

Imnlications

It is clear that our study has not fully addressed all of the potentially important aspects of the

problem we have set forth. However, despite the various limitations mentioned here, we believe that our

study possessed a reasonable degree of external validity for answering the primary questions of interest.

The study’results have, we believe, significance on several levels -- foremost for informing decisions

about the GRE Writing Assessment, but also for understanding more fully both the graduate admissions

process and the role of GRE scores in this process. Our findings may also have some relevance to

performance assessment more generally.

A current, long-standing objective of the GRE Board is to better understand how its test offerings

facilitate and influence graduate admissions decisions. Toward this end, this study has provided some

additional information about the emphases, or relative weights, that departments in two disciplines place

on GRE scores and other traditional pre-admission information. The results contribute modestly, we

believe, to the existing body of knowledge concerning the use of GRE test scores in graduate admissions

(see, for instance, Oltman & Hartnett, 1985). Further, the study provides additional information about the

feasibility and utility of a set of procedures that could be employed for further study of graduate

admissions. Possibly, these procedures could be extended in the future to other disciplines, other

decision-making contexts (such as decisions about fellowships and financial aid), or any new measures

being considered for graduate admissions.

More broadly speaking, we believe the results also have implications for the type of score

reporting that is associated with performance assessment and other forms of constructed-response testing.

50

The general issue here is the extent to which the products or performances generated by test takers may

contain useful information that is not captured solely in summary evaluations of these products -- that is,

the test scores. For example, a potential employer might wish to view an applicant’s videotaped

performance in a certified technical trainer program; similarly, a school administrator might wish to read

a teacher-applicant’s test essay to ascertain the writer’s position on sensitive political, social, or moral

issues. Our study provides some clues, we believe, as to whether one such work product -- the GRE

Writing Assessment essay -- may be welcomed by graduate departments, and also whether any potential

for misuse or misinterpretation may be associated with releasing this product.

There are, however, to the best of our knowledge, no formal guidelines regarding the

circumstances under which it is appropriate to release examinees’ work products. According to the

recently published Standards for Educational and Psychological Testing (AERAlAF’AfNCME, 1999),

“Test scores, per se, are not readily interpreted without other information . . . ” (p.62). It is incumbent on

testing programs, therefore, to provide a variety of interpretive materials that describe, for instance, what

a test covers, what test scores mean, and how scores should (and should not) be used. The standards are

silent, however, with respect to whether examinees’ test responses should be included in these materials.

Increasingly, as test users become more and more informed about testing issues, they are calling

for more information than is contained in a single test score. For some kinds of assessments, test

responses may help to meet test users’ demand for more comprehensive information. It seems clear,

however, that if examinee responses -- essays for example -- are to be provided to test users, it would be

prudent to assess not only the potential value of this information, but also the likelihood that it will be

misused.

Our study detected little evidence of inappropriate (construct-irrelevant) judgments when

examinee essays were released to graduate faculty. Thus, for one large-scale test -- the GRE Writing

Assessment -- making applicants’ responses available to admissions staff will probably not give rise to

abuse, at least with respect to the likelihood that graduate faculty will focus unduly on inappropriate

features of applicants’ writing. From this perspective, therefore, reservations about releasing test takers’

essays -- that test score recipients may focus on trivial errors, apply inappropriate standards, or react to

the essays in a biased or unfair way -- seem less compelling. There is little evidence in this study to

support such reservations: Even without formal training in evaluating GRE essays, the faculty who

participated in our study were not influenced excessively by noticeable but relatively trivial, construct-

51

irrelevant errors of the kind that GRE essay readers are trained to downplay when they evaluate the

overall quality of GRE essays.

Further Considerations

This study has provided some information about how graduate admissions personnel perceive

GRE writing scores when examinees’ essays accompany those test scores. However, if the GRE program

were to consider sending essays to graduate departments, additional questions would probably need to be

answered, and other factors should likely be considered, in order to decide the conditions -- if any --

under which these essays are ultimately released. For example:

l Should the release of essays be automatic, or should it be optional -- for example, only with the test taker’s permission? What are the implications of some candidates denying permission? When might it be to the writer’s advantage to release (or withhold) his or her essays?

l Should the essays be available to graduate departments only under certain conditions? For instance, should the GRE program provide training to test users, or require some assurance that test essays will be used appropriately, before authorizing their release?

l How might the decision to release essays affect the construct being assessed? In particular, would the specification of graduate admissions faculty as a second audience (that is, in addition to trained GRE essay readers) affect test takers’ writing? Might writers put forth greater effort to include or emphasize discipline-specific content? Might they be more cautious in order to limit inaccuracies that might be noticed by faculty in their discipline?

l Would some departments be inclined to use GRE essays for alternative purposes (for example, diagnosing writing problems) without first formally establishing the validity of scores for such purposes?

l What are the implications for challenging the accuracy of GRE Writing Assessment scores? Might there be more requests for restoring as a result of faculty review?

These are some of the additional issues that should probably be considered, along with the

information provided by the study reported here, before a decision is made about providing examinees’

GRE essays to test score users. Finally, as Art Young (personal communication, March 27,200O) pointed

out to us, there may also be problems associated with not releasing GRE essays. We have not speculated

about these potential problems here.

52

References

Aamodt, M. G., Bryan, D. A., & Whitcomb, A. J. (1993). Predicting performance with letters of recommendation. Public Personnel Management, 22, 8 l-90.

American Educational Research Association/American Psychological Association/ National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Ceci, S. J., & Liker, J. K. (1986). A day at the races: A study of IQ, expertise, and cognitive complexity. Journal of Experimental Psychology: General, 115, 255-266.

Cohen, J., 8z Cohen, P. (1983). Applied multiple regression/correlation analvsis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Connors, R. J., & Lunsford, A. A. (1988). Frequency of formal errors in current college writing, or Ma and Pa Kettle do research. College Composition and Communication, 39, 395409.

Council of Graduate Schools. (1993). A policy statement: An essential guide to graduate admissions. Washington, DC: Council of Graduate Schools.

Dawes, R. M. (1971). A case study of graduate admissions: Application of three principles of human decision making. American Psvchologist, 26, 180-l 88.

Dawes, R. M. (1979). The robust beauty of improper linear models. American Psychologist, 34,571-582.

Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81,95- 106

Dawes, R. M., Faust D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243,1668- 1674.

Dougherty, T. W., Ebert, R. J., & Callender, J. C. (1986). Policy capturing in the employment interview. Journal of Applied Psychology, 7 1,9-15.

Educational Testing Service. (1998). 1998-1999 GRE guide to the use of scores. Princeton, NJ: Educational Testing Service.

Educational Testing Service. (1999). 1999-2000 GRE guide to the use of scores. Princeton, NJ: Educational Testing Service.

Ferguson, F. J. (1991). Voices of their own: Students’ biographies and the college application. College Board Review, 158, 18-21,32.

Finkelstein, M. A., & Brannick, M. T. (1997). Making decisions about sexual intercourse: Capturing college students’ policies. Basic and Annlied Social Psvchologv, 19, 101-120.

Freedman, S. (1979). How characteristics of student essays influence teachers’ evaluations. Journal of Educational Psychology, 7 1, 328-338.

Gaeth, G. J., & Shanteau, J. (1984). Reducing the influence of irrelevant information on experienced decision makers. Organizational Behavior and Human Performance, 33,263-282.

Goldberg, L. R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. Psvcholoeical Bulletin, 73,422-432.

Gomey, B. E., & Jaeger, R. M. ( 1995). A pilot study on the use of five new GRE subtests for admissions screening by faculty in two academic disciplines: An apnlication of judgmental policy canturing. Greensboro, NC: The University of North Carolina at Greensboro, Center for Educational Research and Evaluation.

Hairston, M. (198 1). Not all errors are created equal: Nonacademic readers in the professions respond to lapses in usage. Collepe English, 43,794~806.

Hammond, K. R., Mumpower, J. L., & Smith, T. H. (1977). Linking environmental models with models of human judgments: A symmetrical decision aid. IEEE Transaction on Systems, Man, and Cybernetics, SMC-7,358-367.

Harrison, P. D., Ryan, J. M., & Moore, P. S. (1996). College students’ self-insight and common implicit theories in rating of teaching effectiveness. Journal of Educational Psvchologv, 88,775-782.

Hatch, J. A., Hill, C. A., & Hayes, J. R. (1993). When the messenger is the message: Readers’ impressions of writers’ personalities. Written Communication, 10, 569-98.

Hawkins, B. D. (1993). Educators push for diverse grad school admission criteria. Black Issues in Higher Education, 10, 26-30.

Hobson, C. J., Mendel, R. M., & Gibson, F. W. (1981). Clarifying performance appraisal criteria. Organizational Behavior and Human Performance, 28, 164-188.

Janopoulos, M. (1992). University faculty tolerance of NS and NNS writing errors: A comparison. Journal of Second Language Writing. 1,109-121.

Keith-Spiegel, P. (1991). The complete guide to graduate school admission: Psychology and related fields. Hillsdale, NJ: Lawrence Erlbaum.

Kline, T. J. B., & Sulsky, L. M. (1995). A policy-capturing approach to individual decision making: A demonstration using professors’ judgements of the acceptability of psychology graduate school applicants. Canadian Journal of Behavioural Science, 27, 393-404.

&rouse, S. B. (1983). The letter of recommendation: Specificity and favorability of information. Personnel Psychology, 36, 33 l-34 1.

Loher, B. T., Hazer, J. T., Tsai, A., Tilton, K., & James, J. (1997). Letters of reference: A process approach. Journal of Business and Psvchologv, 11,339-355.

McCauley, C. (1991). Selection of National Science Foundation graduate fellows: A case study of psychologists failing to apply what they know about decision making. American Psvchologist, 4, 1,287-1,291.

Murphy, E. (1991). Whomp! Real voices in college admission essays. English Journal, 80, 34-37.

54

Oltman, P. K, & Hartnett, R. T. (1985). The role of Graduate Record Examinations in graduate admissions. Journal of Higher Education. 56,523-527.

Paley, K. S. (1994, March). The college annlication essav: A rhetorical Paradox. Paper presented at the meeting of the Conference on College Composition and Communication, Nashville, TN.

Powers, D. E., & Fowles, M. E. (1997). The personal statement as an indicator of writing skill: A cautionary note. Educational Assessment, 4,75-87.

Powers, D. E., Fowles, M. E., & Boyles, K. (1996); Validating: a writing test for graduate admissions (GRE Board Professional Rep. 96-26b and ETS Research Rep. No. 96-27). Princeton, NJ: Educational Testing Service.

Powers, D. E., Fowles, M. E., & Welsh, C. K. (1999). Further validation of a writing; assessment for graduate admissions (GRE Board Research Rep. No. 96-13R and ETS Research Rep. No. 99-18). Princeton, NJ: Educational Testing Service.

Ralston, S. M., 8z Yoder, D. D. (1989). Effect of a referent’s status on the evaluation of job applications. Journal of Emplovment Counseling;, 26, 84-89.

Range, L. M., Menyhert, A., Walsh, M. L., Hardin, K. N., Ellis, J. B., & Craddick, R. (1991). Letters of recommendation: Perspectives, recommendations, and ethics. Professional Psychology: Research and Practice, 22,389-392.

Rifkin, B., & Roberts, F. D. (1995). Error gravity: A critical review of research design. Language Learning, 45,5 1 l-537.

Roehling, M. V. (1993). Extracting policy from judicial opinions -- The dangers of policy capturing in a field setting. Personnel Psvchologv, 46,477-502.

Santos, T. (1988). Professors’ reactions to the academic writing of nonnative-speaking students. TESOL Ouarterlv, 22,69-90.

Schaeffer, G., Briel, J., & Fowles, M. E. (in press). Psvchometric evaluation of the new GRE Writing Assessment (GRE Rep. No. 96-l 1). Princeton, NJ: Educational Testing Service.

Schmidt, F. L., Johnson, R. H., & Gugel, J. F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2,347-359.

Schneider, L. M., dz Briel, J. B. (1990). Validity of the GRE: 1988-89 summarv report. Princeton, NJ: Educational Testing Service.

Shavelson, R. J. (1973). What is the basic teaching skill? Journal of Teacher Education, 14, 144-15 1.

Shavelson, R. J., & Atwood, N. (1977). Teachers’ estimates of student “states of mind.” British Journal of Teacher Education, 3, 131-138.

Stumpf, S. A., dz London, M. (1981). Capturing rater policies in evaluating candidates for promotion. Academy of Management Journal, 24,752-766.

55

Tommasi, G. W., Williams, K. B., & Nordstrom, C. R. (1998). Letters of recommendation: What information captures HR professionals’ attention? Journal of Business and Psvchologv, 13,518.

Vann, R. J., Meyer, D. E., & Lorenz, F. 0. ( 1984). Error gravity: A study of faculty opinion of ESL errors. TESOL Ouarterlv, 18,427-440.

Wallace, M. J., & Schwab, D. P. (1976). A cross-validated comparison of five models used to predict graduate admissions committee decisions. Journal of Applied Psychology, 61, 559-563.

Weins, A. N., Jackson, R. H., Manaugh, T. S., Jz Matarazzo, J. D. (1969). Communication length as an index of communicator attitude: A replication. Journal of Applied Psvchologv, 53,264-266.

Werner, J. M., & Bolino, M. C. (1997). Explaining U.S. Courts of Appeals decisions involving performance appraisal: Accuracy, fairness, and validation. Personnel Psychology, 50, l-24.

Wickenden, J. W. (1982). Open letter to college students applying to competitive colleges. In H. C. Hegener (Ed.), The competitive colleges: Who are they? Where are they? What are they like? (pp. ix-xvii). Princeton, NJ: Peterson’s Guides.

Wiederman, M. W. (1999). Policy capturing methodology in sexuality research. Journal of Sex Research, 3&91-95.

Wiederman, M. W., & Dubois, S. L. (1998). Evolution and sex differences in preferences for short-term mates: Results from a policy capturing study. Evolution and Human Behavior, 19, 153-170.

Willingham, W. W. (1985). Success in college: The role of personal aualities and academic ability. New York: College Entrance Examination Board.

Willingham, W. W., & Breland, H. M. (1982). Personal qualities and college admissions. New York: College Entrance Examination Board.

Wilson, K. M. (1986). The relationship of scores based on GRE General Test item types to undergraduate grades: An exploratory study for selected subgroups (GRE Board Rep. No. 83- 19P). Princeton, NJ: Educational Testing Service.

Zedeck, S., & Kafry, D. (1977). Capturing rater policies for processing evaluation data. Organizational Behavior and Human Performance, 18,269-294.

Zeleznik, C., Hojat, M., & Veloski, J. (1983). Levels of recommendation for students and academic performance in medical school. Psychological Reports, 52, 85 l-858.

56

APPENDIX A

APPLICATION FOR GRADUATE SCHOOL

57

APPLICATION FOR ADMISSION FOR GRADUATE STUDY

APPLICATION MUST BE MADE ONLY TO ONE DEPARTMENT. Return both copies of this form with supporting documents directly to the Department to which you are applying. Processing of your application will be deluyed if supporting credentials are sent separately. Supporting materials will not be returned to you or forwarded to other schools or agencies. Type or print clearly.

BIOGRAPHICAL INFORMATION FULL LEGAL NAME LAST, FAMILY, OR SURNAME FIRST MIDDLE

MAILING ADDRESS NUMBER & STREET CITY STATE ZIP CODE

If you are a U.S. citizen or permanent resident, please 0 American Indian/Alaska Native 0 African American/ Cl Caucasian 0 Puerto Rican indicate your ethnic origin. (Self-identification is entirely (indicate tribal affiliation) Black 0 Mexican-American 0 Multi-racial voluntarily.) 0 Asian American/ (Chicano) 0 Other

Pacific Islander 0 Other Hispanic

APPLICATION INFORMATION

PROPOSED DEPARTMENT

PROPOSED 0 0 0 0 0 DEGREE: PHD MA/PHD MS/PHD MA/DMA MASTER ONLY

MAJOR DEPARTMENT SPECIALIZATION

LIST EVERY COLLEGE AND UNIVERSITY YOU HAVE ATTENDED FOR ONE YEAR OR MORE FULL TIME: ACTUAL NAME OF DATE

DATES (Month and Year) DEGREE OR DIPLOMA RECEIVED NAME OF INSTITUTION & LOCATION (List chronologically) From To MAJOR FIELD OF STUDY (do not translate) OR EXPECTED

I

1

GRADE POINT AVERAGE GRADUATE RECORD EXAM (GRE) RESULTS

GPA EXAMINATION Score %

Cumulative Verbal

Major Quantitative I

Analytical

Writing Assessment I I

I

ADDlTlONAL INFORMATION

List all other activities related to your academic goals:

ACADEMIC HONORS, FELLOWSHIPS, OR NON-ACADEMIC DISTINCTIONS, OR RECOGNITION

LANGUAGE BACKGROUND List your first language

OTHER GRADUATE SCHOOLS TO WHICH YOU ARE APPLYING:

I HEREBY APPLY FOR ADMISSION TO GRADUATE STUDY AT THE UNIVERSITY AND CERTIFY THAT THE ABOVE INFORMATION AND AlTACHED STATEMENTS ARE CORRECT AND COMPLETE TO THE BEST OF MY KNOWLEDGE. FALSIFICATION OR OMISSION OF REQUESTED INFORMATION WILL BE GROUNDS FOR TERMINATING CONSIDERATION OF THE APPLICATION, OR, IF DISCOVERED AFTER ENROLLMENT, FOR WITHDRAWING REGISTRATION PRIVILEGES.

SIGNATURE DATE

58

APPENDIX B

RECOMMENDATION FORM

59

RECOMMENDATION FORM (GRE ADMISSIONS STUDY)

NAME OF APPLICANT LAST FIRST MIDDLE

DEPARTMENT OF DEGREE

Family Educational Rights and Privacy Act of 1974 guarantees students access to their educational records, Students can waive their right to access records.

I DO WAIVE my right to inspect the contents of the following recommendation. = I DO NOT WAIVE my right to inspect the contents of the following recommendation.

SIGNATURE

RECOMMENDER: This recommendation will remain confidential during the admission process. HOW LONG AND IN WHAT CAPACITY HAVE YOU KNOWN THE APPLICANT?

Please write candidly about the student’s qualifications, potential to carry on advanced study in the field specified, intellectual independence, capacity for analytical thinking, ability to organize and express ideas clearly, and potential for teaching. Descriptions of significant actions, accomplishments, and

i

personal qualities related to scholarly achievement are particularly helpful. I I

Please rate this student in comparison with other individuals whom you have known at similar stages in their careers. Please check the box under the appropriate rating.

1. Academic preparedness/background for proposed program of study

Below average

Cl

Average

cl 2. Intellectual ability/capacity I 0 I q 3. Aptitude (skill or potential) for research 1 0 I cl 4. Teaching ability/potential 0 cl 5. Ability to communicate orally cl Cl 6. Ability to communicate in writing q cl 7. Creativity/originality cl q 8. Independent thinking

9. Dependability/reliability I 0 I 0 I cl I cl

10. Initiative I 0 I cl 11. Seriousness of purpose I cl I cl

I cl I 0 12. Professional expertise

13. Character/integrity I cl I cl 14. Promise of productive scholarship I 0 I cl 15. Emotional maturity I Cl I cl 16. Motivation for proposed program of

study cl cl

17. Overall promise I cl I 17

Somewhat Inadequate above Very Truly opportunity

average Good good Outstanding exceptional to observe

ADMISSION TO GRADUATE STUDY IS:

0 Strongly recommended 0 Recommended 0 Recommended with reservations 0 NOT recommended

DATE SIGNATURE TITLE

[Note: Assume that this recommendation is an adequate reflection of all other recommendations submitted by the applicant.]

60

APPENDIX C

APPLICANT’S STATEMENT OF PURPOSE

61

Summary of Applicant’s STATEMENT OF PURPOSE

NAME OF APPLICANT LAST FIRST MIDDLE

This applicant’s statement of purpose was rated Somewhat Inadequate by a faculty committee as showing evidence of Below above Very Out- Truly opportunity the foil0 wing: average Average average Good good standing exceptional to observe

1. Match of applicant’s interests with those of prospective advisees q 0 q 0 0 Cl 0 q

2. Match of applicant’s professional aspirations with objectives program 0 0 0 q cl 0 c3 cl

3. Applicant’s rationale for graduate study 0 cl cl cl cl 0 0 0

4. Likely commitment to the field 0 0 0 q 0 Cl cl 0

5. Evidence of motivation Cl 0 0 cl 0 cl 0 cl

6. Evidence of maturity q 0 0 0 0 cl q 0 7. Evidence of other personal characteristics

likely to facilitate degree completion II cl 0 I7 0 0 III 0 8. Degree to which applicant has overcome prior

obstacles (economic, social, etc.) 0 0 0 0 cl Cl ci 0 9. Evidence of relevant experiences, special

interests, or achievements cl 0 q 0 cl cl cl 0 10. Depth of knowledge cl 0 q 0 0 0 0 cl

11. Evidence of a “voice” (personality, style, etc.) j-J cl cl 0 0 0 0 0

12. Overall strength of statement 0 0 tl 0 III Cl cl Cl

62

APPENDIX D

ABBREVIATED APPLICATIONS FORM

63

Abbreviated Applications (History) Form D

, Undergrad. YOUR PROB School GRE UGPA FACULTY OVERALL PERSONAL RECOM. ADMIS

Selectivity v * A Writ OveralVMaJor RECOMMENDATION PROMISE STATEMENT Deny/Admit 0 toloo%

very Sel. 560 580 680 3.5 3.67 3.72 3 Strongly Recom’d 6 Outstanding 6 Outstanding 0 1 %

very Sel. 640 680 650 5.5 3.49 3.69 1 Recom. w Resew. 3 Smwt Above Avg 2 Average 0 1 %

Very Sel. 510 670 540 2.0 2.57 2.61 1 Recom. w Reserv. 3 Smwt Above Avg 6 Outstanding 0 1 %

Very Sel. 470 560 520 4.5 2.75 3.05 1 Recom. w Resew. 3 Smwt Above Avg 7 Truly Exceptional 0 1 %

Extrm’ly Sel. 540 610 670 5.0 3.44 3.07 3 Strongly Recom’d 6 Outstanding 6 Outstanding 0 1 %

Extrm’ly Sel. 590 790 730 2.5 3.59 3.25 3 Strongly Recom’d 5 Very Good 5 Very Good 0 1 %

Extrm’ly Sel. 610 640 650 2.0 3.27 3.09 1 Recom. w Reserv. 2 Average 4 Good 0 1 %

Extrm’ly Sel. 670 670 620 4.5 3.44 3.39 3 Strongly Recom’d 7 Truly Exceptional 6 Outstanding 0 1 %

Extrm’ly Sel. 660 550 760 5.0 3.33 3.66 2 Recommended 3 Smwt Above Avg 6 Outstanding 0 1 %

Extrm’ly Sel. 700 770 770 5.5 2.95 3.21 3 Strongly Recom’d 6 Outstanding 3 Smwt Above Avg 0 1 %

Extrm’ly Sel. 770 680 700 6.0 3.55 3.19 3 Strongly Recom’d 5 Very Good 3 Smwt Above Avg 0 1 %

Less Sel. 800 640 770 6.0 2.90 2.89 2 Recommended 4 Good 5 Very Good 0 1 %

Extrm’ly Sel. 670 640 570 2.5 3.21 3.54 3 Strongly Recom’d 7 Truly Exceptional 5 Very Good 0 1 %

Less Sel. 460 390 500 3.0 2.88 2.81 3 Strongly Recom’d 5 Very Good 4 Good 0 1 %

Extrm’ly Sel. 520 700 710 3.0 3.26 3.64 2 Recommended 4 Good 6 Outstanding 0 1 %

Less Sel. 530 550 580 3.5 3.33 3.76 3 Strongly Recom’d 6 Outstanding 5 Very Good 0 1 %

Less Sel. 620 710 680 6.0 3.49 3.59 1 Recom. w Reserv. 3 Smwt Above Avg 6 Outstanding 0 1 %

Less Sel. 520 550 640 2.0 2.72 2.80 1 Recom. w Reserv. 3 Smwt Above Avg 1 Below Average 0 1 %

Less Sel. 530 650 510 4.0 3.55 3.14 1 Recom. w Reserv. 2 Average 3 Smwt Above Avg 0 1 %

Very Sel. 540 720 730 5.0 2.95 3.34 2 Recommended 4 Good 5 Very Good 0 1 %

Very Sel. 540 520 590 4.5 3.11 3.22 1 Recom. w Reserv. 3 Smwt Above Avg 3 Smwt Above Avg 0 1 %

Very Sel. 640 560 570 3.0 3.55 3.56 2 Recommended 3 Smwt Above Avg 7 Truly Exceptional 0 1 %

Less Sel. 560 640 610 4.0 3.44 3.49 2 Recommended 4 Good 2 Average 0 1 %

Less Sel. 620 630 610 3.5 3.58 3.55 2 Recommended 3 Smwt Above Avg 3 Smwt Above Avg 0 1 %

Less Sel. 590 680 730 2.5 3.54 3.55 2 Recommended 5 Very Good 3 Smwt Above Avg 0 1 %

Very Sel. 530 560 630 5.5 3.30 3.48 2 Recommended 5 Very Good 4Good 0 1 %

Very Sel. 710 800 800 4.0 3.56 3.66 2 Recommended 5 Very Good 7 Truly Exceptional 0 1 %

KEY: UNDERGRADUATE SCHOOL SELECTIVITY/REPUTATION: Extrm’ly Sel. = extremely selective undergraduate school/extremely good reputation Very Sel . = very selective school/very good reputation Less Sel. = less selective school/good reputation

FACULTY RECOMMENDATION: OVERALL PROMISE = overall oromise ratina from recommendation form (3) Strongly Recom’d = strongly recommended PERSONAL STATEMENT = overall strenath of Dersonal statement (2) Recommended = recommended (1) Below average (4) Good (7) Truly exceptional (1) Recom. w Reset-v. = recommended with reservations (2) Average (5) Very good

(3) Somewhat above average (6) Outstanding

YOUR RECOM = your recommendation to denv or admit PROB ADMIS = likelihood of admission to vour deot. (0 to 100%)

Please return form in envelope provided. See reverse side for percentiles corresponding to GRE scores b

65

Table Dl. GRE General Test Scores Interpretive Data

Percent of Examinees Scorinu Lower than Selected Scores Score GRE-V GRE-Q GRE-A 800 99 99 99 700 97 80 86 600 84 58 62 500 59 35 34 400 26 15 14 300 5 3 3

Table D2. GRE Writing Assessment Interpretive Data

Score

Percent of Examinees Scoring Lower than Selected Scores

!&

6.0 99 5.0 83 4.5 64 4.0 41 3.5 24 3.0 12 2.0 2

66