The Physics GRE does not help applicants “stand out”

Nicholas T. Young1, 2 and Marcos D. Caballero1, 2, 3, 4

1Department of Physics and Astronomy, Michigan State University, East Lansing, Michigan 488242Department of Computational Mathematics, Science, and Engineering,

Michigan State University, East Lansing, Michigan 488243Center for Computing in Science Education & Department of Physics, University of Oslo, N-0316 Oslo, Norway

4CREATE for STEM Institute, Michigan State University, East Lansing, Michigan 48824

One argument for keeping the physics GRE is that it can help applicants who might otherwise be missed inthe admissions process stand out. In this work, we evaluate whether this claim is supported by physics graduateschool admissions decisions. We used admissions data from five PhD-granting physics departments over a 2-year period (N=2537) to see how the fraction of applicants admitted varied based on their physics GRE scores.We compared applicants with low GPAs to applicants with higher GPAs, applicants from large undergraduateuniversities to applicants from smaller undergraduate universities, and applicants from selective undergraduateinstitutions to applicants from less selective undergraduate institutions. We also performed a mediation andmoderation analysis to provide statistical rigor and to better understand the previous relationships. We find thatfor applicants who might otherwise have been missed (e.g. have a low GPA or attended a small or less selectiveschool) having a high physics GRE score did not seem to increase the applicant’s chances of being admitted tothe schools. However, having a low physics GRE score seemed to penalize otherwise competitive applicants.Thus, our work suggests that the physics GRE does not, in fact, help applicants who might otherwise be missedstand out.

I. INTRODUCTION

While applying to graduate programs requires many com-ponents, perhaps none is as scrutinized as the GraduateRecords Exam (GRE), and in physics, the physics GRE.Indeed, research into graduate admissions in physics sug-gests that the physics GRE is one of the most importantcomponents of the applications for determining which ap-plicants will be admitted, based on both student and fac-ulty perspectives [1, 2] and analysis of the admissions pro-cess [3, 4]. Despite its prominence in the admissions pro-cess, the physics GRE is known to be biased against womenand people of color in physics [5], resulting in lower averagescores compared to white and Asian males. At least one inthree programs use a cutoff score [2], with 700 being a com-mon choice [6], meaning applicants from groups already un-derrepresented in physics graduate programs can be furthermarginalized as they are less likely to achieve these scores.This is in addition to the observation that many physics stu-dents of color already see the GRE as a barrier to applying tograduate school [7–9].

Further, the physics GRE might not even be useful fordetermining which applicants will be successful in graduateschool. For example, Miller et al. suggest that the physicsGRE is not useful for predicting which applicants will earntheir PhDs [6]. Additionally, Levesque et al. argue that usingthe common 50th percentile cutoff score for the physics GREwould have caused admissions committees to reject nearly30% of students who would later receive a national prize post-doctoral fellowship, which can be viewed as a proxy for re-search excellence [10]. Yet despite evidence suggesting thephysics GRE does not predict these typical ways of measur-ing “success” in graduate school and calls from the Ameri-can Astronomical Society and the American Association of

Physics Teachers to eliminate the physics GRE from admis-sions [11, 12], most physics graduate programs still requireapplicants to submit their physics GRE scores. Currently,nearly 90% of physics and astronomy graduate programs stillaccept the physics GRE, with over half requiring or recom-mending submitting a score [11]. Of those that do not ac-cept physics GRE scores from applicants, all of the programsare solely astronomy graduate programs or joint physics andastronomy graduate programs. While it is uncertain whereremoving the physics GRE affects any measure of graduateschool success (e.g. completion rate), initial work by Lopezsuggests that removing the physics GRE does increase the di-versity of applicants [13].

Given these documented issues with the physics GRE, whydo departments continue to use it? First, given that manyprograms are seeing larger number of applicants, the physicsGRE provides a quick way to filter the applications down toa more reasonable number for faculty review. Unlike in un-dergraduate admissions, graduate admissions tend to be de-centralized and done at the departmental level by a facultycommittee. Hence, faculty are asked to review applicationsin addition to their regular teaching and research duties andthus, might not have the time to read the letters of recommen-dation and applicant essays for every applicant.

Second, some faculty view GRE scores as measures of in-nate intelligence [3, 14] or ability to become a PhD-level sci-entist [5]. After all, they and other faculty likely had highGRE scores in order to be admitted to graduate school, andmay exhibit a survivorship bias, believing that a high GREscore is needed to succeed. Further, physics is seen as a"brilliance-required" field, where innate intelligence is re-quired for success [15].

A third argument, and the most interesting one in termsof the scope of this paper, is that standardized tests such as

arX

iv:2

008.

1071

2v1

[ph

ysic

s.ed

-ph]

24

Aug

202

0

2

the physics GRE can help students stand out [16]. The ETS,the creator of the GRE and physics GRE, claims that subjectGREs "can help you stand out from other applicants by em-phasizing your knowledge and skill level in a specific area"[17]. For example, a student with an average grade point av-erage (GPA) might be able to stand out from other applicantsif they did exceptionally well on the physics GRE.

In addition, applicants from smaller universities or univer-sities that are not known to the admissions committee mightbenefit from performing well on a standardized measure. Forexample, the ETS claims that the GRE provide a “common,objective measure to help programs compare students fromdifferent backgrounds” [18] and physics admissions commit-tees worry that removing the GRE would limit their ability tocompare applicants from different backgrounds [19]. Anec-dotally, some faculty claim that a good physics GRE scorecould aid students from small liberal arts colleges in the ad-missions process [20].

We already know that GPAs are interpreted in context ofthe applicant’s university. Posselt has shown that among moreprestigious graduate programs, the applicant’s GPA is viewedin the context of their undergraduate institution with highGPAs from prestigious institutions seen favorably, low GPAsfrom an unknown school as unfavorably, and high GPAs fromunknown schools and middle GPAs from prestigious institu-tions in the middle [21]. Therefore, a standardized test suchas the physics GRE could provide an assumed equal compar-ison for an admissions committee and might allow the appli-cant from an unknown school to stand out or have a similarchance of admission as an applicant from a more well knownschool.

Finally, graduate admissions have been documented to be"risk-adverse," where admissions committees select appli-cants most likely to complete their program [3, 14]. As appli-cants from smaller universities may be judged based on howpreviously enrolled students from their university did in theprogram [21], a risk adverse admissions committee might beless likely to admit applicants from small universities whosestudents have previously struggled in their program. How-ever, perhaps a high standardized test score could overcomethese perceptions and signal that the applicant might indeedbe successful in the program.

Our goal then is to focus on the third argument. Does thephysics GRE help applicants "stand out" in the admissionsprocess? If that is the case, we would expect those disadvan-taged in the admissions process, those who have low GPAs,attended a smaller institution, or identify as part of a groupcurrently underrepresented in physics, to be admitted at sim-ilar rates as their more advantaged peers with similar physicsGRE scores. Specifically, we ask:

1. How does an applicant’s physics GRE score and under-graduate GPA affect their probability of admission?

2. How are these probabilities of admission affected by anapplicant’s undergraduate institution, gender, and race?

As Small points out in his critique of admissions and stan-dardized test studies [22], multiple variables rather than just a

standardized test might best explain our results and therefore,a framework that allows for substitutions and trade-offs be-tween variables is necessary. Therefore, we ask an additionalresearch question:

3. How might the above relationships be accounted forthrough mediating and moderating relationships?

This paper is organized as follows: Sec. II provides anoverview of mediation and moderation analysis. We then de-scribe our data, how we determined what constitutes “stand-ing out," and how we implemented mediation and modera-tion analysis in Sec. III. In Sec. IV, we describe our findingsand in Sec. V, we use those findings to answer our researchquestions and explain our limitations and choices which mayaffect our results. Finally, we describe our future work in Sec.VI and the implications of our work for graduate admissionsin physics in Sec. VII.

II. BACKGROUND

Before we can answer the third research question, it is im-portant to describe what we mean by mediating and moderat-ing relationships.

In a mediating relationship, two variables are only relatedbecause they are also related to some common third variable.For example, a student who played video games the nightbefore an exam might to do poorly because they stayed upplaying video games too late and did not get enough sleep.Therefore, video games and doing poorly on the exam areonly related due the common factor of lack of sleep. Lack ofsleep is then a mediating variable.

In a moderating relationship, the strength of the relation-ship between two variables depends on some third variable.For example, the relationship between someone liking dogsand owning a dog likely depends on whether they are allergicto dogs. That is, we would expect someone who likes dogsbut is allergic to dogs is less likely to own a dog than someonewho likes dogs but is not allergic to dogs is. Being allergic todogs is then a moderating variable.

Mathematically, suppose that some input X has an effecton output Y . We would say that some other input M mediatesthe relationship between X and Y if X only has an effect onY because X has an effect on M and M has an effect on Y[23]. For a simple case, we can represent these relationshipsas

Y = i1 + cX (1)

M = i2 + aX (2)

Y = i3 + c′X + bM (3)

where i represents the intercepts. These relationships arevisually shown in Fig. 1.

3

FIG. 1. Visual representation of eqs. (1) to (3). The top graphicshows eq. (1) while the bottom graphic shows eqs. (2) and (3).

Using this representation, the direct effect of X on Y isrepresented by c′ and the indirect effect is represented by ab.The total effect is then c′ + ab which for a linear regressionmodels, is equal to c. Equivalently, in the case the linear re-gression, the indirect effect is c− c′.

However, if Y is binary, linear regression is not appropriateand logistic regression should be used instead. In this case,Rijnhart et al. recommend using ab as the indirect effect astheir simulation studies found the ab estimate of the indirecteffect exhibited less bias than the c− c′ estimate [24].

To determine if the indirect effect is statistically signifi-cant, a common approach is to use a Sobel test. However,simulations suggest that the Sobel test is underpowered andthat bootstrapping is a good alternative [25]. Specifically,those simulations find that using the percentiles of a boot-strapped estimate of the indirect effect to estimate the confi-dence interval is a good compromise between avoiding typeI errors while maintaining statistical power. From their ap-proach (which has also been used in PER studies before, e.g.[26]), if ab is different than zero, then there is some degree ofmediation.

More specifically, there are three cases.1. If ab 6= 0 and c′ = 0 then M fully mediates the rela-

tionship between X and Y .2. If ab 6= 0 and c′ 6= 0, then M partially mediates the

relationship between X and Y . In that case, we canestimate the amount of mediation as the fraction of thetotal effect attributed to the indirect effect, ab

ab+c′ [27,28].

3. If ab = 0, then M does not mediate the relationshipbetween X and Y .

So far, we’ve assumed that the relationship between themediator M and the output Y does not depend on any othervariables. However, it is possible that the relationship be-tween M and Y could also depend on X or some othervariable, meaning there is a conditional indirect effect (seePreacher et al. [29]). In the case that the relationship betweenM and Y depends on X , we would say that X moderates therelationship between M and Y . Practically, this means we

must add an interaction term to eq. (3), which then becomes[29]

Y = i3 + c′X + b1M + b2XM

= i3 + c′X + (b1 + b2X)M(4)

.The conditional indirect effect is then a(b1+b2X). If b2 =

0, we would say that there is no moderation and the indirecteffect is the standard ab.

In the special case that X is binary, eq. (4) reduces to Y =ix=0 + b1M when X = 0 and Y = iX=1 + (b1 + b2)Mwhen X = 1. Therefore, to test if there is moderation, wecan simply regress M on Y given X = 0 and again givenX = 1 and subtract the slopes to calculate b2.

III. METHODS

A. Data

Data for this study comes from the physics departments atfive selective, research-intensive, primarily white universities.Four of these universities are public and part of the Big TenAcademic Alliance while the remaining university is a privateMidwestern university. During the 2017 and 2018 academicyears, graduate admissions committees at these five univer-sities recorded all physics applicants’ undergraduate GPA,GRE scores, undergraduate institution, and demographic in-formation such as gender, race, and domestic status. In ad-dition, the universities recorded whether each applicant madethe shortlist, was offered admission, and whether the appli-cant decided to enroll. Because our study includes all appli-cants rather than only admitted applicants, we are unlikely tosuffer from the range restrictions noted in critiques of otheradmissions studies (e.g. [22, 30]). However, we do a addressa possible range restriction in the Limitations and ResearcherDecisions section (sec. V B).

Due to different requirements and admissions processes forinternational students and domestic students (e.g. interna-tional students need to submit a test of English proficiency),we only include domestic students in our study. We then re-move any applicant for whom a physics GRE and GPA werenot recorded, leaving us with 2537 applicants. Distributionsand analysis of the physics GRE scores and GPAs appear inthe appendix.

As the applicant’s undergraduate university does not con-tain meaning in itself, we needed to categorize the institu-tions. We chose to categorize the institutions by their size andtheir selectivity. We then used the number of physics bache-lor’s degrees awarded per year as measure of the size of theuniversity. We assume that universities with more graduatesare more well known and hence, would likely be known to theadmissions committees. In contrast, universities that producefewer bachelor’s degrees might not be known to the admis-sions committees and hence, might be unknown programs. It

4

would then be these applicants from “smaller” programs whomight need to “stand out.” We acknowledge some programsthat produce a small number of physics bachelor’s degreeseach year might not be unknown to the admissions commit-tees due to previous applicants from such schools or researchcollaborations or partnerships. However, there is no way inour data to know if this is the case.

To determine whether a university should be counted asa “small university”, we used the undergraduate institutionnames to look up the number of typical physics bachelor’sdegrees from AIP’s public degree data [31, 32]. As of thiswriting, degree data for the 2018 academic year was not avail-able, so we used data from the 2016 and 2017 academic yearsto quantify the number of bachelor’s degrees. Additionally,this would have been the most recent data available when ad-missions committees would have reviewed applications andmany of the applicants would be represented in the data asbachelor degree recipients

To account for the institution’s prestige, we used Barron’sSelectivity Index [33]. Barron’s selectivity index is a mea-sure based on the undergraduate acceptance rate of an institu-tion as well as characteristics of its undergraduate incomingclasses, such as mean SAT scores, high school GPAs, andclass rank. We assume selectivity is a proxy for prestige asprestigious institutions tend to have low acceptance rates andhigh SAT scores and GPAs from incoming students. In con-trast to the AIP data, Barron’s selectivity index applies to theinstitution as a whole rather than only the physics department.

B. Probability of admission procedure

Determining whether an applicant is more or less likely tobe admitted first requires computing admissions probabilities.To do so, we grouped applicants based on their GPAs andphysics GRE scores. Prior work has found that the physicsGRE score and undergraduate GPA are two of the most im-portant aspects of the applications [2–4]. Our previous workspecifically found that the physics GRE score and undergrad-uate GPA were able to predict with 75% accuracy whetheran applicant would be admitted to one public Midwesternphysics graduate program.

In addition, physics is a “high consensus” discipline, mean-ing most programs agree on what consists of a successful ap-plicant [3]. Therefore, despite many other components of theapplications that affect whether an applicant will be admit-ted, we believe using the physics GRE score and undergrad-uate GPA provides a first-order overview of what admissionscommittees would use to admit applicants.

In order to ensure a reasonable number of applicants ineach group to do meaningful analysis, we grouped applicantsinto bins based on their GPA and physics GRE score. Wechoose to use GPA bins 0.1 units in width and physics GREbins 50 points in width. The GPA bins were selected to en-sure that that GPAs with the same tenth digit were in a singlebin. That is, 3.50 through 3.59 would be in a single bin. All

GPAs were already reported on the 4.0 scale and physics GREscores were reported using the standard 200-990 scale so wedid not need to do any conversions.

We then computed the fraction of applicants in each binwho were admitted to the program they applied. As we areinterested in applicants “standing out,” we frame our resultsas whether applicants in a bin are admitted at a higher ratethan the overall rate (all accepted applicants divided by allapplicants). If applicants are admitted at a higher rate thanthe overall rate, it suggests that these applicants did in factstand out to the admissions committee.

To take into account the size of the institution, we first usedthe AIP data to determine the national quartile each appli-cant’s institution ranked in terms of all bachelor’s degree re-cipients for each of the two years of data. Because not all in-stitutions reported data in both years and the number of grad-uates could vary significantly between years, we conductedseparate analyses first with the highest quartile an institutionreached in the two years and second with the lowest quartilethe program reached in the two years. For example, if an in-stitution was ranked in the 3rd quartile the first year and the4th quartile in the second year, our first analysis would usethe 4th quartile and our second analysis would use the 3rdquartile. We then define the large programs as those in the4th quartile and small programs as those in the 1st through3rd quartiles. We address this choice in the discussion.

When using Barron’s Selectivity Index to take into accountthe selectivity of the institution, we used Chetty et al.’s [34]five groupings (Ivy League +, Remaining most selective in-stitutions, highly selective institutions, selective institutions,and non-selective institutions) as a guide. As there was a sin-gle applicant from a non-selective institution, selective andnon-selective were grouped into a single category. Becausewe are interested in smaller, less known programs comparedto larger, well-known programs, we took the selective andnon-selective group to be our "less selective institution" groupand institutions in the first three of Chetty et al.’s categories asour "most selective institutions". This corresponds to group-ing institutions with a Barron’s Index of 1 and 2 together asthe "most selective institutions" and all other values togetheras the "less selective institutions".

To understand how high physics GRE scores might helpapplicants identifying as part of a group currently underrep-resented in physics, we compared women’s admission proba-bility to men’s admission probability and applicants of color’sadmission probability to applicants not of color’s admissionprobability. While it should be noted that gender is not binary[35], the data the admissions committee recorded is only interms of the male and female binary and hence, we cannotcomment on how high physics GRE scores may impact ap-plicants identifying as other genders.

Further, given the limited number of applicants identify-ing as part of a racial group underrepresented in physics, wecombined all applicants identifying as Black, Latinx, Multira-cial, or Native into a single category, which we will refer toas B/L/M/N following the recommendation of Williams [36].

5

TABLE I. Counts of applicants by gender and race who provided both GPAs and physics GRE scoresRace

Gender Asian Black Latinx Multi Native White Unreported TotalMen 247 49 99 166 4 1410 112 2087Women 56 2 19 26 0 308 28 439Unreported 1 0 0 1 0 5 4 11Total 304 51 118 193 4 1723 144 2537

We acknowledge that this may obscure important distinctionsbetween groups, as Teranishi [37] and Williams suggest. Wealso acknowledge applicants identifying as a marginalizedgender and race may face additional barriers and hence couldstand out differently than an applicant identifying as either amarginalized gender or race. However, there are less than 50applicants ( 2% of the sample) identifying as a member ofboth a marginalized gender and marginalized race, limitingstatistical power for analysis. Full demographics are shownin table I. For information about how race and ethnicity cat-egories were constructed and standardized, see Posselt et al.[38] who previously used the 2017 academic year applicationdata from this study in their study.

C. Mediation and Moderation Procedure

Given that to some degree, both the physics GRE scoreand undergraduate GPA measure physics knowledge, we ex-pect that these two measures will be correlated with eachother. Therefore, we first tested whether the physics GRE andGPA have any mediating or moderating effects on each otherwhen predicting admission. Because admissions status is abinary outcome variable, we need to use logistic regressionfor eqs. (1), (3) and (4).

When taking an applicant’s GPA and physics GRE scoreinto account, we first centered and scaled both variables sothey both have means of zero and variances of 1. As we aretreating GPA and physics GRE score as continuous, we canuse linear regression for eq. (2).

To estimate the coefficients in eqs. (1) to (4), we gener-ated 5000 bootstrap samples with replacement as was done inHayes and Scharkow [25]. For each trial, we computed theindirect effect ab. To get the estimate of each parameter, wetook the average of the 5000 bootstraps. To get the lower endof the 95% confidence interval, we used the value that cor-responded to the 2.5th percentile of the values generated bythe bootstrap. Likewise, to get the upper end of the 95% con-fidence interval, we used the value that corresponded to the97.5th percentile.

For the institutional features, we treat institutional selectiv-ity and institution size as binary input variables (most selec-tive or less selective and large institution or smaller institu-tion) and the applicant’s physics GRE score and GPA as con-tinuous mediating and moderating variables. We then com-puted the coefficient estimates using the same bootstrap pro-

cess as above.For demographic features, we treat gender and race as bi-

nary variables. Again, we use B/L/M/N as one category forrace and white and Asian as the other. We also treat the appli-cant’s physics GRE score and GPA as continuous mediatingand moderating variables. We then computed the coefficientestimates using the same bootstrap process as above.

IV. RESULTS

A. Probability of admission results

When comparing the GPAs and physics GRE scores of allstudents, we notice that most students who are admitted haveboth high GPAs and high physics GRE scores (Fig. 2). Fur-ther, while a near perfect GPA or physics GRE score resultedin the highest chance of admission, having either a high GPAor high physics GRE and a modest score on the other seemedto still offer an admission fraction around the overall average.However, having a low GPA or low physics GRE and a mod-est score on the other is usually grounds for rejection. Overalladmissions fractions for a given physics GRE score or GPAare shown in the top and right margins of Fig. 2 respectively.

When it comes to having a high physics GRE score despitea low GPA, we first note that only a small fraction of all ap-plicants fall in this regime. Second, there appears to be nopattern in terms of higher than average fraction admitted forthese applicants. Some combinations of low GPA and highphysics GRE score result in a few applicants being admit-ted, and hence, an above average fraction of applicants beingadmitted, while other score combinations have no applicantsbeing admitted, and hence, a below average change of admis-sion. For example, having a GPA in the 3.3 bin and a physicsGRE score in the 1000 bin resulted in an above average frac-tion admitted while having a GPA in the 3.4 bin and a physicsGRE score in the 1000 bin did not result in an above aver-age fraction admitted, despite the applicants having a higherGPA.

To further understand whether a high physics GRE scorecan highlight those with low GPA, we divided all studentsinto either a high or low GPA and high or low physics GREscore bins, Fig. 3. Based on Fig. 2 in terms of admissionsprobabilities, a low GPA seems to be below a 3.5, while a highphysics GRE score seems to be above 700. However, 700is a common cutoff score which could explain why admis-

6

FIG. 2. Fraction of applicants admitted by undergraduate GPA and physics GRE score. The number of students in each bin is also shown.‘Any‘ corresponds to the corresponding row or column totals. The bin label corresponds to the upper bound of values in the bin exclusivewith the exception of the 4.0 GPA bin which includes 4.0. Values are colored based on whether they are above, below, or equal to the overalladmissions rate. Admissions rates within 10% of the overall rate are colored the same as the overall rate.

FIG. 3. A condensed version of Fig. 2 showing the fraction of ap-plicants admitted by undergraduate GPA and physics GRE score.

sions probabilities increase after that score. Because hittingthe minimum score might not catch the admission commit-tee’s eyes, we instead selected a higher score of 880 which

represents the 80th percentile.

From Fig. 3, we notice two things. First, among appli-cants in the low GPA bin, less than half (44%) even make itabove the typical cutoff score of 700 and less than 10% ofthose applicants with low GPAs score 880 or higher. Theserepresent approximately 11% and 2% of all applicants respec-tively. Comparing the fraction of admitted applicants in eachbin, applicants with high physics GRE scores and low GPAsare admitted at nearly the same rate as applicants with highGPA and low physics GRE scores.

Second, we notice that 16% of all applicants score in thehigh GPA but low physics GRE score bin. That is, more appli-cants could be penalized for having a low physics GRE scoredespite a high GPA than could benefit from a high physicsGRE score despite a low GPA.

When taking the size of the applicant’s undergraduate pro-gram into account, (large or small), using either the highestor lowest quartile of bachelor’s graduates over the two yearperiod did not substantially change the results. Therefore, weonly present results from the highest quartile reached, whichare shown in Fig. 4. Due to the much smaller number of ap-plicants per bin, we reduce the number of GPA and physics

7

FIG. 4. Fraction of applicants admitted by undergraduate GPA and physics GRE score and split by large or small undergraduate university

FIG. 5. Fraction of applicants admitted by undergraduate GPA and physics GRE score and split by selective or non-selective undergraduateuniversity.

GRE bins. We use bins of 3.0 or less, which corresponds to aB or lower, 3.0 to up 3.3, a B+, 3.3 up to 3.7, an A-, and 3.7up to 4, an A under the standard 4.0 scale.

Overall, by looking at the bin in the ‘Any’ row and ‘Any’column of Fig. 4 and Fig. 5, we see that applicants fromlargest undergraduate programs are nearly 40% more likelyto be admitted (0.28 to 0.20) while applicants from selectiveinstitutions are nearly 70% more likely to be admitted (0.31to 0.18). Looking at the individual admission fractions, theredoes not appear to be any advantage to applicants graduatingfrom smaller institutions or less selective institutions. Thephysics GRE scores and GPAs where applicants are admit-

ted at higher than average rates are the nearly same for largeand small programs and selective and non-selective programs.Unsurprisingly, these tend to be higher physics GRE scoresand higher GPAs. Outside of a few bins with a small numberof applicants, no combination of low GPA (B+ or less) andhigh physics GRE score resulted in above average admissionfraction.

For the highest physics GRE scores, 900 and above, appli-cants from the largest or most selective universities seem tobe admitted at a higher rate and a higher fraction of applicantsfrom large or selective universities achieve these high scorescompared to applicants from smaller universities. The frac-

8

TABLE II. Distribution of applicants scoring in each Physics GRErange by size of institution. ETS only publishes overall score dis-tributions and hence, we cannot report national scores from onlydomestic students.

Score LargeSchools

SmallSchools

SelectiveSchools

Non-selectiveSchools

National

(400,500] 0.7% 4.3% 0.8% 2.1% 9%[500,600) 7.5% 21.4% 5.9% 17.7% 19%[600,700) 15.6% 22.9% 14.2% 23.3% 20%[700,800) 21.5% 25.1% 22.1% 23.0% 19%[800,900) 29.3% 18.8% 29.7% 23.1% 16%[900,990] 25.5% 7.5% 27.3% 10.7% 17%

tion of applicants from both large universities, small univer-sities, selective universities, and non-selective universities, aswell as nationally, achieving each score is shown in table II.Thus, it appears that even if higher scores did help applicantsstand out, applicants from smaller and less selective schoolsmost in need of standing out are less likely to achieve thosescores in the first place.

Finally, the results from grouping by gender and race areshown in Fig. 6 and Fig. 7. Interestingly, we find that formost physics GRE scores, women are admitted at higher ratesthan men of equal score are. Likewise, we find that appli-cants identifying as Black, Latinx, Multi-racial, or Native areadmitted at higher or the same rates at applicants identifyingas white or Asian for similar physics GRE scores. In addi-tion, the same trend seems to hold for GPA as well. How-ever, a high physics GRE score does not seem to help womenwith a low GPA. For B/L/M/N applicants, there appears tobe a few places where applicants may stand out (such as the800 physics GRE bin and 3.3 GPA bin). If these applicantswere standing out due to their physics GRE score though, wewould expect that pattern to continue for higher physics GREscores but the same GPA. This does not appear to be the case,suggesting these applicants stood out for a reason other thantheir physics GRE scores. We address this in our discussion.

Because these applicants may have stood out for reasonsother than their physics GRE score, we do not discuss anyinteractions between gender and race and selectivity and in-stitution size. For completeness, plots showing these interac-tions are included in the supplementary material.

B. Mediation and moderation results

1. Physics GRE and GPA

Whether we pick the physics GRE score or GPA as the me-diating or moderating variable does not change the results, sobecause our previous work showed that the physics GRE hasmore predictive power than the applicant’s GPA for admis-sion [4], we only present the results when the independentvariable is the physics GRE score. A visual representation of

our mediation results with the physics GRE score and GPA isshown in Fig. 8. We find that all coefficients are statisticallydifferent from zero.

From Fig. 8, we see that an applicant’s physics GRE scoreand GPA have about the same effect on whether the appli-cant is admitted. Given that applicants who had either a highphysics GRE score or a high GPA had about the same chanceof being admitted, this is not a surprising result.

Second, we find that the indirect effect is not zero, meaningthat there is partial mediation. That is, whether an applicant isadmitted depends on their physics GRE score and their GPA.In terms of the amount of moderation, we find that the indirecteffect accounts for nearly 33% of the total effect.

Finally, doing moderation analysis, we find that b2 =0.024 (−0.114, 0.154). As zero is included in the confidenceinterval, we do not find evidence that the physics GRE scoremoderates the relationship between an applicant’s GPA andwhether they are admitted. That is, relationship between anapplicant’s GPA and whether they are admitted is not influ-enced by their physics GRE score.

2. Institutional features

A visual depiction of our results is shown in Fig. 9. Wefind that the applicant’s physics GRE score partially mediatesthe relationship between the selectivity of their undergraduateinstitution and whether they were admitted and fully mediatesthe relationship between their institution’s size and whetherthey were admitted. The fractions of mediation due to theindirect effects were ab

ab+c′ = 0.456 and 0.969 respectively.In contrast, the applicant’s GPA was not found to be a sig-

nificant mediator in either case (zero was contained in theindirect effects’ 95% confidence intervals).

When looking at the results of the moderation anal-ysis when the physics GRE is the mediating variable,we find that neither b2 value is statistically differentfrom zero (b2,selectivity = 0.235 (−0.015, 0.476) andb2,institutionsize = 0.104 (−0.161, 0.3.63)). However, inthe case of institutional selectivity moderating physics GREand admit, the result is marginally significant. Indeed, zero isnot contained in the 93% confidence interval for b2,selectivity.

3. Demographic features

Our results are shown visually in Fig. 10. Because wechose woman to be "1" and B/L/M/N to be "1" in our logis-tic regression equation, some of the coefficients are negative.For example, the negative a coefficient for gender and physicsGRE score means that women score lower on the physicsGRE than men do. Because the sign depends on our choiceof which category should be "1" and are in that sense arbi-trary, we use the absolute values of c′ and ab to calculate thefraction of mediation.

9

FIG. 6. Fraction of applicants admitted by undergraduate GPA and physics GRE score and split by the applicant’s gender.

FIG. 7. Fraction of applicants admitted by undergraduate GPA and physics GRE score and split by the applicant’s race.

We find that the applicant’s physics GRE score partiallymediates the relationship between both gender and admissionand race and admission. The fractions of mediation for gen-der and admission is |ab|

|ab|+|c′| = 0.281 and for race and ad-

missions is |ab||ab|+|c′| = 0.386.

We also find that GPA partially mediates the relationshipbetween race and admission but not gender and admission.The fraction of mediation for race and admission due to GPAis |ab|

|ab|+|c′| = 0.458 respectively.When investigating whether any moderation effects exist,

we do not find that to be the case. That is, we find that none ofthe b2 values are statistically different from zero. Specifically,

• b2,pGRE,gender = 0.192 (−0.090, 0.483),• b2,GPA,gender = 0.241 (−0.072, 0.578),• b2,pGRE,race = −0.011 (−0.266, 0.259),• b2,GPA,race = −0.209 (−0.517, 0.126).

All results and interpretations from the mediation and mod-eration analyses are summarized in table III.

V. DISCUSSION

Here, we address each of our research questions and possi-ble limitations or confounding factors.

10

FIG. 8. Visual representation of the bootstrapped coefficients ineqs. (1) to (3). We do find evidence of GPA mediating the rela-tionship between physics GRE score and admission status.

TABLE III. Summary of the mediating and moderation results. *signifies partial mediation is present, ** signifies full mediation ispresent, † signifies moderation is present. However no moderationeffects were found.

Independent Mediating Indirect effect Moderating effectPhysics GRE GPA 0.234* 0.024Selectivity Physics GRE 0.377* 0.235Selectivity GPA 0.030 0.132Institution size Physics GRE 0.492** 0.104Institution size GPA -0.017 -0.109Gender Physics GRE -0.679* 0.193Gender GPA -0.040 0.241Race Physics GRE -0.219* -0.011Race GPA -0.401* -0.201

A. Research Questions

How does an applicant’s physics GRE score and under-graduate GPA affect their probability of admission? We findthat scoring highly on the physics GRE and having a highGPA results in the highest chance of admission (Fig. 3). Like-wise, having a low physics GRE score and low GPA resultsin the lowest chance of admission. If either the applicant’sphysics GRE score or GPA is high while the other is not,the chance of admission is approximately equal, regardlessof which one is high.

However, the number of applicants with high GPAs but lowphysics GRE scores is 9 times as large as applicants with lowGPAs and high physics GRE scores (i.e. scoring above the80th percentile; Fig. 3). Even if we consider meeting theminimum cutoff score as a high physics GRE score, the num-ber of applicants who have high GPAs but low physics GREscores is 1.5 times greater than the number of applicants withlow GPAs but high physics GRE scores. Thus, many morehigh GPA applicants could be penalized by the physics GREthan low GPA applicants could stand out or benefit with ahigh physics GRE score.

Finally, we note that for low-GPA applicants with highphysics GRE scores, they are all essentially admitted at the

same rate, regardless of whether they scored in the 700-870range or the 880-990 range. If these applicants were standingout, we would expect low GPA applicants scoring above 880to be admitted at a much higher rate than low GPA applicantsscoring between 700 and 870. Thus, it is hard to determine ifthese applicants actually stood out to the committee or if theysimply met the minimum physics GRE score needed for thecommittee to review the rest of the application.

How are these probabilities of admission affected by an ap-plicant’s undergraduate institution, gender, and race? First,we find that for most physics GRE scores, applicants fromlarger and smaller institutions are admitted at similar rates(Fig. 4). However, for the highest scores (above 900), ap-plicants from larger universities are admitted at higher rates.Interestingly, for applicants from smaller programs scoringabove 900 does not appear to provide any additional benefitin terms of the fraction of applicants admitted compared toscoring between 800 and 900.

In contrast, applicants from less selective institutions areless likely to be admitted than applicants from more selec-tive institutions for all physics GRE scores above the com-mon cutoff score (Fig. 5). That is, the physics GRE doesnot seem to counteract any potential biases from admissionscommittees toward applicants from less selective institutions.

Overall, attending a large or selective institution and scor-ing highly on the physics GRE does result in a higher chanceof admission than scoring highly on the physics GRE and at-tending a smaller or less selective institution.

It is important to note that there might be selection biasin our data because test-takers with high scores from smalleruniversities might not chose to apply to these schools. How-ever, this seems unlikely because 1) these programs are highlyregarded and hence, these would not be "safety schools" tohigh scoring applicants (as indicated by many high scoringapplicants from large programs applying here) 2) while thereis research suggesting students with low physics GRE scoresmight view their scores as barriers to applying [7], to ourknowledge, there is no evidence that students with high scoresdo not apply to physics graduate programs. Given that stu-dents with low test scores might not apply, it is expected thatour data is not representative of test-takers on the lower endof scores (as shown in table II).

When looking at the demographic variables, we find thatwomen are admitted at higher rates than men with similarscores (Fig. 6) and B/L/M/N applicants are also admitted athigher rates than white or Asian applicants (Fig. 7. As priorwork has shown [5], women and B/L/M/N test-takers tend toscore lower than white men on the physics GRE and hence,scoring highly could cause these applicants to stand out toadmissions committees.

How might the above relationships be accounted forthrough mediating and moderating relationships? Our me-diation and moderation analysis further supports the resultsfound through the probability of admissions procedure.

We find that the physics GRE score and GPA have similarregression coefficients when modeling admission, suggesting

11

FIG. 9. Visual representation of the bootstrapped coefficients in eqs. (1) to (3). We do find evidence of the physics GRE score mediatinginstitution size and selectivity but do not find evidence of GPA mediating institution size or selectivity.

FIG. 10. Visual representation of the bootstrapped coefficients in eqs. (1) to (3). We do find evidence of the physics GRE score mediatinggender and race and GPA mediating race but do not find evidence of GPA mediating gender.

they have similar effects (Fig. 8). In addition, we did notfind any evidence of moderation. That means the relationshipbetween GPA and admission is not different due to the appli-cant’s physics GRE score. If a high physics GRE score didhelp a low-GPA applicant stand out, we would expect to see

a moderation effect.Combining the results of probability of admission analysis

and the mediating and moderation analysis, we find that thereis no interaction between an applicant’s physics GRE scoreand their GPA when it comes to admission probability. An

12

applicant with a low GPA cannot simply overcome that lowGPA by scoring highly on the physics GRE.

When we performed mediation analysis on the institutionalfactors, we found that the relation between institutional selec-tivity and admission was partially mediated by the applicant’sphysics GRE score and the relation between institutional sizeand admission was fully mediated by the applicant’s physicsGRE score (Fig. 9). Neither of these relationships was medi-ated by the applicant’s GPA however.

The results of the mediation analysis show that physicsGRE scores seem to explain some of the differences in ad-mission probability based on the applicant’s undergraduateinstitution. Therefore an applicant from a smaller or less se-lective institution may be able to stand out by scoring highlyon the physics GRE. However, looking at the fraction of ap-plicants by physics GRE scores, especially the highest scores,suggests that is not what happens in practice.

Our moderation analysis found a marginally significant dif-ference in the relation between an applicant’s physics GREscore and admission status by institution selectivity, whichwarrant further analysis. If future work does find a moder-ation effect from an applicant’s physics GRE score on therelation between their institutional selectivity and admissionstatus, it would challenge the claim that the physics GRE of-fers a level playing field for all applicants.

In terms of gender and race, we do find some mediatingrelationship, but no moderation relationships (Fig. 10). Wefind that the physics GRE partially mediates the relationshipbetween gender and admission and the relationship betweenrace and admission. We also find GPA partially mediates therelationship between race and admission. That is, some ofthe differences in admission rates between men and womencan be explained by the differences in their physics GREscores and some of the differences in admission rates betweenB/L/M/N applicants and non-B/L/M/N applicants can be ex-plained by differences in their physics GRE scores and GPAs.

These results then suggest that a female or B/L/M/N appli-cant may be able to stand out by doing well on the physicsGRE. In practice, the probability of admission results do sug-gest that women and B/L/M/N applicants are admitted athigher rates than their male, white, or Asian peers are. How-ever, as the five programs studied here were interested in in-creasing their diversity, our data does not allow us to disentan-gle "standing out" from highlighting. Therefore, our resultsshould be interpreted with caution regarding any claims thatthe physics GRE may help applicants from groups underrep-resented in physics stand out.

It should also be noted that women and B/L/M/N appli-cants are less likely to reach these higher scores than theirmale, white, and Asian peers. In our data, 75% of men and72% of white or Asian applicants scored above 700 com-pared to 45% of women and 57% of B/L/M/N applicants.Thus, even if the physics GRE does allow these applicantsto stand out, any potential benefit must be weighed againstknown scoring discrepancies.

B. Limitations and Researcher Decisions

Data Biases As previously noted, applicants with lowerphysics GRE test scores may be less likely to apply, result-ing an over-representation of high scoring applicants. In ad-dition, the programs in this study are well-regarded programsand there is likely a secondary bias toward applicants withhigh GPAs and high physics GRE scores applying overall. Asa result, the results may not generalize to graduate programswhose applicants tend to have lower GPAs or low physicsGRE scores.

In addition, it is possible that an applicant could be rep-resented multiple times in the data set, as an applicant couldhave applied to more than one of the five universities in thisstudy. However, each applicant applies to each program in-dependently and thus, we can treat them as separate eventsfor the admissions probabilities. On the other hand, resultsbased on distributions such as table II and Fig. 11 and Fig.12 would be affected by the duplicates. When we comparethe distributions both with and without possible duplicates,Kolmogorov-Smirnov tests [39] suggest the distributions arenot significantly different. Therefore, because we cannot ac-tually determine which applicants are duplicates and exclud-ing possible duplicates does not change our results, we didnot remove possible duplicates.

Our choice of low GPA and high physics GRE While per-centiles are available for the physics GRE, a “high score” isleft to interpretation. Even among admissions committees,individual members may have different ideas of what a highscore is. In our work, we have taken the common cutoff scoreof 700 as the minimum possible high score [6]. Even aroundthis minimum score, the number of applicants with low GPAswho could benefit from scoring highly on the physics GRE isless than the number of high GPA applicants who could bepenalized by having a score below the cutoff.

We find that the number of low GPA applicants who couldbenefit from a high physics GRE score is approximately equalto the number of high GPA applicants who could be penal-ized by a low score when the high score cutoff is 670, whichis lower than the typical cutoff score and is around the 43rdpercentile. Assuming a high score should be at least abovethe 50th percentile, our specific choice of a high score doesnot affect our result that more applicants could be penalizedthan could benefit.

The previous argument is also affected by what we con-sider a high GPA. We have chosen any GPAs less than 3.5 tobe low based on the results shown in Fig. 2 where applicantswith GPAs at or above 3.5 are nearly twice as likely to beadmitted to as applicants with GPAs below 3.5. If we wereto pick a lower threshold, there would be even fewer appli-cants in the low GPA-high physics GRE score group and moreapplicants in the high GPA-low physics GRE score group,meaning even more applicants would possibly be penalizedrather than standout. If we instead picked a higher GPA suchas 3.6, there would be more applicants who could potentiallybenefit, but even then, the number of applicants who could

13

benefit is nearly equal to the number of applicants who couldbe penalized around a physics GRE score of 730, which isnot a high physics GRE score (approximately 54th percentile)and does not significantly change our results. If we were topick an even higher GPA cutoff, we could be hard-pressed tojustify why anything other than an ‘A’ GPA is considered alow GPA, especially because admissions committees seem togroup applicants with GPAs between 3.5 and 3.6 more closelywith applicants with GPAs between 3.7 and 3.8 than appli-cants with GPAs between 3.4 and 3.5 (based on the fractionof applicants admitted).

Based on our data and the fact that some universities use3.5 as the only separation between a 3.0 and 4.0, using 3.5seems to represent the best option for separating high and lowGPA students. Using any other choice either strengthens ourclaims or seems unrealistic to use as a cutoff.

Our choice of non-selective school We choose to followa modified version of Chetty et al.’s groupings of programs[34]. However, many large, state universities have a Bar-ron’s Selective Index of 3 and fall in Chetty et al.’s fourthgroup. For our analysis, we would have included these large,state institutions as part of the less selective programs. As weare concerned with whether the physics GRE helps applicantsstand out, saying applicants from large, state universities (forexample, the University of Colorado-Boulder, the Universityof Washington, and Michigan State University) may fall inthe traditionally missed category may not be correct.

We reran the analysis with these large, state institutions aspart of what we called the most selective programs. We findthat the conclusions are then more aligned with the large vssmall program results. Using this grouping, applicants fromless selective programs are admitted at similar rates to appli-cants from more selective programs for most physics GREscores. However, applicants from more selective institutionswith physics GRE scores above 900 are still more likely to beadmitted than applicants from less selective institutions withsimilar physics GRE scores.

In terms of the mediating and moderation analysis, our re-sults would be strengthened under this choice. The PhysicsGRE score fully mediates the relationship between selectivityand admission. In addition, selectivity moderates the relation-ship between the physics GRE and admission. That is, underthis grouping, we do find that the relation between physicsGRE score and admission depends on the applicant’s under-graduate institution, with a high score from a selective insti-tution carrying more "weight."

Thus, even though the details change, the overall conclu-sion are not weakened by changing our groupings. In fact,changing the groupings may strengthen our conclusions in-stead.

Our choice of a “small” school We chose to small schoolsto be any university not in the top quartile of yearly bache-lor’s degrees awarded. We acknowledge that using quartilesis a somewhat arbitrary decision. However, when we usedhalves instead of quartiles to divide large and small school,our results were unchanged, both in terms of the probability

of admission analysis and the mediation and moderation anal-ysis. Using the bottom quartile as small schools and all otherprograms as the large school would not have yielded insight-ful results as less than 2% of applicants would have attendeda small school under this choice.

Of the possible physics specific measures, the number ofbachelor’s degrees seems most appropriate because programswith more graduates are more likely to be known by admis-sions committees simply because there are more students toapply from those programs. For example, the programs inthe top quartile by number of bachelor’s graduates producenearly two-thirds of all physics bachelor’s graduates [31, 32].In addition, we assume that programs with strong physicsreputations attract more students and hence, produce moregraduates. While this is likely to be more true at the grad-uate level, not all physics programs offer graduate degreesand hence, using the number of PhDs awarded would not beuseful. Thus, we believe the number of bachelor’s graduatesserves as a rough proxy for physics reputation.

VI. FUTURE WORK

While the five universities included in this study were inter-ested in increasing their diversity and reducing inequities intheir program, their admissions processes still resembled thetraditional metrics-based admissions model. Recently, manyprograms, including the ones studied here, have begun to em-ploy holistic admissions, which looks at the overall appli-cation, taking into account non-cognitive competencies andcontextualizes the accomplishments of the applicant in termsof the opportunities that were available to them [40, 41]. Of-ten these holistic admissions use rubrics to weight the vari-ous components of each applicant (e.g. see [42, 43]). Evi-dence from biomedical science graduate programs suggeststhat the GRE can even be included in holistic admissionswithout reproducing its known gender and racial biases [44].Further, their two-tiered approach to holistic admissions didnot significantly increase the workload of admission commit-tee members. These findings could persuade faculty reluctantto remove GRE due to its ease and supposed ability to mea-sure some innate quality to try holistic admissions. Whetherthese results would hold for decentralized admissions as istypical in physics and for the physics GRE though are stillopen questions.

Our future work will then examine how our results maybe affected when a department uses holistic admissions. Intheory, we should no longer see the discrepancies between ad-mitted applicants from large and small programs and more se-lective and less selective universities. In addition, the samplerubric developed by the Inclusive Graduate Education Net-work (as shown in [42]) suggests ranking applicants by high,medium, or low on each part of their applicant. Therefore, wewould expect to see a flatter distribution of admission frac-tions based on physics GRE scores because for example, allscores within the ‘high’ range should be treated equally in the

14

admissions process. Our future work will determine if this isindeed the case.

VII. CONCLUSION AND IMPLICATIONS

Our work suggests that scoring highly on the physics GREdoes not help applicants from small or less selective schoolsor applicants with a low GPA "stand out". Indeed, having ahigh physics GRE and low GPA is no better than having alow physics GRE score and high GPA in terms of the frac-tion of applicants admitted. Similarly, for average physicsGRE scores, the selectivity or size of the applicant’s insti-tution does not offer any advantage. For the highest scoresthough, attending a smaller or less selective institution doesappear to result in an admissions penalty.

We do find that women and B/L/M/N applicants do havehigher rates of admission based on physics GRE scores.However, given that the departments included in this studywere actively trying to improve the diversity of their graduatestudent population [38], we are unable to attribute that stand-ing out to the physics GRE.

In response to the ETS’s claim that the physics GRE canhelp applicants stand out from other applicants, we do not findevidence to support that claim. In fact, our results suggest theopposite: the physics GRE may penalize applicants due to alow score rather than help applicants due to a high score.

As Small points out, facts and data do not unambiguouslyprescribe a course of action [22] and as other have noted,making such courses of action require a framework of as-sumptions and commitments [45]. Thus, we do not make aspecific recommendation regarding whether the physics GREshould be kept or removed as a result of our work because theanswer to that question depends on the priorities of the de-partment. However, if departments are using the physics GREto identify applicants who might be missed by other metricsto achieve their admissions priorities, we suggest against thispractice as it does not appear to be backed by evidence.

ACKNOWLEDGMENTS

We would like to thank Julie Posselt, Casey Miller, and theInclusive Graduate Network for providing us with the data forthis project. We would also like to thank Rachel Hendersonfor her feedback on this project and guidance on the media-tion and moderation analysis. This project was supported bythe Michigan State University College of Natural Sciencesand the Lappan-Phillips Foundation.

VIII. APPENDIX: ANALYSIS OF THE VARIABLES

In this appendix, we describe the data used to answer re-search questions 2 and 3 to give the reader a better idea ofwhat the distributions of physics GRE scores and GPA in the

data set. Since the data are skewed left and exhibit ceilingeffects (many applicants have 4.0 GPAs or 990 physics GREscores), quartiles are used to describe the various features. Tomaximize the amount of information shown about the data,we use raincloud plots [46, 47], which show the distribu-tion, the density plot, and traditional box plot. Kolmogorov-Smirnov tests suggest the distributions are not significantlydifferent whether we include applicants who may have ap-plied to multiple schools in our data set, so we include possi-ble duplicates in our analysis.

Fig. 11 shows the physics GRE scores and undergradu-ate GPAs of each applicant based on whether they attendeda large undergraduate physics program (top 25% nationallyin yearly physics bachelor‘s degrees) or attended a selectiveuniversity (categorized as most competitive or highly com-petitive based on Barron‘s Selectivity Index). We notice thatthe physics GRE score distributions are shifted to the right forapplicants from large physics departments or selective insti-tutions, signifying higher scores. Indeed, the median physicsGRE scores of applicants from large programs or selectiveinstitutions is nearly 100 points higher than that of appli-cants from smaller or less selective institutions. However, interms, of GPA, the median GPA is approximately the same,regardless of whether the applicant graduated from a larger orsmaller physics department or attended a more or less selec-tive institution.

Fig. 12 shows the physics GRE and undergraduate GPAsby gender and race. As expected, men score higher on thephysics GRE than women do and Asian and white applicantsscore higher than Black, Latinx, Multiracial, or Native appli-cants, though the gaps appear larger than those reported in[6].

When comparing GPAs, we find that men and women havesimilar GPAs, as recently reported in [48] when comparingmen and women’s STEM GPAs. Likewise, our data alsoshows a racial GPA gap with non-B/L/M/N applicants hav-ing a median GPA higher than that of B/L/M/N applicants by0.15.

When looking across both figures, we notice that thephysics GRE score distributions from smaller and less se-lective programs resembles the physics GRE distributions ofwomen and B/L/M/N applicants while the physics GRE scoredistributions of the largest and most selective programs re-sembles the physics GRE score distributions of men and non-B/L/M/N applicants. To see if gender and race are confound-ing variables in our analysis, we examined the fraction ofwomen and B/L/M/N applicants in each group. If this werethe case, the smaller and less selective programs should havea greater fraction of women and B/L/M/N applicants than thelarger and more selective programs.

However, we did not find this to be the case. Applicantsfrom more selective institutions were 16% women while ap-plicants from less selective institutions were 18% women(15% and 14% respectively for B/L/M/N applicants). For in-stitution size, applicants from larger institutions were 16%women compared to 21% women from smaller institutions

15

FIG. 11. Distribution of physics GRE scores and undergraduate GPAs by the size of the undergraduate physics program and institutionalselectivity for each applicant.

FIG. 12. Distribution of physics GRE scores and undergraduate GPAs by gender and whether the applicant identified as a member of racialor ethnic group currently underrepresented in physics.

16

(14% and 17% for B/L/M/N applicants respectively). Thus,it does not appear that differences in who attends (in terms

of gender and race) larger or more selective institutions is re-sponsible for the observed differences in scores.

[1] D. Chari and G. Potvin, Physical Review Physics EducationResearch 15, 023101 (2019).

[2] G. Potvin, D. Chari, and T. Hodapp, Physical Review PhysicsEducation Research 13, 020142 (2017).

[3] J. R. Posselt, Inside graduate admissions (Harvard UniversityPress, 2016).

[4] N. T. Young and M. D. Caballero, in 2019 Physics EducationResearch Conference Proceedings (American Association ofPhysics Teachers, Provo, UT, 2020).

[5] C. Miller and K. Stassun, Nature 510, 303 (2014).[6] C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Silvestrini, and

T. Hodapp, Science Advances 5, eaat7550 (2019).[7] G. L. Cochran, T. Hodapp, and E. E. A. Brown, in Physics

Education Research Conference Proceedings, PER Conference(Cincinnati, OH, 2018) pp. 92–95.

[8] R. Wilson, Doctor of Education in Secondary Education Dis-sertations (2020).

[9] L. Owens, B. Zwickl, and C. Miller, ““Not Required,”“No Re-quired Minimum,” and “Optional” General and Physics GRERequirements: The Impact on Prospective Graduate Students,”(2020), presented at APS March Meeting.

[10] E. M. Levesque, R. Bezanson, and G. R. Tremblay,arXiv:1512.03709 [astro-ph, physics:physics] (2015), arXiv:1512.03709.

[11] American Association of Physics Teacher (2019).[12] “AAS Statement on Limiting the Use of GRE Scores in Grad-

uate Admissions in the Astronomical Sciences,” (2018).[13] L. A. Lopez, “Demographic Effects of Removing the Physics

GRE Requirement in Graduate Admissions,” (2019).[14] R. E. Scherr, M. Plisch, K. E. Gray, G. Potvin, and T. Ho-

dapp, Physical Review Physics Education Research 13, 020133(2017), publisher: American Physical Society.

[15] S.-J. Leslie, A. Cimpian, M. Meyer, and E. Freeland, Science347, 262 (2015), publisher: American Association for the Ad-vancement of Science Section: Report.

[16] K. Langin, Science (2019), 10.1126/science.caredit.aay2093.[17] “About the GRE Subject Tests (For Test Takers),” .[18] “GRE Guide to the Use of Scores,” (2019).[19] N. D. Morrison, W. V. Dixon, C. W. Miller, and T. W. in As-

tronomy IV Graduate School Admissions White Paper Group,Bulletin of the AAS 51 (2020).

[20] E. M. Levesque, R. Bezanson, and G. R. Tremblay, PhysicsToday (2017), 10.1063/PT.5.9090, publisher: American Insti-tute of Physics.

[21] J. R. Posselt, The Review of Higher Education 41, 497 (2018).[22] A. R. Small, arXiv:1709.02895 [physics] (2017), arXiv:

1709.02895.[23] R. M. Baron and D. A. Kenny, Journal of Personality and So-

cial Psychology 51, 1173 (1986).[24] J. J. M. Rijnhart, J. W. R. Twisk, I. Eekhout, and M. W. Hey-

mans, BMC Medical Research Methodology 19, 19 (2019).[25] A. F. Hayes and M. Scharkow, Psychological Science 24, 1918

(2013), publisher: SAGE Publications Inc.[26] N. Amos and A. F. Heckler, Physical Review Physics Educa-

tion Research 14, 010105 (2018), publisher: American Physi-

cal Society.[27] S. Ditlevsen, U. Christensen, J. Lynch, M. T. Damsgaard, and

N. Keiding, Epidemiology 16, 114 (2005).[28] L. S. Freedman, Journal of Statistical Planning and Inference

96, 143 (2001).[29] K. J. Preacher, D. D. Rucker, and A. F. Hayes, Multivariate

Behavioral Research 42, 185 (2007).[30] M. B. Weissman, Science Advances 6, eaax3787 (2020), pub-

lisher: American Association for the Advancement of ScienceSection: Technical Comments.

[31] S. Nicholson and P. J. Mulvey, Roster of Physics Departmentswith Enrollment and Degree Data, 2017, Tech. Rep. (AmericanInstitute of Physics, 2018).

[32] S. Nicholson and P. J. Mulvey, Roster of Physics Departmentswith Enrollment and Degree Data, 2018, Tech. Rep. (AmericanInstitute of Physics, 2019).

[33] National Center for Education Statistics, “NCES-Barron’s Ad-missions Competitiveness Index Data Files: 1972, 1982, 1992,2004, , 2008, 2014,” (2017), library Catalog: nces.ed.gov Pub-lisher: National Center for Education Statistics.

[34] R. Chetty, J. Friedman, E. Saez, N. Turner, and D. Yagan, Mo-bility Report Cards: The Role of Colleges in IntergenerationalMobility, Tech. Rep. w23618 (National Bureau of EconomicResearch, Cambridge, MA, 2017).

[35] J. Blue, A. L. Traxler, and X. C. Cid, Physics Today 71, 40(2018).

[36] T. L. Williams, “’Underrepresented Minority’ ConsideredHarmful, Racist Language,” (2020), library Catalog:cacm.acm.org.

[37] R. T. Teranishi, New Directions for Institutional Research2007, 37 (2007), publisher: Wiley Online Library.

[38] J. Posselt, T. Hernandez, G. Cochran, and C. Miller, Journal ofWomen and Minorities in Science and Engineering 25 (2019),10.1615/JWomenMinorScienEng.2019027863.

[39] F. J. Massey Jr., Journal of the American Statistical Association46, 68 (1951).

[40] C. Miller and J. Posselt, “Broadening Participation in GraduateEducation through Holistic Review,” (2018), presented at theBridge Program and National Mentoring Community Confer-ence.

[41] J. D. Kent and M. T. McCarthy, Washington DC: Council ofGraduate Students (2016).

[42] A. Rudolph, G. Basri, M. Agüeros, E. Bertschinger, K. Coble,M. Donahue, J. Monkiewicz, A. Speck, K. Stassun, R. Ivie,C. Pfund, and J. Posselt, Bulletin of the AAS 51 (2020).

[43] K. Dunlap, “Journey to a more holistic admissions review pro-cess by implementing an evaluation rubric,” (2018).

[44] M. A. Wilson, M. A. Odem, T. Walters, A. L. DePass,and A. J. Bean, CBE Life Sciences Education 18 (2019),10.1187/cbe.18-06-0103.

[45] J. R. Searle, The Philosophical Review 73, 43 (1964), pub-lisher: [Duke University Press, Philosophical Review].

[46] M. Allen, D. Poggiali, K. Whitaker, T. R. Marshall, and R. A.Kievit, Wellcome Open Research 4, 63 (2019).

[47] K. Whitaker, T. R. Marshall, T. V. Mourik, P. A. Mar-

17

tinez, D. Poggiali, H. Ye, and M. Klug, “RainCloud-Plots/RainCloudPlots: WellcomeOpenResearch,” (2019).

[48] K. M. Whitcomb and C. Singh, arXiv:2003.04376 [physics]

(2020), arXiv: 2003.04376.