1

De-mystifying The U.S. News Rankings:

How To Understand What Matters,What Doesn’t, and

What You Can Actually Do About It

Joe Brennan, Ohio UniversityRobert Brodnick, Univ. of the Pacific

Diana Pinckley, Zehno Cross-Media Communications

2

Overview

1. How does U.S. News rank schools?2. Can you affect the peer assessment

score?3. Do the rankings actually differentiate

schools?4. Do prospective students care about

rankings?5. What does all this mean for

marketers?6. What do you think?

3

1. U.S. News ranking system

• Schools are grouped by Carnegie classification

• Editors get data on 15 variables– 14 are objective– 1 (peer assessment) is subjective

• Each variable is weighted by the editors

• A composite score is calculated for each school

• Rank-order is based on composite score

4

Table 1. U.S. News Ranking Model for National Universities Category

weight Sub-factor

weight Relative

weight

Peer assessment 25% 25%

Survey of presidents, provosts, admissions directors Retention and graduation 20%

Freshman retention, fall to fall (4-year average of freshman cohorts) 20% 4%

6-year graduation rates (4r-year average of freshman cohorts) 80% 16%

Faculty resources 20%

Percent of undergraduate classes with fewer than 20 enrolled 30% 6%

Percent of undergraduate classes with 50 or more enrolled 10% 2%

Faculty pay and benefits, 2-year average, adjusted for regional cost-of-living differences 35% 7%

Percent of full-time faculty with terminal degree 15% 3%

Student/faculty ratio, FTE basis 5% 1%

Proportion of FTE faculty members that are full-time 5% 1%

Student selectivity 15%

SAT/ACT (middle 50 percent) 50% 8%

Percent 2005 freshmen in top 10 percent of HS class 40% 6%

Freshman acceptance rate 10% 2%

Financial resources Average spending per FTE student (2-year average) 10% 10% Alumni giving rate 5% 5% Percent of undergraduate alumni of record who made a gift (2-yr. ave.)

Graduation rate performance 5% 5%

Difference between predicted and actual 6-year graduation rate; predictors are SAT/ACT scores and expenditures per student

Total 100% 100%

5

Peer Assessment Score

• Survey of presidents, provosts, chief enrollment officers

• Rate academic programs• Scale of 1 (“marginal”) to 5

(“distinguished”)• Single most important factor in

ranking• Comprises 25% of overall score

6

Peer Assessment Score

Before spending $$$ to move this measure, you need to know:

1. Is it valid (does it actually measure academic quality)?

2. Can it be influenced?

7

What does the Peer Assessment measure?

• We compared peer assessment scores to five other possible indicators of quality:– Median SAT/ACT score– Percent of classes under 20 students– Graduation rate– Alumni giving rate– Public or private control of institution

8

Relationship of other indicators to Peer Assessment Score

Indicator Relationship to Peer Score

Coefficient

Graduation Rate

Direct, large 1.2953

Alumni Giving Direct, moderately large

0.8420

Public Direct, moderately small

0.4332

Freshmen Test Scores

Direct, small 0.0035

Small Class Sizes

No relationship N/A

9

Regression Model

Predicted Score = -2.2149 + 1.2953(Graduation Rate) + 0.8420(Giving Rate) + 0.4332(Control) + 0.0035(Median SAT Score) + e

• How to use this:– Messaging strategies– Compare your predicted to your actual

score

10

Behavior of the Peer Score

• Compared peer scores in 2000 and 2005 for national universities - how much did they change?

• Distribution of peer scores

11

Figure A-1. Distribution of changes in

peer assessment scores, 2000-05

4 7

74

113

46

4

≤ -0.3 -0.2 -0.1 0 +0.1 +0.2

Change in Score

95% of all schools had same score or changed +/- 0.1

12

Figure A-2. Distribution of fi ve-year

mean scores, 2000-05

9

58

83

4227

15 14

1.5-1.9

2.0-2.4

2.5-2.9

3.0-3.4

3.5-3.9

4.0-4.4

4.5-5.0

5-year mean score

(μ = 3.00 σ = 0.73)

Scores cluster tightly around the midpoint and skew towards the higher end of the scale.

13

Do the Rankings Tell Us How Schools are Different?

• Institutional researchers collect data on 100s of variables (magazine uses 15)

• Three questions:1. What factors underlie this vast array of

data?2. Can we create a model to differentiate

schools?3. Does well does the U.S. News model fit

the observed data?

14

Dimensions of higher education

• Examined 170 variables on 247 national universities.

• Pared the data set to 33 variables• Identified seven factors underlying

those variables

15

Factor correlations

IO Control DiversityResearc

h AidAffluenc

e Size

IO 1 -0.349 -0.113 0.333 -0.162 0.501 -0.194

Control -0.349 1 -0.061 0.1 -0.077 -0.341 -0.264

Diversity -0.113 -0.061 1 -0.026 0.058 0.101 -0.027

Research 0.333 0.1 -0.026 1 -0.189 0.315 -0.239

Aid -0.162 -0.077 0.058 -0.189 1 -0.252 0.202

Affluence 0.501 -0.341 0.101 0.315 -0.252 1 -0.059

Size -0.194 -0.264 -0.027 -0.239 0.202 -0.059 1

16

Variable-Factor Relationships

IO Control Research Affluence Aid Size DiversityPERST RATE CLASS < 20 EXP RSCH PCT END / FTES FR AID PCT SUM UNIT % NON WHITE

GRAD RATE PCT UG ENR EXP RSCH / FTES ENDOWMENT INST GRANT % FTES TOTAL FED GRANT %

USN RANK CLASS > 50 EXP INST PCT EXP SVC / FTES LOAN % EXP TOTAL

SAT MEDIAN SF RATIO PCNT FT FAC STATE GRANT %

TOP 10 HS TUITION

AAUP SALARY

REP SCORE

SELECTIVITY

GIVING RATE

AID FR AMT

EXP / INST FTES

EXP / FTES

17

Simple Orthogonal Model

.85FR_PERST_RATE

.85FR_GRAD_RATE_AVG

.88USN_RANK

.90SAT_MEDIAN

.81TOP_10_HS

.76AAUP_SALARY

.83USN_REP_SCORE

.48SELECTIVITY

.58GIVING_RATE

.49AID_FR_AMT

.53EXP_INST_FTES_RATIO

.54EXP_FTES_RATIO

.58CLASS_UNDER_20

.58ENR_PCT_UG_TOTAL

.25CLASS_OVER_50

.72STUD_FAC_RATIO

.70TUITION

.96EXP_RSCH_PCT

.72EXP_RSCH_FTES_GR_PR

.49EXP_INST_PCT

.28USN_PCNT_FT_FAC

1.04END_FTES_RATIO

.72ENDOWMENT

.35EXP_STSVC_FTES_RATIO

2.52AID_FR_AID_PCT

.09AID_FR_INST_GRANT_PCT

.06AID_FR_LOAN_PCT

.12AID_FR_STATE_GRANT_PCT

.50SUM_UNIT

.66FTES_TOTAL

IO

.51EXP_TOTAL

29.79ENR_PCT_UG_NON_WHITE

.01AID_FR_FED_GRANT_PCT

CONTROL

RESEARCH

AFFLUENCE

AID

SIZE

DIVERSITY

e1

e2

e3

e4

e6

e5

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e19

e20

e21

e22

e23

e24

e25

e26

e28

e27

e29

e30

e31

e32

e33

.92.92-.94.95.90.87.91-.69.76.70

.84

-.50-.76.76

.53-.70.85.98

1.02.85.59

1.59.30.24.34

.71

.81

.72

5.46.09

MODEL = OrthogonalGFI = .438

AGFI = .365CFI = .556

RMSEA = .185

.73

.74

-.85

18

Model Used byU.S. News FR_PERST_RATE

FR_GRAD_RATE_AVG

USN_RANK

SAT_MEDIAN

TOP_10_HS

AAUP_SALARY

USN_REP_SCORE

SELECTIVITY

GIVING_RATE

CLASS_UNDER_20

CLASS_OVER_50

STUD_FAC_RATIO

USN_PCNT_FT_FAC

SELECTIVITY

FAC RESOURCES

RETENTION

e11

e21

e31

e41

e61

e51

e71

e81

e91

e131

e151

e161

e211

MODEL = USN onlyGFI = .000

AGFI = .000CFI = .000

RMSEA = .402USN_GRAD

EXP_UG_G

e34

e351

1

1

1

1

19

Model showing what U.S. News

should use.87

FR_PERST_RATE.86

FR_GRAD_RATE_AVG.89

USN_RANK.89

SAT_MEDIAN.80

TOP_10_HS.75

AAUP_SALARY.84

USN_REP_SCORE.46

SELECTIVITY.57

GIVING_RATE

.34CLASS_UNDER_20

.88CLASS_OVER_50

.00STUD_FAC_RATIO

.21USN_PCNT_FT_FAC

IO

SIZE

e1

e2

e3

e4

e6

e5

e7

e8

e9

e13

e15

e16

e21

MODEL = USN CorrectedGFI = .606

AGFI = .486CFI = .718

RMSEA = .228

.00USN_GRAD

.51EXP_UG_G

e34

e35

.58

-.94

-.46

.93

.93-.94.95.90.87.92

-.68

.76

.72

20

Model which best explains observed data

.93FR_PERST_RATE

.92FR_GRAD_RATE_AVG

.88SAT_MEDIAN

.83USN_REP_SCORE

.52SELECTIVITY

.60GIVING_RATE

.91AID_FR_AMT

.92EXP_INST_FTES_RATIO

.95EXP_FTES_RATIO

.49CLASS_UNDER_20

.54ENR_PCT_UG_TOTAL

.47CLASS_OVER_50

.65STUD_FAC_RATIO

.94TUITION

.88EXP_RSCH_PCT

.53EXP_INST_PCT

.48END_FTES_RATIO

.46EXP_STSVC_FTES_RATIO

.38AID_FR_AID_PCT

.84AID_FR_INST_GRANT_PCT

.86FTES_TOTAL

IO

.94EXP_TOTAL

CONTROL

RESEARCH

AFFLUENCE

AID

SIZE

e1

e2

e4

e7

e8

e9

e10

e11

e12

e13

e14

e15

e16

e17

e18

e20

e22

e24

e25

e26

e30

e31

.96.96

.67

.57-.25.45.48

1.08

-.50-.74.70

-.73

.94

.77

.34

.62

.78

.81

.77

-.16-.21

-.61

.46

.35

.72

-.02

.57

-.08

.76

.03

.09

.33

-.13

.38

-.23

MODEL = Best FitGFI = .774

AGFI = .661CFI = .876

RMSEA = .131

-.20

-.11

.47-.20.93.03.09

.38

.46

-.54

.07

.23

.21

.18

.30

-.52.38-.321.05.97

-.10-.49

.40

.51-.26

.45

.28

21

Model Summary

Model GFI AGFI CFI RMSEA

Simple Orthogonal 0.438 0.365 0.556 0.185

USN Only (what they do use) 0.000 0.000 0.000 0.402

USN Corrected (what they should use) 0.606 0.486 0.718 0.228

Best Fit 0.774 0.661 0.876 0.131

22

What does this mean?

• U.S. News model does not explain observed data (only 2 of 7 known factors)

• The kinds of data collected about schools doesn’t help individual students find the school which “fits” them best.

• New measures are needed:– Learning– Service to society– Satisfaction of stakeholder needs and wants

23

Do Rankings Matter to Students?

• Yes and no (but mostly no).• Studies show little impact on

undergraduate decision making.• May be more influential for graduate

students.

24

How Undergrads See Rankings

• Art & Science Group (2002)– Only 20% can recall reports or articles about

rankings– Just 8% used rankings info in college

decision

• Lipman Hearne (2006)– Two-thirds don’t use rankings as info source– Mid-Atlantic students and “academic

superstars” are more likely to use rankings

25

How Grad Students See Rankings

• GMAC studies show:– Rankings are strongest influence of MBA

prospects– Students older than 27, and men, are

more likely to be influenced by rankings– For younger students, Web site more

important– For women, personal contact more

important

26

Focusing on What Matters

• Rankings aren’t going away – but don’t overvalue them

• Use the facts about rankings with your administrators

• Use your data to find competitive advantages that really matter

• Deliver messages which differentiate your school and communicate meaningful benefits

27

Questions?

• Diana Pinckley, Zehno Cross-Media Communications

• Rob Brodnick, University of the Pacific

• Joe Brennan, Ohio University