21
[RESAMPLED RANGE OF WITTY TITLES] Understanding and Using the NRC Assessment of Doctorate Programs Lydia Snover, Greg Harris & Scott Barge Office of the Provost, Institutional Research

[ Resampled Range of Witty Titles]

  • Upload
    adolfo

  • View
    35

  • Download
    0

Embed Size (px)

DESCRIPTION

[ Resampled Range of Witty Titles]. Understanding and Using the NRC Assessment of Doctorate Programs. Lydia Snover, Greg Harris & Scott Barge Office of the Provost, Institutional Research Massachusetts Institute of Technology • 2 Feb 2010. Overview. Overview. - PowerPoint PPT Presentation

Citation preview

Page 1: [ Resampled  Range of Witty Titles]

[RESAMPLED RANGE OF WITTY TITLES]Understanding and Using the NRC Assessment of Doctorate Programs

Lydia Snover, Greg Harris & Scott BargeOffice of the Provost, Institutional Research

Massachusetts Institute of Technology • 2 Feb 2010

Page 2: [ Resampled  Range of Witty Titles]

Overview1. Background & Context2. Approaches to Ranking3. The NRC Model: A Modified Hybrid4. Presenting & Using the Results

OV

ERV

IEW

2*NB: All figures/data in this presentation are used for illustrative purposes only and do not represent a known institution.

Page 3: [ Resampled  Range of Witty Titles]

Background & Context

A. History of NRC RankingsB. MIT Data Collection Process

INTR

OD

UC

TION

3

Page 4: [ Resampled  Range of Witty Titles]

Participating MIT ProgramsAeronautics and Astronautics Computer Science

Applied Biosciences Economics

Applied Mathematics Electrical and Computer EngineeringAstrophysics and Astronomy and Planetary Science Geology and Geochemistry and Geophysics

Atmospheres, Oceans & Climate including MIT/WHOI Oceanography History, Theory and Criticism

Biological Engineering, Health Science and Technology Linguistics

Biology/Biochemistry and Biophysics Material Sciences and Engineering

Biology/Cell and Developmental Mathematics

Biology/Genetics and Genomics Mechanical Engineering

Chemical Engineering Neuroscience

Chemistry Operations Research

Civil and Environmental Engineering Philosophy

Cognitive Science Physics

Computer Engineering Political Science 4

INTR

OD

UC

TION

Page 5: [ Resampled  Range of Witty Titles]

Section 2

2. Approaches to Ranking

APPR

OA

CH

ES TO R

AN

KIN

G

5

Page 6: [ Resampled  Range of Witty Titles]

How do we measure program quality?• Use INDICATORS (“countable” information)

to compute a rating– Number of publications– Funded research per faculty member– Etc.,

• Try to quantify more subjective measures through an overall PERCEPTION-BASED RATING– Reputation– “Creative blending of interdisciplinary perspectives”

APPR

OA

CH

ES TO R

AN

KIN

GS

6

Page 7: [ Resampled  Range of Witty Titles]

Section 3

3. The NRC Approach

THE N

RC

APPR

OA

CH

7

Page 8: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?The NRC used a modified hybrid of the two basic approaches:•In total, a 4-step process, indicator based, by field•Process results in 2 sets of indicator weights developed through faculty surveys:

– “Bottom up” –importance of indicators– “Top-down” – perception-based ratings of a

sample of programs•Multiple iterations (re-sampling) to model “the variability in ratings by peer raters.” *

THE N

RC

APPR

OA

CH

8*For more information on the rationale for re-sampling, see pp. 14-15 of the NRC Methodology Report

Page 9: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?STEP 1: Gather raw data from institutions, faculty & external sources on programs. Random University (RU) submitted data for its participating doctoral programs.

THE N

RC

APPR

OA

CH

9

RU PhysicsIndicator Value# publications/fac 1.07

# citations/article 1.17

Median GRE 746

Gender diversity 44%

Time to degree 5.67

RU ChemEIndicator Value# publications/fac 1.07

# citations/article 1.17

Median GRE 746

Gender diversity 44%

Time to degree 5.67

RU EconomicsIndicator Value# publications/fac 1.07

# citations/article 1.17

Median GRE 746

Gender diversity 44%

Time to degree 5.67

NRC

Page 10: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?STEP 2: Use faculty input to develop weights:– Method 1: Direct prioritization of indicators--

“What characteristics (indicators) are important to program quality in your field?”

THE N

RC

APPR

OA

CH

10

Program Faculty Quality Most Impt Indicator (Mark 4)

Top 2 Indicators

Number of publications per faculty member

Number of citations per publication

Racial/ethnic diversity of the student population

Avg. # of Ph.D.s granted over last 4 years

Gender diversity of program faculty

… … …

Direct WeightsIndicator 1= 0.2

Indicator 2= 0.0

Indicator 3= 0.1

Indicator 4= 0.1

Indicator 5= 0.2

Calculations

Page 11: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?STEP 2: Use faculty input to develop weights:– Method 2: A sample of faculty each rate a sample of 15

programs from which indicator weights are derived.

THE N

RC

APPR

OA

CH

11

Program #2: Yale University Economics

Some Facts About the Program:# of Ph.D.s 2001-2006: _____ Faculty % Female: _____Median Time to Degree: _____ Current Faculty List, etc.

Program #2: Yale University EconomicsProgram #2: Ivy University Economics

Program #1: Land Grant University Economics

Some Facts About the Program:# of Ph.D.s 2001-2006: XX Faculty % Female: YY%Median Time to Degree: Z.Z Current Faculty List, etc.

On a scale from 1 to 3, indicate your familiarity with this program?___ 1 (Little or none)___ 2 (Some)___ 3 (Considerable)

On a scale from 1 to 6, how would you rate this program?___ 1 (Not adequate for doc educ.)___ 2 (Marginal)___ 3 (Adequate___ 4 (Good)___ 5 (Strong)___ 6 (Distinguished)___ 9 (Don’t know well enough)

Regression-based

WeightsInd. 1= 0.3

Ind. 2= 0.04

Ind. 3= 0.2

Ind. 4= 0.15

PrincipleComponents& Regression

Page 12: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?STEP 3: Combine both sets of indicator weights and apply them to the raw data:

THE N

RC

APPR

OA

CH

12

Direct Weights

Ind. 1= 0.3

Regression-based

WeightsInd. 1= 0.2

Combined Weights

Ind. 1= 0.25

DATAIndicatorValue# publications/fac 1.07

# citations/article 1.17

Median GRE 746

Gender diversity 44%

Time to degree 5.67

X= Rating

RANKEDLIST

1. Ivy Univ (98)

2. Random Univ (94)

3. Private Univ (91)

4. Land Grant U (88)

5. Univ of State (87)

Page 13: [ Resampled  Range of Witty Titles]

So how does NRC blend the two?STEP 4: Repeat steps 500 times for each field

THE N

RC

APPR

OA

CH

13

A) Randomly draw ½ of faculty

“important characteristics”

surveys

C) Randomly draw ½ of faculty

program rating surveys

B) Calculate “direct” weights

D) Compute “regression-

based” weights

E) Combine weights

F) Repeat (A) – (E) 500 times to develop 500 sets of weights for each field

G) Randomly perturb institutions’ program data 500 times*

H) Use each pair of iterations (1 perturbation of data (G) + 1 set of weights (F)) to rate programs and

prepare 500 ranked lists

I) Toss out the lowest 125 and highest 125 rankings for each

program and present the remaining range of rankings

*For more information on the perturbation of program data, see pp. 50-1 in the NRC Methodology Report

Page 14: [ Resampled  Range of Witty Titles]

Section 4

4. Presenting & Using the Results

RESU

LTS

14

Page 15: [ Resampled  Range of Witty Titles]

What are the indicators?Program Faculty Quality Student Characteristics Program Characteristics# of publications per faculty member

Median GRE of entering students

Avg. # Ph.D.s granted in past 5 years

# of citations per faculty member

% students receiving full financial support

% entering students who complete

Receipt of extramural grants for research

% students with portable fellowships

Time to degree

Involvement in interdisciplinary work

Racial/ethnic diversity of student population

Placement of students after grad

Racial/ethnic diversity of the program faculty

Gender diversity of student population

% students with individual work space

Gender diversity of the program faculty

High % of international students

% of health insurance premiums covered

Reception of peers of a faculty member’s work as measured by honors/awards

# of student support activities provided

15

RESU

LTS

Page 16: [ Resampled  Range of Witty Titles]

What will the results look like?• TABLE 1: Program values for each indicator plus

overall summary statistics for the field

RESU

LTS

16

RU Econ All Economics Programs (n=117)Indicator Value Min 25th

%tile75th

%tileMax Std.

Dev.# publications/fac 1.07 .049 .369 .655 1.257 .246

# citations/article 1.17 .153 .684 1.771 5.485 1.002

Median GRE 746 353 740 790 800 55

% female students 44% 0% 28.6% 42.9% 76.9% 12%

% female faculty 12.5% 0% 10.5% 21.1% 66.7% 9.9%

Time to degree 5.67 3 5 6 8 .8

Page 17: [ Resampled  Range of Witty Titles]

What will the results look like?• TABLE 2: Indicators and indicator weights – one

standard deviation above and below the mean of the 500 weights produced for each indicator through the iterative process (and a locally calculated mean)

RESU

LTS

17

Indicator Minus 1 SD Plus 1 SD Calculated Mean

# publications/fac 0.130 0.134 0.132# citations/article 0.294 0.267 0.2805Median GRE 0.091 0.089 0.09% female students -0.029 -0.043 -0.036% female faculty n.s.* n.s.*

Time to degree -0.026 -0.031 -0.0285…

*n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.

Page 18: [ Resampled  Range of Witty Titles]

What will the results look like?• TABLE 3: Range of rankings for RU’s Economics

program alongside other programs, overall and dimensional rankings

RESU

LTS

18

Institution Overall Research Activity

Diversity of Acad Environ.

Student Supp/Outcomes

25th %tile

75th %tile

25th 75th 25th 75th 25th 75th

Ivy Univ 30 36 31 32 37 41 28 31

Univ of State 45 54 40 42 42 50 45 46

Random Univ 45 56 38 42 47 51 43 47

Private Univ 48 57 41 42 40 47 45 49

Land Grant U 55 63 59 64 48 50 54 61

Total # of ranked programs = 117

Page 19: [ Resampled  Range of Witty Titles]

What will the results look like?• TABLE 4: Range of rankings for all RU’s programs

RESU

LTS

19

Program Overall Research Activity

Diversity of Acad Environ.

Student Supp/Out

comes

1995 NRC

Ranking

2009 USNWR Ranking

25th 75th 25th 75th 25th 75th 25th 75th

Linguistics 45 56 … 40 38Material Sciences and Engineering

25 26 24 24

Mathematics 21 23 23 25Mechanical Engineering

32 36 33 33

Neuroscience 34 35 34 35Operations Research

54 56 56 53

Philosophy 43 44 … 44 43

Page 20: [ Resampled  Range of Witty Titles]

Q&AQ

&A

20

Page 21: [ Resampled  Range of Witty Titles]

For more information…• The full NRC Methodology Reporthttp://www.nap.edu/catalog.php?record_id=12676

• Helpful NRC Frequently Asked Questions Pagehttp://sites.nationalacademies.org/pga/Resdoc/PGA_051962

RESO

UR

CES

21