Tan 1986 The assessment of quality in higher education- A critical review of the literature and research.pdf

7/28/2019 Tan 1986 The assessment of quality in higher education- A critical review of the literature and research.pdf

http://slidepdf.com/reader/full/tan-1986-the-assessment-of-quality-in-higher-education-a-critical-review-of 1/43

THE ASSESSMENT OF QUALITY IN

HIGHER EDUCATION: A Critical Review

of the Literature and Research

David L. Tan

® , ° ù ° , ù ù , ° , , , ù ° ° ù , ù , , ù , ù , ° ° , ù = , , , , . . . . , ° ù ù ù ° , , , = = , ° , ù = , , ~ ° , , , , , , , , ° ~ ~ ° ° ù = , ~ ® , = ° ° , , . , ° ù , = ~ , °

, , , , , , ° , = , , , ° , , , , ° , , , , , , , = , , , , ù , ° ° ° , ° , , , ° ° , , , , ° . . . . , ~ , ù , , , , , ù , , , , , ù , , ù ° , , , = , , , ù , , , , ° ù ° , o , ,

The art ic le discusses the eiusive concept of qual i ty and the problems involved in at-tempting to assess it. The article provides a comprehensive yet cdtical review of theliterature and em pirical research studies unde rtaken on the subje ct matter. Three type sof studies are iden tif ied and a re reviewe d relative to their m ajor f indings, strengths, andweakn esses. Issues that rem ain to be resolved are also identif ied and discussed .

° , , , , ° , , , ù , , , = , ù , , , , , , , , , , , , ù ù ° ° , , ù , , , , , , ù , , , , , , , , ° , , , , , , , , . . . . ° . . . . . , , , ù , , = , ù ° , ° , . . . . , , , ° °

Q u al i t y . . . you know what i t i s, ye t you don ' t know what i t is . But that ' sself-contradictory. But some things are bet tet than others, that is , they

have more quali ty. But when you try to say what the quali ty is , apart

f lora the things that have i t , i t al l goes poof! There 's nothing to talkabou t . But i f you can 't see what Quali ty is , how do you know w hat i t is ,

or ho w do y ou kn ow tha t i t even exists? I f no one knows w hat i t is , thenfor all practical purposes it doesn't exist at all . But for all practical

purposes i t does exist. Wh at else are grades based on ? Wh y else would

people pay fortunes for sorne things and throw others in the t rash pi le?Obviously some things are bet ter than o th e rs . . , hut what ' s the "bet ter -

ness"? . . . So round and round you go, spinning mental wheels andnowhere f inding anyplace to get t ract ion. What the hel l is Quali ty?(Pirsig, 1974, p. 184).

" W h a t t h e h e l l is q u a l i t y ? " a s k e d P i rs i g . I n d e e d , t h is is a q u e s t i o n t h a t h a s

b e e n d is c u s se d , d e b a t e d , a n d p u z z l e d a b o u t s in c e t h e b e g i n n in g o f h u m a n -

k i n d . Y et n o a b s o l u t e a n s w e r t o t h e q u e s t i o n is p o s s ib l e . T h e b a s i c p r o b l e m

u n d e r l y i n g t h i s d i f f i c u l t y is t h e l a c k o f a t h e o r y o f q u a l i t y a n d t h e ä b s e n c e

o f a g r e e m e n t o n t h e c o n c e p t i ts e lf . S i n c e q u a l i t y i s m u l t i f a c e t e d a n d v a r ie s

w i t h i n d i v i d u a l p e r c e p t i o n , i t c a n n o t b e u n i v e r s a l l y a g r e e d u p o n . A t t h e

David L. Ta n, Palmer College of Chiropractic, 1000 Brady St., Davenport, Iowa 52803.

Research in High er Edu cation © 1986 Ag ath on Press, Inc. Vol. 24, No. 3

223



224 T AN

s a m e t im e , h o w e v e r, w e i n t u it i v e ly k n o w o f a w i d e r a n g e i n q u a l i t y a m o n g

m a n y t h in g s w e c o n f r o n t , u se , o r d e a l w i t h o n a r e g u la r b a s i s - f r o m a p p l i-

a n c e s a n d a u t o m o b i l e s t o f o o d a n d b o o k s .W h y i s q u a l i t y a s a c o n c e p t s o e lu s iv e ? T h e m a j o r r e a s o n r es ts in t h e f a c t

t h a t w e d i f f e r i n o u r p e r c e p t i o n o f q u a l i ty . Q u a l i ty , l i k e b e a u t y , i s i n t h e e y e

o f t h e b e h o l d e r ; i t h a s a d i f f e r e n t m e a n i n g f o r d i f f e r e n t p e o p le . F o r e x a m p l e ,

a T o y o t a a u t o m o b i l e m a y b e r e g a r d e d b y s o m e t o b e o f h i g h q u a l it y b e c a u se

o f i t s e n g i n e r e li a b i li t y a n d e x c e ll e n t m a i n t e n a n c e r e c o r d , w h i l e a F e r r a r i

m a y b e c e l e b r a t e d b y o t h e r s f o r i t s s p o r t y l o o k a n d h i g h s p e e d c a p a b i l i t y ;

o n e f o o t b a l l t e a m m a y b e j u d g e d t o b e o f q u a l i ty b e c a u s e o f i ts w i n n i n g

r e c o r d , w h i le a n o t h e r m a y b e s i m i l a r ly ju d g e d f o r i ts i m p r e s si v e o f f e n s i v e

y a r d a g e g e n e r a t e d p e r g a rn e . S o m e p e o p l e m a y ju d g e m o v i e s a n d t e l e v is i o n

s h ow s t o b e o f q u a l i t y b e c au s e t h e y c o n t a i n n u m e r o u s q u i c k - p a c e d s c e n es ,

w h i le o t h e r s m a y j u d g e t h e m o n t h e b a s is o f s o li d s to r y c o n t e n t a n d s u p e r b

a c t in g . A s t h e s e e x a m p l e s s u g g es t , n o a b s o l u t e c r i t e ri o n o r m e a s u r e m e n t o f

q u a l i t y i s p o s s i b le .

W e k n o w w e c a n n o t u n i v e r s a l l y a g r e e o n t h e c o n c e p t o f q u a l i t y . I f t h e

c o n c e p t i s s o e lu s iv e , w h y a r e w e s o o b s e s s e d w i t h r a n k i n g t h i n g s b a s e d o n

q u a l i t y ? W h y i s i t s o i m p o r t a n t t h a t w e k n o w w h i c h t e l e v i s i o n s h o w o r

m o v i e i s t h e b e s t d u r i n g a g i v e n w e e k , w h i c h c o l l e g e f o o t b a l l t e a m i s r a t e d

f i rs t in t h e n a t i o n , w h i c h b o o k is t h e t o p - s e ll e r o f th e w e e k , a n d w h i c h

r e s t a u r a n t w e o u g h t t o v is it f o r q u a l i t y d i n in g ? S u c h a n o b s e s s i o n is s p e c u -l a t e d t o b e i n h e r e n t i n o u r d e m o c r a t i c c u l t u r e , i n w h i c h c o m p e t i t i o n a n d

e x c e ll e n ce a r e c e n t ra l v al u e s . W h a t e v e r th e r e a s o n , t h e f u n d a m e n t a l p o i n t i s

t h a t w h e n w e t a n k t h i n g s , w e a r e e s s e n t i a l l y i m p l y i n g t h a t s o m e k i n d o f

c o n s i s t e n c y ex i st s i n t h e p e r c e p t i o n o f q u a l it y . W h i l e n o a b s o l u t e c o n s i s t en c y

i s p o s s i b l e , t h e r e m a y b e a f e w a t t r i b u t e s o f q u a l i t y u p o n w h i c h w e c a n a l l

a g re e . I f s u c h i s t h e c a se , t h e r e is h o p e t h a t w e c a n b e t t e r u n d e r s t a n d q u a l it y .

I n h i g h e r e d u c a t i o n , m o s t a c a d e m i c s h o p e f o r s o m e c o n s i s te n c y i n th e

p e r c e p t io n o f q u a l i t y e v en th o u g h t h e h i g h e r e d u c a t i o n e n v i r o n m e n t i s c h a r -

a c t e r i z e d b y d i v e r si ty i n p e o p l e a n d i n s t it u t io n s . W h e n H a r v a r d U n i v e r s i t y is

j u d g e d t o b e a p r e m i e r i n s t i tu t i o n i n th e n a t i o n , m o s t a c a d e m i c s r e a li z e t h ev a r i a n c e i n t h e c r i te r i a u s e d b y p e o p l e t o d e t e r m i n e t h a t s t a t u s . S o m e p e o p l e

m a y o f f e r th e e x c e l le n c e o f H a r v a r d ' s f a c u l t y a n d c u r r i c u l a a s e x p l a n a t io n s

f o r t h e i r c h o i c e w h i le o t h e r s H a r v a r d ' s h i g h l y a b l e s t u d e n t s o r i t s c o n s i d e r-

a b ! e f i n a n c i a l re s o u r c e s . S u c h a r e q u i t e d iv e r s e o p i n i o n s . D e s p i t e t h e d i v e r-

s ity , i t s e e m s lo g i c a l t h a t i f s o m e p o p u l a r o p i n i o n s c a n s o m e h o w b e c a t e g o -

r i z e d i n t o a s e t o f d e t e r m i n a n t s , t h e r e i s h o p e t h a t w e c a n r e a c h a g e n e r a l

a g r e e m e n t o n t h e q u e s t i o n o f q u a l it y . S u p p o s i n g s u c h a s e t o f d e t e r m i n a n t s

o r a t t r ib u t e s o f q u a l i t y c a n b e c o n s i s t e n t l y i d e n t i f ie d f o r H a r v a r d , d o e s i t

n e c e s s a r il y m e a n t h a t t h e q u a l i t y o f a l l t y p e s o f h i g h e r e d u c a t i o n i n s t i t u t io n s

c a n b e j u d g e d b a s e d o n t h e s a m e d e t e r m i n a n t s ? I f so , s h o u l d a ll i n s t it u t io n s

e m b r a c e H a r v a r d a s a m o d e l o f e x c el le n c e?



A S S E S S M E N T O F Q U A U T Y 2 2 5

Ev i d en t l y , n o t a l l i n s t i t u t i o n s s h o u l d emb r ace t h e H a r v a r d t r ad i t i o n n o r

t h i n k t h a t d e t e r mi n an t s o f q u a l i t y f o r H a r v a r d a r e n eces s a r il y ap p l i cab l e to

t h em. C o mmu n i t y co l l eg es , f o r ex amp l e , d i f f e r f r o m H ar v a r d i n t h e i r m i s -s ions , goa l s , cu r r i cu l a r em phas i s , an d s tuden t c lien te le , bu t t hey a l so serve ä

v i ta l f u n c t i o n i n o u r s o c ie ty . U n l i k e H a r v a r d , t h ey o f f e r a cu r r icu l u m s t r o n g

i n th e t e ch n i ca l an d v o ca t i o n a l a r ea s . M o r e i mp o r t an tl y , t h ey p r o v id e h i g h e r

ed u c a t i o n o p p o r t u n i t i e s t o w o r k i n g ad u l t s , s en i o r c it iz en s , an d o t h e r p eo p l e

w h o o t h e r w i s e w o u l d n o t h av e a t t en d ed an y i n s t i t u t i o n . A l t e r i n g t h a t m i s -

s i o n a n d c u r r i c u l u m i n o r d e r t o c o n f o r m t o t h o s e o f H a r v a r d w o u l d n o t

n eces s a r i l y mean t h a t q u a l i t y a t t h e s e co mmu n i t y co l l eg es w o u l d b e en -

h an ced . I n t h e l o n g r u n , s u ch e f f o r t s m ay li k e ly u n d e r m i n e t h e ir ex t r eme l y

i mp o r t an t p u r p o s e o f p ro v i d i n g h i g h e r ed u c a t i o n o p p o r t u n i t i e s to th o s e

w h o o t h e r w i s e w o u l d h av e mi s sed t h e i r ch ance .

W h i l e th e s am e a t t r ib u t e s o f q u a l i ty m ay n o t b e ap p l i cab l e t o a l l i n s ti tu -

t i ons , t hey may , howex~er, be a pp l i cab l e t o t he sam e type o f i n s t i t u t i ons ( such

as r esearch un iver s i t i es o r communi ty co l l eges ) . Given th i s poss ib i l i t y ,

k n o w l ed g e ab o u t a t t ri b u t e s o f q u a l i t y may b e ex t r eme l y i n f o r ma t i v e t o i n s ti -

t u t i o n s a t t em p t i n g t o e n h an ce q u a l it y . I f re s ea r ch p r o d u c t i v i t y o f t h e f acu l t y

i s known to be r e l a t ed t o exce l l ence , t hen a l l f eas ib l e ways and means fo r

f acu l t y to ach iev e th e h i g h es t r e s ea r ch p r o d u c t i v i t y s h o u l d b e p r o v i d ed ; an d

i f l i b r a r y r e s o u r ces a r e a s s o c i a t ed w i t h t h e q u a l i t y o f f acu l t y r e s ea r ch p r o -

d u c t i v i t y an d i n s t ru c t i o n a l e f f ec ti v en es s , t h en ex p an s i o n a n d i n c rea s edf u n d i n g o f t h e l i b r a r y w o u l d n eces s a r i l y i n c r ea s e f acu l t y e f f ec t i v en es s an d

u l t ima te ly overa l l i n s t i t u t i ona l qua l i ty .

Th e n eed f o r t h i s k n o w l ed g e h as c r ea t ed a my r i ad o f r e s ea r ch an d s ch o l -

a r l y p u b l i ca t i o n s . A s mi g h t b e ex p ec t ed , t h e o p i n i o n s , r e s ea r ch me t h o d o l o -

g i e s , an d f i n d i n g s u s ed an d r ep o r t ed i n t h e s e p u b l i ca t i o n s a r e a s co mp l ex

an d v a r i ed a s t h e co n cep t o f q u a l i t y it se lf . N ev e r t h e le s s , t h e s e p u b l i ca t i o n s

can be ca t egor i zed i n to t h ree genera l t ypes : (1 ) r epu ta t i ona l s t ud i es ; (2 )

ob j ec t i ve i nd i ca to r s t ud i es ; and (3 ) qua n t i t a t i ve co r re l a t e s t ud i es . Rep u ta-

t i o n a l s t u d i e s p i o n ee r ed i n q u i r y i n t o q u a l i t y i n h i g h e r ed u ca t i o n . Th es e

s t u d ie s u s e s u b j ec ti v e ev a l u a t io n s f r o m f acu lt y , d ep a r t m en t h ead s , õ r d ean s

as b a s e s f o r r a ti n g p r o g r am s . O b j ec t i v e i n d i ca t o r s t u d i e s, o n t h e o t h e r h an d ,

r e f ra i n f r o m u s i n g an y t y p e o f s u b j ec t i v e ev a l u a t i o n . T h ey a s s e ss p r o g r ams

t h r o u g h t h e u s e o f o b j ec t i v e v a r iab l e s s u ch a s f acu t t y re s ea r ch p r o d u c t i v it y ,

f i nanc i a l r esources , o r s t uden t ou t comes . F ina l ly , quan t i t a t i ve co r re l a t e

s t u d i e s a r e d e s i g n ed f o r t h e p r i mar y p u r p o s e o f i d en t i f y i n g v a r i ab l e s t h a t

have co r re l a t i on wi th f acu l ty o r p rogram qual i t y . These var i ab l es a re ca l l ed

quan t i t a t i ve co r re l a t es o f qua l i t y . Examples o f quanf i t a t i ve co r re l a t es a re

d ep a r t me n t s ize, t h e am o u n t o f f ed e r a l f u n d i n g , l i b r a r y r e so u r ces , f a cu l t y

sa l a r i es , and f acu l ty r esearch p roduct iv i t y . Overa l l , a l l t h ree t ypes o f s t ud i esh av e co n t r i b u t ed t o w ar d a b e t t e r u n d e r s t an d i n g o f q u a l i t y i n h i g h e r ed u ca -

t io n , b u t m o r e k n o w l ed g e is st il l n eces s a r y s i n ce mo s t s t u d i e s h av e s u f f e r ed



226 TAN

from various methodological limitations. In this paper, all major findings,

strengths, and weaknesses of the three types of studies are discussed.

REPUTATIONAL STUDIES

Reputational studies are those that use peer evaluation to rate programs.

Most reputational studies are focused on graduate programs, but some are

focused on professional and undergraduate programs.

Studies on Graduate Education

The best-known reputational studies on graduate education are the two

reports sponsored by the American Council on Education (ACE): Allan

Cartter's Report, An Assessment of Quality in Graduate Education (1966),

and its replication, A Rating of Graduate Education (1970), conducted by

Kenneth Roose and Charles Andersen. Prior to the two ACE studies, how-

ever, major reputat ional reports were conducted by Hughes (1925, 1934) and

Keniston (1959). The first study by Hughes was one of the earliest published

national reputational studies on graduate education. What prompted

Hughes, then President of Miami University in Ohio, to conduct his study

was his concern for the lack of information about graduate schools available

to baccalaureate graduates of bis institution who wanted to pursue graduateeducation. Consequently, Hughes set out to evaluate the qual ity of graduate

instruction in thirty-eight out of sixty-seven institutions then offering the

Ph.D. As the first step in his study, Hughes compiled a list of distinguished

national scholars in twenty fields of study provided by his Miami faculty.

Questionnaires were subsequently sent to these scholars soliciting their rat-

ings of Ph.D. programs. A second report by Hughes in 1934 employed a

similar methodology, but in this report he expanded bis sample to include

thirty-five fields in fifty-nine universities and classified departments into

two categories, adequate or distinguished, using ratings from an expert

panel of forty faculty members. Even though the two Hughes reports wereby no means ideal in terms of the research methodology employed, Hughes

deserved credit as a pioneer into the investigation of quality. As Lawrence

and Green (1980) pointed out, Hughes was the first person to advocate the

use of academicians as evaluators, the first to emphasize the importance of

reviewing programs att he graduate level, and the first to use faculty quality

as a major criterion for evaluating program quality.

Despite such praise, the two Hughes' studies have drawn much criticism.

Two of the more frequently cited criticisms have been centered on the small

size of the panel of experts (between twenty and sixty) and the dispropor-

tionate geographical representation of the experts, who tended to be concen-



ASSESSMENT OF QUALITY 227

t r a t ed i n t h e N o r t h eas t an d M i d w es t . D es p i t e t h e s e cr it ic i sms , H u g h es ' f in d -

i n gs co n t i n u e t o b e h eav i ly c i ted , u s u a l l y f o r co m p ar i s o n p u r p o s es .

Ev en t h o u g h t h e H u g h es r ep o r t s h av e r ece iv ed m u ch c r it ic i sm , n o a t -t emp t s w e r e mad e t o i mp r o v e o r u p d a t e b i s r ep u t a t i o n a l r a t i n g s u n t i l t h e

s tud y cond ucte d by Ken i s ton i n t he l a ie 1950s. Ken i s ton ' s (1959) p r im ary

g o a l w as t o d e t e r mi n e t h e acad emi c s t an d i n g o f t h e U n i v e r s i t y o f P en n s y l -

van i a i n r e l a t i on t o a g roup o f twen ty - four s imi l a r un iver s i t i es be long ing to

t h e A s s o c i a t i o n o f A m er i can U n i v e r s it ie s ( A A U ) . H i s s tu d y , li ke t h o s e o f

H u g h es , u s ed p ee r ev a l u a t i o n s b y a p an e l o f ju d g es . S p ec if ic a ll y , t h e s t u d y

w as b as ed o n co n s u l t a t io n s e s s i o n s w i t h d ep a r t m en t ch a i r p e r s o n s , w h o w er e

as k ed t o n am e t h e t o o f i v e g r ad u a t e d ep a r t men t s i n t h e i r f ie l d s o n a co m-

b i n ed m eas u r e o f p r o g r am an d f acu l t y q u a li ty . B y ag g reg a t in g t h e d a t a f r o mthe sess ions , Ken i s ton gene ra t ed a r ank-o rd ered l is t o f t he twe n ty - f ive i ns ti -

t u t i o n s , an d f o r co mp ar i s o n p u r p o s es , h e ma t ch ed h i s l i s t ag a i n s t t h a t o f

H u g h e s ( 19 25 ) t o d e t ec t an y ch an g e s i n t h e r an k i n g s o f t h e s am e i n s t it u t io n s .

I n d eed s o m e ch an g es w e r e f o u n d . H a r v a r d U n i v e rs i ty , f o r ex amp l e , h ad

mo v ed f r o m s eco n d p l ace i n t h e H u g h es r ep o r t t o f i r s t p l ace i n t h i s r ep õ r t ;

t h e U n i v e r s i ty o f C a l i f o r n i a ( B e rk e l ey ) m o v ed f r o m n i n t h t o s eco n d p l ace ;

Y a le U n i v e r s i t y mo v ed f r o m f i f th t o f o u r t h ; an d t h e U n i v e r s it y o f M i ch i g an

moved f rom e igh th t o f i f t h (Ken i s ton 1959) .

L i k e mo s t r ep u t a t i o n a l s t u d ie s , t h e K en i s t o n s t u d y h as r ece iv ed i ts s h a r e

o f c r it i c isms . S imi l a r t o c r i ti c i sms o f the H ugh es r epo r t s , t he pa ne l o f ex -p e r ts u s e d i n t h e s t u d y h a s b e e n j u d g e d b y s o m e t o b e t o o s m a l l, a n d t h e u s e

o f d ep a r t m en t h ead s a s j u d g es o f r ep u t a t i o n h as b een c r i ti c ized a s le s s t h an

an id ea l ap p r o ach t o g a th e r i n g i n f o r m a t i o n a b o u t r ep u t a t i o n . R eg a r d i n g t h e

l a t t e r , s o me c r i t i c s h av e a r g u ed t h a t d ep a r t men t h ead s w e r e n o t t h e b e s t

j u d g es f o r t h e s e r ea so n s : (1 ) S i n ce m o s t d ep a r t m en t ch a i r s w e r e f u ll p r o f e s -

so r s , w ere genera l l y o lder i n age , and were f r equen t ly no t t he mo s t d i s t in -

g u i s h ed s ch o l a r s i n t h e d ep a r t men t s , t h ey w er e n o t n eces s a r i ly rep r e s en ta t iv e

o f t h e i r p ee r s i n s p ec i a li z a ti o n a n d k n o w l ed g e o f t h e i r a cad em i c d i s c ip l in e

(Car t te r , 1966); (2 ) s i nce de pa r tm en t heads were genera l l y o lder i n age t han

t h e r eg u l a r f acu lt y , t h ey t en d ed a l s o t o b e tr ad i ti o n a l is t s an d h en ce m ay h av e

r a t ed mo r e p r e s t i g i o u s d ep a r t men t s h i g h e r an d l e s s p r e s t i g i o u s o n es l o w er

(Car t te r , 1966 ; Law rence and Gree n , 1980); (3 ) a lum ni b i as m ay have been a

p r o b l e m s in c e a m a j o r i t y o f t h e s el e ct e d d e p a r t m e n t h e a d s h a d g r a d u a t e d

f r o m t h e i n s t i t u t i o n s b e i n g s amp l ed ( Law r en ce an d G r een , 1 9 8 0 ) . A n o t h e r

f r eq u en t c r i ti c is m o f t h e K en i s to n s t u d y h as b een a b o u t t h e a l leg ed f a i lu r e o f

t h e r e s ea rch e r t o d i s ti n g u is h m eas u r e s o f f acu l t y r ep u t a t i o n f r o m meas u r e s

o f genera l educa t iona l r ep u ta t i o n (Car t t e r , 1966) .

C o n s t r u c t i v e c r i t i c i s ms w h en h eed ed can b e u s ed t o i mp r o v e s u b s eq u en t

m e t h o d o l o g i ca l d e s i g n s , an d t h a t w as i n d eed th e ca s e i n t h e t w o a f o r emen -t i o n ed A C E s t u d i e s . I n t h e 1 9 6 6 C a r t t e r r ep o r t , g r ea t c a r e w as t ak en i n



228 TAN

addressing several methodological inadequacies reflected in the earlier re-

ports by Hughes and Keniston. For example, an improvement was made in

achieving a more equitable geographical representation of institutions. Cart-ter sampled 106 universities located in various regions of the country.

Amidst some improvements, however, some highly criticized features of the

previous studies remained. One such feature was the use of a panel of

experts as judges. Despite strong criticisms about its use, its validity was

defended by the researcher:

The present study is a survey of informed opinion. The opinions we have soughtare what in a court of law would be called "the testimony of expert witnesses"-those people in each field who are weil qualified to judge, who by training are bothknowledgeable and dispassionate, who by their scholarly participation withintheir chosen fields have earned the respect of their colleagues and peers (Cartter,1966, p. 8).

Conscious of several other criticisms brought against Hughes and Kenis-

ton, Cartter identified three purposes for his study. First, he intended to

update the two earlier studies to see what changes had occurred in the same

programs. Second, he wanted "to widen the assessment to include all major

universities in the United States on the assumption that major expansion

will come from the 10-15 traditionally distinguished universities" (1966,

p. 3). Third, he wanted to examine and compare reputational and objective

measurement techniques.Cartter worked with a total of four thousand raters consisting of depart-

ment heads, distinguished senior scholars, and knowledgeable junior

scholars. All were asked to rate each doctoral program in their own field of

study on two variables: (1) the quality of the graduate faculty, and (2) the

effectiveness of the doctoral program. Response categories for the first vari-

able ranged from d i s t i n g u i s h ed to n o t s u f f i c i en t , while categories for the

second ranged from ex t remely a t t rac t i ve to no t at tractive. A n insufficient

information option was available for both variables. Important to note

about the second variable is that Cartter confounded program effectiveness

and Personal attractiveness. Nevertheless, from this scale, Cartter compiled

scores for each department and generated nat ional ratings of 1,663 graduate

programs in twenty-nine academic disciplines in 106 universities. Eight of

the programs ranked in the top ten (namely, the programs at the University

of California [Berkeley], the University of Chicago, Columbia University,

Harvard University, the University of Michigan, Princeton University, the

University of Wisconsin, and Yale University) were similar to those ranked

by Hughes and Keniston. More significant, however, Cartter improved the

research methodology used in the previous studies. Careful attention was

paid to the statistical validity and reliability of the research methods used inthe study. First, representativeness of the sample of raters was checked and




analyzed using biographical information of raters. Second, detalled reliabil-

ity testing of the derived rankings was performed for three selected fields of

study: economics, English, and physics. Cartter found his reputational rat-ings to be reliable when they had remalned unchanged after the ratings were

separated according to the raters' academic rank, their institution of em-

ployment, their highest degree and geographical origin of the degree, and

their field of study. Cartter also investigated the widespread claim that a

small panel of experts may blas reputational results. His findings revealed

that a minimum of fifty knowledgeable persons would be required in a

sample before reliability could be insured.

Finally, Cartter compared his ratings to those derived by objective meas-

ures, such as ratings based on department size, faculty scholarship, faculty

salaries, library resources, and publication records. Cartter found consis-

tency between the two types of ratings and concluded:

it seems likely that if one were to include enough factors in constructing a so-called objective index- allowing for variations in institutional size and university'scommitments to certain fields of study- the results of our subjective assessmentwould be almost exactly duplicated (1966, p. 18).

This first ACE report received a strong response from the academic com-

munity, including both severe criticism and unstinting praise. In 1970, a

replication of the first ACE study was conducted, this time by Roose andAndersen. The original purpose of this second report was to fulfill Cartter's

commitment to undertake a five-year follow-up to the first report. Not

surprisingly, the second report consisted of mostly reassessments of the

same departments in the first report. Roose and Andersen, however, empha-

sized that they would not undertake any study that would "bolster or inflate

[institutional] egos" (1970, p. 3). Rather, their study was intended to be

informational. The researchers hoped that readers of their report would

"think in terms of quality ranges rather than specific pecking orders" (p. 2).

As a way to encourage readers to think in this direction, ranges of scores

rather than absolute raw departmental scores (as was the case in the Cartter

report) were presented.

Roose and Andersen also saw a need "to protect the potential consumer of

graduate education from inadequate programs" (ibid., p. 2). Accordingly,

their study was expanded to include the less visible institutions. Specifically,

their sample included 131 institutions (106 were used in the first study) and

thirty-six fields of study (twenty-nine were used in the first study). Alto-

gether, 2,626 graduate programs were examined, 55% increase over the first

ACE-sponsored study. Like Cartter, Roose and Andersen refrained from

aggregating departmental scores into institutional rankings. Originally artic-ulated by Cartter, three reasons were cited for this refräinment. First, since



230 TAN

no t a l l i n s t i t u t ions su rveyed had doc to ra l p rog ram s in a ll f i e lds , accura t e

ins t i t u t iona l r ank ings c ou ld n o t be der ived . Se cond , even i f al l i n s t i t u t ions

had doc to ra l p rog ram s in a l l f i e lds , ass ign ing we igh t s t o var ious f i e lds w ou ldbe d i f f icu l t , i f no t imposs ib l e . Th i rd , s i nce dep ar tm en ta l sp ec i a l i za t ion i s t he

ch i e f o r g an i z i n g p r i n c i p l e i n acad emi a , l i t t l e p u r p o s e w o u l d b e s e r v ed t o

r ep o r t i n f o r m a t i o n o n an i n s t it u t io n a l b a s i s . D es p i t e t h e s e rea s o n s , t h e r e

h av e b een s o m e r e sea r ch e rs w h o h av e d i s ag r eed w i th C a r t t e r a n d h av e d ev el -

o p ed i n s t it u t io n a l r an k i n g s b y ag g r eg a ti n g d e p a r t m en t a l s co r es . S u ch r e -

searcher s i nc lude Magoun (1966) , t he Nat iona l Sc i ence Board (1969) ,

Pe t rowsk i , Brown, and Duf fy (1973) , and Webs t er (1983) .

E v e n t h o u g h t h e t w o A C E r e p o r t s s h o w e d a m a r k e d i m p r o v e m e n t i n

r e s ea rch m e t h o d o l o g y o v e r t h e ea r l ie r r ep u t a t i o n a l s t u d ie s , t h ey h av e d r aw n

m u ch c r it ic i sm . P e r h a p s t h e m o s t f r eq u e n t l y c i ted c r it ic i sm h as b een t h a t t h e

A C E s t u d i e s i g n o r ed a l a r g e n u mb er o f h i g h e r ed u ca t i o n i n s t i t u t i o n s

( Law r en ce an d G r een , 1 9 8 0 ; C o n r ad an d B l ack b u r n , 1 98 5a ; P e t r o w s k i e t a l .,

1973). The su rveys a l so have be en c r i t i c ized fo r no t assess ing t he l ess popu lar

p r o g r am s i n l e s s e r- k n o w n i n s t it u ti o n s a n d p r o g r ams w i t h f ew n a t i o n a l l y

r en o w n ed f acu lt y . F u r t h e r , t h e r ep o r t s d i d n o t r a t e d ep a r t m en t s i n t h e l a rg e

f i e ld o f educa t ion , and the smal l e r bu t s t i l l s i gn i f i can t p ro fess iona l f i e lds o f

ag r i cu l tu re , bus iness admin i s t r a t i on , l aw, den t i s t ry , and med ic ine (Pe t rowsk i

et al . , 1973).

A n o t h e r f r eq u en t l y c i t ed c r i t i c i s m o f t h e A C E r ep o r t s h a s b een t h a t t h eyme as u r ed r ep u t a t i o n r a t h e r t h an q u a l i ty ( Law r en ce an d G r een , 1 9 8 0 ; D o l an ,

1 97 6). D o l an a r g u ed t h a t ev en i f t h e r ep o r t s w e r e meas u r i n g r ep u t a t i o n , t h e

m e t h o d o l o g y em p l o y ed in th e s e r ep o r t s w as n o t s u i t ed f o r th e t a s k . R ep u t a -

t i o n , D o l an p o i n t ed o u t , is o f f en a s d if f i cu l t t o meas u r e a s q u a li ty , an d t h e

A C E r ep o r t s m eas u r ed n e i t h e r in an ad eq u a t e man n e r . Th i s d i ff i cu l t y w as

a l s o n o t ed b y K n u d s en an d V au g h an ( 19 69 ), w h o s u g g es t ed t h a t t h e r ea s o n

f o r t h e d i f f i cu l ty is t h a t r ep u t a t i o n is t o o s t r o n g ly i n f l u en ced b y d ep a r t m en t

s ize an d n am e f am i li a r it y f r o m f ac u l t y p u b l i ca t i o n . G a l l an t an d P r o t h e r o

(1972) were a l so i n ag reemen t bu t l ooked a t t he s i t ua t i on i n a d i f f e ren t

p a r ad i g m. Th ey s aw th e r ep u t a t i o n o f t h e f acu l t y a s n o t j u s t a f f ec t ed b y t h es h ee r n u m b er o f f acu l t y b u t b y th e ex i s t en ce o r n o n ex i s ten ce o f w h a t t h ey

t ermed "cr i t i ca l mass ." Cr i t i ca l mass , accord ing to t he au thor s , i s t he min i -

mu m d ep a r t men t a l s i z e r eq u i r ed f o r a h i g h q u a l i t y p r o g r am b u t w h i ch d o es

n o t g u a r an t ee i t. F o l l o w i n g th i s r ea s o n in g , r ep u t a t i o n can b e a s i n f lu en ced

by the p resen ce o f t h is c r i t ica l mass as d epa r tm en t s ize .

A n o t h e r c r it ic i sm o f th e A C E r a t in g s h a s b een t h a t t h e r a t in g s m ay cau s e

a " h a l o " o r t h e s o - ca ll ed H e r t z - A v i s e f f ec t b y e s t ab l is h i n g a p eck i n g o r d e r

am on g ins t i t u t i ons . The bas i s fo r th i s c r it i c ism i s t ha t s i nce t he A CE ra t ings

h av e man y me t h o d o l o g i ca l l i m i t a t i o n s , t h ey co u l d b e i n co r r ec t l y u s ed a s

bases fo r es t ab l i sh ing a mi s l ead ing peck ing o rder (Webs t er , 1981 ; Do lan ,



ASSESSMENT OF QUALtTY 231

1976; Lawrence and Green, 1980). Furthermore, some critics have argued

that the halo effects created by the ratings may reinforce institut ional status

quo and impede innovations and improvements. Dolan (1976), for example,argued that only the large, orthodox academic departments would be

rewarded by these ratings. What Dolan was implying is that highly rated

departments would not be as motivated or enthusiastic about implementing

other creative or nontraditional ideas into their programs at the expense of

losing their high ratings. Especially affected would be programs that develop

in an unorthodox fashion, such as interdisciplinary work or experimentation

in graduate education, since proper recognition and the time necessary for

their development and eventual success would not be given.

The ACE studies also have been criticized for not examining all valid

measures of the quality of doctoral programs (Dolan, 1976; Webster, 1981;

Lawrence and Green, 1980). The ACE ratings were based on only two vari-

ables (reputation of the faculty and reputation of the program) and there-

fore, according to the critics, should not be regarded as though they were

measuring the overall quality of doctoral programs. Some other critics have

argued that the ACE ratings were also measuring general academic excel-

lence, not just the overall quality of doctoral programs. Petrowski and his

colleagues (1973), however, found this observation to be unfounded when

they discovered little or no relationship between the reputation of surveyed

doctoral programs and that of baccalaureate or unsurveyed graduate pro-grams. Nevertheless, the lack of adequate measures of quality continues to

be criticized.

Another criticism of the ACE studies has centered on the controversial use

of a panel of experts as evaluators. In addition to rater bias, critics have

argued that since a disproportionately large number of evaluators were

graduates or faculty members in the larger and more prestigious programs,

their evaluations also would be affected by alumni blas (Lawrence and

Green, 1980; Webster, 1981; Cartter, 1966). Both biases were said to affect

ratings in a misleading manner. To be sure, some critics have offered their

suggestions for improvement. Lawrence and Green (1980), for example, rec-ommended that nonacademicians and other individuals who may be more

knowledgeable about program quality be included as evaluators.

Finally, the ACE reports have been criticized for the lag that existed

between the time data were collected and when ratings were reported. This

time lag, critics argue, would render the ratings to be unreflective of current

conditions that existed at a given university at the time of the reports (Web-

ster, 1981; Petrowski et al., 1973).

A unique group of reputational studies that has not been discussed is a

group of studies that ranked institutions by aggregating departmental scores

into an institutional score despite Cartter's plea that they should not. Such



232 T AN

studies include those conducted by Magoun (1966), the National Science

Board (1969), Petrowski et al. (1973), and Webster (1983).

Magoun (1966) compiled data from the Cartter report and aggregateddepartmental scores into two types of institutional standings-comprehen-

sive and div isional-covering the humanities, social sciences, life sciences,

physical sciences, and engineering. Since two types of scores were reported

by Cartter, one based on the quality of the graduate faculty and the other on

the effectiveness of the program, Magoun generated a total of four institu-

tional standings. Divisional institutional standings were derived by dividing

the sum of all departmental scores in each division by the total number of

rated programs in the division. Comprehensive institutional standings, on

the other hand, were derived by dividing the sum of all divisional scores by

four (four was used as the denominator because engineering was excluded).

Magoun compared the divisional standings with the comprehensive stand-

ings and found similarities between the two. Institutions that appeared in the

upper clusters of the divisional standings list also appeared in the upper half

of the comprehensive standings list. An example of such an institution was

the University of Iowa, which achieved a divisional ranking of third in two

fields (the humanities and the social sciences), a divisional ranking of sec-

ond in the life sciences, and a comprehensive institutional ranking o f first

place. This and other similarities led Magoun to conclude that the aggrega-

tion o f department scores into an institutional score was not only possible, itwould be reflective of the overall institutional reputation (mathematically,

some similarities were expected because institutional scores were nothing

more than mathematical computations of divisional scores).

Another attempt to derive institutional rankings by aggregating depart-

mental scores was a report published by the National Science Board (1969)

entitled Graduate Educa t ion: Parameters fo r Pub l ic Po l icy . This study was

based on Cartter's ratings of the reputation of the graduate faculty. Essen-

tially, the National Science Board aggregated departmental scores (Cartter's

ratings of faculty reputation) into institutional scores and rank-ordered

institutions. The list revealed fifty institutions in the top three ranks of aseven-rank system. Even though the top-rated institutions were not identi-

fied, they were divided into nine geographical regions. The geographical

distribut ion of the institutions among the nine regions was not as even as the

Board had expected. The heaviest concentration of the institutions was

located in the Mid-Atlantic region (with thirteen institutions), followed by

the East North-Central (with ten institutions), while the sparsest concentra-

tions were found in the Mountain and East South-Central regions with only

one each.

The third and more recent attempt to derive institutional rankings was

made by Petrowski et al. (1973). In this study, the researchers employed three




methods to identify what they termed "national universities." National uni-

versities were regarded as top-rated institutions that should be funded by the

federal government for the undertaking of graduate education and research.The first method involved the derivation of an institutional score by averag-

ing the scores of all departments (ranked or not) within a given institution

(data came from the ACE studies). The second method ranked institutions

according to the number of programs that had achieved ACE rankings. The

third method used a weighting scheine which considered both the number of

rated departments and the reputational rankings of the departments. In this

method, a weighted institutional mean score was derived by multiplying the

mean scores derived by the first method with the proportion of rated depart-

ments to the total number of disciplines surveyed. These weighted scores

were subsequently used as bases for deriving institutional rankings.

To check for the plausibility of the rankings, Petrowski et al. compared

them to those of the National Science Board. The National Science Board

ranked institutions by nine geographical regions; so Petrowski et al. sepa-

rated their ranked institut ions into the same regions. Similarities between the

two rankings for seven of the nine regions were found. The two regions in

which both lists did not match were the Mid-Atlantic and the South-Atlantic

regions. In addition, the researchers noted several consistencies in institu-

tional characteristics among the top-rated institutions. For e×ample, the top

thirty universities were all large institutions and had more than sixteenranked departments. Finally, the researchers offered a few words of caution

regarding their research findings. They pointed out that their rank-ordered

list of institutions or any other for that matter that has attempted to identify

national universities for exclusive financial support would produce a guild

of preferred institutions in which entry would be severely restricted. The

researchers insisted that mediocrity should be discouraged, not entry into

the guild (ibid.).

Webster (1983) made the fourth at tempt at deriving institutional rankings

by aggregating departmental scores. Webster arrived at his institutional

score by aggregating reputational scores of the faculty cornpiled by the

Conference Board of Associated Councils (Jones, Lindzey, and Coggeshall,

1982). Graduate schools in the fields of the physicat and mathematical

sciences, the humanities, engineering, the biological sciences, and the social

sciences were ranked based on their institutional score. Thirty top-rated

graduate schools were listed. The top ten institutions were the University of

California (Berkeley), Stanford University, Harvard University, Yale Univer-

sity, Princeton University, the University of Chicago, the University of Cali-

fornia (Los Angeles), the University of Michigan, and the University of

Wisconsin (Webster, 1983).In addition to the ACE studies and studies that aggregated departmental



234 T AN

scores i n to i ns t i t u t i ona l r ank ings , o the r r epu ta t i ona l s t ud i es o f g raduat e

e d u c a t i o n w e r e c o n d u c t e d b y L a d d a n d L i p se t ; C l a rk , H a r n e t t , a n d B a i rd

( 1 9 7 6 ) ; an d , mo r e r ecen t l y , t h e C o n f e r en ce B o a r d o f A s s o c i a t ed R es ea r chCo unci l s ( Jon es e t a l . , 1982) . L ad d an d L ipse t , u t i li z ing bas i ca l l y the sam e

m e t h o d o l o g y e m p l o y e d b y H u g h e s a n d K e n i st on , a s k e d 4 ,0 0 0 f a c u lt y m e m -

ber s a t fou r -year co l l eges and un iver s i t i es t o i den t i fy f i ve depar tmen t s

kn ow n fo r t he ir d i s t i ngu i shed f acu l ty i n t he i r r espec t ive d i sc ip l i nes . De par t -

m en t s r ece iv ing vo t es f rom a t l eas t 10°70 o f t he r espo nde n t s i n each o f t he

n i n e t een s u r v ey ed f ie l d s q u a l i f ied t o b e r an k ed . Th e r an k s r ece iv ed b y m an y

ins t i t u t i ons i n t h i s s t udy were s imi l a r t o t hose genera t ed by p rev ious r e -

searchers (cf . Scul ly , 1979, p . 77) . For example, the Univers i ty of Chicago

was r anke d s ix th i n t h is s t udy , f i f t h by H ugh es (1934) , s i x th by Ken i s ton

(1959) , n ine th by Car t t e r (1966) , and seven th by Ro ose and A nd erso n (1970) .

A n o t h e r ex amp l e w as H a r v a r d U n i v e r s i t y , w h i ch w as r an k ed s eco n d i n t h i s

s tud y and by Car t t e r (1966) , and f i r s t by Hu ghe s (1934) and Ken i s ton (1959).

T h e s t u d y b y C l a rk a n d h e r a s s o c ia t es , c o n d u c t e d u n d e r t h e jo i n t s p o n -

s o r s h i p o f t h e C o u n c i l o f G r ad u a t e S ch o o l s an d t h e E d u ca t i o n a l Tes ti ng

S e rv ice , u s ed r ep u t a t i o n a l ev a l u a t io n s an d o b j ec t i v e v a r i ab le s t o r an k p r o -

g r ams . R ep u t a t i o n a l ev a l u a t i o n s o f s ev en t y - t h r ee d ep a r t men t s i n t h e f i e l d s

o f ch emi s t ry , h i st o ry , an d p s y c h o l o g y w ere o b t a i n ed f r o m f acu l t y m em b er s ,

s t u d en t s , an d a l u mn i . T h es e r ep u t a t i o n a l ev a l u a t io n s w e r e co m b i n ed w i t h

th i r t y o b j ec t i ve var i ab l es t o a r r i ve a t tw o c lus t e r s o f i nd i ca to r s : (1) r esearch -o r i en t ed i nd i ca to r s ( i nc luded var i ab l es such as depar tmen t s i ze , f acu l ty r epu-

t a t i on , s t uden t academic ab i l i t y , f acu l ty pub l i ca t i ons , and phys i ca l and f i -

nanc i a l r esources ) and (2 ) educa t iona l exper i ence i nd i ca to r s ( i nc luded

facu l ty i n t e rper s ona l r e l a t i ons , a lum ni r a t ings o f d i s ser t a t i on exper i ences ,

an d t h e acad emi c c l i ma t e ) . A h i g h co r r e l a t i o n amo n g v a r i ab l e s w i t h i n each

c lus t e r was found bu t no t across c lus t e r s . Ev iden t ly , r esponden t s s t rong ly

be l i eved in the d i s t inc t n a tu re o f e ach c lus te r. As a r esu l t, C l ark e t a l.

q u i ck l y p o i n t ed o u t t h a t t h e ir r a ti n g s w e r e " n o t r a t in g s o f o v e r a ll d o c t o r a l

p r o g r am q u a l it y , b u t , r a th e r, r a ti n g s o f th e f ac u l t y emp l o y ed i n th e s e p r o -

grams, ref lect ing pr imar i ly thei r research records" (1976, pp . 1313-1314) .

Th i s f i n d i n g s u p p o r t s t h e v i ew p o i n t s h a r ed b y s o me c r i t i c s t h a t s o me r e -

s ea r ch e rs h av e i n co rr ec t ly o r a t l e a s t t o o q u i ck l y eq u a t ed f acu l t y r ep u t a t io n

t o p r o g r am q u a li ty .

Th e mo s t r ecen t ex t en s i v e r ep u t a t i o n a l s t u d y o f g r ad u a t e ed u ca t i o n w as

t h e f i v e - v o lu m e r ep o r t o f t h e C o n f e r en c e B o a r d o f A s s o c i a t ed C o u n c i l s

( J o n es e t a l ., 1 98 2).* Th i s B o a r d is co mp r i s ed o f th e A me r i can C o u n c i l o f

Lea r n ed S o c i e t i e s , t h e A mer i can C o u n c i l o n Ed u ca t i o n , t h e N a t i o n a l R e -

s ea r ch C o u n c i l , an d t h e S o c i a l S c i en ce R es ea r ch C o u n c i l. T h e B o a r d ' s s t u d y

*Further reference o this stud y will be 1982a, 1982b, etc.




as s e s s ed t h e q u a l i t y o f d o c t o r a l p r o g r am s i n th i r t y - tw o d i sc i p li n e s in m o r e

t h an t w o h u n d r ed d o c t o r a t e - g r an t in g u n i v er s it ie s ( a to t a l o f 2 ,6 9 9 p r o -

g r ams ) . Th e C o n f e r en ce B o a r d co n d u c t ed t h is s t u d y b as ed o n f o u r c lea r l ys t a t ed p r inc ip l es :

1. Th e i m p o r t an c e o f t h e s t u d y re s u lt s t o n a t i o n a l an d s t a te b o d i e s ;

2 . Th e d es i r e t o s t imu l a t e co n t i n u i n g em p h as i s o n q u a l i ty i n g r ad u a t e ed u -

ca t i on ;

3 . Th e n eed f o r cu r ren t ev a l u a t i o n s t h a t t ak e i n t o acco u n t t h e man y ch an g es

t h a t h av e o ccu r r ed i n p r o g r ams s i n ce t h e R o o s e - A n d e r s en s t u d y ; an d

4 . Th e v a l u e o f ex ten d i n g t h e r an g e o f meas u r e s u s ed i n ev a lu a t iv e s t ud i e s

o f g radua t e p rog ram s ( Jo nes e t a l . , 1982a , p . 7 ).

W i t h i n t h e co n t ex t o f t h e s e p r i n c i p l e s , t h e B o a r d t o o k g r ea t c a r e i n ad -

d r e s si n g s o m e o f t h e c r i ti c is ms b r o u g h t ag a i n s t p r ev i o u s r ep u t a t i o n a l s t u d ie s

an d a t t emp t ed t o re s o lv e s o m e o f t h e s e m e t h o d o l o g i ca l i n ad eq u ac i e s . O n e

s u ch i n ad eq u acy w as r e s o l v ed b y t h e u s e o f mu l t i p l e meas u r e s . Th e B o a r d

r eco g n i zed , h o w ev e r , t h a t n o ma t t e r h o w man y meas u r e s w e r e u s ed , n o t

ev e r y b o d y w o u l d b e f u l ly s a t is f ied , e s p ec i a ll y t h o s e w h o h av e i n si st ed t h a t

t h e q u a l i ty o f g r ad u a t e e d u ca t i o n can n o t b e q u an t i f i ed . F u r t h e r mo r e ,

acco r d i n g t o t h e B o a r d , t h e u s e f u l n es s o f an y a s s e s s men t w o u l d b e d ep en -

d en t o n t h e v a l i d it y an d r e l i abi li ty o f th e v a r i ab l e s u s ed . T h e B o a r d co n s i d -ered t h ree c r i t e r i a fo r t he i nc lus ion o f var i ab l es . The f i r s t c r i t e r i on was t ha t

v a r i ab le s m u s t b e r e la t ed t o th e q u a l i ty o f r e s ea r ch - d o c t o r a t e p r o g r am s . Th e

second cr i t e r i on concerned the f eas ib i l i t y o f ga ther ing and compi l i ng r e l i -

ab l e d a t a o n t h e s e v a r i ab l e s f o r mak i n g n a t i o n a l co mp ar i s o n s o f p r o g r ams

in par t i cu l a r d i sc ip l i nes . Th i rd , t he Board on ly cons idered var i ab l es t ha t

were app l i cab l e t o a majo r i t y o f t he d i sc ip l i nes su rveyed (1982a) .

U s i n g t h i s ap p r o ach , t h e B o a r d co mp i l ed , an a l y zed , an d r ep o r t ed d a t a

and f i nd ings i n f i ve s tud i es cover ing p rogram s in t he f i e lds o f (1) hum ani t i es ,

(2 ) soc i a l and behav io ra l sc i ences , (3 ) b io log i ca l sc i ences , (4 ) mathemat i ca l

and phys i ca l sc i ences ; and (5 ) eng ineer ing . Because t hese s tud i es con ta in

var i ab l es and da t a , t hey a re e×amined in g rea t e r de t a i l .

H um a n i t i e s

In th i s s tudy (1982a) , a to tal of 522 programs in ar t h i s tory , c lass ics ,

Eng l i sh l anguage and l i t e ra tu re , F rench l anguage and l i t e ra tu re , German

language and l i t e ra tu re , Span i sh l anguage and l i t e ra tu re , l i ngu i s t i cs , mus i c ,

an d p h i l o s o p h y w er e ex ami n ed . Tw e lv e v a r i ab le s w e r e u s ed i n t h e s t u d y an d

were g rou ped und er fou r genera l a reas : (1 ) p rog ram s ize , (2) charac te r i s ti cso f g rad uat es ; (3) r epu ta t i ona l su rvey resu l t s; a nd (4) un iver s i ty l i b ra ry si ze .



236 TAN

T A B L E 1 . V a r i ab l e s U s e d b y th e C o n f e r e n c e B o a r d a

N a m e o f V a r i ab l e H u m a n i t i e s S o c i a l B io . M a t h . E n g i n e e r in g

N o . o f f a c u l t y x x x × ×

N o . o f g r a d u a t e s × × × x x

N o . o f s t u d e n t s × × × × ×

N o . o f g r a d u a t e s w i t h × × × × ×

f e l l o w s h i p s

M e d i a n n o . o f y r s . t a k e n x × × x ×

t o a t t a in P h . D .

P r o p o r t i o n o f g r a d u a t e s × × × × x

w i t h e m p l o y m e n t

P r o p o r t i o n o f g r a d u a t e s x × × x ×w i t h a c a d e m i c

e m p l o y m e n t

R e p u t a t i o n o f f a c u l t y × × × × ×

P r o g r a m e f f e c t i v e n e s s × × × × x

P r o g r a m i m p r o v e m e n t × × x × x

E v a l u a t o r s ' f a m i l i a r i t y x × × × x

C o m p o s i t e l i b r a r y i n d e x x x × x x

P r o p o r t i o n o f f a c u l t y x × × ×

w i t h g r a n t s

R e s e a r c h e x p e n d i t u r e s x × × ×

F a c u l t y p u b l i c a t i o n s x × × ×

a × denotes that the var iable was used in the s tudy

U n d e r p r o g r a m s iz e , t h r e e v a r i a b l e s w e r e i n c l u d e d : ( a) t h e r e p o r t e d n u m -

b e r o f f a c u l ty m e m b e r s i n t h e p r o g r a m i n D e c e m b e r 1 98 0 ( th i s i n f o r m a t i o n

w a s o b t a i n e d d i r e c t l y f r o m p a r t i c i p a t i n g u n i v e r s i t i e s) ; ( b ) t h e r e p o r t e d n u m -

b e r o f p r o g r a m g r a d u a t e s i n t h e f iv e y e a r s f r o m 1 97 5 t h r o u g h 1 98 0 ( th i s

f ig u r e w a s b a s e d o n d a t a c o m p i l e d i n t h e N a t i o n a l R e s e a r c h C o u n c i l ' s Sur-

vey o fEarned Doctora tes ) ; a n d (c ) t h e r e p o r t e d t o t a l n u m b e r o f f u l l - t i m e

a n d p a r t - t i m e g r a d u a t e s t u d e n t s ( t a k e n in D e c e m b e r 1 98 0) w h o w e re e n -r o l le d in th e p r o g r a m a n d w h o i n t e n d e d t o e a r n d o c t o r a t e s . T h e v a r i ab l e s

u s e d b y t h e C o n f e r e n c e B o a r d i n th e f i ve r e p o r t s a r e s u m m a r i z e d i n T a b l e 1.

I n th e s t u d y o n h u m a n i t i e s ( 19 8 2 a) , u n d e r c h a r a c t e r i s t i c s o f g r a d u a t e s ,

d a t a f o r f o u r v a r i a b le s w e re c o m p i l e d : ( a ) t h e p r o p o r t i o n o f p r o g r a m g r a d u -

a t e s w h o h a d r e ce i ve d s o m e n a t i o n a l f e l l o w s h i p o r t r a in i n g g r a n t s u p p o r t

d u r i n g t h e i r g r a d u a t e e d u c a t i o n d u r i n g f i s c a l y e a r s 1 97 5 t h r o u g h 1 97 9; ( b)

t h e m e d i a n n u m b e r o f y e a r s f r o m f i r s t e n r o l l m e n t i n g r a d u a t e s c h o o l to

r e c e i p t o f t h e d o c t o r a t e ( d a t a w e r e t a k e n d u r i n g f i sc a l y e a r s 1 97 5 t h r o u g h

1 97 9) ; ( c) t h e p r o p o r t i o n o f f is c a l y e a r 1 9 7 5 - 7 9 p r o g r a m g r a d u a t e s w h o a t

t h e t i m e t h e y c o m p l e t e d r e q u i r e m e n t s f o r t h e d o c t o r a t e r e p o r t e d t h a t t h e y




had made definite commitments for postgraduate employment either inside

or outside academia; and (d) the proportion o f fiscal year 1975-79 program

graduates who at the tirne they completed requirements for the doctoratereported that they had made definite commitments for postgraduate em-

ployment in Ph.D.-granting universities (this variable was intended to meas-

ure the placement success of graduates in gaining academic positions.

The third area of variables in this study, reputational survey results, which

were based on responses to the Board's survey conducted in 1981, included

(a) the mean rating of the perceived scholarly quality of the program fac-

ulty; (b) the mean rating of the effectiveness of the program in educating

future research scholars/scientists; (c) the mean rating of the improvement

in program quality in the last five years; and (d) the mean rating of the

evaluator's familiarity with the work of the program's faculty. The last

variable was used as a control for checking the reliability of the first three

reputational ratings.

Finally, under university library size, a composite index (taken in 1979-80)

was given. This index, based on data compiled by the Association of Re-

search Libraries, described the library size in the university in which the

program was located.

S o c i a l a n d Be h a v i o r a l S c i en ces

Data for a total of 639 programs in anthropology, economics, geography,

history, political science, psychology, and sociology were presented in this

study (1982b). In addition to the four groups o f variables used in the human-

ities study, two additional ones were added: (1) research support data, which

were represented by (a) the proportion of program facul ty rnembers holdlng

research grants from the Alcohol, Drug Abuse, and Mental Health Adminis-

tration, the National Institutes of Health, or the National Science Founda-

tion at any time during the FY 1978-80 period, and (b) the total expendi-

tures reported by the individual university for research and development

activities in a specified field during FY 1979; and (2) publication records,

which contained (a) the number of published articles attributed to the pro-

gram faculty members for 1978-80, and (b) the proportion of program

faculty members with one or more published articles during 1978-80.

Bio log ica l Sc i ences

A total o f 616 programs in biochemistry, botany, cellular/molecular biol-

ogy, microbiology, physiology, and zoology were reviewed (1982c). All six

areas of variables used in the previously discussed social and behavioralsciences study were used in this study. The only difference was that instead



2 3 8 T A N

of using the proportion of faculty members with one or more published

articles as a variable (this was one of the variables under the area of publica-

tion records), a replacement variable that estimated the overall influence ofpublished articles was used. The Board claimed that this new variable was

not designed to measure the impact o f articles by individual authors/f aculty

members but rather to measure the impact of the journals in which articles

associated with a particular program had been published.

M athem at ica l a nd P hys ica l Sc iences

Data for 596 programs in chemistry, computer science, geosciences, math-

ematics, physics, and statistics/biostatistics were compiled and presented

(1982d). The variables included in this study were similar to those used in the

study of biological sciences.

Engineer ing

Exactly 326 programs in chemical, civil, electrical, and mechanical engi-

neering were reviewed in this report (1982c). The variables included in this

analysis were similar to those used in the study of biological sciences.

Som e Se lec ted F ind ings o f the Conference Board

From the five sets of data, the Conference Board reported differences

across disciplines. For example, in the humanities study, the English pro-

grams, on the average, had the largest number of faculty members (thirty-

one in December 1980), while music was second with twenty faculty mem-

bers. The English programs also graduated the most students (an average of

forty-four Ph.D. recipients per program in the FY 1975-79 period) and had

the largest enrollment (an average of sixty-two doctoral students in Decem-

ber 1980). In contrast, the German programs had an average of only nine

faculty members, thirteen graduates, and fifteen doctoral students.The Board also performed Pearson product-moment correlation analyses

for all possible combinat ions of variables used in each study. In the humani-

ties study, for example, the Board found two variables, the number of

program graduates and the survey rating of the reputation of the faculty, to

be highly correlated. No explanation for this correlation, however, was pro-

vided. Correlation results for other disciplines were similarly reported.

Perhaps the most significant part of the Board's study was that the Board

proved the reliability of its three reputational survey items. Using the split-

half correlation analysis, the Board established the overall reliability of the

survey results in each discipline in all five studies. The Board found a high



ASSESSMENT OF QUALFTY 239

degree of reliability in the judgments of the evaluators in all three survey

items.

As a final step, the Board addressed and defended some of the criticismsbrought against reputational studies. In response to the criticism that repu-

tational ratings may be affected by inaccurate judgments among evaluators

who are unfamiliar with the program they are asked to evaluate, several

observations were made by the Board. First, the Board found its ratings to

vary as a funct ion of the level of familiarity among evaluators. Specifically,

the reputation of the facul ty was rated higher by evaluators who had c o n s i d -

e r a b l e f a m i l i a r i t y (this was a category used to measure the level of familiar-

ity) than by evaluators who had s o m e or l i t t l e f a mi l i a r i t y . This finding,

however, came as no surprise to the Board. In a subsequent analysis, the

Board found ratings from unfamiliar evaluators to have insignificant effects

on the overall ratings (1982a, p. 196).

In addressing the criticism that reputational evaluations may be affected

by the geographical proximity of evaluators, the Board regrouped programs

into two categories, nearby and outside, depending on the proximity of the

evaluators to the programs. By recalculating the mean ratings of the reputa-

tion of the faculty, the Board found that even though nearby programs were

given higher ratings than those outside the evaluators' region, the differences

were insignificant. The Board called these differences a representation of no

more than a "secondary effect" since evaluators tended to rate higher thoseprograms they were familiar with, whether nearby or not. Also, the high

correlations found between the mean ratings of the two groups of programs,

the nearby and the outside, showed that the relative standings of programs

were not significantly influenced by the geographical proximity of evalua-

tors.

In addressing yet another criticism, namely, that reputat ional surveys may

be affected by alumni bias, the Board analyzed its own data and found

evidence supporting this claim: differences existed in the ratings of the

reputation of the faculty between alumni and nonaIumni. Fortunately, the

proportion of the evaluators who were alumni of the programs they wereasked to evaluate was small and therefore had "very little" effect on the

overall survey results. The Board came to this conclusion by recalcutating

the mean ratings of faculty reputation in some selected programs with eval-

uations from alumni excluded and comparing them to the original mean

ratings. In the humanities study, for example, the mean ratings of 485 out of

522 humanities programs had remained unchanged (to the nearest tenth of

a unit). Similar insignificant differences were observed for the social and

behavioral sciences (1982b, pp. 198-199), the biological sciences (1982c,

pp. 186-187), the mathematical and physical sciences (1982d, pp. 182-183),

and engineering (1982e, p. 132). Convincingly, the Board concluded that



240 TAN

alumni bias did not play a significant part in its reputational survey data

and, therefore, alumni ratings were not excluded.

The Board also addressed the criticism that reputational evaluation mayhave been affected by the interaction of research interests of evaluators and

the focus of the research interests of the program to be rated. Some critics

have argued that since some programs may be strong in some narrowly

focused areas of research, this strength may not be known and hence not

evaluated. The Board recognized the difficulty in identifying research inter-

ests within programs; so it only attempted to do so for the fields of physics

and statistics/biostatistics. Evaluators in each discipline were separated into

two groups according to their specialized area. In physics, for example, the

first group included specialists in elementary particles and nuclear structure,

while the second group was comprised of specialists in all other areas of

physics. Mean ratings of the reputation of the faculty for each of the two

groups were recalculated, and the overall findings indicated that the evalua-

tors' specialty field had an effect on their ratings, but not significant enough

to alter the relative standings of the programs. A similar conclusion was

reached for the field of statistics/biostatistics (1982d, p. 184).

In summary, the Conference Board has developed a rauch improved repu-

rational study of doctoral programs. The methodologies used in gathering

institutional and program data and reputational evaluations were clearly a

marked improvement over those used in the past. Nonetheless, there weresome limitations. The first was that the Board did not attempt to define

quality. The closest the Board came to a definition was when it specified that

its reputational surveys were measures of perceived program quality-not

quality in some ideal or absolute sense. Therefore, the Board refrained from

ranking programs. It reasoned that any ranking attempt would not fully

capture the actual quality of programs. Another limitation of the study was

that the Board merely assumed the reputation of the faculty to be the major

variable indicating quality. While this is not a major criticism, the Board

could have combined its reputational survey results with its data on objective

variables to arrive at a more adequate def inition or measurement of quality.Future studies should address this lacuna.

S t u d i e s o n P r o f e s s i o n a l / U n d e r g r a d u a t e P r o g r a m s

One of the major criticisms of reputational studies is that a majority of

these studies are focused on programs at the gräduate level. In addressing

this criticism, some researchers have focused on undergraduate and profes-

sional programs.

One major attempt to assess professional programs was a study by

Margulies and Blau (1973), which rated programs in seventeen professional




f ields: archi tecture, business , educat ion, engineer ing, law, l ibrary science,

nur s ing , pha rma cy , soc i a l w ork , den t i s t ry , fo res try , j ou r na l i s m , m ed ic ine ,

o p t o me t r y , p u b l i c h ea l t h , t h eo l o g y , an d v e t e r i n a r y med i c i n e . Emp l o y i n g anap p r o ach s i mi la r to t h a t u s ed b y R o o s e an d A n d e r s en , 1 ,1 80 d ean s o f p r o -

f e s s i o n a l s ch o o l s w e r e a s k ed t h r o u g h a q u es t i o n n a i r e t o i d en t i f y t h e b e s t

p rograms in t he i r r espec t ive f i e lds o f spec i a l t y . Responses were subsequen t ly

ag g r eg a t ed an d p r o g r am s r an k ed b y i n s t it u t io n . A s ex am p l e s , th e t o p - r a t ed

a r ch i tec t u r e p r o g r am s w er e t h o s e a t t h e U n i v e r s it y o f C a l i fo r n i a ( B e rk e ley ) ,

M as s ach u s e t t s I n s t i t u te o f Tech n o lo g y , H a r v a r d U n i v e r si ty , t h e U n i v e r s i ty

o f P en n s y l v an i a , an d P r i n ce t o n U n i v e r s i t y , w h i l e t h e t o p - r a t ed s o c i a l w o r k

p r o g r am s w er e t h o s e a t C o l u m b i a U n i v e r si ty , th e U n i v e r s i ty o f C h i cag o , t h e

U n i v e r s i t y o f M i ch i g an , B r an d e i s U n i v e r s it y , an d C as e W es te r n R es e r ve

Univer s i t y . In a fo l l ow-up s tudy , B lau and Margu l i es (1974) added mus i c as

an e igh t een th f i e ld and r esponded to t he c r i t i c i sms b rough t aga ins t t he i r f i r s t

r epor t . F i r s t , B l au an d M argu l i es o f fe red a r ebu t t a l t o the c r i ti c i sm tha t t he i r

f i n d i n g s w o u l d o n l y c r ea t e an i n s t i t u t i o n a l " h a l o e f f ec t . " Th ey t o o k t h e

p o s i t i o n t h a t p r o f e s s i o n a l s ch o o l s d i d ac t u a l l y d i f f e r i n q u a l i t y an d t h a t

p rov id ing such in fo rmat ion t o consumers was a pub l i c se rv i ce . Second ly ,

t h ey d e f en d ed t h e l o w r e s p o n s e r a t e f r o m d ean s ( o n l y 3 6 % ) i n t h e i r f i r s t

r epo r t ( c r i ti cs a rgu ed tha t t he l ow ra t e ma y have b i ased r esu lt s ) by p rov id ing

ev i d en ce t h a t h i g h o r l o w r e s p o n s e r a t e d i d n o t mar k ed l y a f f ec t t h e i r r an k -

ings . In t he i r fo l l ow-up s tudy , B lau and Margu l i es go t a h igher r esponse r a t e( 7 9 % ) an d f o u n d t h a t th e i r r an k in g s h a d r em a i n ed b as i ca l ly u n ch an g e d

f rom the o r ig ina l s t udy .

A n o t h e r s t u d y o f p r o f e s si o n a l p r o g r a m s w a s c o n d u c t e d b y C a r t t e r a n d

S o l m o n i n 1 9 77 . Th i s s t u d y r a t ed p r o g r am s i n law , ed u c a t i o n , an d b u s i n es s .

T h i s s t u d y b a s i c a l l y e m p l o y e d t h e s a m e m e t h o d o l o g y u s e d b y B l a u a n d

M argu l i es (1974), t ha t i s , ques t i onnai res were sen t t o f acu l ty and dean s

s o li c it in g r a ti n g s o f th e r ep u t a t i o n o f t h e f acu l t y an d p r o g r am o n a L i k e r t-

t y p e s ca l e . Th e t o p f i v e l aw s ch o o l s i n t h e n a t i o n w er e l o ca t ed a t H a r v a r d

Univer s i t y , Yale Un iver s i t y , S t an fo rd Univer s i t y , t he U niver s i t y o f M ich igan ,

an d t h e U n i v e r s i t y o f C h i cag o ; t h e t o p f i v e s ch o o l s o f ed u ca t i o n w er e l o -

ca t ed a t S t an f o r d U n i v e r si ty , H a r v a r d U n i v e r si ty , t h e U n i v e r s i ty o f C h i cag o ,

t h e U n i v e r s i t y o f C a l i f o rn i a a t Lo s A n g e l e s , an d t h e U n i v e r s i ty o f C a l i f o r n i a

( B e r ke l ey ) ; an d t h e t o p f iv e b u s i n es s s ch o o l s w e r e l o ca t ed a t S t an f o r d U n i -

v e rs it y, H a r v a r d U n i v e rs it y , M as s ac h u s e t t s I n s t it u t e o f Tech n o l o g y , th e U n i -

v e r s it y o f C h i cag o , an d C a r n eg i e - M e l l o n U n i v e rs it y . C a r t t e r an d S o l m o n

f o u n d v as t si m i la r it ie s b e t w een th e i r f i n d in g s an d t h o s e o f B l au an d M ar g u -

l i es . There were some d i f f e rences as wel l , hu t t he d i f f e rences , accord ing to

M u n s o n an d N e l s o n ( 1 97 7), m ay s i mp l y b e d u e t o d i ff e r en ces in t h e s amp l e

an d t h e i n s t r u men t u s ed .

O t h e r r ep u t a t i o n a l s t u d ie s o f p r o f e s s i o n a l p r o g r am s i n c l u d ed a s t u d y b y



242 T AN

C ar p e n t e r an d C a r p e n t e r (1 97 0), w h i ch r an k ed f o r t y - f o u r l ib r a r y s c ien ce

p r o g r am s ; a s t u d y b y C o l e an d L i p t o n ( 1 97 7), w h i ch r an k ed m ed i ca l

s ch o o l s ; t w o s t u d ie s b y t h e MBA Magazine (1974, 1975) , which ranked thet o p f i f t e en b u s i n es s p r o g r ams ; an d a s t u d y r ep o r t ed i n t h e Juris Doctor

Magazine (1976) , wh ich ranked the t op tw en ty l aw schoo l s in the na t ion . The

l aw s ch o o l s t u d y w as u n i q u e i n th a t , i n ad d i t i o n t o f i ft y - e ig h t d ean s , 1 ,3 00

m agaz ine r eade r s were used as eva lua to r s . In t e res t ing ly , the s tud y foun d

co n s i s t en cy i n t h e r a t i n g s b e t w een t h e t w o g r o u p s ( c f . Law r en ce an d G r een

1980, p. 21).

F i n a l l y , an o t h e r g r o u p o f r ep u t a t i o n a l s t u d i e s a s s e s s ed u n d e r g r ad u a t e

p rogra m s . A s tud y by John so n (1978) i den t i f i ed a li s t o f t en lead ing ins t i t u -

t i ons i n t e rms o f t he i r na t ion a l i n f luence . Th e top t en ins t i tu t i ons w ere

C o l u m b i a U n i v e rs i ty , H a r v a r d U n i v e rs it y , M as s ach u s e t t s I n s t i tu t e o f Tech-

no logy , Pr ince ton Univer s i t y , S t an fo rd Univer s i t y , t he Univer s i t y o f Cal i fo r -

n i a ( B e r k e ley ) , t h e U n i v e r s i ty o f C a l i f o r n i a a t Lo s A n g e l e s , t h e U n i v e r s i t y o f

Ch icago , t he U niver s i t y o f Mich igan , and Yale Univer s i ty . Th i s l is t t u rne d

ou t t o be s imi l a r t o t he l i s t o f l ead ing na t iona l un iver s i t i es genera t ed by

P e t r o w s k i e t a l. ( 19 73 ). I n a n o t h e r a s s e s s men t o f u n d e r g r ad u a t e ed u ca t i o n ,

As t in and So lmon (1981) , u s ing a r a ther d i f f e ren t approach , i den t i f i ed the

t o p - r a t ed u n d e r g r ad u a t e d ep a r t men t s i n t h e f i e l d s o f e co n o mi cs , h i s t o r y ,

Eng l i sh , chem is t ry soc io logy , and b io logy . Th e researcher s asked ra t e r s to

rank depar tmen t s based on s ix c r i t e r i a : (1 ) t he overa l l qua l i t y o f underg rad-u a t e ed u ca t i o n ; ( 2 ) t h e l ev e l o f p r ep a r a t i o n o f s t u d en t s f o r g r ad u a t e o r

p r o f e s s i o n a l s ch o o l s ; (3 ) t h e l ev el o f p r ep a r a t i o n o f s t u d en t s f o r emp l o y m en t

af t e r co l l ege ; (4 ) f acu l ty commi tmen t t o underg raduate t each ing ; (5 ) scho l -

a r l y o r p r o f e s s i o n a l a cco mp l i s h men t s o f f acu l t y ; an d ( 6 ) i n n o v a t i v en es s o f

cu r r i cu l u m an d p ed a g o g y (A s t i n an d S o l mo n , 1 98 1, p . 1 7 ). P r o g r am s r an k ed

in the t o p t en in a t l eas t one o f the s ix c r i t e ri a and which w ere no t r an ked by

R o o s e an d A n d e r s en (1 97 0) w e re li st ed . A s ex am p l e s , t h e t o p - r a t ed p r o g r am s

i n eco n o mi cs f i tt in g t h e t w o co n d i t i o n s w e r e t h o s e a t S w ar th mo r e , D a r t -

m o u t h , H a v e r f o r d , A m h e r s t , a n d C l a r e m o n t M e n ' s , w h il e t h e t o p - ra t e d

p r o g r a m s i n b i o l o g y w e r e a t R e e d , A m h e r s t, C a r l e to n , P o m o n a , a n d B o w -d o i n . M o r e im p o r t a n t , A s t in a n d S o l m o n f o u n d t h a t r e p u t a t io n o f o n e f ie ld

a t an i n s t i tu t i o n co u l d p r o v i d e a rea s o n ab l e e s t i ma t e o f r ep u t a t i o n o f o t h e r

u n d e r g r ad u a t e f i e l d s a t t h e s ame i n s t i t u t i o n . R a t e r b i a s an d h a l o e f f ec t s ,

A s t i n an d S o l m o n ex p l a i n ed , h ad a f f ec t ed t h e i r r ep u t a t i o n a l r a t in g s . F u r -

t he r , A s t i n an d S o l m o n f o u n d t h a t r ep u t a t i o n c o u l d b e e s t i ma t ed f a i rl y

accu r a t e l y f r o m k n o w n f ac t s o f t h e in s t i tu t i o n , s u ch a s ex p en d it u r e s f o r

l ib r a ri e s, p h y s i ca l p l an t , an d ed u ca t i o n a l p u r p o s es , an d t h e n u m b er o f d e -

g rees aw arded in the na tu ra l sc i ences and eng ineer ing . Given the h igh cos t o f

co n d u c t i n g r ep u t a t i o n a l s t u d i e s an d t h e l i m i ta t io n s i n h e ren t i n t h e ir r ep u t a -

t i o n a l r a t in g s , A s t i n an d S o l m o n s till s aw a n eed f o r mo r e r ep u t a t i o n a l



AS8ES8MENT OF QUALITY 243

studies, especially those covering more fields and more diverse quality

criteria.

Cr i t ique of R eputat iona l S tudJes

Reputational studies have been criticized for their methodological limita-

tions. The lack of a clear definition or direct measurement of quality is

usually criticized. Some critics insist that what is actually measured in these

studies is renown or reputation (Dolan, 1976; Lawrence and Green, 1980).

Second, a problem with rater and alumni bias is likely to occur if members

of the panel of evaluators are graduates of the programs they are rating

(Webster, 1981; Cartter, 1966; Lawrence and Green, 1980) or if they are

unfamiliar with the programs they are asked to evaluate (Blackburn and

Lingenfelter, 1973; Lawrence and Green, 1980). Third, a program can be

judged to be better in reputation than another only when they are compared

on a relative scale. Fourth, reputational studies have been criticized for

having focused mostly on graduate programs and the top 20 to 150 institu-

tions in the country. Critics point out that virtually no information exists

about the status of teachers' colleges, regional state colleges and universities,

women's colleges, predominantly black colleges and universities, profes-

sional schools, and other institutions which also perform a vital higher

education function (Petrowski et al., 1973; Conrad and Blackburn, 1985a).Fifth, reputational studies have been criticized for their part in creating the

so-called Hertz-Avis effect by establishing a pecking order among institu-

tions of higher education (Lawrence and Green, 1980; Roose and Andersen,

1970). Sixth, some reputational studies have been criticized for not consider-

ing the institutional environment, such as institutional size and student

cultural mix, which can affect reputat ional ratings (Astin and Solmon, 1981;

Dolan, 1976; Lawrence and Green, 1980). Finally, some critics argue that

some reputational rankings do not reflect the current reputation and prac-

tice of the departments owing to the fact that many ratings are based on

dated data and impressions (Lawrence and Green, 1980; Cox and Catt,1977).

Such criticisms notwithstanding, reputational studies have many merits.

In particular, they have been very informative about the excellence of aca-

demic programs at the doctoral level. Accordingly, many reseärchers urge

the continuance of reputational studies, particularly those with improved

methodologies. Such a sentiment is expressed by Conrad and Blackburn

(1985a, p. 23).

Controversy over reputational studies should not deter researchers from conduct-

ing such studies in the future. If reputational studies are designed to respond to thecriticisrns raised here, we are persuaded that they can make a rnore irnportant



244 TAN

con tribution to evaluating quali ty in higher education. Especially if method ologi-cal refinements are made, if quality is evaluated through multiple criteria, if

normat ive s tandards of qua l i ty a re used , and i f qua l i ty i s eva lua ted not jus t a t"leading " schools, reputa tional studies ma y help to shed fu rther l ight on programquali ty at the und ergrad uate an d professional as weil as the graduate level.

O B J E C T I V E I N D IC A T O R S T U D I E S

I n s t e a d o f u s i n g r e p u t a t i o n a l e v a l u a t io n s f r o m f a c u lt y , d e p a r t m e n t h e a d s ,

a n d d e a n s , o b j e c ti v e i n d i c a t o r s t u d i e s h a v e b e e n a i m e d a t a s s es s in g q u a l i t y

b y t h e u s e o f o b j e c t i v e m e a s u r e s . S i n c e li t tl e t h e o r y e x i s ts t o g u i d e r e s e a r c h -

e r s a b o u t w h i c h v a r i a b l e s a r e a c c u r a t e f o r m e a s u r i n g q u a l i t y , r e s e a r c h e r s

h a v e s e l e c te d t h e m a t t h e i r o w n d i s c r e t i o n . A s a r e s u l t , a v a r i e t y o f v a r i a b l e sh a v e b e e n u s e d . T h e s e v a r i a b l e s , h o w e v e r, c a n b e c a t e g o r i z e d i n t o f i v e g e n -

e r a l t y p e s : t h o s e d e a l i n g w i t h ( 1 ) f a c u l t y , ( 2 ) s t u d e n t s , ( 3 ) i n s t i t u t i o n a l o r

d e p a r t m e n t a l r e s o u r c e s , (4 ) o u t c o m e s , a n d ( 5) m u l t i p l e c r i t e ri a .

S t u d i e s B a s e d o n F a c u l t y

P e r h a p s t h e m o s t w i d e l y k n o w n o b j e c t iv e i n d i c a t o r s t u di e s a re t h o s e

b a s e d o n f a c u l t y . R e s e a r c h e r s o f s u c h s t u d i e s c o n t e n d t h a t d e p a r t m e n t a l

q u a l i t y i s d e p e n d e n t o n t h e q u a l i t y o f t h e f a c u l t y . T h e y s u g g e s t t h a t i f

f a c u l t y is s t r o n g i n q u a l it y , t h e n i t n e c e s s a r il y f o ll o w s t h a t t h e d e p a r t m e n tw o u l d o f h i g h q u a l i t y a s w e l l . T h e p r o b l e m , h o w e v e r , l i e s i n t h e l a c k o f a n

a g r e e m e n t o n h o w t h e q u a l i t y o f f a c u l t y c a n b e s t b e m e a s u r e d . M o s t r e -

s e a r ch e r s h a v e u s e d f a c u l t y re s e a rc h p r o d u c t i v i t y t o m e a s u r e q u a l i t y w h i l e

o t h e r s f a c u l t y a w a r d s o r a c a d e m i c o r i g in s o f f a c u lt y .

A n e x a m p l e o f a s t u d y t h a t d i d n o t u s e f a c u l t y r e s e ar c h p r o d u c t i v i t y t o

r a t e p r o g r a m s w a s t h e B o w k e r ( 1 9 6 5 ) r e p o r t . I n t h i s r e p o r t , t h e r e s e a r c h e r

r a t e d g r a d u a t e s c h o o l s i n t w o g r o u p s o f d i s c i p l i n e s , t h e s o c i a l s c i e n c e s a n d

h u m a n i t i e s a n d t h e s c i e n c e s , u s i n g a q u a l i t y i n d e x b a s e d o n f o u r c r i t e ri a : (1 )

t h e n u m b e r o f f o r m e r W o o d r o w W i l s o n f e l l o w s o n t h e f a c u l t y ; ( 2 ) t h e

n u m b e r o f W o o d r o w W i l s o n f e ll o w s a t t e n d i n g t h e i n s t i t u t io n ; ( 3) t h e n u m -

b e r o f A m e r i c a n C o u n c i l o f L e a r n e d S o c i e t i e s A w a r d w i n n e r s o n t h e f a c -

u l t y ; a n d (4 ) t h e n u m b e r o f G u g g e n h e i m M e m o r i a l f e l lo w s o n t h e f a c u l ty .

T h e t o p t e n i n s t i t u t i o n s i n t h e t w o d i s c i p l i n e s w e r e C o l u m b i a U n i v e r s i t y ,

H a r v a r d U n i v e r s i ty , Y ale U n i v e r s i ty , t h e U n i v e r s i t y o f C a l i f o r n i a ( B e r k e le y ) ,

t h e U n i v e r s i ty o f C h i c a g o , S t a n f o r d U n i v e r si ty , th e U n i v e r s i ty o f M i c h i g a n ,

P r i n c e t o n U n i v e r s i t y , I n d i a n a U n i v e r s i t y , a n d t h e U n i v e r s i t y o f P e n n s y l -

van ia (Bowker , 1965) .

W i t h t h e e x c e p t i o n o f t h e B o w k e r s t u d y , m o s t s t u d i e s b a s e d o n f a c u l t y

h a v e u s e d f a c u l t y re s e a rc h p r o d u c t i v i t y a s t h e v a r i a b le f o r m e a s u r i n g q u a l it y .E x a m p l e s o f s u c h s t u d i e s i n c l u d e t h o s e b y S o m i t a n d T a n e n h a u s ( 1 9 6 4 ) ,



ASSESSM EN T OF QU AU T Y 2 45

Lewis (1968), Knudsen and Vaughan (1969), Glenn and Villemez (1970),

Clemente and Sturgis (1974), Wispe (1969), Cox and Catt (1977), Liu (1978),

Gill (1975), and House and Yeager (1978).Somit and Tanenhaus (1964), in their attempt to rate political science

departments, used faculty publication records as the objective measure of

departmental quality: the higher the number of publication records, the

better the quality. The authors, however, were cautious about drawing infer-

ences about lower-ranking institutions. They stated that poor publication

records in lower-ranking institutions did not necessarily mean that faculty at

these institutions were of low quality or deficient in training their students in

research productivity. Heavier teaching loads and the lack of access to ade-

quate library facilities, the authors suggested, might have caused relatively

poor publication records among faculty at these institutions.

In another study, Lewis (1968) also used publication records of faculty as

the measure of quality, this time to assess the top seventeen sociology de-

partments as identified in the Cartter report o f 1966. The publication record

of doctoral students also was considered (this served as a variable for meas-

uring the effectiveness of graduate training). Publication records of both

faculty and students were measured by the number of articles, research

reports and notes, and extended commentaries (excluding letters to the edi-

tor) published in the A mer i ca n S o c io lo g i ca l R ev i ew from 1955 to 1965.

These records formed the bases for ranking departments. Lewis comparedhis derived rankings to those of Cartter and found that, with the exception

of a few programs (such as those at the University of Minnesota, Northwest-

ern University, and Washington University), a consistent relationship existed

between prestige (the reputational measure developed by Cartter) and re-

search productivity (the objective measure).

Utilizing a broadly similar methodology, Knudsen and Vaughan (1969)

also examined the relationship between reputational and objective measures

of quality among sociology departments. The objective measures used hefe

to tank programs were publication records of faculty and program grad-

uates. The publication records were measured in two ways: (1) the totalnumber of publication records, and (2) the number of publication records

per person (to control for size). Scholarly contributions to these •eading

journals were counted: the A mer i ca n S o c io lo g i ca l R ev i ew , the A m e r i c a n

Jour na l o f Soc io logy, and Socia l Forces. A weighted scheme was devised to

measure the significance of each type of scholarly contribution. A weight of

48 was assigned to a theoretical or research monograph, 24 to a textbook, 16

to an edited book, 16 to an article in the A m er i ca n S o c io lo g i calR ev iew , 12 to

an article in the Am er ic an Journa l o f Sociology , and 8 to an article in Socia l

Forces or a research note in the Amer ican Soc io log ica l Rev iew. This

weighted scheine enabled every department's research productivity to be



2 4 6 T A N

me as u r ed an d r an k ed . R an k i n g s d e r i v ed t h is w ay w er e co m p ar e d t o t h e

r ep u t a t i o n a l r a t in g s o f C a rt te r . O v e r a ll , K n u d s en an d V au g h an f o u n d s i mi -

l a r it ies be tw een the two ra t i ngs bu t fo r on ly the d i s t i ngu i shed o r c l ear lysup er io r i n s t i t u ti ons . For example , t hese t h i rt een i ns t i tu t i ons were r anked in

t h e t o p f i f t e en o n b o t h l i s t s - t h e U n i v e r s i t y o f C a l i f o r n i a ( B e r k e l ey ) , H a r -

v a r d U n i v e r s i t y , t h e U n i v e r s i t y o f C h i cag o , t h e U n i v e r s i t y o f M i ch i g an ,

C o l u m b i a U n i v e rs i ty , P r i n ce t o n U n i v e r si ty , t h e U n i v e r s i t y o f W i s co n s i n , t h e

U n i v e r s i t y o f C a l i f o r n i a ( Lo s A n g e l e s ) , S t an f o r d U n i v e r s i t y , N o r t h w es t e r n

Univer s i t y , Yale Un iver s i t y , Corne l l U n iver s it y , and the U niver s i t y o f M inne-

so t a . Below the t op f i f t een i ns ti t u t ions , however , t he t he r a t i ngs were no t as

co n s i s t en t (K n u d s en an d V au g h an , 1 96 9).

A n o t h e r o b j ec t i v e i n d i ca t o r s t u d y o f s o c i o l o g y d ep a r t men t s w as a r ep o r t

b y G l en n an d V i ll imez ( 1 97 0) w h i ch ex am i n ed f o r t y - fi v e p r o g r am s . Th i s

s t u d y w as a r ep l i ca ti o n o f t h e K n u d s en - V a u g h an s t u d y ex cep t t h a t i t co v e r ed

a d i f f e r en t p e r i o d an d i n c l u d ed a w i d e r ran g e o f p u b l i ca ti o n s . R eca l li n g t h e

K n u d s en - V au g h an s t u d y , p u b l i ca t i o n r eco r d s f r o m t h e American Sociologi-

cal Review, th e Amer ican Journa l o f Soc io logy , a n d Social Forces were

coun ted . In t he Glenn-Vi l l imez s tudy , however , con t r i bu t ions t o n ine t een

o t h e r j o u r n a l s w e r e co n s i d e r ed , i n c l u d i n g S o c io l o g y o f E d u ca ti o n , A m er i -

can Sociologis t, Social Prob lems, a n d Soc ial Science Quarterly ( f o r a co m-

p le t e l is t , see Gle nn an d V i ll imez , 1970 , p . 246) . An o th er d i f f e rence f rom the

K n u d s en a n d V au g h an s t u d y w as t h a t G l en n an d V i ll imez co u n t ed r e s ea r chn o t e s an d l o n g co mmen t a r y p ap e r s i n ad d i t i o n t o b o o k s an d a r t i c l e s . Th e

r e s ea rch e r s a l s o r ev is ed t h e w e i g h ted s ch eme d es i g n ed b y t h e i r p r ed eces s o r s

an d ca l led i t t h e G l en n -V i l li mez C o m p r eh en s i v e I n d ex . F o r ex amp l e , i n s tead

o f a w e i g h t o f 4 8 a ss i g n ed to t h eo r e t ica l o r r e s ea rch m o n o g r ap h s , a 3 0 w as

as s i g n ed ( w e i g h t s w e r e b a s ed o n q u es t i o n n a i r e r e s p o n s es f r o m a s amp l e o f

109 soc io log i s t s who were asked to ass ign weigh t s t o each k ind o f pub l i ca-

t i o n ) . U s i n g t h e G l en n - V i l l i mez C o mp r eh en s i v e I n d ex , w h i ch w as c l a i med

a s a m o r e a d e q u a t e m e a s u r e o f t h e q u a n t i ty a n d q u a l it y o f r e se a rc h p r o d u c -

t i v it y o f s o c i o l o g is t s t h an t h e K n u d s en - V au g h an i n d ex , d o c t o r a l p r o g r am s

w er e r a t ed an d t h e r a t i n g s co mp ar ed t o t h o s e g en e r a t ed b y K n u d s en an d

V au g h an . G l en n an d V i l li mez f o u n d co n s i s ten cy b e t w ee n b o t h r a t in g s f o r

o n l y th e t o p e l i te in s t it u t io n s . Th e h i g h l y ra t ed p r o g r am s a t C o l u m b i a U n i -

v e r s i t y , t h e U n i v e r s i t y o f C h i cag o , t h e U n i v e r s i t y o f M i ch i g an , H a r v a r d

U n i v e r s i t y , t h e U n i v e r s i t y o f W i s co n s i n , an d t h e U n i v e r s i t y o f C a l i f o r n i a

(Berke l ey ) ha d r ece ived a s imi l a r r ank on b o th l is t s. B e low these i ns t i t u t ions ,

t he d i f f e rences i n t he r anks were qu i t e s i gn i f i can t (Glenn and Vi l l imez ,

1970).

In an a t t empt t o r ep l i ca t e t he Glenn-Vi l l imez s tudy , C lemen te and S tu rg i s

(1 97 4) u s ed t h e G l en n -V i l li mez C o m p r eh en s i v e I n d ex t o co n v e r t t h e p u b l i ca -t i o n d a t a o f 2 ,20 5 P h .D . h o l d e r s i n s o c i o lo g y f r o m t h e y ea r s 1 9 4 0 -7 0 in t o



ASSESSMENT OF QUALfTY 247

f acu l t y r e s ea rch p r o d u c t i v i t y s co r e s. Th e ma i n p u r p o s e o f t h e s t u d y w as t o

ex ami n e i f a co r r e la t i o n ex i st ed b e t w ee n t h e q u a l i ty o f g r ad u a t e t ra i n in g an d

r e s ea rch p r o d u c t i v it y . Th e q u a l i t y o f g r ad u a t e t r a i n in g w as r ep r e s en ted b yfou r r epu ta t i ona l r a t i ngs . T he f i r s t t h ree r a ti ngs were com pi l ed d i r ec t l y f rom

the Ken i s ton (1959), Ca r t t e r (1966) , and Ro ose -A nde rsen (1970) s tud i es . Th e

f o u r t h r a ti n g, c r ea t ed b y C l emen t e a n d S t u rg i s , w as e s s en t ia l ly a co mp o s i t e

d e r i v ed b y co m b i n i n g t h e f i rs t t h ree r a t in g s . W h en ze r o - o r d e r co r r e l a t io n

an a l y s is w as u s ed t o t e s t t h e co r r e l a t i o n b e t w ee n g r ad u a t e t ra i n in g ( i n d ica t ed

by a ll fou r r a t i ngs ) and r esearch p rodu ct iv i t y , t he r esu l ts r evea l ed on ly a

w eak r e l a t io n s h i p b e t w ee n th e t w o ( C l em en t e an d S t u rg i s , 1 97 4).

I n a s t u d y d ea l in g w i th p s y ch o l o g y d ep a r t me n t s , W i s p e (1 96 9) u s ed p u b l i-

ca t i o n r eco r d s o f t h e facu l t y t o ra t e d ep a r t me n t s . B u t t h e ma i n p u r p o s e o f

h e r s t u d y w a s to e x a m i n e t h e r e a s o n f o r p r o d u c t i v it y o f p s y c h o l o g y d e p a r t -

m e n t s : W a s o n e d e p a r t m e n t m o r e p r o d u c t i v e ( as m e a s u r e d b y p u b l i c a t io n

r eco r d ) t h an an o t h e r b ecau s e i t w as ac t u a l l y b e t t e r i n q u a l i t y o r b ecau s e i t

w as b ig g e r ? W i s p e w o r k ed w i t h a s amp l e o f tw en t y - e i g h t p s y ch o l o g y d ep a r t -

m en t s w h i ch b e t w een 1 8 9 5 an d 1 9 6 5 h ad g r an t ed a t l e a s t o n e d o c t o r a t e i n

p s y c h o l o g y an d f o u n d t h a t d e p a r t m en t s ize w as r e la t ed to r e s ear ch p r o d u c -

t iv i ty . W ispe foun d a r a ther h igh co r re l a t i on be twe en the two var i ab les : .82

(p < .01 ) fo r 1950 and .86 (p < .01) fo r 1960 . Other r esearcher s such as Drew

(1975) , Guba and C lark (1978) , E l ton and Rodger s (1971) , E l ton and Rose

(1972) , Hags t rom (1971) , and Knudsen and Vaughan (1969) had found s imi -l a r r esu l t s . At t he underg raduat e l eve l , however , depar tmen t s i ze has no t

b een f o u n d t o b e r e l a t ed t o q u a l it y . Th i s f in d i n g w as co n f i r m ed b y A s t i n an d

S o l m o n (1 98 1), w h o f o u n d a n eg a t iv e co r r e l a t i o n b e t w een t h e t w o v a r i ab l e s.

A n o t h e r s t u d y o f p s y c h o l o g y d e p a r t m e n t s w a s a r e p o r t b y C o x a n d C a t t

( 19 77 ) w h i ch r a ted p s y ch o l o g y d ep a r t m en t s o n t h e b a s i s o f f acu l t y s ch o l a r ly

c o n t r i b u ti o n s t o t h ir te e n j o u r n a l s s p o n s o r e d b y t h e A m e r i c a n P s y c h o l o g i c a l

A s s o c i a t i o n b e t w een th e y ea r s 1 9 70 an d 1 97 5. W h en t h e r a ti n g s w e r e co m-

p a r e d t o r e p u t a ti o n a l r a ti n gs o f th e s a m e d e p a r t m e n t s b y R o o s e a n d A n d e r -

s en ( 1 9 7 0 ) , C o x an d C a t t f o u n d t h a t r ep u t a t i o n ( t h e R o o s e - A n d e r s en r a t -

i n gs ) w as n o l o n g e r r e f l ect iv e o f t h e c u r r en t s ch o l a r l y p r o d u c t i v i t y o f t h e s edep ar tm en t s ( c f. Law rence and Green , 1980 , p . 10 ).

N o t a l l ob j ec t i ve i nd i ca to r s t ud i es us ing f acu l ty research p ro duc t iv i t y as

t h e m e a s u r e m e n t o f q u a l i ty h a v e f o c u s e d o n p s y c h o l o g y a n d s o c i o lo g y p ro -

g rams; a f ew have focu sed on o th er d i sc ip l ines . L iu ' s s t udy (1978) , fo r

ex amp l e , ran k ed m ech an i ca l en g i n ee r in g d ep a r t m en t s o n t h e b a s is o f facu l t y

c i t a ti o n co u n t s . A s am p l e o f 2 3 2 p r o f e s s o r s an d a t o t a l o f 3 ,2 6 4 c it a t io n s i n

t h e 1 9 7 5 ed i t io n o f t h e Science Citation Index were used in t he s tudy . L iu

u s ed th e av e r ag e c it a t io n co u n t t o d e t e r mi n e d e p a r t m en t a l r an k i n g s b ec au s e

s h e b e li ev ed i t w as a b e t t e r m eas u r e t h a n t h e t o t a l n u m b er o f c i t a t io n s .

W h en L i u co m p ar ed h e r r an k in g s t o t h o s e d ev e l o p ed b y G il l (1 97 5), s h e



248 TAN

found vast differences between the two rankings except for one interesting

aspect. The top five departments (those at Stanford University, the Univer-

sity of Minnesota, Massachusetts Institute of Technology, the CaliforniaInstitute of Technology, and the University of California) were the same for

both studies except that the rank order was different. Another study on a

different discipline was conducted by House and Yeager (1978), which

assessed economics departments based on the total number of pages pub-

lished by full professors in forty-five leading journals in the discipline.

It should be pointed out that not all studies have used faculty research

productivity as the indicator of quality. In addition to the faculty character-

istics used by Bowker (1965), a few scholars have used years of teaching

experience, academic origins of the faculty, faculty awards, and the like, but

as Blackburn and Lingenfelter (1973) explained, most of these indicators

were not as valid as faculty research productivity in indicating departmental

quality.

The use of faculty characteristics as indicators of quality is not as widely

accepted as might be expected. The first problem causing a difference in

opinion has been the assumption that faculty research productivity derer-

raines faculty quality and therefore the two are equal to each other. Some

researchers have argued that the two are not necessarily the same. The

second problem causing the difference in opinion has been the assumption

that departmental quality is best characterized by the quality of faculty(measured by faculty research productivity). Student ability, student out-

comes, and the extent of financial resources have been cited as factors

equally important in characterizing high quality departments. Therefore,

future research is needed to explore the extent to which faculty and other

characteristics play in the enhancement of quality.

Studies Based on Students

While faculty characteristics have often been used to measure quality,

some researchers have used student characteristics. Researchers that haveused student characteristics to measure program quality include Knapp and

Greenbaum (1953), Krause and Krause (1970), Dubé (1974), Tidball and

Kristiakowski (1976), Astin and Henson (1977), Astin and Solmon (1979),

and Glower (1980).

In the Knapp-Greenbaum study, institutions were rated based on the pro-

port ion of their alumni who earned doctoral degrees; in the Krause and

Krause study, undergraduate colleges were rated according to the number of

their baccalaureate graduates who had contributed articles to Sc ien t i f i c

Arne r i can between 1962 and 1967; in the Dubé study, one hundred under-

graduate colleges were rated according to the number of graduated students



ASSESSMENT OF QUALJTY 249

who were successful in gaining admission to medical schools in 1973-74;

in the Tidball-Kristiakowski study, colleges were rated according to the

number of their alumni who earned doctoral degrees; in the Astin-Hensonstudy, the student selectivity index (which measured the average academic

ability of entering freshmen) was used to measure institutional quality; in

the Astin-Solmon study, college preferences of highly able students were

used as a basis for measuring quality; and finally, in the Glower study,

successful careers of alumni evident in biographical data in the W.ho's Who

in Engineering were used to measure the quality of engineering schools.

The use of student characteristics as indicators of program quality is not

without criticisms. Even though "student quality can stand in its own right

as a criterion of excellence" because "well-qualified students are an essential

element of an excellent program" (Blackburn and Lingenfelter, 1973, p. 8),

little empirical evidence exists supporting the direct linkage between student

excellence and program quality. Most studies at best have been able to show

certain student characteristics (such as student ability and the number of

students) to be positively correlated with program quality, especially at the

graduate level, but they have not by themselves explained a large amount of

variance in prograrn quality. Furthermore, an agreement cannot be reached

on the type of student variables that could be used to indicate student

quality. Kuh (1981) suggested the use of the student undergraduate experi-

ence as an indicator of student excellence, but that too is difficult to meas-ure. Most researchers, according to Kuh, have relied on surrogate indices of

the student experience instead of direct measurements. Evidently, the devel-

opment of more direct indices of the student experience and of student

excellence in general is needed.

Studies Based on Resources

In addition to faculty and student characteristics, some researchers have

used resources to measure quality. These researchers suggest that quality cän

be measured by departmental, institutional, and human resources. Exam-

ples of such measurements include the number of faculty, staff, and stu-

dents; the value of physical facilities (libraries, laboratories, office spaces,

computer facilities); endowments; expenditures per student and per facul ty;

faculty salaries; research funds; departmental program offerings; and the

diversity of programs. Although rauch discussion has been made about

resources being an important factor in nurturing quality, few studies have

been able to establish the direct linkage between the two.

Examples of studies that have used resources to measure quality were

reports by Kelso (1975), Glower (1980), and the Carnegie Council on PolicyStudies in Higher Education (1976). Kelso ranked law schools on the basis o f



250 TAN

a resources index. Glower (1980) used the total number of graduates listed in

the 1977 issue of the Who's Who In Engineering and the total amount of

research spending to rank engineering schools. Glower speculated that theuse of resources as the measure of departmental quality was more reliable

than reputational surveys because this method can rank all programs, not

just the highly visible ones.

Perhaps the best known study that used resources to rank institutions was

a report by the Carnegie Council on Policy Studies in Higher Education

(1976). In this report, the Carnegie Council classified institutions into six

broad categories based on the amount of federal financial support received

by an institution, the number of Ph.D.s awarded by the institution, and the

institution's student selectivity index. Institutions that led in terms of the

amount of federal funding received and the number of Ph.D.s (including

M.D.s if the institution has a medical school) awarded in 1973-74 were

classified as doctorate-granting institutions. The next classification of insti-

tutions were comprehensive universities and colleges, followed by liberal arts

colleges, two-year colleges and institutes, professional schools and other

specialized institutions, and finally institutes for nontradi tional study.

The extent to which resources can be used as indicators of quality has not

been entirely understood. While on the one hand speculations have been

made about financial resources being an important factor in nurturing qual-

ity, little research evidence exists proving the direct linkage between the two.Abbott and Barlow (1972), for example, in their study of sociology pro-

grams, could not find enough evidence to substantiate this linkage. Instead,

they found variables such as faculty research productivity and the number

and type of degrees awarded to be intervening variables between resources

and prestige. Resources may be a factor in the enhancement of qual ity but

evidently not in a direct way.

Studies Based On Outcomes

Researchers in this category of studies suggest that the quality of depart-ments or programs is not as dependent on inputs (institutional and depart-

mental resources) as outcomes. Outcome variables are products of students

and alumni.

A study that used outcomes to measure quality was by Knapp and Good-

rich (1952). Knapp and Goodrich used the proportion of institutional

alumni who had earned doctoral degrees to measure insti tutional quality. In

a follow-up study, that variable was replaced by the proportion of institu-

tional alumni who had won graduate fellowships (Knapp and Greenbaum,

1953).

Other studies in this category included those by Tidball and Kristiakowski



ASSESSMENTOF QUALITY 251

(1976) , t he Carneg ie Counci l on Po l i cy S tud i es i n Higher Educa t ion (1976) ,

an d t h e N a t i o n a l S c i en ce B o a r d ( 1 9 6 9 ) . T i d b a l l an d K r i s t i ak o w s k i u s ed t h e

n u m b e r o f a l u m n i w h o e a r n e d d o c t o r a l d e g re e s as th e m e a s u r e o f q u a li ty ;an d b o t h t h e C a r n eg i e C o u n c i l an d t h e N a t i o n a l S c i en ce B o a r d u s ed t h e

n u m b e r o f d o c to r a t e s a w a r d e d .

Th e ag r eemen t t o t h e u s e f u l n es s o f o u t ¢ o mes a s i n d i ca t o r s o f q u a l i t y i s

n o t a s d ec i s i v e a s mi g h t b e ex p ec t ed . S o me r e s ea r ch e r s c l a i m t h a t h i g h l y

p r o d u c t i v e s t u d en t s an d s u cces s f u l a l u mn i m ak e a h i gh q u a l i t y d ep a r t m en t ,

bu t l i t t l e empi r i ca l ev idence i s ava i l ab l e t o suppor t t h i s c l a im. As a r esu l t ,

s o m e re s ea r ch e rs an d acad em i cs a r e n o t en ti re l y co n v i n ced t h a t o u t co m es

are accura~e ind i ca to r s o f p rog ram qual i ty . A s t in (1962), fo r example , p ro -

v i d es ev i d en ce t h a t m o s t s t u d en t l e a r n in g o u t co m es w er e m o r e d ep e n d en t o n

t h e q u a l i t y o f s tu d en t s t h an o n t h e q u a l i t y o f t h e p r o g r am. Th e r e l a ti o n s h i p

b e t w een i n p u t s ( su ch a s s t u d en t ab i l it y ), o u t co m es , an d p r o g r am q u a l i t y

n eed s t o b e ad d r e s s ed i n f u t u r e r e s ea rch .

Studies Based On Mul t ip le Cr i ter ia

Stud ies d i scusse d so f a r i n th i s sec t i on hav e genera l l y u t il ized t he un ivar i -

a t e ap p r o ach t o meas u r i n g q u a l i t y . H o w ev e r , s o me s t u d i e s h av e emp l o y ed

t h e mu l ti v a r ia t e ap p r o a ch . P e r h a p s t h e m o s t v i s ib l e y e t co n t r o v e r si a l s tu d i e s

u t il iz ing t h is app roa ch a re t he G ou rm an repo r t s (1967 , 1977a , 1977b , 1982 ,1983). These r epo r t s have be en severe ly and jus t i f i a b ly c r i ti c ized b eca use t he

m e t h o d o l o g y u s ed t o d e r iv e t h e r a ti n g s w as n o t ad eq u a t e l y ex p la i n ed . S i n ce

f ig u r e s w e r e mer e l y r ep o r t ed w i t h o u t an a d eq u a t e ex p l an a t i o n , Law r en ce

and Green (1980 , p . 36 ) ca l l ed t he Gourman ra t i ngs " id iosyncra t i c and

unrep l i cab l e . " Yet ano ther c ri tic , W ebs t er (1984) ques t i on ed the c red ib i l it y o f

J ack G o u r m an a s a ra t e r o f i n s t it u t io n s a n d p r o v i d ed g o o d ev i d en ce t h a t t h e

G o u r m a n r e p o r ts h a v e m a n y s e r io u s fl aw s .

A m o r e s c h o l a r l y w o r k b y B r o w n ( 1 96 7) u s ed e i g h t o b j ec ti v e v a r i ab le s t o

ra t e co l leges : (1 ) t he p r op or t i o n o f f acu l ty wi th t he doc to ra t e ; (2) t he average

s a l a r y an d f r i n g e b en e f i t co mp en s a t i o n p e r f acu l t y me mb er ; ( 3 ) t h e p r o p o r -

t i o n o f s t u d en t s co n t i n u i n g t o g r ad u a t e s c h o o l ; ( 4) t h e p r o p o r t i o n o f g r ad u -

a t e s tuden t s ; (5 ) t he number o f vo lumes i n l i b rary per fu l l - t ime s tuden t ; (6 )

the t o t a l number o f fu l l - t ime facu l ty ; (7 ) t he f acu l ty - s tuden t r a t i o ; and (8 )

t h e t o ta l cu r r en t i n co m e p e r s t u d en t . A t o t a l o f 1 ,1 21 an o n y m o u s i n s ti tu -

t io n s w e r e r an k ed ag a i n s t e ach o t h e r o n each o f t h e e i g h t c ri te r ia ab o v e ;

hence a sco re f rom 1 to 1 ,121 was ass igned to e ach ins t i t u t ion fo r each

cr i te r i a . B y averag ing the e igh t sco res fo r each ins t i t u t i on , Brow n ar r ived a t

an o v e ra l l r an k o r w h a t h e ca l l ed a co m p o s i t e r a t in g f o r each i n s t it u t io n .

S ince qua l i t y i s a mul t i d imens iona l en t i t y , t he use o f mul t i p l e var i ab l esmay p e r h ap s b e t h e i d ea l ap p r o ach t o d e r i v i n g an i n d i ca t o r o f q u a l i t y .



252 TAN

U n f o r t u n a t e l y , f ew r e s ea rch e r s h av e u t il iz ed t h is ap p r o ach . Th e b as i c p r o b -

l em h as b een t h e d i f f i cu l ty in v o l v ed i n s e lec t in g t h e " r i g h t" co m b i n a t i o n o f

var i ab l es t o i nd i ca t e qua l i t y . Th i s wi l l con t inue t o be a p rob lem in t heab s en ce o f a t h eo r y o f q u a l i t y . B u t t h e b e s t ch an ce f o r a t h eo r y t o b e

d ev e l o p ed is th r o u g h t h e u s e o f m u l t ip l e v a r i ab le s . Th i s ch i ck en - eg g p r o b l em

wi l l need to be r eso lved in fu tu re s tud i es .

Cr i t i que o f Ob j ec t ive I nd i ca t o r S t ud i es

O b j ec t i v e i n d i ca t o r s t u d i e s c l ea r l y h av e co n t r i b u t ed a g r ea t d ea l t o w ar d

o u r u n d e r s t an d i n g o f q u a l i t y i n h ig h e r ed u ca t i o n . S i g n if ican tl y , t h ey hav e

b r o u g h t t o l i g h t an en t i r e l y d i f f e r en t ap p r o ach t o t h e meas u r emen t o f q u a l -

i ty : th e u s e o f o b j ec t iv e v a r i ab le s . T h u s , t h e m a j o r s t ren g t h o f th i s ap p r o ach

is th a t t h e r e i s an o b j ec t i v e m eas u r em en t o f q u a l it y .

A t t h e s am e t ime , o b j ec t i v e i n d i ca t o r s t u d i e s a l so s u f f e r f r o m v a r i o u s

me t h o d o l o g i ca l l i m i t a t i o n s . O n e l i m i t a t i o n i s t h a t mo s t s t u d i e s h av e o f t en

u s ed f acu l t y r e s ea r ch p r o d u c t i v i t y a s t h e o n l y meas u r emen t o f q u a l i t y . I n

m an y o f t h e s e s t u d i e s , r e s ea rch e r s h av e b een c r i ti c ized f o r eq u a t i n g f acu l t y

q u a l i t y t o p r o g r am q u a l i t y . S o me c r i t i c s h av e a r g u ed t h a t f acu l t y an d p r o -

g r am q u a l i t y a re n o t n eces s a r i l y t h e s am e t h in g an d t h e r e f o r e s h o u l d n o t b e

regarded as such (Sm i th and F i ed ler , 19 71 ; L iu , 1978) . Th i s i s a m ajo r

l imi t a t i on o f m os t ob j ec t i ve i nd i ca to r s t ud i es . Oth er c i t ed l imi t a t ions i n -c l u d e ( a ) t h e f a i l u r e o f man y s t u d i e s t o i n c l u d e a s amp l e o t h e r t h an t h e

h igh ly v i s ib le i n s t i t u t ions (Co nra d a nd B la ckb urn , 1985a), (b ) t he i nab i l i t y

o f t h e s e s t u d ie s t o co m e u p w i t h a co n s i s t en t s e t o f o b j ec t i v e meas u r em en t s

t h a t c an b e u s ed t o m eas u r e q u a l i ty f o r a v a r i e ty o f i n s t it u ti o n s , n o t j u s t t h e

t o p - r a t ed o n es ( C o n r ad an d B l ack b u r n , 1 98 5a ), an d ( c) t h e f a il u r e t o i n c lu d e

m ul t i p l e var i ab l es . W ebs t er (1981) , fo r example , a rgued tha t s i nce p ro gram

q u a l i t y is mu l t i d ime n s i o n a l , i t s h o u l d b e me as u r ed u s in g mu l t i p le v a r i ab l e s .

S o m e o f t h e s e l im i t a t io n s n eed t o b e ad d r e s s ed an d p e r h ap s r e s o lv ed in

fu tu re s tud i es .

Q U A N T IT A T IV E C O R R E L A T E S T U D I E S

Un l ike r epu ta t i ona l and ob j ec t i ve i nd i ca to r s t ud i es , quan t i t a t ive s tud i es

h av e n o t b e en d es i g n ed t o d e r i v e a m eas u r em en t o f q u a l it y . R a t h e r, t h i s t y p e

o f s t u d y h as a t t emp t ed t o i d en t i f y v a ri ab l e s t h a t a r e co r r e l a ted w i t h h ig h l y

r ep u t ed p r o g r am s ( r ep u t a t i o n a l r a ti n g s, u s u a l l y t h e A C E r a ti n g s, h av e b een

u s ed a s t h e b a s e s f o r i d en t i fy i n g r ep u t ed p r o g r ams ) . V a r i abl e s f o u n d t o h av e

assoc i a t i on wi th t he chosen r epu ta t i ona l r a t i ngs a re ca l l ed quan t i t a t i ve co r -

r e l a tes o f qua l i ty . N o t a l l s t ud i es , however , have used r ep u ta t i on a l r a t ings as

t h e d ep e n d en t v a r i abl e . N o t i ceab l y , t h e t w o ex cep t io n s t o t h e r u l e a r e s t u d i e s

co n d u c t e d b y A x e l s o n ( 19 60 ) an d b y C o n r ad an d B l ack b u r n ( 19 86 ).



ASSESSMENT OF QUALtTY 253

One of the first researchers who identified quantitative correlates of high

quality programs was Cartter (1966). Even though his study was primarily

concerned with reputational evaluations and ratings, Cartter identified twocorrelates of quality: faculty compensations and library resources. He found

high faculty compensations and library resources to be associated with

highly reputed programs. Note, however, that only a correlation existed

between the two variables, not a cause and effect relationship. That meant

increased library resources or raised faculty compensations would not neces-

sarily cause an increase in the reputat ion of the faculty or vice versa.

Utilizing an approach similar to Cartter's, Oromaner (1970) found depart-

ment size (measured by the number of faculty), the chronological age of the

faculty, and the academic origins of the faculty to be correlates of highly

reputed sociology programs (as rated by Cartter). Specifically, Oromaner

found that the most prestigious sociology departments were characterized bY

department size, a relatively younger faculty, and faculty who were grad-

uates from more prestigious departments. Furthermore, Oromaner found

that distinguished and strong departments were the most productive in

awarding Ph.D.s to students. These departments conferred 45% of the 1,560

degrees awarded between the years 1955 and 1964.

In another study of sociology departments, Abbott (1972) investigated

institutional and departmental variables as potential quantitative correlates

of quality (as indicated by the ACE ratings). Institutional variables exam-ined were (a) the total university research productivity, (b) the total number

of doctorates conferred, and (c) the total number of full-time faculty em-

ployed. Departmental variables were the same. The results of the study

indicated that institutional variables were more highly correlated with the

ACE ratings than departmental variables. Accordingly, Abbott (1972) con-

cluded that the rating of sociology departments was not so much a function

of what went on in a department in isolation as it was of the university as a

whole.

Another study of sociology departments by Abbott and Barlow (1972)

initially examined annual faculty income, library volumes, the number offull-time faculty, and the value of pltysical facilities within an ins titution as

potential correlates of quality. However, Abbott and Barlow did not find any

substantial evidence to show these variables as being strong correlates of

quality. Instead, they found outcome variables (or functions, as the re-

searchers identified them), namely, faculty research productivity and the

number and type of degrees awarded by an institution, to be stronger corre-

lates. Accordingly, these functions were called intervening variables between

resources and prestige.

In still another study of sociology programs, Axelson (1960) attempted to

identi fy the correlates of sixty-five doctoral programs. A marked difference

of this study was that the researcher did not rely on Cartter's ratings to



25 4 TAN

i d en t i f y h i g h l y r ep u t ed p r o g r ams . I n s t ead , h e r a t ed p r o g r ams acco r d i n g t o

t h e av e r ag e n u mb er o f f acu l t y p u b l i ca t i o n s i n t h e American Sociological

Review, th e Amer ican Journa l o f Soc io logy , a n d Social Forces. U s i n g h i so w n ra t in g s a s th e d ep en d en t v a ri ab le , A x e l s o n f o u n d t h a t t h e mo s t p r o d u c -

t i v e p r o g r ams w er e ch a r ac t e r i zed b y f acu l t y memb er s w h o w er e mo s t l y

a lum ni f rom l a rger in s t i t u t ions ( espec i aUy f rom l a rger m idwe s t ern un iver -

s it ie s ) w h il e t h e l e a s t p r o d u c t i v e p r o g r ams w er e ch a r ac t e ri zed b y f acu l t y

m em b er s w h o w er e g r ad u a t e s f r o m s m a l le r i n s ti t u ti o n s ( s u ch a s fr o m

smal l e r Catho l i c un iver s i t i es ) . Ev iden t ly , t he o r ig in o f g raduat e t r a in ing

am o n g s o c i o lo g i s ts w as a g o o d p r ed i c t o r o f h ig h r e s ear ch p r o d u c t iv i ty .

S t u d i e s d is cu s s ed s o f a r h av e f o cu s ed o n s o c i o l o g y p ro g r ams ; t h e r e w e r e

s o m e t h a t f o c u s ed o n o t h e r d i s c ip l in e s . A s t u d y b y E l t o n a n d R o d g e r s

(1971) , fo r example , examined the co r re l a t es o f phys i cs depar tmen t s . Var i -

ab l e s ex ami n ed a s p o t en t i a l co r r e la t e s o f C a r t t e r 's r a ti n g s o f p h y s ic s d ep a r t -

men t s w e r e ( 1 ) t h e n u mb er o f a r ea s o f s p ec i a l i z a t i o n w i t h i n a d ep a r t men t ;

( 2 ) t h e n u mb er o f f acu l t y ; ( 3 ) t h e n u mb er o f P h .D . s aw ar d ed b e t w een 1 9 6 0

and 1964 ; (4) t he nu m be r o f fu ll - time s tud en t s ; (5) t he nu m be r o f f ir s t -year

s tuden t s ; and (6) t he r a t i o o f par t - t ime to fu l l- t ime s tud en t s . A m ul t i p le

d i scr iminan t ana lys i s us ing t hese var i ab l es r evea l ed t ha t co l l ec ti ve ly t hey

were very goo d co r re l a t es o f t h e A CE ra t ings ( t he t a was 75 % ) . Spec i f ica l ly ,

t h e s e v a r i ab l e s w e r e f o u n d t o b e mo s t e f f i c i en t a t p r ed i c t i n g d ep a r t men t s

r a t ed b y C a r t t e r t o b e accep tab lep lus and l eas t e f f i c i en t wh en p red i c t i ng lessthan adequate p lus d ep a r t men t s . O v e r a l l, t h e r e sea r ch er s p o i n t ed o u t , t h e s e

" o b j ec t i v e v a r i ab le s p r o v i d ed a g o o d a p p r o x i m a t i o n o f t h e n u m er i ca l r a ti n g s

o f t h e g r ad u a t e p r o g r ams i n t h e C a r t t e r s t u d y " ( E l to n an d R o d g e r s , 1 97 1,

p. 568).

A f o l lo w - u p o f t h a t s t u d y a y ea r la t e r w as co n d u c t e d b y E l t o n an d R o s e

(1972) . In t h i s s tudy , t he r esearcher s exam ined psy cho logy depar tm en t s wi th

the same s ix p red i c to r var i ab l es used in t he i r ear l i e r s t udy . Thei r f i nd ings

revea l ed t ha t extremely at tractive d ep a r t men t s a s r a t ed b y C a r t t e r w e r e ch a r -

ac t e r ized b y a r e l at iv e l y l a rg e n u m b er o f f acu l ty , a l a rg e n u m b er o f P h .D . s

awarded , and h igh fu l l - t ime s tuden t en ro l lmen t s ; attractive d e p a r t m e n t s( the second bes t ) were charac t e r i zed by depar tmen t s hav ing r e l a t i ve ly h igh

per fo rmances i n a l l s i x var i ab l es ; acceptable plus d ep a r t men t s w e r e ch a r ac -

t e r i zed by a l a rge num be r o f a reas o f spec i a l i za t ion ; and f ina l ly , the less tha n

acceptable plus d ep a r t men t s ( t h i s c a t eg o r y o f d ep a r t men t s w as c r ea t ed b y

E l t o n a n d R o s e t o i n c l u d e d ep a r t me n t s n o t r an k ed b y C a r t te r ) w e r e ch a r ac -

t e r i zed by a h igh par t - t ime to fu l l - t ime s tuden t r a t i o . F rom these r esu l t s , t he

r e s ea rch e r s co n c l u d ed t h a t t h e " 1 9 66 [ C a r tt e r] r a ti n g s m ay h av e b een b as ed

m o r e o n t h e h a l o e f f ec t d e r i v ed f r o m s ize f ac t o rs t h an o n t h e s t a ted c r i te r ia

i n C a r t t e r 's i n s t ru c t i o n s , s u ch a s acces s ib i li ty o f p r o f e s s o r s . . . " ( E l t o n an d

Rose , 1972, p . 200) . The r esearcher s add ed tha t C ar t t e r h ad to o qu i ck ly




eq u a t ed q u a n t i t y to q u a li ty . E l t o n an d R o s e p o i n t ed o u t t h a t " a l t h o u g h

ed u ca t i o n a l q u an t i t y an d q u a l i t y n eed n o t b e mu t u a l l y ex c l u s i v e , i t may b e

u n w i s e . . , t o co n c l u d e th a t w h e r e q u a n t i t y ex is ts , t h e r e a l s o r e si d e s q u a l it y ,o r converse ly , t ha t i s imposs ib l e fo r qua l i t y t o ex i s t i n t he absence o f l a rge

size" (1972, p. 200).

P e r h a p s t h e b e s t ex amp l e o f a q u an t it a t iv e co r r e l a te s t u d y is t h e H ag s t r o m

(1 97 1) r ep o r t . I n t h is r ep o r t , H a g s t r o m a t t em p t ed t o i d en t i f y t h e co r r e la t e s

o f q u a l i t y t h r o u g h a s er ie s o f z e r o - o r d e r p r o d u c t - r n o m en t co r r e l a t io n an d

mu l t i p l e l in ea r r eg r e s s io n an a l y s e s w i t h t h e C a r t t e r r a t i n g s u s i n g a s am p l e o f

125 dep ar tm en t s o f ma them at i cs , phys i cs , chemis t ry , and b iologyo The inde-

p en d en t v a r i ab le s u s ed i n t h e s t u d y w er e g r o u p ed u n d e r s ix a rea s : (1 ) d ep a r t -

men t s i ze ; (2 ) r esearch p roduct ion ; (3 ) r esearch oppor tun i t i es ; (4 ) f acu l ty

b ac k g r o u n d ; ( 5) s t u d en t ch a r ac t e ri s ti c s ; a n d ( 6) facu l t y aw ar d s an d o f f i ce s .

D e p a r t m e n t s iz e w a s m e a s u r e d b y t h e n u m b e r o f g r a d u a t e f a c u lt y. U n d e r

research p rod uc t ion , f i ve var i ab l es were used : ( a ) t he average research a r t i -

c l es f rom facu l ty dur ing 1961-66 ; (b ) t he average c i t a t i ons o f pub l i shed

r e s ea rch f o r 1 9 6 6 ; (c ) t h e n u m b e r o f r ev] ew a r ti c le s , i 9 6 1 - 6 6 ; ( d ) t h e n u m b er

o f t ex t b o o k s i n car ee r; a n d (e ) t h e n u m b er o f b o o k s i n ca reer. U n d e r r e -

s ea r ch o p p o r t u n i t i e s , t h e t h r ee v a r i ab l e s u s ed w er e ( a ) t h e av e r ag e p r o p o r -

t i o n o f t i me s p en t o n r e s ea r ch ; ( b ) t h e mean r ep o r t ed ea s e o f o b t a i n i n g

ex t r amu r a l r e s ea rch s u p p o r t ; an d ( c) t h e p e r cen t o f t h e f acu l t y h o l d in g

r e s ea rch g r an ts . F acu l t y b ac k g r o u n d , t h e f o u r t h a r ea , co n s i s t ed o f ( a ) t h eq u a l i ty o f t h e d e p a r t m e n t s f r o m w h i c h f a c u lt y g r a d u a t e d w i th t h ei r P h . D . s

an d ( b ) t h e s e l ec t iv i ty o f t h e f acu l t y 's u n d e r g r ad u a t e co ll eg e . U n d e r s t u d en t

ch a r ac te r i st ic s , f o u r v a r i ab l e s w e re u s ed , w h i ch i n c l u d ed ( a) th e m ean n u m -

b e r o f p o s t s ec o n d a r y f e ll o w s i n t h e d ep a r t m en t ; ( b ) t h e u n d e r g r a d u a t e s e lec -

t iv i ty o f t h e en ti re i n s t i tu t i o n ; ( c) t h e m ean n u m b er o f g r ad u a t e r e s ea rch

as s is t an ts ; an d ( d ) t h e me an n u m b er o f t e ch n ic i an s w o r k i n g f o r f acu lt y .

F i na l ly , t h e aw ar d s an d o f f i ce s a r ea w as mea s u r ed b y (a ) t h e ma x i m u m s co r e

f o r aw ar d s r ecei v ed b y an y f acu l t y w i t h i n a d ep a r t men t ; ( b ) t h e m ax i m u m

s c o re f o r t h e n u m b e r o f p a r t ic i p a ti o n s o n g o v e r n m e n t a l a d v i s o r y c o m m i t-

t ee s; an d ( c) th e p e r cen t ag e o f f acu l t y h o l d i n g s o c i e t y o r ed i t o r o f f i ce s .

Th r o u g h mu l t i p l e l i n ea r r eg r e s s i o n , H ag s t r o m f o u n d t h a t t h e b e s t co r r e -

l a t es o f qua l i t y w ere var i ab l es r e l a t ed to r esearch p ro du ct io n ( r 2= .542),

e s p ec i a ll y t w o v a r i ab l e s, t h e av e r ag e n u m b er o f r e s ea r ch a r t ic l e s an d t h e

av e r age n u m b er o f c i t a ti o n s t o p u b l i s h ed r e s ea r ch . T h e n ex t b e s t co r r e la t e s

o f q u a l i t y b e l o n g e d t o t h e s t u d e n t c h a r a c t e r i s t i c s b l o c k o f v a r i a b l e s

( r2= .542) , fo l lowed by the aw ards an d o f f i ces b lock ( r2= .436), t he r esearch

oppor tun i t i es b lock ( r2= .356) , and f ina l ly depar tmen t s i ze ( r2= .323) . Even

t h o u g h d ep a r t m en t s ize w as la s t , i t ex p l a i n ed a l mo s t a t h i r d o f t h e v a r ian ce

i n d ep a r t men t a l p r e s t i g e . Th i s co n ce r n ed H ag s t r o m. H e co n t r o l l ed f o r d e -p a r t m en t s ize i n a la t e r an a l y s is an d f o u n d t h a t i t d i d n o t a f f ec t t h e o v e ra l l



256 TAN

e f f ec t s o f t h e o t h e r v a r i ab le s . H en ce , H ag s t r o m s u g g es ted t h a t " s ize is an

a l mo s t n eces s a r y co n d i t i o n f o r ex ce l l en ce i n mo d e r n s c i en t i f i c e s t ab l i s h -

men t s . La r g e s ize p e r mi ts b r ead t h t o b e c o m b i n ed w i t h t h a t s p ec ia l iz a t io ne s s en ti a l fo r w o r k o n r ap i d l y d ev e l o p in g r e sea r ch f r o n t s " ( H ag s t r o m , 1 97 1,

p . 382) . Hags t rom a l so con t ro l l ed fo r o ther var i ab l es , and h i s r esu l t s i nd i -

ca t ed t h a t a l l v a r i ab l e s , w i t h t h e ex cep t i o n o f t h e r e s ea r ch o p p o r t u n i t i e s

b lock (espec i a l ly one var i ab l e ) , were s t i l l h igh ly co r re l a t ed wi th depar t -

men ta l p res t i ge . The one var i ab l e which had a nega t ive par t i a l co r re l a t i on

w i t h p r e s t i g e w as t h e mean t i me s p en t o n r e s ea r ch . H ag s t r o m s p ecu l a t ed

t h a t a p o s s i b l e r ea s o n f o r t h is o ccu r r en ce m i g h t b e t h a t f acu l t y me m b er s i n

l e s s p r e s t i g i o u s d ep a r t men t s h av e n o t f aced a g r ea t e r d eman d f o r ad mi n i s -

t r at iv e w o r k t h an f acu l t y i n p re s t ig i o u s d ep a r t m en t s .

A n o t h e r s t u d y t h a t i d en t if i ed q u an t i ta t i v e co r r e la t e s o f q u a l i t y w as b y t h e

Nat iona l Sc i ence Board (1969) . The Board iden t i f i ed the fo l lowing var i ab l es

as quan t i t a t i ve co r re l a t es o f t he Car t t e r r a t i ngs :

1 . M ag n i t u d e o f t h e d o c t o r a l p r o g r am ( th e n u mb er o f d eg r ees aw ar d ed ) ;

2 . A m o u n t o f f ed e r a l f u n d i n g f o r acad em i c r e s ea rch an d d ev e l o p men t ;

3 . N o n f ed e r a l cu r r en t f u n d i n co me f o r ed u ca t i o n a l an d g en e r al p u r p o s es ;

4 . Bac ca l au re a t e o r ig ins o f g rad uate f e l l owsh ip r ec ip i en ts (NS F fe llow-

ships) ;

5 . B acc a l au r ea t e o r ig i n o f d o c t o r a t e s ;6 . F r e s h m an ad m i s s io n s s e lec t iv i ty ;

7 . Se l ec t ion o f i n s t it u t i ons b y rec ip i en ts o f g rad uate f e l lowsh ips (NS F fe l-

l owsh ips ) ;

8 . Po s tdo c to ra l s tude n t s in sc i ence and eng ineer ing ;

9 . D o c t o r a t e s aw ar d ed p e r facu l t y me m b er ;

1 0. D o c t o r a t e s aw ar d ed p e r g r ad u a t e s t u d en t ;

1 1. R a t i o o f d o c t o r a t e s t o b acca l au r ea t e d eg r ees ;

1 2. C o m p en s a t i o n o f f u ll p r o f e s s o r s ;

1 3. Th e p r o p o r t i o n o f f u ll p r o f e s s o r s o n t h e f acu l t y ;

1 4. H i g h e r g r ad u a t e s t u d en t / f ac u l t y r a ti o s ;1 5. D e p a r t m en t a l s iz e o f s ev en o r mo r e f acu l t y m em b er s ( N a t i o n a l S c i en ce

Bo ard , as c i t ed by Blac kbu rn an d Lingenfe l t e r , 1973, p . 11).

S i g n if ican tl y , w h e n t h e N a t i o n a l S c i ence B o a r d (1 96 9) ex ami n ed t h e n u m b er

o f P h . D . s a w a r d e d p e r f a c u lt y m e m b e r a n d p e r s t u d e n t e n ro l lm e n t , i t f o u n d

t h a t h i g h l y r ep u t ed i n s t i t u t i o n s h ad p r o d u ced t h e l a r g es t n u mb er o f P h .D . s

p e r f a c u l ty m e m b e r a n d p e r s t u d e nt .

A n o t h e r s t u d y t h a t i d en t i fi ed t h e co r re l a te s o f t h e C a r t te r r a ti n g s w as a

repo r t by M org an , Kearney , and Re gens (1976). Th i s s tud y inves t iga t ed g rad -

ua t e dep ar tm~ nt s i n t he hum ani t i es , soc i a l sc i ences, na tu ra l sc i ences , and




eng ineer ing . The independen t var i ab l es used in t h i s s t udy were (1 ) r evenue

per s tu den t du r ing 1966-67 ; (2 ) f acu l ty sa l a ry ; (3) s t ude n t - f ac u l ty r a t i o

(1969-70) ; an d (4) the t o t a l l i b ra ry vo lum es (1969). Us ing mu l t i p le r eg ress ionan a l y s is , t h e r e s ea r ch e r s f o u n d t h e f o u r v a r i ab l e s to ex p l ai n o v e r 8 0 % o f t h e

v a r i an ce i n t h e q u a l i t y o f t h e g r ad u a t e f acu lt y . Th e t o t a l n u m b er o f l ib r a r y

v o l u mes w as t h e s t r o n g es t p r ed i c t o r o f t h e f o u r , f o l l o w ed b y r ev en u e p e r

s tuden t , s t u den t - f acu l ty r a t io , and f acu l ty sa l a ry .

Other quan t i t a t i ve co r re l a t e r esearcher s i nc lude t he fo l l owing :

J an es (1 96 9) an d Lav en d e r, M a t h e r s , an d P eas e ( 1 9 7 1) , w h o f o u n d s t u -

den t - f acu l ty r a t i o t o be a co r re l a t e o f Car t t e r ' s r e pu ta t i ona l r a t ings ;

C r an e (1 97 0) an d S h i ch o r ( 1 97 0) , w h o f o u n d p a t t e r n s o f f acu l t y m o b i l i t y

an d emp l o y men t t o b e co r r e l a t e s ;Knudsen and Vaughan (1969) , Drew (1975) , and Guba and C lark (1978) ,

w h o f o u n d r e s ea r ch p r o d u c t i v i t y t o b e a co r re l a te ;

D r ew an d K a r p f (1 98 1), w h o f o u n d d e p a r t m en t a l r a te o f p u b l i ca t i o n t o b e

a co r re l a t e ;

So lm on and As t in (1981) , w ho fou nd s ize , p res ti ge , se l ec t iv i ty , per s t u den t

f i nanc i a l expe nd i tu res , and ins t i t u t i ona l cu r r i cu l a r co nce n t ra t i on t o be t he

co r r e la t e s o f th e q u a l i t y o f u n d e r g r ad u a t e p r o g r am s ;

Perk ins and Snel l (1962) , Jo rda n (1963) , Car t t e r (1966) , who f oun d l i b rary

s t r eng th ( as meas ured b y l i b rary expend i tu res and l i b rar ians ' sa l a r i es ) t o be

an i m p o r t an t co r r e l a te o f q u a l i ty ;

Gregg and S ims (1972) , w ho fo un d s tud en t c harac t e r i s t ics , espec i a l l y t he

q u a l i ty o f s t u d en t s an d g r ad u a t e s , t o b e th e m a j o r co r r e la t e s o f t h e q u a l i ty

o f p r o f e s s i o n a l p r o g r ams i n ed u ca t i o n a l ad m i n i s tr a t io n .

Th e mo s t r ecen t q u an t i t a t i v e co r r e l a t e s t u d y w as a s t u d y b y C o n r ad an d

B l ack b u r n ( 1 9 8 6 ) w h i ch ex ami n ed co r r e l a t e s o f d ep a r t men t q u a l i t y a t t h e

m as t er ' s and doc to ra l l eve l i n reg iona l co l l eges and un iver s it ies . A to t a l o f

fo r ty - f i ve depar tm en t s i n fou r t een p ub l i c un iver s i ti es i n f ive d isc ip li nes

( ma t h em a t i c s , b i o lo g y , ed u c a t i o n , h i st o ry , an d ch emi s t r y ) w e re an a ly zed .

Th e d ep en d en t v a r i ab l e i n t h i s s t u d y w as n o t b a s ed o n r ep u t a t i o n a l r a t i n g s

b u t o n p r o g r am r evi ew s g en e r a ted b y s ev er a l t e ams o f ex t e r n a l r ev iew er s.

B as ed o n t h e p r o g r am r ev iew s , d ep a r t me n t s w e r e r a t ed o n a f iv e - p o i n t s ca le

b y th r ee o t h e r ev a l u a t i o n ex p e r t s. Th i s r a t in g s e r v ed as t h e d ep e n d en t v a r i-

ab l e i n t he s tudy . The seven ty - th ree i ndependen t var i ab l es used in t h i s s t udy

were g roupe d u nde r f i ve a reas : (1 ) t he f ac u l ty ; (2) t he s tud en t ; (3) t he p ro -

g ram; (4 ) t he f ac i l i t y ; and (5 ) t he suppor t . Overa l l f i nd ings i nd i ca t ed t ha t

i n d i v i d u a l an d co mb i n ed meas u r e s o f t h e f acu l t y s ch o l a r l y p r o d u c t i v i t y ,

g ran t sm ansh ip , age and t enu re s ta tus , geograp h ica l o r ig in o f h ighes t degree ,

an d t each i n g w o r k l o ad ) , t h e s t u d en t ( t h e n u mb er an d ab f l i t y ) , t h e p r o g r am( p r o p o r t i o n o f d eg r ee p r o g r am s a t t h e ad v an c ed g r ad u a t e leve ! an d cu r r icu -



258 TAN

lar concentration), and the facility (library size) were the best correlates of

departmental quality. Some of these variables had not been identified as

correlates in previous studies. Conrad and Blackburn therefore argued thatit was premature for some researchers (such as Hagstrom, 1971) to assume

that all correlates of quality have been adequately identified or examined,

especially when less visible institutions have not been investigated. FinaUy,

the researchers concluded that "factors associated with departmental quality

are more multidimensional in regional colleges and universities than in de-

partments at leading research universities" (1986).

Crit ique of Qu antitative Co rrelate Studies

Quantitative correlate studies have been successful at identifying major

correlates of reputation, particularly of graduate programs in highly visible

institutions. Such success, according to Conrad and Blackburn (1985a),

does not necessarily mean that future research intended for the purpose of

identifying correlates of quality at the graduate level would not add new

knowledge. On the contrary, if the many methodological limitations associ-

ated with quantitative correlate studies are addressed, new knowledge can be

found.

The first limitation of most quantitative correlate studies is that they are

too dependent on the ACE's reputational ratings. As a result, many quanti-tative correlate studies are subjected to all the limitations associated with the

ACE ratings (Conrad and Blackburn, 1985a). The si tuation can be ad-

dressed by using more recent and improved reputational ratings, such as

those developed by the Conference Board (1982), Conrad and Blackburn

(1986), or recent ratings derived by objective variables.

The second limitation is that most quantitative correlate researchers have

used an atheoretical approach in identifying the correlates of quality. Re-

searchers "seem to have 'rummaged' through their data in search of any

factors that might conceivably be linked empiricaUy to quality instead of

identifying potential correlates on the basis of a theory of quality" (Conradand Blackburn, 1985a, p. 39). Therefore, it is suggested that a theory of

quality be developed in order to provide a framework for interpreting corre-

lates of quality in many of these studies.

Another limitation of quantita tive correlate studies is their exclusive focus

on programs at the graduate level. Since little or no information about the

correlates of quality of programs at the professional and undergraduate level

is available, future studies should be focused at these levels (Conrad and

Blackburn, 1985a).

These criticisms notwithstanding, quant itative correlate studies have con-

tributed to our understanding of program quality. As Conrad and Black-




burn (1985a) pointed out, eren though researchers have been criticized for

not linking their findings to a theoretical scaffolding, the fact remains that

certain correlates have been consistently identified as strong factors relatedto quality and that these factors have important implications for institu-

tional planning. Furthermore, if most methodological limitations associated

with quantitative correlate studies are addressed, this type of study can

generate more knowledge.

CONCLUDING REMARKS

The three types of quality assessment studies have clearly contributed a

great deal toward our understanding of quality, but at least four critical

issues remain yet to be resolved.

First, no one is certain what reputational studies are measuring-reputä-

tion or quality? Some researchers of reputational studies have claimed that

even though their studies were meant to measure reputation of programs,

quality was also measured because the two were closely related. Critics have

argued that reputational ratings were nothing more than a popularity con-

test based on the raters' perception of departmental renown and their famil-

iarity with the programs they were asked to evaluate. Reputational studies,

many have argued, do not assess quality. Furthermore, reputational studies

are said to suffer from various methodological flaws, such as rater andalumni bias. If reputation and not quality is being assessed in reputational

studies, we should seek an alternative approach, such as the use of objective

variables.

The use of objective variables in the assessment of quality shows great

promise, but unfortunate ly it has met with limited success. The main prob-

lem has been the assumption by many researchers that faculty research

product ivity is the major indicator of quality. Since little empirical evidence

exists supporting the direct linkage between faculty research productivity

and quality, some academics have argued that faculty research productivity

should not be the major indicator of departmental qualityo Other equally asimportant variables, such as student ability, financial resources, and curric-

ular concentration, should be used as criteria. Assuming faculty research

productivity is the major criterion of quality, does that necessarily mean

that institutions which do not emphasize research over instruction (such as

liberal arts and community colleges) are not quality institutions? This is

evidently not the case. Clearly, the best way to assess quality is by the use of

multiple variables. Yet little success has been gained. Part of the problem lies

in the fact that there is little theory to guide researchers in their selection of

the "right" combination of variables to measure quality. This will continue

to be a problem as long as we do not have an adequate theory of quality.



260 TAN

T h i s c h i c k e n - e g g s i t u a t i o n , h o w e v e r , s h o u l d n o t d e t e r u s f r o m o u r a t t e m p t s

t o d e v e l o p a t h e o r y o f q u a l it y .

T h i r d , n o o n e i s c e r t a i n w h a t t h e d e f i n i t i o n o f q u a l i t y s h o u l d b e . S h o u l dq u a l i t y b e d e f i n e d b y c h a r a c t e r is t i c s o f q u a l i t y p r e v a l e n t a t e l i te i n s t it u t i o n s

o r a t n o n e l i t e ( " r e p r e s e n t a t iv e " ) i n s t i tu t i o n s ? G r a n t e d t h a t c h a r a c t e ri s ti c s o f

q u a l i t y a t e l i t e i n s t i t u t i o n s m a y b e d i f f e r e n t f r o m t h o s e a t " r e p r e s e n t a t i v e "

i n s t i tu t i o n s a n d v i c e v e r s a , w h i c h s h o u l d w e c h o o s e a s m o d e l s f o r o u r

e f f o r t s i n e n h a n c i n g q u a l i t y ? P e r h a p s w e n e e d t o e s t a b l i s h a f e w s e t s o f

d e t e r m i n a n t s o f q u a l i t y , e a c h a p p l i c a b l e t o a p a r t i c u l a r t y p e o f i n s t i t u t i o n .

F i n a ll y , e v e n t h o u g h m o s t s t u d ie s h a v e b e e n s u c c e s sf u l at i d e n t i f y i n g a

v a r i e t y o f c o r r e l a t e s o f q u a l i ty , n o n e h a s a d e q u a t e l y e x a m i n e d t h e r e l a ti o n -

s h ip b e t w e e n q u a l i t y a n d t h e e d u c a t i o n a l d e v e l o p m e n t o f s tu d e n t s . A s t in

a n d S o l m o n ( 19 8 1 , p . 1 9) p u t i t c l e a r l y : " D o s t u d e n t s i n h i g h l y r a t e d . . .

i n s t i tu t i o n s d e v e l o p d i f f e r e n t l y f r o m s t u d e n t s i n i n s ti t u t io n s w i t h m e d i o c r e

o r l o w r a t i n g s ? " T h i s " v a l u e - a d d e d " q u e s t i o n , u s i n g A s t i n a n d S o l m o n ' s

t e r m i n o l o g y , c o u l d b e a n s w e r e d b y c a r r y i n g o u t l o n g i t u d i n a l s t u d ie s s p e c i fi -

c a ll y t a r g e te d a t e x a m i n i n g t h e v a r y i n g d e g r e e o f s t u d e n t d e v e l o p m e n t i n

d i f f e r e n t t y p e s o f i n s t i t u t io n s . T h e h y p o t h e s i s is t h a t e x c e ll e n t i n s t i tu t i o n s

w o u l d b e t h o s e w h i c h c a n a f f e c t s ig n i f i c a n t ly th e i n t e l le c t u a l a n d s c h o l a r l y

d e v e l o p m e n t o f s t u d e n t s ( a n d f a c u l t y a s w e il ). T h i s " v a l u e - a d d e d " o r n o w

c a l l e d « t a l e n t d e v e l o p m e n t " v i e w o f q u a l i t y h a s b e e n a d v o c a t e d b y A s t i n

( 1 9 8 5 ) i n h i s n e w b o o k , Achieving Educational Excellence.F i n a ll y , i f a l l f o u r i ss u e s a n d m o s t m e t h o d o l o g i c a l f l a w s o f q u a l i t y a ss e ss -

m e n t s t u d i e s c a n b e r e s o l v e d , t h e r e m a y b e a c h a n c e w e c a n a r r i v e a t a

w o r k a b l e d e f i n i t i o n o f q u a l i t y , a d e f i n i t i o n w h i c h c a n g u i d e t h o s e o f u s

c o m m i t t e d t o e n h a n c i n g t h e q u a l i t y o f o u r i n s ti tu t io n s .

R E F E R E N C E S

Abbott , Walter F. (1972) . Universi ty and departmental determinants of the prest igeo f soc i o logy depa r t m en t s . American Sociologist 7 (Novem ber): 14-15.

Ab bot t , Wal te r F ., and He nry M. Barlow (1972). S t ra t i f ica t ion theo ry and organiza-t iona l tank: resources, func t ions and universi ty prest ige in the United States .Pacific Sociological Review 15 (October): 401-424.

Ast in , Alexander W . (1962). Prod uct iv i ty of und ergraduate ins t itu t ions . Science 136:129-135.

- - (1985). Achieving Educational Excellence. San Francisco: Jossey-Bass.

Ast in, Alexander W., and James W. Henson (1977). New measures of col lege selec-tivity. Research in Higher Education 6: 1-8 .

Ast in , A lexander W. , and Lewis C. Solm on (1979) . Measur ing academ ic qua l ity : anin te r im re por t . Change 11: 48-51 .

- - (1 98 1). A r e r e p u ta t io n a l ra tin gs n ee d e d t o m e a su r e q u a li ty ?Change

13 (Oc-tober) : 14-19.



ASSESSMENT OF QU AL ITY 261

Axelson, Leland J. (1960). Graduate schools and the productivity of their graduates.Am er i can Journa l o f Soc io logy, 66 (September): 171-175.

Blackburn, Robert T., and Paul E. Lingenfelter (1973). Assess ing Qua l i ty in Doc toralProg ram s: Criteria an d C orrelates o f Excellence. Ann Arbor: University of Michi-gan, 1973.

Blau, Peter M., and Rebecca Z. Margulies (1974). The reputations of Americanprofessional schools. Change ó (Winter): 42-47.

Bowker, Albert H. (1965). Quality and quantity in higher education. Journa l o f t heArner ican S ta t is t ica l Assoc ia t ion 60 (March): 1-15.

Brown, David G. (t967). The M ob i l e Pro f es sors . Washington, D.C.: The AmericanCouncil on Education.

Carnegie Council on Policy Studies in Higher Education (1976). A Classification ofIn s t i tu t i ons o f H igher Educa t ion . Berkeley, Calif.: The Carnegie Foundation for

the Advancement of Teaching.Carpenter, Ray L., and Patricia A. Carpenter (1970). The doctorate in librarianship

and an assessment of graduate library education. Journa l o f Educa t ion f o r L ib rar-ianship 11: 3-45.

Cartter, Allan M. (1966). A n Asse s s rnen t o f Qua l i t y in G radua te Educa t ion . Wash-ington, D.C.: The American Council on Education.

Cartter, Allan M., and Lewis C. Solmon (1977). The Cartter report on the leadingschools of education, law, and business. Change 9: 44-48.

Clark, Mary J., Rodney T. Harnett, and Leonard L. Baird (1976). A s s e s si n g D i m e n -s ions o f Q ua l i ty i n Doc tora l Educa t ion : A Teehn ical Repor t o f a Na t iona l S tu dy i nThree Fields. Princeton: The Educafional Testing Service, 1970.

Clemente, Frank, and Richard Sturgis (1974). Quality of department of doctoraltraining on research productivity. Soc io logy o f Eduea t ion 47: 287-299,

Cole, Jonathan R. and James A. Lipton (1977). The reputation of American medicalschools. Soc ia lForces 55: 662-684.

Conrad, Clifton E and Robert T. Blackburn (1985a). Program quality in highereducation: a review and critique o f literature and research. In Higher Edu ca t ion :Handbook o f Theory and Research , Vol. I, pp. 283-308. New York: AgathonPress.

(1985b). Correlates of departmental quality in regional colleges and univer-sities. Am er i can E duea t iona l Research Journa122 (Summer): 279-295.

- -

(1986). Current views of departmental quality: an empiricat examinafion.R e v i e w o f H i g h e r E d u c a ti o n , 9 (Spring), forthcoming.

Cox, W. Miles, and Viola Catt (1977). Productivity ratings of graduate programs inpsychology based on publication in the journals of the American PsychologicalAssociation. A m e r i c a n P s y c h o lo g i st 32: 793-813.

Crane, Diana (1970). Scientists at major and minor universities: a study of productiv-ity and recognition. Arner ican Soc io logical Rev iew 30 (October): 699-714.

Dolan, W. Patrick (1976). The Rank ing Garne: The Power o f t he Aca dem ic E lit e.Lincoln, Neb.: The University of Nebraska Printing and Duplicating Service.

Drew, David E. (1975). Se i ence Deve lop men t : A n Eva lua tion S tudy . Washington,D.C.: The National Academy of Sciences.



262 TAN

Dr ew, Da v id E . , a n d R o n a ld S . Ka r p f (1 98 1) . Ra n k in g a c a d e m ic d e p a r tm e n t s : e m p i r -ica l f ind ings and a theore t ica l pe rspec t ive . Research in Higher Educat ion 14:

3 0 5 - 3 2 0 .Du b é , W . F . ( 19 74 ). Un d e r g r a d u a t e o r i g in s o f U .S . m e d ic a l s t u d e n t s. Journa l o f

Med ica l Educa t ion 49: 1005-1010.

E l to n , Ch a r l e s E , a n d S a m u e l A . Ro d g e r s ( 1 9 7 1 ) . P h y s i c s d e p a r tm e n t r a t i n g s : a n -o th e r e v a lu a t i o n . Science 1 74 ( No v e m b e r ) : 5 6 5 - 5 6 8 .

E l to n , Ch a r l e s F . , a n d Ha r r i e t t A . Ro s e ( 1 9 7 2 ) . W h a t a r e t h e r a t i n g s r a t i n g ? A m e r i -can Psychologis t 27 : 197-201 .

Ga l l a n t , J o n a th a n , a n d J o h n W . P r o th e r o ( 1 97 2) . W e ig h t - wa tc h in g a t th e u n iv e r s it y :t h e c o n s e q u e n c e s . Science 175 (no. 381): 385-386.

Gil l , W. N. (1975). Rank ings o f Gradua te Eng ineering Depar tmen t s . Bu f f a lo : S t a t eUn iv e r s i t y o f Ne w Yo r k a t Bu f f a lo .

G le n n , No v a l e s D . , a n d W a y n e V i l l im e z ( 1 97 0) . T h e p r o d u c t i v i t y o f s o c io lo g i s t s a t 4 5A m e r i c a n u n iv e rs i ti e s. Am er i can Soc iolog is t 5 ( Au g u s t ) : 2 4 4 - 2 5 2 .

G lo we r , Do n a ld D . ( 1 9 8 0 ) . A r a t i o n a l m e th o d f o r r a n k in g e n g in e e r in g p r o g r a m s .

Engineer ing Educat ion 7 0 : 7 8 8 - 7 9 4 ; 8 4 2 .

Go u r m a n , J a c k ( 1 9 6 7 ) . The G ourma n R epor t : Ra t ings o f Am er i can Co ll eges . P h o e -

n ix , Ar i z o n a " T h e Co n t in u in g Re s e a r c h I n s t i t u t e .

- - (1977a). The Gourman Repor t : A Rat ing o f Undergraduate Programs inAmerican and In ternat ional Univers i t ies . L o s A n g e l e s : T h e N a t i o n a l E d u c a t i o nS ta n d a r d s .

- - (1977b). The Gourman Repor t : A Ra t ing o f Gradua te and Pro f e s s iona lProgra ms in Am erica n a nd In ternat ional Univers it ies . L o s A n g e l e s: T h e N a t i o n a lE d u c a t i o n S t a n d a r d s .

- - ( 1 9 8 2 ) . The Gourman Repor t : A Rat ing o f Undergraduate Programs inAmerican and In ternat ional Univers i t ies . L o s A n g e l e s : T h e N a t i o n a l E d u c a t i o n

S ta n d a r d s .

- - (1983). The Go urma n Repo r t : A Ra t ing o f Gradua te and Pro f e s siona l Pro -grams in Am erica n and In terna t ional Universi ties , rev . 2nd ed . Los Ange les : TheN a t i o n a l E d u c a t i o n S t a n d a r d s .

Gr e g g , Ru s s e l l T . , a n d P a u l D . S im s ( 1 9 7 2 ) . Qu a l i t y o f f a c u l t i e s a n d p r o g r a m s o f

g r a d u a t e d e p a r t m e n t s o f e d u c a t i o n a l a d m i n i s t r a ti o n . Educa t iona l Ad min i s t ra t ionQuarter ly 8 : 6 7 - 9 2 .

Gu b a , E g o n G . , a n d Da v id L . C l a r k ( 19 78 ). L ev els o f R & D p r o d u c t i v i t y i n s c h o o l so f e d u c a t i o n . Educat ional Researchers 7" 3 -9 .

Ha g s t r o m , W a r r e n O . (1 97 1) . I n p u t s , o u tp u t s , a n d t h e p r e s t i ge o f u n iv e r s i ty s c ie n c e

d e p a r t m e n t s . Soc io logy o f Educa t ion , 44 (Fall) : 3 75-3 97.

Ho u s e , D o n a ld R . , a n d J a m e s H . Ye a ge r, Jr . (1 97 8) . T h e d i s t r i b u t i o n o f p u b l i c a t i o ns u c c e s s w i th in a n d a m o n g to p e c o n o m ic s d e p a r tm e n t s : a d i s a g g r e g a t e v i e w o frecen t ev idence . Econ om ic Inqu iry , 1 6 ( Oc to b e r ) : 5 9 3 - 5 9 8 .

H u g h e s , R a y m o n d M . (1 92 5) . A S tud y o f Graduate Schoo l s o f Amer i ca . O x f o r d ,

Oh io : T h e M ia m i Un iv e r s i t y P r e s s , 1 9 25 .

- - ( 19 34 ). Re p o r t o f th e Co m m i t t e e o n G r a d u a t e I n s t r u c t i o n . Educa t iona l Rec -ord, 15 (1934): 192 -23 4.




Jones , R. W. (1969) . The s tudent facul ty ra t io in graduate programs of se lec teddepa r tm en t s o f socio logy . American Sociologist , 4: 123-127.

Johns on , R icha rd R . Leade r sh ip amo ng Am er i can col leges. Change 10 (Nove mbe r) :50-51 .

Jones , Lyle V. , Ga rdne r Lindzey, a nd Por te r E . C oggesha l l , eds. (1982a) . A n A s s e s s -m en t o f Research-Doctorate Programs in the Un i ted S ta tes : Hum ani t ies . Washing-ton , D .C . : Th e Na t iona l Academ y P re ss .

- - ( 1 9 8 2 b ) . A n A sse ssm en t o f Re~varch-Doetorate Progra ms in the Uni tedStates: Social an d Beha vioral Sciences. W ash ing ton , D .C . : The Na t iona l Academ y

Press .

- - ( 1 9 8 2 c ) . A n Asse s sm en t o f Research -Doc tora t e Programs in t he U n i t edStates: B iological Sciences. W ash ing ton , D .C . : The Na t iona l Academ y P re s s.

- - ( 1 9 8 2 d ) . A n Asse s sm en t o f Researeh -Doc tora t e Programs in the Un i t ed

Sta tes: Mathemat ica l and Phys ica l Sc iences . Wash ing ton , D .C . : The Na t iona lAcad emy P re s s.

- - ( 1 9 8 2 c ) . A n Asse s sm en t o f Research -Doc tora t e Programs in the U n i tedStates: Engineering. Wash ing ton , D .C . : Th e Na t iona l Academ y P re s s.

Jordan , Rober t T . (1963) . Library charac te r i s t ics of co l leges ranking h igh in aca-dem ic excellence. College an d Resea rch Libraries, 24: 369-376.

J u r is D o c t o r M a g a z i n e (1976) . The popular vote : rankings of top schools . Vol . 6 :17-18, 21.

Kelso , Char les D. (1975). Ho w does yo ur law scho ol measu re up? S t u d e n t L a w y e r 4:

20 -24 .

Kenis ton , Hayward (1959) . Graduate S tu dy in Research in the A r ts and Sc iences a tthe Univers i ty o f Pennsy lvania . Ph i l ade lph i a : The Un ive r s i t y o f Pennsy lvan i a

Press.

Knapp , R . H . , and J . J . Greenbaum (1953) . The Younger Am erica n Scholar: H isCollegiate Origins. Chicago : T he U n ive r si t y o f Ch icago P re s s .

Knapp , R . H . and H . B . Goodr i ch (1952) . Orig ins o f Amer i can Sc i en t i s t s . N ew

York: Russell and Russell.

Knudsen , Dean D. , and Ted R. Vaughan (1969) . Qual i ty in graduate educa t ion: a re -

eva lua t ion o f t he r ank ings o f soc io logy depa r tmen t s i n t he C a r t t e r r epo r t . A m e r i -can Sociologist 4 (February)" 12-19.

Krause , Erv in D. and Lore t ta Krause (1970) . The co l leges tha t produce our bes t

scien ti st s: a s tudy o f the academ ic t ra in ing ground s o f a la rge grou p o f d is tin-guished American scientis ts . Science Educat ion 54: 133-140.

Ku h, G eorge D. (1981). Indices o f Qual i ty in the Undergraduate Exper ience . A A H E -ERIC/Highe r Educa t i on Resea rch Repor t No . 4 . Wash ing ton , D .C . : Amer i canAssoc i a t i on fo r H ighe r Educa t i on .

Lavender , A. D. , R. A . M athers , a nd J . Pease (1971). Th e s tudent - facul ty ra t io ing radua t e p rog rams o f s e l ec t ed depa r tmen t s o f soc io logy : a supp l emen t t o t heJanes r epo r t . Amer i can Soc io log i s t 6: 29-30 .

Lawrence , Judi th K. , and Kenneth C. Green (1980) . A Ques t ion o f Qua l it y : TheHighe r Educa t ion R a t ings Game . A A H E - E R I C / H i g h e r E d u c a t io n R e se ar ch R e-po r t No . 5 . Wash ing ton , D .C . : Am er i can Assoc i a t i on fo r H ighe r Educa t i on .



264 TAN

Lewis, Lionel S. (1968). On subjective and objective rankings of sociology depart-ments. Amer i can Soc io log i s t 3 (May): 129-131.

Liu, Helen C. (1978). Faculty citation and quality of graduate engineering depart-ments, Engineer ing Educat ion 68 (April): 739-741.

Magoun, H. W. (1966). The Cartter report on quality in graduate education. Journa l

o f H i g h e r E d u c a t i o n , 37 (December): 481-492.

Margulies, Rebecca Z., and Peter M. Blau (1973). America's leading professionalschools. Change, (November): 21-27.

M B A M a g a z i n e (1974). The 15 top-ranked business schools in the United States.December, 21-25.

- - (1975). The top 15: The MBA survey of graduate business schools. Decem-ber, 33-35.

Morgan, David L., Richard C. Kearney, and James L. Regens (1976). Assessingquality among graduate institutions of higher education in the United States.Socia l Sc ience Quarter ly 57 (December): 670-679.

Munson, C. Eric, and Pat Nelson (1977). Measuring the quality of professionalschools. U C L A E d u c a to r 19: 41-52.

National Science Board (1969). Graduate Educa t ion: Param eters fo r Pub l ic Pol icy .Washington, D.C.: The National Science Foundation.

Oromaner, Mark J. (1970). A note on analytical properties and prestige of sociologydepartments. American Sociologist , 5 (August): 240-244.

Perkins, Dexter, and John L. Snell (1962). The Educat ion o f His tor ians in the Uni tedStares. New York: McGraw-Hill.

Petrowski, William R., Evan L. Brown, and John A. Duffy (1973). National univer-sities and the ACE ratings. J o u r n al o f H i g h e r E d u c a ti o n, 44 (October): 495-513.

Pirsig, Robert M. (1974). Zen and t he Ar t o f Mo torcyc l e Ma in t enance . New York:Morrow.

Roose, Kenneth D., and Charles J. Andersen (1970). A Ra t ing o f Uradua te Pro -grams. Washington, D.C.: The American Council on Education.

Scully, Malcolm G. The well-known universities lead in rating of faculties' reputa-tions. Chron ic l e o fH igher Educa t ion , Jan. 15, 1979, pp. 6-7.

Shichor, David. "Prestige of Sociology Departments and the Placing of NewPh.D.'s" American Sociologist , 5 (May 1970): 157-160.

Smith, Richard, and Fred E. Fiedler (1971). The measurement of scholarly work: acritical review of the literature. Educa t iona l Record 52 (Summer): 225-232.

Solmon, Lewis, and Alexander W. Astin (1981). Departments without distinguishedgraduate programs. Change 13 (October): 23-28.

Somit, Alfred, and Joseph Tanenhaus (1964). Am erica n Po l i tica l Sc ience: A Prof i leofa Discipl ine. New York: Atherton Press.

Tidball, M. Elizabeth, and V. Kristiakowski (1976). Perspective on academic womenand affirmative action. Educa t iona l Record 54: 130-135.

Webster, David S. (1981). Methods of assessing quality. Change 13 (October): 20-24.

- - ( 1 9 8 3 ) . America's highest ranked graduate schools, 1925-1982. Change(May/June): 14-24.




- - ( 19 84 ). W h o i s J a c k G o u r m a n a n d wh y i s h e s a y in g a ll t h e se t h in g s a b o u t m y

co l lege? Change ( No v . /De c . ) : 1 4 - 1 9 .

Wispe , Lauren G. (1969) . The b igger the be t te r : p roduc t iv i ty , s ize , and tu rnover in as a m p l e o f p s y c h o l o g y d e p a r tm e n t s . American Psychologist 2 4 : 6 6 2 - 6 6 8 .

Received February 10, 1986

Documents

Tan 1986 The assessment of quality in higher education- A critical review of the literature and research.pdf