FINAL RIPORT 01 013 CONTRACTXMII 100014-85-K-0113, 31 150-535
ROBT . TSUTAZAVA
, .T,, B., 196, DT IC: -.LECTE
*H DPrepared under contract No. 100014-85-K-0l13, SK 150-535
with the cognitive Science prosramOffice of laval Research
Approved for public release: distribution unlimited.Reproduction Is whole or part in pernitted forany purpose of the United States Government.
88 9 26 15
w - --
SECURITY CLASSIFICATION OF THIS PAGE
RNForm ApprovedREPORT DOCUMENTATION PAGE OMB No. 0704-01"
la REPORT SECURITY CLASSIFICATION lb RESTRICTIVE MARKINGS
Unclassified2a SECURITY CLASSIFIC;TIOni AUTHORITY 3 DISTRIBUTION/AVAILABILITY OF REPORT
Dpproved for public release; distri-2b. DECLASSIFICATION/DOWNGRADING SCHEDULE Ut ion unlimited
4 PERFORMING ORGANIZATION REPORT NUMBER(S) S MONITORING ORGANIZATION REPORT NUMBER(S)
None6a. NAME OF PERFORMING ORGANIZATION 6b OFFICE SYMBOL 7a NAME OF MONITORING ORGANIZATIONDepartment of Statistics (fapplicable) Cognitive Science ProgramUniversity of Missouri Office of Naval Research (Code 1142PT6c. ADDRESS (City, State, and ZIPCodre) 7b ADDRESS (City, State, and ZIP Code)222 Math Sciences 00 North Quincy StreetColumbia, MO 65211 rlington, VA 22217-5000
8a. NAME OF FUNDING/SPONSORING Sb OFFICE SYMBOL 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGANIZATION (If applicable) 100014-85-K-0113
8c. ADDRESS (City, State, and ZIP Code) 10 SOURCE OF FUNDING NUMBERSPROGRAM PROJECT TASK WORK UNITELEMENT NO NO NO JACCESSION NO
61153N RR04204 RR4204-0 4421-53511 TITLE (Include Security Classification)
Final Report on ONR Contract Number N00014-85-K-0113, NR 150-53512 PERSONAL AUTHOR(S)
Tsutakawa, Robert K13a. TYPE OF REPORT 13b TIME COVERED 14 DATE OF REPORT (Year, Month, Day) 15 PAGE COUNT
Fin FROM R n TO 30 88 Sep 30 1516 SUPPLEMENTARY NOTATION
17 COSATI CODES 18. SUBJECT TERMS (Continue on reverse if necessary and identify by block number)FIELD GROUP SUB-GROUP Bayesian IRT, calibration, ability estimation,
2PL, 3PL.19 ABSTRACT (Continue on reverse if necessary and identify by block number)
This is the final report on ONR Contract number N00014-85-K-0113,NR 150-535, Calibration of Test Items and Measurement of Abilities. Thereport reviews the results obtained during the period January 1985through September 1988. The results include the development of a generalBayesian framework for item response analysis and Bayesian estimation ofitem and ability parameters. A listing of technical reports andpublications resulting from the contract is also presented.
20 DISTRIBUTION/AVAILABILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION- UNCLASSIFIED/UNLIMITED 0 SAME AS RPT 0 DTIC USERS
22a. NAME OF RESPONSIBLE INDIVIDUAL 22b. TELEPHONE (Include Area Code) I22c. OFFICE SYMBOLDr. Charles Davis 202-696-4046 pNR 1142CS
aDD Form 1473, JUN 36 Previous editions are obsolete. SECURITY CLASSIFICATION OF THIS PAGE
FINAL REPORT ON OUR CONTRACTNUMBER N00014-85-K-0113, MR 150-535
Calibration of Test Item and Measurement of Abilities
by
Robert K. TsutakawaPrincipal Investigator
SEPTEMBER 1988
Department of StatisticsUniversity of MissouriColumbia, MO 65211
II
-7i
Introduction
) The overall goal of this project is to develop new Bayesianprocedures for mental testing. A typical test, which is studiedhere, consist of k test items administered to n examinees. Thedata consists of an nxk matrix of binary responses indictingwhich of the k items are scored correctly and which incorrectlyby each of the n examinees.-
The statistical procedures are based on the assumption thatthere is a model which specifies the probability of a correctresponse to each item as a function of an unidimensionalability. Such functions are assumed to belong to certainfamilies such as the two-parameter logistic (2PL) or three-parameter (3PL) curves. These curves are identified byparameters called item parameters.\
When these models are used for testing, a set of items isinitially calibrated using a moderately large value for n (thesample size). The calibration consists of estimating the itemparameters. The calibrat d, curves are then used to ,_soreabilities of new examinees. , .
It is standard practice to ignore the uncertainty in the ritems once the calibration is complete and to estimate abilitiesassuming that the item parameters are known. This practice canlead to serious inferential errors in the measurement ofabilities. In particular, an interval estimate of an ability canbe too narrow giving a false impression of the accuracy of theestimate.
The sequential nature of first calibrating and then scoringmakes the Bayesian approach particularly appropriate. Accordingto this approach, an analysis is made of the uncertainties inthe estimated items at the calibration phase. This uncertaintyis then taken into account when abilities are measured. Theuncertainty in the measured ability is not only due to therandomness of responses from individuals with the same ability,but also due to the uncertainty in the calibrated items.
The Bayesian paradigm can be extended to on-linecalibration, where new items are introduced with items which havealready been calibrated. In this situation the uncertainties ofabilities based on the calibrated items are incorporated into theuncertainties of the new items. Again the typical standardpractice is to ignore the uncertainties in the abilities ofindividuals used for the calibration of the new items.
2
In order to develop this general Bayesian approach to mentaltesting, the research was divided into the following four topicsand the results are outlined below.
I. Development a general Bayesian framework for item responseanalysis.
II. Estimation of item parameters.
III. Estimation of abilities.
IV. On-line calibration.
I. Bayesian Framework
The general framework for Bayesian item response theory hasbeen described in Tsutakawa and Lin (Psychometrika, 1986). Giventhat the item response curves belong to a certain parametricfamily, a prior distribution for the item parameters are assumed.The joint likelihood function of ability and item parameters isbased on the assumption of local independence. The ability areassumed iid N(0,1). The marginal likelihood function is then theaverage of the joint likelihood function weighted by the N(O,1)prior. The marginal likelihood function is then multiplied tothe prior to get the (unnormalized) posterior for the itemparameters. The marginal posterior for the ability parameter canbe similarly expressed but is not easy to work with due to themultiple integrals involved.
II. Estimation of item parameters
The general approach developed for item parameter estimationis to use as point estimate the posterior mode and as measure ofuncertainty the posterior covariance matrix. The use of the EMalgorithm for computing the posterior mode is described inTsutakawa and Lin (1986). A novel feature of this paper is theuse of the ordered bivariate beta to form a prior distributionfor the item parameters in 2PL. This paper proposes the use of9the inverse posterior information matrix to approximate theposterior covariance matrix. It also illustrates the relativecloseness of estimated values in repeated samples, when comparedto standard methods such as LOGIST (Wingersky, Barton & Lord1982). L0
In Rigdon and Tsutakawa (JES 1987) an empirical Bayesprocedure is developed for the case in which both ability anditem parameters are sampled from population distributions withunknown hyperparameters. Here the EM algorithm (Dempster, Laird,& Rubin, JRSS-B 1977) is modified to simultaneously estimate the Code$hyperparameters. Simulations are used to show the robustness of Iorthis approach relative to marginal maximum likelihood in thecase of the Rasch model.
The' use of the Dirichlet distribution to form a priordistribution for item parameters in 3PL is studied in Tsutakawa(TR143, 1988). The conventional Bayesian approach assume priorindependence of parameters within items. This paper suggests asimple device to represent the prior dependence among parameterswithin items. The emphasis here is on looking at curves ratherthan parameters. Bayesian modal estimates are compared withLOGIST (Wingersky, Barton, and Lord, 1982) and marginal maximumlikelihood. The robustness of the Bayesian estimate relative toweights placed on the prior is also illustrated. One notablefeature of the Bayesian method is that there are much feweroutliers with unreasonable values.
III Ability estimation
Bayesian approximations to the posterior mean and varianceof ability are proposed and illustrated for 2PL in Tsutakawa andSoltys (JES, 1988). The standard empirical Bayes approximationsare posterior moments conditional on assuming the unknown itemparameters to equal those estimated at the calibration phase.The new approximation modifies this by adding terms representing,and correcting for, the uncertainties of the calibrated itemparameters. It is a special case of Lindley's (1980)approximation when the 3rd partial derivatives of thelogposterior vanish. The new approximation shows that theempirical Bayes approximation consistently underestimates theposterior variance. Other approximations, including those byLeonard (1982) and Tierney and Kadane (1986), have also beenexamined and found to require an excessive amount of computingand therefore not suitable for routine use in ability estimation.
The Bayesian approximation was then extended to 3PL inTsutakawa and Johnson (TR147, 1988). This paper demonstratesthat maximum likelihood and empirical Bayes, both of whichreplace unknown item parameters by those estimated, grosslyunderestimate the variance of the ability parameters. Thenumerical examples, upon which much of the conclusion is reached,is based on a sample of n-400. Although the discrepanciesbetween the procedures should decrease as n increases, there issome feeling at even at n-1600 the differences might not benegligible.
IV On-line calibration
Work on this topic remains incomplete due to the delayencountered in developing computer programs for Bayesian abilityestimation under 3PL. The delay was due to untimely personnelchanges, which required finding and training a new computerprogrammer each time a person left.
4
Summary
The development of Bayesian item response theory requiresconsiderable amount of computation and new techniques forapproximating posterior distributions. This researchdemonstrates that computational problems (though far from solved)can be dealt with by careful use of asymptotic approximations.It also demonstrates that reasonable prior distributions can beformulated in spite of the complexities of IRT models. But moreimportantly it shows the feasibility of developing acomprehensive and complete theory which can be adapted to largescale testing environments.
References
Dempster, A.P., Laird, N.M. & Rubin, D.B. (1977). Maximumlikelihood from incomplete data via the EM algorithm (withdiscussion). Journal of the Royal Statistical Society,Series B, 39, 1-38,
Leonard, T. (1982). Comment on "A simple predictive densityfunction", by M. Lejeune and G.D. Faulkenberry. Journal ofthe American Statistical Association 77, 657-658.
Lindley, D.V. (1980). Approximate Bayesian methods TrabajosEstadistica 31, 223-237.
Tierney, L. and Kadane, J.B. (1986). Accurate approximations forposterior moments and marginal densities. Journal of theAmerican Statistical Association 81, 82-86.
Wingersky, M. S., Barton, M.A., & Lord, F.M. (1982). LogistUser's Guide, Educational Testing Service, Princeton, NJ.
Technical Reports and Publications Prepared Under Contract
Tsutakawa, R.K. & Lin, H.Y. (1986). Bayesian estimation of itemresponse curves. Psychometrika 51, 251-267.
Rigdon, S.E. & Tsutakawa, R.K. (1987), Estimation for the Raschmodel when both ability and difficulty parameters arerandom. Journal of Educational Statistics, 12, 76-86.(previously distributed under Mathematical SciencesTechnical Report No. 133, Department of Statistics,University of Missouri.)
Tsutakawa, R.K. & Soltys, M.J. (1988). Approximation forBayesian ability estimation. Journal of Educationalstatistics, 13, 117-130. (Previously distributed underMathematical Sciences Technical Report No. 134, Departmentof Statistics, University of Missouri.)
5
Tsutakava. R.K. (1988). Dirichlet prior in Bayesian estimationof item response curves. Mathematical Sciences TechnicalReport No. 143, Department of Statistics, University ofMissouri.
Tsutakava, R.K. & Johnson, J.C. (1988). Bayesian abilityestimation via 3PL with partially known item parameters.Mathematical Sciences Technical Report No. 147, Departmentof Statistics, University of Missouri.
6
1 98/07,/0O6
University of Missouri-Columbia/Tsutakawa
Dr. rerry Ackerman Dr. R. Oarrell BockAmerican College Testing Programs University ot ChicagoP.O. Box 168 NORCIowa City, IA 52243 6030 South Ellis
Chicago, IL 60637Dr. Robert AhlersCode N711 Cdt. Arnold BohrerHuman Factors Laboratory Sectie Psycholoqisch OnderzoekNaval Training Systems Center Rekruterings-En SelectiecentrumOrlando, FL 32813 Kwartier Koninqen Astrid
Bru i jns traatDr. James Algina 1120 Brussels. BELGIUM1403 Norman HallUniversity of Florida Dr. Robert BreauxGainesville, FL 32605 Code 76
Naval Training Systems CenterDr. Erling B. Andersen Orlando. FL 42813-1100Department of StatisticsStudiestraede 6 Dr. Robert Brennan1455 Copenhagen American Collegie VeitinqDENMARK Protirams
P. 0. Box 168Dr. Eva L. Baker Iowa City. IA t;2243UCLA Center for the Study
of Evaluation Dr. James Carlson145 Moore Hall American College lestingUniversity of California ProgramLos Angeles, CA 90024 P.O. Box 168
Iowa City, IA 52243Dr. Isaac BejarMail Stop: 10-R Dr. John 8. CarrollEducational Testing Service 409 Elliott Rd., NorthRosedale Road Chapel Hill, NC 27514Princeton, NJ 08541
Dr. Robert M. CarrollDr. Menucha Birenbaum Chief of Naval OperationsSchool of Education OP-0182Tel Aviv University Washington. DC 20350Ramat Aviv 69978ISRAEL Dr. Raymond E. Christal
UES LAMP Science AdvisorDr. Arthur b. Blaiwes AFHRL/MOELCode N/12 Brooks A'B. fX /8235Naval Training Systems CenterOrlando, FL 32813-7100 Dr. Norman Cliff
Department of PsychologyDr. Bruce Bloxom Univ. of So. CaliforniaDefense Manpower Data Center Los Anoeles. CA 90089-1061550 Camino El Estero,
Suite 200Monterey, CA 93943-3231
7
1968/0Y/06
iJniversitv of Missouri-Columbia/Isutakawa
Director. Or. Fritz OrasQowManpower Support and University of Illinois
Readiness Program Department of Psycholoqv
Center for Naval Analysis 603 E. Daniel St.
2000 North Beauregard Street Champaign. IL 61820
Alexandria, VA 22311
Defense lechnical
Dr. Stanley Collver Information Center
Office of Naval rechnoloqy Cameron Station. Bldq 5
Code 222 Alexandria, VA 22314
800 N. Quincy Street Attn: TC
Arlington, VA 22217-5000 (1Z Copies)
Dr. Hans F. Crombag Dr. Stephen Dunbar
Faculty of Law 2248 Lindquist Center
University of Limburq for Measurement
P.O. Box 616 University of Iowa
Maastricht Iowa City, IA 5I2242
The NETHERLANDS 6200 MD
Dr. James A. EarlesDr. Timothy Davey Air Vorce Human Resources Lab
Educational resting Service Brooks AFB, 1X /8235
Princeton, NJ 08541
Dr. Kent Eaton
Dr. C. M. Dayton Army Nesearch Institute
Department of Measurement 5001 Eisenhower Avenue
Statistics & Evaluation Alexandria, VA 22333College of Education
University of Maryland Dr. John M. Lddins
College Park, MD 20/42 University of Illinois252 Eni]neerii1Q kes.arch
Dr. Ralph J. DeAvala Laboratory
Measurement, Statistics, 103 South Mathews 51.reet
and Evaluation Urbana, IL 61801
Benjamin Bldg., Rm. 4112UJniversity of Maryland Dr. Sus.in Lmbretson
College Park, MD 20/42 University of Kansas
Psycholoqv Departiment
Dr. Dattprasad Divqi 4,b FraserCenter for Naval Analysis Lawrence. KS 66045
4401 Ford Avenue
P.O. Box 16268 Dr. Georqe Englehard, Jr.Alexandria, VA 22302-0268 Division of Educational Studies
Emory University
Dr. Hei-Ki Dong 210 Fishburne Bldg.
Bell Communications Research Atlanta, GA 30322
6 Corporate Place
PYA-IK226 Dr. Benjamin A. FairbankPiscataway, NJ 08854 Performance Metrics, Inc.
5825 CallaghanSuite 225
San Antonio, TX 78228
8
1988/07/06
University of Missouri-Columbia/lsutakawa
Dr. P-A. Federico Or. Robert GlaserCode 51 Learning Research
NPRDC & Development CenterSan Diego, CA 92152-6800 University of Pittsburgh
3939 O'Hara StreetDr. Leonard Feldt Pittsburgh, PA 15260
Lindquist Centerfor Measurement Dr. bert Green
University of Iowa Johns Hopkins University
Iowa City, IA 52242 Department of PsychologyCharles & 34th Street
Dr. Richard L. Ferguson Baltimore, MD 21218American Colleqe TestingP.O. Box 168 DORNIER GMBHIowa City, IA 52243 P.O. Box 1420
D-7990 Friedrichshafen IDr. Gerhard Fischer WESI GERMANYLiebiggasse 5/3A 1010 Vienna Dr. Ronald K. HambletonAUSTRIA Universitv of Massachusetts
Laboratory of PsychometricDr. Myron Fischl and Evaluative ResearchU.S. Army Headquarters Hills South, Room 152DAPE-MRR Amherst, MA 01003The PentagonWashington, DC 20310-0300 Dr. Delwvn Harnisch
Universitv of IllinoisProf. Donald Vitzqerald 51 Gertv DriveUniversity of New England Champaign, IL 61820Department of PsychologyArmidale, New South Wales 2351 Dr. Grant HenningAUSIRALIA Senior Research Scientist
Division of MeasurementMr. Paul Foley Research and ServicesNavy Personnel R&D Center Educational restinq ServiceSan Diego, CA 92152-6800 Princeton, NJ 08541
Dr. Alfred R. Fregly Ms. Rebecca HetterAFOSR/NL, Bldg. 410 Navy Personnel R&D CenterBolling AFS, DC 20332-6448 Code 63
San Diego, CA 92152-6800Dr. Robert D. GibbonsIllinois State Psychiatric Inst. Or. Paul W. HollandRm 529W Educational Testing Service, 21-T1601 W. Taylor Street Rosedale RoadChicago, IL 60612 Prinoeton, NJ 08541
Dr. Janice Gifford Prof. Lutz F. HornkeUniversity of Massachusetts Institut fur PsychologieSchool of Education RW1H AachenAmherst, MA 01003 Jaegerstrasse 17/19
0-5100 AachenWES1 GERMANY
9
1988/0?/06
University of Missouri-Columbia/Tsutakawa
Dr. Paul Horst Or. G. Gage Kingsbury
677 G Street, #184 Portland Public Schools
Chula Vista, CA 92010 Research and Evaluation Department
501 North Dixon Street
Mr. Dick Hoshaw P. 0. Box 3107
OP-]3t Portland, OR 97209--3107
Arlinqton Annex
Room 2834 Dr. William Koch
Washington, DL 20330 i.- t Mea;. 3nl L/aI. Atr.
University of lexis-Au-tin
Dr. Lloyd Humphrevs , i ,i in J.rsi~v of JI in S
- ;u .?it- D.3Sed Lducat ion
1-hamp3iqg, LL bldeJO Resear.h Laboratory
Dr. Steven tiLuf, rthana. IL Li:')1
iUlniversity of Alberta Dr. Leonard Kroeker
Edmonton, Alberta Navy Personnel RW Center
CANADA TFW 2G5 Code 62
San DieQo. CA 92152-6800
Dr. Huynh Huynh
College of Education Dr. Jerry Lehnus
Univ. of South Carolina Defense Manpower Data Center
Columbia, SC 29208 Suite 400
1600 Wilson Blvd
Dr. Robert Jannarone Rosslvn. VA 22209
Elec. and Computer Eng. Dept.
University of South Carolina Dr. rhomas Leonard
Columbia, SC 29208 University of Wisconsin
Department of Statistics
Dr. Douglas H. Jones 1210 West Dayton Street
Thatcher Jones Associates Madison. WI 53705
P.O. Box 664010 Trafalga- Court Dr. Michael Levine
Lawrenceville, NJ 08648 Educational Psychology
210 Education Bldg.
Dr. Milton S. Katz University of illinois
European Science Coordination Champaign, IL 61801
OfficeU.S. Army Research Institute Dr. Charles Lewis
Box 65 Educational lesting Service
FPO New York 09510-1500 Princeton, NJ 08541-0001
Prof. John A. Keats Dr. Robert L. Linn
Department of Psychology Campus Box 249
University of Newcastle University of Colorado
N.S.W. 2308 Boulder, CO 80309-0249AUSIRALIA
10
1988/07/06
University of Missouri-Columbia/Tsutakawa
Dr. Robert Lockman Dr. William Montaque
Center for Naval Analysis NPRDC Code 13
4401 Ford Avenue San Diego, CA 92152-6800
P.O. Box 16268Alexandria, VA 22302-0268 Ms. Kathleen Moreno
Navy Personnel R&D Center
Dr. Frederic M. Lord Code 62
Educational Testing Service San Dieqo. CA 92152-6800Princeton, NJ 08541 Headquarters Marine Corps
Dr. George B. Macready Code MPI-20
Department of Measurement Washington, DC 20380
Statistics & EvaluationCollege of Education Dr. W. Alan Nicewander
University of Maryland University of Oklahoma
College Park, MD 20742 Department of PsychologyNorman, OK 73071
Dr. Gary MarcoStop 31-E Deputy lechnical Director
Educational Testing Service NPRDC Code 01A
Princeton, NJ 08451 San Diego, CA 9-lb?-6800
Dr. James R. McBride Director, Training Laboratory.
The Psychological Corporation NPRDC (Code 05)
1250 Sixth Avenue San Dieiqo, CA 92t52-6800
San Diego, CA 92101Directr. Manpower ana Person-,el
Or. Clarence C. McCormick Laboratorv.
HO, JSM upCM/MEpC r NPROt (Code 06)
2500 Green Bay Road .)a;( u iio. J-, i ,
North 1.hicao. It b0)064
-N uroanizational Systems Lab.
Educational Testing Service NPROX (Coie ()7)
16-T .-, . , --
Library, NPRDC
Dr. James McMichael Code P2O1L
Technical Director San Dieqo, CA 92152-6800
Navy Personnel R&D CenterSan Diego, CA 92152-6800 Commanding Officer,
Naval Research Laboratory
Dr. Barbara Means Code 2627
SRI International Washington, DC 20390
333 Ravenswood AvenueMenlo Park, CA 94025 Dr. Harold F. O'Neil. Jr.
School of Education - WPH 801
Dr. Robert Mislevy Department of Educational
Educational Testing Service Psychology & fechnology
Princeton, NJ 08541 University of Southern CaliforniaLos Angeles, CA 90089-0031
11
I
~~1988,'0!/O06
University of Missouri-Columbia/isutakawa
Dr. James B. Olsen Oepartment of Operations Research,WICAT Systems Naval Postgraduate School1875 South State Street Monterey, CA 93940Orem, UT 84058
Dr. Mark D. ReckaseOffice of Naval Research. ACT
Code 1142CS P. 0. Box 168800 N. Quincy Street Iowa City. IA 52243Arlinqton, VA 22217-5000(6 Copies) Dr. Malcolm Ree
AFHRL/MOAOffice of Naval Research, Brooks AlB, IX 13235
Code 125800 N. Quincy Street Dr. Barry RieqelhauptArlington, VA 22211-5000 HumRRO
1100 South Washington StreetAssistant for MPT Research, Alexandria. VA 22314
Development and StudiesOP OB7 Dr. Carl Ross
Washinqton, DC 203(0 CNLT-PDCDBuilding 90
Dr. Judith Orasanu Great Lakes NrC, IL b0088Basic Research OfficeArmy Research Institute Dr. J. Ryan5001 Eisenhower Avenue Department of EducationAlexandria, VA 22333 University of South Carolina
Columbia, SC 29208Dr. Jesse OrlanskyInstitute for Defense Analyses Dr. Fumiko Samejima1801 N. Beauregard St. Department of PsychologyAlexandria, VA 22311 University of lennessee
3106 Austin Peay Bldg.Dr. Randolph Park Knoxville. I'N 37916-0900Army Research Institute5001 Eisenhower Blvd. Mr. Drew SandsAlexandria, VA 22333 NPRDC Code 62
San Diego, CA 92152-6800Wayne M. PatienceAmerican Council on Education Lowell SchoerGED Testing Service, Suite 20 Psycholoqical & GuantitativeOne Dupont Circle, NW FoundationsWashington, DC 20036 College of Lducation
University of IowaDr. James Paulson Iowa City, IA 52242Department of PsychologyPortland State University Dr. Mary SchratzP.O. Box 751 Navy Personnel R&D CenterPortland, OR 97207 San Diego, CA 92152-6800
Dept. of Administrative Sciences Dr. Dan SegallCode 54 Navv Personnel R&D CenterNaval Postgraduate School San Dieqo, CA 92152Monterey, CA 93943-5026
12
I-w I W
t988/07/06
IJniversity of, Missouri-Columbia/rsutakawa
Dr. W. Steve Sellman Dr. Hariharan SwaminathanOASD(MRA&L) Laboratory of Psychometric and28269 rhe Pentagon Evaluation ResearchWashington, DC 20301 School of Education
University of MassachusettsDr. Kazuo Shigemasu Amherst, MA 010037-9-24 Kugenuma-KaiganFujisawa 251 Mr. Brad SympsonJAPAN Navy Personnel RPM Center
Code-62Dr. William Sim-, San Diego, CA 92152-6800Center for Naval Analysis4401 Ford Avenue Dr. John TangneyP.O. Box 16268 AFOSR/NL, bldg. 410Alexandria, VA 22302-0268 Boiling AIEB. DC 20332-6448
Dr. H. Wallace Sinaiko Dr. Kikumi latsuokaManpower Research CERL
and Advisory Services 2b2 Engineering ResearchSmithsonian Institution Laboratory801 North Pitt Street. Suite 120 103 S. Mathews AvenueAlexandria, VA 22314-1/13 Urbana, IL 61801
Dr. Richard E. Snow Dr. Maurice latsuokaSchool of Education 220 Educal:ion BldgStanford University 1310 S. Sixth St.Stanford, CA 94305 Champaign, IL 61820
Dr. Richard C. Sorensen Dr. David IhissenNavy Personnel R&D Center Department of PsychologySan Diego, CA 92152-6800 University of Kansas
Lawrence, KS 66044Dr. Paul SpeckmanUniversity of Missouri Mr. Gary lhomassonDepartment of Statistics University of IllinoisColumbia, MO 65201 Educational Psychology
Champain, IL 61820Dr. Judy SprayACT Dr. Robert IsutakawaP.O. Box 168 University of MissouriIowa City, IA 52243 Department of Statistics
222 Math. Sciences Bldg.Dr. Martha Stocking Columbia, MO 65211Educational Testing ServicePrinceton, NJ 08541 Dr. Ledyard Tucker
University of IllinoisDr. William Stout Department of" PsychologyUniversity of Illinois 603 E. Daniel StreetDepartment of Statistics Champaiqn, IL 61820101 Illini Hall725 South Wright St.Champaign, IL 61820
13
1988/07/06
University of Missouri-Columbia/isutakawa
Dr. Vern W. Urry Or. Douglas Wetzel
Personnel R&D Center Code 51
Office of Personnel Management Navy Personnel R&D Center
1900 E. Street, NW San Diego, CA 92152-6800
Washington, DC 20415Dr. kand R. Wilcox
Dr. David Vale University of Southern
Assessment Systems Corp. California
2233 University Avenue Depariment of Psychology
Suite 440 Los Angeles. CA 90089-1061
St. Paul, MN 55114German Military Representative
Dr. Frank L. Vicino ATTN: Wolfqang Wildgrube
Navy Personnel R&D Center Streitkraefteamt
San Diego, CA 92152-6800 D-5300 Bonn 24000 Brandywine Street, NW
Dr. Howard Wainer Washington, DC 20016
Educational Testing ServicePrinceton, NJ 08541 Dr. Bruce Williams
Department of Educational
Dr. Minq-Mei Wang Psychology
Lindquist Center University of Illinois
for Measurement Urbana, IL 61801
University of IowaIowa City, 1A 52242 Dr. Hilda Wing
NRC MH-176
Dr. Thomas A. Warm 2101 Constitution Ave.
Coast Guard Institute Washington, DC 20418
P. 0. Substation 18Oklahoma City, OK 73169 Dr. Martin F. Wiskoff
Defense Manpower Data Lenter
Dr. Brian Waters 550 Camino El Estero
HumRRO Suite 200
12908 Argyle Circle Monterey. CA 93943-3231
Alexandria, VA 22314Mr. John H. Wolfe
Dr. David J. Weiss Navy Personnel R&0 Center
N660 Elliott Hall San Diego, CA 92152-6800
University of Minnesota75 E. River Road Dr. George Wong
Minneapolis, MN 55455-0344 Biostatistics LaboratoryMemorial Sloan-Kettering
Dr. Ronald A. Weitzman Cancer Center
Box 146 1275 York Avenue
Carmel, CA 93921 New York, NY 10021
Major John Welsh Dr. Wallace Wulfeck, III
AFHRL/MOAN Navy Personnel R&D Center
Brooks AFB, TX 78223 Code 51San Diego, CA 92152-6800
14
II 1988,0//06
Universiiv of MSsour.-C.uiumbia/lsutakawa
Dr. Kentaro Yamamoto(3-TEducational Testing ServiceRosedale RoadPrinceton. N.J 08541
Dr. Wendy YenCTB/McGraw H i IIDel Monte Research ParkMonterey. CA 93940
Dr. Joseph L. YoungNational Science F-oundationRoom 3201800 G Street, N.W.Washinqton. DC 20550
Mr. Anthony R. ZaraNational Council of State
Boards of Nursing, Inc.625 North Michigan AvenueSuite 1544Chicago, IL 60611
Dr. Peter StoloffCenter for Naval Analysis4401 Ford AvenueP.O. Box 16268Alexandria, VA 22302-0268
15