ED 273 685 · successfui accomplishment of its mdssion. Tbe panel recognized that the center will require increased funding, but funding requests should be judged agaiust the progress

ED 273 685

AUTHORTITLE

INSTITUTION

SPONS AGENCYPUB DATECONTRACTNOTE

AVAILABLE FROM

PUB TYPE

EDRS PRICEDESCRIPTORS

DOCUMENT RESUME

TM 860 622

Levine, Daniel B., Ed.Creating a Center for Education Statistics: A Timefor Action.National Academy of Sciences - National ResearchCouncil, Washington, DC. Commission on Behavioral andSocial Sciences and Education.; National Academy ofSciences - National Research Council, Washington, DC.Committee on National Statistics.Department of Education, Washington, DC.8630084016487p.; A report of the Panel to Evaluate the NationalCenter for Education Statistics.Committee on National StatisticsCouncil, 2101 Constitution Ave.,20418.Reports - Evaluative/Feasibility

MF01/PC04 Plus Postage.*Data Collection; Educational Assessment;*Educational Policy; *Educational Research; FederalPrograms; Longitudinal Studies; National Programs;National Surveys; *Program Evaluation; ProgramImprovement; Public Agencies; Research Design;*Research Problems; Resource Allocation; Standards;*Statistical Data

IDENTIFIERS *National Center For Education Statistics;Timeliness

, National ResearchN.W., Washington, DC

(142)

ABSTRACTAt the request of the Office of the Assistant

Secretary for Educational Research and Improvement, the Panel toEvaluate the National Center for Education Statistics (NCES) wasestablished in 1985 by the National Academy of Sciences. This reportculminates an extensive two-year study of NaS and reflects thepanel's broad view of the center's serious problems. The study hadfour major aspects: (1) to review, describe, and assess existing dataquality and quality assurance processes; (2) to review, describe, andassess program content and services and procedures used forestablishing priorities consistent with the Congressional mandate;(3) to review and assess the timeliness of data collected anddisseminated; and (4) to identify issues hindering NCES in thesuccessfui accomplishment of its mdssion. Tbe panel recognized thatthe center will require increased funding, but funding requestsshould be judged agaiust the progress of the center in meetingstandards; and reflect the expected expansive of data collection. Thepanel's comprehensive recommendations addressed the following areas:(1) mission, role, and responsibilities; (2) content and direction ofdata collection programs; (3) establishing a frame of reference; (4)the advisory council; (5) improving data quality (redesigning theelementary/secondary data programs, reducing errcr, and statisticalstandards); (6) timeliness; (7) staff and budget resources; (8)contracting out; and (9) publications policy. The panel recommendedthat serious consideration be given to abolishing the center andfinding other means to obtain and disseminate education data ifwide-ranaina actions are not undertaken to damage both the image andreality of the center. Appendices include the center's organizationalstructure and biographical sketches of panel and staff. (UMO)

Creating a Center forEducation Statistics:A Time for Action

U.S. DEPARTMENT OF EDUCATIONOffice el Elucatienel Reassach anal Imenmement

EDUCATIONAL. RESOURCES INFORMATIONCENTER (ERIC)

..21latus document has been reproduced asreceived from the person or organizationoriginating It

0 Minor changes have been made to improvereproduction qualcy.

Points of view or opinions stated in Mrs docu-ment do not necessarily represent officialOERI position or policy.

Creating a Center forEducation Statistics:A Time for Action

Daniel B. Levine, editor

Panel to Evaluate the National Center for Educatiol. tatisticsCommittee on National StatisticsCommission en Behavioral and Social Science and EducationNational Research Council

NATIONAL ACADEMY PRESSWashington, D.C. 1986

NOTICE: The project that is the subject of this report wasapproved by the Governing Board of the National ResearchCouncil, whose members are drawn from the councils of theNational Academy of Sciences, the National Academy ofEngineering, and thc Institute of Medicine. The members ofthe committee responsible for the report were chosen fortheir special competences and with regard for appropriatebalance.

This report has been reviewed by a group other than theauthors, according to procedures approved by a ReportReview Committee consisting of members of the NationalAcademy of Sciences, the National Academy of Engineering,and the Institute of Medicine.

The National Research Council was established by theNational Academy of Sciences in 1916 to associate the broadcommunity of science and technology with the Academy'spurposes of furthering knowledge and of advising the federalgovernment. The Council operates in accordance with gen-eral policies determined by the Academy under the authorityof its congressional charter of 1863, which establishes theAcademy as a private, nonprofit, self-governing membershipcorporation. The Council has become the principal operatingagency of both the National Academy of Sciences and theNational Academy of Engineering in the conduct of their ser-vices to the government, the public, and the scientific andengineering communities. It is administered jointly by bothAcademies and the Institute of Medicine. The National Aca-demy of Engineering and the Institute of Medicine wereestablished in 1964 and 1970, respectively, under the charterof the National Academy of Sciences.

This report was prepared for the Department of Educa-tion under contract number 300840164. Points of view oropinions stated in this document are those of the contractorand do not necessarily represent the official position or poli-cies of the Department of Education.

Available from:

Committee on National StatisticsNational Research Council2101 Constitution Ave., N.W.Washington, D.C. 20418

4

Panel to Evaluate the National Centerfor Education Statistics

VINCENT P. BARABBA (Chair), Executive Director, MarketResearch and Planning, General Motors Corporation

ANTHONY BRYK, Department of Education, University ofChicago

ROBERTO M. FERNANDEZ, Department of Sociology,University of Arizona

CHRISTOPHER JENCKS, Center for Urban Affairs,Northwestern University

F. THOMAS JUSTER, Institute for Social Research, Universityof Michigan

STEPHEN KAAGAN, Commissioner of Education, State ofVermont

GRAHAM KALTON, Survey Research Center, Institute forSocial Research, University of Michigan

ALEXANDER LAW, California Department of Education,Sacramento, Calif.

RICHARD LIGHT, Kennedy School of Government andGraduate School of Education, Harvard University

SHIRLEY MALCOM, American Association for theAdvancement of Science, Washington, D.C.

ANDREW PORTER, Institute for Research on Teaching,College of Education, Michigan State University

WRAY SMITH, Harris-Smith Research, Inc., Arlington, Va.

DANIEL B. LEVINE, Study DirectorMICHAEL L. COHEN, ConsultantBERNARD MICHAEL, ConsultantANNE M. SPRAGUE, Administrative Secretary

Committee on National Statistics

STEPHEN E. FIENBERG (Chair), Department of Statistics,Carnegie-Mellon University

LEO BREIMAN, Department of Statistics, University ofCalifornia, Berkeley

WAYNE A. FULLER, Department of Statistics, Iowa StateUniversity

SEYMOUR GEISSER, School of Statistics, University ofMinnesota

JERRY A. HAUSMAN, Department of Economics,Massachusetts Institute of Technology

F. THOMAS JUSTER, Institute for Social Research, Universityof Michigan

NAN M. LAIRD, Department of Biostatistics, Harvard Schoolof Public Health

JANE A. MENKEN, Office of Population Research, PrincetonUniversity

LINCOLN E. MOSES, Department of Statistics, StanfordUniversity

JOHN W. PRATT, Graduate School of Business, HarvardUniversity

S. JAMES PRESS, Department of Statistics, University ofCalifornia, Riverside

COURTENAY M. SLATER, CEC Associates, Washington, D.C.JUDITH M. TANUR, Department of Sociology, State

University of New York, Stony BrookKENNETH W. WACHTER, Department of Statistics, University

of California, Berkeley

EDWIN D. GOLDFIELD, DirectorMIRON L. STRAF, Research DirectorMICHELE W. ZINN, Administrative Associate

iv

6

Contents

ACKNOWLEDGMENTS vii

SUMMARY STATEMENT OF THE PANEL 1

1 EDUCATION STATISTICS: HISTORYAND PROBLEMS 5

A Brief History, 5What the Center Collects, 7The CenterWhom Should 1,t Serve? 11The Center's Problems. 13

2 MISSION, ROLE, AND FRAME OF REFERENCE 23

Mission, Role, and Responsibilities, 24Content and Direction of Data

Collection Programs, 25Establishing a Frame of Reference, 26

3 DESIGNING A PROGRAM 31

The Advisory Council, 32Improving Data Quality. 34Timeliness, 41Resources--Staff and Budget, 43Contracting Out--Or Not, 48A Publications Policy, 51

4 RECOMMENDATIONS 54

Mission, Role, and Responsibilities, 55Content and Direction of Data

Collection Programs, 56Establishing a Frame of Reference, 57The Advisory Council, 57Improving Data Quality, 58Timeliness, 59Resources--Staff and Budget, 60Contracting Out--Or Not, 61A Publications Policy, 61

REFERENCES AND BIBLIOGRAPHY 63

APPENDICES

A ORGANIZATIONAL STRUCTURE OF THECENTER 68

B BIOGRAPHICAL SKETCHES OF PANELMEMBERS AND STAFF 72

vi

8

Acknowledgments

The panel wishes to thank the many people who contributedto the preparation of this report. The staff of the Centerfor Statistics (formerly, the National Center for EducationStatistics, NCES) has been extremely forthco:ling in it:: will-ingness to share with and educate both panel members andstaff. First Lee Eiden and then Francis Corrigan, whoserved as liaisons from the center to the panel, gave freelyand often of their time and expertise and were tireless inmeeting our many requests for both information and exposureto the center, its staff, end its programs. The number ofcenter staff who contributed to our knowledge is beyond ourcapacity to acknowledge individually; to each of them, how-ever, we extend our appreciation.

Special thanks are due Emerson Elliott, director of thecenter, who generously made himself available throughout thestudy and ;hared his thoughts and experience. DonaldSenese, Assistant Secretary, Office of Educational Researchand Improvement, who initiated the study, and his successor,Chester Finn, who strongly encouraged the panel in itsendeavors, provided invaluable support. Dorothy Gilford andMarie Eldridge, who served previously as administrators ofNCES, shared their wealth of experience with the panel andprovided unique insights into the workings of the Center andits constituencies.

The panel also benefited from information and expertiseoffered by staff of other groups within the Department ofEducation that have been involved with the work of the cen-ter, including the Office of Vocational and Adult Education,the Office of Planning, Budget and Evaluation, and the Officeof the Assistant Secretary for Educational Research andImprovement. Among the broader community of users ofeducation data, *e are indebted to William Pierce, GeorgeRush, and Ramsey Selden of the Council of Chief State

vii

School Officers; John Stiglmeier of the New York StateDepartment of Education; Frank Windsor of the MarylandState Board of Education; Douglas Connors of the AmericanAssociation of Collegiate Registrars and Admissions Officers;James Hyatt of the National Association of College and Uni-versity Business Offices; Gene Bottoms of the American Voc-ational Association; Charles Hopkins of the Oklahoma StateDepartment of Vocational Education; Gordon Swanson of theUniversity of Minnesota; Katherine Wallman of the Council ofProfessional Associations on Federal Statistics; and Elaine El-Khawas of the American Council on Education.

Thoughtful and helpful comments on our report werereceived from the many reviewers within the NationalResearch Council, including members of the Committee onNational Statistics and of the Commission on Behavioral andSocial Sciences and Education. The efforts of EugeniaGrohman, associate director for reports of the commission, inediting, commenting on, and reorganizing the report wereespecially valuable in preparing the report for publication.

I also wish to thank the panel members for their generouscontributions of time and expert knowledge. Many of themprepared background or supporting materials for discussionand contributed draft sections for the report, and detailedcomments on the various drafts were forthcoming from virtu-ally all of the members. Our report, and especially, therecommendations, are the result of syntheses of their efforts,ideas, and insights on the issues addressed and, for that rea-son, my task was made easier and more rewarding.

Finally, both for the panel and myself, I would like toexpress thanks to the staff for their commitment, dedication,and efforts. Daniel B. Levine, who served as study director,in collaboration with Anne Sprague, and assisted by BernardMichael and Michael Cohen, consultants, worked long andhard, endured (and sometimes dodged) the misaimed slingsand arrows of panel members, produced a cohesive synthesisof our debates, discussions, and conclusions, and rewrotepatiently to accommodate comments from all directions. Weand the education community are indebted to them.

Vincent P. Barabba, ChairPanel to Evaluate the NationalCenter for Education Statistics

viii

10

Summary Statement of the Panel

In late 1984 the National Academy of Sciences was asked bythe Office of the Assistant Secretary for EducationalResearch and Improvement of the U.S. Department ofEducation to undertake an evaluation of the National Centerfor Education Statistics (NCES). The request was areflection of the expressed concern that the center had lostthe confidence both of those providing data to it and ofthose who used its products, that the quality of its productshad declined, that it failed to provide its data in a timelyfashion, and that its interpretations of those data it did pro-vide were flawed.

At the same time, an educational reform movement hadbrought education statistics into what is likely to be thecenter of policy debate for years to come. The educationalreform movement burst onto the public agenda in early 1983with the issuance of the first of the reform reports from theDepartment of Education's Commission on Excellence in Edu-cation (U.S. Department of Education, 1983). The spate ofreports that followed from a multitude of groups laying claimto the issue relied on data from the center, among othersources, to provide the evidence for declines in quality andto argue for specific solutions, such as increasing high schoolgraduation requirements in mathematics and science.

This situation provides a dramatic illustration of thepower of data to fuel a policy debate and of the changingdemands for data that come with the recognition of a crisis.How can policy makers identify needed intervention points?Hcw can society increase the amount of educational output?How will people know if reforms have been effective? Thedemand for data to help answer these and other questionshas thrust the center forward and promises to keep it therefor a long time.

1

ii

2

To carry out the study requested by the department, thePanel to Evaluate the National Center for Education Statis-tics was established in January 1985 under the Committee onNational Statistics of the National Research Council. Thestudy had four major aspects:

o to review, describe, and assess existing data qualityand quality assurance processes by NCES, includingstandards and guidelines for conducting surveys andreleasing and publishing the data, and proceduresused to ensure accuracy, reliability, and validity ofthe data as well as ensuring consistency anduniformity over time.

o to review, describe, and assess program content andservices of NCES and the procedures used for estab-lishing priorities consistent with the congressionalmandate.

o to review and assess the timeliness of data collectedand disseminated by NCES.

o to identify any issues that obstruct or hinder NCESin the successful accomplishment of its mission.

The panel's activities included interviews with appropriateofficials and experts within the center, the Department ofEducation, and the federal government; with members of thedepartment's Advisory Council on Education; with representa-tives of many of the education organizations and professionalgroups concerned with education; with other users of educa-tion data; and with providers of the data to the center. Thepanel drew on the expertise and advice of other committeesand panels of the National Research Council whose interestsextend to the use of education statistics. The panel alsoreviewed an extensive body of materials, including the annualreports of the advisory council, reports of previous reviews,internal memoranda, and relevant correspondence.

The panel began its activities with the full intention ofapproaching the issues through examination of individual pro-grams. We soon found, however, that the center's problems,rather than being unique to particular programs, extendedthroughout the center. Thus our report focuses on issuesrather than programs.

The report culminates an extensive two-year study andreflects the panel's very broad view of the problems of thecenter. We have addressed the problems both from the reali-

12

3

ties of the past and present and from the longer-range per-spective of the center as we perceive it in the years ahead.Simply put, we have found the center wanting.

We believe that the discussion and recommendations in ourreport provide a realistic approach to dealing with the issuesand problems that the center faces and that actions such asthose we recommend are critical to its survival as a credibleorganization. Neither our findings nor many of our recom-mendations are startling in nature; rather, they tend to echoor expand what has been said by previous groups that haveexplored the same issues. However, because we believe edu-cation issues are now so urgent, we place more emphasis oncomprehensive action than did past reviews.

Given that the need for information about education isbeyond argument, in this report we outline a goal that iswithin the grasp of a statist:m1 agehey working in the dif-ficult environment of the federal government with its dispar-ate objectives and multitudinous demands on budgets andstaff. To a large extent, the actions we propose, to a largeextent, are inherent elements in the operating philosophiesthat guide such respected statistical organizations as theBureau of the Census, the Bureau of Labor Statistics, theNational Center for Health Statistics, and the Energy Inform-ation Administration. In essence, we seek to institutechanges at the Center for Statistics for which there is ampleprecedent.

As we make clear in the report, the center's problemsextend throughout its full program; thus, our recommenda-tions are a comprehensive and integrated set. Our proposalsprovide a package that, when taken as a whole, will initiateand reinforce a process of improvement. The approach isrealistic--not radical. The changes we propose cannot beeffected overnight, or even in a year. But they can beginwith the initial and fundamental steps of improvement.

Simple tinkering--selecting a suggestion here, reorganizinga bit there, adding a memo, deleting a regulation--has notworked and will not work, as the history of the past 25years shows. Such self-deception, represented by quick-fixapproaches, merely reinforces the already unacceptable statusquo. The overriding issue before the Department of Educa-tion and the Center for Statistics is one of intent, commit-ment, and leadership.

It is appropriate at this point to state unequivocally thatwe are not proposing to solve the center's problems merelyby throwing money at them. While the center does requireimmediate additional staff, and thus some incremental addi-tional funding, just to carry out its current workload, we

13

4

firmly believe that thc centcr, by implementing many of ourrecommendations, can significantly improve its operationswithout additional large infusions. Certainly, the panelrccognizes and accepts that over time the center will requireincreased funding, but such requests, first, should be judgedagainst the progress of the center in meeting the standardsexpected of it and, second, should reflect an expected expan-sion in thc types and scope of data the center collects as itdemonstrates an ability to meet thc diverse nceds of itsuscrs.

We wish to emphasize the seriousness with which we viewthe ccnter's problems. We believe that there can be r.odcfense for allowing the center to continue as it has for alltoo long. If, indeed, "the nation is at risk" in the area ofcducation, it is past time for those in positions of responsi-bility to acknowledge the risks and dangers of perpetuatingthc myriad and continuing problems of the center. Withoutstrong and continuous commitment and demonstrated determ-ination to undertake wide-ranging actions to change both theimage and reality of the center, we are unanimous in ourconviction that serious consideration should be given to themore drastic alternatives of abolishing the center and findingother means to obtain and disseminate education data. A

number of potential alternatives to the center exist, such asplacing leadership responsibility elsewhere in the departmentand collecting the needed data through contracting withanother federal statistical agency or with the private sector,through a federally funded nonprofit research centerestablished by the department at the direction of the Con-gress, or through directing other entities within the depart-ment to accomplish the task.

We emphasize strongly, however, that we believe the pre-ferred course of action is to begin the process of improve-ment. As we have noted, the center's problems are long-standing and pervasive, but if faced openly they can, in time,be overcome. As for this report--and its many predecessors--perhaps our thought is best expressed in the words of SirFrancis Bacon:

Crafty men condemn studiesSimple men admire themWise men use them.

14

1

Education Statistics:History and Probkms

A BRIEF HISTORY

In 1867 Congress established a Department of Education and,in the first section of the legislation, gave it a primary mis-sion of "collecting such statistics and facts as shall show thecondition and progress of education in the several States andTerritories, and of diffusing such information respecting theorganization and management of schools and school systemsand methods of teaching." Congress further provided for theCommissioner of Education to present annually "a reportembodying the results of his investigations and labors,together with a statement of such facts and recommendationsas will, in his judgment, subserve the purpose for which thisdepartment is established." Thus in the department's earliestincarnation, the need for and importance of statistics wasboth recognined and established.

During the ensuing 30 years, the Commissioners of Educa-tion focused the efforts of what by then had become aBureau of Education on the collection and reporting of schoolstatistics, such that one scholar observed "that fae phrase[reporting of school statistics] became a definition of thebureau itself" (Warren, 1974:163). The first index to its pub-lications was issued in 1891 and covered the Bureau's effortsbetween 1867 and 1890. Containing over 100 pages, the list-ings are almost equally divided among historical sketches,descriptive pieces, and statistical summaries (U.S. Bureau ofEducation, 1891). Even at this early point there was recog-nition of the need to interpret the facts. For example, inits annual report for 1897, two chapters were devoted to adescriptive anallysis of program data acquired by the office;in 1902, the annual report dealt with the subject of blackstudent enrollment in higher education; and in 1903, the edu-cation of women was a focus of discussion.

5

15

6

This focus on thc collection and dissemination of cduca-tion statistics by the Bureau of Education (and, later, theOffice of Education) continued more or less unabatedthroughout the first half of the twentieth century, even asthe organization lost its indcpendent status and became apart of the Department of the Interior (1868) and eventuallysettled within the Federal Security Agency (1939), whichlater became the Department of Health, Education, andWelfare.

The National Center for Education Statistics appeared inJanuary 1965, having been established as a staff officereporting to the Commissioner of Education. The idea for accnter originated from the recommendations of a public advi-sory commission and a White House task force "to providefor cducation, the statistical expertise and products whichthe Bureau of Labor Statistics provides for Labor, theNational Center for Health Statistics for Health, and theAgricultural Reporting Service for Agriculture" (Ottina, 1973).The center assumed responsibility for all cf the general pur-pose data athering within the Office of Education and forhelping states, through a program of grants, improve theirown data-gathering and statistical capabilities.

The federal education programs of the Great Society, ini-tiated dui ing the administration of President Johnson, placednew and heavy demands on the very limited resources of theyoung center. However, the needs of the center for addi-tional staff and funding failed to win the support of theCongress. Rather, Congress sought to strengthen the centerby changing it in 1974 to a statutory entity reportingdirectly to the Assistant Secretary for Education. With theestablishment of a separate Department of Education in early1980, the center became part of Office of EducationalResearch and Improvement and continued to report to anassistant secretary. In October 1985 yet another reorganiza-tion was carried out which, among other changes, resulted inthe renaming of the National Center for Education Statis-tics--"henceforth it will be known as the 'Center for Statis-tics'"--and removed some of its information disseminationfunctions. As is always the case, some internal restructuringoccurred, with appropriate name changes for the operatingelements. A description of the organizational structurethe center, as of April 1986, is in Appendix A.

16

7

WHAT THE CENTER COLLECTS

Until the late 1960s the statistical programs of the centerand its predecessors were restricted primarily to generalpurpose data--enrollments, staff, and finance in the elemen-tary, secondary, and postsecondary schools. The past quartercentury, however, has seen the demands for additional anddifferent kinds of info.mation, which has resulted both in theinitiation of special scudies designed to meet the specificneeds of Congress and in the development of new and expan-ded data collection systems at all levels of education. Forexample, the elementary/secondary school program wasenlarged and expanded to reflect the effects of preprimaryeducation and of adult and noncollegiate postsecondary edu-cation, to provide data on libraries and, most importantly, toimplement longitudinal studies to measure and report on theeffects of cumulative student development and change. Inaddition, information on inputs (such as enrollments, numberof schools, and funding) and process (such as expendituresand curricula) was supplemented to reflect the increasedemphasis in outcome measures--what students learn, whatthey know, and what happens to them after they completetheir formal education and training. During this period, thecenter also initiated a series of publications, established afast response survey system to meet data needs for policydetermination, became responsible for the National Assess-ment of Educational Progress (NAEP) and, overall, grew inprogram and importance. The center lost responsibility forNAEP later in the decade, but regained it in October 1985.The principal center programs at the time the panel's studywas initiated are described briefly below.

Common Core of Data

The Common Core of Data (CCD) program currently is theprimary source of basic statistical data about public elemen-tary and secondary educational institutions. Much of thedata is derived from administrative records maintained by thestate education agencies (SEAs), which compile the data inthe desired formats and transmit them to the center. TheCCD consists of the following six separate parts.

Part I. The Public School Universe File. This annualcensus of SEAs provides information on all public elementaryand .eco.Idary schools in operation during a school year.This file includes information on school type, grade span, fall

17

8

membership, and the number of classroom teachers. Theschool data are aggregated to the state level.

Part II. Local Education Agency (LEA) Universe File.This annual census of SEAs ptovides information for the uni-verse of LEAs in a state on of agency, operating status,fiscal status, and control st.. The LEA data are aggre-gated to the state level.

Part III. Local Education Agency Nonfiscal Report. Thisannual census of SEAs provides data on LEAs on number ofschools operated by the agency, membership, and full-timeequivalent teachers and other staff. The LEA data areaggregated to the state level.

Part IV. Public School District Finance Report. Thisannual census of SEAs provides data on LEAs on averagedaily attendance, revenues by source, and expenditures bymajor function.

Part V. State Aggregate Nonfiscal Report. This annualcensus of SEAs provides data for states on fall membershipby grade level, full-time-equivalent staff by major category,and high school graduates.

Part VI. State Aggregate Fiscal Report. This annualcensus of SEAs and other state agencies that provideresources to LEAs provides aggregate data for states onaverage daily attendance, school district revenues by source,and expenditures by major function.

Sample Surveys

In addition to the administrative data collected through theCommon Core of Data, the center conducts sample surveysamong education institutions to obtain other data on publicand private eementary and secondary education to addressemerging poli'y issues. Some of these surveys are doneperiodicall...: others are of a one-time nature. Some recentexamples include:

Public School Survey. This survey of a nationallyrepresentative sample of both schools and teachers within theschools obtained core summary data and supplementary dataon topics of policy interest. The core data collected fromschool admiaistrators included: grade span, fall membership,

18

9

minority enrollment, full-time equivalent (FTE) teachers andother school staff, class size, and high school graduates.Data collected from teachers included: age, sex, and race;education and training; teaching assignments; time utilization;and compensation. The data from the surveys of schools andteachers were aggregated to the national level.

Private School Survey. This survey nf a nationallyrepresentative sample of private elementary and secondaryschools, supplemented by an area sample of schools, obtainedcore school summary data and supplementary data on topicsof policy interest. The data collected on schools were, ingeneral, parallel to data collected in the public school survey,but the survey also included data on school policies, programofferings, tuition rates, and participation in federal program.As in the public school survey, data were aggregated to thenational level.

Survey of Teacher Demand and Shortage. This survey ofa nationally representative sample of LEAs and privateschools obtained data on the number of teachers byassignment, the number of new hires, the number of positionsthat ould not be fill-.d, and recruiting and employmentpractices. The data from LEAs and private schools wereaggregated to form national estimates.

National Assessment of Education Progress

The National Assessment of Educational Progress (NAEP) wasmandated by Congress as a program to continuously monitorthe knowledge, skills, and performance of the nation'schildren and youth. Begun in 1969, NAEP has each yearconducted assessments of students at ages 9, 13, and 17 inten school-related subject areas: reading, mathematics, writ-ing, science, citizenship, social studies, art, music, literature,and career and occupational development. NAEP alsoconducts special assessments in other educational areas, suchas young adult literacy, which is an assessment of the natureand extent of literacy among young adults aged 21-25.

Longitudinal Surveys

Beginning in 1972 the center initiated the first in a series oflongitudinal studies, known as the National LongitudinalStudy of the High School Class of 1972. The fifth follow-up

19

10

survey of this study was conducted in 1985. The secondstudy, High School and Beyond, is a national longitudinalstudy of cohorts of 1980 high school sophomores and seniors,who are being interviewed over time to determine whathappened to them after high school completion. Parents,teachers, and school administrators were also interviewed inconnection with /he study. Beginning in 1988, this surveywill be complemented by the National Education LongitudinalSurvey (NELS:88), which will begin with an eighth gradecohort. Thc major objectives of the longitudinal studies isto provide trend info, mation about the quality, equality, anddiversity of educational opportunity and the effect of thesefactors on individual development and educational and careerou!comes; changes over time in educational and careeroutcomes; and transition rates for such changes as highschool to college, two-year colleges to four-year colleges anduniversities, and school to work.

Higher Education G:neral Information Survey

The Higher Education General Information Survey (HEGIS),begun in the mid-1960s, is a coordinated effort designed toacquire and maintain statistical data on the characteristicsand operations of institutions of higher education. A series ofsurveys, HEGIS collects information from the universe ofinstitutions of higher education on five subjects: institu-tional characteristics of colleges and universities; opening fallenrollment; salaries, tenure, and fringe benefits of full-timeinstructional faculty; degrees and other formal awards con-ferred; and financial statistics.

Concurrent with the conduct of HEGIS, the center wasdeveloping a new program, titled the Integrated Postsecon-dary Education Data System (IPEDS), which will replaceHEGIS and provide a data base covering ooth traditional andnontraditional postsecondary institutions. A major rationalefor the development of an integrated system is its improve-ment over some currently used surveys of postsecondary edu-cation that are recognized as having statistical deficiencies.As the panel's study was under way, forms and procedureshad been established and a pilot study successfully completed.Plans call for this revised approach to be introduced to thcpostsccondary cducation community beginning in late 1986.

20

11

Vocational Education Data System

Mandated in 1976, the objective of the Vocational EducationData System (VEDS) program was to collect data on voca-tional education for purposes of policy analysis, programaccountability, program evaluation, and manpower planning.The program did not meet the expectations, despite under-going a number of revisions, and was terminated in December1983. At the present time, the center is exploring a numberof alternative means of collecting data on vocational educa-tion.

Other Data Collection Efforts

In addition to the sources described above, the centerconducts ad hoc surveys, as needed, on subjects of immediateinterest or obtains data undet contractual agreements withSEAs, with private contractors, or with one of the federalstatistical agencies.

It also must be borne in mind that the data collected bythe center represents only a rezlatively small proportion ofthe data collected by the department, since progiam officesdirectly collect program and compliance data.

With the appointment of a new administrator in late 1984,the center initiated E. thorough review oi all of its data col-lection programs, addressing issues of suitability, scope, relia-bility, and timeliness of its products from the viewpoints ofthe variety of purposes to be served, including instructional,administrative, and policy. This endeavor is unique in that itincorporates direct public comment and participation by theeducation community (and by other interested parties) on allaspects of the center's programs, from objectives throughdata needs and from collection methodologies to output andproduct. The extent to which this approach is successful andis reflected in the future data-gathering and information-pro-ducing activities of the center cannot be assessed at thistime, but the panel commends the center leadership fortaking this step and for attempting to involve its communityof interested and concerned users in the development of themodel which will guide its program for the future.

THE CENTER--WHOM SHOULD IT SERVE?

With 120 years of tradition to call upon, the appropriatenessof the federal government's role in the collection and provi-

21

12

sion of statistical information on education is no longer anissue. Recently, in fact, a report by the Heritage Foundationcalling for the abolition of the Department of Educationcategorically exempted the collection of statistics (Butler etal., 1984). Nonetheless, there has been, and continues to be,ongoing and ofttimes stridcnt debate, if not confusion, on themore fundamental issue, namely, the role of a centcr foreducation statistics: What is it? What should it be?

The arguments and disagreements are long standing and,like dcmons, appear and reappear in a variety of forms.Perhaps the broadest and most fundamental disagreement isexpressed as the conflict between two philosophies: one thatmaintains the availability of wide and ready access to basicinformation is an important federal responsibility, and onethat justifies federal data collection only as required to meetspecific program needs. That this is not a new problem isborne out by reference to a management evaluation conductedby the Department of Health, Education and Welfare in 1978,which noted "confused opinions on what NCES' role is andshould be in relation to the Education Division, the Off icr.: ofthe Secretary, and Congressional mandate . . ." The reportwent on to note: "this confusion . . . is a result of onemajor factor: the lack of an established role in the Depart-ment for NCES" (U.S. Department of Health, Education, andWelfare, 1978:13). This conccrn also was expressed by theAdvisory Council on Education Statistics (ACES) in its secondannual report, issued in 1977. The council expressed itsconcern in terms of the degree of autonomy and indepen-dence available to NCES, to "insure its products and activi-ties have an actual and perceived non-political character" andstated, further, that its role in "collecting data for purposesof compliance enforcement, program evaluation or regulation. . . must not impair the effectiveness of the center as ageneral purpose statistical agency" (U.S. Department ofHealth, Education, and Welfare, 1977:6).

A clear view of the role of a federal statistical agency, inthe panel's view, is found in remarks presented at the CensusBureau's first annual research conference in March 1985. Indiscussing the role of statistics in government decisionmaking, the then Under Secretary for Economic Affairs ofthe Department of Commerce, Dr. Sidney L. Jones (1985),said:

I believe that the mission of statistical agencies shouldbe to fulfill their specific responsibilities within theoverall information system, rather than responding tothe ad hoc functional priorities of their parent institu-

22

13

tions. . . . These two institutions (Census and theBureaL of Economic Analysis), along with others likethe Bureau of Labor Statistics, the Bureau of JusticeStatistics, the National Centers of Health and Educa-tion Statistics, and the Statistical Reporting Service ofthe Department of Agriculture, are all part of aninformation system whose principal responsibility is theintegrity of the numbers, not responsiveness to politi-cal needs. That should be their number-one priority.The fact that they happen to be parts of . . . some[other] department is, to me, not the controlling factof their missions.

The panel fully shares Dr. Jones' view. To restrict thecenter solely or mainly to the data requirements of thosewith program responsibility, in the panel's opinion, would beextremely shortsighted. It would result in a statisticalorganization stilted in scope, limited in flexibility, and viewedas a partisan arm of the department. And it would violatethe history of and congressional intent for the center.

THE CENTER'S PROBLEMS

The center has experienced relatively little success over theyears in obtaining support for funds, staff. or program initia-tives from its department, from the Office of Managementand Budget, or from Congress. This lack of support has con-tributed to the inability of the center to develop both theimage and the reality of a competent and objective majorstatistical organization serving the wide need for statisticsabout education in the United States. The reason for persis-tent lack of support is not clear. One suggestion is that thehighly decentralized nature of education has placed the gath-ering of statistics at the vortex of a continuing struggle, atug-of-war, if you will, between the diverse players in theeducation field--local and state officials, associations andinterest groups, the Congress, and groups with programresponsibilities within the department--few of whom could ordid agree on what should or could be collected, or how. Arecent report by the Committee for Economic Development,Investing in Our Children, summarized the situation in thesewords: "Private industry could not succeed with a data-col-lection system and research base as weak as this naCion hasin the field of education" (quoted in AnthropologicalNewsletter 26(9), 1985).

23

14

Reports of prior advisory committees--in 1957, in 1960,and in 1963--and the report of an internal departmentalstudy in 1978 all testify to the persistent problems that haveplagued the center for many years. The most recent ofthese advisory committee reports on the state of educationalstatistics was explicit in their discussions of "persistent prob-lems" (Advisory Panel on Education Statistics, 1963):

o long delays in the completion of many studies andreports;

o difficulties in the reporting of basic information by theinitiating sources;

o less than adequate understanding of the needs of usersand anticipation of new needs;

o failure to utilize fully and effectively modern methodsof collecting and analyzing data, including sampling;and,

o lack of adequate planning, organizing, and schedulingof the work.

It is somewhat sobering to contemplate the conclusion ofthe 1963 report of the Advisory Panel on Education Statis-tics, almost a quarter of a century ago, that "the indefinitecontinuance of these chronic problems is entirely unsatisfac-tory" (U.S. Department of Health, Education, and Welfare,1963:3). It remains so today, as concluded by the clepart-ment's own Advisory Council on Education Statistics, whichinitiated the work of this panel because of its own conceras,as well as by many others who had made their views knownto the council. Another important conclusion to be drawnfrom this history is that, whatever the reason, the adviceand counsel that have been sought and obtained appears tohave been ignored and forgotten.

The most serious problems repeatedly cited to the panelconcern thP quality and lack of timeliness of the dataproduced b) the center. These problems are widely knownand have been openly expressed--in many of the more than50 papers prepared in connection with the center's ongoingredesign of its elementary and secondary data system and, aswe noted, by the center itself as a major rationale for thedevelopment of its revised system of postsecondary data col-lection.

9 4

15

Quality

The seriousness of this issue of quality is clearly illustratedby a quote from one of the redesign papers: "If the datacontinue to be as inaccurate in the future as they have inthe past, an other issues are moot" (Dr Vid, 1985:143). Thepoor quality of the data is generally .:ttributed to the factthat the data are collected, in large pat t, from administrativerecords maintained at the local level, which record "official"rather than "real" behavior; that the data are the product ofdiverse record-keeping systems that lack comparability hidefinitions and time periods; that the data provided to thecenter are at such gross levels of aggregation--such as for astate as a whole--as to seriously limit anyone's ability tocheck them for accuracy, consistency, and reasonableness;and that the data as published are at such summary levels ofgeography--such as a region--as to seriously limit their ana-lytical usefulness.

Our review fozind that, for the most part, the centerlacks written standards to guide many, if not all, of its tech-nical activities, including those concerned with collectingdata, monitoring contracts, and publishing reports. Many ofthe problems the center has encountered also lead us to con-clude that unwritten standards of performance either do notexist or are not known to the staff. Furthermore, the appli-cation of general statistical standards is highly variablethroughout the organization and its products. In publica-tions, for example, technical detail is often lacking, errorstatemems do not provide the required information, tablecolumns or rows may not add to totals, table titles are notdescriptive of the content, and headlines are not borne outby the data. In reviewing the background of the publica-tions, it is also clear that the comments of reviewers havenot been taken into account.

The effort to collect data on vocational education, asmandated by Congress in 1976, illustrates virtually everyproblem encountered in our review. The history of theVocational Education Data System (VEDS), from its inceptionto its termination in 1983, is marked with failed attempts bya wide range of interested and invtAved parties, including thecenter, other parts of the department, states, and Congress,to establish, agree on, and accomplish realistic goals.Furthermore, the congressional mandate for VEDS failed toprovide either staff or financial resources for the new effort,nor did the department shift resources to assist the center;consequently, the center had to consume large amounts of its

25

16

scarce resources on VEDS, to thc detriment of othcr ongoingprogra ms.

Bctween f;scal 1978 and 1983, the center spent about $2.1million on start-up costs for VEDS and an average of$600,000 per year in operating costs. The costs to thestates--and thus the eventual loss--was manyfold greater; thecenter estimated that state and local start-up costs werebetween $10 and $35 million and that operating costs were at$15 to $20 million per year. As notcd, however, dcspite thisconsiderable expense and effort, VEDS failed to meet itsgoals: it collected thc full scope of planned data requirementsonly once in its five-year life.

Furthermore, in late August 1983 the center notified thedepartment that apparent problems had bccn identified in thedata that had been collected and, in early 1984, the centerconfirmed its earlier statements and added that the VEDSdata were unreliable and subject to serious misinterpretation.The center listed many reasons for the unreliability: lack ofcornpnrability among states, resulting in misleading nationaltotals; year-to-year variability in the data, severely limitingthe usefulness in trend analysis; within-state discrepancies,identified when data from VEDS wcre compared with datafrom independent sources; variability in collection methodol-ogy, reflecting poor management within states; a lack of uni-form reporting, resulting from difficulty in applying uniformdefinitions; duplicated counts--in a study of nine states in1981, duplicated enrollments ranged from 10 to 50 percent;changing universes, reflecting the changing composition ofstate plans: differential response rates, reflecting yearly vari-ations in the number of reporting units; missing data--anumber of states were unable to provide the detail requested,particularly for financial data.

In December 1983 the Office of Management and Budgetdisapproved the department's request for VEDS collet.:tion for1983-84. The center had begun the process of revising theprogram in August 1983; the sixth revision was submitted inSeptember 1984; as of August 1986, collection of data onvocational education had not becn resumed. Given this chainof events, one can easily agree with the thought expressedby a former administrator that "VEDS is a four-letter word."

That the project failed is certainly unfortunate. Mostdamaging, though, was the high price exacted from thecenter--it bore the brunt from all sides, with substantialadditional loss to its already fragile professional image and toits morale.

VEDS, however, is not the only center program withproblems. Response rates in the collection of the Common

17

Core of Data components from elementary and secondaryschools have been far below any acceptable standard, yetlittle if any concern appears to have been expressed or anyaction taken until very recently. At one of its earlymeetings, the panel was astonished to be told that it wasusual practice to have received less than half of theresponses from the states by the cutoff data specified by thecenter, that six or more additional months might elapsebefore most of the delinquent statcs complied, and that somestates just failed to comply. In the case of the most recentgroup of CCD surveys (those for the 1983-84 school year orlater), not a single one of the six component surveys wascompleted by the due date. And two months later, the samesituation existed--not a single survey had achieved a 100percent state return rate: three of the surveys were missing10 percent of the replies, one was missing 15 percent, andthe remaining two were still awaiting replies from one-fourthof the states. And one year after the due date, three of thesurveys were still incomplete. The panel was further aston-ished to learn that no procedures, such as organized follow-up, existed for dealing with noncompliance. Rather, thecenter sent a serics of advance notices of a forthcominginquiry to the states. Techniques for improving responserates, of course, have been standard procedures among datacollection organizations for many years.

In addition to lack of completeness, evaluation of results,investigations into possible sources of error, exposure to peergroups, record checks--standard operating procedures forstatistical surveys--seem not to have been part and parcel ofthe center's operating philosophy, and their absence has con-tributed to the center's difficulties. The center's problemsof data quality are further compounded by the lack of anyspecific individual or office with the assigned and exp1icitresponsibility either for putting standards in place or seeingthat they are carried out.

The center's publications can also be faulted with regardto quality and accuracy. For example, a bulletin issued inApril 1982 (NCES 82.3116) presented the results of a surveyto determine the labor market experiences of students whocompleted vocational education programs in 1979. The meth-odological description is both so brief and incomplete as tobe of little value to users. The role of the individual statesin the collection of the data is confused, as is that of theeducation components within the states; response rates asgiven do not distinguish clearly between those for states andthose for completers (a term that is never defined), andresponse rates for those states using sampling are not shown.

27

18

The weighted total of replies (which is erroneously labeled asrespondents) appears in the body of the hulletin, but thenumber is omitted from the methodological Jiscussion. Thatdiscussion does note the failure of three states containinghalf of one universe to participate in the survey, but thisfact, or an appropriate caveat, are not noted in the body ofthe bulletin. A proportion noted in the text differs consid-erably from that which appears in the appropriate table.Finally, the data are presented to the unit digit (e.g.,591,377), ascribing a wholly misleading and totallyunwarranted degree of precision to the results.

In similar fashion, data on enrollment in vocational educa-tion by provider and state appears in Table 3.2 of the 1983Condition of Education. In no fewer than nine states, thenumbers of enrollees in vocational education in public secon-dary schools in 1979, as shown in this table, exceeded thetotal reported as enrolled in secondary schools, as shown ina table from another center publication, the 1979 Digest ofEducation Statistics. In yet another instance, review by anassistant secretary resulted in two pages of comments on acenter report on nonresident alien enrollments in higher edu-cation, most of which addressed the issues of clarity, accura-cy, consistency, and ease of presentation; the reportpreviously had been reviewed fully and approved at all levelswithin the center.

The perception of the center is not helped when anotherpart of the department openly criticizes a report issued bythe center. In one case, the Office of Vocational and AdultEducation (OVAE) challenged a report issued by the center inMarch 1983 and based on expenditure data collected throughVEDS. The NCES Bulletin, which was headlined "State andLocal Expenditures for Vocational Education Continue SteepDecline" (National Center for Education Statistics, 1983b),does in fact represent an unsatisfactory piece of statisticalpresenteion that passed through the center's review processwithout major change or criticism at any level. Both thebasic work and the review were clearly inadequate. Theheadline refers to declines in expenditures; the data in thereport refer to changes in the ratios of state and localexpenditures to federal expenditures--a very different matter.The report also suffered from sloppy and inconsistentpresentation, such as rounding errors and switching betweencurrent and constant dollars. The reaction of OVAE to theBulletin, however, missed most of the inconsistencies andinstead focused on the differences between earty and revisedreports from states.

28

19

The controversy did contribute to the center's initiationof en evaluation of the VEDS data quality (though nothorough process of checking back to the states was evercarried out), to the institution of formalized review processeswithin the department for center products and, ultimately, tothe demise of VEDS.

Of course, not all the center's work bad, and theintegrity of a statistical organization is seriously compromisedwhen its advice, based on sound statistical procedures, isignored. Such was the case when, in early 1984, at theurging of the American Association of University Professors(AAUP), the Secretary of Education directed the center, overthe objections of both the center and the Assistant Secretaryfor Educational Research and Improvement, to provide AAUPwith a tape that contained both incomplete and not fullyedited data on salaries and tenure of staff in colleges anduniversities, collected as part of the Higher EducationGeneral Information Survey (HEGIS). Subsequently, an articlein The Chronicle of Higher Education raised serious questionsabout the data and their release by the center. The finalirony was the concern expressed at the top level when thecenter was subsequently accused of releasing the incompleteand incorrect files.

Timeliness

One of the major concerns expressed by users of the center'sdata, and reflected in the decision of ACES to seek thisreview, has been the lack of timeliness in the data. Thislack, which has resulted in inordinate delays in the publica-tion of important information, reflects a variety of causes.For example, poor planning or inadequate management of asurvey process can result in an overly extended data collec-tion period, as in the case of CCD as noted earlier; clericalprocessing and data production requirements can be underes-timated; the complexity of activities can be ignored or notfully understood; time schedules can be allowed to slip; fundscan be inadequate for the needs; and, finally, long delays canoccur in the preparation of results for publication and in theprocess itself. The end result is that the data are lessuseful and users are frustrated. All of these factors havebeen present, either singly or together, in the center'sefforts to produce its products. The result is that the mostrecent edition of the Digest of Education Statistics, publishedin December 1983, contains state-level data on studentenrollment only through fall 1982, instructional staff data

29

20

only through fall 1981, and finance data only through the1980-81 school year. As of August 1986, the latest publishedstate-level data in each of these areas appears in the 1985edition of The Condition of Education and refers to 1983-84for school enrollments, 1981-82 for instructional staff, and1982-83 for school revenues and expenditures. The 1986 edi-tion of The Condition of Education, originally scheduled forpublication in June 1986, is now promised by the end of theyear.

The occasional bulletins issued by the center appear tohave fared equally poorly. A Directory of Computer rapestook six months to complete internal center review, followingwhich an outside reviewer still found numerous problems inthe publication. And, unfortunately, the time period betweencollection and publication remains long--one bulletin report-ing on data for 1982 was still in review in early 1985.

Conceptual Obsolescence

Another issue to be dealt with in the field of education sta-tistics is that of conceptual obsolescence. As pointed out byBonnen (1977), two types of conceptual obsolescence occur indata: one, because of changes in the organization andframework of that which is being studied, in this case, theeducation system, and one, because of the changes in thedemands on the system. Changing the design of data or itscollection system--which the center is now attemptingthrough its different redesign efforts--invariably involves aconflict between those who strongly favor the status quo inorder to maintain comparability over time and those whoseneeds will best be met by the proposed changes. Resolutionof conceptual issues is neither simple nor immediate. On theother hand, according to Bonnen (1977:399): "Failure to keepup with the changes in the policy agenda and in the realityof (education) leads to significant conceptual obsolescence,and the system begins to lose its capacity as an accurateguide for problem identification and solution or management."That this has been one of the concerns expressed about thecenter is borne out by the total of 59 articles, letters, andcomments covering 785 printed pages and containing hundredsof recommendations from more than 50 experts, organizations,and agencies that the center received in response to a broad,public request to the education community to comment on theproposed elementary/secondary redesign program and tosuggest ideas for improving the nation's store of data foreducation policy.

30

21

Funding and Staff Resources

Finally, the center has long faced serious questions about itsprofessionalism, staff competence, and management abilities.The management evaluation referred to earlier (U.S. Depart-ment of Health, Education, and Welfare, 1978), for example,listed ten problems, varying from a lack of response on thepart of the center to policy needs, dissatisfaction with thetimeliness of the products, general lack of responsiveness onthe part of NCES to special requests or to users, to dis-agreement over the nature and extent of data analysis to beperformed.

In its response to the 1978 internal management review,the center noted two major problems that it believed wer-;the cause of ell other problems--funding and staff resourcesthat were much lower than those of other statistical agen-cies. Whether the sole cause or not, what was true in 1978is equally so today: the center continues to be the smallestof the five primary federal statistical agencies in both dollarsand staff. In fiscal 1985, its budget stood at $14.2 millionwith 134 authorized positions. In contrast, the NationalCenter for Health Statistics, which, until the establishment ofthe Department of Education in 1980 was a sister statisticalagency in the same department, had a budget of $42.8 millionand 493 authorized Positions. Both the ccater's budget(adjusted for inflation) and its staffing level have decreasedmore than 25 percent since fiscal 1978; other federal age...ciesprimarily involved in statistical activities were cut by lessthan 10 percent (Congresiional Research Service, 1984).Whatever the reason for this treatment, it is quite clear thatthe center and its products have been seriously affected.

The low level of funding for education statistics becomeseven more startling when examined against the pattern oftotal expenditures for education. Comparing education andhealth, both highly decentralized program areas, we find thattotal government expenditures (including federal, state, andlocal) are at similar levels, on the order of $150 billion peryear for health against $180 billion for education (Bureau ofEconomic Affairs, I984:Tables 3.15-3.17). But expendituresfor federal statistical programs show enormons divarity:obligations for health statistics are estimated to be 5176.5million; for fiscal 1986, those for education statistics areestimated to be $18.9 million (U.S. Office of Management andBudget. 1985). In relative terms, then, the ratio of totalgovernment health expenditures to education expenditures isclose to 1 to 1, while the ratio of expenditures on health

31

22

statistics to those on education statistics is more than 9to 1.

This extreme difference certainly raises some questionsabout the understanding of the need for and the commitmentto high-quality education statistics, especially in light of thecurrent public concern about educational quality. To citeanother example, several years ago when the energy crisiswas a major public issue, more than $100 million was pro-vided to the Energy Information Administration (EIA) toremedy the lack of energy statistics. Even in today'srestrictive budget climate, the EIA budget for fiscal 1986 isapproximately $63 million, or more than four times that foreducation statistics. It seems clear that, whatever thereason, the center has been seriously underfinanced.

It would be quite simple if all the difficulties of thecenter could be blamed only on a lack of money. Unfortu-nately, as we have shown, the situation is otherwise. Thecenter continues to face three more bas.ic problems:

o it lacks a clearly defined mission and role supported byagency leadership;

o there is no agreement on an explicit framework as towhat data should be collected and how they should beobtained or on the establishment of priorities withinavailable choices to ensure that the center is success-ful in carrying out its mission; and

o it lacks the technical capability and resources to do itsjob.

The panel's reactions to the first two problems are addressedin Chapter 2; the iast concern is dealt with in Chapter 3.

32

2

Mission, Role, and Frame of Reference

A continuing thread throughout the panel's review has beenthe consistent lack of support for the center or, expressedanother way, its seeming isolation from participation andin,rolvement at the policy-making levels of the department.Although not a solution in itself, the center will benefitthrough having its role detailed more clearly in its mandatedcharter and, in turn, Congress and the Secretary of Educa-tion can and must contribute to the growth of the center asa respected, impartial voice on the condition of education inthe United States. To this end, our initial recommendationsin this chapter deal with the role, mission, and responsibili-ties of the center; the second set of recommendations con-cern data collection strategy and, in a general way, datasubstance; our final recommendations concern the center'sframe of reference.

The panel's recommendations establish a role for thecenter and recognize its importance in reaching out to policymakers and to its user community and involving them in dia-logue on data needs. The center's recent efforts in thisregard in connection with the redesign of the elementary/secondary data collection system are to be applauded, butthis process must become an inherent part of the center'sactivities and not an occasional activity resulting from theconcerns and thus the pressures of concerned data users. Inthis connection, we provide some general guidance in termsof needed program content and program direction thatdeparts from the state-oriented collection of administrativedata and points to the need for sample-based national datathat are essential for understanding the functioning of thenation's educational system.

23

33

24

MISSION, ROLE, AND RESPONSIBILITIES

It is essential that support for the center and its indepen-dence be clearly articulated by both Congress and the rele-vant departmental officials and that the appropriate actionsbe taken to demonstrate the full extent of the support. His-tory shows that the center has seldom, if ever, shared itsknowledge and expertise, even in the determination of itsresource needs, either with the Office of Management andBudget or with the appropriations subcommittees of Congress;the center invariably has been represented by other entities.Thus, its needs and their justification have been presented byothers, not the optimum way for the center to gain respectand credibility. In the panel's view, the department mustprovide the center with the opportunities to be seen and tobe heard, to have the opportunity to respond directly andopenly to challenges and to criticism, to be held responsiblefor what it says and what it does, to understand throughdirect colloquy what is expected of it, and to understand whyand how its requests have been disposed of. It is alsoessential that the center be viewed as objective, unbiased,and nonpartisan in carrying out its mission.

Even beyond support, the center requires recognition ofits role in participating in the determination of data needsand of its capabilities in determining if and how the needscan best be met. Furthermore, the center must be recog-nized as the key source for compiling and issuing data,including the release of key indicators and especially forassessing progress in education among the states. Given thehistory of the center, the panel believes that its positionwould be strengthened by broadening and detailing its man-date and by positive actions on the part of Congress. Wepropose several first essential steps in the process of refo-cusing the center.

o The panel recommends that the mission of the Centerfor Statistics, as stated in Section 406(b) of theGeneral Education Provisions Act, as amended (20U.S.C. 1221e-1), be strengthened to clearly establishand define the role of the center in assisting the Sec-retary of Education in determining the data needs forassessing the condition of education in the UnitedStates, and for accepting responsibility and account-bility for ensuring the availability of the necessarydata.

34

25

o The panel recommends that Congress demonstrate itssupport for the center and its mission through its bud-get actions, through its requests for informed statis-tical advice, through requests for direct testimony fromcenter officials, and through its calls for educationdata.

o The panel recommends that the Secretary of Educationdemonstrate strong and continued support for the cen-ter and its mission through department budget requests,public statements, and in appearances before Congress.

o The panel recommends that Congress and the secretaryrecognize and state their support both for the nonpar-tisan nature of the center and for its statistical inde-pendence.

o The panel recommends that the secretary designate thecenter as the functional agency responsible for coor-dination and technical review of all data collectionwithin the department.

o The panel recommends that the secretary designate thecenter as the focal point of releasing statistical infor-mation on education.

o The panel recommends that the center assume account-ability for the process that leads to the dete:minationof the content of the data to be collected.

CONTENT AND DIRECTIONOF DATA COLLECTION PROGRAMS

Given the ongoing debate on education, it is quite clear thatthe center needs to make substantive changes in programsand in overall strategy. We beLeve the following recommen-dations are both basic and essential to providing the under-pinning for the more difficult steps to follow.

o The panel recommends that the leadership of the cen-ter develop mechanisms to assist policy makers indetermining their data needs and that the center con-tinue to involve its users in dialogue to determine themost relevant and appropriav; content of its dlta col-lection efforts.

35

26

o The panel recommends that the center continue thecompilation of program, staiT, and financial data fromthe states, but that it undertake an analysis, jointlywith representatives from the states, to ensure thatthe present program meets both center and staterequirements for usefulness, relevance, quality, andreliability. In addition, the content of the data on thecondition of education, originating with the states,should be monitored periodically with Oe intent ofimproving the content and thus its usefulness to thccenter and the states.

o The panel recommends that the center initiate asample-based progrun of daLa collection focused onindividual classrooms and students, designed to facil-itate better undersvading of the relat,onships betweeneducational inputs, processes, and outcomes. The sam-ple would not nect2sarily be able to provide adequateestimates for all individual states, although i: shouldbe organized so that statcs could choose to augmentthe ample to provide state data.

o The panel recommends that sample-based data on edu-cational inputs, processes, and outcomes be madereadily available, on a timely basis, to all interestedusers in the form of pnblic-use tapes, with appropriatemasking of characteristics to preclude the identificationof individual schools, teachers, or students. The panelbelieves that the usefulness of center data would begreatly enhanced if its data become widely available asa resource for both policy and research use.

o The panel recommends that the center continue toexplore the inclusion of longitudinal features in itssample-based survey data and that, as a m:nimum,small-scale longitudinal studies be mounted regularly aspart of the center's data collection efforts.

ESTABLISHING A FRAME OF REFERENCE

Every organization in carrying out its mission and role oper-ates within a set of guidelines, a frame of reference againstwhich it measures and weighs what it should be doing, whatit is doing, and, finally, how well its completed tasks meetits criteria. In many cases, the framework is fully articu-lated and spelled out in directives or other pronouncements

36

27

of the organization; in other instances, the history of perfor-mance stands as the object illustration, and little if anythingis found in written, explicit form. The center appears to fallvery loosely into the latter group--the panel was unable tofind a clearly defined conceptual framework that has servedas the guide for the center. In fact, v.e would argue thatthe history of the ceLter suggests that it has operated pri-marily in a reactive mode, facing each problem or challengesomewhat independent of the one before, and that its frame-work or operating model has neither been documented nor isreadily discernible through observation or from conversationwith staff.

o The panel recommends that the center develop a con-ceptual framework for organizing its program and forsetting priorities in light of available resources.

Such a step will provide a basis for planning and a basis forreview and evaluation of problems. This section presents thepanel's views on some aspects of a framework and SOT%desirable charactcristics of a data colllztion system.

A Model for Data Gathering

A model for the miter's data collection activities should becharacterized by its ability to meet two major objectives thatthe panel s-es as essential to a system of educational statis-tics. First, almost all of the data collected by an educa-tional statistics agency snould reflect the interplay of educa-tional input:, extucational resources, and educational out-comes. Longitudinal data are required to trace the influenceof resources applied at one point in time to outcomesobserved at a subsequent point in time. Second, a data col-lection system must be guided by the needs of users andreflect the joint concerns of both the research communityand the policy community.

The role of the research community is to ensure thecollection and reporting of data most relevant to analysis ofeducational processes and outcomes. The involvement of theresearch community also contributes to the continuing evalu-ation of the adequacy of the data, both in coverage and instatistical quality. The policy community sets the agenda fordata collection and thus establishes the needs for data rele-vant to policy decisions (even if, at that time, the data haveno well-defined role in an existing model).

37

28

The center's data collection efforts should comprise ele-ments that can be classified along several dimensions, such asserving primarily analytic uses, in which they form an essen-tial element in an analytic model of the educational process,or as serving primarily descriptive uses, in which they informusers about the size or distiibution of a particular character-istic of the educational system. Data elements can describemicro-environments, such as activities in a particular schoolor classroom, or they can describe macro-environments, suchas all schools or classrooms along a particular dimension.Data can be cross-sectional, such as describing the state of aparticular education variable at a specific time, or longitudin-al, describing a process of change that takes place over agiven time period. Finally, data can relate to any of fivelevels of the educational system--preschool, primary or ele-mentary school, secondary school, postsecondary school, andeducation at work.

As we have noted earlier, a data system must beperceived as an evolving collection of measures in whichdesign of the system not only reflects existing or past inier-ests and needs of the education community, but also attemptsto anticipate future interests. As understanding of relation-ships among inputs, processes, and outcomes changes withnew research knowledge, and as policy needs change, mea-sures of central importance at one stage of development of asystem are likely to be replaced by other measures.

This process of change will be enhanced if opportunitiesfor analyzing the data are regularly made available to outsideusers as well as to center staff. The quality and utility ofdata are tested through the close scrutiny of data analystsand researchers trying to understand the effectiveness of theeducational system or the relationships between inputs andoutputs or the mechanism by which various system inputs aregenerated. In similar fashion, use of data by policy makersand the public serves as a test of both quality and utility.

Implementing a Framework

Fulfilling the center's mission--to collect and disseminatestatistics and other information related to education in theUnited States and other nations and to design a system ofstatistics that not only satisfies current information needsbut also anticipates the future information needs--requiresthat the center improve its approach toward setting prioritiesand allocating resources. In so doing, it must make clearhow what it collects augments information available on edu-

38

29

cation in the United States from other sources and how, col-lectively, the available information fits within a structurethat also makes clear what information is not being collected.

One might begin this process by exploring the issue ofthe outcomes of education. For example, answers might besought to such questions as the extent to which studentslearned what they were taught; the extent to which studentswere prepared for the next steps--further education or work;and the extent to which students have accumulated skillsthat permit them to function as adults. Currently, criticismis heaped on the meager data currently available on studentoutcomes, whether they be achievement rates, reading accom-plishment, dropout measures, or conclusions on labor-markettransition success--all are considered inadequate. Taking thelead in identifying and seeking agreement on appropriate out-come measures and devoting imaginative and innovativethinking to possible data collection programs would certainlybe a forward-looking and much-needed step for the center.

Ideally, the center should provide information that allowsexamination of the differences among students and schoolsystems and other potential influences on outcomes. Theseinfluences include the available resources--funding, qualifica-tions of teachers and administrators, curriculum quality,characteristics of the setting in which education takes place(e.g., policies, organizational structure, physical facilities),and context--characteristics of the students and their fami-lies, (e.g., cognitive and social skills, and public attitudes.)For each potential influence, the statistics system must tryto describe the parameters of the potential influences (e.g.,the knowledge, skills, and dispositions of teachers) as well asto model the origins of those characteristics (e.g., studies ofteacher education, recruitment, and selection). Because eachof these influences is complex and represents a cost, thecenter must be responsible for establishing the priorities todetermine the area to be explored and developed, for deter-mining the information to be coliected, and for providing thelogic behind the choices. Although this brief discussion ofeducational outcomes and of potential influences is writtenlargely from the perspective of describing the education thatis provided in schools and colleges, the general frameworkalso applies to preschool education, both formal and informal,and to education provided in the work environment. Ideally,a system of educational statistics would also be capable ofdescribing transitions from education at one level to educa-tion in another. Key points of transition are from preschoolto elementary school, from elementary school to middleschool, from mijdle school to high school, from high school

39

30

to postsecondary education or to work, and from postsecon-dary education to work. These points of transition raisequestions concerning how well the educational outcomes fromone level serve as appropriate inputs for productive educa-tional experiences in another level. They also raise questionsabout the continuity of learning experiences and the effectsof continuity or discontinuity on educational outcomes.

Because the mandate for education in the United States isbroad, because the U.S. educational system is complex, andbecause the influences on educational outcomes are so littleunderstood, an ideal system of education statistics goes wellbeyond the resources now available to the center. It isespecially important, therefore, that the center design andmaintain its system of education statistics in a public andopen forum, giving full access to users, policy makers, andresearchers. The center is to be commended for havingrecently moved in this direction by involving a wide varietyof constituencies in assessing the nature of its current workand in soliciting suggestions for future work. At the sametime, it must be recognized that a system of education sta-tistics that supports descriptions of trends over time isabsolutely essential. Thus, responsiveness to current needamust be tempered with the requirement that a core set ofdata be maithained over time.

4 0

3

Designing a Program

Given our view that the Center for Statistics requires a fun-damental change in its methods of operation, we stress onceagain the need to view our recommendations as a unifiedprogram. It is true, particularly in the areas covered in thischapter, that individual recommendations address what appearto be unique problems, but we believe the necessary anddesired transformation of the center will occur only if all ofthe panel's recommendations are considered as a whole. Theappropriate analogy is that of a complex machine that hasmany interdependent parts: fixing what appears to be onedeficient part does not result in a workable machine; further-more, fixing or replacing parts one at a time is both ineffi-cient and counterproductive, since action (or its lack) in onearea directly affects the progress in others.

Although we recognize that some additional funding willbe required to fully implement our recommendations, we havenot proposed a solution that rests solely on immediate, largeinfusions into the center. In the short run, the center willrequire modest additional support for the staff needed just tocarry out its present program; however, we believe it unrea-sonable (as well as unwise) for the center either to requestor to expect further support in order to accomplish the pan-el's recommendations without first having demonstrated acommitment and a strong beginning towards improvement.Although difficult choices may be required of the center, itis our view that much can be done within its existing budgetand staffing level.

To this end, as needed, the panel strongly urges that thecenter plan and be prepared to divert resources or curtailexisting programs or their scope in order to implement andaccorip1ish the panel's recommendations. Given the widerange of possible actions and choices before the center, thepanel does not believe it is the appropriate body to sugge:st

31

41

32

either the specific programs to bc curtailed or whichresources might be diverted; rather the panel emphasizes theneed for the center to develop a mechanism to ensure thatpriorities and choices are openly discussed and chosen andmade known to users and other interested parties.

This chapter is concerned with the realities of carryingout the program of the Center for Statistics. We begin byreviewing the role of the Advisory Council on Education Sta-tistics, and then provide guidelines and recommendations forimproving each of the areas involved in the production ofdata, including staffing and budget, quality control, design,collection, processing, and publication.

THE ADVISORY COUNCIL

The Advisory Council on Education Statistics was establishedby statute in 1974, and the initial members were appointed inmid-1975. It meets a minimum of four times a year, and itsmembers serve three-year terms. The Assistant Secretary forEducational Research and Improvement serves as the presid-ing officer. In that the ACES charter directs it to "reviewgeneral policies for thc operation of the Center," it followsthe practice of virtually all advisory groups--namely, toadvise. What distinguishes ACES and sets it somewhat apartfrom other groups is the part of its charter that states thatthe Council ". . . shall be responsible for establishing stan-dards to insure that statistics and analyses disseminated bythe center are of high quality and are not subject to politi-cal influence."

The usefulness of any advisory group, of course, isdependent on a number of different factors, including thestature of its members and the extent of their competence inthe topics under review; the types of information providedthe group, the rroblems brought to it, and the cooperation itreceives; the interest, concern, and responsiveness of thoseto whom the group provides its advice and counsel; and thedegree of understanding and reaction on the part of the topmanagement in the activities and actions of the group.Overriding these factors, of course, is the fact that any suchgroup has neither operating nor policy authority.

The panel's review reveals that ACES has had a checkeredpast. In its early incarnation, its members indeed were tech-nically oriented and extremely competent to review operatingpolicies and comment on how the center carried out its mis-sion. Later, the membership changed to reflect a broadvariety of experience, but not in the collection of education

42

33

data or in topics necessarily related to the concerns of thccenter. As such, ACES appears to have become considerablyless relevant and useful to the center over timc. Moreover,ACES does not appear to have received any reaction to theconcerns that it has raised in its reports to the secretary orto the Congress. Furthermore, for a variety of reasons,including the budget problems already noted, there appears tohave been little or no reaction on the part of the centeritself to the concerns raised by ACES. This lack of in.erestalso appears to be fostered by having an assistant secretary,rather than the administrator of the center, sit as thepresiding officer.

With regard to the mandate to establish standards, ACES,in its first annual report prepared in March 1976, expressedits support for the addition of a senior mathematical statisti-cian to the staff and indicated it would return to discussionof the issue at future meetings. Successive reports dealtwith one or another aspect of this issue and, as noted ear-lier, it was ACES that initiated the present study because ofmany concerns, including its perception of the coatinuinglack of standards within the center. As the panel was com-pleting this report in mid-1986, it is informative to note thatfor the previous year or so the center had bcen seeking (buthad not yet found) a principal statistician to assume respon-sibility, among other areas, for "the setting of standards anddeveloping ways to ensure they are known and observed sys-tematically in the Center" (letter from the di, ctor of thecenter).

With a goal of once again ensuring the presence of arange of technical competence, the present leadership ofthe department replaced the membership of ACES. It isobvious that if ACES is to be of any value to the center, itmust consistently represent technical competence, both opera-tional and substantive, and the panel urges that this be thekey criterion for member selection. Nevertheless, it is theview of the panel that ACES alone cannot provide the widerange of technical advice that the center seeks and requires.Furthermore, we strongly believe that it is wholly inappro-priate for ACES to be 1..eld responsible for establishing stan-dards, a mandate that it never was able to implement. Aswe noted earlier, an advisory group can advise and providean oversight function, but given ACES's lack of authority andintermittent operation with no staff, it should not beexpected to establish standards. Responsibility for establish-ing and implementing standards--and ensuring that its prod-ucts are of high quality and not subject to political influ-

43

34

ence--must and can only be the inhcrent responsibility of anoperating entity, in this case, the centcr.

o The panel recommends that Congress modify themandate of the Advisory Council on Education Statis-tics to a rcle of technical review and oversight, ratherthan responsibility for the establishing of statisticalstandards.

o The panel recommends that the Secretary of Education,in support of the center, provide timely writtenresponse to the concerns raised by ACES in its a nnualreports to the secretary, and that Congress periodicallyhold hearings at which both the secretary and repre-sentatives of ACES provide testimony on the status,problem3, and prosress of the center and the concernsand advice of ACES.

o The panel recommends that rnembcrs of ACES beselectea for their expertise and competence in areasdirectly related to the center% program.

o The panel recommends that the center establish one ormore technical advisory groups, in addition to ACES,to meet at least twice per year to review, comment,and advise on the methodology used in the conduct ofits work, and to provide the director with objective,outside, independent review of the technical capabilityof the center's staff or of its products. As requiredby its changing program needs, the center also shouldestablish ad hoc advisory groups representing uniquetechnical and subject area skills.

IMPROVING DATA QUALITY

Statistical Standards

The essential underpinning of data quality is the establish-ment and implementation of standards. Standards are thenorms of the research community as to what constitutes ade-quate practice, and they provide mechanisms for the publicexposure and censure of deficient practice. As noted earlier,ACES, by law, has the responsibility for establishing thestandards which both guide the center's work and ensure thatits products are free from political influence. In fact, thisresponsibility can only be carried out by the center.

44

35

Our earlier exposition of the many different pfaced by the center also serves to detail what themust do to improve the quality of its data and its pWith regard to standards, we note that they have rcand meaning only when they are required and expestaff as a normal part of any task. Staff must uncthe need for standards and must be technically compimplementing them. Data quality is the product ofment, competence, and pride. The center should nnot--accept less from its staff. It should also interathose who provide its data to ensure that they, iunderstand what is required of them and why, and itassist the states and local school districts in carrytheir functions in a satisfactory and timely manner.

The panel notes that certain of the center's prcfor example, the longitudinal studies--are currentlyaized by users as providing high-quality, valuable, ancneeded data. In some cases, the products have been pby staff; in other cases, the center has been very su(in and is benefiting from the selection of contractorsreputation, operating philosophy, and accomplishmentsdemonstrate high standards. Simply put, the center trequires more of each.

o The panel recommends that the center develop, Idisseminate, and implement standards to guide tduct of all phases of its work, from developnobjectives through collection, follow-up, and prcand including the preparation, review, analyspublication of results.

o The panel recommends that the center establOffice of Statistical Standards and Methods amexpeditiously to recruit and appoint a chief statat the level of assistant director, with responfor the establishment and maintenance of stastandards throughout the center.

o The panel recommends that the center, in conceappropriate state and local education agenciesand representatives of institutions of higher eduinstitute the development and publication of udefinitions and their continuing review, to enstcollection of consistent data, such as between diLEAs and states. Such information should beavailable regularly in reference handbooks.

45

36

o The panel recommends that the center undertake acontinuing program of evaluation of its methodologyand its programs.

o The panel recommends that the center disseminatewidely the results of its methodological research. Pub-lications in professional journals and presentations atconferences should be strongly encouraged.

o The panel recommends that the center require that itsreports contain information on definitions used. in stud-ies and a description of the data collection methodol-ogy, provide a measure of reliability, if applicable, anddiscuss possible sources of error.

Redesign of the Elementary/Secondary Data Program:An Example

Quality is a goal of all statistical organizations; recognizingand measuring it are the tools by which the goal is accom-plished. The approach is called quite simply "quality con-trol," and error is the villain. The many possible sources oferror in statistical data and what can be done to expose,measure, and control them are well known and expounded onin textbooks on quality assurance. It is important to recog-nize that errors can arise from many sources at every stagein the collection, processing, and analysis of data and, fur-thermore, that some error is unavoidable (such as samplingerror when conducting a sample survey, or recording errorwhen using administrative records as the source of data).When designing a system to provide statistical dau, all thepoteutial errors need to be considered as well as the costsinvolved in reducing them.

The current effort to redesign th;:, elementary/secondarydata system provides a clear example of how these and manyother issues surface and must be dealt with when fundamen-tal changes r,re proposed. For example, should the commoncore of data be collected as at present--that is, tluough acomplete enumeration of all school districts using, for themost part, administrative records--or should the data or somevariant be obtained through a sample survey approach? Onone hand, a sample survey approach--in which a sample ofLEAs would provide a sample of schools and, in turn, a sam-ple of pupils, teachers, and officials in these same schools--as proposed to the center by a panel )f consultants (Hall etal., 1985)--is seen as providing more timely and more accu-

46

37

rate data. Proponents argue that it would permit greatercontrol of the entire data collection process, provide foranalyses of relationships within and across major areas of theeducation system, and result in reduced burden on respon-dents and that such a system also would provide longitudinalcapabilities to relate outcomes to earlier experiences. Eventhis minimal list of potential benefits seems to lead to anobvious conclusion. On the other hand, the costs of imple-menting such a system would be great. Critics note that thebasis for assuming the data would be either more timely oraccurate is not firmly established and that the problems ofintegrating or linking the diverse elements would be manyand dif f icult.

Although the current system raises serious concerns aboutthe presenze of large response and other types of nonsam-piing errors, the Common Core of Data program is institu-tionalized and has been in operation for a number of years.It has been adapted by the individual states to provide formany of their continuing needs for data, as well as beingfinancially supported virtually in full by the states. In thewords of the National Education Association (1985:738):

The Core represents the most basic data seri.es withinthe NCES. It enables assessments of what was, whatis, and what will be in a statistical sense. Annualupdates to Core surveys provide basic statistical infor-mation on public schools, their pupils, personnel, andfinances . . . the Common Core may represent the mostheavily used series of public school statistics. TheCore is the cornerstone of educational information inthe United States. No other public or private institu-tion collects and maintains public education data to theextent that NCES does via the Core.

The choice of design also is often complicated by themultipurpose nature of statistical datathe preferred designfor one purpose may not suit the needs of another. Thus,were all other factors equal, collecting common core data ona sample basis with controlled data collection almost certain-ly would be the preferred method for national and regionalestimates; however, this method may not adequately serve theneed for local area estimates. And if local school officialsare unwilling to support the desired approach, that fact mayoutweigh all other factors.

On balance, however, the panel favors and supports amove to collect elementary and secondary data through asurvey-based approach as a long-range strategy. Our under-

4 7

38

standing of the present character of the center's relationshipwith its data suppliers and the center's limited experience incarrying out such a survey lead us to conclude that abruptand wholesale replacement of the existing system would entailan unwise and unnecessary risk. The center must not makethe mistake of moving precipitously without undertaking thenecessary testing and development, wiLhout first demonstra-ting to the states and local areas, as well as to itself, how itwill deal with the issues that have been raised, what thecosts of the program will be, and how and why a surveyapproach will produce consistent, accurate, and reliable data.The center also has a responsibility to the states to ensurethat there is agreement on what their respective responsibil-ities will be under the new program and how the uniqueneeds of different states will be met and supported. Thus,the panel supports serious study, development, and testing ofan integrated system based on sample surveys, but urges cau-tion, judgment, open discussion, and continuous review w:thall parties prior to the adoption of a new system. Above all,it urges that there be adequate testing to confirm the feasi-bility, probable cost, and the center's ability to accomplishthe objectives of the program. In fact, the center may wishto overlap some survey-based components with the presentCCD program as a learning experience, to demonstrate itsfeasibility, and to provide comparative data.

o The panel recommends that the center continue toexplore most carefully the appropriate balance betweenthe continuing use of state and local administrativerecord sources and the use of sample surveys.Although supportive of the use of sample surveys, thepanel does not believe that the center has fullyresolved the issues involved in shifting the elementary/secondary data collection system from one based onadministrative records to one based solely on an inte-grated sampling survey approach. The panel endorsesthe efforts of the center in seeking new approaches toaccomplishing its mission and suggests that small-scaletesting within one or a few states be undertaken toestablish procedures, define problems and develop solu-tions, refine the approach and, overall, assess thefeasibility and cost of the survey proposal.

48

39

Detecting and Reducing Error

Another major reason for needing information on the magni-tude of errors from different sources is in order to be ableto provide data users with an assessment of the quality ofthe statistics produced. Sophisticated users will demandinformation on quality, but unsophisticated users may not.However, even without specific demands, the center has theresponsibility not only to produce education statistics butalso to inform its data users of the limitations of those sta-tistics.

o The panel recommends that the center routinely makeits users aware of the quality of the data and thelimitations inherent in their use.

The nature of error, its many sources, and its measure-ment need to be understood as a proper basis for decisionmaking with regard to the best way to collect, process, andpublish data. It is only realistic to accept the inevitabilityof some error; at the same time, much of the error in a datacollection process can be reduced by an effective and willingstatistical agency.

The fundamental requirements, of course, are a generalinquisitiveness on the part of the staff toward the entireprocess and toward the numbers collected, a distrust of theinformation conveyed by a sequence of digits, and an ever-present realization of the ease with which error can beintroduced in data. Together, these factors combine tocreate an atmosphere in which efforts to minimize errorbecome basic to the planning and execution of the data col-lection process and, later, in which the data as collected areexamined for enor and appropriate actions taken.

There are ri variety of opportunities to exercise an errordetection and corrcction process. The use of appropriatecontrol procedures providei the ability to monitcr the opera-tion on a timely brsis: sample checks made throughout thedata collection prr,cess ensure that the operations are beingconducted as specified; internal consistency checks are donewhen the data are obtained; for data that are collected peri-odically from the same units (e.g., schools or universities),present and past responses are compared to identify anoma-lies. Similarly, independent checks against record sourcesserve as a valuable quality control measure. Reviews by sub-ject-matter specialists point out analyses that compare statis-tics that are not directly comparable, noting the need foradditional variables in regression models.

49

40

All of these quality control measures, if they are to besuccessful, require that they be fully documented--both theprocedures and the results must be available on a timelybasis during the process. In addition, historical records mustbe maintained and readily available to guide in the planningfor new projects with similar characteristics, to assess theeffectiveness and efficiency of the quality control mechanismover time, as a teaching device, to correct faulty memoriesand, above all, to be exposed to critks and supporters alike,for comment and suggestion.

Particular care is needed in checking final estimates priorto publication. Errors made in the analysis and in the prep-aration of estimates for publication are potentially the mostdamaging. Results can be compared with those producedfrom other sources to see whcther they are reasonably con-sistent; if not, an explanation of the discrepancy can besought. Checks on data that are part of a time series canbe made by examining the reasonableness of the changes thathave occurred since the previous data collection. Errors alsoare less likely to be introduced into a publication if thoseresponsible for its preparation are knowledgeable about itssubject mattcr and thus able to assess the reasonableness ofthe data as they complete the analyses. At the most basiclevel, users expect that published data have been checked toensure that detail adds to totals; that figures found in thetext are those in the tables; that table titles reflect tablecontent; and that highlights reflect the data. The panel hasreviewed a number of examples of center publications thatfail to meet these criteria and finds such performance by astatistical 'agency to be unacceptable. We believe that theCenter for Statistics has a responsibility to ensui.: I:Iat itsreports are a model of statistical integrity--complete, correct,and consistent.

Our recommendations deal in part with this lack of sen-sitivity to the !ibiquitousness of error and thc responsibilityof a statistical agency and its staff to track down, diagnose,and correct as much of this error as possible before dissemi-nation.

The usefulness of the publications produced by a statis-tical agency also is affected by the piccess of the staff indiagnosing and identifying the sources of error, as well as inmeasuring the amount of error that cannot be corrected.Interaction with data users helps focus error detectionefforts on the areas that will benefit users the most. Theopenness and frequency of interaction of a statistical agencywith its data providers helps to ph omote the establishment ofstandardized definitions and data collection procedures among

50

41

the providers. Thus, the relationships of the center with itscommunities of data providers and data users become animportant and essential element in the achievement of qual-ity.

o The panel recommends that the center foster a climateof openness in dealing with all elements of its consti-tuency, including other parts of the department, othergovernment agencies, Congress, providers and users ofits data, and its professional peer groups.

o The panel recommends that the center initiate acomprehensive program to assess and improve, wherenecessary, the quality, consistency, and reliability ofdata obtained from state and local agencies, from in-stitutions of higher education, and from other sources.Meetings, discussions, visits, and technical assistance,including the training of local and state cfficials onwhat is required of them and their systems, must com-prise a major component of the program.

o The panel recommends that the center establish stan-dards for review of data releases and institute pro-cedures to effect them.

TIMELINESS

It is essential that any system of collecting education datarecognize, reflect, and react to the issue of timeliness.Timeliness has two dimensions: collecting timely data andproviding results in a timely fashion. Analysis becomes vir-tually useless if long-term gaps are allowed between collec-tion points, particularly for topics that may show largedirectional shifts in relatively short time frames. Even moreimportant, failur T. to anticipate or recognize a shift inemphasis or need inevitably results in a lack of timely andrelevant information when the policy agenda takes one of itsmany turns.

An example of such a lack is the case of teaching andteachers. With the publication of A Nation at Risk (NationalCommission on Excellence in Education, 1983), these topicsemerged as fundamental issues of concern, and the need fordata was sudden and immediate. Unfortunately, informationon the number of teachers and other professional staff--which we would think would be an essential element of anycontinuing data system--was last collected at the elementary

51

42

ard secondary level in 1979-80. Data on minority teacheeven more archaic, having last been collected in 1968.the postsecondary level, the most recent data are for 1

hardly informative or useful for the concerns of the1980s. Turnover rates among teachers provide yet ancexample, having last been collected in 1969.

Although hindsight is certainly easier than foresight, texamples clearly illustrate several key -facts. First, hadcenter's planning and decision process been guided byorganizing framework that required a balanced appnacross and description of the phases of education--injprocesses, and outcomes--at least minimum informationteachers and teaching would have been anticipated and a'able as a regular component of the center's data collecprogram. Second, an appropriate system of education sttics must strive to anticipate the needs, both of researcand the policy community. Finally, it is important thatperiodicity of data being collected be reviewed regularlensure that trends are being monitored on an appropitime continuum.

Of the need to provide data in a more timely fashthere can be no argument. Too often, the center has faitself embarrassed by an excessive and unacceptable timebetween collection and release of data. Perhaps there i$better example than the 1985 Condition of Educe(National Center for Education Statistics, 1986), which stits objective as providing "accurate and timely statistics""the center's major response to its continuing mandateto report full and complete statistics." But the vast ma joof the data presented in this report are for 1983 or earand in many cases the 1983 data are marked as "preliminor "projected." As to the references to "accurate" and "and complete," one needs only to note again the exampleChapter 1 or to refer to the Synthesis of Invited Paper:the Elementary/Secondary Education Data Redesign Pro(Silverman and Taeuber, 1985c). Some of the concerns N

highlighted in an article by Jonathan Friendly in the 1

York Times of March 11, 1986: "Last summer, for exarrresearchers asking the center for state-by-state totalsschool enrollment had to make do with 1983 figures.they were luckier than the people who needed statisticsteacher training and salaries: the center's most recent nbers for that dated to 1980 and 1981, before most ofstate measures to improve education were enacted."timeliness of the center's performance has been abysmamust move aggressively and rapidly to overcome bothproblems and its poor reputation. That the center appear

52

43

be facing up to its critics and their criticisms is to itscredit, but the proof of its intentions will lie in the extentand speed of change and the degree of commitment toimprovement, rather than in repeating the words in its statu-tory mandate.

o The panel recommends that the center explore anddevelop approaches for expediting response from par-ticipants in its studies and techniques for producingusable advance results; use management informationsystems for monitoring progress through all phases ofdata collection and reporting; and establish proceduresfor taking prompt and appropriate actions whenrequired. Similar techniques and monitoring controlsshould be required as integral components of workdone by contractors to the center and should be fol-lowed closely to ensure prompt adherence to schedule,with penalties for failure to accomplish work on time.

RESOURCES--STAFF AND BUDGET

The presence of competent, experienced, and dedicated staff,with the required mix of disciplines and expertise, is basic tothe success of the center. Given its mission and responsibil-ities, its needs range across a broad spectrum of talentsrequired in collecting and producing education statistics,ranging from organizational leadership and management skillsto clerical and semiprofessional support, and at a rr.inimum,the expertise necessary to collect, process, analyze, andpresent statistics, whether collected from existing records orobtained directly through surveys or censuses. Furthermore,the staff must have the capacity, capability, and interest toprepare and publish regular analytic reports based on thedata. Such efforts develop both a substantive and technicalcompetence and insight on the part of the staff, whichresult, first, in better data, and, second, in better under-standing of the meaning of the data. On a continuing basis,efforts must also be devoted to evaluation and review of thecenter's methodology and products as one step in exploringand developing future methodology.

Such a diverse menu of needs calls for survey and math-ematical statisticians or others with similar expertise, forstaff trained in the subject of education and education policyand with analytic skills, and for staff experienced in theneeds and practices of states, local education agencies, andother data providers. Competent leadership must set realistic

53

44

boundaries of what can be accomplished to protect theorganization from unrealistic demands. Once having achievedand demonstrated an acceptable level of competence andaccomplishment, an organization can expand and stretch asnew needs are encountered. As needed, it also can seekshort- or long-term outside assistance. In either case, how-ever, it is essential that what is to be done be done well andwithin the capabilities, knowledge, and experience of avail-able staff resources.

The panel wishes to emphasize most forcefully its beliefthat the center staff itself must include sufficient expertiseand the required leadership capability to understand, address,and respond to both the technical and related requirementsof the demands placed upon it if it is to be perceived as(and in fact is to be) a competent statistical organization.Although it may require outside assistance to meet all of itscommitments, it cannot, should not, and must not turn toothers for its basic needs.

Given the long history of education statistics and thealmost universal agreement on their importance, our exposi-tion of necessary skills, related requirements, and essentialattributes for a responsible statistical organization could beviewed as both naive and belaboring the obvious. Unfortu-nately, such is not the case--the situation at the centerleaves much to be desired. During a period of risingdemands for more and better information, staffing and bud-gets have declined significantly. Between fiscal years 1980and 1984, the number of full-time equivalent employees in thecenter decreased from 173 to 134, a 23 percent decrease(Congressional Research Service, 1985) and the number inearly 1986 has dropped to 111. Adding insult to injury, areport issued in 1984 by the Committee on Government Oper-ations noted that during this same period of overall decline"the Department has required the Agency o fill 14 positions. .. with assigned staff because of RIFs (reductions in force)and reorganizations (elsewhere) within the Department" (Con-gressional Research Service, 1984:172).

That the problem of adequate and competent staff is notof recent vintage is illustrated by a rather succinct quotefrom the report of the 1963 advisory group (U.S. Departmentof Health, Education, and Welfare, 1963:14): "The Office ofEducation [the predecessor organization to the center] doesnot have adequate professional staffing. . . ." Of somewhatmore recent vintage, it is instructive to note that in a mem-orandum to division directors in mid-1985, the director notedthat efforts to improve quality at the center will not succeed

54

45

unless existing staff attitudes that "it doesn't make muchdifference what I do" are countered and overcome.

During its study, the panel heard not only of the scarcityof staff, but also of the scarcity of skillq. Serious questionswere raised as to technical competence, particularly in theareas of sampling and survey methodology and of evaluationof the adequacy of products. As an example of this concern,of a staff of 128 permanent full-time members in March 1985,only 11 were classified as mathematical statisticians, andnone was located in the office of the director. The panelwas also presented with a number of examples of inadequateor incorrect presentation and interpretation of data by centerstaff. Furthermore, the center was unable to document anyrecent instances in which staff had preparee and presentedtechnical papers before, or had participated in, meetings ofprofessional associations, such as the Americ:n EducationalResearch Association, the American Statistical Association,the American Sociological Association, or the PopulationAssociation of America.

We note the recent establishment by the American Sta-tistical Association of a research fellowship and associateshipprogram, in cooperation with the center, which stands as avery positive development for the center. The program, sup-ported by a grant from the National Science Foundation, willbring to the center for periods of a year or so seniorresearchers and advanced graduate students to pursueresearch on any area related to the center's data or method-ology.

The inability or unwillingness to respond to the existenceof vacancies, particularly when the agency employment levelis consistently below its ceiling and when its programobviously is in a state of growth, has been demoralizing anddestructive. For example, an interval of about nine monthsto fill a mid-level statistical position (GS-11) raises doubts asto the commitment of the department to the objectives andneeds of the center. Similarly, a personnel ceiling of 129compared with an on-board staff of 1 1 1 (of whom some 90are classified as professionals) at the time that the Congresshas before it a request for a 50 percent budget increase(spring 1986) and that work has already begun on a new,major reimbursable project for another office in the depart-ment must also provide some pause.

The recent reports prepared for the Committee onGovernment Operations of the House of Representatives(Congressional Research Service, 1984, 1985) also document adecrease of 28 percent in budget (when adjusked for infla-tion) over the five-year period beginning 1980, which resulted

46

in sharp curtailments in the ability of the biaff to visitstates to coordinate efforts, review activities, or otherwiseinteract with providers and users of its data. The effect ofthe budget and staffing curtailments also has had a directnegative effect on programs. The center's publicationsshifted away from more detailed analytic report to more lim-ited bulletin or statistical summaries, and even more discour-aging has been the absence in recent years of any efforts invalidating any of the center's surveys or programs.

These examples inevitably raise the question of leadership.The absence of leadership was mentioned by many whoshared their views with the panel. Therefore, it seemsobvious that a first requirement of any attempt to deal withthe center's problems is for the leaders of the center todemonstrate their capabilities and regain the respect and sup-port, as well as control, of the staff. It also is essentialthat the lines of responsibility and authority throughout thecenter be clearly delineated, followed, and supported, andthat leaders be held accountable and responsible for theiractions or lack thereof.

Recent changes in leadership, both at the center (in late1984) and in the department (in early 1985) appear to beresulting in some positive changes both in attitudes towardthe center and in increased resources. The new leadersappear particularly realistic in their outlook: in the words ofthe present Assistant Secretary, Office of EducationalResearch and Improvement, thus far, the center "is not doingenough and not doing it well enough." Both the assistantsecretary and the director recognize that a large task liesahead to "repair errors of the past and nurture for thefuture;" they have articulated three goals for the center:

to repair the data base;

to add qualitative information on the quantitative base;and

to ensure that the center's products have both utility andquality.

The work that is done by the statistical agency chieflyresponsible for understanding the status of the U.S. educa-tional system requires stable and competent leadership andstaff that must have substantial quantitative and subject-cmtter expertise.

56

47

o The panel recommends that the center identify theprofessional and technical staff required to successfullyaccomplish its mission and initiate an independentreview and assessment of the technical and subject-matter qualifications of the present staff.

o The panel recommends that the center initiate anactive and continuing recruiting program to obtainstaff with the needed skills that are lacking among thepresent staff.

o The panel recommends that the center devote resourcesfor the purposes of developing innovative approaches,including training and work assignments, to ensure thecontinuing technical growth and competence of thestaff, and that .itaff accomplishment be rewarded byopportunity for further professional development.

o The panel recommends that the lines of responsibilityand authority throughout the center be clearly delin-eated.

o The panel recommends that the center, as needed,actively solicit the assistance and cooperation of itssister federal statistical agencies in addressing tech-nical issues or problems, to obtain short-run staffingassistance, or in carrying out various facets of its pro-gram.

Given the extreme importance the panel places on thedevelopment of competent staff--and, in fact, in carrying outits other recommendations for improvements in the center'sprogram--the panel fully expects that the center will curtailits program as may be necessary to free the resources neces-sary to improve its current operations.

To institutionalize the central place of quality in the workof the center, it is vital that the issue of statistical qualityexplicitly be made a central component of the center, with afocal point in an Office of Statistical Standards, as recom-mended above by the panel.

In addition to technical competence, many factors enterinto the perception and reality of an organization recognizedas outstanding, one responsive to demands and innovative inmeeting needs. We have already highlighted the importanceof leadership; without vision, direction, and commitment, noorganization can long sustain excellence. At a somewhatlower level--but vital nonetheless--are a host of activities or

5 7

48

requirements that set the tone, sustain the momentum, andprovide the basis for continual growth and improvement.These include staff recruitment, training, and development;the conscious fostering of communication within the organi-zation and between the organization and its users and pro-viders; coordination of efforts among all the participants;exposure of the organization to outside review and critiqueof its procedures, methodologies, and products; efforts atconsultation, both to provide others with the benefits of thecenter's expertise and to obtain from others the fruits oftheir wisdom and experience; and finally, the product of allthe above, a sense of morale, that elusive element that bindsan organization and makes possible continued accomplishment.

o The panel recommends that the center staff be givenboth responsibility and authority for the conduct ofindividual projects and, accordingly, be held account-able for meeting the established standards ofacceptable performance, including timeliness.

o The panel recommends that the center recognize thevalue and importance of participating in meetings andactivities of professional and technical organizations,including the preparation, presentation, and publicationof papers and articles describing aspects of the workof the center and its problems, and that the centerdevote, and if necessary divert, resources to ensureand support staff participation.

CONTRACTING OUT--OR NOT

Federal government agencies--statistical and nonstatistical--have a range of options in arranging for the performance oftheir statistical activities. The first level of choice iswhether to perform the project with in-house staff or to usethe services of an outside organiration. If an outside organi-zation, choices include another federal statistical agency, aprivate survey firm, a university survey center, a nonprofitresearch institute, a public-interest association, or a unit ofstate or local government.

Major federal statistical agencies exhibit many differentconfigurations in the arrangements by which they performtheir mainstream activities. The Bureau of the Census, forexample, serves as a survey data collection and processingagent for many federal statistical and nonstatistical agencies.In contrast, the Center for Statistics has relied heav:ly on

. 58

49

statc and local education agencies for the primary collectionof elementary and secondary school data and has contractedout most of its other data collection activities, including suchmajor longitudinal surveys such as High School and Beyond.In addition, it has contracted out for some processing of datareceived from state agencies and from institutions of highereducation. Over all, very little of its work has been carriedout directly by its own staff.

The question of the proper role for outside contract ser-vices in the performance of a statistical center's primary orcore activities is a very difficult one. In the case of thecenter, a basic problem is that a professional staff the sizeof the center's (about 90 in spring 1986) cannot be expectedto develop and maintain an appropriate level of expertiseacross all of the technical and subject-matter fields that arerequired in carrying out its mission and program.

The general adequacy and quality of the survey work andother collection and processing activities carried out by thecenter depend heavily on its operational capacity to plan,manage, and evaluate technical projects. Similarly, theability to carry out supervisory or overview functions to themaximum benefit of the center depends heavily on the exper-ience of the staff in having participated directly in one oranother of the many facets of direct data collection and cro-cessing. Only with such experience can one be expected torecognize fully, understand the meaning of, and react to demeasures of quality control, the implications of actions takenor deferred, the significance of patterns of expenditures, andthe proper choices among alternatives when plan and perfor-mance differ.

o The panel recommends that the center undertake toconduct some projects directly in-house, in order toprovide the opportunity for staff to become exposed toand experienced in the variety and multitude of tech-nical activities, problems, and difficulties inherent insuch an undertaking and to provide tec,inical compe-tence for the staff in subsequently supervising outsidecontractors.

With regard to contracting with private organizations, thecenter has had a range of experiences. The longitudinalstudies have received generally favorable reviews fromknowledgeable observers. In contrast, the center has hadsome unpleasant experiences in the nonperformance or inade-quate performance of support contractors in the area ofoperational processing and checking of data from educational

59

50

institutions. Although it is not possible to guarantee impec-cable, continuing performance on the part of any contractor,it is a major responsibility of center management to recog-nize early warnings of problem areas and to take immediateand decisive action to correct them. From the oppositeviewpoint, contractors have indicated their share of difficul-ties in dealings with the center. Qualified technical staff toevaluate and review the contractor's work have been andcontinue to be in short supply, with consequent delays to theprogram. Specifications once agreed upoL have been changedwithout notice or much discussion, needed decisions havebeen delayed, the center has spoken often with many differ-ent voices--all adding to confusion, loss of time, and lessthan efficient performance. In each instance, the availabilityof knowledgeable, competent, and experienced staff is anessential element, first, in anticipating and preventing prob-lems and, second, in dealing with them expeditiously andefficiently if and when they occur. As noted earlier, ensur-ing data quality must be the center's responsibility.

The panel believes it would be very much to the center'sadvantage to develop close relationships with other majorfederal statistical agencies, in particular, the Bureau of theCensus. Given the bureau's special role in carrying out workfor other federal agencies; given its mix of resources of staffexperienced in all aspects of data collection and processing,including the use of both sample surveys and administrativerecords; and given its past experience in undertaking workfor the center, the neI urges the center to explore theestablishment of v. continuing, substantial relationshipbetween itself and the bureau. Such a relationship mightinclude, among other activities, the conduct of at least onemajor program for the center by the bureau. Such an under-taking, if feasible, would permit staff of the center to beexposed regularly to technical staff of the bureau and to theproblems inherent in all phases of data collection; to partici-pate in the determination of all aspects of the undertaking,including dealing with problems and developing solutions andobserving data collection nd processing; and to call on thebureau's staff for technical and other assistance in areasother than those involved in their joint relationship. Theperiod of outstanding cooperation a decade ago among staffof the Department of Health, Education, and Welfare, of thecenter, and of the Census Bureau during the development andconduct of the Survey of Income and Education can serve asa model.

60

51

o The panel recommends that the center establish a con-tinuing relationship with the Bureau of the Census andexplore the feasibility of joint participalkm in one ormore major programs.

A PUBLICATIONS POLICY

The data products of the Center for Statistics are the cul-mination of the certer's effort, and therefore are the pri-mary vehicle through which the work of the center should bejudged. Ineffective publications and incomplete or incorrectdata tapes and documentation that fail to serve the needs ofdata users, publications that contain outdated data, reports inwhich text and data are inconsistent, reports that fail todetail Methodology and provide error measures--all these havebeen ascribed to the center. Unfortunately, our review ofexamples of the center's publications confirms the view thatthe center has had more than its share of problems. In fact,the initial support within the department for a study of thecenter resulted in large part from concerns with the presen-tation of results in the center's publications. Many of therecommendations already proposed in the report, if implemen-ted, would result in improvements in th: center's publica-tions.

The critical lacks in many of the center's publicationscome quickly to the fore when its publications are comparedwith those of other federal statistical agencies: timeliness;the absence of detail on the definitions used or on thesources of the data themselves; any sense of how the datawete treated to get them into the form in which they werepresented; the treatment of problematic data elements; esti-mates of the sampling or nonsampling error present in thenumbers; disc1osure avoidance techniques; information on theuse of imputation or weighting procedures. Users find them-selves in the dark, unable to make appropriate and necessaryjudgments on quality and left with a feeling that the claware from an unknown source, of unknown quality, and havebeen subject to unknown editing and imputation procedures.As a consequence, their ability to make maximum use of thedata is restricted by their necessary caution. Such deficien-cies lead the panel to emphasize the need for the center toestablish standards and to developing mechanisms for moni-toring their implementation. Of course not all center publi-cations exhibit the problems we have identified; their exis-tence merely heightens the contrast with those publications

61

52

and reports that fail in their objective to inform andenlighten the user.

Our review also indicates clearly the need to upgradc thetechnological base for producing publications. Few if any ofthe present publications demonstrate concern with readability,clarity in presentation, and the use of graphs or charts toaid users. The center--and its publications--would _benefitthrough the application of graphic tools for display and ana-lysis. Similar attention should be directed to disseminationand distribution of publications.

An additional area of concern regarding publications andthe dissemination of information arose as a result of therecent reorganization of the center, which moved much, ifnot all, of the dissemination function Out of the center.Given that a major function and responsibility of a statisticalorganization is to understand how and by whom its data arebeing used, the panel believes that the organization mustinteract directly and continually with the users of its prod-ucts. Direct interaction also serves a number of other veryimportant functions, such as allowing those who best under-stand the data to explain its limitations and meaning topotential users; providing users with the opportunity oflearning about other types of data that might better servetheir needs or provide a broader understanding of issuesbeing studied; and, finally, providing producers with theopportunity of determining future data needs. The importantnature of this activity leads the panel to conclude that closeattention must be paid to the operation of the informationfunction under the new organizational arrangement.

o The panel recommends that the center institute a pub-lications policy that sets forth the types of publica-tionsanalytic reports, news releases, descriptivereports, statistical summaries and digests, methodolog-ical reports, etc.--and other forms of information, suchas data tapes, diskettes, on-line reporting capabilitiesto be made available; audiences to be served; and thefrequency of release of data.

o The panel recommends that the center establish, pub-lish, and adhere to a set of fixed release dates forselected key education statistks.

o The panel recommends that the publications policyrequire that center reports contait information on def-initions used; a description of the data collectionmethodology; provide a measure of reliability, if

62

53

applicable; discuss other posF1'ui4 bc,irces of error; anddetail other sources of relevant or similar data. Com-pendia of statistics, such as the Condition of Educa-tion, should, at a minimum, follow the example of theStatistical Abstract of the United States in providingreferences to source information; similarly, advance orbrief highlights reports should contain appropriatereferences to the sources of such information.

o The panel recommends that the Office of EducationalResearch and Improvement and the Center for Statis-tics, given the current organizational separation of thedissemination of data from the preparation phase, mon-itor and review closely the release of informationthrough the new arrangement and the quality andextent of interaction with users to ensure thatappropriate guidance on sources, uses, interpretations,and limitations of the data is provided to those receiv-ing the information.

o The panel recommends that the center initiate acomprehensive review of its technological capabilities,with the objective of developing a short-range plan forproviding state-of-the-art capability in the dissemina-tion and distribution of its products.

4

Recommendations

This chapter brings together all the panel's recommendationsfor the Center for Statistics. They are presented in theorder in which they appear in the report.

The panel believes that its recommendations provide arealistic approach to addressing the issues and problems thatface the center, both from the realities of the past andpresent, anti from the longer-range perspective of the centeras we envision it in the years ahead. Given our findings, webelieve that what we propose is critical to the survival ofthe center as a viable and credible organization. Our recom-mendations are a comprehensive and integrated set, a packagethat, taken as a whole, can make the center into a crediblestatistical agency.

In setting forth its recommendations, tile panel is fullymindful of the current budget situation. Fitst, many of itsrecommendations can be implemented within the current bud-getary capacity of the center and will result in significantimprovements in the quality, timing, and accuracy of what itdoes and how it does it. Second, the center must be pre-pared as neee,ed to curtail program, scope, or activity inorder to improve the quality of its work and products. Inthe view of the panel, less, done well, is far superior tomore, done poorly. The panel recognizes and accepts thatover time the center will require additional funding; however,it believes that such increases must be judged against theprogress of the center in meeting the standards expected ofit and should reflect an expected expansion in its program asit demonstrates its progress.

Finally, the panel wishes to emphasize the seriousness ofits view that there can be no defense for allowing the centerto continue as it has for all too long. It is past time forthose in positions of responsibility to race up to the risksand dangers of perpetuating the myriad and continuing prob-

54

64

55

lems of the center. Without strong and continuous commit-ment and demonstrated determination to undertake wide-rang-ing actions to change both the image and reality of the cen-ter, we are unanimous in our conviction that serious consid-eration should be given to the more drastic alternatives ofabolishing the center and finding other means to obtain anddisseminate education data. For our part, we continue tobelieve strongly that the center has a future and that thepreferred course of action is to begin the process ofimprovement.

MISSION, ROLE, AND RESPONSIBILITIES

The panel recommends that the mission of the Center forStatistics, as stated in Section 406(b) of the GeneralEducation Provisions Act, as amended (20 U.S.C. 1221e-l), bestrengthened to clearly establish and define the role of thecenter in assisting the Secretary of Education in determininfthe data needs for assessing the condition of education inthe United States, and for accepting responsibility andaccountability for ensuring the availability of the necessarydata.

The panel recoramerv4for the center ar,.th-ough it., reque:',.requests for directthrough its calls for

ralgress demonstrate its supporta through its budget actions,-A statistical advice, through

. from center officials, and

The panel recorame,, .1at the S:;notary of Educationdemonstrate strong and ccntinved supi.k.irt for the center andits mission through depart..-Ient budget requests, public state-ments, and in appearances before Congress.

The panel recommends That Congress and the secretary rec-ognize and state their support both for the nonpartisannature of the center and its statistical independence.

The panel recommends that the secretaTy designate the cen-ter as the functional agency responsible far coordination andtechnicai review of all data collection within the department.

The panel recommends that the secretary designate the cen-ter as the focal point of releasing statistical information oneducation,

65

56

The panel recommends that the center assume accountabilityfor the process that leads to the determination of the con-tent of the data to be collected.

CONTENT AND DIRECTIONOF DATA COLLECTION PROGRAMS

The pane: recommends that the leadership of the centerdevelop mechanisms to assist policy makers in determiningtheir data needs and that the center continue to involve itsusers in dialogue to determine the most relevant rnd appro-priate content of its data collection efforts.

The panel recommends that the center continue the compila-tion of program, staff, and financial data from the states,but that it undertake an analysis, jointly with representativesfrom the states, to ensure that the present program meetsboth center and state requirements for usefulness, relevance,quality, and reliability. In addition, the content of the dataon the condition of education, originating with the states,should be be monitored periodically with the intent ofimproving the content and thus its joint usefulness to thecenter and the states.

The panel recommends that the center initiate a sample-basedprogram of data collection focused on individual classroomsand students, designed to facilitate better understanding ofthe relationships between educational inputs, processes, andoutcomes. The sample would not necessarily be able to pro-vide adequate estimates for all individual states, although itshould be organized so that states could choose to augmentthe sample to provide state data.

The panel recommends that sample-based data on educationalinputs, processes, and outcomes be made readily available, ona timely basis, to all interested users in the form of public-use tapes, with appropriate masking of characteristics to pre-clude the identification of individual schools, teachers, orstudents. The panel believes that the usefulness of centerdata would be greatly enhanced if its data become widelyavailable as a resource for both policy and research use.

The panel recommends that the. center continue to explorethe inclusion of longitudinal features in its sample-based sur-vey data and that, as a minimum, small-scale longitudinal

6p

57

studies be mounted regularly as part of the center's datacollection efforts.

ESTABLISHING A FRAME OF REFERENCE

The panel recommends that the center develop a conceptualframework for organizing its program and for setting priori-ties in light of available resources.

THE ADVISORY COUNCIL

The panel recommends that Congress modify the mandate ofthe Advisory Council on Education Statistics to a role oftechnical review and oversight, rather than responsibility forthe establishing of statistical standards.

The panel recommends that the Secretary of Education, insupport of the center, provide timely written response to theconcerns raised by ACES in its annual reports to the secre-tary, and that Congress periodically hold hearings at whichboth the secretary and representatives of ACES provide tes-timony on the status, problems, and progress of the centerand the concerns and advice of ACES.

The panel recommends that the members of ACES be selectedfor their expertise and competence in areas directly relatedto the center's program.

The panel recommends that the center establish one or morecontinuing technical advisory groups, in addition to ACES, tomeet at least twice per year to review, comment, and adviseon the methodology used in the conduct of its work, and toprovide the director with objective, outsidc, independentreview of the technical capability of the center's staff or ofits products. As required by its changing program needs, thecenter also should establish ad hoc advisory groups repre-senting the unique technical and subject area skills.

67

58

IMPROVING DATA QUALITY

Statistical Standards

The panel recommends that the center develop, publish, dis-seminate, and implement standards to guide the conduct ofall phases of its work, from development of objectivesthrough collection, follow-up, and processing and includingthe preparation, review, analysis, and publication af results.

The panel recommends that the center establish an Off;'-e ofStatistical Standards and Methods and move expeditiousii torecruit and appoint a chief statistician at the leve of assis-tant director, with responsibility for the establishment andmaintenance of statistical standards throughout the center.

The panel recommends that the center, in concert withappropriate state and local education agencies (LEAs) andrepresentatives of institutions of higher education, institutethe development and publication of uniform definitions andtheir continuing review, to ensure the collection of consis-tent data, such as between different LEAs and states. Suchinformation should be made available regularly in referencehandbooks.

The panel recommends that the center undertake a continuingprogram of evaluation of its methodology and its programs.

The panel recommends that the center disseminate widely theresults of its methodological research. Publications in pro-fessional journals and presentations at conferences should bestrongly encouraged.

The panel recommends that the center require that itsreports contain information on definitions used in studies anda description of the data collection methodology, provide ameasure of reliability, if applicable, and discuss possiblesources of error.

Redesign of the Elementary/Secondary Data Programs:An Example

The panel recommends that the center continue to exploremost carefully the appropriate balance between the continuinguse of state and local administrative record sources and theuse of sample surveys. Although supportive of the use of

68,t

59

sample surveys, the panel does not bl!lieve that the centerhas fully resolved the issues involved iD shifting the elemen-tary/secondary data collection system from one based onadministrative records to one based solely on an integratedsampling survey approach. The panel endorses the efforts ofthe center in seeking new approaches to accomplishing itsmission and suggests that small-scale testing within one or afew states be undertaken to establish procedures, defineproblems and develop solutions, refine the approach and,overall, assess the feasibility and cost of the survey proposal.

Detecting and Reducing Error

The panel recommends that the center routinely make itsusers aware of the quality of the data and the limitationsinherent in their use.

The panel recommends that the center foster a climate ofopenness in dealing with all elements of its constituency,including other parts of the department, other governmentagencies, Congress, providers and users of its data, and itsprofessional peer groups.

The panel recommends that the center initiate a comprehen-sive program to assess and improve, where necessary, thequality, consistency, .;4 n d reliability of data obtained fromstate and local agencies, from institutions of higher educa-tion, and from other sources. Meetings, discussions, visits,and technical assistance, including the training of local andstate officials on what i7 required of them and their systems,must comprise a major component of the program.

The panel recommends that the center establish stane..rds forreview of data releases and institute procedures to effectthem.

TIMELINESS

The panel recommends that the center explore and developapproaches for expediting response from participants in itsstudies and techniques for producing usable advance results;use management information systems for monitoring progressthrough all phases of data collection and reporting; andestablish procedures for taking prompt and appropriateactions when required. Similar techniques and monitoringcontrols should be required as integral compzinents of work

69

60

done by contractors to the center and should be followedclosely to ensure prompt adherence to schedule, with penal-ties for failure to accomplish work on time.

RESOURCES--STAFF AND BUDGET

The panel recommends that the center identify the profes-sional and technical staff required to successfully accomplishits mission and initiate an independent review and assessmentof the technical and subject-matter qualifications of thepresent staff.

The panel recommends that the center initiate an active andcontinuing recruiting program to obtain staff with the neededskills that are lacking atring the present staff.

The panel recommenos that the center devote resources forthe purposes of developing innovative approaches, includingtraining and work assignments, to ensure the continuingtechnical growth and competence of the staff, and thataccomplishment be rewarded by opportunity for further pro-fessional tievelopmtnt.

The rAnel :recommends that the lines of t:sponsibility andanthority threughout the center be clearly ddineated.

The nel recommends that the center, as needed, activelysolicsit Ole assistance and cooperation of its sister federalstatistical agencies in addressing technical issues or problems,to obtain short-run staffing assistance, or in carrying outvarious facets of its program.

The panel recommends that the center staff be given bothresponsibility and authority for the conduct of individualprojects and, accordingly, be held accountable for meetingthe established standards of acceptable performance, includingtimeliness.

The panel recommends that the center recognize the valueand importance of participating in meetings and activities ofprofessional and technical organizations, incluoing the pre-aration, presentation, and publication of papers and articles

t ibing aspects of the work of the center and its prob-lei..., and that the center devote, and if necessary divert,resources to ensure and support staff participation.

70

61

CONTRACTING OUT--OR NOT

The panel recommends that the center undertake to conductsome projects directly in-house, in order to provide theopportunity for staff to become exposed to and experiencedin the variety and multitude of technical activities, problems,and difficulties inherent in such an undertaking and to pro-vide technical competence for the staff in subsequentlysupervising outside contractors.

The panel recommends that the center establish a continuingrelationship with the Bureau of the Census and explore thefeasibility of joint participation in one or more major pro-grams.

A PUBLICATIONS POLICY

The panel recommends that the center institute a m!blicationspolicy that sets forth the types of publications--analyticreports, news releases, descr; c reports, statistical sum-maries and digests, methodological reports, etc.--and otherforms of information, such as data tapes, diskettes, on-linereporting capabilities to be made available; audiences to beserved; and the frequency of release of data.

The panel recommends that the center establish, publish, andadhere to a set of fixed release dates for selected key edu-cation statistics.

The panel recommends that the publications policy requirethat center reports contain information on definitions used; adescription of the data collection methodology; provide ameasure of reliability, if applicable; discuss other possiblesources of error; and detail other sources of relevant orsimilar data. C:mpendia of statistics, such as the Conditionof Education, shcm1d, at a minimum, follow the example ofthe Statistical Abstract of the United States in providingreferences to source information; similarly, advance or briefhighlights reports should contain appropriate references tothe sources of such information.

The panel recommends that the Office of EducationalResearch and Improvement and the center, given the currentorganizational separation of the dissemination or data fromthe preparation phase, monitor and review cl oso.:1. the release

71

62

of information through the new arrangement and the qualityand extent of interaction with users to ensure that appropri-ate guidance on sources, uses, interpretations, and limitationsof the data is provided to those receiving the information.

The panel recommends that the center initiate a comprehen-sive review of its technological capabilities, with the objec-tive of developing a short-range plan for providing state-of-the-art capability in the dissemination and distribution of itsproducts.

72

References and Bibliography

Bonnen, James T.1977 Assessment of the current agricultural data base: an

information system approach. Pp. 386-407 in L.R.Martin, G.G. Judge, et al., eds., A Survey ofAgricultural Economics Literature. Vol. IL AmericanAgricultural Economics Association. St. Paul, Minn:University of Minnesota Press.

Bureau of Economic Affairs1984 Total Government Expenditures Allocated to Edu-

cation and Health: 1983. Survey of CurrentBusiness, National Income Accounts. Washington,D.C.: U.S. Department of Commerce.

Butler, M. Stuart, Michael Sanera, and W. Bruce Weinrod1984 Mandate for Leadership 11: f.7nntinuing the

Conservative Revolution. Wash'ngton, D.C.: HeritageFoundation.

Congressional Research Service, Library of Congress1984 The Federal Statistical System. 1980 to 1985. Report

prepared by Baseline Data Corporation, 98th Congre-ss, 2nd session. Washington, D.C.: U.S. GovernmentPrinting Office.

1985 An Update on the Status of Major Federal StatisticalAgencies, Fiscal Year, 1986. Report prepared forthe Committee on Government Operations, 99thCongress, 1st session. Washington, D.C.: U.S.Government Printing Office.

Cooke, Charles, Alan Ginsburg, and Marshall Smith1985 The Sorry State of Education Statistics. Unpublished

paper. U.S. Department of Education, Washington,D.C.

63

73

64

Council of Chief State School Officers1984 Education Evaluation and Assessment in the United

States: Position Paper and Recommendations forAction. Washington, D.C: Council of Chief State.S.:hool Officers.

Currie, James T.no Historical Overview of NCES. Unpublished paperdate prepared for the National Center for Education

Statistics, U.S. Department of Education, Washington,D.C.

David, Jane L.1985 Improving the quality and utility of NCES data. Pp

142-148 in L.J. Silverman and R.C. Taeuber, eds.,Invited Papers, Elementary/Secondary Education DataRedesign Project. National Center for EducationStatistics. Washington, D.C.: U.S. Department ofEducation.

Hall, G., R.M. Jaeger, C. Philip Kearney, and D.E. Wiley1985 Alternatives for a National Data System of

Elementary and Secondary Education. Reportprepared for the Office of Educational Research andImprovement. Washington, D.C.: U.S. Department ofEducation.

Hill, Kenneth1985 Controversy in the Department of Education: A

Highly Criticized Report. Unpublished paper preparedfor the Panel to Evaluate the National Center forEducation Statistics, Comidittee on NationalStatistics, National Research Council, Washington,D.C.

Jones, Sidney L.1985 Statistics in government decision making Pp. 473-

478 in Bureau of the Census, First Annual ResearchCvnference Proceedings, March 20-23, 1985.

Washington, D.C.: U.S. Department of Commerce.Kursh, H.

1965 The United States Office of Education: A Centuryof Service. Philadelphia, Pa.: Chilton Books.

Lykes, Richard W.19./5 Higher Education and the United States Office of

Education (1867-1953). Bureau of PostsecondaryEducation. Office of Educatiot., Washington, D.C.:U.S. Department of Health, Educaiion, arh:' Welfare.

National Center for Education Statistics1983a The Condition of Education, Part 2: NCES Program

and Plans. Washington, D.C.: U.S. Department ofEducation.

.7 4

65

1983b State and Local Expenditures for VocationalEd lcation Continue Steep Decline. NCES Bulletin83-310b. Washington, D.C.: U.S. Department ofEducation.

1984 Digest of Education Statistics, 1983-1984.Washington, D.C.: U.S. Departmcnt o' Education.

1985a The Condition of Education. 1985 Edition, AStatistical Report. V.W. Plisko and J.D. Stern, eds.Washington, D.C.: U.S. Department of Education.

1985b Invited Papers, Elementary/Secondary Education DataRedesign Project. L.J. Silverman and R.C. Taeuber,eds. Washington, D.C.: U.S. Department ofEducation.

1985c Synthesis of Invited Papers, Elementary/SecondaryRedesign Project, A Public Discussion Draft. LJ.Silverman and R.C. Taeuber, eds. Washington, D.C.:U.S. Department of Education.

1986 Plan for the Redesign of the Elementary andSecondary Data Collection Program. Working paper.Office of Educational Research and Improvement,U.S. Department of Education, Washington, D.C.

National Commission on Excellence in Education1983 A Nation at Risk: The Imperative for Educational

Reform. A report to the nation and the Secretary ofEducation. Washington, D.C.: U.S. Department ofEducation.

National Education Association1985 Suggestions for the NCES redesign project. Pp. 730-

747 in L.J. Silverman and R.C. Taeuber, eds., InvitedPapers, Elementary/Secondary Redesign Project.Washington, D.C.: U.S. Department of Education.

National Research Council1976 Setting Statistical Priorities. Report of the Panel on

Methodology for Statistical Priorities, Committee onNational Statistics, National Research Council.Washington, D.C.: National Acath;my of Sciences.

Ottina, John1973 Testimony of John Ottina, Commissioner of

Education, U.S. Department of Health, Education, andWelfare, before the General Subcommittee onEducation, House Committee on Education and Labor,House of Respresentatives, 93rd Congrcss, 1stsession.

75

66

Plisko, V.W., A. Ginsburg, and S. Chaikind1985 Education statistics: assessing national data. Pp.

407-430 in L.S. Silverman, and R.C. Taeuber, eds.,Invited Papers, Elementary/Secondary Education DataRedesign Project. National Center for EducationStatistics. Washington, D.C.: U.S. Department ofEducation.

U.S. Bureau of Education1891 The U.S. Bureau of Education from 1867 to 1890 with

Subject and Index. Washington, D.C.: U.S. Bureauof Education

U.S. Congress, House1965 Hearings on the U.S. Office of Education. Special

Subcommittee on Education, Committee on Educationand Labor, 89th Congress, lad session. Washington,D.C.: U.S. Government Printing Office.

1967 Study of the United States Office of Education underthe Authority of House Resolution 614: Report ofthe Special Subcommittee on Education. Committeeon Education and Labor, 90th Congress, 1st session.House Document No. 193. Washington, D.C.: U.S.Government Printing Office.

U.S. Department of Health, Education, and Welfare1957 Report of the Advisory Committee on Educational

Statistics. niTice of Education. Washington, D.C.:U.S. Department of Health, Education, and Welfare.

1960 Report of the Advisory Committee of Users ofEducational Statistics. Office of Education.Washington, D.C.: U.S. Department of Health.Education, and Welfare.

1961 A F( Jeral Education Agency for the Future. Reportof the Committee on Mission and Organizat'in,Office of Education. Washington, D.C.: U.S.Department of Health, Education, and Welfare.

1963 Report of the Advisory Panel on EducationalStatistics. Office of Education. Washington, D.C.:U.S. Department of Health, Education, and Welfare.

1976 First Annual Report of the Advisory Council onEducation Statistics, A Description of Activities.Unpublished report. Office of Education, U.S.Department of Health, Education, and Welfare,Washington, D.C.

7 6

67

1977 Second Annual Report of Ihe Advisory Council onEducation Statistics, A Description of Activities.Unpublished report. Office of Education, U.S.Department of Health, Education, and Welfare,Washington, D.C.

1978 Management Evaluation: National Center forEducation Statistics. Unpublished report prepared bythe Office of Management Analysis and Systems.U.S. Department of Health, Education, and Welfare,Was'aington, D.0

U.S. Office of Management and Budget1984 Federal Statistics: A Special Report on the

Statistical Program and Activities of the U.S.Government, FY 1985. Statistical Policy Office,Office of Information and Regulatory Affairs.Washington, D.C.: U.S. Government Printing Office.

1985 Federal Statistics: A Special Report on theStatistical Program and Activities of the U.S.Government, FY 1986. Statistical Policy Office,Office of Information and Regulatory Affairs.Washington, D.C.: U.S. Government Printing Office.

Warren, Donald R.1974 To Enforce Education. Detroit, Mich.: Wayne State

University Press.

Appendix A

Organizational Structure of the Center

During the panel's tenure, the Center underwent a number ofreorganizations, including a renaming, from the National Cen-ter for Education Statistics to the Center for Statistics.Shown on the following pages are the organizational struc-ture of the center, as of April 1986, and a listing of theprogram content and staffing of each of the organizationalcomponents.

78

68

CENTER FOR STATISTICS

Acl:fisory Council on

Education Statistio

Offir of

the

INUMMIN VIOft0111.11MP MIME=

Condition of

Education

Division

Elem/Sec

Education

Statistics

Division

General Surveys

and

Analysis Branch

1 Special Surveys

and

Analysis Branch

79

Postsecondary

Education

Statistics

Division

General Surveys

and

Analysis Branch

Special Surveys

and

Analysis Branch

Education

Outcomes

Division

Longitudinal

Studies Bran]ch

Cross Sectional

and Special

Studies Branch

70

CONDITION OF EDUCATION DIVISION

Condition ofEducationDivision

20 professional3 support

IndicatorsCondition of EducationDigest of Education StatisticsProjections of Education StatisticsAllocationsData on Vocational Education

ELEMENTARY/SECONDARY EDUCATIONSTATISTICS DIVISION

Elem/SecEducationStatisticsDivision

1 professionalI. support

Elementary-Secondary Redesign

General Surveys 1and

Analysis Branch i14111r . 4

11 professionc..

1 support

Special Surveys

andAnalysis tlronch9 professionalI support

Common Core of Data (CCD)CCSSO Education Data Improvement Project

Public/Private School (including libraries) SurveysSchool and Staffing SurveysLEA Teacher Supply and Demand Survey

80

71

'OSTSECONDARY EDUCATION STATISTICS DIVISION

General Surveysand

Analysis Branch13 profeuional2 support

PostsecondaryEducationStatisticsDivision


Higher Education General Information System(HEGIS)

Integrated Postsecondary Education Data System(IPEDS)

Special Surveys National Postsecondary Student Aid Study (NPSAS)and Recent College Graduates (including transcript) Surveys

Analysis Branch State Higher Education Executive Officers (SHEE0)/CS Network8 professional2 support

EDUCATION OUTCOMES DIVISION

LongitudinalStudies Branch

8 professionalI support

1

Cross Sectionaland Specir.1

Studies Branch9 professional2 support

EducationOutcomesDivision


National Education Longitudinal Study of 1988 (NELS 88)High School and BeyondNational Longitudinal Survey (NLS-72)

National A ... ment of Educational Progrese (NAEP)International StudiesFast Response Survey System (FRSS)

Appendix BBiographical Sketches

of Pane" Members and Staff

VINCENT BARA BBA is executive director 3f market researchand planning at General Motors Corporation, Detroit, Michi-gan. He was formerly director of the U. S. Bureau of theCensus and director of market intelligence, Eastman KodakCompany. His current research interests are in knowlegeutilization. He is a fellow and former vice president of theAmerican Statistical Association, a member of the Interna-tional Stalistical Institute, and on the board of directors ofthe Social Science Research Council. He received undergrad-uate degrees in advertising and marketing from WoodburyBusiness College and California State University and anM.B.A. degree in marketing from the University of California,Los Angeles.

ANTHONY S. BRYK is associate professor in the Departmentof Education at the Universky of Chicago, chairs the depart-mental computer committee, and oversees the operation ofthe department's computer c,...nter. He has been both admin-istrator and senior research associate at the Huron Institute,a firm of applied research, Ivaluation, and policy analysis.His current research interests are in applications of hierarch-ical linear models to educational research, the sociopoliticalorganaation of the evaluation context and its implicationsfor the role and training of applied social scientists, and thesocial organization of Catholic schools. He received a B.S.degree from Boston College in chemistry and an M.Ed. degreefrom Harvard University in measurement and statistics.

MICHAEL L. COHEN is a faculty research associate in theSchool of Public Arfairs zst the University of Maryland. Heis a former research associate to the Con. "ittee on NationalStatistics and served as consultant to the Panel to Evaluatethe National Center for Education Statistics. His primary

72

82

73

interests are in data analysis, regression, and sample designand estimation. He received a B.S. degree from the Univer-sity of Michigan and M.A. and Ph.D. degrees in statisticsfrom Stanford University.

ROBERTO M. FERNANDEZ is an assistant professor in theDepartment of Sociology at the University of Arizona and aRockefeller Postdoctoral Fellow. His recent work has beenon schooling and early labor force activities of Hispanicyouth, using data from the the center's longitudinal study,High School and Beyond, and the Department of Lec.lr'snational longitudinal study of youth labor market experience.His work also includes the institutional factors that deter-mine the structure of labor markets, the role of social net-works in recruitment to social movements, and the role ofinterorganizational networks in policy domains. He recentlyserved on the National Research Council's committee onyouth employment. He received a B.A degree from HarvardUniversity and M.A. and Ph.D. degrees in sociology from theUniversity of Chicago.

CHRISTOPHER JENCKS is professor of sociology and urbanaffairs at Northwestern University. He has previously beenprofessor of sociology at Harvard University and the Uniw.7-sity of California, Santa Barbara, and taught at the Harva,,4,Graduate School of Education. His research has dealt withthe development of higher education in the United States,the role of education in social stratification, and standardizedtesting, among other subjects. He received a B.A. degree inEnglish from Harvard College.

F. THOMAS JUSTER is director of the Institute for SocialResearch and professor of economics at the University ofMichigan. He is currently a member of the Commit ...c onNational Statistics, chair of its Panel on Statistics of Supp!vand Demand for Precollege Science and Mather-hat is...Teachers, and chair of the American Economic Assoei tionCommittee on the Quality of Economic Statistics. He is af.',..:!ow of the American Statistical Association. He received aB.S. degree from Rutgers University and a Ph.D. degree ineconomics from Columbia University.

STEPHEN B. KAAGAN is a Commissioner of Educam of thestate of Vermont. Previously, he was provost at Pratt Insti-tute in Brooklyn, New Yerk, served as deputy commissionerfor the Massachusetts Depal iioent of Education, sper0 oneyear in Washington as special assistant to the Deputy Com-

83

74

missioner of Education for Development, and was director ofadmissions and firr ncial aid for the Harvard School of Edu-cation. He has also served as a teacher of English in Can-berra, Australia, and Arlington, Massachusetts. He recentlyled an effort by thc of Chief State School Officerson the assessment and evaluation of education in the UnitedStates. He has a B.A. degree from Williams College and anM.A. degree in teaching and a Ph.D. degree in t.lucation,both from Harvard U giversity.

GRAHAM K ALTON is a research scientist in the SurveyResearch Center and professor of biostatistics at the Univer-sity of Michigan. Previously, he was professor of social sta-tistics at the University of Southhampton and reader insocial statistics at the London School of Economics. Hisresearch interest are in survey sampling and general surveymethodology,. He received a B.Sc. degree in economics, anM.Sc degree in statistics from the University of London, anda Ph.T1. degree in survey methodology from the University ofSouthha p m n .

ALE:%..* .7)ER LAW is director of program evaluation andresearch, California State Department of Education.Previously, he was a professional associate with thc Educa-tional Testing Service. He has served as visiting scholar at

,1ord and UCLA. His interests are primarily in the areassearch management, analyses of large scale data sets and

educational policy development. He received his doctorate inEducational Psychology frotr: the University of Southern Cali-fornia.

DANIEL B. LEVINE, study director of the iy.r,c1, wss formerlywith the Bureau of the Census; he was leputy directo:between 1979 and 1982 and also served as acting director.His interests are in the management of statist.ca, systemsand in the collection, processing, ar4.! presentation of statis-tical informatior, particularly thrky,bb the conduct of large-scale surveys and censuses. He Ls a fellow of the AnwricanStatistical Association and a member of tiv: InterneienljStatistical Institute. He received a B.A. degree from GeorgeWashington University and an M.A. degree in economics hornColumbia University.

RICHARD J. LIGHT is professor -ur ..he graduate school ofeducation and the Kennedy School or Government at Harvarduniversity. He is also chair of the seiruinar on assessment, agroup of faculty and administroors working on ways to

64

75

assess college effectiveness. His interests center on aggre-gating and synthesizing findings from multiple reports. He isa member of the American Statistical Association and theAmerican Educational Research Association and president ofthe American Evaluation Association. He recently chairedthe Panel on Outcome Measurement in Early ChildhoodDemonstration Programs at the National Research Council.He received a B.A. degree from the University of Pennsyl-vania and M.A. and Ph.D. degrees in statistics from HarvardUniversity.

SHIRLEY M MALCOM is head of the Office of Opportunitiesin Science of the American Association for the Advancementof Science (AA AS). Formerly, she served as program officerin the Science Education Directorate of the National ScienceFoundation, held various positions in the AAAS Office ofOpportunities in Science, and was assistant professor of bio-logy at the University of North Carolina-Wilmington. SheShe serves as AAAS staff liaison to the Commission on Pro-fessionals in Science and Technology, was chair of the Na-tional Science Foundation's Committee on Equal Opportunitiesin Science and Technology from 1984 to 1986, arid is a mem-ber of the Advisory Council of the Carnegie Foruai on Edu-cation and the Econ:ttny and the Forum's Task Force onTeaching as a Profession. She received a B.S. degree inzoology from the University of Washinston, an M.S. degree inzoology from the University of California, Los Angeles, and aPh.D. degree in teology from the Pennsylvania State Univer-sity.

BERNARD MICHAEL is a consti!,::t. c:itication and man-power. Previously, he was executi ve Oitt:ctot of the FederalInteragency Committee on Education of the U.S. Departmentof Health, Education, and Welfare. He also served s direc-tor of evaluation on vocational education and manpowertrail:ng in the Office of Education :nd as a senior manpowereconomist an d. statistician with the Bureau of Lacir Statis-tics. On retirefilent from the federal service, he became exe-cutive vice-president of Information Dynamics, Inc., a con-sulting firm specializing in publishing, planning, and rrinagingnational conferences and evaluation studies. He received aB.A. degree in economics from George Washington Universityand an M.A. degree in economics from Columbia University.

ANDREW C. PORTER is professor of education and codirectorof the Institute for Research on Teaching )f the College ofEducation at Michigan State 'Iniwasity. Previously, he was

85

76

associate director of the National Institute of Education, U.S. Department of Education. His research interests includeresearch on teaching and psychometric research. He receiveda B.S. degree in education from Indiana State University ladhis MS. and Ph.D. degrees ir educational psychology fromthe University of Wisconsin, Madison.

WRAY SMITH is chair of Harris-Smith Research, Inc..; inArlington, Virginia, and teaches stcitistics and operationsresearch at the George Washington University. Previously,he was assistant administrator of the Energy InformationAdministration and was technical director on the staff of theAssistant Secretary for Planning and Evaluation, U.S. Depart-ment of Health, Education, and Welfare. His research inter-ests are in data analysis and stochastic modeling, statisticalcomptating, and survey research. He is a fellow of the Amer-ican Statistical Association and a member af the OperationsResearch Society of America. I:: as IA. degree in mathe-matics and physics from Georg.- ...ngton University, anM.S degi:e in instrumentation ant, .cience from the Univer-sit of Michigan, and a D.Sc. dcgree in operations research-ad statistics from George Washington University.

ANNE M. SPRAGUE is a staff member of the Committee onNational Statistics and served as adrrinistrative secretary tothe panel. Previously, she served as conference coordinatorat Marymount College of Virginia. She has designed andimplemented numerous commercial surveys and, as a graduatestudent, conducted participant interviews of the NomadicRom. She has a B.A. agree in anthropology from the Uni-versity of Maryland.

86

Documents

ED 273 685 · successfui accomplishment of its mdssion. Tbe panel recognized that the center will require increased funding, but funding requests should be judged agaiust the progress