Putting Data Into Practice

Embed Size (px)

Text of Putting Data Into Practice

  • 8/8/2019 Putting Data Into Practice



    PUTTING DATA INTO PRACTICE:Lessons From New York City

    By Bill Tucker

    October 2010

  • 8/8/2019 Putting Data Into Practice



    Thanks to my Education Sector colleagues, in particularElena Silva and Kelly Bathgate, for their help in thinkingabout issues related to data and learning, and to KevinCarey and Robin Smiles for their support in the writingand editing of this paper. Susan Headden deservesspecial thanks for her careful editing and thoughtfulfeedback. Catherine Cullen provided invaluable help withthe initial research and ideas contained in this report. Mysincere appreciation also goes to the many people whowere kind enough to read and comment on an earlier draftof this paper, including Ben Boer, Vincent Cho, BeverlyDonohue, Eric Osberg, Baron Rodriguez, Jeffrey Wayman,and Emily Weiss. Finally, thank you to the dozens ofeducators, researchers, policy analysts, and experts whograciously offered their insights and knowledge to me

    throughout the research and writing of this report.This report was funded by the Charles Stewart MottFoundation. Education Sector thanks the foundation fortheir support. The views expressed in the paper are thoseof the author alone.

    ABOUT THE AUTHORBILL TUCKER is managing director at Education Sector. He canbe reached at btucker@educationsector.org.

    ABOUT EDUCATION SECTOREducation Sector is an independent think tank that challengesconventional thinking in education policy. We are a nonprot,nonpartisan organization committed to achieving measurableimpact in education, both by improving existing reform initiativesand by developing new, innovative solutions to our nations mostpressing education problems.

    Copyright 2010 Education Sector

    Education Sector encourages the free use, reproduction, and distributionof our ideas, perspectives, and analyses. Our Creative Commons

    licensing allows for the noncommercial use of all Education Sector authored or commissioned materials. We require attribution for all use.For more information and instructions on the com mercial use of our

    materials, please visit our website, www.educationsector.org.

    1201 Connecticut Ave., N.W., Suite 850, Washington, D.C. 20036 www.educationsector.org

  • 8/8/2019 Putting Data Into Practice


  • 8/8/2019 Putting Data Into Practice


    2 EDUCATION SECTOR REPORTS: Putting Data Into Practice www.educationsector.org

    ARIS, a repository of statistical information aboutstudents.

    With a couple of mouse clicks, classroom teacherscan now get such data as interim test scores, subjectgrades, attendance records, and English language

    learner status on a single computer screen. Thanksto ARIS, a high school instructor who may have astudent for just one period a day can now see howthat student is progressing across all courses, andcan identify students at risk of academic failure.Teachers are now also able to spot long-term learningtrends, even for students who have moved oftenamong schools and who have only just arrived intheir class. After initial resistance, ARIS has won the

    cautious support of the local principals union, andmore than 65 percent of the districts teachers nowparticipate in inquiry teams. 7

    But ARIS has been fraught with problems, as well.Developers have confronted a tangle of antiquatedsystems that cant talk to each otherinformationsilos that prevented any one person from getting acomplete picture of a student. And they continue tostruggle with making the data timely and accurateand giving educators the time and training they needto use it well. In the process, they have learned thattechnology holds little value unless it is exible,relevant, and provides the ne-grained informationthat teachers really need. Above all, the district hasrealized that building a data system is only the rststepwhat educators do with the data is the criticalsecond. Building the conditions and demand for data-based analysis is often more difcult than collectingthe data itself.

    While New Yorks size and the scope of its initiativemake it unique, the district provides a rich and timelyillustration of how data is being used in an urban

    school systemand how it can be employed insmaller districts, as well. Seen from the perspectiveof both its obstacles and successes, New Yorksexperience holds valuable lessons for all schooldistricts about how to succeed with the criticalsecond component of the drive for datausing theinformation to improve student performance.

    The Promise and Perils of DataData is used to inform decisions in almost every eldof endeavor, from health care to sports, from criminal

    justice to nance. Film studios use data to decidewhat movies to produce. 8 Police departments use it topredict and prevent crime. And credit card companiesinstantly compare transactions with historical patternsto spot potential fraud.

    In health care, providers are beginning to use datafrom electronic records to reduce errors, cut costs,and improve patient outcomes. Effective use ofelectronic health records is one reason why theVeterans Health Administration now surpasses otherhealth systems on standardized quality measureseven though its patients are for the most part older,sicker, and poorer. Not only do these records allowfor coordinated care by multiple physicians, they alsoenable more rapid identication of risk factors, makingit possible, for instance, to detect kidney disease in

    veterans often before symptoms emerge. 9

    Likewise, the best teachers have always usedinformation about their students to help them improveinstructionand they know that more and betterinformation can lead to even better results. Yet,unlike for almost all other professionals who performcomplex, demanding work, the information toolsavailable to teachers have been remarkably limited.Most teachers still work isolated in their classrooms,with only their own eyes and rudimentary assessmenttools to guide them. For the most part, they arent

    beneting from sophisticated information-gatheringtools, from their colleagues knowledge, or fromanalyses of thousands of similar situationsthe verykind of information that physicians, police ofcers,and even sports executives use on a daily basis.Concludes a recent article from SRI InternationalsCenter for Technology and Learning: Teachers donot have the data-rich, performance-support, andinformation-feedback work environment that virtually

    Above all, the district hasrealized that building a data

    system is only the rst stepwhat educators do with the data

    is the critical second.

  • 8/8/2019 Putting Data Into Practice


    3EDUCATION SECTOR REPORTS: Putting Data Into Practice www.educationsector.org

    all other high-performance professionals have attheir disposal. 10

    That is not to say that educational data doesntexist. On the contrary, it has been centrally collectedsince at least 1867 when the rst federal Education

    Department was charged with collecting suchstatistics and facts as shall show the condition andprogress of education in the several States andTerritories, and of diffusing such information. 11 Butuntil recently, data almost always owed one way:up. It went from school to district, from district tostate, and from state to the federal government. Itwas retrospective in nature and designed almostexclusively to show compliance with state and federalregulations. Because the information was reported inaggregate, state agencies could provide policymakerswith descriptive snapshots of student populations, but

    they couldnt give them data about individual studentsand how those students changed over time.

    The 2001 federal No Child Left Behind Act, whichimposed new reporting requirements on states,sharpened the focus on data. The law required statesto collect test data for individual students each year,meaning states could no longer rely on aggregatedachievement information reported by local districts.(Student-level data can now be sorted by school,race, and a variety of other subgroups). At the sametime, researchers and policymakers urged states to

    develop longitudinal data systems that would allowthem to track individual students over time, acrossschools, through college, and into the work force. In2002, only 15 states had a longitudinal data system. 12 By 2008, 48 states did. 13

    But these systems often fail to give educatorsthe information they need. While the amount ofeducational data collected continues to growTexasschool districts alone respond to 104 data collectionsby the Texas Education Agency each yearthe qualityand utility of much of it remains questionable. 14 Many

    systems have become de facto data morgues, 15 usedmore often to perform autopsies of failed programsthan to help educators and policymakers improveexisting ones. 16

    Other elds have tackled similar problems. Hospitals,for example, collect data on patient outcomes andmortality rates so policymakers, administrators, andconsumers can use it to make judgments about

    entire institutions. But physicians require differenttypes of information, such as measurements of vitalsigns and results of blood tests, to diagnose andtreat individual patients. 17 Likewise, in education,state and district ofcials want data that shows broadtrends so they can assess a schools or a districtsoverall effectiveness. (This is accountability data.)Teachers want additional information, such as resultsfrom classroom assessments that may track weeklyprogress.

    Health reformers have also demonstrated thatelectronic data systems will not improve performanceon their own. Although they are essential, improvedtechnology and better data are just the infrastructurefor more subs