Putting Data Into Practice

  • Published on

  • View

  • Download

Embed Size (px)


<ul><li><p>8/8/2019 Putting Data Into Practice</p><p> 1/19</p><p>EDUCATION SECTOR REPORTS</p><p>PUTTING DATA INTO PRACTICE:Lessons From New York City</p><p>By Bill Tucker</p><p>October 2010</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 2/19</p><p> ACKNOWLEDGEMENTS</p><p>Thanks to my Education Sector colleagues, in particularElena Silva and Kelly Bathgate, for their help in thinkingabout issues related to data and learning, and to KevinCarey and Robin Smiles for their support in the writingand editing of this paper. Susan Headden deservesspecial thanks for her careful editing and thoughtfulfeedback. Catherine Cullen provided invaluable help withthe initial research and ideas contained in this report. Mysincere appreciation also goes to the many people whowere kind enough to read and comment on an earlier draftof this paper, including Ben Boer, Vincent Cho, BeverlyDonohue, Eric Osberg, Baron Rodriguez, Jeffrey Wayman,and Emily Weiss. Finally, thank you to the dozens ofeducators, researchers, policy analysts, and experts whograciously offered their insights and knowledge to me</p><p>throughout the research and writing of this report.This report was funded by the Charles Stewart MottFoundation. Education Sector thanks the foundation fortheir support. The views expressed in the paper are thoseof the author alone.</p><p> ABOUT THE AUTHORBILL TUCKER is managing director at Education Sector. He canbe reached at btucker@educationsector.org.</p><p> ABOUT EDUCATION SECTOREducation Sector is an independent think tank that challengesconventional thinking in education policy. We are a nonprot,nonpartisan organization committed to achieving measurableimpact in education, both by improving existing reform initiativesand by developing new, innovative solutions to our nations mostpressing education problems.</p><p> Copyright 2010 Education Sector </p><p>Education Sector encourages the free use, reproduction, and distributionof our ideas, perspectives, and analyses. Our Creative Commons</p><p> licensing allows for the noncommercial use of all Education Sector authored or commissioned materials. We require attribution for all use.For more information and instructions on the com mercial use of our </p><p> materials, please visit our website, www.educationsector.org.</p><p>1201 Connecticut Ave., N.W., Suite 850, Washington, D.C. 20036 www.educationsector.org</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 3/19</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 4/19</p><p>2 EDUCATION SECTOR REPORTS: Putting Data Into Practice www.educationsector.org</p><p> ARIS, a repository of statistical information aboutstudents.</p><p>With a couple of mouse clicks, classroom teacherscan now get such data as interim test scores, subjectgrades, attendance records, and English language</p><p>learner status on a single computer screen. Thanksto ARIS, a high school instructor who may have astudent for just one period a day can now see howthat student is progressing across all courses, andcan identify students at risk of academic failure.Teachers are now also able to spot long-term learningtrends, even for students who have moved oftenamong schools and who have only just arrived intheir class. After initial resistance, ARIS has won the</p><p>cautious support of the local principals union, andmore than 65 percent of the districts teachers nowparticipate in inquiry teams. 7</p><p>But ARIS has been fraught with problems, as well.Developers have confronted a tangle of antiquatedsystems that cant talk to each otherinformationsilos that prevented any one person from getting acomplete picture of a student. And they continue tostruggle with making the data timely and accurateand giving educators the time and training they needto use it well. In the process, they have learned thattechnology holds little value unless it is exible,relevant, and provides the ne-grained informationthat teachers really need. Above all, the district hasrealized that building a data system is only the rststepwhat educators do with the data is the criticalsecond. Building the conditions and demand for data-based analysis is often more difcult than collectingthe data itself.</p><p>While New Yorks size and the scope of its initiativemake it unique, the district provides a rich and timelyillustration of how data is being used in an urban</p><p>school systemand how it can be employed insmaller districts, as well. Seen from the perspectiveof both its obstacles and successes, New Yorksexperience holds valuable lessons for all schooldistricts about how to succeed with the criticalsecond component of the drive for datausing theinformation to improve student performance.</p><p>The Promise and Perils of DataData is used to inform decisions in almost every eldof endeavor, from health care to sports, from criminal</p><p> justice to nance. Film studios use data to decidewhat movies to produce. 8 Police departments use it topredict and prevent crime. And credit card companiesinstantly compare transactions with historical patternsto spot potential fraud.</p><p>In health care, providers are beginning to use datafrom electronic records to reduce errors, cut costs,and improve patient outcomes. Effective use ofelectronic health records is one reason why theVeterans Health Administration now surpasses otherhealth systems on standardized quality measureseven though its patients are for the most part older,sicker, and poorer. Not only do these records allowfor coordinated care by multiple physicians, they alsoenable more rapid identication of risk factors, makingit possible, for instance, to detect kidney disease in</p><p>veterans often before symptoms emerge. 9 </p><p>Likewise, the best teachers have always usedinformation about their students to help them improveinstructionand they know that more and betterinformation can lead to even better results. Yet,unlike for almost all other professionals who performcomplex, demanding work, the information toolsavailable to teachers have been remarkably limited.Most teachers still work isolated in their classrooms,with only their own eyes and rudimentary assessmenttools to guide them. For the most part, they arent</p><p>beneting from sophisticated information-gatheringtools, from their colleagues knowledge, or fromanalyses of thousands of similar situationsthe verykind of information that physicians, police ofcers,and even sports executives use on a daily basis.Concludes a recent article from SRI InternationalsCenter for Technology and Learning: Teachers donot have the data-rich, performance-support, andinformation-feedback work environment that virtually</p><p>Above all, the district hasrealized that building a data</p><p>system is only the rst stepwhat educators do with the data</p><p>is the critical second.</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 5/19</p><p>3EDUCATION SECTOR REPORTS: Putting Data Into Practice www.educationsector.org</p><p>all other high-performance professionals have attheir disposal. 10 </p><p>That is not to say that educational data doesntexist. On the contrary, it has been centrally collectedsince at least 1867 when the rst federal Education</p><p>Department was charged with collecting suchstatistics and facts as shall show the condition andprogress of education in the several States andTerritories, and of diffusing such information. 11 Butuntil recently, data almost always owed one way:up. It went from school to district, from district tostate, and from state to the federal government. Itwas retrospective in nature and designed almostexclusively to show compliance with state and federalregulations. Because the information was reported inaggregate, state agencies could provide policymakerswith descriptive snapshots of student populations, but</p><p>they couldnt give them data about individual studentsand how those students changed over time.</p><p>The 2001 federal No Child Left Behind Act, whichimposed new reporting requirements on states,sharpened the focus on data. The law required statesto collect test data for individual students each year,meaning states could no longer rely on aggregatedachievement information reported by local districts.(Student-level data can now be sorted by school,race, and a variety of other subgroups). At the sametime, researchers and policymakers urged states to</p><p>develop longitudinal data systems that would allowthem to track individual students over time, acrossschools, through college, and into the work force. In2002, only 15 states had a longitudinal data system. 12 By 2008, 48 states did. 13 </p><p>But these systems often fail to give educatorsthe information they need. While the amount ofeducational data collected continues to growTexasschool districts alone respond to 104 data collectionsby the Texas Education Agency each yearthe qualityand utility of much of it remains questionable. 14 Many</p><p>systems have become de facto data morgues, 15 usedmore often to perform autopsies of failed programsthan to help educators and policymakers improveexisting ones. 16</p><p>Other elds have tackled similar problems. Hospitals,for example, collect data on patient outcomes andmortality rates so policymakers, administrators, andconsumers can use it to make judgments about</p><p>entire institutions. But physicians require differenttypes of information, such as measurements of vitalsigns and results of blood tests, to diagnose andtreat individual patients. 17 Likewise, in education,state and district ofcials want data that shows broadtrends so they can assess a schools or a districtsoverall effectiveness. (This is accountability data.)Teachers want additional information, such as resultsfrom classroom assessments that may track weeklyprogress.</p><p>Health reformers have also demonstrated thatelectronic data systems will not improve performanceon their own. Although they are essential, improvedtechnology and better data are just the infrastructurefor more substantive changes in the daily practicesof providers. For example, ofcials at CentralPennsylvanias Geisinger Health System, a model in</p><p>its efforts to improve quality and control costs, quicklyrealized that the ability to share data across a varietyof systems and contexts was not enough to improve</p><p>care. What was needed was a cultural change amongits employeesa shift from working in isolation onsingle tasks to working together on tasks that arealigned. 18</p><p>New York Citys Data StrategyIndividual teachers can benet from access toanalytical tools like ARIS. But New York Citysgoalto build evidence-based school culturesiseven more ambitious and requires both the toolsand collaborative processes to use the informationto improve student performance. Shael Suransky,deputy chancellor for the Division of Performance and</p><p> Accountability, identies the two key components ofthe districts strategy: regular collection and analysisof assessment datamuch of which can be found in</p><p> ARISand the use of teacher inquiry teams. 19 </p><p>The best teachers have alwaysused information about their</p><p>students to help them improveinstruction.</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 6/19</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 7/19</p></li><li><p>8/8/2019 Putting Data Into Practice</p><p> 8/19</p><p>6 EDUCATION SECTOR REPORTS: Putting Data Into Practice www.educationsector.org</p><p>Signicantly, the New Dorp team included teachersfrom all subject areas. They observed each othersclasses to see how, and whether, critical writingskills were being taught. And what they found wasilluminating: They discovered that grammar wasrarely addressed across the curriculum, and thateven in English classes, learning literary terms, likethe difference between a metaphor and a simile,occupied more instructional time than lessons in suchmatters as syntax, punctuation, and verb agreement.The school responded by injecting grammar acrossthe curriculum, including it as part of the grade onassignments such as history essays. The studentsexamined by the inquiry team were canaries in theschools coal mine. The theory is that if you canmove those students, Zoleo says, then we canmove groups of students overwhelmingly. 23</p><p> A Rocky Start As encouraging as its efforts have been, New Dorp isthe exception when it comes to using data to improveinstruction. Elsewhere in the district, getting this sortof information into the right hands, let alone gettingteachers to use it to improve learning, has beenanything but easy.</p><p>From the beginning, ARIS has fallen short of thegrandiose promises made about itthat it would</p><p>transform instruction, that it would provide all of theinformation teachers need, that it would allow parentsto get involved in their childrens educations as neverbefore. And it has been bedeviled by countlessdelays. 24 Designers had several systems to integrate,and whatever errors these systems held made theirway into ARIS. The district also had to replace itsinitial vendor, IBM, with another contractor. Thedelays served to only further iname existing politicalopposition to the accountability strategy of schoolsChancellor Joel Klein. Citing budget cuts and layoffs,many teachers and parents felt the money could havebeen better spent elsewhere.</p><p>In ARISs rst year and a half, the results weredecidedly underwhelming. A February 2008 article inthe New York Post called the project a disaster andnoted that almost half the budget had already beenspent. 25 And in October 2008, the New York Times reported that the system was still largely inoperable. 26 </p><p>The technical challenges proved much greater thananticipated. None of the underlying data on studentsresides in a single database: Attendance data livesin one system, grades in another, and test scores inothers. Replacing all of these systems would havebeen almost impossible and certainly unaffordable.So, rather than collect all this information on itsown, ARIS was developed to sit on top of all thesesystems, requiring just a single user name andpassword for access. But the systems used by NewYorkand many other districtswere not designedfor this purpose, and many come from competingvendors. Each required costly software upgrades toallow ARIS to understand its data.</p><p> A series of rapid xes followed. Wireless Generation,an education technology company that served asa subcontractor to IBM on the project, took over</p><p>the development of ARIS, and an improved versionwas launched in November 2008. A site that allowsparents to track their childs progress was launchedthe following May. 27 Soon critics were changing theirtune. In August 2009, a poll by the New York CityOfce of the Public Advocate reported that most ofthe principals surveyed supported ARIS, even if alarge majority still challenged the costs. 28 AntoinetteIsable, a representative for the principals union, theCouncil of School Supervisors and Administrators,told the New York Times: While we were skeptical inthe beginning, we have come to understand much of[ARISs] long-term value and importance. 29</p><p> At rst, the ARIS training program focused onteaching the teachersprincipals includedhowto use the tools. There was less emphasis on howand why to apply them. During the 200809 schoolyear, each school selected at least two educatorsto be trained. The district also offered instructionfor principals so they could build awareness andsupport for the tool. But ofcials soon learned...</p></li></ul>