Defining Data Literacy for Educators

Ellen Mandinach, WestEdEdith Gummer, Education Northwest and NSF

A Collaborative Project of WestEd and Education Northwest

Sponsored by the Bill & Melinda Gates Foundation

STATS-DC 2012 Data ConferenceJuly 12, 2012

Problem Space

• No common definition of data literacy• Professional development providers • Teacher/Administrator preparation

• Confusion between data literacy and assessment literacy

• Lack of understanding of what it means to be data literate and use data effectively

The Project

• Bring together a diverse group of expert stakeholders who know data-driven decisionmaking

• Scope out a common definition of data literacy

• Discriminate between data literacy and assessment literacy, understanding the requisite skills and knowledge

• Perform a content analysis of major professional development books and models. Identify the key knowledge and skills.

• Identify commonalities, differences, and perform a gap analysis.

The Stakeholder Groups

• Experts from:• Research• Professional development• Government• Policy• Funding agencies• Assessment• Other stakeholders

Data Sources

• Pre-conference data definitions from experts

• Professional development text analyses

• A hands-on activity to distinguish between data literacy and assessment literacy

• Transcripts from the conference

Pre-Conference Definitions

• Really interesting and diverse• Some succinct; others quite involved• Some quite philosophical• Focus on data; focus on assessment• Plethora of skills and knowledge

components

A Wordle for Data Literacy

Some Overarching Thoughts and Select Quotes from Pre-Conference

• Differentiation between declarative, procedural, schematic and strategic knowledge; the content versus the performance/process components

• Data literacy is more than the sum of the component skills and knowledge

• Continuum of novice to expert• “Defining data literacy will require careful

consideration of the level of ability necessary to define literacy – somewhere between basic skills to complete fluency.”

Conflicting Perspectives(in one quote)

“At the state department and college/university levels we are doing a damn lousy job of teaching data literacy.  Too few states demand courses on testing and measurement.  Too few colleges have those courses taught (when taught at all) by people who could plausibly be described as psychometrically competent.  We spend too much time on the statistics of data and too little on the ‘how to make constructive use of data’ and the ethics of data gathering and use.”

A Hands–On Activity

• Purpose – to differentiate between data and assessment literacy

• Outcomes• Data literacy is seen as broader than

assessment literacy – but contested• Assessment literacy is a component of

data literacy – but has unique components

• The question remains what are the unique components – open quandary

Analysis of the Materials

• Purpose – to characterize the terrain of professional development in data literacy • What knowledge and skills are suggested by

the materials as being the core of what educators need to know and be able to do?

• Representative rather than exhaustive search

• Provide an initial operational framework

Analysis of the Materials

• Process for analysis• Had to start somewhere – 2 texts we knew

were being used extensively in professional development» Love, Stiles, Mundry, & DiRanna (2008) The data

coach's guide to improving learning for all students.

» White (2011) Show me the proof: Tools and strategies to make data work with the Common Core Standards.

• Read through the texts identifying the key concepts and skills being presented on each page – tons of post-it notes

Analysis of the Materials

Grain size decisions – 30,000 feet versus weeds• Examined commonality in language

across the 2 texts• Distilled 4 pages of codes to 8 key

terms – negotiated process• Sorted remaining codes into 8 key

terms

Analysis of the Materials

Coding process• 2 research associates used the coding

spreadsheets – negotiated practice• Code values

» Mentioned – page # and m» Whole page addressed code – page #-1» More than one page addressed code page # +» Multiple pages addressed code – page #-page #

Relative emphasis of the text material

Findings – Common Elements

Collaborative processesMultiple forms of data – triangulationKey focus on student achievementLittle emphasis on statistical knowledgeGeneral analysis processes described

with examplesTemplates for examining and displaying

data and making interpretationsConnection to instruction overly general

Findings – Variations

Level of description and instruction – a number of texts provide broad general description and few details

Data use versus formative assessment • Focus on multiple forms of data beyond

achievement and student performances• Educator versus teacher

Analysis of the Conference Transcripts

• Overarching theme: It’s complex• Data literacy is more than just its

definition• There is an entire educational

contextual landscape that surrounds the definition

• The issue is systemic

Working Definitions for Data Literacy

• Data Literacy is………• Complexity• 95/5 issue – importance of the 5%• Consensus definition

Complexity of Defining Data Literacy

• Literacy – generic to specific• Messaging• Language• Role or stakeholder

influence/perspective• Systems nature• Technical knowledge and skills

• Data• Pedagogical/administrative

Knowledge and Skills95%/5%

• Problem Focus• Identify purpose• Frame questions• Understand context• Apply data to instructional or administrative action

• Data Focus• Select the right data• Know how to access data• Design instruments• Collect data• Organize data• Use multiple sources of data• Understand data properties• Understand different kinds of data (formative, summative, diagnostic)• Use a variety of data sources (qualitative, quantitative)

Knowledge and Skills95%/5%

• Data Focus - continued• Use of statistics and measurement concepts - understand

reliability, validity, sources of error• Understand scaled scores, percentiles, performance levels• Drill down to different levels of data• Summarize data• Synthesize information• Analyze data• Analyze different levels of data – cohort, course, grade• Examine patterns and trends• Make interpretations• Troubleshoot data

Knowledge and Skills95%/5%

• Process Focus• Formulate hypotheses• Engage in collaborative inquiry• Transform data into actionable knowledge• Know how to use data systems, tools, and applications• Comprehend data displays and data reporting• Draw inferences• Consider consequences• Test assumptions• Evaluate outcomes• Critique arguments

• Habits of Mind - Dispositions

Educational Context

• The need for a conceptual model• Account for organizational factors• Issues around data properties and adequacy for use• Issues around professional development• The need for best practices and models• The role-based nature of data literacy• Roles of schools of education, state agencies, and policymakers• The SLDS• Other technologies and supporting resources • The role of research• The need for instrumentation• The role of funding organizations• The need for collaboration• Messaging

A Road Map for Next Steps• For the researchers• For the professional development

providers• For the policymakers• For the funders