Rigorous Research on the Effects of Learning Technology ... · titles commercially available. Thus, it is very hard to state that all educational ... NS2. Slide 22 NS2 slight edits

Rigorous Research on the Effects of Learning Technology:Implications for Policy

Barbara MeansCenter for Technology in Learning

SRI InternationalPresentation for the National Conference of State Legislators

San FranciscoJune 13 2009June 13, 2009

NATIONAL STUDY OF THE EFFECTIVENESS OFEDUCATIONAL TECHNOLOGY INTERVENTIONSEDUCATIONAL TECHNOLOGY INTERVENTIONS

Sponsored byI tit t f Ed ti S iInstitute of Education Sciences U.S. DEPARTMENT OF EDUCATION

Conducted by

WithWith

http://ies.ed.gov/ncee/pubs/20074005Some of this material is based on work supported by the US Department of Education under Contract ED-01-CO-0039/007 with Mathematica Policy Research Any opinions, conclusions or recommendations expressed in this presentation are those of the presenter and do not necessarily reflect the views of the Department of Education or Mathematica Policy Research.

The ContextThe Context

C i l d t i NCLB i d th Ed tiCongressional mandate in NCLB required the Education Secretary to:

IES tasked with implementing the mandatep g

Design contract made key decisions Focus on mathematics and reading software with commonly used

d di d hi l istandardized achievement tests as learning outcome Emphasize schools serving Title I students Cluster similar products; individual product results confidential Use an open self-nomination process to select products for inclusion

The Technologies Studied

1 2 3 4 6 8 9 10 11 12

g

K 1 2 3 4 5 6 7 8 9 10 11 12

A B C DAEarly Reading

1. Headsprout2. Plato Focus

BReading Comprehension

6. Academy of

CPre-Algebra

10. SmartMath11. Achieve Now12 L P l b

DAlgebra

13. Carnegie Cognitive Tutor14 PLATO Al b3. Waterford

4. Academy of Reading5. Destination

Reading

Reading7. Read 1808. KnowledgeBox9. Leaptrack

12. Larson Prealgebra 14. PLATO Algebra15. Larson Algebra

ead g

Copyright © 2007 SRI International

Research Questions

Does the use of software products increase reading and mathematics achievement?

Wh t diti d ti i t d ith ff ti

I li i i h h i i li d h

What conditions and practices are associated with effective software implementations?

Implicit in the way these questions were operationalized was the decision to study the kinds of effects on achievement that can be expected when a district purchases a reading or mathematics software product.

The EETI study attempted to test the effectiveness of typicaly p yptechnology implementations under real-world conditions.

Sample & Recruiting

Schools recruited through their districts: 33

6

Schools recruited through their districts: 33 districts, 132 schools, 439 teachers, and over 9,400 students participated. Within each school, volunteering teachersWithin each school, volunteering teachers

were randomly assigned to use the software or teach as normal. If necessary to run the software efficiently,If necessary to run the software efficiently,

schools received technology upgrades from the study budget. Vendors trained teachers before

implementation started and provided ongoing support as they deemed appropriate.

Learning Outcome MeasuresLearning Outcome Measures

1st Grade: SESAT & TOWRE

4th G d SAT R di 4th Grade: SAT Reading

6th Grade: SAT Math

Algebra: ETS Algebra Test

Findings: The Short StoryFindings: The Short Story

A hi i ifi l hi h i Achievement test scores were not significantly higher in classrooms using the selected reading and mathematics software products than in control classrooms. p

There was substantial variation between schools in theimpacts on student achievementimpacts on student achievement.

Although the study collected data on many school andclassroom characteristics and practices they did notclassroom characteristics and practices, they did notexplain the variation in impacts between schools onstudent achievement except for two factors in reading.

Experimental Results for First Grade

Source: Dynarski et al., 2007.

Issues of Alignment & Integrationg g

Vendors said that their products were aligned with nationalVendors said that their products were aligned with nationalstandards, and most of the products could be tailored to state ordistrict standards if teachers wished.

Teachers’ reports of whether or not the product had beenaligned with local standards did not predict effectiveness.

P d t t d ith th ti iti d d f Products competed with other activities and agendas forteachers’ and students’ time and attention.

12 out of 15 of the products are described as “supplemental”12 out of 15 of the products are described as supplemental instruction rather than the core curriculum.

There were cases where students were confused by differencesin representation or terminology between regular class workand the software.

Understanding the findings in contextg g

The EETI study involved a certain class of educational The EETI study involved a certain class of educational technology applications. Most of them were drill-and-practice at heart. The study does not show that these kinds of software can’t

be effective. Sometimes they were and sometimes they weren’tweren t. Report analyzes effectiveness in treatment teachers’ first year

of using the product.of using the product.

Recently released report on teachers’ second year of implementation found significant average effect size for twoimplementation found significant average effect size for two products.

Actual Headlines

EDUCATION WEEK: Federal Study Finds No Edge for Students Using Technology-Based Reading and Math Products

W hi t P t Software’s Benefits On Tests In Doubt: StudyWashington Post: Software s Benefits On Tests In Doubt: StudySays Tools Don’t Raise Scores

CoSN: Washington Update: USDE Releases ControversialCoSN: Washington Update: USDE Releases Controversial Study on Efficacy of Educational Software

Associated Press: No Benefit of Adding Math and SciencegSoftware Products to Classrooms, Study Says

CNN.com: Study: No benefit going high-tech for math and science

Study Mythsy y

Myth: The study showed that using technology to teach science produces b fi f hino benefit for achievement.

There was NO SCIENCE software in the study.

Myth: The U.S. Department of Education wanted this study to show that y p ytechnology is ineffective to justify cutting off educational technology funding. IES had hoped that this study would find convincing positive effects i d d h d ili f d i d fi ldin order to demonstrate the power and utility of randomized field trials.

Myth: The study authors had it in for educational technology.We wanted to do an exemplary RCT study.

Myth: The study did not take implementation factors into account.O 200 i l t ti i bl d i hOver 200 implementation variables were measured in each classroom using the software. Very few variables had a significant correlation with effect size.

Response to Industry Criticismsespo se to dust y C t c s s

CRITIQUE: “only surveyed 16 educational software titles out of hundreds of suchCRITIQUE: only surveyed 16 educational software titles out of hundreds of such titles commercially available. Thus, it is very hard to state that all educational software does not impact learning.”

RESPONSE: What the report actually says: This study . . . was not designed to assess the effectiveness of educational technology across its entire spectrum of uses, and the study’s findings do not

t l i b t t h l ’ ff ti b d th t d ’ t tsupport conclusions about technology’s effectiveness beyond the study’s context, such as in other subject areas.

M b l ti b t f i l d t ith th b t il blMoreover, by selecting a subset of commercial products with the best available prior evidence of effectiveness, the study was if anything biased toward finding a positive effect relative to what could be expected if all software products were included.

Response to Industry Criticisms cont’dp y

CRITIQUE: “the study itself essentially admits that the teachers using the softwareCRITIQUE: the study itself essentially admits that the teachers using the software did not feel comfortable with it, even after training.” [emphasis added]

RESPONSE: What the report actually says: “by the time of the first classroom observation (generally about mid fall) when mostby the time of the first classroom observation (generally about mid-fall) when most teachers had begun to use products, the proportion of teachers indicating that the initial training had adequately prepared them had declined from 95 percent at the end of initial training to 60 percent.”

Moreover, the product vendors did the teacher training and ongoing support themselves.

CRITIQUE: “the teachers reported that they used the product only 10% to 15% of all classroom time over the course of one year devoted to a particular subject.”

RESPONSE: This is what happens in the real world. Further, amount of time onRESPONSE: This is what happens in the real world. Further, amount of time on the product did not correlate with effect size in Grades 1, 6 or algebra. There was a small association in Grade 4.

So where does this leave us?So where does this leave us?

What can research tell us about whether to invest in educational technology?educational technology?

Insights from a contrasting case . . .

16

The Technology StudiedThe Technology StudiedV

Integration of Curriculum, Technology & PD

Key Strategy -Visual, dynamic, linked representations

“Democratizing access to the mathematics of change” - Jim Kaput17

A decade of SimCalc research

SRI’s large-scale RCT built on many years of prior research.

Studies grew in size from 10 students to 10 teachers to 10 schools to 10 regions.

Many research methods were used to investigate how best to support students’investigate how best to support students understanding of proportionality and rate.

Research Questions

Does SimCalc support complex math learning with a wide variety of teachers and students?

Which kinds of variability in classrooms are most important to the effectiveness of the programimportant to the effectiveness of the program with students?

This study attempted to test the robustness of theapproach under realistic conditions of scaling up.

Scaling Up g p

L l d i d t l Large-scale randomized control study

1,621 students, 95 teachers, 73 h lschools

Research team developed i l l t itcurriculum replacement units,

instructional strategies, assessments and teacher trainingg

Ongoing professional development provided by UT’s Dana Center

Experimental DesignSUMMER SCHOOL YEAR

TeacherCl I i

TreatmentPart 1

Mathematical

Professional Development Classroom Instruction

TeachSimCalc Replacement

Part 2Teaching with

Part 3PlanningTreatment

SimCalcIntervention

Foundations(TEXTEAMS)

2 days

pUnit

[Technology + Curriculum]

about 3 weeks

gSimCalc

3 days

g

1 day

Control

Teach

Rate and Proportionality

Part 1MathematicalFoundationsControl Rate and Proportionality

as usual

about 3 weeks

Foundations(TEXTEAMS)

2 days

StudentPretest

StudentPosttest

NS3

Slide 21

NS3 Added diagramNicole Shechtman, 5/23/2007

Student learning measuresg

P ti l R i P ti fProportional Reasoning as Preparation for Algebra and Calculus“Simple” proportionality “Simple” proportionality– Solving for a specific value in a/b=c/d or y=kx– Reading a specific value in a graph or table

“Complex” proportionality– Solving problems that invoke the function y=kx

• Function implies a mapping• Function implies a mapping• Within various representations• Making connections or comparisons across varying rates,

f ti d t tifunctions, and representations

NS2

Slide 22

NS2 slight editsNicole Shechtman, 5/23/2007

SimCalc students scored higher2323

The overall effect size (classroom level) was 0.84, considered large in ed cation st dieseducation studies

(t(93) = 9.1, P < 0.0001)

Outcomes by EthnicityOutcomes by Ethnicity

(Student-Level Data)

NS4

Slide 24

NS4 changed graph again (change did not persist from yesterday for some reason)Nicole Shechtman, 5/23/2007

Implications for Ed Tech Implementation

Move beyond promotion of technology for its own sakey p gy

Need to think in terms of Instructional Activity Systems Content Content

Learning activities, some of which are technology based

Articulation with other aspects of instruction

Teacher professional development and collaboration around implementation

Assessment for learning Assessment for learning

Use of data to refine the Instructional Activity System on an ongoing basis

Need ways to measure the things we care about

THE NATIONAL EDUCATIONAL TECHNOLOGY PLANTECHNOLOGY PLAN

Prior National Ed Tech Plans

19962004

2000

Planning Activitiesg

Outreach to key stakeholders

Technical Working GroupTechnical Working Group

Commissioned research and data analyses

Solicitation of white papers, viewpoints and comments

Public web site

Plan Development Processp

ED Internal Interviews

Information Gatheringgg

Potential Theme Draft Plan

Web Site with Potential

O h I f P bli

Identification Draft PlanPotential Themes/Goals

Outreach Activities

Input from Public & the Field

29

Four Bases for Goal Settingg

Research on

implementation

effective technology

implementation

New Education

New and emerging NATIONAL EDUCATIONAL Education

Prioritiesg g

technology capabilities

TECHNOLOGY GOALS

TechnologyTechnology-induced changes in

30

qeducation

requirements

Education Priorities

Rigorous college- and career-ready standards and high-qualityRigorous college and career ready standards and high quality assessments that are valid and reliable for all students, including English language learners and students with disabilities;

Pre-K-to college and career data systems that track progress andPre K to college and career data systems that track progress and foster continuous improvement;

Improvements in teacher effectiveness and in the equitable distribution of qualified teachers for all students, particularlydistribution of qualified teachers for all students, particularly students who are most in need;

Intensive support and effective interventions for the lowest-performing schools.p g

Universal access to affordable, high-quality education, including early learning and college

R&D and infrastructure to dramatically accelerate innovation − the creation and adoption of effective educational solutions.

New Technology Capabilitiesgy p

Web 2.0 capabilities for knowledge building Learning 24/7g Virtual schools and open educational

resourcesresources Immersive environments

M bil ti d i Mobile computing devices Sensor networks

Much wider options for educational content and ways to learncontent and ways to learn

Technology-induced changes in education requirementseducation requirements

Predictable functions, even those requiring complex processes, can be automated or outsourced.

Effective participation as a worker, citizen, or community member requires new skills, including use of technology toolstools.

The nature of subject domains—particularly in the sciences—is being changed by technologysciences is being changed by technology.

An Examplep

Biology Today(bioinformatics)(bioinformatics)

SS

BioCyc Pathway/Genome DatabasesThe BioCyc databases are divided into three tiers, based on their quality.

BioCyc Pathway/Genome DatabasesThe BioCyc databases are divided into three tiers, based on their quality.

Tier 1 databases have received person-decades of literature-based curation, and are the most accurate. Tier 2 and Tier 3 databases contain computationally predicted metabolic pathways, predictions as to which genes code for missing enzymes in metabolic pathways, and predicted operons.

BioCyc Tier 1: Intensively Curated Databases

Tier 1 databases have received person-decades of literature-based curation, and are the most accurate. Tier 2 and Tier 3 databases contain computationally predicted metabolic pathways, predictions as to which genes code for missing enzymes in metabolic pathways, and predicted operons.

BioCyc Tier 1: Intensively Curated DatabasesDatabase Scope Highlights

EcoCyc Escherichia coli K-12 MG1655Model-Organism Database

* Literature curation of complete genome* Information from 17,000 publications* Transcriptional regulatory network* Protein complexes and transporter functions* Gene Ontology assignments

MetaCyc Multiorganism Metabolic Pathway and Enzyme * 1 200 metabolic pathways

Bi C Ti 2 C t ti ll D i d D t b S bj t t M d t C tiBi C Ti 2 C t ti ll D i d D t b S bj t t M d t C ti

MetaCyc Multiorganism Metabolic Pathway and Enzyme Database

1,200 metabolic pathways* Pathways elucidated from 1,500 organisms* Extensive commentary* Information from 19,000 publications

BioCyc Tier 2: Computationally-Derived Databases Subject to Moderate Curation23 databases are available. [list of tier 2 DBs]

BioCyc Tier 3: Computationally-Derived Databases Subject to No Curation

389 databases are available [list of tier 3 DBs] and ready for adoption [more] by interested scientists for curation and updating.

C t Y O P th /G D t b

BioCyc Tier 2: Computationally-Derived Databases Subject to Moderate Curation23 databases are available. [list of tier 2 DBs]

BioCyc Tier 3: Computationally-Derived Databases Subject to No Curation

389 databases are available [list of tier 3 DBs] and ready for adoption [more] by interested scientists for curation and updating.

C t Y O P th /G D t bCreate Your Own Pathway/Genome Database

Interested in creating a pathway/genome DB for your genome? [learn more]

Create Your Own Pathway/Genome Database

Interested in creating a pathway/genome DB for your genome? [learn more]

Research on effective education technology implementationstechnology implementations

Technology by itself is not the answer.

We need to think about instructional activity systems, not individual technology tools by themselves.

Like any education innovation, those involving technology need tho ghtf l implementation plans reflecting a s stemicneed thoughtful implementation plans reflecting a systemic perspective.

It makes sense to try out technology-based innovations on It makes sense to try out technology based innovations on a modest scale with provision for examining both practice and outcomes to identify areas for improvement.

All too often our policies assume we can solve our fproblems if we just buy the right technology

Insert into Schools &Make T h Insert into Schools &

ClassroomsTech

Purchase

Better instruction & improved outcomes for students

38

We actually need something more like thisWe actually need something more like this

We actually need something more like thisWe actually need something more like this• What learning goal do we want to address?• What research-based approaches show promise in addressing this goal?addressing this goal?• What technology is part of that?• What else do we need to change to implement this intervention?


• Try out the intervention in a small number of schools or classrooms.• Involve teachers in deciding how to implement.


• Gather information on both implementation and learning outcomes.• Involve teachers and students in

l i h i ievaluating the intervention.


• Aggregate data and feedback across pilot classrooms and schoolsclassrooms and schools.


• Determine whether intervention merits scaling up.• Identify ways in which the intervention can be improved.


• Plan for larger-scale implementation. I t t th i t ti ith th • Integrate the intervention with other

curriculum, assessment and pd activities.

Closing Thoughtg g

It’s time we moved from a “Silver Bullet” mentality to a “Continuous Improvement” mindset around educational technology.

Documents

Rigorous Research on the Effects of Learning Technology ... · titles commercially available. Thus, it is very hard to state that all educational ... NS2. Slide 22 NS2 slight edits