EVALUATING WEST VIRGINIAS EETT MODEL SCHOOLS PROJECT NECC 2006 San Diego, CA

EVALUATING EVALUATING WEST VIRGINIA’S WEST VIRGINIA’S EETTEETT

MODEL SCHOOLS PROJECTMODEL SCHOOLS PROJECT

NECC 2006NECC 2006

San Diego, CASan Diego, CA

West Virginia StatewideWest Virginia StatewideEducational TechnologyEducational Technology

NAEP 50-State Measures of NAEP 50-State Measures of Student Computer UseStudent Computer Use

(SOURCE: Jonathan Becker, Ph.D., (SOURCE: Jonathan Becker, Ph.D., Professor, Hofstra University, Professor, Hofstra University, Hempstead, NY)Hempstead, NY)

*Only one state has had its *Only one state has had its students’ use of computers students’ use of computers at the highest interval (“a at the highest interval (“a lot”) exceed the national lot”) exceed the national average in average in every assessed every assessed yearyear from 1997 to 2003 from 1997 to 2003 ((pp = <.05) = <.05)

*That consistently high-use *That consistently high-use state is West Virginia.state is West Virginia.

West Virginia StatewideWest Virginia StatewideAchievement GainsAchievement Gains

Q: Can computer Q: Can computer use improve use improve student learning?student learning?

A: YESA: YES

From BS/CE to From BS/CE to TMSTMS

Question:Question:

When does technology improve achievement?When does technology improve achievement?

Answer:Answer:

When it is used by teachers and students.When it is used by teachers and students.

Question:Question:

How can more teachers be encouraged to use more How can more teachers be encouraged to use more technology?technology?

Answer:Answer:

In-schoolIn-school technology integration specialists (TISs, the technology integration specialists (TISs, the Technology Model Schools Program)Technology Model Schools Program)

A Study Question of A Study Question of National InterestNational Interest

QuestionQuestion::

Did the in-school, TIS strategy work and how Did the in-school, TIS strategy work and how do we know?do we know?

AnswersAnswers::

• Teacher computer useTeacher computer use

• Student computer useStudent computer use

• Achievement gainsAchievement gains

RESEARCH DESIGNRESEARCH DESIGN

Experimental/ControlExperimental/Control& Pre/Post& Pre/Post

EXPERIMENTALEXPERIMENTAL

With TIS specialistWith TIS specialist

CONTROLCONTROL

Without TIS specialistWithout TIS specialist

PREPRE

Data at beginning of Data at beginning of TIS yearTIS year

POSTPOST

Data at end of TIS work Data at end of TIS work with teachers with teachers and and beyondbeyond

Using Technology Using Technology To Measure TechnologyTo Measure Technology

• Meters in desktopsMeters in desktops– Software runs in the Software runs in the

background 24/7background 24/7

– Records file activity only Records file activity only

– Installed on classroom Installed on classroom computers onlycomputers only

– Data e-mailed directly to Data e-mailed directly to Interactive, Inc.Interactive, Inc.

– ZERO burden on teachersZERO burden on teachers

• Pagers on teachers + Pagers on teachers + web surveysweb surveys – Free pager rings once per Free pager rings once per

day, every other weekday, every other week

– When pager rings, teacher When pager rings, teacher and a randomly selected and a randomly selected student take a web-surveystudent take a web-survey

– Pages are randomly Pages are randomly scheduled during alternate scheduled during alternate weeks (not more than 5 per weeks (not more than 5 per week)week)

USING TECHNOLOGY TO STUDY USING TECHNOLOGY TO STUDY TECHNOLOGY:TECHNOLOGY:

How well did it work (Year 1 – [2004-How well did it work (Year 1 – [2004-05])?05])?

Pagers + Web-SurveysPagers + Web-Surveys

• 2,100 pager-triggered responses from 2,100 pager-triggered responses from teachers. teachers.

• 2,000+ responses from students. 2,000+ responses from students. • Responses from 110 different teachers. Responses from 110 different teachers. • Responses from individual teachers ranged Responses from individual teachers ranged

from 36 to 1.from 36 to 1.

Data from Metering SoftwareData from Metering Software

• Data from 184 computers in 25 schools. Data from 184 computers in 25 schools.

• Installed on 287 computers. Installed on 287 computers.

• 2 reports a day X 184 computers X 5 2 reports a day X 184 computers X 5 monthsmonths

• = 35,000 e-mail reports of computer use= 35,000 e-mail reports of computer use

RESULTS: RESULTS: Teacher and Student Teacher and Student

Computer Use Computer Use (Web-Based Survey Data – Year 1 (Web-Based Survey Data – Year 1

[2004-05])[2004-05])

More Teacher Computer Use More Teacher Computer Use In TIS SchoolsIn TIS Schools

% of “Yes” Responses% of “Yes” Responses

Treatment (n=933)Treatment (n=933) 2121

Control (n=646)Control (n=646) 1111

““At the time the pager was activated, were At the time the pager was activated, were YOU using computers?”YOU using computers?”

More Teacher Productivity Use More Teacher Productivity Use In TIS SchoolsIn TIS Schools

Treatment %Treatment % Control %Control %Word ProcessingWord Processing 3030 2525PowerPointPowerPoint 33 00SpreadsheetsSpreadsheets 1010 33MS PublisherMS Publisher 11 00Compass / RiverdeepCompass / Riverdeep 44 33Internet researchInternet research 2424 1717E-mailE-mail 44 3838OtherOther 2525 1515

““How were you using your computer?”How were you using your computer?”

More Student Computer Use More Student Computer Use In TIS SchoolsIn TIS Schools

% “Yes”% “Yes”

Technology Model Technology Model Schools (n=941)Schools (n=941)

24% 24%

Comparison schools Comparison schools (n=649)(n=649)

22% 22%

Another jurisdictionAnother jurisdiction 0.1%0.1%

““At the time the pager was activated, were At the time the pager was activated, were any of your students using computers?”any of your students using computers?”

RESULTS: RESULTS: Student Achievement Student Achievement

(Year 1 [2004-05])(Year 1 [2004-05])

RESULTS:RESULTS:Achievement GainsAchievement Gains

Q: Can computer use improve student Q: Can computer use improve student learning?learning?

A: YES: 4A: YES: 4thth and 5 and 5thth Grade students in the Grade students in the TIS schools made (statistically TIS schools made (statistically significantly; p<.05) greater score gains significantly; p<.05) greater score gains (compared to similar students in non-(compared to similar students in non-TIS schools) on the state’s Math TIS schools) on the state’s Math WESTEST (2004-2005). WESTEST (2004-2005).

Summary (Year 1): Summary (Year 1): IN THE TIS SCHOOLSIN THE TIS SCHOOLS

• Students used computers moreStudents used computers more

• Teachers used computers moreTeachers used computers more

• Teachers used computers more for Teachers used computers more for productivity applicationsproductivity applications

• There were statistically significant gains in There were statistically significant gains in WESTEST Math scores from students in WESTEST Math scores from students in the 4the 4thth and 5 and 5thth grades. grades.

Coming AttractionsComing Attractions

Year 2 [2005-06]Year 2 [2005-06]

National (Year 2) Study QuestionsNational (Year 2) Study Questions

• Since the TIS specialists completed Since the TIS specialists completed their funding year (June 2005), their funding year (June 2005), what happened? Did teachers what happened? Did teachers continue to use technology? Did continue to use technology? Did achievement gains hold?achievement gains hold?

Year 2 Data Collection SuccessYear 2 Data Collection Success

• 337 teacher responses to pager-inspired Web-337 teacher responses to pager-inspired Web-surveyssurveys

• 297 student responses to pager-inspired Web-297 student responses to pager-inspired Web-surveyssurveys

• 134 (of a possible 182) teachers in 48 schools 134 (of a possible 182) teachers in 48 schools completed the end-of-year questionnairecompleted the end-of-year questionnaire

• 45,434 e-mails with data from TAM-monitored 45,434 e-mails with data from TAM-monitored computerscomputers

• 21 schools visited during academic year 2005-0621 schools visited during academic year 2005-06

(Very) Preliminary Conclusions(Very) Preliminary Conclusions

• Most places are:– getting most teachers to be familiar with different applications    

– and getting some teachers to mastery.

•  West Virginia's multi-year effort has produced:– near universal mastery with

– many teachers as fluent users.

•  West Virginia's fluent users:– pick and choose the strongest features for their student needs

– invent new ways to use technology

– emphasize small group and individual work.

– work collaboratively with other teachers to develop local applications.

Questions?Questions?Interactive, Inc.Interactive, Inc.

61 Green Street 61 Green Street

Huntington, New York 11743Huntington, New York 11743phone: 631 351 1190phone: 631 351 1190fax: 631 351 1194fax: 631 351 1194

e: interinc@aol.come: interinc@aol.com