THE RESULTS OF PROFESSIONAL … revised.doc · Web viewThis research documents the success of using technology to study technology. Teachers and school administrators cooperated with

THE RESULTS OF PROFESSIONAL DEVELOPMENT ABOUT TECHNOLOGY:

A Report of West Virginia's Statewide Technology Model Schools Program

Prepared for: West Virginia Department of EducationDivision of Curriculum & InstructionOffice of Technology and Information SystemsCharleston, West Virginia

By: Dale Mann, PhD., Managing DirectorJonathan Becker, LLD & Ph.D., Research DirectorInteractive, Inc.

Monday, June 18, 2007

61 Green Street • Huntington, New York 11743-6913Phone: 631-351-1190 • Website: www.interactiveinc.org • Fax: 631-351-1194

THE RESULTS OFPROFESSIONAL DEVELOPMENT ABOUT TECHNOLOGY:


Contents1.0 SUMMARY...................................................................................................1

1.1 Was student achievement in TMS schools higher than in other schools?...21.2 How did students use their computers?.......................................................31.3 Were teachers benefited in their instructional and other duties?.................31.4 Was the turnkey expectation realized?........................................................31.5 Did the effects of the TISs survive their departure?.....................................41.6 Did the new methods for evaluating the outcomes of professional development work?............................................................................................41.7 Did the program---as a whole---work?.........................................................41.8 Can the Technology Model Schools program be replicated?......................4

2.0 THE TECHNOLOGY MODEL SCHOOLS PROGRAM................................62.1 The US Department of Education and support for educational technology. 62.2 The West Virginia Department of Education and support for educational technology.........................................................................................................72.3 The Technology Model Schools program....................................................7

3.0 STUDENT OUTCOMES: ACHIEVEMENT................................................103.1 Were the TMS students and the control group students equivalent?........103.2 Did the TMS program have a positive impact on student achievement?...11

3.2.1 After the first year of TMS services to the school................................113.2.2 After the second year of TIS services.................................................153.2.3 Achievement for the cohort with three years of test score data...........213.2.4 Achievement by background characteristics.......................................22

4.0 STUDENT USE OF COMPUTERS.............................................................274.1 Student web-survey responses..................................................................27

5.0 TEACHER OUTCOMES.............................................................................305.1 Introduction to teacher outcomes...............................................................305.2 Using technology to study technology.......................................................30

5.2.1 Teacher direction of student computer use.........................................315.2.2 Teacher own use of computers...........................................................34

5.3 Teachers' attitudes and opinions about technology (second year)............375.3.1 Changes in the TMS program and the samples of teachers...............375.3.2 Self-reported confidence/expertise after 2 years.................................375.3.3 Growth in positive attitudes toward technology 2004-2006.................395.3.4 Growth in proficiency with technology 2004-2006...............................415.3.5 Teacher use of computers...................................................................455.3.6 Teacher-directed student use of technology during class time...........465.3.7 Teacher-directed student use of technology to create products..........475.3.8 Teacher use of computers for productivity..........................................485.3.9 Continuing effects................................................................................495.3.10 The “turnkey” effect: Teachers training teachers..............................50

6.0 RESEARCH AND EVALUATION METHODS............................................52

6.1 The R&E methods used in this inquiry.......................................................526.1.1 Methods..............................................................................................52

6.1.1.1 Data collection..............................................................................526.1.1.2 The Pilot study..............................................................................546.1.1.3 Results of the data collection methods.........................................55

6.1.2 Threats to validity................................................................................586.2. Comments about the methods of this study..............................................59

6.2.1 The practical importance of evaluation research methods..................596.2.2 The limits of conventional methods for the study of technology integration into classroom instruction...........................................................606.2.3 New methods for the study of the integration of technology into classroom instruction....................................................................................60

7.0 RECOMMENDATIONS...............................................................................617.1 Recommendation #1: Use learning technology to improve student achievement....................................................................................................617.2 Recommendation #2: Apply a critical mass of a proven model of professional development................................................................................617.3 Recommendation #3: Calibrate expectations and strategies by curriculum topics...............................................................................................................627.4 Recommendation #4: Use technology to measure technology.................627.5 Recommendation #5: Use evaluation results to improve practice............62

APPENDICES.....................................................................................................65A. Questionnaires: Technology Integration Specialists (EOY)........................65B. Conventional questionnaires: Teachers (EOY)..........................................65C. Teachers random-interval, pager-triggered web survey..............................65D. Students random-interval, pager-triggered web survey...............................65E. About Interactive, Inc..................................................................................65

TablesTable 1: Higher Achievement for Each Year in the TMS Schools.........................2Table 2: Higher Achievement by How Much TMS Service Was Provided.............3Table 3: State-Sponsored Training for TISs: The First Summer (2004)...............8Table 4: Student Cohorts....................................................................................10Table 5: Reading/Language Arts (Spring 2004 - Pretest)....................................10Table 6: Mathematics (Spring 2004 - Pretest).....................................................11Table 7: Mathematics (Spring 2004 and Spring 2005)........................................12Table 8: Reading/Language Arts (Spring 2004 and Spring 2005).......................12Table 9: Mathematics Tests of Between-Subjects Effects...................................12Table 10: Mathematics Estimated Marginal Means.............................................13Table 11: Reading/Language Arts Tests of Between-Subjects Effects...............14Table 12: Reading/Language Arts Estimated Marginal Means...........................14Table 13: Mathematics (Spring 2005 and Spring 2006)......................................16Table 14: Reading/Language Arts (Spring 2005 and Spring 2006).....................17Table 15: Mathematics Tests of Between-Subjects Effects.................................18

Table 16: Mathematics Estimated Marginal Means.............................................18Table 17: Reading/Language Arts Tests of Between-Subjects Effects...............19Table 18: Reading/Language Arts Estimated Marginal Means...........................20Table 19: Mathematics Tests of Between-Subjects Effects.................................21Table 20: Mathematics Estimated Marginal Means.............................................21Table 21: READING/LANGUAGE ARTS.............................................................22Table 22: Mathematics Achievement:.................................................................22Table 23: Mathematics Achievement:.................................................................23Table 24: Mathematics Achievement:.................................................................24Table 25: Reading/Language Arts Achievement:................................................24Table 26: Reading/Language Arts Achievement:................................................25Table 27: Reading/Language Arts Achievement:................................................26Table 28: Student Computer Use: Pager + Web-Survey Reports.......................27Table 29: Place of student computer use:...........................................................28Table 30: Student reports of curriculum use of computers:.................................29Table 31: Teacher and Student Pager + Web-Survey Reports...........................29Table 32: Teacher and Student Pager + Web-Survey Reports...........................29Table 33: Frequency of Student Computer Use by Month:.................................31Table 34: Numbers of Students Using Computers:.............................................34Table 35: Locations of Student Computers:........................................................34Table 36: Curriculum Area of Student Computer Use:........................................35Table 37: Frequency of Teacher Computers Use by Month:...............................36Table 38: The Applications Used in Classrooms:................................................37Table 39: Teacher confidence about the future use............................................38Table 40: Teachers ability to use technology independently: Most to least.........39Table 41: Analysis of Variance on Future and Independent Use of Technology

Between Teachers with and without TIS Assistance....................................39Table 42: “Indicate how much you agree or disagree.........................................40Table 43: Technology Attitudes Scale.................................................................41Table 44: Teacher changes in attitudes toward technology................................41Table 45: Teachers' Self-Estimate of Computer Expertise..................................42Table 46: Changes in teacher self-reported computer expertise over time and

duration of TMS services..............................................................................42Table 47: “Please tell us how proficient you are with each of the following

computer applications:”................................................................................43Table 48: Teachers' Self-Reported Proficiency with Technology Applications

Over Two Years by Study Condition............................................................43Table 49: Repeated Measures Analysis of Variance...........................................44Table 50: “Please tell how good you are at integrating the following software

programs into the curriculum:”......................................................................44Table 51: Technology Integration Scale..............................................................45Table 52: Teacher directed frequency of student use of technology...................46Table 53: Teacher-directed Student use of Computers.......................................46Table 54: Teacher report of frequency of requests for student products using

technology....................................................................................................47Table 55: Teacher-directed Student use of Technology to Create Products.......47

Table 56: Teacher report of frequency of use of computers for own work..........48Table 57: Teacher Use of Computers for their Own Work...................................48Table 58: Teacher self-reports of turnkey training: Most to least........................50Table 59: Teacher self-ranking of classroom technology integration:.................50Table 60: Local Agencies Ranked by Most-to-Least Improvement in Teacher

Classroom Technology Integration Factors..................................................63Table 61: Teacher Classroom Technology Integration Factors...........................64

FiguresFigure 1: Computer Use: Student Pager + Web-Survey Reports (TMS/treatment

vs Control Schools)......................................................................................28Figure 2: Percents of students using computers per day per month: Teacher

pager + Web Survey Responses.................................................................32Figure 3: Teacher Computer Use by Month:.......................................................36Figure 4: TrueActive Setup Screen.....................................................................54

AuspicesThis analysis was prepared under a contract from the West Virginia Department of Education to Interactive, Inc. While Interactive, Inc. is grateful for the support, encouragement and counsel of the West Virginia Department of Education, the Company is responsible for the conduct of this work as a third-party evaluator. This research, the analysis, interpretations and recommendations are the sole responsibility of Interactive, Inc.

AcknowledgementsThis work was performed in connection with a grant awarded to the West Virginia Department of Education from the US Department of Education, PR Award # S318A040014, the Evaluating State Educational Technology Program (ESTEP).

THE RESULTS OFPROFESSIONAL DEVELOPMENT ABOUT TECHNOLOGY:


1.0 SUMMARY This is the final report of a three-year analysis of West Virginia's Technology Model Schools program. The study was supported by a grant from the US Department of Education to the West Virginia State Department of Education: Interactive, Inc. was the R&E contractor for the state. Documenting the multiple effects of professional development on the integration of technology into classroom instruction by elementary school teachers is the chief purpose of this analysis.

The Technology Model Schools program dramatically increased the use of technology by teachers and students in classrooms and that increase is associated with gains in Mathematics and Reading/Language Arts.

Why train teachers about technology?Questions Answers

Does professional development about technology improve test scores?

Yes, in Reading/Language Arts and in Mathematics for 4th and 5th graders.

Does professional development about technology contribute to closing the achievement gap?

In Math, Title I-eligible students in versions of the TMS program outperformed others.In Reading, Title I-eligible students did as well as others.

Does professional development increase the use of technology in classrooms?

Yes, as a result of the TMS model, trained teachers used technology 22% of the classroom day compared to 1% for untrained teachers.And, TMS students used computers twice as much as students without the program’s support.

How much professional development is necessary?

Assistance across at least one academic year.

How should it be delivered? In schools, in classrooms, on demand.Who should deliver it? Classroom teachers with special

training to work with their adult colleagues.

How much and for what do the trained teachers ask students to use computers in the classroom?

One-fifth (21%) of the school day and that use is targeted on Reading/Language Arts.

How much do the technology-trained teachers use computers?

One-fifth (22%) of the school day including intensive use of productivity applications.

Are there other benefits for teachers? TMS teachers used their own computers more, more expertly and for more productivity applications than others.

This is a detailed analysis of the ability of a professional development program to impact the classroom integration of technology and, through that, the achievement of students. The study is unusual in its: (1) multiple, independent and corroborating data sources; (2) statewide scope; and (3) multi-year horizon.

This evaluation answers the following questions. As a result of the Technology Model Schools program:

1. Did student achievement go up and for whom?2. How did students use their computers?3. Were teachers benefited in their instructional and other duties?4. Was the turnkey expectation (that teachers would train other teachers)

realized?5. Did the effects of the TMS program continue beyond the departure of the

TISs?6. Did the program---as a whole---work?7. Can the Technology Model Schools program be replicated?

1.1 Was student achievement in TMS schools higher than in other schools?

Over the two years of this study, teacher professional development from the Technology Model Schools program was delivered in three configurations. In general, the achievement of students whose teachers had any of those three versions of the program's help was higher than students in classes without the program. And, in general, two years of TMS help is associated with higher achievement than one year or no years of TMS help.

The table below summarizes the points at which students in schools served by TISs scored higher than their counterparts in control schools in Math and Reading/Language Arts. [Note: In order to account for the differences among schools at the beginning of the study, we applied analyses of covariance. That allows us to compare scores between schools in different conditions and to comment on which group has higher achievement at points in time, for example “Scores from group X were 620 and scores from Group Y 604”.]

Table 1: Higher Achievement for Each Year in the TMS SchoolsDifferences after the 1st

Year(EOY 2004 - EOY 2005)

Differences after the 2nd

Year(EOY 2005 - EOY 2006)

Math Yes YesReading/Language Arts No Yes

Table 2: Higher Achievement by How Much TMS Service Was ProvidedOne Year of TIS Service(EOY 2006 from the first

year TIS schools) Two Years of TIS ServiceMath Yes YesReading/Language Arts No Yes

For Math, the performance of students in different configurations of the Technology Model School program was consistently higher than in the control schools. That was not the case for Reading/Language Arts.

We examined whether or not the TMS program was helpful to students from low-income, Title I-eligible backgrounds. For Math achievement, the estimate for the two-year treatment group (TIS both years) and the control and treatment group (TIS 2nd year only) is higher than for the control group (no TIS either year). Students eligible for Title I services had higher math achievement in both the one-year and the two-year TMS schools than their counterparts in the control schools.

For Reading/Language Arts achievement the estimate for one-year treatment/turnkey group (TIS in the first year only) and the control-plus-treatment group1 (TIS 2nd year only) is higher than for the control group (no TIS either year). In two of the three TMS-service delivery groups, there were no statistically significant differences in Reading achievement between students who were and were not eligible for Title-1 assistance. To that extent, the TMS program may be contributing to closing the achievement gap.

1.2 How did students use their computers?

TMS students used computers twice as much as students without the program’s support and they used them significantly more for Reading/Language Arts.

1.3 Were teachers benefited in their instructional and other duties?

The TMS trained teachers were much more confident in their expertise and in their independent use of technology than were the others. The TMS teachers used their own computers more and used them more for productivity applications.

1.4 Was the turnkey expectation realized?

All the teachers in the state, whether or not they were in TMS schools, report that they share technology hints with their colleagues.

1 Some schools that began as control sites added TMS services in the second year.

1.5 Did the effects of the TISs survive their departure?

Yes, while teachers' use of computers, in the control group schools, plummeted from 9% of the school day to 1% of the school day between the first and second years of this study, for the TMS teachers it remained constant at 22% of the day. Second, confidence (or expertise) in their own use of computers and in the prospective contribution of computers increased among the TMS teachers from year to year but not among the other teachers.

1.6 Did the new methods for evaluating the outcomes of professional development work?

As a result of the methods we used, we have the first independently verified and objective estimate of how much teachers and students are using technology in classrooms---as a result of sustained professional development using the TMS model.

1.7 Did the program---as a whole---work?

Yes, the TISs performed according to plan and they got better at their work over the two years. Technology use increased due to their training. Student achievement increased in the TMS schools and the teachers have continued to use technology after the departure of the TISs. The data indicate that: (1) any version of the TIS service was preferable to none; (2) two years of help was better than one; and (3) the TIS changed their services to become more effective in the second year than in the first. Thus, the Technology Model Schools program has helped teachers and students be more successful.

1.8 Can the Technology Model Schools program be replicated?

Yes. This analysis supports five recommendations about practice improvement.

Recommendation #1: Use learning technology to improve student achievementThis is the second large-scale, multi-year study, in West Virginia, to document that state policies and practices, consistently applied, do improve student achievement. West Virginia demonstrates the conditions necessary to school improvement with technology.

Recommendation #2: Apply a critical mass of a proven model of professional developmentThe state created a cadre of classroom teachers and trained them in skills relevant to the state's other classroom teachers. Those Technology Integration Specialists worked (1) on-site, (2) in classrooms and (3) on-demand.

Recommendation #3: Calibrate expectations and strategies by curriculum topicsAlthough learning-related technology can improve several subject matters, the relationship is neither even nor well understood. Educators seeking to apply

technology should remain sensitive to likely modifications by subject matter and by level of schooling.

Recommendation #4: Use technology to measure technologyThis study demonstrates the feasibility of going beyond self-report and retrospective data. The West Virginia example demonstrates that it is possible for cost-effective evaluation research to inform practice. .

Recommendation #5: Use evaluation results to improve practiceThe West Virginia Department of Education has been refining its technology support policies for more than a decade. The successive improvements in the Technology Model Schools program and the state's general policies and practices demonstrate the value of data-driven decision-making.

Final report 061807 6

2.0 THE TECHNOLOGY MODEL SCHOOLS PROGRAM This is an analysis of the teacher and student outcomes associated with West Virginia’s Technology Model Schools program, a full-featured and statewide initiative to apply professional development to increase technology use and thus student achievement2. In 2005, the National School Boards Association asked districts, "What is the biggest challenge facing your school district in the area of technology?" The number one response (45% of the districts) said, "Integrating technology into the classroom"3 which is a polite way to say, "We bought it, now we need to figure out how to get teachers to use it". The next sections discuss the national and state backgrounds of this program followed by a description of the program's operational features.

2.1 The US Department of Education and support for educational technology

Advocates have sought to harness teaching and learning technology to school improvement. And, most believe that professional development is the key to getting technology used in classrooms. Thirty-four states have standards for teachers that include their use of technology and 12 states require technology-related professional development4. In 2000, 70% of all teachers reported participating in some kind of professional development about technology5 but in 2005 district officials estimated that only 13% of their teachers were "well prepared to integrate technology into the classroom to improve academic learning"6.

The following is excerpted from a review of Enhancing Education Through Technology's (EETT, Title IID of NCLB) evaluation requirements by the U.S. Department of Education, Office of Policy, Evaluation, and Program Development, Policy and Program Studies Service.

Each year since fiscal year (FY) 2002, the U.S. Department of Education (the Department) has awarded educational technology block grants to the states through the Enhancing Education Through Technology (EETT) program, as authorized under Title II D of the No Child Left Behind Act of 2001 (NCLB). EETT allocated $496 million to states in FY 2005, and it is the second-largest federal educational technology program after the eRate program, which provides funding specifically for computer

2 The grant to the West Virginia Department of Education is part of the US Department of Education’s ESTEP Program (Evaluating State Education Technology Programs). (PR#: S318A030014)3 NSBA 2005 Technology Survey Results, October 27, 2005, National School Boards Association, Alexandria, Virginia. [email protected] 4 Education Week, “Pencil’s Down: Technology’s Answer to Testing”, May 8, 2003, p 59.5 Education Week, ibid, p 59.6 NSBA, ibid.

mailto:[email protected]


networking and Internet access. As stated in NCLB, the primary goal of EETT is ‘to improve student academic achievement through the use of technology in elementary schools and secondary schools.’ Additional goals of the program include assisting all students in becoming technologically literate by the eighth grade and encouraging the effective integration of technology resources and systems with teacher training and curriculum development to establish research-based instructional methods that can be widely implemented as best practices by State educational agencies and local educational agencies.

…[And] at least 25 percent of EETT funds must be spent on professional development opportunities for teachers, principals, and school administrators in order to develop their capacity to integrate technology effectively into teaching and learning practices.7

2.2 The West Virginia Department of Education and support for educational technology

West Virginia is among America’s most experienced states in advancing student achievement with learning technology. Beginning with two sequenced programs---Basic Skills/Computer Education (BS/CE) for elementary grades and Project SUCCESS for secondary grades---West Virginia pioneered the application of critical masses of technology resources to student learning8. BS/CE deployed intensive waves of hardware and software coupled to targeted professional development. Of the 50 American states, only West Virginia has had is students' use of computers measured at the highest interval ("a lot") exceed the national average in every year from 1997 to 2003. The lessons learned from the State's earlier initiatives were reflected in the intervention being evaluated---the "Technology Model Schools Program, School-Based Specialized Training”.

2.3 The Technology Model Schools program

West Virginia’s Office of Technology and Information Systems created the Technology Model Schools program (TMS) to help teachers integrate learning technology into curriculum and instruction. Using EETT funds, grants of $150,000/year were made on a competitive basis to counties. The key feature was the training in 2004-05 and school-based deployment of “technology integration specialists” (TISs). In the second year (2005-06), the teachers trained in the first wave of TIS assistance were expected to take over additional

7 Source: U.S. Department of Education, Office of Policy, Evaluation, and Program Development, Policy and Program Studies Service, Summary of the U.S. Department of Education’s Evaluation Institute for State Educational Technology Directors, Washington, D.C., 2005 page 1.8 Mann, D., Shakeshaft, C., Becker, J., and Kottkamp, R., (1999) The West Virginia Story: Achievement Gains from a Statewide Comprehensive Instructional Technology Program, Milken Family Foundation, Santa Monica, CA. (http://web.mff.org/pubform.taf)

http://web.mff.org/pubform.taf


dissemination, also through professional development in this case peer-to-peer. That “turnkey” expectation is a question for this study: it assumed that the school staff would (1) continue integrating computer-related technology into their instruction and (2) train additional teachers.

The WV initiative was linked to the state's content standards, which are, themselves aligned to its Adequate Yearly Progress benchmarks for NCLB. In 2002-03, the State jury-selected the best teacher-developed standards-aligned content and placed it on the WVDE website.

The TMS program reflects general agreement about professional development best practice. Counties selected 18 teachers for the TIS role. The state provided 40-days of (off site) training on classroom-relevant instructional and other technology. The county grantee then deployed the newly-trained TISs on a part-time or full-time basis to work in schools and classrooms in response to the requests of classroom teachers. Thus, the TMS program used homophilous trainers and delivered just-in-time assistance on-site and during the school day.

Federal guidelines require that EETT money be distributed in connection with a competitive grant proposal review and selection process. The WVDE specified that applicants must use the funds to hire and deploy Technology Integration Specialists but left other aspects of their work to the counties. For example, 15 of the 18 TISs selected by their counties for that role had no previous experience as technology experts (only one of the 18 was previously known by the State's technology group9). None were experienced at providing professional development to their peers. Counties also decided how much service a TIS could give to a school. The availability of TISs in schools ranged from the full-time equivalent of 20% to 100%: five of the TISs worked full-time in single schools, others were split among 3, 4 and 5 schools.

The Department took responsibility for training the TISs prior to the inception of their local work. The training was centrally provided over one residential summer month: the major components of that training were:

Table 3: State-Sponsored Training for TISs: The First Summer (2004)1. Teacher Universe – Lesson Planning (major emphasis)2. Teacher Universe – Lesson Preparation (major emphasis)3. TIS Home Page Construction (major emphasis)4. Technology Integration Modules5. Kidz Online6. Try Science/Digital Story-Telling7. Web Discoveries8. Reinventing Education – Lesson Planning9. SAS in Schools (SIS)

9 Ten of the 18 had been elementary school teachers, two were Title I literacy specialists and one was a librarian.


10.M3 Online

The Department created a schedule of “Collaborative Exchanges” that moved among the participating jurisdictions and challenged all the TISs to identify and share best practices in at least two Collaborative Exchanges.

The TISs were convened periodically for additional training and information exchanges. Classroom teachers and school administrators were also invited to annual, late summer State Technology Conferences.

Finally, the WVDE actively managed the TMS program. Two professionals were assigned to the program, nearly full-time. They created and delivered the training, monitored the school activities and provided technical assistance at the county, school and sometimes classroom level. The Department required TISs to complete a weekly log of their program-related activities. Monitoring of those reports by WVDE employees was intended to encourage service delivery and accountability at the local and state levels. The Department employees assigned to the program also helped trouble-shoot the evaluation, for example, answering questions about pagers or the desktop meters.


3.0 STUDENT OUTCOMES: ACHIEVEMENT This study was designed as an analysis of the state's 4th and 5th grade students over two years (computers are often used to augment learning in those grades). Thus, over the two years of the study, there were three cohorts of students. Throughout this section, we refer to the cohorts as indicated in the following table.

Table 4: Student CohortsSPRING 2004

(Pretest)SPRING 2005

(Posttest 1)SPRING 2006

(Posttest 2)COHORT 1 4th Grade 5th GradeCOHORT 2 3rd Grade 4th Grade 5th GradeCOHORT 3 3rd Grade 4th Grade

3.1 Were the TMS students and the control group students equivalent?

The groups were not equivalent---the schools originally identified as control schools began with higher achievement in Reading/Language Arts and Mathematics. State department officials and Interactive, Inc. worked together to identify schools that would match the student achievement characteristics of the TMS/treatment schools as closely as possible at baseline (Spring 2004). The treatment schools were those that: (1) were in need of improvement10; and (2) competed for and won grants to support state-defined TMS program activities. After the state asked schools to serve as comparison sites, a few opted out of the study and that diminishes the validity of the TMS-control comparisons11. The table below indicates the statistically significant differences between the two sets of schools at the beginning of the study--- the Spring 2004 WESTEST ("West Virginia Education Standards Test Results") scores for the (then 3rd and 4th grade) students. The next table displays the Reading/Language Arts test scores of the two groups of schools at the beginning of the study.

Table 5: Reading/Language Arts (Spring 2004 - Pretest)

Cohort/ Grade Condition N Mean

Std. Deviation

Std. Error Mean

Cohort 2 - 3rd Grade

Scaled Score Reading/Language

Control 996 631.7781 38.90424 1.23273TMS 1289 627.2459 39.89267 1.11113

Cohort 1 – 4th Grade

Scaled Score Reading/Language

Control 1040 645.4212 34.11910 1.05799TMS 1047 638.0334 39.01794 1.20584

10 Because the WVDE adhered to federal requirements in selecting low-performing agencies for the EETT grants, it was to that extent more difficult to find other, equally low-performing agencies to serve as controls.11 Collisions between empirical desirability and school priorities are common in researching school programs. The WVDE did everything possible to encourage schools to take part as (uncompensated) control sites. Interactive, recognizes the tension and respects the decisions of school professionals.


For cohort 2, the difference between the mean Reading/Language Arts score for the control group students (631.78) and the TMS school students (627.25) is statistically significant (t=2.722, p = .007). Also, for cohort 1, the difference between the mean Reading/Language Arts score for the control group students (645.42) and the TMS school students (638.03) is statistically significant (t=4.603, p < .001).

Table 6: Mathematics (Spring 2004 - Pretest)

Cohort/ Grade Condition N Mean

Std. Deviation

Std. Error Mean

Cohort 2 - 3rd Grade

Scaled Score Math

Control 996 620.3554 33.91008 1.07448TMS 1289 615.9845 33.42117 .93088

Cohort 1 – 4th Grade

Scaled Score Math

Control 1039 643.5419 29.29594 .90887TMS 1047 639.8329 31.27115 .96643

For cohort 2, the difference between the mean Mathematics score for the control group students (620.36) and the TMS school students (615.98) is statistically significant (t=3.080, p = .002). Also, for cohort 1, the difference between the mean Mathematics score for the control group students (643.54) and the TMS school students (639.83) is statistically significant (t =2.795, p = .005).

To account for the different starting points between the two groups, we used analysis of covariance (ANCOVA) in examining student achievement. That is, we examine posttest scores while controlling for pretest scores (i.e. holding pretest scores constant). Essentially, this approach computes estimated posttest scores (while holding pretest scores constant) and tests for differences between the groups on those estimated marginal mean (posttest) scores.

3.2 Did the TMS program have a positive impact on student achievement?

The TMS program is associated with gains in student achievement in some circumstances.

We have data on the same students in the same schools over two years of TMS program operation. One benefit of multi-year data is that it allows us to look at the cumulating impact of technology with students. Also, the program was fielded differently in the second year than in the first and those changes can be associated with changes in achievement.

3.2.1 After the first year of TMS services to the school

The following two tables show the mean scores for the two cohorts of students on the baseline test (Spring 2004) and the first posttest (Spring 2005).

Table 7: Mathematics (Spring 2004 and Spring 2005)


Cohort Condition Year of Exam MeanStd.

Deviation NCohort 2 - 3rd Grade (2004) to 4th Grade (2005)

Control Scaled Score Math (2004) 620.3554 33.91008 996

Scaled Score Math (2005) 649.1708 31.60417 1289

TMS Scaled Score Math (2004) 615.9845 33.42117 1054


Cohort 1 - 4th Grade (2004) to 5th Grade (2005)

Control Scaled Score Math (2004) 643.5419 29.29594 1040


TMS Scaled Score Math (2004) 639.8329 31.27115 1084


Table 8: Reading/Language Arts (Spring 2004 and Spring 2005)



Control Scaled Score Reading (2004)

631.7781 38.90424 996

Scaled Score Reading (2005)

648.2372 35.71615 1289

TMS Scaled Score Reading (2004)

627.2459 39.89267 1054


646.2365 37.61492 1353


Control Scaled Score Reading (2004)

645.4212 34.11910 1040


659.3653 33.59235 1047

TMS Scaled Score Reading (2004)

638.0334 39.01794 1084


653.1776 38.56482 1098

The next four tables show the results for the analyses of covariance; the first two for Math scores, the last two for Reading/Language Arts.


Table 9: Mathematics Tests of Between-Subjects Effects

Dependent Variable: Scaled Score Math (2005)

Cohort SourceType III Sum of Squares df

Mean Square F Sig.

Partial Eta

Squared

Cohort 2 - 3rd Grade (2004) to 4th Grade (2005)

Corrected Model 1529379.60a 2 764689.802 1600.55

4 .000 .584

Intercept 204088.806 1 204088.806 427.173 .000 .158Scaled Score Math (2004) 1529205.773 1 1529205.77

33200.74

3 .000 .584

TMS 8595.775 1 8595.775 17.992 .000 .008Error 1090261.593 2282 477.766Total 967210716.00

0 2285

Corrected Total 2619641.197 2284Cohort 1 - 4th Grade (2004) to 5th Grade (2005

Corrected Model 1418956.21b 2 709478.106 1916.10

1 .000 .648


73829.59

1 .000 .648

TMS 1872.897 1 1872.897 5.058 .025 .002Error 771646.301 2084 370.272Total 921857226.00

0 2087

Corrected Total 2190602.514 2086a. R Squared = .584 (Adjusted R Squared = .583)b. R Squared = .648 (Adjusted R Squared = .647)

The tests of between-subjects effects indicate that, for the cohort 2 (the students who moved from 3rd to 4th grade), the difference between the estimated marginal mean Math posttest score for the control group students (647.51) and the TMS school students (651.43) is statistically significant (F=17.992, p < .001). Also, for cohort 1 (students moving from 4th to 5th grade) the difference between the estimated marginal mean Math posttest score for the control group students (662.87) and the TMS school students (664.87) is statistically significant (F=5.058, p = .025).

Table 10: Mathematics Estimated Marginal MeansDependent Variable: Scaled Score Math (2005)

Cohort Condition Mean Std. Error95% Confidence Interval

Lower Bound Upper BoundCohort 2 - 3rd Grade (2004) to 4th Grade (2005)

Control 647.512(a) .693 646.153 648.872

TMS 651.432(a .609 650.237 652.627


)Cohort 1 - 4th Grade (2004) to 5th Grade (2005)

Control 662.872(b) .598 661.700 664.044

TMS 664.771(b) .595 663.604 665.937

a. Covariates appearing in the model are evaluated at the following values: SS Math (2004) = 617.8897.b. Covariates appearing in the model are evaluated at the following values: SS Math (2004) = 641.6363.

Thus, for both cohorts of students, the differences in Math (posttest) scores at the end of the first year are statistically significant and in favor of the TMS students. Students moving from 3rd to 4th grades gained slightly more than those who went from 4th to 5th grades.

Table 11: Reading/Language Arts Tests of Between-Subjects Effects

Dependent Variable: Scaled Score Reading (2005)

Cohort SourceType III Sum of

Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model 2171938.654(a) 2 1085969.32

72748.30

4 .000 .707

Intercept 218312.079 1 218312.079 552.491 .000 .195Scaled Score Read. (2004) 2168193.991 1 2168193.99

15487.13

0 .000 .706

TMS 551.882 1 551.882 1.397 .237 .001Error 902108.477 2283 395.142Total 961045817.00

0 2286

Corrected Total 3074047.131 2285Cohort 1 - 4th Grade (2004) to 5th Grade (2005)

Corrected Model 1834808.607(b) 2 917404.303 2348.13

0 .000 .693


54659.80

1 .000 .691

TMS 275.193 1 275.193 .704 .401 .000Error 814209.822 2084 390.696Total 902060913.00

0 2087

Corrected Total 2649018.429 2086a R Squared = .584 (Adjusted R Squared = .583)b R Squared = .648 (Adjusted R Squared = .647)

The tests of between-subjects effects indicate that, for cohort 2, the difference between the estimated marginal mean Reading posttest score for the control group students (646.79) and the TMS school students (647.78) is not statistically


significant (F=1.397, p = .237). Also, for cohort 1 the difference between the estimated marginal mean Reading posttest score for the control group students (656.11) and the TMS school students (656.84) is not statistically significant (F=0.704, p = .401).

Table 12: Reading/Language Arts Estimated Marginal MeansDependent Variable: Scaled Score Reading/Language Arts (2005)

Cohort Condition Mean Std. Error95% Confidence Interval

Lower Bound Upper BoundCohort 2 - 3rd Grade (2004) to 4th Grade (2005)

Control 646.789(a) .630 645.552 648.025

TMS 647.781(a) .554 646.695 648.867


Control 656.109(b) .614 654.904 657.314

TMS 656.839(b) .612 655.638 658.040

a. Covariates appearing in the model are evaluated at the following values: SS Reading (2004) = 617.8897.b. Covariates appearing in the model are evaluated at the following values: SS Reading (2004) = 641.6363.

For both cohorts of students, the differences in Reading (posttest) scores at the end of the first year are not statistically significant.

In general, student achievement in the program's second year is stronger than in the first year (see discussion below). Because we know that achievement is related to the presence or absence of TIS services, it is reasonable to ask what were the specialists doing? It turns out that their work during the first and second years was very different.

Recall that 15 of the 18 people recruited by their counties to be Technology Integration Specialists had no prior experience with technology and none had ever served as teacher trainers. The TISs told us how reluctant they were to engage their former colleagues as "experts" during the first year. Teaching adults is, after all, different from teaching children. Instead, the TISs spent most of the first year setting up computers, installing software and trouble shooting applications. It was only in the second year that the cumulation of training from the state and their on-the-job experience encouraged them to press the teacher training aspect of their role.

3.2.2 After the second year of TIS services After the first year of the study, a series of judgments by local educators seeking to improve schooling compromised the quasi-experimental design. (All the judgments represented a vote of confidence in the TMS program.) First, some of the original control group schools applied for and received EETT grants for TISs in the second year.


Second, some the first year experimental schools continued their TISs into an additional year. The "turnkey assumption" was part of the original design of the Technology Model Schools program and one of the questions for this study. The plan had been that after a year of specialized assistance, the TISs would move on, the "key would turn" and the faculty would be sufficiently confident and competent to (1) persist in what they had been shown during the TIS year and (2) train their colleagues. From the methods point of view, the treatment would end and Interactive, Inc. would measure how much they continued to use the technology after the TMS services had moved to another school and how much the originally trained teachers would train their colleagues. Instead, some schools in the original treatment group extended the services of their TIS either through a second round of EETT funds or through locally generated funds12.

To accommodate those world-of-practice circumstances, we divided our study schools into four groups:

1. The one-year treatment/turnkey group: a TIS in year 1 but not in year 22. The two-year treatment group: TISs in both years 3. The control and treatment group: no TIS in year 1 but a TIS in year 213 4. The control group: no TIS in either year

The local decisions cut the size of our sample groups and complicated the analysis. But, they also allowed us to ask, "Is two years of TMS service better than one?" and, "Do the effects of the TMS services differ between the first and second years?"

Also, for the second year of the study, cohort 1 graduated to the 6th grade and out of this analysis. At the same time, cohort 3---3rd graders in 2005---entered the analysis.

The following tables show the mean scores for the two cohorts of students on the Spring 2005 WESTEST (i.e. the first posttest for cohort 2 and the pretest for cohort 3) and the final posttest (Spring 2006 WESTEST).

Table 13: Mathematics (Spring 2005 and Spring 2006)

Cohort ConditionYear of Exam Mean

Std. Deviation N


Control Scaled Score Math (2005)

621.3791 35.70827 517

Scaled Score Math (2006)

646.6544

33.81250 515

12 This is another example of school practice over research procedure. As is common in federal systems of government, the state department honored local judgment about school practice. The alternative would have denied services in pursuit of school improvement in order to preserve a research design.13 For a complete discussion of the sample changes over time, please see the Appendix.


One-Year Treatment/Turnkey


627.0895 36.77641 961


654.1417 38.19265 960

Two-Year Treatment


622.7121 41.77322 330


655.4878 35.32211 328

Control and Treatment


622.2102 34.90431 333


651.6517 33.12352 333


Control Scaled Score Math (2005)

647.9783 33.25249 506


670.5494 32.60320 506



648.4944 31.96210 710


671.2370 30.96741 709

Two-Year Treatment


653.5861 33.60185 360


673.7632 33.18439 359



651.0826 30.50771 327


675.2382 31.50142 319

Table 14: Reading/Language Arts (Spring 2005 and Spring 2006)



Control Scaled Score Reading (2005) 631.1431 35.88962 517

Scaled Score Reading (2006) 646.7825 36.10904 515

One-Year Treatment/ Turnkey



Two-Year Treatment






Cohort 2 - Control Scaled Score 649.4427 35.07883 506


4th Grade (2005) to 5th Grade (2006)

Reading (2005)Scaled Score Reading (2006) 660.4249 33.19118 506

One-Year Treatment/ Turnkey



Two-Year Treatment






The next two tables show the results for the analyses of covariance, looking at the Spring 2006 WESTEST scores while holding constant the Spring 2005 scores. NOTE: in the tables below, “TMS Duration” is the name for the 4-category grouping variable.

Table 15: Mathematics Tests of Between-Subjects EffectsDependent Variable: Scaled Score Math (2006)


Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1618136.276(a) 4 404534.069 740.778 .000 .582


22920.81

3 .000 .578

TMS Duration 12461.018 3 4153.673 7.606 .000 .011Error 1163725.431 2131 546.094Total 911235681.00

0 2136

Corrected Total

1618136.276(a) 4 404534.069 740.778 .000 .582


Corrected Model

1225565.584(b) 4 306391.396 819.539 .000 .635


03262.48

5 .000 .633


0 1893

Corrected 1931410.239 1892


Totala R Squared = .582 (Adjusted R Squared = .581)b R Squared = .635 (Adjusted R Squared = .634)

The tests of between-subjects effects indicate that, for Cohort 3, there are differences between the estimated marginal mean Math posttest scores by condition (i.e. across the four groups).

Table 16: Mathematics Estimated Marginal MeansDependent Variable: Scaled Score Math (2006)

Cohort Condition Mean Std. Error

95% Confidence IntervalLower Bound

Upper Bound


Control 648.811(a) 1.031 646.790 650.832


652.091(a) .755 650.610 653.572

Two-Year Treatment 656.555(a) 1.290 654.025 659.086


653.176(a) 1.281 650.664 655.688


Control 671.926(b) .860 670.240 673.612


672.116(b) .726 670.692 673.541

Two-Year Treatment 670.762(b) 1.022 668.758 672.766


674.478(b) 1.083 672.354 676.601

a. Covariates appearing in the model are evaluated at the following values: SS Math (2005) = 624.2767.b. Covariates appearing in the model are evaluated at the following values: SS Math (2005) = 649.7327.

Through post-hoc pairwise comparisons, we can make the following statements about which groups differ significantly from each other:

Math Achievement for Cohort 3 The estimate for the control group (no TIS either year) is lower than all

three other groups. The estimate for the two-year treatment group (TIS for both years) is

higher than the estimate for the one-year treatment/turnkey group (TIS first year).

Math Achievement for Cohort 2 The estimate for the control and treatment group (TIS 2nd year only) is

higher than the estimate for the two-year treatment group (TIS both years).


Table 17: Reading/Language Arts Tests of Between-Subjects EffectsDependent Variable: Scaled Score Reading (2006)


Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1961926.431(a) 4 490481.608 1107.23

9 .000 .675


44415.16

4 .000 .675


0 2135

Corrected Total 2905467.567 2134


Corrected Model

1491458.633(b) 4 372864.658 1049.46

0 .000 .690


24194.34

4 .000 .690

TMS Duration 598.208 3 199.403 .561 .641 .001Error 670435.597 1887 355.292Total 824943547.00

0 1892


a. R Squared = .675 (Adjusted R Squared = .675)b. R Squared = .690 (Adjusted R Squared = .689)

The tests of between-subjects effects indicate that, for Cohort 3 there are differences between the estimated marginal mean reading posttest scores by condition (i.e. across the four groups).

Table 18: Reading/Language Arts Estimated Marginal MeansDependent Variable: Scaled Score Reading (2006)

Cohort Condition MeanStd. Error


Upper Bound


Control 645.833(a) .928 644.014 647.652


647.334(a) .680 646.000 648.667


Control and Treatment 649.899(a) 1.152 647.640 652.158



Control 659.306(b) .838 657.662 660.949


659.274(b) .708 657.886 660.663

Two-Year Treatment 658.925(b) .995 656.974 660.877

Control and Treatment 660.662(b) 1.057 658.589 662.735

a. Covariates appearing in the model are evaluated at the following values: SS Reading/Language (2005) = 629.9007.b. Covariates appearing in the model are evaluated at the following values: SS Reading/Language (2005) = 648.0085.

Through post-hoc pair-wise comparisons, we can make the following statements about which groups differ significantly from each other:

Reading Achievement for Cohort 3 The estimate for the control group (no TIS either year) is lower than the

two-year treatment group (TIS both years) and the control and treatment group (TIS 2nd year only).

The estimate for the two-year treatment group (TIS for both years) is higher than the estimate for the one-year treatment/turnkey group (TIS first year).

Reading Achievement for Cohort 2 There are no significant differences between the groups.

3.2.3 Achievement for the cohort with three years of test score data We have three sets of test scores for Cohort 2. They were in classes for both years of the research process. Thus, for Cohort 2, we utilized ANCOVA with the Spring 2006 WESTEST scores (taken as 5th graders at the end of the study) as the dependent variable and the Spring 2004 scores (taken as 3rd graders before the study began) as the covariate (i.e. holding constant the Spring 2004 scores).

Table 19: Mathematics Tests of Between-Subjects EffectsDependent Variable: Scaled Score Math (2006)


Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1165017.459(a) 4 291254.365 719.792 .000 .626


82866.07

0 .000 .625

TMS Duration 10954.588 3 3651.529 9.024 .000 .015Error 696785.025 1722 404.637Total 782738213.00 1727


0Corrected Total 1861802.484 1726

a. R Squared = .626 (Adjusted R Squared = .625)

The tests of between-subjects effects indicate that there are differences between the estimated marginal mean Math posttest scores by condition (i.e. across the four groups).

Table 20: Mathematics Estimated Marginal Means




Upper Bound


Control 670.064(a) .932 668.237 671.892


671.058(a) .802 669.485 672.631



a. Covariates appearing in the model are evaluated at the following values: SS Math (2004) = 618.3619.


For Math, the estimate for the control group (no TIS either year) is lower than the two-year treatment group (TIS both years) and the control and treatment group (TIS 2nd year only).

For Math, the estimate for the one-year treatment/turnkey group (TIS 1st year only) is lower than the two-year treatment group (TIS both years) and the control and treatment group (TIS 2nd year only).

Table 21: READING/LANGUAGE ARTSTests of Between-Subjects Effects

Dependent Variable: Scaled Score Reading (2006)


Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1306370.222(a) 4 326592.556 835.556 .000 .660


93341.04

6 .000 .660

TMS Duration 2987.844 3 995.948 2.548 .054 .004


Error 673075.324 1722 390.868Total 753468854.00

0 1727

Corrected Total 1979445.546 1726a. R Squared = .660 (Adjusted R Squared = .659)

For Reading, the tests of between-subjects effects indicate that there are no differences between the estimated marginal mean reading posttest scores by condition (i.e. across the four groups) (F = 2.548, p = .054).3.2.4 Achievement by background characteristics: All TMS students compared to controls

3.2.4 Achievement by background characteristics It is possible that the achievement of children from different backgrounds may vary by their background characteristics (here, sex and eligibility for Title I services). For Cohort 2, we ran similar analyses as in section 3.2.3 with sex and eligibility for Title I services14 included.

Girls and boys were equally benefited by the TMS program: there were no statistically significant differences in achievement by the sex of the student.

The data about Math achievement are presented in the next tables. (Note: “TMS duration” expresses the period of time or recency of Technology Model Schools services.)

Table 22: Mathematics Achievement:Tests of Between-Subjects Effects



Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1178601.879(a) 16 73662.617 183.395 .000 .631


62716.41

9 .000 .613

Sex 1193.010 1 1193.010 2.970 .085 .002Title I 41.082 1 41.082 .102 .749 .000TMS Duration 3990.832 3 1330.277 3.312 .019 .006Sex x Title I 125.836 1 125.836 .313 .576 .000Sex x TMS Duration 143.197 3 47.732 .119 .949 .000

Title I x TMS Duration

4405.497 3 1468.499 3.656 .012 .006

14 Because, in the West Virginia sample, 94% of the students are Caucasian, we did not include race in the analyses.


Sex x Title I x TMS Duration 1636.793 3 545.598 1.358 .254 .002

Error 689249.738 1716 401.661Total 785293736.00

0 1733



The tests of between-subjects effects indicate that there are differences between the estimated marginal mean Math posttest scores by condition (i.e. across the four groups defined by service delivery). There are no significant differences by sex or by Title I eligibility status.

Table 23: Mathematics Achievement:

Estimated Marginal MeansDependent Variable: Scaled Score Math (2006)



Upper Bound


Control 669.831(a) .929 668.008 671.654


671.712(a) .924 669.900 673.525



a. Covariates appearing in the model are evaluated at the following values: SS Math (2004) = 618.3619.


For Math, the estimate for the two-year treatment group (TIS both years) and the control and treatment group (TIS 2nd year only) is higher than for the control group (no TIS either year).

Also, there is a significant interaction effect between Title I eligibility and TMS duration. The TMS “treatment” has differential effects for Title I-eligible students. It is hard to determine exactly what those effects are, but the table below shows the estimated marginal means for the eight possible groups of students. Students eligible for Title I services had higher math achievement in both the one-year and the two-year TMS schools than their Title I-eligible counterparts in the control schools.

Table 24: Mathematics Achievement:


Estimated Marginal MeansDependent Variable: Scaled Score Math (2006)

Title I Condition Mean Std. Error


Upper Bound

Not Eligible Control 672.192(a) 1.311 669.621 674.763


673.080(a) 1.610 669.921 676.238



Eligible Control 667.470(a) 1.326 664.869 670.072


670.345(a) .926 668.529 672.162



a Covariates appearing in the model are evaluated at the following values: SS Math (2004) = 618.2660.

The results of a similar analysis for Reading/Language Arts are discussed next.

Table 25: Reading/Language Arts Achievement:Tests of Between-Subjects Effects

Dependent Variable: Scaled Score Reading/Language Arts (2006)


Squares dfMean

Square F Sig.

Partial Eta

Squared


Corrected Model

1321464.722(a) 16 82591.545 212.821 .000 .665

Intercept 305077.586 1 305077.586 786.119 .000 .314Scaled Score Reading (2004)

1239810.161 1 1239810.161

3194.723 .000 .651

Sex 80.727 1 80.727 .208 .648 .000Title I 330.790 1 330.790 .852 .356 .000TMS Duration 4306.514 3 1435.505 3.699 .011 .006Sex x Title I 29.480 1 29.480 .076 .783 .000Sex x TMS Duration 2343.841 3 781.280 2.013 .110 .004

Title I x TMS Duration 4514.152 3 1504.717 3.877 .009 .007

Sex x Title I x 311.648 3 103.883 .268 .849 .000


TMS DurationError 665946.408 1716 388.081Total 755850324.00

0 1733



Unlike the simpler ANCOVA, these tests of between-subjects effects do indicate that there are differences between the estimated marginal mean Reading/Language Arts posttest scores by condition (i.e. across the four groups). There are no significant differences by sex or by Title I eligibility status.

Table 26: Reading/Language Arts Achievement:

Estimated Marginal MeansDependent Variable: Scaled Score Reading/Language Arts (2006)



Upper Bound


Control 657.625(a) .913 655.834 659.416


661.384(a) .903 659.613 663.154



a Covariates appearing in the model are evaluated at the following values: SS Reading/Language (2004) = 629.6555.


For Reading/Language Arts, the estimate for one-year treatment/turnkey group (TIS in the first year only) and the control and treatment group (TIS 2nd year only) is higher than for the control group (no TIS either year).

Also, there is a significant interaction effect between Title I eligibility and TMS duration, i.e., the TMS “treatment” has differential effects for Title I-eligible students. The table below shows the estimated marginal means for the eight possible groups of students. Even though the achievement of the Title I-eligible students in the TMS schools was hampered by the low-income status of their families, there were no differences between their achievement and their non-Title 1 counterparts. To that extent, the TMS program may be contributing to closing the achievement gap in the state.


Within the one-year treatment/turnkey schools, students not eligible for Title I services did better than those who were eligible. That is unsurprising since Title I eligibility is a proxy for families with low income: those not eligible for Title I assistance are presumed to have achievement advantages.

Table 27: Reading/Language Arts Achievement:

Estimated Marginal MeansDependent Variable: Scaled Score Reading/Language Arts (2006)

Title I Condition Mean Std. Error


Upper Bound

Not Eligible Control 657.443(a) 1.286 654.919 659.966


664.910(a) 1.564 661.842 667.979



Eligible Control 657.808(a) 1.301 655.256 660.360


657.857(a) .910 656.073 659.642



a Covariates appearing in the model are evaluated at the following values: SS Reading/Language (2004) = 629.6555.


4.0 STUDENT USE OF COMPUTERS

4.1 Student web-survey responses

In the second year of the study, in addition to the teacher responses, Interactive, Inc. received 297 usable pager-triggered web survey responses from students. Those 297 responses came from 33 different classrooms. The range in the number of responses from individual classrooms was one to 31 (i.e. a few teachers only once asked a student to respond, and one teacher had students respond 31 different times).

When the teacher’s pager rang, teachers asked a randomly-designated student to fill out a parallel web-survey on a student desktop monitor. The first question that students saw was, “Before you started this survey, were YOU OR ANY OTHER STUDENTS IN YOUR CLASS using a computer?” For this question, data were aggregated to the classroom level and then weighted by the number of times a given teacher had students respond.

Over the year, students in TMS schools were twice as likely to report that they were using computers compared to students in the control group schools (35% and 18%, p < .001). On the 195 instances when students in TMS schools responded to the question, they responded, “Yes” 35% of the time. Students in control school classrooms only responded, “Yes” in 18% of the 77 instances.

We also analyzed the responses to the ‘Were you using a computer’ question by month. (Note: While the differences between TMS and other schools over the year are statistically significant, the differences within the month are not, perhaps because of the numbers of respondents.)

Table 28: Student Computer Use: Pager + Web-Survey ReportsTMS Vs Control Schools

“Before you started this survey, were YOU OR ANY OTHER STUDENTS IN YOUR CLASS using a computer?”

MonthTMS vs. Control Mean N Std. Deviation

December '05 TMS .5000 12 .52223Control .2500 16 .44721

January '06

TMS .3265 49 .47380Control .2000 15 .41404

February '06

TMS .4000 70 .49344Control .2500 24 .44233

March '06

TMS .2206 68 .41773Control .0714 28 .26227

April '06

TMS .3750 8 .51755Control .0000 2 .00000


May ‘06 TMS .6667 3 .57735Control .0000 2 .00000

Figure 1: Computer Use: Student Pager + Web-Survey Reports (TMS/treatment vs Control Schools)

One goal of the TMS program was to encourage computer use in classrooms, not in computer labs. The previous year, students in the control group were more likely to report classroom-use of computers (87% to 80%). During this (continuation) year, that reverses sharply with students in TIS schools more likely to report using computers in their classrooms (see table below).

Table 29: Place of student computer use: Student pager + web-survey reports

TMS Vs. Control Schools

PlaceTMS Schools Control Schools

Valid Percent Frequency Valid Percent FrequencyIn the classroom 87 61 80 12In a computer lab 10 7 20 3In the library or media center 1 1 0 0Somewhere else 1 1 0 0

Students were next asked about curriculum topics and computer use. About two-thirds of the student responses omitted to indicate exactly what they were doing on the computer at the moment queried. Where they did indicate a topical application, Reading was cited twice as frequently for students in TMS schools as for the control schools and that reverse the previous year’s emphasis. Similarly, where the TMS students had previously been most likely to report “Internet” as the most frequently used application, this year it was Reading.


Table 30: Student reports of curriculum use of computers: Pager + Web-Survey Reports

TMS Schools Vs. Control SchoolsTMS Schools Control Schools

Valid Percent Frequency Valid Percent FrequencyUsing the Internet 15 10 20 3A Reading program 28 19 27 4A Math program 18 13 20 3Something else 32 22 33 5I don't know 7 5 0 0

We wanted to compare teacher and student self-reports of computer use in the two types of schools. In both cases (teachers and students), the differences between the Technology Model Schools and others favor the TMS schools (i.e., there is more use) and are statistically significant. And, in both types of schools, the students report slightly more computer use than do the teachers, a use pattern not unlike other studies.

Table 31: Teacher and Student Pager + Web-Survey Reportsof Computer Use in Model Schools and Comparison Schools

Were you using a computer when paged?

% reporting “Yes” from TMS schools

% reporting “Yes” from Control schools

Teacher responses 21 14Student responses 35 18

We asked students what they had been doing before they began the survey. There appears to be less time out of class for the TMS-school students and a bit more working independently.

Table 32: Teacher and Student Pager + Web-Survey Reportsof Previous Classroom Activities: TMS Schools Vs. Control Schools

TMS Schools Control SchoolsValid Percent Frequency Valid Percent Frequency

Listening to the teacher with the rest of my class 23 48 17 15

Working in a small group of students 11 22 5 4

Working quietly by myself 22 46 12 10Taking a test 8 16 8 7Out of the classroom (recess, lunch, etc.) 29 60 43 37

Other 9 18 16 14


5.0 TEACHER OUTCOMES

5.1 Introduction to teacher outcomes.

For classroom instruction, the teacher is the narrow spot in the intersection between students and technology15. Teachers determine how much or how little technology they and/or their students will use. The West Virginia State Board’s regulations recognize teacher practice as a priority and teacher technology use is supported at the State and RESA levels. The next two sections report teacher knowledge, attitudes and behavior as they were documented by teacher responses to: (1) random-interval, pager-triggered web surveys deployed across the school year; and (2) conventional self-report questionnaires circulated early and late in the study school years.

5.2 Using technology to study technology

Studies of technology use have asked respondents to reconstruct their use after-the-fact. Even with sincere cooperation, the results are unlikely to do justice to the complexity of the classroom day and are vulnerable to inaccurate or selective recall. The reliance of evaluators on fixed response questionnaire items about technology integration into the classroom is cueing responses and misleading data analysis and thus conclusions about policy and practice. Teachers see the pre-determined range of uses and some are tempted to over-report. Other teachers who have never thought much about issues of, for example, "fluent use", are taught by the item language what that means and then respond in invalid ways. The result is data that are more artifacts of the data collection method than valid descriptions of practice. (Qualitative, f2f interviews and field visits are a useful if incomplete amelioration.)

As a supplement to the conventional retrospective self-reports, we sampled the classroom day by asking teachers to keep telephone pagers close at hand on alternate weeks and by activating16 those pagers at randomly selected intervals. When the pagers rang, teachers answered a 5-item web-survey and they asked a (randomly selected) student to complete a parallel 5-item web-survey.

The focus for the second year was “continuing effects” or "Did the TIS delivered skills last?" so we limited the pager data collection to only teachers in the "one-year treatment/turnkey” condition schools (a TIS in the first year but not in the second) or “control schools” (no TIS in either year17). 15 That is obviously less true outside the school and/or inside schools when students have independent access to the Internet, for example, in 1:1 laptop or ubiquitous computing environments.16 Teachers could set the pagers to ring audibly or to vibrate silently.17 The control schools were also "digital" schools in the first year, that is, we used technology to measure their use and integration. Teachers in three of the formerly 'digital control' schools declined to take part in the study's second year.


In Year 2, we received 319 usable random interval, pager-triggered web survey responses from 33 different teachers: in Year 1, we had 1,563 responses from 91 teachers. In the second year, we rang teachers 40 times: individual teachers provided from one to 32 responses. If a teacher responded fewer than five times, we doubted that we had sufficient information to characterize that teacher's classroom and we excluded their responses from further analysis (we used this procedure in the first year as well). This eliminated 24 responses from 13 teachers bringing the total pool for analysis in the second year to 295 responses from 20 teachers. (For a complete discussion of the Methods, see Section 6.0 below.)

5.2.1 Teacher direction of student computer use When the pagers were activated and when teachers turned to the web survey on their desktop monitor the first question they saw was: “At the date and time your pager was most recently activated, were ANY OF YOUR STUDENTS using a computer?” Technology Model Schools teachers were more likely than others to report student technology use, 21.3% of the times paged compared to 14.7%. The difference is statistically significant (p < .001). For analysis of this question, data were aggregated to the teacher level and then weighted by the number of times any particular teacher responded. This assumes that data from the most diligent teachers are also the most reliable. Giving the more cooperative and diligent teacher responses greater weight---for both treatment and control groups---increases our ability to discern patterns in teacher work. Schools are seasonal organizations. The Fall begins with organizing for instruction and the Spring is occupied with testing. In order to respect school priorities, we did not sample teaching in September/October or in May/June. The following table shows student computer use by months of the school year between the Model Schools and other teachers.

Table 33: Frequency of Student Computer Use by Month: Teacher Pager + Web Survey Responses

TMS Compared to Control Schools

MONTH TMS/Control Mean NStd.

Deviation Sig. Diff.?December ‘05 TMS .2500 8 .4629 NoControl .1765 17 .3930January '06 TMS .1957 46 .4011 NoControl .1875 16 .4031February ‘06 TMS .2464 69 .4341 NoControl .2083 24 .4149March ‘06 TMS .1618 68 .3710 NoControl .0741 27 .2669 April ‘06 TMS .2308 13 .4385 NoControl .0000 2 .0000


“At the date and time your pager was most recently activated, were ANY OF YOUR STUDENTS using a computer?”

Note: In the previous table and in Table 37, there are no statistically significant differences between TMS and control teachers during the months measured. Over the course of the TMS project, those services did cumulate so that the TMS teachers and students outperform the controls in their use of computers. The lack of statistically significant results for Tables 33 and 37 is likely attributable to the small numbers of respondents, the brevity of the intervals and the availability of equipment and some amount of professional development to both groups.

Figure 2: Percents of students using computers per day per month: Teacher pager + Web Survey Responses

(TMS/treatment Compared to Control Schools)

For the teachers that reported technology use during a sampled moment, we asked, “How many students were using the computer?” TMS and control teachers reported similar student group sizes at the computers. Both TMS and control teachers had similar amounts of hardware in their classrooms---generally, four student desktops and that similarity would dampen differences. It may be that with more student desktops, a TMS-trained teacher would be more likely to assign students to use them.

Table 34: Numbers of Students Using Computers: Teacher Pager + Web Survey Responses



Size of student computer-using group

TMS Schools Control SchoolsValid

PercentFrequenc

yValid

PercentFrequenc

y1 student 18.2 8 23.1 32-5 students working individually 63.6* 28 61.5 82-5 students working as a small group 2.3 1 0 0

More than 5 students working individually 9.1 4 15.4 2

More than 5 students working in small groups 0.0 0 0.0 0

More than 5 students; some working individually, some working as a group 6.8 3 0.0 0

*Up from 45% in Year 1.

The modal practice is several students working individually at computers: teachers do not ask groups of students to work at a single computer.

Our focus is on classroom integration of technology. When computers first began to be used in schools, they were concentrated in “computer laboratories” for reasons of security and the scarcity of personnel who knew how to use them. The practice restricted student use only to scheduled trips. It was not uncommon for students to have one 45-minute period a week on a computer. And just as 45 minutes of basketball practice a week isn't likely to produce fluent play, 45 minutes of "computer" a week has not been associated with changes in academic performance. Putting computers closer to where students spend most of the day---in classrooms---opens the possibility for greater use18. West Virginia has 10 years of consistent investment in equipping each classroom with four computers. When we asked teachers about student use, we also asked about the location of that use. Most of the student computer use across all the teachers (with or without the TMS program) was in the classroom.

Table 35: Locations of Student Computers: Teacher Pager + Web Survey Responses

TMS Compared to Control SchoolsTMS vs. Control Location Valid Percent FrequencyTMS

Classroom 86.4 38Computer Lab 13.6 6Library / Other 0.0 0

Control

Classroom 84.6 11Computer Lab 15.4 2Library / Other 0.0 0

18 Whether the possibility translates into reality is still controlled by the teacher, hence the state's emphasis on integrating computers into classroom instruction.


Reading, because it is central to a child's success and because it is measured relentlessly on state and national tests is the ne plus ultra of elementary school teaching. When teachers were asked what their students were supposed to be working on at the computers, the TMS teachers were much more likely to say “Reading” (43% to 17%) while non-Model Schools teachers reported more use of “various subjects or other”. The TMS teachers believe in computers as an assist to Reading instruction far more than their counterparts19.

Table 36: Curriculum Area of Student Computer Use: Teacher Pager + Web Survey Responses

TMS Vs Control Schools

Curriculum area of student use

TMS Schools Control SchoolsValid

PercentFrequenc

yValid

PercentFrequenc

yReading-based program20 43.2 19 16.7 2Math-based program 18.2 8 25.0 3Internet surfing or researching 6.8 3 0.0 0Productivity software (e.g. word processing, spreadsheets, presentations, etc.)

15.9 7 0.0 0

Various subjects or other 15.9 7 58.3 7

5.2.2 Teacher own use of computers Model Schools teachers assigned computer use more frequently and to more students than the comparison teachers. We next inquired about their own practice---Yes, the students were using computers but what were they, the teachers doing? TMS teachers were more much more likely than others to be using computers (when the pagers rang). On the 206 instances when teachers in TMS schools responded to the question, they responded “Yes” 22% of the time. Teachers in control school classrooms only responded “Yes” on 1% of the 88 instances they were paged. That difference is statistically significant (p < .001). Computer use stayed constant for the TMS group---about 20% of the day---over the two years but it plummeted from 9% of the day reported by the comparison teachers to 1% of the day during the second year.

19 The conundrum is that in this analysis, technology use is more consistently related to higher Math achievement than Reading achievement.20 The emphasis on using computers to support Reading between the two groups reverses over the two years of this analysis. In the first year, teachers in the TMS schools reported that their students were using computers for Reading 34% of the times when they were paged: in the second year, the per cent for Reading rose to 43%. Teachers in the control schools went even more strongly in the opposite direction: first year student use of computers for Reading was 46% and that fell to 17% in the second year. Note that there are no significant differences in Reading achievement between the two groups in Year 2.


We also looked at teacher computer use by month.

Table 37: Frequency of Teacher Computers Use by Month: Teacher Pager + Web Survey Responses


MONTH TMS/Control Mean NStd.

Deviation Sig. Diff.?December ‘05 TMS .0000 8 .0000 NoControl .0000 17 .0000January '06 TMS .1739 46 .3832 NoControl .0000 16 .0000February ‘06 TMS .2029 69 .4051 NoControl .0417 24 .2041March ‘06 TMS .2941 68 .4590 NoControl .0000 27 .0000 April ‘06 TMS .3077 13 .4385 NoControl .0000 2 .0000

“At the date and time your pager was most recently activated, were YOU using a computer?”

Figure 3: Teacher Computer Use by Month: Teacher Pager + Web Survey Responses

(TMS/treatment Compared to Control Schools)

Teachers served by the Model Schools programs report more (and constantly increasing) use across the school year than their counterparts.

We next asked about the applications being used (“If yes to question 2… If you were using a computer when your pager was most recently activated, the application was...).


Table 38: The Applications Used in Classrooms: Teacher Pager + Web Survey Responses


Application

Model Schools Control SchoolsValid

PercentFrequenc

yValid

PercentFrequenc

yWord processing 28.3 13 0.0 0PowerPoint or another graphics program 2.2 1 0.0 0

Spreadsheets or database 15.2 7 0.0 0Publisher or other desktop publishing program 0.0 0 0.0 0

Kidspiration 2.2 1 0.0 0Inspiration 0.0 0 0.0 0Compass Learning or Riverdeep 0.0 0 0.0 0

Internet surfing or research 17.4 8 100 1E-mail 8.7 4 0.0 0Other 26.1 12 0.0 0

The state encourages teachers to use Compass Learning and Riverdeep for instruction in basic skills. While no teacher reported "using" those applications, it is still likely that their students were working with them.

5.3 Teachers' attitudes and opinions about technology ( second year)

5.3.1 Changes in the TMS program and the samples of teachers West Virginia funded a model of embedded professional development but necessarily relied on the state's county agencies to deliver the services. The second year of the program was different from the first. The most relevant change has already been discussed. Newly recruited "specialists" were reluctant to present themselves as “experts” before teachers who until recently had been colleagues and peers21.

In addition, services from the different TISs varied. While counties received lump sum grants for the program, they allocated them in different ways. For example, some schools had a full-time TIS, others had as little as one-quarter of a full-time equivalent TIS. Those decisions honor local judgment but, from the measurement perspective, produce different treatments.

21 The TISs also had to consider the likelihood that they would return, as teachers, to the same faculties, the same colleagues that they had just been training. “The nail that sticks up gets hammered down.”


Note: Running schools takes precedence over doing research and it should. To examine the 'continuing effects' and 'treatment/turnkey' questions, we had hoped to follow all of the TMS schools into a second year without the services of TISs. But some TMS schools and some counties concluded that the schools with that help for one year would benefit from a second year. Our results from the first year of this evaluation suggest that continuing those services was a good decision since, in the first year, we documented that the newly appointed TISs spent more time learning about technology than teaching about technology. Other schools that had served as non-TMS control sites added TIS services in the second year22.

5.3.2 Self-reported confidence/expertise after 2 years Teachers are practical people: they focus on what works, what is possible. At the end of the first year, teachers who had the benefit of a TIS were (statistically) significantly more positive about their prospects for integrating technology into classroom instruction than teachers in control schools. And, at the end of the second year, teachers who had help from a TIS continued to be more positively inclined toward technology integration than were the control group.

Table 39: Teacher confidence about the future use of technology in the classroom: Most to least

Next September, I will be better able to use technology in my classroom than I was last September1= Strongly Agree2 = Agree3 = Disagree4 = Strongly Disagree Post (Year 2)TIS year 1, TIS year 2 (the “two-year treatment” group)(n=14) Agree (1.6429)

No TIS year 1, TIS year 2 (the “control and treatment” group)(n=17) Agree (1.9412)

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group)(n=49)

Agree (2.1224)

No TIS year 1, No TIS year 2 (the “control” group)(n=13) Disagree (2.8462)

In the table above, the most confident group are those with TIS services for two succeeding years, followed by those who had TIS services during the past year only. Teachers who had help from a TIS only in the first year believed that their technology expertise would grow and thus the TIS effects may last. The groupwithout TIS services are noticeable for their lack of confidence.

22 For a complete discussion of the sample changes over time, please see the Appendix.


Table 40 below reports the mean responses of teachers to a question about their perceived ability to move forward with technology integration on their own, an idea that is central to the program's continuing effects.

Table 40: Teachers ability to use technology independently: Most to leastNext September, when I use technology in my classroom, I will be able to do so on my own (i.e. without the aid of colleagues or a TIS). 1= Strongly Agree2 = Agree3 = Disagree4 = Strongly Disagree Post (Year 2)TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group)(n=49) Agree (2.2041)

TIS year 1, TIS year 2 (the “two-year treatment” group)(n=14) Agree (2.3571)




In both the future use and the independent use of technology, an analysis of variance shows significant differences in group means. Teachers who never had TIS help report much less confidence in their future and independent uses of technology. The mean score of the control group teachers is statistically significantly lower than the mean score of each of the other three groups of teachers for both questions (see ANOVA table below). Thus, TIS-supported teachers are better able to use technology than those without that service.

Table 41: Analysis of Variance on Future and Independent Use of Technology Between Teachers with and without TIS Assistance

Sum of

Squares dfMean

Square F Sig.Next September, I will be better able to use technology in my classroom than I was last September

Between Groups 10.586 3 3.529 8.029 .000

Next September, when I use technology in my classroom, I will be able to do so on my own (i.e. without the aid of colleagues or a TIS)

Between Groups

9.296 3 3.099 6.729 .000

The TISs succeeded in making the teachers more confident and more independent. The question of 'independence' is important because it signals mature use. Teachers have their own incentives to stay dependent on 'experts'---teachers believe themselves to be notoriously busy so recruiting someone else to "do technology" helps (that includes delegating to the computer lab). Second, until expectations about computer use are built into job descriptions and performance monitoring, it is 'not my job'. Just as help to governments had to shift from technical assistance (doing something for someone) to capacity-building (teaching them to do things for themselves), the TISs had to learn to work from dependence to independence even it means they were working themselves out of a job. One TIS described working with a faculty where individuals made the same requests for the same help repeatedly. She concluded that they needed a successful experience in order to become more independent. She focused on helping the faculty deploy, analyze and use online assessments. As they began to master that, they were willing to try other new things on their own.

5.3.3 Growth in positive attitudes toward technology 2004-2006 Each of the three times that we surveyed teachers, we asked them to self-report their attitudes toward technology. Those questions were based on a previously validated scale composed of the following six items:

Table 42: “Indicate how much you agree or disagree with each of the following statements.”

1 = strongly agree


2 = agree3 = disagree4 = strongly disagree[NOTE: LOWER SCORE MEANS MORE POSITIVE ATTITUDE]

Technology makes my work easier. Technology in the classroom helps teachers implement better lessons.

New technology is easier for teachers to learn.Technology in the classroom improves student achievement.

Students work more creatively when they use computers.Students learn more if teachers use more technology

Scores on this scale were computed by taking the mean response across all six items. The results are displayed in Table X below. As with future and independent technology use (reported above) there is evidence of the effectiveness of the program. The teachers that worked with a TIS for both years of the study reported the most drastic improvements in attitudes. Teachers who had any assistance from TISs developed more positive attitudes and teachers that never worked with a TIS showed no change in attitudes.

Table 43: Technology Attitudes Scale[NOTE: LOWER SCORE MEANS MORE POSITIVE ATTITUDE]

PrePost

(Year 1)Post

(Year 2)No TIS year 1, No TIS year 2 (the “control” group) (n=10)

Disagree (2.73)

Disagree (2.70)

Disagree (2.78)

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=44)

Agree (2.27)

Agree (2.34)

Agree (2.12)

TIS year 1, TIS year 2 (the “two-year treatment” group) (n=7)

Disagree (2.50)

Agree (2.48)

Agree (1.71)

No TIS year 1, TIS year 2 (the “control and treatment” group) (n=6)

Disagree (2.64)

Agree (2.47)

Agree (2.39)

In the table above, the control group teachers began and ended negative about the contributions of technology. All the TIS teacher groups ended by endorsing technology and some groups switched from negative attitudes to positive ones.

Furthermore, repeated measures analyses of variance show that there are different growth trajectories across two of the groups (the control group and the one-year treatment/turnkey group); there are statistically significant differences in the growth in positive attitudes between those two groups (F=3.721, p=.016). (The differences in the trajectories among the other "mixed" groups are not significantly different.) Additionally, the effect size (partial eta2 = .151) for “groupness” is reasonably large. That is, whether or not a teacher had TIS help accounts for 15% of the variance in their positive-negative attitudes.


Recall that the scale captures attitudes that are pivotal to teacher use of technology---'does it help me?', 'does it help students?' etc.

Table 44: Teacher changes in attitudes toward technology

over time and by duration TMS services

Tests of Between-Subjects EffectsMeasure: MEASURE_1 Transformed Variable: Average

Source

Type III Sum of Squares df

Mean Square F Sig.

Partial Eta2

Intercept 654.773 1 654.773 1089.987 .000 .945TMS Duration 6.705 3 2.235 3.721 .016 .151Error 37.845 63 .601


5.3.4 Growth in proficiency with technology 2004-2006 We have just reported the positive changes in confidence about technology and estimates about its use. Do those gains translate into differences in proficiency and use of technology by teachers and/or their students? We have data from teachers across three time points to answer those questions. Teachers completed questionnaires at the beginning of the study (Fall 2004), the end of the first year (Spring 2005), and at the end of the study (Spring 2006). The results reported below are from only the teachers that completed all three questionnaires.

We asked teachers how expert they were in using computers. The results are displayed in Table 45 below. Teachers from all the study groups thought they had become more expert from Fall 2004 to Spring 200523 (the first year of the study) but the only teachers that reported improved expertise for the second year were those who had worked with a TIS in the first year. That suggests that the effect of the TIS survives his or her departure.

Table 45: Teachers' Self-Estimate of Computer Expertise Over Two Years by Study Condition

"How would you describe your overall computer-related expertise?"1= Advanced2 = Intermediate3 = Novice Pre

Post (Year 1)

Post (Year 2)

No TIS year 1, No TIS year 2 (the “control” group) (n=10)

Novice (2.70)

Novice (2.50)

Novice (2.50)

TIS year 1, No TIS year 2 (the one-year “treatment/ turnkey” group) (n=42)

Intermediate (2.24)

Intermediate (1.98)

Intermediate (1.90)


Intermediate (2.29)

Intermediate (2.00)

Intermediate (1.86)


Intermediate (2.17)

Intermediate (2.00)

Intermediate (2.00)

Furthermore, repeated measures analyses of variance show that there are different growth trajectories across groups; there are statistically significant differences in the changes in self-reported computer-related expertise across the groups (F=4.010, p=.011). Additionally, the effect size (partial eta2 = .165) for duration of TMS services is reasonably large. That is, belonging to one group or another accounts for 16.5% of the variance in growth of self-reported expertise.

Table 46: Changes in teacher self-reported computer expertise over time and duration of TMS services

Tests of Between-Subjects EffectsMeasure: MEASURE_1 Transformed Variable: Average

23 Although the categories do not change, the average numbers do.


Source

Type III Sum of

Squares dfMean

Square F Sig.Partial Eta Squared

Intercept 525.156 1 525.156 904.970 .000 .937TMS Duration 6.981 3 2.327 4.010 .011 .165Error 35.398 61 .580

In addition to that single item on computer-related expertise, each of the teacher questionnaires contained a previously validated technology proficiency scale that was composed from responses to 13 items (see below).

Table 47: “Please tell us how proficient you are with each of the following computer applications:”

1 = I can teach others how to do this (advanced)2 = I can do this independently (intermediate)3 = I can do this, but sometimes I need help (novice)4 = I've never tried this (beginner)

MS Word or other word processing program(s)MS Excel or other spreadsheet program(s)MS Access or other database program(s)

MS PowerPoint or other presentation program(s)MS Publisher or other desktop publishing program(s)

Basic Skills Software (e.g. Compass Learning, Riverdeep, etc.)Kidspiration

Reinventing Education websiteE-mail

Web page authoring (e.g. MS FrontPage, Dreamweaver, etc.)World Wide Web searching

MS Windows (e.g. moving files, creating folders, etc.)Video production software

Teachers who knew how to use one of these applications tended to know also how to use the others. The fact that teacher responses are similar indicates that TIS services worked across several functional areas or, contrarily, teachers not in the TMS schools did not know how to use the array of applications. Scores on this scale were computed by taking the mean response across all 13 items. The results are displayed in the table below.

Table 48: Teachers' Self-Reported Proficiency with Technology Applications Over Two Years by Study Condition

13-Item Technology Proficiency Scale

PrePost

(Year 1)Post

(Year 2)No TIS year 1, No TIS year 2 (the “control” group) (n=10)

Beginner (3.3846)

Novice (3.2692)

Novice (3.2077)

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=45)

Novice (2.7845)

Novice (2.7043)

Intermediate (2.4667)



Novice (2.9890)

Novice (2.8132)



Novice (2.8077)

Novice (2.6923)


Again, we see that the control group teachers remain novices at the end of two years while the groups with access to TIS services have become intermediate technology users.

A repeated measures analysis of variance demonstrates that there are different growth trajectories across groups; there are statistically significant differences in the changes in self-reported technology proficiency across the groups (F=4.103, p=.010).

Table 49: Repeated Measures Analysis of Variance

Tests of Between-Subjects Effects Measure: MEASURE_1 Transformed Variable: Average

SourceType III Sum of Squares df

Mean Square F Sig.

Intercept 891.064 1 891.064 1086.555 .000TMS Duration 10.096 3 3.365 4.103 .010Error 52.485 64 .820

Finally, each of the teacher questionnaires contained a previously validated technology integration scale that was composed of the following 9 items:

Table 50: “Please tell how good you are at integrating the following software programs into the curriculum:”

1 = I can teach others how to do this (advanced)2 = I can do this independently (intermediate)3 = I can do this, but sometimes I need help (novice)4 = I've never tried this (beginner)[NOTE: LOWER SCORE MEANS MORE ADVANCED]

Accelerated MathAccelerated ReaderCompass Learning

Homeroom.comInformal Math Inventory

JumpStart TypingKeys to ReadingMax’s Sandbox

Riverdeep

Scores on this scale were computed by taking the mean response across all 9 items. The results are displayed in Table 40 below. Generally and across the


board, teachers did not report much change in their ability to integrate these software programs into the curriculum.


Table 51: Technology Integration Scale[NOTE: LOWER SCORE MEANS GREATER INTEGRATION ABILITY]

Pre Post (Year 1) Post (Year 2)No TIS year 1, No TIS year 2 (the “control” group) (n=10) Novice (3.49) Novice (3.47) Novice (3.48)

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=44) Beginner (3.53) Novice (3.45) Novice (3.32)

TIS year 1, TIS year 2 (the “two-year treatment” group) (n=7) Novice (3.33) Beginner (3.54) Novice (3.16)

No TIS year 1, TIS year 2 (the “control and treatment” group) (n=6) Novice (3.22) Novice (3.44) Novice (3.19)

Repeated measures analyses of variance show no differences in growth trajectories over time across the groups (F=0.461, p=.711).

5.3.5 Teacher use of computers Generally speaking “integration” has been confused with “use”. For example, if a student is doing independent skill practice, that indicates more about classroom management than the teacher’s integration of technology into instruction. We approached the issue of definition in two complementary ways. First, teacher self-reports have been the only available indicators of integration. Because we have data from multiple sources on the same phenomenon, measures of “integration” will emerge from those data.

In addition, we adopted the measures developed and validated for use in the USEIT (Use, Support, and Effect of Instructional Technology) Study, a 3-year study undertaken by the in TASC group housed in the Center for the Study of Testing, Evaluation and Educational Policy and the Lynch School of Education at Boston College. For the USEIT study, some results of which were recently published in the online, peer-reviewed journal Education Policy Analysis Archives, the researchers utilized multiple measures of teacher technology use. Recognizing that teacher technology use is a multidimensional construct, the researchers developed and validated the following seven scales (composite variables):

Teachers’ use of technology for delivering instruction Teacher-directed student use of technology during class time Teacher-directed student use of technology to create products Teachers’ use of technology for class preparation Teachers’ professional e-mail use Teachers’ use of technology for (special needs) accommodation Teachers’ use of technology for grading

Those composite variables, consisting of anywhere from one to five items, are still measures of “use” that do not necessarily capture “integration,” but they are validated and move us collectively toward a better understanding of integration.


Furthermore, they are an appropriate set of outcomes for the Technology Model Schools program.

NOTE: these USEIT technology integration scales were not included on the baseline survey. They were added to the teacher questionnaires at the end of the first year based on feedback from the USDOE expert panel review team. Therefore, we only have data from two time points for these scales.

5.3.6 Teacher-directed student use of technology during class time

Table 52: Teacher directed frequency of student use of technology

“During class time this year, how often did students perform the following activities…”1 = Never2 = Once or twice this year3 = Several times this year4 = Several times a month5 = Several times a week

Students work individually on school work using computersStudents work in groups on school work using computers

Students perform research or find information using the Internet or CD_ROMStudents use a computer to solve problems

Students use a computer or portable writing device for writingStudents present information to the class using a computer

Scores on this scale were computed by summing the responses across all 6 items. Scores, therefore, could possibly range from 6 to 30. The results are displayed in Table 42 below.

Table 53: Teacher-directed Student use of Computers[Range = 6 (low) – 30 (high)]

Post (Year 1)

Post (Year 2)

No TIS year 1, No TIS year 2 (the “control” group)(n=12) 13.75 14.25

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=46) 15.61 17.37

TIS year 1, TIS year 2 (the “two-year treatment” group) (n=11) 14.27 17.73

No TIS year 1, TIS year 2 (the “control and treatment” group) (n=17) 15.94 17.65

Repeated measures analyses of variance show that there was significant growth across the whole sample (F=16.695, p <.001), but no differences in growth trajectories across the groups (F=1.275, p =.289).


Additionally, analysis of data from the first year indicated that teachers in TMS schools sent their students to computer labs more frequently and for longer sessions than the control group teachers.

5.3.7 Teacher-directed student use of technology to create products

Table 54: Teacher report of frequency of requests for student products using technology

“This year, how often did you ask students to produce the following products using technology…”

1 = Never2 = Once or twice this year3 = Several times this year4 = Several times a month5 = Several times a week

Multimedia projectsWeb pages, web sites, or other web-based publications

Pictures or artworkGraphs or chartsVideos or movies

Table 55: Teacher-directed Student use of Technology to Create Products

[Range = 5 (low) – 25 (high)]Post (Year 1)

Post (Year 2)

No TIS year 1, No TIS year 2 (the “control” group) (n=12) 6.92 6.75TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=46) 8.02 8.85




Additionally, repeated measures analyses of variance show that there was no significant growth across the whole sample (F = 2.103, p = .151), and no differences in growth trajectories across the groups (F=1.895, p =.137).

5.3.8 Teacher use of computers for productivity

Computers have transformed the workplace in other sectors of the economy. This section discusses the extent to which teachers made use of computers for their own responsibilities.

Table 56: Teacher report of frequency of use of computers for own work

“This year, how often did YOU perform the following computer-related activities…”1 = Never2 = Once or twice this year3 = Several times this year4 = Several times a month5 = Several times a week

Make handouts for students using a computerCreate a test, quiz or assignment using a computer

Perform research and lesson planning using the InternetRecord student grades using a computer

E-mail to administrators, teachers or staff in your schoolCommunicate electronically with parents

Post student work, suggestions for resources, or ideas and opinions on the World Wide Web

Scores on this scale were computed by summing the responses across all seven items. Scores, therefore, could possibly range from 7 to 35. The results are displayed in Table 46 below.

Table 57: Teacher Use of Computers for their Own Work[Range = 7 (low) – 35 (high)]

Post (Year 1)

Post (Year 2)

No TIS year 1, No TIS year 2 (the “control” group) (n=12) 15.25 15.92

TIS year 1, No TIS year 2 (the one-year “treatment/turnkey” group) (n=46) 19.37 20.65



Repeated measures analyses of variance show that there was significant growth across the whole treatment group (F=11.544, p=.001), and differences in growth trajectories across the groups (F=4.687, p=.005). Specifically, the growth exhibited by the “control and treatment” group was significantly greater than the growth shown by the control group. This finding is consistent with a Year 1


finding; over the course of the first year working with a TIS, teachers tend to show the most growth in their own computer use (which does not necessarily translate to teacher-directed student use of computers). The “control and treatment” group worked with a TIS for the first time during the time span that these data cover.

5.3.9 Continuing effects The history of school innovation is often melancholy. New things are introduced, teachers are trained, they use the innovations for a while and they then revert to their previous practice. Will the TIS teachers go back? The question of continuing effects is critical for public policy because, if the intervention does not last, why spend the money?

In this instance, there are signals of the stability of classroom integration of technology among the TMS trained teachers. First, while teachers' use of computers, in the control group schools, plummeted from 9% of the school day to 1% of the school day between the first and second years of this study, it remained constant at 22% of the day for the TMS teachers. Second, confidence (or expertise) in their own use of computers and in the prospective contribution of computers increased among the TMS teachers from year to year but not among the others.

Observations in two of our field study schools document that those faculties: (1) continued their use of technology from the first to second years; (2) increased and deepened their use of technology; (3) said they would not revert to their previous paper-pencil practices; (4) had become independent of their TIS; and (5) were teaching other, newly arrived teachers how to do things. In some instances those effects are reaching parents through their children’s use of county websites, at home.

In both of these positively impacted schools, the teachers spontaneously described functions and applications to teaching, not simply software by vague titles. They talked of assessment, reporting, lesson presentation, diagnosis of individual needs, grouping of students by documented needs, Internet research and assignments to students. That discourse described work that they do when the students are in “the computer lab” and when they are in their home classrooms. Additionally, the teachers offered critiques and evaluations of various software by functionality, by contribution to things like matching quiz items to the States’ required CSO’s (learning objectives). Taken together, the remarks of those teachers, in those schools, is evidence of emerging fluent and classroom integrated use.

In one school, the principal attributed the impact to the limitation of the TIS’s availability (one day a week followed by one day a month): he said that forced teachers to be independent. In the other school, the faculty had been so resistant in the initial year, that the county extended the TIS’s 4-day a week


availability for a second year and that second year paid off in big and probably permanent gains.

5.3.10 The “turnkey” effect: Teachers training teachers Turnkey activities, sometimes called ‘teachers-training-teachers’, have been a staple of school change strategy for decades. The hope has been that teachers with new skills would enlist colleagues in the new activities. The idea is attractive because it is inexpensive (only the first generation of trainers needs to be paid) and respects the professionalism of teachers. The TMS program was designed so that, after the Specialist moved on, teachers would (a) continue to use technology and (b) train others. Regardless of having had the TMS services or not, all teachers report helping “other teachers use technology”. (The differences between the groups with and without TMS services are not statistically significant.) In the context of the initial reluctance of newly-dubbed Technology Integration Specialists to act as experts with their former colleagues, it is interesting to observe this informal interaction among peers.

Table 58: Teacher self-reports of turnkey training: Most to leastI have helped other teachers use technology.1= Strongly Agree2 = Agree3 = Disagree4 = Strongly Disagree Post (Year 2)TIS year 1, TIS year 2 (the “two-year treatment” group)(n=15) Agree (1.8621)



Agree (1.9455)

No TIS year 1, No TIS year 2 (the “control” group)(n=15) Agree (2.3333)

We also asked teachers to compare their classroom technology integration abilities to others. All the teachers---both with and without the TMS program--- rejected the idea that other teachers knew more about this than did they (no statistically significant differences among the study conditions).

Table 59: Teacher self-ranking of classroom technology integration: Most to least (higher is better)

Most of my colleagues know more about how to integrate technology into the curriculum than I do.1= Strongly Agree2 = Agree3 = Disagree

Post (Year 2)


4 = Strongly DisagreeTIS year 1, TIS year 2 (the “two-year treatment” group)(n=15) Disagree (2.9310)


Disagree (2.7818)

No TIS year 1, TIS year 2 (the “control and treatment” group)(n=18) Disagree (2.7222)


On this evidence, a year after the TMS program was originated, teachers in those schools were not more willing to help other teachers although all the teachers already share technology notes.


6.0 RESEARCH AND EVALUATION METHODS

6.1 The R&E methods used in this inquiry

6.1.1 Methods To triangulate the conventional pre-post, self-report questionnaires we used technology to study technology, i.e., metering software on classroom computers and random-interval web-based surveys triggered by the pager. That data collection was supplemented by face-to-face interviews, classroom observations and close cooperation with the responsible personnel from the WVDE. Interactive, Inc. conducted made 100 visits to schools over more than a month of face-to-face interviews and observations.

6.1.1.1 Data collectionThe Experience Sampling Method (ESM) was developed to “detect variations in emotional states over time”.24 The design has been used to study student stress25, teacher motivation and job satisfaction26, and student affective experiences of studying.27 ESM asks participants to answer questions at multiple random moments when a timer (e.g. a pager, a watch, a personal digital assistant, etc.) prompts a response. The “unique advantage of ESM is its ability to capture daily life as it is directly perceived from one moment to the next, affording an opportunity to examine fluctuations in the stream of consciousness and the links between the external context and the contents of the mind.”28

We modified the ESM to document activities (i.e. teacher and student use of computers) over the course of the school year. Teachers were equipped with pagers that were activated once or twice a day at random times every other week. The web-based scheduling program only allowed pages scheduled in advance to occur on the hour, once a day.

When the pagers were activated, teachers were to complete a short web-based questionnaire at their earliest convenience.29 Additionally, one randomly selected student was to complete a different, even shorter web-based survey.

Setting up the pagers required selecting a paging company30, getting the pagers to the teachers, showing them how to use them and how to respond to the web survey and scheduling the pages. Because the pager service maintained a

24 A modified version of the Experience Sampling Method (ESM) (Hektner, Schmidt & Csikszentmihalyi, 2006; Csikszentmihalyi & Hunter, 2003, p.186)25 Verma, Sharma & Larson, 200226 Bishay, 199627 Asakawa & Csikszentmihalyi, 199828 Hektner, Schmidt & Csikszentmihalyi, 2006, p. 629 See Appendix C30 We used Rampage Communication of Charleston, WV.


web-based paging “scheduler” and because each device was pre-programmed into eight different group numbers we had only to dial 8 numbers not 100.

For the treatment schools, the TISs helped with delivery and training. Pagers and pager manuals were sent to the TIS as part of an “evaluation kit”31 with directions for teachers use. The TISs were very helpful with their schools.

In the control schools, we called the principals to ask for cooperation. If the principal had a “technology contact” in the school, the evaluation kit was sent to the school directly. In some cases, a member of the research team visited schools to work with the teachers.

The web-based questionnaires were designed and tested with roughly parallel items for teachers and students. We used SNAP Survey software and the hosting capabilities of a cooperating university for the brief (3-5 item) teacher and student pager-triggered web questionnaires.

The frequency of paging was partly empirical and partly political---what would teachers accept. Because we needed the (uncompensated and voluntary) cooperation of teachers over two years we agreed on 5 to 7 randomly determined pages during alternate weeks.

In addition to the ESM, we monitored file activity on individual computer workstations through metering software. We selected TrueActive Monitor (TAM) version 5.0. TrueActive Software, Inc. (formerly known as WinWhatWhere Corp.) was founded in 1991 to support the debugging of client software by monitoring every activity (down to individual keystrokes) on a Windows-based computer. Also, TAM could capture screenshots and store the images for later viewing. Figure 3 is an image of the TAM setup screen. For this study, only file activity was recorded. Data on file activity were automatically forwarded every seven days to Interactive, Inc. by an e-mail function of TAM. (Getting TAM installed on hundreds of workstations required the cooperation of the TISs and WVDE technology staff in a process similar to the pager deployment.) Early in the second year of the study, TAM was sold to another company. The programmers and developers who had customized the software for this study disappeared. The acquiring company orphaned TAM and refused to cooperate with this study. Thus, the data retrieved by TAM over the course of the second year could not be analyzed.

Figure 4: TrueActive Setup Screen

31 The evaluation kit included instructions on how to create a desktop icon/link to the web-survey, the TrueActive monitoring software with installation instructions and a CD-ROM with a multimedia presentation describing the study and its research methods.


6.1.1.2 The Pilot studyOur data collection methods were pilot tested with two teachers in a single school in April 2004 before the 2004-05 launch. Interactive, Inc. visited the school, met with the technology integration specialist, the principal and the teachers and reviewed the data collection feasibility.

During the pilot, we learned that virtually every desktop computer in any West Virginia classroom had been labeled with the same ID---"cl" which stands for the most commonly used instructional software in the state, Compass Learning. The consistent application of the same two-letter identification avoids the problem of students (and teachers) not knowing or forgetting a workstation password. Unfortunately, that practice defeated our ability to link particular desktops with particular students. Because of the understandable 'fix' to the access problem, we lost the ability to link the amount of computer use by student #1409 to student #1409's achievement. That was also the case for workstations in computer labs. It was impossible to isolate the data captured by TAM and attribute it to the students in the study32. Therefore, we monitored only classroom computers and not computer lab workstations.

From the beginning the WVDE and Interactive, Inc. were committed to protecting the confidentiality of all participants including those who's computers were being monitored by the TAM procedure. We installed a splash advisory message that appeared on all study computers when the device was turned on.

"This computer has a meter that records the applications that are accessed. The data are collected as part of study funded by the US

32 By agreement, teacher participants were identified by the last four digits of their Social Security numbers.


Department of Education and conducted by the West Virginia Department of Education. All data are confidential; no school or individual names will be reported in the research. Anyone using this computer expressly consents to such monitoring."

It should also be noted that all computers on the state's network are already subject to monitoring by the West Virginia Department of Education. And all public employees at all levels have agreed to “acceptable use” policies.

At our initial meeting with the teachers in the pilot school, they agreed to wear the pagers, to respond to the Web-based surveys, and to use their roll books to select students to complete the surveys essentially randomly without repeating requests to individual students. We placed shortcut icons linked to the Web-based surveys on the desktops of each of the classroom workstations and two of the computers in the computer labs. This provided convenient, direct access to the survey without having to remember a URL.

We sought to balance minimal intrusion on classroom instruction with accurate data collection about the school day. Because teachers were concerned not to interrupt whole-group instruction, we printed paper versions of the brief student web-survey. At the teacher's discretion, some of the data collection from students was paper-based and the teachers then posted the results to the web site as soon as was practical. Also, the teachers agreed that when they were absent, they would (a) check the pager for missed messages and times and (b) complete the web survey as "not available". Substitutes were not asked to respond to any queries.

The pilot teachers had begun with skepticism and even resistance but concluded the pagers were unobtrusive and simple to use. The students like the process and were eager to get their turn at the computer. The pilot study had a relatively low response rate that turned out to be related to signal reception not teacher cooperation. Even with the satellite origin of the signals, we had problems with basements, thick walls, electronic interference and West Virginia mountains (the state is officially the "Mountain State”). Nonetheless, the pilot study gave us practical data and confidence in the cooperation of teachers and schools.

6.1.1.3 Results of the data collection methodsOur modification of the Experience Sampling Method (ESM, i.e., the pagers + web-based questionnaires) was successful. Technical hurdles and teacher reluctance were both manageable. Teachers were cooperative, they took their role seriously and their students were enthusiastic about providing data. In the first year of the study, 109 teachers (61 teachers in treatment schools and 48 teachers in control schools) were equipped with pagers. From November 15, 2004 through April 15, 2005, the pagers were activated 41 times. In the end,


1,598 usable33 pager-triggered web survey responses were received from teachers. Those 1,598 responses came from 109 different teachers (i.e. every teacher responded at least once). The range in the number of responses from individual teachers was one to 41 (i.e. a couple of teachers responded once, and one teacher responded to all 41 pages).

Students sent 1,31134 usable pager-triggered web survey responses. Those 1,311 responses came from 102 different classrooms. The range in the number of responses from individual classrooms was one to 34 (i.e. a couple of teachers only once asked a student to respond, and one teacher had students respond 34 different times).

Because the second year was focused on “continuing effects,” pager data collection was limited to teachers who continued from the first year into the second year. Teachers in three schools refused to use the pagers in the second year. In the second year 51 teachers were paged 40 times from December 2005 through March 2006. During that time, 33 different teachers sent 319 usable random-interval, pager-triggered web survey responses. The range of responses from individual teachers was one to 32.

In addition to teacher surveys, Interactive, Inc. received 297 usable pager-triggered web survey responses from students. Those 297 responses came from 33 different classrooms. The range in the number of responses from individual classrooms was one to 31 (i.e. a few teachers only once asked a student to respond, and one teacher had students respond 31 different times).

Documenting file activity with desktop metering software was more problematic especially because the new owners orphaned the application and refused cooperation. Throughout the study, it was necessary to troubleshoot the use of data on computer file activity captured automatically and unobtrusively by TAM. For example, the TAM developers had to revise the TAM 5.0 system to run at least, on Windows 98 computers.

33 By “usable”, we mean a few things: first, the response came on a day we were expecting a response; some teachers responded on a day during a week when no pages were scheduled. Second, the response came at a time close to when it was expected. If a response came well before 9 a.m. or well after 2:00 p.m., we did not use the response as we only scheduled the pagers to go off between 9 and 2. Finally, there were some instances when the same teacher submitted two or more responses nearly simultaneously with the exact same responses. In those instances, we assumed that the teacher clicked “submit” at the end of the survey more than once. We discarded all but one response in those cases.34 There are a couple of reasons why this number is smaller than the number of teacher responses. First, because of a necessary change in the Web-survey for the students, the student web-survey data start with December 1, 2004 (compared to November 15, 2004 for teachers). Second, seven teachers never had any students respond to the web-survey.


They also had to reprogram the e-mail function that sent the data automatically to Interactive, Inc., ideally every seven days. Initially, many of the computers were sending data every seven minutes. That slowed the sending workstations and buried Interactive, Inc. in unnecessary data. The “7 minute” problem was never entirely solved. Despite multiple visits, some computers remained continued to send data every seven minutes. In each of the two years of the study, almost 35,000 e-mails with a data file attached were forwarded to the Interactive, Inc.

Eventually, usable data from the metering software arrived. Between October 1, 2004 and April 1, 2005, TAM was installed on approximately 287 computers. Over the course of the two years of the study, data was ultimately received at one point or another from 184 computers in 25 schools. The difference between 287 and 184 arose for a number of reasons, including the fact that some classroom computers were simply never turned on (and that, given the purpose of this study, is a finding).

Attrition among respondents is inevitable in survey research and it increases with the press of other business, the duration of the research and the absence of compensation. In the Fall of 2004 (baseline Year 1), we started with responses from 199 teachers. We ended with:

68 teachers from 24 different schools who completed all three questionnaires over the two year interval for this analysis

25 additional teachers completed both end-of-year surveys (Spring 2005 and Spring 2006, Years 1 and 2) and

7 additional teachers completed the first and last surveys (the Fall 2004 baseline survey and the final Spring 2006 survey (plus the original 68 who also completed these).

For all analyses reported, associated n’s are reported.

It is common to have the cooperation of about 60% of a pool of unpaid respondents. We are grateful that between 74% and 92% of the West Virginia teachers cooperated with the questionnaires (and the desktop meters and the random-interval, pager-triggered web-surveys)35. Our response rates varied over time and the different waves of data collection as follows:

For the baseline administration (Fall 2004), conventional paper-pencil questionnaires were sent to 227 teachers; they were returned by 199 teachers, for a response rate of 88%.

At the end of the first year, the conventional teacher questionnaires were moved to a Web-based format. 194 teachers were expected to take the survey; 178 teachers completed the online survey, for a response rate of 92%.

35 The percents underestimate the cooperation of eligible teachers. Some attrition had nothing to do with cooperativeness but with a reassignment to a new grade, a new school or other life changes (retirement, resignation). The cooperation of the stable cohort of teachers is high and appreciated.


At the end of the second year, 182 teachers were expected to take the conventional EOY Web-based survey; 134 teachers completed it, for a response rate of 74%.

6.1.2 Threats to validity Because of the state’s commitment to technology, all teachers in the state were getting equipment, training and software (for example, Kidspiration and Compass Learning were available on state contracts for schools to purchase). Although the other schools did not have Technology Model Schools resources, given the State's long-standing press for learning technology, their faculties may have had technology-related professional development of some sort.

All the sampled teachers knew this was a study of technology. Some teachers who did not use technology may have obscured their non-use by failing to cooperate. But, web surveys are commonly used and cell phones (akin to pagers) are nearly ubiquitous. Nonetheless, those technologies may have discouraged participation from techno-phobic teachers. Weighting by response, along with excluding the infrequent responders might result in overestimating technology use. Note that we used the same rules to analyze treatment and control respondents.

Early on, the technical advisors to this study suggested an analogue to the "what gets measured, gets done" phenomenon. They suspected that if teachers knew we were monitoring the applications accessed by their classroom desktops, they would artificially boost the use of those desktops36. This is the question of a Hawthorne Effect: Does collecting data digitally on teacher computer use change that use?

To test that possible distortion, in the first year, we asked half the teachers who were receiving TIS services and half the control group teachers to wear pagers. Obviously, we also told them that we were installing monitoring software in their desktops. It seemed possible that those obtrusive and unobtrusive additions acted as an intervention and encouraged their use of technology. The most likely place to look for an effect was with control group teachers who, by definition, did not have professional development assistance from the TIS program. In 78% of the proficiency areas we measured, there were no differences between teachers who were and were not “digitally measured” (i.e. wore pagers and had the monitoring software installed on classroom computers). And, where there were differences, the non-digital control teachers (those assessed without pagers or computer software) were more enthusiastic and advanced users than the digital group.

36 Nota bene: It was never a question that teachers would not know what we were doing. We sought and they provided explicit and continuing consent to the data collection procedures, among other things, a reminder showed up on the monitor every time the desktop was booted.


Thus, we concluded that putting monitors in teacher desktops and adding pagers plus web surveys to the teacher’s day is not associated with more use of instructional technology.

Teacher indifference to our monitoring runs contrary to that staple of undergraduate classes about scientific method, the "Hawthorne Effect" where workers became more productivity when they were watched by researchers with clipboards. These teachers did not do that but why not? Although we do not have data to test the explanation, several forces may be at work. First, teachers are very busy: they worry about things closer to the pressures of this morning's 22 4th graders. Second, the computer monitors required nothing of them, a virtue in data collection that encouraged their indifference. (The pagers were another matter since they rang during class time.) Third, most teachers are secure in their employment and we assured them early on that the data would not be used to supervise or evaluate them. And finally, some of our respondents allowed us access to their daily work because they shared our interest in a more accurate, more complete understanding of classroom instruction.

6.2. Comments about the methods of this study

6.2.1 The practical importance of evaluation research methods Why are evaluation research methods important? The billion dollar expenditures on technology are often justified more on promise than evidence. Asking, "how do we know what works" is important because it can inform practical decisions where money and student and professional outcomes are at stake. People ask questions about empirical inquiry whether they recognize them or not:

A state superintendent meets with her cabinet to preview the next fiscal year's budget request. "Tell me again why we're going to spend money to put more computers in classrooms instead of fewer students?"

A state legislature asks an SEA official, "What happened with all those computers we bought two years ago?"

A state technology director walks out of the hundredth classroom he has visited this year and wonders privately, "Am I sure about my conclusions and what I'm recommending? What if I'm wrong? Is there a better way to test what is to be learned about technology integration?"

A legislator looks at the requirement that one-fourth of EETT dollars be spent on professional development and wonders "How do we know what we're buying?"

The research methods of social science can make a difference for practice if those methods surface relevant phenomenon, documented unambiguously.


6.2.2 The limits of conventional methods for the study of technology integration into classroom instruction

The other argument for better methods is, of course, that previous methods are not very good. We have better ways to measure gas mileage than asking drivers what they feel yet for the most part that is how we have been measuring computer use in schools, i.e., with so-called "smile check" data. “Surveys are no longer adequate as the single measure for determining the impact of a technology project, mainly because self-reported data are often unreliable.37

Interactive, Inc. has pioneered the use of technology to study technology. We deployed multiple pre and post data collection methods--- (1) random interval telephone pagers coupled to web surveys for teachers and students, (2) meters in desktops that monitor and report file activity; (3) conventional paper-and-pencil questionnaires; and (4) field interviews and observations. Thus, our data triangulate teacher and student use of technology with unusual detail38.

6.2.3 New methods for the study of the integration of technology into classroom instruction

This research documents the success of using technology to study technology. Teachers and school administrators cooperated with the pagers plus web surveys procedures and the use of automatic meters to study computer desktop applications. The meters required nothing of the teachers except their assent. The pagers were quickly accepted. The data from both methods were inexpensive supplements to conventional retrospective data collection and are available for wider use.

A program evaluator summarizes the disadvantages of self-report data as follows: “Low response rate; no control over misunderstanding or misinterpretation; missing data, or inaccurate responses; not suited for people who have difficulty reading and writing; not appropriate for complex or exploratory issues.”39

37 Elizabeth Byrom, “Tips for Writing an Evaluation Plan for a Technology Grant”, SEIR-TEC News Wire, v 5, n 3, 2002, p 2.38 See the comparative responses in the table below:

Teacher and Student Reports about Student Use of Computers When Paged: Years 1 and

2TeachersStudentsYear 1Year 2Year 1Year 2TMS Schools25213135Comparison Schools21152518

39 Anna Li, “Thinking Beyond Surveys”, SEIR-TEC News Wire, ibid., p 10.


7.0 RECOMMENDATIONS

7.1 Recommendation #1: Use learning technology to improve student achievement

It is reasonable (if contentious) to expect improvement in student learning as a result of technology. There is a path from teachers to computers to students to learning although it is not straight. This is the second large-scale, multi-year study, in West Virginia, to document that state policies and practices do improve student achievement40.

Because the hope that technology will improve achievement is so widely shared, the recommendation seems obvious. But it is not. Districts have bought hardware and connectivity expecting those things ipso facto to improve achievement. There is no 'auto-install' button on learning technology that one can press and watch while the meter reaches "100%!"

7.2 Recommendation #2: Apply a critical mass of a proven model of professional development

The West Virginia procedures are available to other jurisdictions with an interest in fielding effective professional development about technology. The state created a cadre of classroom teachers and trained them in skills relevant to the state's other classroom teachers. Those Technology Integration Specialists were then returned to schools in the counties from which they had been recruited. The specialists worked (1) on-site, (2) in classrooms and (3) on-demand.

It is hard to over-estimate the importance of those three properties (sometimes summarized as "embedded professional development"). They contrast sharply with the more common professional development that is (1) determined by superordinates like a central office, (2) delivered by "experts" external to the faculty and (3) provided after school or worse yet, on the teacher's "own" un-paid time.

The West Virginia TMS program model is effective and expensive41. Because of the cost, many jurisdictions will continue to deploy "training" that is ineffective but cheap.

40 C.f., Mann et al., The West Virginia Story, ibid.41 In the near future, interactive computer simulations of school practice may give educators a chance to see vignettes and choices similar to those they face every day, make decisions and see consequences. The realism, privacy, ubiquitous availability and low cost of interactive computer simulations have made them the preferred method to upgrade skills in the private sector and the military. Eventually, some educational institutions will adopt them.


7.3 Recommendation #3: Calibrate expectations and strategies by curriculum topics

If the matter of how to do professional development for technology integration is settled, the question of its variation by subject matter is not. Why, for example, are the technology-associated improvements in Math more consistent than those in Reading42 even though, compared to the control group, teachers in the TMS schools directed students to use computers more for Reading than Math. This study does not have evidence that bears directly on that question but, because it is a pivotal matter of public policy, we ought all to ponder answers.

For example, is the difference in the ranks of elementary teachers who are more likely to trust their expertise in Reading instruction than in Math instruction and are therefore more willing to delegate aspects of Math instruction to technology? Or were the independent, parallel statewide efforts to promote Reading instruction so unanimously successful that they washed out our TIS/treatment-to-control differences? Or are the differences associated with the different software applications? On this evidence, we cannot answer those questions but we can recommend them for further inquiry.

7.4 Recommendation #4: Use technology to measure technology

Passing out 'smile-check' self-report questionnaires at the end of a workshop is not evaluating results. Although No Child Left Behind is famous for mentioning "data-driven decision-making" 111 times schools and districts are not philanthropies or universities: money spent measuring outcomes is taken away from producing outcomes unless the two can be related so that research informs practice.43 There are inexpensive ways to get proxy data for teacher and student use, for example, server records. We listened to West Virginia principals who were monitoring those lists bi-weekly and inviting teachers to discuss their use or non-use. Similarly, the six factors that describe teacher use of technology are now efficient, reliable and valid indicators that (1) describe high-end teacher performance and (2) are related to achievement. They are measures that should be used to benchmark other programs.

7.5 Recommendation #5: Use evaluation results to improve practice

The West Virginia Department of Education has been refining its technology support policies for more than a decade. Early on (1995), the state committed to a "follow-through" strategy beginning with a technology saturation of the early grades and following that initial group of students as they moved through the

42 Previous research indicates that teachers use computers less for math and science (C.f., H.J. Becker, “Findings from the Teaching, Learning and Computing Survey: Is Larry Cuban Right?” Education Policy Analysis, v 8, n 51, November 15, 2000.)”]

43 Plato Learning, (2003) "Choosing and Using Learning Technology: Making Evidence-Based Decisions: A Guide for Educational Leaders", Plato Learning, Bloomington MN.


grades. Also, WVDE made mid-course changes to the training of the TISs when they saw how little key variables had changed at the end of the first year.

For example, when it became clear that the TISs were not willing to impose themselves, as "experts", on their colleagues during the first year, the WVDE allowed a second year of TIS services and added training to feature more attention to peer-to-peer relations. In the future, experience with training other adults might be a pre-condition to employment in that role.

And, to improve its own organizational capability to use data to improve practice, the WVDE assigned its own employees to participate in and/or to shadow this R&E effort over its three years. As a result, the Department's institutional capability to commission meaningful evaluations and to use the results to inform decisions may have been improved.

For example, the state made grants of EETT funds to local projects. Most states defer to grantees who decide what to evaluate and how. One result is inattention to achievement outcomes; another is a proliferation of non-comparable, non-illuminating (and therefore non-accountable) case studies. In West Virginia, ten local agencies were encouraged to use evaluation methods that were comparable (and linked to this statewide analysis). That made possible comparative performance tables similar to the following. Because the evaluation methods were comparable (and validated), it is possible to see from the Table 47 that the top-ranked agency's professional development was almost three times more effective in encouraging teachers to use technology than the bottom-ranked agency.

Table 60: Local Agencies Ranked by Most-to-Least Improvement in Teacher Classroom Technology

Integration FactorsRank County* Points %s

1. d'Artenau 16 89%2. Waldrich 15 83%3. North Essex 14 78%4. Oak 13 72%5/6. Baywater 12 67%5/6. Big Bend 12 67%7. Sunder 11 61%8. Phillips 10 56%9. Blackwell 9 50%

10. June 6 33%* Although the data reported are real the place names are fictitious.

Alternatively, if a supervising agency is more interested in the ability to change aspects of the teachers' use of technology than it is in ranking jurisdictions, the next table displays which factors have changed the most. On this evidence,


professional development was impacting proficiency and attitudes but not how much teachers were willing to encourage students to use technology.

Table 61: Teacher Classroom Technology Integration Factors Ranked by Most-to-Least Accessible to Change:

Local Agency Data

Got betterStayed

the same Got worseProficiency with software 4 5 0Attitude toward contributions of technology 2 7 0Integration of technology into instruction 2 7 0Use of technology to deliver and support instruction

1 6 3

Encouragement of students to use technology

1 7 2

Encouragement of students to go beyond basic skills

2 5 3

Totals 12 37 8


APPENDICES

A. Questionnaires: Technology Integration Specialists (EOY)

B. Conventional questionnaires: Teachers (EOY)

C. Teachers random-interval, pager-triggered web survey

D. Students random-interval, pager-triggered web survey

E. About Interactive, Inc.

/tt/file_convert/5c8a3d1109d3f2bb5b8c5f3f/document.doc

Documents

THE RESULTS OF PROFESSIONAL … revised.doc · Web viewThis research documents the success of using technology to study technology. Teachers and school administrators cooperated with