A Multifaceted Study of Change in Seven Inner-City Schools

A Multifaceted Study of Change in Seven Inner-City SchoolsAuthor(s): Russell Gersten, Douglas Carnine, Leslie Zoref and Denis CroninSource: The Elementary School Journal, Vol. 86, No. 3 (Jan., 1986), pp. 257-276Published by: The University of Chicago PressStable URL: http://www.jstor.org/stable/1001544 .

Accessed: 06/01/2014 17:40

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact [email protected].

.

The University of Chicago Press is collaborating with JSTOR to digitize, preserve and extend access to TheElementary School Journal.

http://www.jstor.org

This content downloaded from 128.223.86.31 on Mon, 6 Jan 2014 17:40:44 PMAll use subject to JSTOR Terms and Conditions

http://www.jstor.org/action/showPublisher?publisherCode=ucpress

http://www.jstor.org/stable/1001544?origin=JSTOR-pdf

http://www.jstor.org/page/info/about/policies/terms.jsp


A Multifaceted Study of Change in Seven

Inner-City Schools

Russell Gersten Douglas Carnine Leslie Zoref University of Oregon

Denis Cronin Stanford University

The Elementary School Journal Volume 86, Number S

? 1986 by The University of Chicago. All rights reserved. 0013-5984/86/8603-0001$011.00

The 1970s produced a rich (if somber) series of case studies depicting the processes by which school districts modify, distort, or discontinue educational innovations (Berman & McLaughlin, 1975; Gross & Herriott, 1979). However, rarely did researchers pay much attention to the nature of an instructional intervention or the level of implementation. As Fullan (1983) notes, "The failure of educational change may be related just as much to the fact that many innovations were never implemented in practice . .." (p. 12). Leithwood and Montgomery (1982) criticize even the few studies that measure level of use of an innovation (often by means of teacher self- report): "Studies of school change typically considered degree of use of a new program or practice in the classroom as the definition of the dependent variable. These studies did not attempt to link such use to improved student outcomes ..." (p. 315).

This situation has begun to change. Good and Grouws (1979) demonstrate how concerted training in an empirically based model of teaching can have a significant impact on both teacher performance and student achievement in mathematics; An- derson, Evertson, and Brophy (1979) and Stallings (1980) found similar effects in reading. All these studies include a comprehensive assessment of implementation and empirical evidence of the relationship between model implementation and student learning.

Examining attempts to improve the quality of urban schools over about the past 15 years, Fullan (1983) concludes that successful change requires a thorough under-



258 THE ELEMENTARY SCHOOL JOURNAL

standing of both the process of change and the exact nature of the instructional innovation. He argues that it is futile to try to understand one without the other. We see a need for comprehensive studies that would not only document the classroom

processes that affect achievement (as Good and Grouws, Stallings, and others have done) but also examine the administrative

processes and decisions that affect classroom instruction. We also see a need for studies that extend beyond the 1-year time

period that previous researchers utilized. The current study is a modest first step

in that direction, an attempt at multilevel

analyses of a large-scale school improvement effort in seven inner-city schools over a 2-year period. The study examined im-

plementation of the Direct Instruction educational model (Becker, Engelmann, Car- nine, & Rhine, 1981), a model with a

history of effectiveness in inner-city set-

tings (Becker, 1978; Kennedy, 1978; Meyer, Gersten, & Gutkin, 1983). Before

proceeding further, it is important to note that there is still some controversy con-

cerning the effectiveness of direct instruction (e.g., House, Glass, McLean, & Walker, 1978) and the use of standardized achievement tests for determining which educational interventions are effective. How- ever, if one is willing to allow that standardized achievement tests are a reasonable way to assess academic and cog- nitive growth in the primary grades (as most communities do), then direct instruction appears to be a relatively consistent, effective intervention for low-income children (see Cooley & Leinhardt, 1980; Fischer et al., 1980; Rosenshine & Ber- liner, 1978).

The article is not intended as a com-

prehensive report of primary data; primary data are reported in numerous technical reports (Cronin, 1980; Gersten, 1979; Gersten, Carnine, & Williams, 1982; Ger- sten, Carnine, Zoref, & Cronin, 1982). Our intention is to synthesize the major findings of these reports and delineate tenta-

tive conclusions and areas in need of further exploration. First, we describe the

setting for the study, the nature of teacher

training, how implementation was measured, and the studies examining the re-

lationship between level of implementation and growth in achievement. Next, we

report case study findings. Important events in the implementation process are

reported. The reactions of teachers, paraprofessional aides, and administrators to a rather drastic change process are then described. Both the effects of the innovation on the community and the effects of the

community on the educational program are discussed.

The study Subjects and settings The school district is one of the 10 larg-

est in the United States. The federally supported Follow Through program was conducted at the seven schools with the

greatest concentration of low-income students. The students were predominantly black (52%) and Hispanic (43%). Twenty- three percent began as limited-English- speaking students. The mobility of the student population is extremely high, ranging from 37% to 74% student turnover per classroom per year. The teachers were primarily Caucasian; the paraprofessionals, black and Hispanic.

During the first year of the intervention (1978-1979), 10 kindergarten and 11 first-grade classrooms in seven schools were involved. During the second year (1979- 1980), eight kindergarten, nine first-grade, and nine second-grade classrooms were involved. (One school was forced officially to terminate the program after one year, in

June 1979, when it became a magnet school. The school still utilized the model, but in a less formal fashion; this school was dropped from the study. However, an eighth school began the Direct Instruction Follow Through program in 1979.)

JANUARY 1986



SCHOOL CHANGE 259

The intervention

Although in recent years the term "direct instruction" has been used in a variety of ways, in this article Direct Instruction (with initial capital letters) refers to a comprehensive educational model involving a particular curriculum, a specific way of teaching, a specific system for monitoring both student and teacher performance, and intensive staff development.

The program included only reading and oral language instruction during this 2-year period. (Subsequently, it has expanded to include mathematics.) The Distar reading curriculum materials used in this program explicitly teach students rules and strategies for oral reading and comprehension of written material. The Distar language programs teach a wide range of receptive language skills, as well as rudimentary log- ical skills.

Principles of mastery learning are ad- hered to; instructional groups do not pro- ceed to the next lesson until mastery (i.e., 85%-95% accuracy) is reached on the material from the previous lesson. Students are given individually administered, instruction-referenced tests on material covered during the preceding 2 months. Scores on these tests are used to determine whether the child (or group) needs re- mediation and whether individual students need to be placed in either an accelerated or a slower-paced group (Carnine & Ger- sten, in press). (The model is discussed in greater detail in Becker et al., 1981, and Gersten, Carnine, & White, 1984.)

Successful implementation of the Di- rect Instruction Follow Through Model hinges on intensive, ongoing interaction between Direct Instruction Model consultants (from the University of Oregon), the local supervisors (employees of the school district), and the teachers. During the first year of implementation, consultants from the University of Oregon spent almost 100 days in the district working with local supervisors and teachers on how to present the Distar programs, set up homogenous

instructional groups, and set up classroom schedules to maximize time in academic areas. The strategy that the consultants followed in training teachers was quite similar to the way Direct Instruction teachers train students in new skills: all training assignments were easily understandable and achievable, and adequate practice with corrective feedback was provided in both group practice sessions (using role playing techniques) and the classroom. Assessment of teachers' progress was frequent; remedial steps were taken when necessary. Al- though consultants from the University of Oregon initially trained the teachers, as the year progressed, the emphasis gradu- ally shifted to training the local supervisors, called resource teachers. After several months of intensive training, both the model consultants and resource teachers served as change agents and quality control agents. The fate of the implementation rested primarily in their hands.

Effects of the intervention on student achievement

A general sense of the effects of the program on student achievement can be gained by examining the median percentile reading scores on the CTBS (Compre- hensive Test of Basic Skills) (CTB, 1975) for first graders prior to the intervention (spring 1978) and the years after the intervention (spring 1979 and spring 1980). The median score in total reading on the CTBS, level B, was at the twenty-fifth percentile prior to the intervention, the thirty- eighth percentile 1 year after the beginning of the intervention, and the forty-sixth percentile 2 years after the intervention had begun.

Variation in achievement among classes In a conventional evaluation report

(such as the original Follow Through evaluation by Abt Associates), only the statis- tics for the total sample would be reported. However, in a class-by-class analysis of these scores from the second year of the inter-




vention, it is apparent that-even within this highly structured model in which the curriculum, the amount of time to be spent in each curricular area, and the mode of teaching are clearly specified and moni- tored-there was wide variance in student growth. During the second year of the intervention, a pretest, posttest, norm-referenced evaluation design was introduced. First graders were pretested in the spring of their kindergarten year on level A of the CTBS, which is essentially a preread- ing or readiness test. A year later, they were tested on level B, a test of formal reading skills including items on word attack, comprehension, and vocabulary. Ta- ble 1 presents the results for the total Fol- low Through sample for each class. (Two classes were omitted because no pretests were given at their school.)

Some classes made no progress (e.g., the classroom where students began at the thirty-fifth and ended at the thirty-seventh percentile), some made weak progress (from the twelfth to the twenty-ninth percentile), and others made tremendous gains (from the eleventh to the sixty-first, and from the twelfth to the seventy-ninth percentile). The purpose of the implementation assessment segment of the study was to help understand and explain this within- site variation. The hypothesis was that observed level of implementation of the ed-

ucational model would explain much of the variance in student achievement.

Studies relating level of implementation to student achievement

Pilot study In September 1978, a pilot study was

initiated (Gersten, Carnine, & Williams, 1982). The objectives of the study were to: (a) empirically describe stages in the implementation process (i.e., highlight trends in acquisition of new skills by teachers), (b) develop reliable and valid instru- ments to assess observed level of implementation, and (c) begin to explore relationships between teacher performance variables specified in the model and student academic growth. Despite the current widespread use of interview techniques to assess levels of implementation (Hall & Loucks, 1977), our working as- sumption was that the most valid way to measure implementation was direct observation of teacher and student behaviors in classrooms in the tradition of Good and Brophy (1974).

A final set of five variables was selected on the basis of reviews of research, interviews with supervisors, and several feasi- bility studies. Though the variables selected for this study evolved specifically from the Direct Instruction Model, they have generality well beyond this approach to instruction (e.g., Good & Grouws, 1979;

TABLE 1. Mean Total Reading Scores for Follow Through First Graders on CTBS

Pretest (Level A), May 1979 Posttest (Level B), May 1980 Follow

Through Scale Scale Class N Score SD Percentile Score SD Percentile

1 15 167.2 55.3 20 269.1 22.4 55 2 9 154.2 73.8 12 301.8 29.3 79 3 11 171.8 90.1 24 269.5 38.0 56 4 11 166.5 64.1 20 261.5 33.9 48 5 12 184 ... 35 248 ... 37 6 8 151.9 66.1 11 276.1 24.1 61 7 15 157.5 23.2 13 237.5 71.2 29

Total 78 164.2 58.9 18 259.3 53.5 46

NOTE.-SDs unavailable for class 5.

JANUARY 1986



SCHOOL CHANGE 261

Stallings, 1980). The variables were: (1) Correction procedures (teacher corrects student errors), (2) Student accuracy (during small-group instruction), (3) Pacing (teacher presentation rate), (4) Format accuracy (teacher follows manual), and (5) Signals (teacher signals with hand or voice so that group responds in unison).

One variable, student accuracy during small-group instruction, may seem more like a dependent variable to readers. How- ever, our contention was that this variable is largely under a teacher's control. By properly placing students in instructional groups, moving through the curriculum at an optimal rate for each instructional group, and providing motivation, teachers can control the student accuracy (or success) rate during each lesson. The importance of this variable has been documented by Block (1980) and Brophy and Evertson (1976). As the reader will see, it proved to be one of the most important variables in the study.

Interobserver reliabilities on the five variables ranged from .80 to .92. Internal consistency (coefficient alpha) coefficients were calculated, treating each of the six observational occasions as a test item. These coefficients were reasonably high for all variables, ranging from .60 to .82. For details on procedures, see Gersten, Car- nine, and Williams (1982).

Results: contrasted groups study. The mean total reading scores of the two first- grade teachers with the highest Direct In- struction Supervision Code (DISC) scores and the two with the lowest DISC scores were compared (see Table 2). Mean performance of students taught by teachers with high DISC scores was at or above grade level on the CTBS. The classes of the low-DISC teachers were at the twenty- second and twenty-ninth percentiles, the level reached prior to the intervention. Note that the three teacher performance variables that seem most clearly linked to student achievement are correction procedures, student accuracy, and, to a lesser

extent, pacing. No relationship was found between achievement and (a) the precision with which teachers followed the wording of the teaching format or (b) their use of hand or vocal signals to encourage unison responses from the group. These results suggest that these two variables may not be essential components of Direct Instruc- tion, unlike the other three variables. If teachers covered the material with a high student success rate, at a brisk pace, with frequent correction of errors, and without using the specified signals or exact wording, no negative effects were found. Sig- nals were conceived as a tool to achieve certain ends; these teachers achieved the same ends without using them. Appar- ently, deviations from the teaching format are not problematical, as long as the intent of the format is accurately conveyed.

This pilot study indicated two major limits of direct classroom observations: (a) the cost is high, and (b) only a relatively narrow range of teacher behavior is assessed. In the second year, it was decided that an implementation rating form (IRF), filled out by a trained supervisor, should be administered to supplement the observation system. Items on the IRF (Gersten, Meyer, & Zoref, 1979) dealt only with specific, concrete, observable phenomena, and the form was only completed by supervisors who had received extensive training in observing and monitoring the behaviors included in the form. Items were selected to include critical teacher behaviors in the domains of teaching procedures, classroom management and organization, and monitoring of student progress. A 26-item instrument was developed with internal consistency index (coefficient alpha) of .93 and interrater reliability of .81.

Intensive study of seven teachers

During the second year, two studies were conducted simultaneously. The first involved an intensive evaluation of seven teachers (four in first grade, two in second grade, one in a mixed first-/second-grade




TABLE 2. Contrasted-Groups Analysis: High and Low Total Observational Scores in First Grade

Performance Variables

Discriminating Nondiscriminating

Student Corrections Accuracy Pacing Format Signals

Dependent Variable:

CTBS Total

Reading (Mean

Percentile)a

High scores: Teacher A (83.5) 89.5 10.0 95.1 99.7 52b Teacher B 100 89.3 9.9 (74.5) 100 59b

Low scores: Teacher C (61.9) (76.8) 10.0 98.7 98.9 27 Teacher D (66.5) (55.2) (8.6) 95 (62) 22

NOTE.-Entries in parentheses indicate below-mastery-level performance. aMean scale score converted to percentile equivalent. bAbove national median (and significantly above expected score for low-income minority

students).

classroom). All seven teachers were vol- unteers. The intensive evaluation consisted of (1) direct observations of teacher performance using a streamlined version of the DISC code; (2) assessment of implementation, based on the IRF, by both a model consultant and a local supervisor; (3) assessment of implementation based on supervisors' global ratings; (4) Levels of Use (LOU) interviews (Hall & Loucks, 1977) conducted by an independent research agency.

A "streamlined" version of the DISC was used, including only the three variables that seemed to discriminate between teachers: corrections, pacing, and student accuracy. A total DISC score was com- puted by aggregating the three variables across the six observations. Internal consistency reliabilities were .96 for pacing, .84 for student accuracy, .67 for corrections, and .91 for the total DISC score. Interrater reliability for each variable ranged from .86 to .99, with a value of .97 for total DISC scores. The internal consistency reliability for the IRF was .93; interrater reliability was .81 for teachers' ratings.

The purpose of this intensive evaluation (Zoref, 1981) was to examine the relationship between level of implementa-

tion and growth in student achievement using a multi-instrument approach. The study attempted to (a) look at the relationship between the various approaches toward assessing level of implementation (concurrent validity) and (b) explore relationships between each measure and gains in reading achievement on the CTBS for a sample of seven teachers.

Concurrent validity. The DISC and IRF measures demonstrated moderate to high levels of concurrent validity. The Pearson correlation between the DISC and IRF scores was .63 (p <.10); the Spearman correlation was .82 (p <.05). The correlations of each of those measures with a supervisor's assessment of overall quality of implementation were .55 (Pearson) and .94 (Spearman); p <.05. Correlations between Levels of Use scores and two of the other measures were appreciably lower and non- significant, .17 for DISC and .27 for IRF. Because of the small sample size, all these concurrent validity correlations are exploratory; however, they do indicate consistent relationships between DISC, IRF, and global ratings, but not the Levels of Use interviews.

Relationship between level of implementation and achievement. Results are presented in Table 3. Both Pearson and

JANUARY 1986



SCHOOL CHANGE 263

TABLE 3. Correlations between Implementation Measures and Mean Class Gains in CTBS Total Reading for Teachers in Intensive Evaluation (N = 7)

Pearson r to CTBSa Spearman r to CTBSa

DISC .59 .82* IRFb .90** .93** Supervisors' global ratings .93** . LOU .61

aGain scores in expanded scale score units. bBased on the mean IRF score of two supervisors. cNot performed because there were at least two ties in the seven instances (Ferguson,

1966). *p < .05. **p < .01.

Spearman correlation coefficients are reported when appropriate. Note that all correlations are in the moderate-to-high range; four are in the .80-.90 range. Again, because of the small sample size, no strong inferences can be drawn. However, there is a trend toward moderate relationships between level of teacher performance and growth in achievement. This is true for the direct observational measure (DISC), the supervisors' rating form (IRF), and the global ratings by the supervisors.

These results suggest that the extent to which teachers followed the essential components of the model correlated with the gains that their students made in reading. Providing informational feedback when- ever student errors are made, maintaining a brisk pace during lessons, and ensuring student success of at least 80% for unison group responses appear to be directly related to how much the students learn. To some, these may seem like fairly mechanical skills. Yet this is not the case. In an earlier study (Gersten, Carnine, & Wil- liams, 1982), we observed that over half the Direct Instruction teachers had not reached a satisfactory level of performance after 1 year of experience and training. To achieve a consistently high level of student accuracy, a complex combination of teaching skills is required. Stu- dents must be sensibly placed in groups, the teacher must move through the curriculum at a reasonable pace (neither too

fast nor too slow), and the lesson must be interesting and dynamic enough to sustain students' attention. The importance of these variables is not limited to Direct In- struction curricula; in our observations of non-Direct Instruction classrooms, we found that teachers corrected student errors less than 20% of the time, that the student success rate rarely reached 80%, and that pacing was generally half the rate of these classrooms. We cannot help but wonder if concerted work in these three areas would help the non-Direct Instruc- tion teachers maximize achievement growth.

The supervisors' rating form (and global ratings) encompassed a wide range of teacher performance, including vocal inflection, classroom organization, management of seatwork, and details of procedures for correction of errors and re- mediation of chronic problems. The correlations between these broad-range measures and achievement were even higher than the DISC correlations. The fact that the relationships were replicated across a battery of three measures indicates they are unlikely to be due to chance findings (Carver, 1978).

Full-sample study. The intensive study of seven teachers included teacher observations using DISC. The full-sample study did not use DISC, making it much more cost efficient. It included all seven first- grade and six second-grade teachers in-



https://www.researchgate.net/publication/243787207_The_Case_Against_Statistical_Significance_Testing?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==


volved in the program. (Actually, two teachers at each grade level could not be included in the study because their school did not pretest students; thus, there was no way to assess their classes' growth in achievement.) The local supervisor com-

pleted the IRF, and IRF scores were correlated with achievement gains in reading on the CTBS and the Wide Range Achievement Test (WRAT) (Jastak & Jas- tak, 1966). First graders were pretested in

April 1977 on level A, form S, of the CTBS. They were posttested on level B, a

reading test. Second graders were pretested on level B in the spring of 1979 and

posttested on level C. Table 4 presents the profiles for the

seven first-grade classrooms. Note that there is a good deal of variability in gains in reading achievement on the CTBS-with class 5 showing no growth, merely going from the thirty-fifth to the thirty-seventh percentile, and others, such as class 2, going from the twelfth to the seventy-ninth percentile.

The IRF scores correlate fairly closely to the degree of academic growth regard- less of the dependent measure or model used to assess student progress. Pearson correlations were .86 for mean CTBS gain in total reading and .79 for the residual- ized gain on the CTBS (using the letter- sounds pretest score as a predictor variable). Both are significant at the .05 level. For WRAT reading, the correlation was a bit lower: r = .66, p = .06.

Table 5 presents the classroom-level implementation and achievement data for the second grade. Note that these achievement data do not show the same type of growth as the first-grade data. In fact, most classes show losses against the standardiza- tion sample of the CTBS. These students did not begin in the Direct Instruction program until sometime during the first grade, whereas the other sample began sometime in kindergarten. However, it is interesting to note that four of the six

classes show gains in word attack skills (as measured by the WRAT).

The data, again, support the hypothesis that growth in reading on both the WRAT and CTBS is correlated with assessed level of implementation. The Pearson correlations for the Implementation Rating Scale scores were .58 for CTBS gain and .94 for WRAT gain. Global supervisor ratings correlated .63 with CTBS gains and .94 with WRAT gains. These results show moderate correlations with CTBS gains and strong correlations with gains in word attack skills (WRAT gains).

Summary. All the studies indicate consistent relationships between measured level of implementation and gains in student reading. Though the sample size for any individual study is small, the fact that the essential finding is replicated across all three studies, using a range of measures, leads to the conclusion that a relationship between operationally defined "level of implementation" and gains in reading achievement does exist. Further research using larger samples is needed to ascertain exactly which components of the model are most strongly linked to achievement.

Case study As Joyce, Hersh, and McKibbin (1983) observe, educational change is socially complex. Introducing new teaching practices such as scripted formats and specified correction procedures is not as simple as it may seem. The next section of the article looks at an often neglected area, yet one that is likely to have a profound impact on educational change efforts-how teachers feel about a new practice, about how they are trained, and how they are treated. In the final section, the overall implications of the school improvement effort on the school district are discussed.

An independent research agency, un- affiliated with either the University of Or- egon or the district, conducted the case study. The investigation had two major foci: (a) teachers' and paraprofessionals'

JANUARY 1986



I 0 po -

04 04 ~lCI; 0

CIS

CA CISC C)Lnt d C CI 4 (C

oqb

D0 - 0

"0 ~ 0 CIS N

CUCIS

loot 0

?ICIS CIS >( ocDt~L

214-1 4-1 4-1 CISc, b c 4-~1 -4 k

0 0 D C)

0"

C CU

~~tUC -.Q

CIS CIS 414 ~ C II 0

po CU-

0ISCI C

CIS C N u CU

CUCI

CU 0 00

POo

V0 ~

0 r. C, poy

V V5 0~ CUIS

c "0 4

"04 0

0 >

CO0 bV

8 - u

0C

> Qj b a et - w

CU o .C

NPI V

'O 0 V b

0 ~ -d U t-t-fkC)0000

I3 *C "~

00



I "0

Cis

o oo

Cis v

"0 0

0 X0 X0k k f) CIS

C,,

0

00

0

00

CuCIS C'C

Cisu Cis o4 "0

C/)) r- t 0 Cu

"0) Q J 0Q0 Q)~

ct

Cu

V "0

0 : Cu ~jO 1kC O ~ Cu

V V o V P

0 u

3~ a

-I Cu 0 Cu

C3J UCnu

1~ 'o a I I C-

Cu U 4O(dC 0



SCHOOL CHANGE 267

attitudes and feelings about the change process and (b) administrative issues and activities relevant to implementation of the innovation. Each will be discussed sepa- rately. First, though, the context or background of the change effort is described.

Background of the study In 1978, the U.S. Department of Ed-

ucation notified a large urban school district with a self-sponsored Follow Through program that unless it immediately adopted a clearly defined approach that differed from conventional district procedures, federal funds would be terminated (Em- rick & Peterson, 1979). The program was operating in seven schools, all located within a 3-mile radius of each other. Over 95% of the school population was minority; over 90% was low-income. The De- partment suggested that the district align itself with an educational sponsor, that is, a university or educational laboratory that had a coherent curriculum and program for improving the language and reading skills of minority children.

Since a district needs assessment iden- tified oral language as a priority, the district investigated three structured programs with an oral language emphasis. By the last day of school in June 1978, no decision had been reached. During the summer the district decided to align itself with the University of Oregon's Direct In- struction Model, which utilized the highly structured Distar programs in reading, language, and arithmetic. Several reasons were cited for the decision: the favorable evaluation of Direct Instruction in the national Follow Through evaluation by Abt Associates (Bereiter & Kurland, 1981- 1982; Stebbins, St. Pierre, Proper, An- derson, & Cerva, 1977), the high test scores from the one bilingual school in the district that used Distar during the self-sponsored years, and the program's oral language emphasis. The district wanted to use only the language program and continue with the variety of reading and math pro-

grams then being used in the schools. The national Follow Through office decided that partial implementation of a model's program did not meet the Follow Through guidelines. In August, the district agreed to implement Direct Instruction reading and language in all the kindergarten and first-grade classrooms participating in the Follow Through program. A total of 10 kindergarten and 11 first-grade classrooms were involved. The delayed nego- tiations meant that most Follow Through teachers did not know about the change until early September when they reported to school for 2 days of preservice training.

Under the district's own loosely defined former Follow Through approach, teachers determined the reading, arithmetic, and language arts curriculum for their classrooms-whether to use a phonics or language experience approach, have an open classroom, or provide a structured classroom atmosphere. They also determined the amount of time spent in each area, the relative proportion of time spent in oral reading versus supervised seatwork, and even whether to implement an oral language program. Kindergarten teachers could choose whether to teach reading. Little systematic work was done in oral language. Local supervisors served as resource teachers by tutoring children with reading problems and occasionally offer- ing suggestions on curriculum programs, but only upon a teacher's request.

Follow Through teachers returned to school in the autumn of 1978 to a curriculum and instructional techniques that were completely new to most of them. Teachers were required to arrange their class schedules so that all students received 30 min per day of intensive small-group instruction in reading and language. Con- sultants from the University of Oregon, as well as the local supervisors, made unannounced visits to classrooms to observe the extent to which the educational model was being implemented and to assess areas in which teachers needed help or technical




assistance. In-service training was provided to all teachers on a weekly basis, whether or not they requested such assistance.

Change was massive for many teachers-and overwhelming to some. To move, in so short a time, from a loose, laissez- faire educational model with no clear ac- countability system to a highly structured model with clear-cut techniques for monitoring both teacher and student performance is an unusual phenomenon in public education.

Perceptions of teachers and paraprofessionals Year one. Each of the teachers and

paraprofessionals in Follow Through (21 teachers and 21 aides) was interviewed in March 1979, toward the end of the first year of implementation. The administration of the Levels of Use interview (Loucks, Newlove, & Hall, 1975) took about 10-15 min; after that, the interviewer asked a series of questions related directly to issues in the implementation of Direct Instruc- tion. The total interview lasted approxi- mately 60 min. After the interview, each participant filled out a Stages of Concern questionnaire (Hall & Loucks, 1978).

These semistructured interviews (Em- rick & Peterson, 1979) revealed a wide range of teacher concerns. The early stages of the implementation effort were char- acterized by a great deal of misunderstand- ing and miscommunication. Virtually all teachers cited the lack of lead-in time (which had consisted of only 2 days of in- service training) as catastrophic. Some teachers indicated that they were upset at not having any input in the decision to change Follow Through models. They felt that the district and the federal Follow Through office were imposing it on them: "Preservice would have been more helpful if I hadn't been so angry. I didn't really listen to very much after we were told (by a district staff member), 'This is what we will be doing this year. Like it or get out.' "

The hurriedly scheduled preservice training was most teachers' first exposure to Direct Instruction; many of these first impressions were not favorable: "There was alienation in the beginning because Oregon presented themselves as 'asked to come here to get you out of a problem.' Some teachers would have liked a choice." It is tempting to say that, in the future, teachers should always be informed about major changes in policy well in advance. However, as Crandall et al. (1983) report, the median length of time between a district's adoption of a new educational program and its implementation tends to be a mere 3.5 months.

During the beginning of the year, outside consultants from the University of Or- egon were unanimously criticized for un- realistically high expectations: "I have worked extremely hard to do Distar cor- rectly, but I have had no reward or rec- ognition for this from the consultants." "They expect you to do everything per- fectly without any experience. Their atti- tude is bad; it seems that they're criticizing you instead of helping."

A number of teachers voiced concerns over the strong emphasis on academics (leaving little or no time for "fun" activities). This was especially true for the kindergarten teachers. Previously, kindergarten was unstructured with little time on academics. Teachers recalled questioning the use of scripted presentations, feeling they were overly mechanical. At least half the teachers experienced some sort of philosophical conflict between the highly structured, basic skill approach and their own views that the teacher of young children should attend to the "whole child," fostering his or her social and emotional as well as academic growth.

According to the Oregon consultants, these initial reactions were not unexpected; they had experienced similar problems in other districts. They believed initial resistance could be overcome with clear demonstrations of district commitment to

JANUARY 1986



SCHOOL CHANGE 269

the program. Also, they felt that once teachers used the program successfully and saw its impact on the students, their attitudes would change.

An early, crucial conflict centered on the weekly monitoring activities by the consultants, particularly the unannounced classroom visits. Teachers felt so strongly about this issue at two schools that they pressured several principals into asking the consultants to announce the exact time of the observation (so that teachers could prepare). The consultants felt this de- feated part of the purpose of the visit.

A strongly worded memo of support for the consultants' policy from the director of compensatory education (with the active support of the superintendent's office in November) helped reduce principals' interference. By late winter, classroom personnel generally came to accept the intensive monitoring and assistance as inevitable, even if some still found it anx- iety provoking. Several teachers seemed to find in-classroom assistance more helpful than the formal group training sessions. One reported, "Demonstrations in the classroom are the most helpful part of training. It's a real-world test of how Dis- tar operates. More demonstrations in my own classroom would have been helpful, especially in the beginning." By the end of the year, even those teachers who expressed strong initial conflicts about the Direct Instruction model and the events surrounding its implementation indicated they had seen dramatic improvements in pupil performance and half saw Distar as a powerful tool that increased their effectiveness as teachers. The following comments were made prior to receiving the end-of-the-year achievement test data: "The effects are extraordinary; the children have real reading skills. The effects are generalized; I see them in other subject areas." "Never before did children sit on the edge of their seats, waiting to give me the answer. With Distar they do. I'll never go back to the old ways."

Even with these statements of support, however, the interviews indicated that the first year had been a difficult and often

painful time for teachers. We know that significant change extracts its costs (Fullan, 1983). In this case, the costs to the district, the classroom staff, and the external sponsor were high.

Year two. During March and April of the second year of implementation, 23 of the 27 teachers involved in the program were interviewed (Cronin, 1980, 1983). The same basic procedure was followed. There were more teachers the second year because Follow Through now included second grade, as well as kindergarten and first grade. Ten of the teachers were new to the model; 13 had some experience.

By this time it appeared that major problems in the implementation process had been worked out. The fundamental theme in this set of interviews was attitudes toward the highly structured program. All 23 teachers indicated that they now "agreed with the main objectives of Direct Instruction as a program for educationally disadvantaged students." All but one felt that "Direct Instruction was compatible with their educational philosophy," al-

though some said Distar had initially clashed with their holistic philosophy (Cronin, 1980, p. 23). One of the veteran teachers explained that her initial percep- tion of disharmony between Direct In- struction and her former "person-centered" educational philosophy during the prior school year had been more apparent than real. She emphasized the importance of actually teaching Distar before an ac- curate appraisal of its underlying philosophy could be made.

The newer teachers reported no such clear philosophical conflicts. Most became interested in Direct Instruction when they heard of its success the preceding year. Al- though a number of teachers recognized that they should be able to supplement Distar with other activities, most had found




this to be more the ideal, rather than the actual, state of affairs in their classrooms.

There appeared to be three distinct sets of teachers. The first group, comprising about one-fifth of the sample (five teachers), indicated that they always liked structured programs; these teachers seemed to thrive on the regular feedback and support that the supervisors and consultants gave. They had no philosophical objec- tions to Direct Instruction and tended to praise the Distar curricula and the consultants consistently.

A second group of about six teachers disliked the Direct Instruction philosophy and approach, finding it overly mechanical and rigid, both for themselves and their students. They thought that "fun" activities were essential for primary-grade students and felt it was impossible to include these in the highly structured, academic Direct Instruction program. Though they acknowledged that the program helped raise test scores, they doubted these gains would endure and believed that students- and teachers-needed a more natural, less routinized learning environment. They indicated that they would leave the program as soon as they could. A few stated that they did not wish to continue teaching disadvantaged students and would prefer middle-class schools.

The third and largest group of teachers (about 55%) is the most fascinating. They tended to be ambivalent. On the one hand, they said that the "mechanical nature" of Direct Instruction with its scripted lessons went against their educational values and philosophies and made them feel a little less like teachers. Many of these teachers indicated that Follow Through needed "a program like Distar." Yet when the interviewer asked them to explain what they meant, to describe what kind of alterations they would make, they were unable to ar- ticulate these. Despite their qualms, most were eventually convinced of the efficacy of the model because of the improvement in reading and oral language that students

in their classrooms made. Many said the academic progress was at a level they had not previously believed possible. Cronin (1980) summarized the results as follows: "Teachers in all cohorts seemed to derive great satisfaction from seeing their children read, speak in correct sentences, and attain a more positive self-concept from these skills. In addition to the progress that their students achieved in the basic skills areas teachers mentioned increased self- reliance, greater social maturity, and a de- crease in 'acting out' behaviors from their students as additional byproducts of their enrollment in the Direct Instruction program" (p. 24).

Cronin also reported marked changes in the attitudes toward supervision: "A solid 50% of the teachers reported deriv- ing benefit from classroom visits by the Or- egon consultants, particularly from visits by the Project Manager, about whom 75% of the teachers (and most principals) had some very positive things to say. She was

typically viewed as approachable, techni- cally knowledgeable, and highly respon- sive to the needs of teachers and principals alike. Her feedback in the classroom was generally considered clear and very relevant to the teachers' day-to-day problems" (p. 28).

Thirty percent still found some incon- sistencies between the three outside consultants, or between the consultant and the local supervisor. About 40% of the teachers did indicate that, at times, they felt some of the Oregon consultants were "checking up" on them, and they wished for fewer classroom observations and more actual demonstrations of teaching techniques by the consultants in the classroom.

Whereas the supervisors and consultants were ultimately perceived in a positive light, virtually every teacher and aide in this site found the building principal to be irrelevant to the success of the implementation, indicating that principals seemed neither to support nor to hinder the program. When interviewed, most

JANUARY 1986



SCHOOL CHANGE 271

principals regretted their inability to deal with the day-to-day details of classroom instruction. All agreed that they were forced to operate as managers because of the

presence of as many as eight disparate budgets at their schools and the multitude of federal programs. Three of the seven

principals indicated that they would have liked to become more involved in supervising classroom instruction.

Visible support by the central administration (deputy superintendent, director of compensatory education) was seen as critical by 53% of teachers interviewed. These were predominantly the veterans of the first year's struggles, when several

principals attempted to change the supervision procedures, until the central administration prohibited them. During the second year, there were no overt conflicts between the building principals and the Direct Instruction staff.

Only 30% of the teachers felt there was evidence of peer support. This modest amount of collegial support is interesting. School-level analyses that Peterson (1980) conducted revealed that most new teachers did feel supported by their more ex-

perienced colleagues in the school. Vet- eran teachers did help new teachers with small procedural and technical problems and told them it would take a while to get the feel of the program. However, at the one school newly added to the program, where all teachers were new to Direct In- struction, no collegial support was re-

ported. Stress (as measured on the Stages of Concern questionnaire) was at its highest here. The blending of veterans and newcomers to a new model appears to be the least stressful, most productive combination, if and when it is feasible.

Implementation in a broader context: the community, the courts, the press

Early in the second year of the program, the court, which had been overseeing the district's voluntary desegregation program, ordered an examination of the

quality of education in the district's 19 mi-

nority-isolated schools. This represents a slow but growing trend in court-ordered

desegregation decisions, in which the quality of classroom instruction, as well as the racial composition of schools, is considered. The judge appointed three educators from outside the district to observe educational practices and then recom- mend procedures likely to improve the

quality of education in these schools. The seven Follow Through schools were among the 19 schools targeted. The court-appointed team cited the following concerns about the typical educational practices in these schools:

1. A tendency to blame poor academic achievement on the students, their fami- lies, and the ethos of poverty, rather than on school and classroom practices. Prin-

cipals cited factors such as poverty, eth-

nicity, and lack of parental interest as the cause of academic failure rather than se-

riously looking at what occurred in the classroom. The complacency of the principals at most of these schools was sharply criticized. Generally, the observers found that principals were rarely knowledgeable on instructional issues.

2. The extensive use of "pull-out" procedures for remedial reading, Title I, bi-

lingual and special education, and so forth, and the consequent diffusion of responsibility for student academic growth. If students were not learning to read, the blame was shifted between the classroom teacher, the Title I or remedial reading experts, the bilingual specialist, and the curriculum

specialist, with no one assuming ultimate

responsibility. 3. The absence of intensive, high-qual-

ity staff development programs that might provide classroom teachers with specific suggestions for improving educational ser- vices in the classroom.

The observers found two exceptions to this generally gloomy picture, the seven schools with the Direct Instruction Follow

Through program and two schools with a




bilingual program specially developed by the district. In both these programs, they found that students spent most of their time

actively engaged in reading, math, and lan-

guage activities. They also cited the prag- matic, down-to-earth technical assistance that the supervisors in both programs offered.

The judge's decision reflected the im-

portant aspects of the court-appointed team's report. Here are some excerpts:

Recent experience with schools in poor minority urban districts reveals that if the schools are properly run, virtually all students, no matter how disadvantaged, can be taught to read, write and calculate at a level sufficient to function in American society....

Traumatic though it may be to the community, busing is an easy way out- temporarily. Busing can be carried out by superintendents reluctant or unwill- ing to do so. It is visible, easily enforced by the Court, and immediately satisfy- ing to those minorities who see it as a symbol of victory over the white community or who have been misled to be- lieve that it will magically produce bet- ter educational results for their children.

Educating children requires willing cooperation on the part of the top administrators, principals, teachers, and the organizations that represent them. It requires hard work, inspiration, imag- ination and perseverance. It is less news- worthy and less visible, but more ful- filling. It works to lift the children out of isolation-permanently! [Welsh, 1980, p. 15, 26]

In addition to attention from the court, the Direct Instruction Follow Through program also received extensive media coverage with a total of nine articles ap- pearing in the city's major newspapers, beginning in the summer of 1980, when the district was considering expansion to other schools. The following is an excerpt from a lead editorial in a prominent local paper:

Direct Instruction has achieved results far surpassing any of the other 15 or so experimental techniques tested in

minority schools around the country since the late 1960s... It delivers what other methods only promise, namely, raising the reading and math test scores of disadvantaged children to national norms of their grade levels. That being the case, it seems absurd to oppose Dis- tar on the grounds of an arbitrary, and discredited, philosophical bias....

The superior court judge, who is overseeing the district's court-ordered voluntary integration plan, has indicated he favors a wider implementation of the ... method. So do we. And we trust that a majority of the school board will agree when it meets on Tuesday.

The district responded to the judge's decision by expanding Direct Instruction programs into 18 new classrooms. More important, a new instructional program was devised for all 19 schools that incorporated many, but not all, of the components of the Direct Instruction Model. Although the non-Follow Through schools main- tained traditional basal reading and arithmetic programs, they modified them sub-

stantially to approximate more closely some

aspects of the Direct Instruction curriculum. Other features of Direct Instruction incorporated were (a) reorganization of the day to include at least 21/2 hours of highly structured, small-group instruction in reading and math; (b) daily structured practice in word attack skills and vocabulary (appreciably more than the reading series provided); (c) the use of instruction- ally referenced tests to assess student mastery of each 2-week unit; and (d) intensive classroom observation of teachers to insure that they were implementing the program.

Principals were given biweekly training sessions by the assistant superintendent on issues in supervising classroom instruction, criteria for observing classrooms, use and interpretation of criterion-referenced tests, and procedures for upgrading classroom management skills and minimizing time wasted in transition.

The district's expansion and dissemination of Direct Instruction exemplifies

JANUARY 1986



SCHOOL CHANGE 273

what Berman and McLaughlin (1975) label a mutally adaptive mode-several components are taken from the original innovation, but many are altered to meet the current situation in the district.

Conclusions In this study, consistent high-to-moderate relationships were found between observed level of model implementation and classroom achievement gains in reading. At least for highly structured models of instruction, degree of implementation can be measured in a reliable and valid fashion.

Assessing level of implementation directly is neither easy nor inexpensive. The type of information gathered can be extremely valuable, however. In this case, we found that certain teacher performance variables (such as use of hand signals to insure unison response or following the exact wording of the teachers' manual) did not appear to be related to achievement gains, whereas others (such as provision of immediate corrective feedback when students make errors) appear to be strongly related to academic growth. By examining relationships over a 2-year period in a natural setting, we were able to explore the replicability of findings. Even in a highly structured model such as Direct Instruction, in which time allocation, procedures for grouping students, and curriculum are tightly con- trolled, we found considerable variability in implementation.

The case study component on the ev- olution of attitudes of the teachers and paraprofessional aides was certainly more exploratory in nature. Two conclusions seem of special interest, especially in view of the wave of programs recently initiated in major urban areas to improve the teaching of basic skills. The first is that teachers may at first react negatively-or be con- fused by-intensive, structured, in-class training (or technical assistance). Yet, ultimately, at least half the teachers found this to be one of the most positive features of the intervention. This result parallels

major findings by Berman and Mc- Laughlin (1975), Fullan and Pomfret (1977), and Walberg (1983) on successful strategies for inducing and maintaining change in educational settings. It is also similar to recent findings concerning the training of business managers (Dreilinger, McElheney, Rice, & Robinson, 1982).

The other key finding is that many teachers altered their reactions to structured educational models after they saw the effects of this program on their students on a day-to-day basis. Often this transfor- mation took many months. At the beginning, teachers were far from enthusiastic about the program and tended to feel that too much time was devoted to academics. Not enough was set aside for "fun" or cre- ative activities. Yet their strong support by the second year was unequivocal. From teacher interview data collected through- out the 2 years, there can only be one main explanation for this, namely, the effect of the model on student performance. Time and again the teachers marveled at the new academic skills their pupils demonstrated. Teachers reported anecdotal evidence of growth well before the standardized achievement tests were administered.

Traditionally, when researchers examine teachers' attitudes and motivation, they study teachers' attitudes and values prior to an intervention and see how these affect how well teachers implement the innovation. Our results and those of Guskey (1984) demonstrate that participation in an effective intervention may radically al- ter teachers' value systems. More research on this phenomenon needs to be done.

Both of these findings have implications for the study of implementation of innovative practices. When examining intensive change efforts that utilize educational programs with histories of consistent, positive effects-programs such as active teaching, mastery learning, and direct instruction-the issues of "mutual ad- aptation" and fidelity of implementation are cast in a very different light. Kennedy




(1978) concluded that the assertive, non- adaptive tactics utilized by the Direct In- struction Model were probably the major reason for its success in dealing with the intricate bureaucracies of a large urban school system. These same assertive tactics would not be warranted (or tolerated) in the implementation of a procedure in which a strong, positive impact on children could not be empirically documented.

The positive response from the courts and the newspapers to the innovation was, by and large, unexpected. The irony of the situation was that community interest grew because of the support of teachers who were

initially resentful and antagonistic toward Di- rect Instruction. The situation in this large urban district was unique in that both the U.S. Department of Education and the courts were unusually assertive. In most urban districts, policies and plans for school improvement do not find their way into the classroom or are implemented in a very diluted form (David, 1983).

On the basis of these findings and those of other researchers, we contend that dramatic improvement in inner-city schools is possible. The recently completed Carne- gie study of school improvement found that the quality of instruction in problem schools improved only when there was someone in the school with a clearly artic- ulated vision of what should be going on in classrooms and who helped teachers with the details of instruction (David, 1983). The interviews with teachers in this project would substantiate David's findings. These findings suggest a very different concept of teacher education from what is currently in vogue, viewing it as an ongoing (occasionally tempestuous) process that goes on well beyond the day a teacher graduates from college with a teaching certificate. We have argued earlier (Ger- sten, Carnine, & Green, 1982) that it is neither feasible nor reasonable to ask the principal to serve as instructional leader, trainer, observer, consultant, and expert in diagnosis and placement. This study

suggests that programs can succeed with virtually no principal involvement, as long as teachers are well trained and consultants and supervisors are working with teachers. Ideally we see a combination, with a principal (or vice-principal) and a supervisor working as a team. Our current research is examining how these teams operate in inner-city schools and documenting teachers' perceptions of their success and failure.

Many potential questions remain: Can these administrative and supervisory func- tions also be empirically linked to student achievement? How do teachers' attitudes evolve over 5- or 6-year periods? What are viable strategies for maintaining innovative projects in schools? And how can implementation of massive change in schooling be done with minimal turmoil? Knowing what to do is moot if educators do not know how to implement effective programs and keep them going.

Note

This research was supported in part by U.S. Office of Education grant G007507234 and Na- tional Institute of Education grant NIE-G-81- 0010. The views expressed here do not represent those of either agency. The reports on which this paper are based were presented at the annual conferences of the American Edu- cational Research Association in 1981, 1982, and 1983. The editorial excerpt is quoted with permission from the San Diego Union. Without the active support ofJohn Rodriguez and Susan Green of the U.S. Department of Education, a project of this scope would have been impossible. Adrienne Allen was the coordinator for the observational component of the project and greatly helped the first author see the often unrealized potential in the process of teacher training of paraprofessionals. The authors wish to thank Mardi Klotz, Sissel Lemke, and Bob Ollar for their assistance in preparing numerous drafts of this manuscript. Jane Arends, Tom Good, Glen Fielding, Susan Green, Mary Ken- nedy, and Mary Moffat offered many helpful comments on earlier drafts; Bill Green offered incisive feedback on later versions. Direct re-

JANUARY 1986



SCHOOL CHANGE 275

quests for reprints or inquiries to: Russell Ger- sten, Follow Through Project, #1751A Alder Street, College of Education, University of Or- egon, Eugene, OR 97403.

References

Anderson L. M., Everston, C. M., & Brophy, J. E. (1979). An experimental study of effective teaching in first-grade reading groups. Elementary School Journal, 79, 193- 220.

Becker, W. C. (1978). The national evaluation of Follow Through-behavior theory based programs come out on top. Education and Urban Society, 10, 431-458.

Becker, W. C., Engelmann, S., Carnine, D., & Rhine, W. R. (1981). The Direct Instruc- tion Model. In W. R. Rhine (Ed.), Encour- aging change in America's schools: A decade of experimentation (pp. 95-154). New York: Ac- ademic Press.

Bereiter, C., & Kurland, M. (1981-1982). Were some Follow Through models more effective than others? A constructive look at the Follow Through evaluation. Interchange, 12, 1-22.

Berman, P., & McLaughlin, M. W. (1975). Fed- eral programs supporting educational change: Vol. 4. The findings in review. Santa Monica, CA: Rand.

Block, J. H. (1980). Success rate. In C. Denham and A. Lieberman (Eds.), Time to learn (pp. 95-106). Washington, DC: National Insti- tute of Education.

Brophy, J., & Evertson, C. (1976). Learningfrom teaching: A developmental perspective. Boston: Allyn & Bacon.

Carnine, D., & Gersten, R. (1985). The logistics of educational change. In J. Osborn, P. Wilson, & R. Anderson (Eds.), Reading education: Foundations for a literate America. Boston: Heath.

Carver, R. (1978). The case against significance testing. Harvard Educational Review, 48, 378- 399.

California Test Bureau (CTB). (1975). Compre- hensive test of basic skills. Monterey, CA: CTB & McGraw-Hill.

Cooley, W. W., & Leinhardt, G. (1980). The instructional dimensions study. Education Evaluation and Policy Analysis, 2, 7-25.

Crandall, D., Loucks, S., Bauchner,J., Schmidt, W., Eiseman, J., Cox, P., Miles, M., Hub- erman, M., Taylor, B., Goldberg, J., Shive, G., Thompson, C., & Taylor,J. (1983). Mas-

ter report series of the study of dissemination efforts supporting school improvement. Andover, MA: Network.

Cronin, D. P. (1980). Implementation study, year 2: Instructional stafinterviews. Los Altos, CA: Emrick.

Cronin, D. P. (1983). Personal and organizational factors affecting educational program implementation: A case study. Ann Arbor, MI: Uni- versity Microfilms International.

David, J. (1983, October). The Carnegie studies of school improvement. Paper presented at annual conference of Oregon Educational Re- search Association, Newport, OR.

Dreilinger, C., McElheny, R., Rice, D., & Ro- binson, B. (1982). The promise of leadership-style training: An outdated myth? Training and Development, 36(8), 69-75.

Emrick, J. A., & Peterson, S. M. (1979). Imple- mentation study: Case study report and first-year measurement development: Pt. I. Introduction and case study, report. Los Altos, CA: Emrick.

Ferguson, G. (1966). Statistical analysis in psychology and education (2d ed.). New York: McGraw-Hill.

Fisher, C. W., Berliner, D. C., Filby, N. N., Marliave, R., Cahen, L. S., & Dishaw, M. M. (1980). Teaching behavior, academic learning time and student achievement: An ov- erview. In C. Denham & A. Lieberman (Eds.), Time to learn (pp. 7-32). Washington, DC: National Institute of Education.

Fullan, M. (1983). The meaning of educational change. New York: Teachers College Press.

Fullan, M., & Pomfret, A. (1977). Research on curriculum and instruction implementation. Review of Educational Research, 47(2), 335-397.

Gersten, R. (1979). A naturalistic stud)' of implementation of a structured educational model in a large urban school district: Interim report (Technical Report No. 79-1). Submitted to Office of Education, U.S. Department of Health, Education and Welfare.

Gersten, R., Carnine, D., & Green, S. (1982). The principal as instructional leader: A second look. Educational Leadership, 40, 47-50.

Gersten, R., Carnine, D., & White, W. (1984). The pursuit of clarity: Direct Instruction and applied behavior analysis. In W. He- ward, T. E. Heron, D. S. Hill, & J. Trap- Porter (Eds.), Focus on behavior analysis in education (pp. 38-57). Columbus, OH: Mer- rill.

Gersten, R., Carnine, D., & Williams, P. (1982). Measuring implementation of a structured educational model in an urban setting: An observational approach. Educational Evalu- ation and Policy Analysis, 4, 67-79.




Gersten, R., Carnine, D., Zoref, L., & Cronin, D. (1982, March). A study of educational change in an urban setting: Integrating teacher effectiveness and implementation research. Pa- per presented at the annual meeting of the American Educational Research Associa- tion, New York.

Gersten, R., Meyer, L., & Zoref, L. (1979). Im- plementation rating scale. Unpublished manuscript, University of Oregon, Eugene.

Good, T., & Brophy, J. (1974). An empirical investigation: Changing teacher and student behavior. Journal of Educational Psy- chology, 66, 399-405.

Good, T. L., & Grouws, D. A. (1979). The Mis- souri Mathematics Effectiveness Project. Journal of Educational Psychology, 71, 355- 362.

Gross, N., & Herriott, R. E. (1979). Theoretical and practical implications. In R. E. Herriott & N. Gross (Eds.), The dynamics of planned educational change. Case studies and analyses. Berkeley: McCutchan.

Guskey, T. R. (1984). The influence of change in instructional effectiveness upon the affective characteristics of teachers. American Educational Research Journal, 21(2), 245-260.

Hall, G. E., & Loucks, S. (1977). A developmental model for determining whether the treatment is actually implemented. Ameri- can Educational Research Journal, 14, 264- 276.

Hall, G. E., & Loucks, S. F. (1978). Teacher concerns as a basis for facilitating and personalizing staff development. Teachers Col- lege Record, 80, 36-53.

House, E. R., Glass G. V., McLean, L. D., & Walker, D. F. (1978). No simple answer: Critique of the "Follow Through" evaluation. Harvard Educational Review, 28, 128- 160.

Jastak, T., & Jastak, B. (1966). Wide range achievement test. Wilmington, DE: Guidance Associates.

Joyce, B., Hersh, R., & McKibbin, M. (1983). The structure of school improvement. New York: Longman.

Kennedy, M. (1978). Findings from the Follow Through planned variation study. Educa- tional Researcher, 7, 3-11.

Leithwood, K. A., & Montgomery, D. J. (1982). The role of the elementary school principal in program improvement. Review of Educa- tional Research, 52(3), 309-339.

Loucks, S. F., Newlove, B. W., & Hall, G. E. (1975). Measuring levels of use of the innovation:. A manual for trainers, interviewers, and raters. Austin, TX: Research and Develop- ment Center for Teacher Education, Uni- versity of Texas.

Meyer, L., Gersten, R., & Gutkin, J. (1983). Direct Instruction: A Project Follow Through success story in an inner-city school. Elementary School Journal, 84, 241- 252.

Petersen, S. M. (1980). Implementation study, year 2: Results from administration of pre-stages of concern questionnaire. Los Altos, CA: Em- rick.

Rosenshine, B. V., & Berliner, D. C. (1978). Ac- ademic engaged time. British Journal of Teacher Education, 4, 3-16.

Stallings, J. (1980). Allocated academic learning time revisted, or beyond time on task. Educational Researcher, 9, 11-16.

Stebbins, L., St. Pierre, R. G., Proper, E. C., Anderson, R. B., & Cerva, T. R. (1977). Ed- ucation as experimentation: A planned variation model (Vols. 4A-4D). Cambridge, MA: Abt.

Walberg, H. (1983). Staf, school and workshop influences on knowledge use in educational improvement efforts. Paper presented at the annual meeting of the American Educational Research Association, Montreal.

Welsh, L. M. (1980, September). Kari Karlin et al. v. Board of Education of the School District. Memorandum of Intended Decision. Superior Court of California.

Zoref, L. S. (1981). A prototypefor how to evaluate implementation of a structured educational program. Unpublished doctoral dissertation, University of Oregon, Eugene.

JANUARY 1986



https://www.researchgate.net/publication/258183073_The_Role_of_the_Elementary_School_Principal_in_Program_Improvement?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==




https://www.researchgate.net/publication/249134258_Direct_Instruction_A_Project_Follow_Through_Success_Story_in_an_Inner-City_School?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==





https://www.researchgate.net/publication/279742170_No_Simple_Answer_A_Critique_of_the_Follow_Through_Evaluation?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==





https://www.researchgate.net/publication/239328434_A_Developmental_Model_for_Determining_Whether_the_Treatment_Is_Actually_Implemented?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==





https://www.researchgate.net/publication/233224293_Academic_engaged_time?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==



https://www.researchgate.net/publication/250184661_The_Influence_of_Change_in_Instructional_Effectiveness_Upon_the_Affective_Characteristics_of_Teachers?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==




https://www.researchgate.net/publication/284853689_Teacher_concerns_as_a_basis_for_facilitating_and_personalizing_staff_development?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==




https://www.researchgate.net/publication/239958576_Education_as_experimentation_A_planned_variation_model?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==




https://www.researchgate.net/publication/250182379_Findings_from_the_Follow_Through_Planned_Variation_Study?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==



https://www.researchgate.net/publication/232548730_Changing_teacher_and_student_behavior_An_empirical_investigation?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==




https://www.researchgate.net/publication/250182355_Allocated_Academic_Learning_Time_Revisited_or_Beyond_Time_on_Task?el=1_x_8&enrichId=rgreq-b27bf52e605efae5deb12ec2f24b06ff-XXX&enrichSource=Y292ZXJQYWdlOzI0OTEzNDI1OTtBUzoxNTk4OTc3NDQzMjI1NjBAMTQxNTEzNDAwMTQ3Mw==



Documents

A Multifaceted Study of Change in Seven Inner-City Schools