Instructional Conversations and Literature Logs · PDF fileWWC Intervention Report Instructional Conversations and Literature Logs ... language arts program. ... broadening English

1WWC Intervention Report Instructional Conversations and Literature Logs October 26, 2006

WWC Intervention Report U.S. DEPARTMENT OF EDUCATION

What Works ClearinghouseEnglish Language Learners October 26, 2006

Instructional Conversations and Literature LogsProgram description This WWC report examines the effect of Instructional Conversa-

tions and Literature Logs used in combination. The goal of Instruc-

tional Conversations is to help English language learners develop

reading comprehension ability along with English language profi-

ciency. Instructional Conversations are small-group discussions.

Acting as facilitators, teachers engage English language learners

in discussions about stories, key concepts, and related personal

experiences, which allow them to appreciate and build on each

others’ experiences, knowledge, and understanding. Literature

Logs require English language learners to write in a log in response

to writing prompts or questions related to sections of stories.

These responses are then shared in small groups or with a partner.

Research Two studies of Instructional Conversations and Literature

Logs met the What Works Clearinghouse (WWC) evidence

standards with reservations.1 The two studies included over

200 Hispanic English language learners from grades 2–5. The

two studies reviewed for this report assess program impacts

in two different contexts; one focuses on the short-term (use

of the intervention over a few days) and the other focuses on

the long-term (use of the intervention over a few years) with

the intervention delivered as key components in a broader

language arts program.

Effectiveness Instructional Conversations and Literature Logs was found to have potentially positive effects on reading achievement and English

language development.

Reading achievement Mathematics achievementEnglish language development

Rating of effectiveness Potentially positive effects Not reported Potentially positive effects

Improvement index2 Average: +29 percentile points

Range: +24 to +33 percentile

points

Not reported Average: +23 percentile points

Range: +21 to +24 percentile

points

1. The evidence presented in this report is based on available research. Findings and conclusions may change as new research becomes available.2. These numbers show the average and range of improvement indices for all findings across the studies.

2Instructional Conversations and Literature Logs October 26, 2006WWC Intervention Report

Additional program information

Developer and contactThe Instructional Conversations and Literature Logs intervention

was developed by William Saunders and Claude Goldenberg

and is available from the publisher, the Center for Research on

Education, Diversity & Excellence (CREDE). Address: Graduate

School of Education, 1640 Tolman Hall, University of California,

Berkeley, CA 94720-1670. Email: [email protected]. Web:

http://crede.berkeley.edu/research/crede/research/llaa/1.5pubs.

html. Telephone: (510) 643-9024.

Scope of useInformation is not available on the number or demographics of

students, schools, or districts using this intervention.

TeachingThis WWC report focuses on a specific approach to combining

the Literature Logs and Instructional Conversations used in the

Saunders and Goldenberg (1999) and Saunders (1999)3 studies. In

the Saunders and Goldenberg (1999) study, the instructional com-

ponents were used on alternating days with the same story. For the

Literature Log component, teachers assigned entries for different

segments of readings. English language learners were then asked

to write about personal experiences relevant to a character in the

story, provide a detailed description of an event that occurred in

the story, and evaluate a theme from the story. They wrote in their

logs independently and then participated in a discussion led by

the teacher comparing their own personal experiences to those of

the character in the story. For Instructional Conversation lessons,

the teacher facilitated discussions with English language learners

in small groups about the content of the story. They participated

in discussions for approximately 45 minutes per week. The

discussions provide an opportunity for teachers to assess story

comprehension and assist in broadening English language learn-

ers’ understanding of the story content and themes.

The two studies reviewed in this report used the intervention in

different ways. In the Saunders and Goldenberg (1999) study,

Instructional Conversations and Literature Logs was used as a

short-term intervention and delivered within four days. In the Saun-

ders (1999) study, the intervention was delivered as a key compo-

nent of a broader language arts program delivered from grades 2–5.

Research CostNo information is currently available on the cost of the intervention.

Two studies reviewed by the WWC investigated the effects of

Instructional Conversations and Literature Logs. One study

(Saunders & Goldenberg, 1999) was a randomized controlled trial

that met WWC evidence standards with reservations due to dif-

ferences between groups in the amount of instruction provided.4

The other study (Saunders, 1999) was a quasi-experimental

design that met WWC evidence standards with reservations.

Saunders and Goldenberg’s (1999) short-term study matched 58

students from two fourth grade and three fifth grade classrooms on

language and reading proficiency and randomly assigned them to

either the intervention or comparison group within each classroom.5

Thirty-two of these students were English language learners and

are the focus of this intervention report. The intervention group was

3. In addition to Instructional Conversations and Literature Logs, Saunders (1999) included 10 common instructional components in the experimental group and employed them over a five-year period. Readers may want to view the results of this study and Saunders and Goldenberg (1999) separately in order to judge the effectiveness of the Instructional Conversations and Literature Logs alone versus in combination with a variety of other more common instructional components.

4. The study reported a confound between the intervention and the amount of instructional time because students in the intervention condition had more instructional time than those in the comparison group. This makes it difficult to ascertain whether the observed group differences resulted from the dif-ference in the amount of instruction or type of instruction.

5. Saunders and Goldenberg (1999) included two additional study groups (Instructional Conversations only and Literature Logs only). The results pertaining to these groups were not reviewed in this intervention report.

http://crede.berkeley.edu/research/crede/research/llaa/1.5pubs.html

http://crede.berkeley.edu/research/crede/research/llaa/1.5pubs.html


compared to the comparison group on multiple reading achieve-

ment and English language development measures.

Saunders (1999) conducted a longitudinal study of the effects

of a transitional bilingual program on Hispanic English language

learners. The program featured Instructional Conversations

and Literature Logs as two key components integrated with a

reading language arts curriculum, and was provided to English

language learners from grades 2–5. Ninety English language

learners from five program schools and 90 English language

learners from five matched neighboring schools were randomly

selected from the pool of eligible students6 to participate in

the study at the beginning of the second grade. From the 125

English language learners who remained in the study through

the end of fifth grade, 84 were randomly selected and matched

within subgroups7 based on first grade Spanish reading and

language scores to form the analytic sample. The analysis

compared the 42 English language learners in the intervention

group and the 42 English language learners in the comparison

group on multiple reading achievement and English language

development measures. Instruction and assessment for these

fifth grade English language learners were in English and the

focus of this report is for that grade only.

Research (continued)

Effectiveness FindingsThe WWC review of English language learner interventions

addresses student outcomes in three domains: reading

achievement, mathematics achievement, and English language

development.

Reading achievement. Saunders and Goldenberg (1999)

found that Instructional Conversations and Literature Logs had

statistically significant positive effects on both measures of fac-

tual comprehension and interpretive comprehension. The WWC

was only able to confirm the statistical significance of the factual

comprehension measure. Saunders (1999) reported a statisti-

cally significant difference favoring the fifth grade intervention

group on performance assessment. The WWC confirmed the

statistical significance of this finding. In both studies, the inter-

vention had statistically significant positive effects on reading

achievement according to WWC criteria.

English language development. Saunders (1999) reported

a statistically significant difference favoring the fifth grade

intervention group on English language development. The WWC,

however, was not able to confirm the statistical significance of

this finding. In this study, the effect of Instructional Conversa-

tions and Literature Logs on English language development was

substantively important.

Rating of effectivenessThe WWC rates the effects of an intervention in a given outcome

domain as: positive, potentially positive, mixed, no discernible

effects, potentially negative, or negative. The rating of effective-

ness takes into account four factors: the quality of the research

design, the statistical significance of the findings (as calculated

by the WWC8), the size of the difference between participants

in the intervention and the comparison conditions, and the con-

sistency in findings across studies (see the WWC Intervention

Rating Scheme).

6. All students in the analysis sample were English language learners.7. Subgroups were formed based on whether the student qualified and began transitional language arts in fourth or fifth grade. This report does not sepa-

rate findings by subgroup and instead focuses on findings based on the overall sample.8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within class-

rooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch. See the Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate the statistical significance. In the case of Instructional Conversations and Literature Logs, corrections for multiple comparisons and clustering were needed.


The WWC found Instructional Conversations and Literature Logs to have potentially positive effects

for reading achievement

The WWC found Instructional Conversations and Literature

Logs to have potentially positive effects for English

language development

Improvement indexThe WWC computes an improvement index for each individual

finding. In addition, within each outcome domain, the WWC

computes an average improvement index for each study and

an average improvement index across studies (see Technical

Details of WWC-Conducted Computations). The improvement

index represents the difference between the percentile rank

of the average student in the intervention condition versus

the percentile rank of the average student in the comparison

condition. Unlike the rating of effectiveness, the improvement

index is entirely based on the size of the effect, regardless of

the statistical significance of the effect, the study design, or the

analysis. The improvement index can take on values between

–50 and +50, with positive numbers denoting favorable results.

The average improvement index for reading achievement is +29

percentile points across the two studies, with a range of +24 to

+33 percentile points across findings. The improvement index for

English language development is +23 with a range of +21 to +24

percentile points across findings.

SummaryThe WWC reviewed two studies on Instructional Conversations and

Literature Logs. One of these studies was a randomized controlled

trial that met WWC evidence standards with reservations due to

differences between groups in the amount of instruction given.

This study focused on the short-term impacts of the intervention.

The other study was a quasi-experimental design that met WWC

evidence standards with reservations. This study focused on

the long-term impacts where the intervention was administered

within a language arts program. Based on these two studies, the

WWC found the intervention to have potentially positive effects on

reading achievement and potentially positive effects on English

language development. The evidence presented in this report is

limited and may change as new research emerges.

References Met WWC evidence standards with reservationsSaunders, W. M. (1999). Improving literacy achievement for

English learners in transitional bilingual programs. Educational

Research and Evaluation, 5(4), 345–381.

Saunders, W. M., & Goldenberg, C. (1999). Effects of instruc-

tional conversations and literature logs on limited- and

fluent-English-proficient students’ story comprehension and

thematic understanding. Elementary School Journal, 99(4),

277–301.

For more information about specific studies and WWC calculations, please see the WWC Instructional Conversations and Literature Logs Technical Appendices.


Appendix

Appendix A1.1 Study characteristics: Saunders & Goldenberg, 1999 (randomized controlled trial with reservations)

Characteristic Description

Study citation Saunders, W. M., & Goldenberg, C. (1999). Effects of instructional conversations and literature logs on limited- and fluent-English-proficient students’ story comprehension and thematic understanding. Elementary School Journal, 99(4), 279–301.

Participants All 138 English language learners enrolled in three fifth-grade and two fourth-grade classrooms in one school participated in this study. They were matched by language proficiency (limited or fluent) and teachers’ rating of reading skills, and then randomly assigned to four study conditions within each classroom. Twenty-two English language learners were excluded from the final analysis (3 special education students, 4 students enrolled just prior to the study, 12 students not present for some of the activities, and 3 students randomly excluded to provide for a balanced design). The remaining 116 participants were evenly distributed among the four groups (29 per group). Only 32 of these participants in two study groups are of interest for this intervention report: 16 English language learners in the comparison condition and 16 English language learners in the Instructional Conversations and Literature Logs (intervention) condition.1 These students participated in the study with students fluent in English.

Setting The study took place in a K–5 elementary school located in an urban area. The majority of students at the school were Hispanic (82%), had limited English proficiency (69%), and qualified for the free or reduced-price lunch program (62%). More than three quarters of the fourth-grade students were performing below grade level in reading, language, and math. The school ranked among the lowest 20% of schools in the district. Schoolwide efforts were underway at the time of the study to improve bilingual programs, English language development program, language arts instruction, and overall academic infrastructure at the school.

Intervention The study lasted for a period of 10–15 days, including pre- and post-intervention activities. The intervention was implemented over four days. On the first day, English language learners in the Literature Log only group and the Instructional Conversations and Literature Logs group received instruction on Literature Logs in two consecutive 45-minute lessons. On the second day, English language learners in the Instructional Conversation only group and the Instructional Conversations and Literature Logs group received two 45-minute lessons on Instructional Conversations. The same procedures were followed on the third and fourth days, with the order of the lessons reversed (that is, Instructional Conversations lessons on day three and Literature Logs lessons on day four). English language learners in each group also spent at least 45 minutes on creat-ing an illustration and caption summarizing their interpretation of the story “Louella’s Song.”

Comparison English language learners in the comparison group participated in reading and writing activities related to social studies either independently or with a teaching assistant. They also devoted at least 45 minutes to creating an illustration and caption summarizing their interpretation of the story “Louella’s Song.” Saunders and Goldenberg (1999) reported that English language learners in the comparison group did not receive as much direct instructional time from teachers as those in the Instructional Conversations and Literature Logs group.2

Primary outcomes and measurement

The effects of the intervention were assessed using several measures, including factual comprehension, interpretive comprehension, theme-explanation essay, and theme-exemplification essay. Although theme-explanation and theme-exemplification essays were measures used in the study, outcomes were not reported in the WWC report because the data reported in the study were the percentage of English language learners whose essays received a high score. Therefore, the measures did not meet WWC standards (see Appendix A2.1 for more detailed descriptions of outcome measures).

Teacher training The five teacher participants in the study were members of the research and development team implementing the school’s language arts model in Spanish, transition, and mainstream English language arts classrooms. The team was led by two instructional advisors who were able to co-teach and provide assistance in the classrooms on a daily basis. The team met on a bi-monthly basis to study instructional components, view videotapes and live demonstrations, plan instructional units, and evaluate English language learners’ work. All study conditions (including the control) were carried out in each classroom by each of the five teachers.

1. The other two study groups (Instructional Conversations only and Literature Logs only) in Saunders and Goldenberg (1999) were not of interest to this intervention report.2. The different amounts of teacher-led instruction in the intervention and comparison conditions might have confounded the effects of the intervention; therefore this study was downgraded to

meeting WWC evidence standards with reservations.


Appendix A1.2 Study characteristics: Saunders, 1999 (quasi-experimental design)

Characteristic Description

Study citation Saunders, W. M. (1999). Improving literacy achievement for English learners in transitional bilingual programs. Educational Research and Evaluation, 5(4), 345–381.

Participants Eligible participants for the study were students who enrolled in their respective schools from first grade through fifth grade, considered Spanish their primary home language, had limited English proficiency at the time of enrollment, and were in a Spanish transitional bilingual program. Among the pool of eligible English language learners, 180 were randomly selected at the beginning of second grade from five intervention schools and five neighboring schools, then matched on academic achievement, percentage of students with limited English proficiency, percentage of students in the free or reduced-priced lunch program, school enrollment, and ethnic composition. Of the 180 English language learners, 125 remained at their respective school through fifth grade, from which 42 intervention-group English language learners and 42 comparison-group English language learners matched on first grade Spanish reading and language and transition status were selected for data analysis.

Setting The study took place in a school district in southern California. The average enrollment for the 10 participating schools was 880. Over 90% of the students enrolled in these schools were Latino or Hispanic, 74% were limited English proficient, and 95% qualified for free or reduced-price lunches.

Intervention English language learners in the intervention group participated in a transitional bilingual program, which covered several transitional phases as they progressed from receiving instruction in their primary language to instruction in English. English language learners were in the Pre-Transition phase in the second and third grades, in Transition I in the fourth grade, and in Transition II in the fifth grade. Transition II is the only phase during which instruction was dedicated completely to reading and writing in English, and English language learners were assessed with English language measures. Transition II services were provided only in grade 4 or 5, depending on whether a student was ready for English-only instruction. The earlier transition phases involved academic support and assessments in both English and Spanish. This transition program incorporated 12 instructional components, including Instructional Conversions and Literature Logs. Instructional Conversations involved ongoing discussions, while Literature Logs (and writing projects) entailed writing activities. Teachers assisted English language learners in understanding how the experiences of a character in a story relate to their own experiences. Writing assignments were intended to encourage English language learners to think about and express ideas, as well as extract meaning from the stories. The Instructional Conversations were intended to provide a forum for sharing ideas and a more complex understanding of story themes. English language learners studied literature during each phase, practiced using reading comprehension strategies, and read assigned books independently. The assignments were based on each English language learner’s reading level. Additionally, for 45 minutes per day during the Pre-Transition phase, teachers provided instruction using the English Language Development through Literature program. Instruction was provided in small groups comprised of English language learners with similar levels of English language proficiency. Three or more times per week, teachers read aloud for approximately 20 minutes. English language learners also chose a book or story to read independently each day and completed short assignments for the ones that they found most engaging. Additionally, they wrote in journals during the beginning stages of English writing and received written responses from the teacher. Instructional Conversations and Literature Logs were two of the 12 components of the language arts program that was implemented.

Comparison English language learners in the comparison group met the same criteria as those in the intervention group. English language learners in the comparison group participated in the district’s typical bilingual education program. In the primary grades, for approximately 20–30 minutes each day, English language learners received instruction in English language development. Instruction for the rest of the day was in Spanish. When they demonstrated proficiency in Spanish reading and writing, and basic proficiency in relation to oral English skills (the end of second grade through the beginning of third grade), they continued Spanish language arts, and began receiving reading and writing instruction in English (transition stage). After approximately 3–6 months, English language learners began a mainstream English program. The comparison group did not receive the transition program that incorporated 12 instructional components, including Instructional Conversations and Literature Logs, which were taught to English language learners in the intervention group.

Primary outcomes and measurement

English language learners’ reading and English language development were assessed using the Comprehensive Test of Basic Skills (CTBS) and the Criteria for Addition of Reading in English (CARE). Reading outcomes were also assessed using performance assessments. Because most English language learners in grades 1–4 took the CTBS and performance assessment in Spanish, findings from these grades are not included in this report. Only findings for fifth-graders, who took the tests in English, are included in this report (see Appendix A2.1 and A2.2 for more detailed descriptions of outcome measures).

Teacher training Saunders (1999) noted that the teachers involved in the study worked with project staff for the full five years during which it was implemented. No further information about teacher training was reported.


Appendix A2.1 Outcome measures in the reading achievement domain

Outcome measure Description

Comprehensive Test of Basic Skills (CTBS), Reading Subtest

The CTBS is a standardized achievement test used to assess English language proficiency (as cited in Saunders, 1999). It includes subtests of reading and language. Perfor-mance on the CTBS was reported in terms of Norm Curve Equivalent (NCE) scores.

Performance assessments: Reading achievement

The author developed this assessment, which consists of six tasks: narrative comprehension (measures story comprehension), informational comprehension (measures how well English language learners understand informational text), writing communication (measures the lucidity of the information that English language learners provide to peers about story content), writing conventions (measures spelling, punctuation, capitalization, and usage), dictation (measures spelling, punctuation, and capitalization), and independent reading and library use, in which English language learners indicate the books and stories that they have chosen to read, and items borrowed from the library (as cited in Saunders, 1999). The assessments were administered in a small-group setting and scored by trained project staff and teachers under the direction of the author. The inter-rater reliability was 61% based on exact agreement and 97% based on adjacent agreement (agreement within one point with a five-point scale).

Factual comprehension This measures required English language learners to recall factual details about a story. The measure was made up of 10 items (as cited in Saunders & Goldenberg, 1999).

Interpretive comprehension This measure included five questions related to the interpretation of events that occurred in a story (as cited in Saunders & Goldenberg, 1999).

Appendix A2.2 Outcome measures in the English language development domain

Outcome measure Description

Comprehensive Test of Basic Skills (CTBS), Language Subtest

The CTBS is a standardized achievement test used to assess English language proficiency (as cited in Saunders, 1999). It includes subtests of reading and language. Perfor-mance on the CTBS was reported in terms of Norm Curve Equivalent (NCE) scores.

Performance assessments: Writing achievement

The author developed this assessment, which consists of six tasks: narrative comprehension (measures story comprehension), informational comprehension (measures how well English language learners understand informational text), writing communication (measures the lucidity of the information that English language learners provide to peers about story content), writing conventions (measures spelling, punctuation, capitalization, and usage), dictation (measures spelling, punctuation, and capitalization), and independent reading and library use, in which English language learners indicate the books and stories that they have chosen to read, and items borrowed from the library (as cited in Saunders, 1999). The assessments were administered in a small-group setting and scored by trained project staff and teachers under the direction of the author. The inter-rater reliability was 61% based on exact agreement and 97% based on adjacent agreement (agreement within one point with a five-point scale).


Appendix A3.1 Summary of study findings included in the rating for the reading achievement domain1

Authors’ findings from the study

WWC calculationsMean outcome

(standard deviation2)

Outcome measureStudy

sample

Sample size (students/ schools)

IC & LL group

Comparison group

Mean difference3

(IC & LL – comparison) Effect size4

Statistical significance5

(at α = 0.05)Improvement

index6

Saunders & Goldenberg, 1999 (randomized controlled trial)7

Factual comprehension Grades 4–5 32/1 14.88 (3.03)

12.12 (3.50)

2.76 0.82 Statistically significant

+29

Interpretive comprehension Grades 4–5 32/1 5.56 (2.73)

3.62 (1.96)


+29

Average8 for reading achievement (Saunders & Goldenberg, 1999) 0.81 Statistically significant

+29

Saunders, 19999 (quasi-experimental design)10

Performance assessment Grade 5 84/10 2.79 (0.65)

2.25 (0.47)


+33


Grade 5 84/10 41.90 (10.64)

34.51 (12.40)

7.39 0.63 ns +24

Average8 for reading achievement (Saunders, 1999) 0.79 Statistically significant

+28

Domain average8 for reading achievement across all studies 0.80 Statistically significant

+29

ns = not statistically significant

1. This appendix reports findings considered for the effectiveness rating and the average improvement index. Subgroup findings from the same studies are not included in these ratings, but are reported in Appendix A4.1. 2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can

take on values between –50 and +50, with positive numbers denoting favorable results.7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the cluster-

ing correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Saunders & Goldenberg (1999), no correction for clustering was needed. The WWC did, however, correct for multiple comparisons.

8. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. 9. Note that Instructional Conversations and Literature Logs were 2 of the 12 components of the language arts program that was implemented as part of the intervention. Therefore, findings were based on all 12 components, including

Instructional Conversations and Literature Logs.10. In the case of Saunders (1999), corrections for clustering and multiple comparisons were needed, which changed the statistical significance of outcomes on the CTBS.


Appendix A3.2 Summary of study findings included in the rating for the English language development domain1

Author’s findings from the study




sample


IC & LL group

Comparison group

Mean difference3




index6



Grade 5 84/10 45.38 (13.94)

37.80 (12.87)

7.58 0.56 ns +21

Writing Grade 5 84/10 2.87 (0.81)

2.37 (0.69)

0.50 0.66 ns +24

Domain average9 for English language development (Saunders, 1999) 0.61 ns +23


1. This appendix reports findings considered for the effectiveness rating and the average improvement indices. Subgroup findings from the same studies are not included in these ratings, but are reported in Appendix A4.2.2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group.4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can

take on values between –50 and +50, with positive numbers denoting favorable results.7. Note that Instructional Conversations and Literature Logs were 2 of the 12 components of the language arts program that was implemented as part of the intervention. Therefore, findings were based on all 12 components, including

Instructional Conversations and Literature Logs.8. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the

clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Saunders (1999), a correction for clustering was needed, which changed the statistical significance of outcomes on the CTBS.

9. This row provides the study average, which in this instance is also the domain average. The WWC-computed domain average effect size is a simple average rounded to two decimal places. The domain improvement index is calculated from the average effect size.


Appendix A4.1 Summary of subgroup findings for the reading achievement domain1

Author’s findings from the study




sample


IC & LL group

Comparison group

Mean difference3




index6



Grade 5 (Transition 4)8

56/10 40.56 (10.18)

33.62 (13.85)

6.94 0.56 ns +21



28/10 44.56 (11.41)

36.29 (9.04)

8.27 0.78 ns +28


1. This appendix presents subgroup findings for measures that fall in the reading achievement domain. Total group scores were used for rating purposes and are presented in Appendix A3.1.2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values between

–50 and +50, with positive numbers denoting favorable results.7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings

not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Saunders (1999), a correction for clustering was needed, which changed the statistical significance of outcomes on the CTBS.

8. These students began transitional English language arts (i.e., Transition II services) during the first semester of grade 4.9. These students began transitional English language arts (i.e., Transition II services) during the first semester of grade 5.


Appendix A4.2 Summary of subgroup findings for the English language development domain1

Authors’ findings from the study




sample


IC & LL group

Comparison group

Mean difference3




index6

Saunders & Goldenberg, 1999 (randomized controlled trial)7

Factual comprehension Grades 4–5 (Literature Logs only versus comparison)

32/1 10.38 (3.72)

12.12 (3.50)

–1.74 0.47 ns –18

Interpretive comprehension Grades 4–5 (Literature Logs only versus comparison)

32/1 3.56 (1.86)

3.62 (1.96)

–0.06 0.03 ns –1

Factual comprehension Grades 4-5 (Instructional

Conversations only versus comparison)

32/1 13.19 (4.10)

12.12 (3.50)

1.07 0.27 ns +11

Interpretive comprehension Grades 4-5 (Instructional

Conversations only versus comparison)

32/1 4.88 (2.47)

3.62 (1.96)

1.26 0.55 ns +21




56/10 47.33 (15.63)

40.11 (12.81)

7.22 0.50 ns +19



28/10 41.49 (8.99)

33.18 (12.13)

8.31 0.76 ns +28

Writing Grade 5 (Transition 4)9

56/10 3.15 (0.78)

2.67 (0.60)

0.48 0.68 ns +25

Writing Grade 5 (Transition 5)10

28/10 2.29 (0.54)

1.76 (0.40)


+36


1. This appendix presents subgroup findings for measures that fall in the English language development domain. Total group scores were used for rating purposes and are presented in Appendix A3.2.2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are; a smaller standard deviation on a given measure would indicate that participants had more similar outcomes.3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations.5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups.


Appendix A4.2 Summary of subgroup findings for the English language development domain (continued)

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the percentile rank of the average student in the comparison condition. The improvement index can take on values between –50 and +50, with positive numbers denoting favorable results.

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch. See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Saunders & Goldenberg (1999), no correction for clustering was needed.

8. In the case of Saunders (1999), a correction for clustering was needed, but since the results were non-significant prior to the correction for clustering, there would have been no change in statistical significance after the correction was made.

9. These students began transitional English language arts (i.e., Transition II services) during the first semester of grade 4.10. These students began transitional English language arts (i.e., Transition II services) during the first semester of grade 5.


Appendix A5.1 Instructional Conversations and Literature Logs rating for the reading achievement domain

The WWC rates the effects of an intervention in a given outcome domain as: positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1

For the outcome domain of reading achievement, the WWC rated Instructional Conversations and Literature Logs as having potentially positive effects. It did not

meet the criteria for positive effects because there was no study that met the WWC evidence standards for a strong design. The remaining ratings (mixed effects, no

discernible effects, potentially negative effects, and negative effects) were not considered, as the intervention was assigned the highest applicable rating.

Rating received

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect.

Met. In both studies on Instructional Conversations and Literature Logs, the intervention showed statistically significant positive effects.

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate

effects than showing statistically significant or substantively important positive effects.

Met. No studies showed a negative effect.

Other ratings considered

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

Not met. The Instructional Conversations and Literature Logs intervention has two studies that examined reading achievement, both of which met

WWC evidence standards with reservations and therefore did not qualify for a strong design.

• Criterion 2: No studies showing statistically significant or substantively important negative effects.


1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.


Appendix A5.2 Instructional Conversations and Literature Logs rating for the English language development domain

The WWC rates the effects of an intervention in a given outcome domain as: positive, potentially positive, mixed, no discernible effects, potentially negative, or negative.1

For the outcome domain of English language development, the WWC rated Instructional Conversations and Literature Logs as having potentially positive effects.

It did not meet the criteria for positive effects because there was no study that met the WWC evidence standards for a strong design. The remaining ratings (mixed

effects, no discernible effects, potentially negative effects, and negative effects) were not considered, as the intervention was assigned the highest applicable rating.

Rating received

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence.

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect.

Met. The one study that examined the intervention’s effect on English language development showed substantively important positive effects.

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate

effects than showing statistically significant or substantively important positive effects.


Other ratings considered

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence.

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design.

Not met. Instructional Conversations and Literature Logs had only one study that examined English Language Development, which met WWC

evidence standards with reservations and thus did not qualify for a strong design.

• Criterion 2: No studies showing statistically significant or substantively important negative effects.


1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description.

(continued)

Documents

Instructional Conversations and Literature Logs · PDF fileWWC Intervention Report Instructional Conversations and Literature Logs ... language arts program. ... broadening English