WCPSS US HistoryA JOINT PROJECT

EVALUATION AND RESEARCHGlenda Haynie

CURRICULUM AND INSTRUCTIONMelinda Stephani

PROJECT GOALS• Study, Improve, and Implement a WCPSS

Value-Added Instructional Improvement Analysis Model• Collect WCPSS specific data that will help teachers,

school, and district leadership understand their current practices in US History.

• Identify and share best practices in US History.

• Contribute to a series of studies that identify targets for overall systemic improvement• Identify the role of teachers, academic departments,

principals, schools, and central services’ administrators in the school improvement process.

• Identify the practices of effective improvement.

Starting Points for the WCPSS US History Study

• A High School Graduation Course Requirement

• Entering HS class of 06-07 must pass EOC Exam

• 90 teachers at 19 High Schools in 2005-06. • New curriculum in 2003-04, first tested in 2005-

06

• 55% of classes are academic, 33% honors, and 11% AP. Two schools have honors and academic mixed. Two schools have a Paideia class.

DISTRICT-WIDE HIGH SCHOOL

STUDENT SUCCESS

24340

1838

388150

4755

0

400

800

1200

1600

2000

2400

2800

3200

3600

4000

4400

4800

7661 Students Took U S History in 2005-06

69.5% of all students scored at level III or IVPerformance by Subgroups

83.3 83.3

46.6

65.7

79.484.6

75.077.871.3

60.8

48.4

72.7

41.4

77.1

0

10

20

30

40

50

60

70

80

90

100

perc

en

tag

e a

t L

evel

III

or

IV

Male

Female

Gaps in Performance

0

10

20

30

40

50

60

70

80

90

100

Gender Ethnicity

per

cen

tag

e at

Lev

el I

II o

r IV

Male

Fem

ale

Wh

ite

Bla

ck

9.2

%

37.1

%

0

20

40

60

80

100

120

140

160

180

200

220

240

260

280

300

320

340

360

scale scores

nu

mb

er

of

stu

den

ts

Level Performance 2005-06

Level I Level II

Level III

Level IV

MEASURES OF

SCHOOL SUCCESS

% AT LEVEL III OR IV 2005-06% at Level III 0r IV

HIGHEST WCPSS SCHOOL 86.6%

79.0%

79.0%

78.4%

78.1%

73.1%

72.9%

71.3%

70.4%

WCPSS DISTRICT 69.5%

69.1%

68.9%

66.9%

66.8%

59.0%

57.4%

STATE 56.8%

56.6%

55.4%

LOWEST SCHOOL 50.6%

AVERAGE SCALE SCORES

HIGHEST WCPSS SCHOOL 86.6% HIGHEST 159.4 HIGHEST

79.0% 156.0

79.0% 157.2

78.4% 156.5

78.1% 156.8

73.1% 154.1

72.9% 155.5

71.3% 154.4

70.4% 154.7

WCPSS DISTRICT 69.5% 154.3

69.1% 153.9

68.9% 153.5

66.9% 153.2

66.8% 153.4

59.0% 150.0

57.4% 148.2 LOWEST

STATE 56.8% 151.4

56.6% 152.0

55.4% 151.0

LOWEST SCHOOL 50.6% LOWEST 150.1

WCPSS EFFECTIVENESS

INDICES

What are Student Residuals?

• A student residual is the difference between a student's EOC scale score and the expected scale score of 'like' WCPSS students. It is a result of a regression analysis that uses previous pretest scores and controls for factors such as special program services, and free or reduced lunch status.

• > + 1 standard deviation (+5.89 in 2005-06 U. S. History) is significantly higher than other ‘like’ WCPSS students.

• < - 1 standard deviation (-5.89 in 2005-06 U. S. History) is significantly lower than other ‘like’ WCPSS students .

• NOTE: Standard deviation is at bottom of student residual roster and varies by subject and year.

RESIDUALS SHOW VALUE ADDED BY TEACHERS and SCHOOLS

Algebra I Regression Scatter Plot

30

40

50

60

70

80

90

100

140 150 160 170 180 190 200 210

8th Grade EOG Score (Converted To Edition One Scale)

20

04

-05

Alg

eb

ra I

Sc

ale

Sc

ore

Teacher A

Teacher B

What are School EOC Effectiveness Indices?

• A School EOC Effectiveness Index is a z-score calculated by averaging all of the student residual scores for a particular test (e.g. U.S.History EOC) at a particular school and standardizing on the z scale. It can be used to compare a school’s performance with other school’s in the district.

• > + 1 is significantly higher than other WCPSS schools

• < - 1 is significantly lower than other WCPSS schools

WCPSS School US History Effectiveness

School 01-02 02-03 05-06

1 M- M- M+

2 M- M- M-

3 M- M- M+

4 M+ M+ H

5 H M+ L

6 H M+ M+

7 M+ H M-

8 M+ H M+

9 L M- L

10 M+ M- M+

11 M+ M+ H

12 M- H M+

13 M- M- M-

14 M+ M- M+

15 M+ M- L

Subjects in Study

• Identified 29 2006-07 teachers of US History (32% of the 90 from 2005-06) that taught US History in 2001-02, 2002-03, 2005-06 and were teaching in 2006-07.

• Calculated the average student residual by teacher across all classes for 2005-06, 2001-03 (2 years), and a combined 2001-03/2005-06.

• Ranked teacher effectiveness from highest to lowest for 2005-06 and compared to previous and combined averages

Top Effective Teachers in 7 Schools

• Top 10 teachers in 9 schools• Bottom 10 teachers in 7 schools• Three schools with top and bottom

teachers. Bottom 4 teachers have a school colleague among the top teachers.

• Residual averages ranged from -2.53 to 4.33

• B1- B4 taught at the same schools as T1,T2, and T5 .

Residual Means and Standard DeviationsTeacher School Res_mean Res_stdev

T1 S1 4.33 5.03T2 S15 3.65 5.29T3 S11 3.65 6.27T4 S6 3.20 5.26T5 S10 2.88 4.40T6 S4 2.78 5.31T7 S3 2.69 6.13T8 S9 2.67 5.62T9 S10 2.36 5.45

T10 S2 2.32 5.45B10 S7 0.50 5.53B9 S5 0.46 4.58B8 S8 0.29 5.66B7 S14 0.15 5.48B6 S5 -0.37 6.79B5 S8 -0.68 5.12B4 S1 -1.08 5.86B3 S10 -1.22 5.60B2 S15 -2.22 5.16B1 S15 -2.53 6.00

EOC PERFORMANCEPercent at Level III or IV

White Student Proficiency

Top teachers ranged from 77% to 100% with 9 teachers above 90%

Bottom teachers ranged from 25% to 99% with 1 above 90%

Black Student ProficiencyTop teachers ranged from 33 % to 100% with 5 teachers above 90%

Bottom teachers ranged from 30% to 86% with 0 above 90%

Percentage Proficiency 2005-06Teacher School % prof white n %prof black n

T1 S1 94 66 100 9T2 S15 100 79 100 4T3 S11 77 48 75 12T4 S6 94 32 100 12T5 S10 94 32 100 12T6 S4 96 118 100 7T7 S3 93 88 50 2T8 S9 97 29 33 9T9 S10 93 92 64 11T10 S2 98 50 na noneB10 S7 83 54 71 24B9 S5 99 86 86 7B8 S8 73 40 53 40B7 S14 25 12 30 20B6 S5 72 78 56 25B5 S8 86 15 50 6B4 S1 87 47 41 22B3 S10 75 36 56 16B2 S15 65 79 42 31B1 S15 83 18 80 5

Student Assignments in courses that take the U. S. History EOC exam

• Top Teachers• 3 AP only, 1 honors only, 1 honors and AP, 3 honors and academic, and 2 all three levels

• Bottom Teachers• 1 honors and AP, 3 honors and academic, 3

academic, and 3 all three levels

There were Top teachers who exhibited success with each level of student.

Yet there were more AP classes among Top teachers and more academic classes among Bottom teachers.

Top Teachers Teach only American History

Teacher% of classes that were American History Related in 2006-07

T1 100%T2 100%T3 100%T4 100%T5 100%T6 100%T7 100%T8 100%T9 100%T10 100%

B10 100%B9 100%B8 100%B7 67%B6 100%B5 83%B4 67%B3 67%B2 100%B1 17%

Level Performance

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

AP TOP AP Bottom Honors Top HonorsBottom

AcademicTop

AcademicBottom

ALLstudents

IV top

IV bottom

III

II

I

Comparing Top with BottomStudying Teacher Effectiveness

• Teacher Survey• Classroom Observations• School Focus Groups• Teacher Focus Groups

RESULTS

Pacing of Curriculum Consistent

• All teachers of the study (Top and Bottom) were teaching the North Carolina standard course of study using pacing consistent with WCPSS Pacing Guide.

How do teachers get to the top?

• Top teachers had a more complete package of rigor, relevance, and relationship strategies than bottom teachers.

• Bottom teachers were lacking in one or more key areas found in ALL Top teachers.

Four Areas of Comparison

• A. Intellectual Demand

• B. Intellectual Enrichment

• C. Time Management

• D. Student-Centered versus Teacher- Centered Classrooms

A. Intellectual Demand

• Top teachers use: • higher-ordered thinking skills in

questioning and in discussion• directed questioning. All students are

called upon to participate and are kept engaged throughout the class period

• Top teachers had high academic expectations of students.

Expectations of students

• Top teachers expect that students can learn to read, take and organize notes, make connections, analyze, and respond to higher-order thinking skill questions. The acquisition of facts happens within a sense-making context.

Examples: teaching how to read a history text, quizzing on pre-readings, requiring a set organization of notes, posing questions on connections and themes, using political cartoons and short video clips.

Examples of Higher-Order Questions

• What is the difference between a colony and a protectorate?

• Why is President Roosevelt drawn so large in this cartoon?

• What role did heroes play in the 1920s?

Examples of Lower-Order Questioning

• Who was the first Chief Justice of the Supreme Court?

• When did the Civil War occur?• Who was President during the Vietnam

War?

Summarizing

• Top teachers provide a clear note taking structure for students.

• Top teachers lead students in seeing similarities and differences.

• Top teachers hold students responsible for all history studied and use cumulative questions regularly on quizzes and tests.

B. Intellectual Enrichment

• Top teachers:• have basic content knowledge and show evidence

of continued learning as a professional.• infuse supplementary content information into

the classroom• develop many of their own materials rather than

relying solely on publisher-supplied materials.

• More top than bottom teachers have efficient focused planning times with colleagues.

Teachers build content knowledge

• ALL TOP Teachers use outside class time reading and studying their content knowledge. They develop background knowledge through outside pleasure reading.

A representative quote was, “I teach history because I love to study and read about history. I am a history nerd”

Content Planning

• Top teachers use their background knowledge to prepare student guides, graphic organizers, warm-up questions, etc. They do not rely solely on textbooks, handouts, worksheets that are prepared by commercial vendors. They plan with peers (in their schools or other schools)

A representative quote was, “The textbook is not the curriculum. No boring worksheets used in my class.”

C. Time Management

• Top teachers:• Teach bell to bell.

• Use an invigorated delivery rate.

• Use an effective Lecture/Discussion method.

D. Student-Centered Classrooms

• Top teachers:• connect the content to current events and

student’s lives.

• teach students to be historians.

• build positive relationships with kids.

Connections

• Top teachers connect history to current events.• Top teachers connect history to their lives and their

students, and help students imagine themselves in historical settings.

• Top teachers connect history in themes across time into the present.

Examples: student reports on newspaper articles, connecting Seward’s folly to Ms.----’s folly in class yesterday, asking the question, “What would happen if you ------?, explaining “safety valve” in connection to the teacher’s prom experiences in high school, connecting John Brown to Pancho Villa to bin Laden (all individual fugitives that were/are hard to catch)

Student Affirmation

• Top Teachers consistently give positive feedback on the quality of student work, effort, and ability. Top teachers believe that all students can succeed at a high level.

Representative quotes: ”You are the greatest kids. I am so proud of what my students can do.”, “My job is to connect the dots. You have so much knowledge.” “I know that you know this.” “You are so smart. The jeopardy questions are too easy for you.”

A student said, “We did so good on this.”

Class Atmosphere

• Top teachers create an atmosphere of mutual respect where inquiry, wrong answers, personal challenge, collaboration, and disequilibrium provide opportunities for learning for all students.

• Top teachers are enthusiastic. Class is FUN! Students are actively engaged. There is no sleeping or off-task behaviors.

SUMMARY OF CLASSROOM OBSERVATIONS TOP VERSUS BOTTOM TEACHERS

IN FOUR COMPARISON AREASComparison Areas Key indicators Top teachers Bottom Teachers

A. Intellectual Rigor/Demand All students expected to participate as they learn to read, take and organize notes, make connections, analyze, and respond to higher-order thinking skill questions. The acquisition of facts happens within a sense-making context. Directed questioning with all students.

10 out of 10 0 out of 10

B. Intellectual Content and Enrichment

Teachers: have basic content knowledge and show evidence of

continued learning as a professional. infuse supplementary content information into the

classroom develop many of their own materials rather than relying

solely on publisher-supplied materials. have efficient focused planning times with colleagues.

10 out of 10 4 out of 10

C. Time Management Teachers teach bell to bell, use an invigorated delivery rate, and use an effective Lecture/Discussion method.

9 out of 10 2 out of 10

D. Teachers Student-Centered not Self-Centered

Teachers connect the content to current events and student’s lives, teach students to be historians, and build positive relationships with kids.

10 out of 10 5 out of 10

Conclusions about Top Teachers

• TOP TEACHERS:• Expected all students to participate in class,• Taught all students to read, take and organize notes, make

connections and analyze historical texts,• Placed the acquisition of facts within a sense-making

context,• Used directed higher-order thinking skill questions with all

students,• Have basic content knowledge and show evidence of

continued learning as a professional,• Infuse supplementary content information into the

classroom,• Develop many of their own materials rather than relying

solely on publisher-supplied materials,• Have efficient focused planning times with colleagues,• Teach bell to bell,• Use an invigorated delivery rate,• Use an effective Lecture/Discussion method,• Connect the content to current events and student’s lives,• Teach students to be historians, and• Build positive relationships with kids.

NEXT STEPS

SCHOOL LEADERSHIP

• Develop a school plan that aligns to the standard course of study, emphasizing rigor and relevance.

• Support and expect meaningful common planning for teachers.

• Share results data with teachers including effectiveness rosters and indices.

• Develop a scheduling plan that encourages a concentration in American history and places first-year teachers in non-EOC courses.

• Support a school culture that promotes open discourse at all levels.

TEACHERS

• Study standard course of study and EOC testing program.

• Study goal summaries. • Study residual rosters .• Plan with other teachers.• Understand and implement Marzano

strategies.• Use and/or develop a school plan.• Study American history to deepen depth of

content knowledge and enrich classroom instruction.

• Reflect on performance data and instructional practice.

• Have high, rigorous expectations of all students.

DISTRICT LEADERSHIP

• Make observations of most effective schools and teachers that can be shared district-wide.

• Provide workshops on implementing Marzano strategies.

• Support school wide improvement efforts based on top school models.

• Support teacher improvement efforts.• Provide data to teachers and schools on

their effectiveness.