Figure 1. From professional development to student achievementWe predict that student achievement will improve by providing teachers with improved mathematics content and pedagogy. These treatments are applied to teachers in grades 4-8.

Modeling student achievement in K-12 math and science classrooms:  An experimental approachM.J. Bryant, R.A. Cardullo, K. Bocian, K.A. Hammond. University of California, Riverside, CA   92521

Figure 2. Comparison of possible experimental designs

The treatment () can be applied to either entire schools or to individual classrooms within a school. Schools or classrooms that do not receive the treatment serve as controls (). Ignoring issues of human subjects, the treatment would be applied to half of the schools or classrooms. The bottom two panels illustrate and example of increasing the proportion of treatment to control groups which is desirable in an educational experiment.

Year 1 Year 2 Year 3 Year 4 Year 1 Year 2 Year 3 Year 4

Treatment Applied to Schools Treatment Applied to Classrooms

1 2 3 4 5 6 7 8 910 11 12 13 14 15 16

Tre

atm

ent:

Con

trol

Con

stan

t(S

choo

l)T

reat

men

t:C

ontr

ol I

ncre

ases

(Sch

ool)

1 2 3 4 5 6 7 8 910

11 12 13 14 15 16

0.00

0.05

0.10

0.15

0.20

0.0 0.2 0.4 0.6 0.8 1.0 0.2 0.4 0.6 0.8 1.0

Alp

ha (

< 0

.05

cons

ider

ed s

igni

fica

nt)

Probability of P < Alpha

Treatment Applied to:SchoolsClassrooms

Classroom design has the most statistical power

School design has the most statistical power

The experimental design with the highest statistical power is dependent on where the variation is greatest.

Variation greatest among schools Variation greatest among classrooms

0.0

Figure 3. Monte Carlo based power analyses inform which design has the highest statistical power. The choice between schools or classrooms as the experimental unit is subject to many logistical considerations. The design impacts statistical power (the probability that a significant difference is found when a difference exists). Computer simulations reveal that statistical power is optimized as a function of where random variation lies. For example if all classrooms in a school are equal but variation exists among schools, randomly assigning treatment to classrooms has greater statistical power.

Figure 4. Assignment of treatment and control groups as a stratified random design.In educational settings, demographic factors (i.e. socio-economic status or SES) are known to have a great impact on student performance, but random assignment of treatment is essential. Random treatment assignment may lead to unintended pooling on either end of an SES continuum. By selecting schools at random from matched pairs, we can control for differences in SES. Here schools were matched by principal components analysis. Blue arrows connect matched pairs ( = Treatment, = Control). Perfect pairing may not be possible for all schools (▲ = schools with SES characteristics outside the range of the other schools).

-3

-2

-1

0

1

2

3

-5 -4 -3 -2 -1 0 1 2 3 4 5

Schools matched by demographic variablesfor stratified random assignment

Index Based on Aggregated Student Level Characteristics (i.e. % reduced or free lunch)

Inde

x B

ased

on

Cha

ract

eris

tics

of t

he S

choo

l (i

.e. C

lass

Siz

e)

Treatment

4th GradeTeachers

5th GradeTeachers

6th GradeTeachers

7th GradeTeachers

4th GradeStudents

5th GradeStudents

6th GradeStudents

7th GradeStudents

Time (2002-2006)

A core team consisting of mathematicians, scientists, and educators develops a treatment

The treatment is administered to teachers in the treatment group

Teachers teach with learned techniques

Efficacy of the treatment is measured through student test scores

This several year long study measures the growth in both teachers and students over time

INTRODUCTION AND STATEMENT OF PROBLEM

Mathematical Achievement through Collaboration with Teachers and Students (Mathematical ACTS) is a multi-faceted NSF-funded research project designed around specifically tailored initiatives for teachers and students in the 4th – 8th grades. ACTS seeks to raise achievement and close gaps in mathematics for all students in the Jurupa Unified School District through a series of programs (150 hours annually) for teachers and students. For the purpose of this project, student achievement in mathematics is defined as preparation for and success in Algebra I, the gatekeeper class for higher mathematics courses and entrance colleges and universities. We targeted students in the Jurupa Unified School District 4th through 9th grades, beginning in September 2003 through June 2007. Low mathematics performance in K-12 impedes progress in college mathematics and science and its toll is being felt at many institutions of higher education. At the University of California, Riverside the percentage of students who take remedial mathematics (defined as pre-calculus or below) has risen from 65% in Fall, 2000 to 73% in Fall, 2003 (UCR Learning Center Fall, 2003 report). In addition, the majority of under-represented high school students in the 66th State Assembly District are not prepared for college. Less than 15% of students from this region attend any university (California Postsecondary Education Commission Report, 2003) and the percentage of students that take any mathematics beyond Algebra I is exceedingly low. Left unchecked, any cohort of students that is not well prepared in both mathematics and science will not be prepared to function in a technologically advanced society.

The effect of professional development on teaching and the subsequent achievement of students are studied in a research design that includes:•Comparison of teacher baseline and post ACTS intervention knowledge and use of mathematics and pedagogy. •Assessment of the implementation of ACTS teaching strategies in the classroom.•Assessment of the value added by the treatment in the classroom by tracking the rate of student achievement growth in targeted and control (wait-listed) classrooms.•Case studies of teacher and school implementation within one school site.•Assessment of benchmarks of systemic change for both UCR and JUSD. The initial study group includes teachers and students from 16 elementary and 3 middle schools. In dealing with a small group a number of questions arise concerning the research design, the quality of assessments used for evaluating both content and pedagogy, and the types of analysis used to maximize statistical power in quantifying the effectiveness of treatments. Since scientifically designed and controlled experiments are rare in education, Mathematical ACTS provides an opportunity to quantify the effectiveness of various teacher interventions.

Question: How can student achievement in mathematics be improved in public schools so that a larger fraction of students is eligible for Algebra I by eighth grade?

 Hypothesis: Since students receive mathematics training from teachers, improving student achievement in elementary and middle school math classes can be attained by improving the math proficiency and pedagogy of teachers.

 Prediction: As a result of ACTS professional development, student achievement, as measured by standardized tests, will significantly increase resulting in a measurable increase in Algebra I enrollment.

Figure 5. Mathematics content knowledge and pedagogy are related but distinct.Primary and secondary school teachers tend to have higher pedagogical skills (as measured by a University of Michigan designed pedagogy assessment) when their mathematics content proficiency is higher (CSU/UC Algebra Readiness Test). Secondary school teachers have significantly higher mathematics pedagogical skills, perhaps due to the fact they are single subject teachers. Despite the correlation, there remains considerable variation in pedagogical skill for all levels of content knowledge. The secondary school teachers’ baseline data serves as benchmark goal for the primary schools teachers.

0

0.2

0.4

0.6

0.8

1.0

0 0.2 0.4 0.6 0.8 1

Relationship Between Mathematics Proficiency and Pedagogy

Score on Mathematics Proficiency Test

Sco

re o

n M

athe

mat

ics

Ped

agog

y T

est

Elementary School TeachersSecondary School Teachers

Effect of Teacher’s Grade level: P = 0.0002

Percentage of Elementary School Teachers Correctly Answering Question

Per

cent

age

of S

econ

dary

Sch

ool T

each

ers

Cor

rect

ly A

nsw

erin

g Q

uest

ion

1 to 1

5050

60

60

70

70 80

80

90

100

90 100

Question 39

Question 1

1) 3.01 + 0.301+ 30 = ?(A) 0.631(B) 3.611(C) 33.311(D) 33.32

39) In the right triangle ABC shown to the right, what is the length of AC ?

(A) 10 (B) 14 (C) 24 (D) 48

B

A

C

6

8

Sample Questions

Figure 6. Baseline data can serve to inform professional development content.Primary and secondary school teachers do not understand all mathematics subject areas equally. The data points above the 1 to 1 line suggest relatively higher proficiency on a given question by secondary school teachers. By examining the relative performance on each question by the different groups of teachers we can identify concepts where a need for professional development is not indicated (i.e. Question 1). Questions with a great disparity between the two groups can be used to fine tune professional development in areas of content knowledge where it is most needed (Question 39 builds off of State Standards from the grade 4 through grade 7).

CONCLUSIONS 1. Any experimental design in education must take into account both traditional scientific design

motifs as well as ethical considerations (i.e., ultimately providing treatments for all subjects during the study). 

2. To optimize statistical power the choice of schools or classrooms as experimental units depends on the amount of variation within that unit. 

3. Pairs of experimental and control groups must be established by taking into account multiple variables (i.e., socio-economic status, school test scores, etc.). 

4. In our study, there are significant differences in both math content and pedagogy between elementary (Grades 4-6) and Middle School (Grades 7 & 8) teachers. Further, there are defined areas within mathematics where there are significant differences between these two groups.

  ACKNOWLEDGEMENTS

1. This work is funded by a Math Science Partnership grant from the National Science Foundation.2. We would like to acknowledge the support of the ALPHA Center at the UCR for assisting in the preparation of

assessment materials and data collection.3. We thank the Jurupa Unified School District for their cooperation and assistance with this project.