Upload
agatha-bryant
View
218
Download
1
Embed Size (px)
Citation preview
Evaluating the Vermont Mathematics Initiative (VMI) in a Value Added Context
H. ‘Bud’ Meyers, Ph.D.College of Education and Social Services
University of VermontDouglas Harris, Ph.D.
The Evaluation Center The Vermont Institutes, Montpelier, VT
Purpose
Describe the context, purpose, structure, staffing and content of the instructional program for teachers
Describe the logic model that provides the framework for evaluation of the program
Examine the methodology for assessing the value added to student learning by teacher preparation in mathematics
Present findings from cross-sectional analyses of eight cohorts of students who have been taught by VMI teachers and four cohorts of control students.
Present findings from a qualitative evaluation of the process of implementation of the components of the program from the perspectives of participants and administrators
Present recommendations arising from the findings of the evaluation
Context, purpose, structure, staffing
A field based masters degree in Education with a primary emphasis on mathematics content
Partially funded by local districts, partially MSP
Focus on Grades k-8 Includes focus on teacher leadership Includes action research Three years Mathematicians and educators
Content-12 Courses
Course 1: Mathematics as a Second Language (3 credits)
Course 2: Functions and Algebra for Elementary Teachers (3 credits)
Course 3: Trigonometry for Elementary Teachers (2 credits) and Algebra and Geometry II (1 credit)
Course 4: Measurement, Geometry, and Probability for Elementary Teachers (3 credits)
Course 5: Number Theory for Elementary Teachers (3 credits)
Courses…contd.
Course 6: Statistics, Action Research, and Inquiry into Effective Practice, I (3 credits)
Course 7: Statistics, Action Research, and Inquiry into Effective Practice, II (3 credits)
Course 8: Algebra and Geometry for Elementary Teachers, III (3 credits)
Course 9: Statistics, Action Research, and Inquiry into Effective Practice, III (3 credits)
Course 10: Calculus for Elementary Teachers, I (3 credits)
Courses…contd.
Course 11: Calculus for Elementary Teachers, II (3 credits)
Course 12: Capstone VMI Experience (3 credits)
Methods
Quantitative- Cross-sectional and longitudinal Two intervention groups
Group 1=16 schools, multiple VMI teachers (HiEnd)
Group 2=22 schools, single teacher per school (RegIntervention)
Control group=22 schools, matched to Groups 1 and 2 (combined) on demographics
Longitudinal comparisons for 2 cohorts, 1999 and 2000, grades 4 to 8. (See Figure 2)
Methods
Qualitative Interviews
Participants Graduates Administrators
Observations Review of course materials and participant
portfolios Course evaluations Interviews of School Staff and Leadership
Sample Characteristics
Comparison of Free/Red Lunch- Schools (%)
10
15
20
25
30
35
40
Groups of Schools
Per
cen
t F
ree/
Red
Lu
nch
2001 Free/Red Lunch- School (%)
2001 Free/RedLunch- School (%)
32 36
VMI Control
Figure 1
Free or reduced price lunch eligibility is a metric for poverty used by the Vermont Department of Education in the calculation of Adequate Yearly Progress. At grade 4 for each cohort when it was measured in the present study, it yields a statewide average of about 34%. Both the VMI and Control schools were about at the state average. Despite a 4 percent differential in poverty level between the VMI and Control groups, comparisons of the VMI and Control groups of schools for both of the cohorts indicated no significant differences among the matched schools with respect to poverty levels as documented by the Vermont Department of Education in 2001.
Current and Potential Trend Studies with 3 data points for VT Data
Others are cross-sectional, independent group comparisons
Year Gr 4 Gr 5 Gr 6 Gr 7 Gr 8 Gr 9 Gr10
1999 x
2000 x
2001 x
2002
2003 x
2004 x x
2005 x x x
2006 xx = Comparisons
Longitudinal comparisonsFigure 2
Question 1: What do comparisons with comparable schools show, over time, when students are matched from grade 4 through grade 8 and grade 10 during the two cohort groups (1999-2005, and 2000-2006)?
Intervention and Control Groups from 1999 to 2005 Matched Pairs Scaled Scores
620
670
720
Years of Intervention
Sca
led
Sco
res
VMI
Control
VMI 662 720 745
Control 657 716 734
1999 2003 2005
Significant differences in 2005. t=3.22, p<.001, df 1, 1300.
Intervention and Control Groups from 2000 to 2006 Matched Pairs Scaled Scores
620
670
720
Years of Intervention
Sca
led
Sco
res
VMI
Control
VMI 671 720 728
Control 668 708 720
2000 2004 2006
Significant differences in 2004. t = 5.27, p<.001, df 1, 1488Significant differences in 2006. t = 3.39, p<.001, df 1, 1107
Figure 3
Figure 4
Discussion
As indicated in Figures 3 and 4, each of the VMI cohorts outperforms the Control schools in similar patterns of difference. Results for comparisons at the first data point for each cohort (1999 and 2000) are not significantly different while they emerge as significant in 2005 and 2006.
Intervention and Control Groups from 1999 to 2005 Matched Pairs Scaled Scores
650
675
700
725
750
Years of Intervention
Scal
ed S
core
s Reg I
Control
HiEnd
Reg I 650 708 735
Control 653 709 735
HiEnd 667 725 753
1999 2003 2005
Significant differences in 2005. adjusted for differences in 1999,
HiEnd v Reg and Control, F = 7.96, df 2, 1235, p <.001
Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (1999 Cohort)
Figure 5
Intervention and Control Groups from 2000 to 2006 Matched Pairs Scaled Scores
650
675
700
725
750
Years of Intervention
Sca
led
Sco
res
Reg I
Control
HiEnd
Reg I 664 710 722
Control 667 706 719
HiEnd 674 727 730
2000 2004 2006
Significant differences in 2006 adjusted for differences in 2000, HiEnd v Reg and Control, F = 9.91, df 2, 1093, p <.001 Note: The average standard deviation for groups is about 40 points across all groups.
Mean differences between groups is thus approximately .25 S.D. between HiEnd and Control groups.
Question 2: How do levels of implementation of the VMI curriculum in classrooms relate to levels of achievement of students who experience them? (2000 Cohort)
Figure 6
Discussion:
VMI HiEnd Schools significantly out-performed the matched schools and appear to carry most of the difference between the VMI and matched schools in both cohorts from 1999 through 2006. Analyses of covariance adjusted the score differences at the first two data points in each cohort. The final data point for each cohort represents group differences at grade 10.
Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (1999 Cohort) Do gaps close?
Free Lunch vs Others Scaled Scores 1999 CohortANCOVA
580
600
620
640
660
680
700
720
740
Year
Sca
led
Sco
res
Reg I (E)
Control (E)
HiEnd (E)
Reg I (NE)
Control (NE)
HiEnd (NE)
Reg I (E) 638 692 694
Control (E) 646 697 720
HiEnd (E) 650 699 725
Reg I (NE) 654 715 722
Control (NE) 656 714 721
HiEnd (NE) 670 729 727
1999 2003 2005
Figure 7
Question 3. Are school level effects related to income level of students and levels of content implementation of the VMI curriculum? (2000 Cohort) Do gaps close?
Free Lunch vs Others Scaled Scores 2000 CohortANCOVA
580
600
620
640
660
680
700
720
740
Year
Sc
ale
d S
co
res
Reg I (E)
Control (E)
HiEnd (E)
Reg I (NE)
Control (NE)
HiEnd (NE)
Reg I (E) 635 695 723
Control (E) 639 700 722
HiEnd (E) 643 706 723
Reg I (NE) 652 713 727
Control (NE) 656 705 726
HiEnd (NE) 661 723 728
2000 2004 2006
Figure 8
Discussion:
The baseline, grade 4 performance is comparable for all groups. Not surprising is the observation that whether in intervention schools or matched schools, students eligible for free or reduced lunch score significantly lower than their non-eligible for free or reduced price lunch peers. This pattern continues through the grade 8 testing. Secondly, the grade 10 Free-lunch eligible students are significantly outscored by their non-eligible peers in the matched schools. No surprise there. But, rather surprising is that given the lack of cohort differences at grades 4 and 8, the free or reduced lunch eligible students are significantly out-scoring their eligible peers at grade 10 and they are gaining on students who are not eligible for free or reduced lunch in the matched schools. The mean score for the group of VMI Free-lunch eligible students (723, in 2006) is approaching that of the Matched Schools non-eligible students (726). In other words, the gap is narrowing between the free or reduced lunch eligible students and their non-eligible peers in the matched schools. These results were not obtained for the regular VMI schools in 2005 (Figure 5) and we have no explanation for that result at this time.
Quantitative Summary
Longitudinal comparisons of VMI and Control schools indicate a persistent pattern of advantage for VMI schools
Longitudinal comparisons indicate that the VMI schools, with more VMI teachers, had a significantly greater long term gain on percentile ranks. Stronger teachers may not have as great an effect in the shorter term when isolated.
Longitudinal comparisons suggest that SES differences are reduced over time when high quality mathematics instruction is offered to all students
Qualitative Findings (2005 and 2006)
Impact on Participants and Their Teaching Finding 1: Mathematics Content An overarching theme is the impact of the VMI experience on the teacher’s own
understanding of mathematics content. This is consistent among participants in all three years of the VMI experience and remains the highest priority of graduates as they reflect on the VMI experience
Finding 2: Increased Confidence Related to Mathematics Increased understanding of mathematics content impacts the confidence of participants as
related to mathematics, to teaching mathematics, and to their enthusiasm for and enjoyment of mathematics.
Finding 3: Impact on Instructional and Assessment Practice Participants graduates (many of who are math teacher leaders), and principals report that
the instruction in VMI, increased content knowledge, and increased confidence have had major impact on instructional and assessment practice in the classrooms of participants and of teachers impacted by VMI-trained math teacher leaders
Finding 4: The Impact of Action Research on Classroom Practice The impact of action research is mixed, with some participants, graduates, and principals
reporting considerable impact, others less so. Finding 5: Principal Support The active support of principals makes a profound difference in VMI participants’ work in
the classroom and in leadership positions.
More Qualitative Findings
Finding 6: Personal Impact on Participants Beyond the impact of VMI on teachers in relation to math content, instruction, and leadership, the program
also has profound personal impact on many participants. Impact on Students Finding 7: Transfer of VMI Content to the Classroom Teachers report direct transfer of mathematics content used in VMI to the math experiences of their
students. Finding 8: Impact on Student Problem Solving Teachers and teacher leaders report that the problem solving emphasis in VMI has significant impact on
their understanding of how to engage students in problem solving in the classroom. Finding 9: Impact of Action Research on Students Some teachers believe that the interventions begun in their action research projects will continue to impact
their students over time. Impact on Teacher Leadership in Schools and Districts Finding 10: Impact on Teacher Leaders Teachers working as teacher leaders credit VMI for providing the knowledge, confidence, and support for
them to take on leadership roles. This is true of teachers working in leadership at the school, district, and state levels.
Finding 11: Impact on curriculum, instruction, and assessment Teacher leaders report that the knowledge base and contacts developed during VMI have significant
impact on the development and implementation of mathematics curriculum, instruction, and assessment. Finding 12: Need for Ongoing Professional Community There is a common desire among VMI graduates to maintain the type of professional learning community
afforded them through VMI.
Recommendations
Recommendations Related to Continued Study of the Vermont Mathematics Initiative
Continue to gather longitudinal data from the State of Vermont’s Assessment System. .
As Vermont transitions to statewide assessment utilizing the New England Common Assessment, carefully analyze the logic and structure of the NECA and review VMI course content in relation to the Grade Level Expectations upon which this assessment is built.
Continue qualitative analysis utilizing existing data sources and consider adding series of observations within VMI participants’ classrooms to better understand what exactly is happening in those classrooms.
Select and implement a “theory into practice” change model considering, for example:
o the IBM/Harvard School of Business Change Toolkit and/or o the McREL Balanced Leadership Model. o the CBAM model of school change o the New Structure of School Improvement model (Joyce and Calhoun)
Educational or Scientific Importance of the StudyThe National Research Council, (2004) US Department
of Education (2004) and researchers at RAND (2004) have cited the need for better evaluations of teacher professional development programs. The combination of mixed method approaches with multiple strategies such as longitudinal studies of students acting as their own controls over time is the present attempt to respond to current concerns. In addition, intervention students in combination with matched groups at multiple intervals is a further attempt to provide additional confidence in results.
Want a copy of the paper or PowerPoint?? [email protected]