School - University Partnerships for Science and Mathematics ReformA one-year proposal for $100,000 submitted in Dec. 2001to the Arizona Board of Regents: Eisenhower Math-Science Program

David Hestenes, Research Professor of Physics, ASUJane Jackson, Academic Associate, Department of Physics, ASU480-965-8438, Jane.Jackson@asu.edu

Final Report by Jane Jackson. June 2003 (revised and updated in fall 2008, with underlining to highlight Mesa Public Schools activities and results, since that was our primary partner district)

Introduction and overview: The idea that K-12 schools need help from universities to upgrade the science/math

curriculum has emerged as a major initiative in the National Science Foundation: the Mathematics and Science Partnerships Program. It provides substantial federal funding for the kind of school–university partnership created in this project. Such school–university partnerships provide schools with a much-needed mechanism for continuous research-based science/math education reform – – perhaps an essential mechanism

Schools and school districts are ill-equipped to conduct science and math professional development on their own, because they lack necessary expertise in mathematics, science and technology as well as resources to keep up-to-date with advances in science and math curriculum materials and pedagogy. Those resources reside primarily in the nation’s universities.

It is imperative that we learn how to make such partnerships work effectively as soon as possible so that federal money will be well spent and Congress will have justification for continuing this program. For this reason we did not wait until the new program took effect last year; we applied immediately for this “proof of concept” project on partnership design and implementation, with high potential to serve as a model for other universities. As we have been laying the groundwork for statewide partnerships for a decade, we are well prepared to get Arizona partnerships up and running rapidly.

The Arizona Science and Technology Education Partnership (AzSTEP) began as a statewide partnership between the ASU physics department and high school physics teachers to support the teachers as local leaders of K-12 science and technology education reform. In the four years since it started, AzSTEP has engaged half of Arizona high school physics teachers in Modeling Workshops. Physics teachers cultivated in AzSTEP provide a cadre of leaders for expansion of AzSTEP to a statewide network of school district - university partnerships. Immediate Goal: To set up trial school-university partnerships in local public schools and

demonstrate the effectiveness of this mechanism for providing school districts with access to university resources to drive science/math/technology education reform.

Long-range Goal: To establish the Arizona Science and Technology Education Partnership (AzSTEP) to broker school-university partnerships statewide and secure their funding to support continuous upgrades in science/math/technology education.

1

Although ultimately science and mathematics reform must encompass the entire twelve years of schooling, this project addressed only one component, albeit a crucial one: the need for an integrated physical science and mathematics workshop for seventh through ninth grade teachers. Among other things, it established a much-needed curricular link between high schools and their feeder schools. Physical science courses in middle school and ninth grade must provide students with the conceptual underpinnings needed for success in high school physics and chemistry. At the same time, these courses must develop basic scientific literacy for students who will not go on to physics and chemistry.

As AzSTEP's goal is to establish a permanent mechanism for statewide reform affecting all students, we targeted districts with the greatest likelihood of success in this “proof of concept” project. Participating school districts were Mesa Public Schools, Peoria Unified, and Glendale Union High School District (GUHSD) with its two feeder districts, Washington (WESD) and Glendale Elementary (GESD). They were selected on the basis of administrative commitment and teacher qualifications. Administrative commitment is, of course, essential for district-wide reform. But the most important requirement is the existence of a cadre of teachers capable of leading implementation of the program. Each of the selected districts already has such a cadre, composed of high school teachers who have been active in the Modeling Program for years and have shown that they are fully attuned to the aims and methods of the project. In fact, their reputation for outstanding teaching was the most important factor in gaining administrative support from their districts. Thus two expert high school teachers, one in physics and one in math, were selected to lead the appropriate district-wide workshop. In Glendale, two workshops were held, one including high schools to which GESD feeds and the other similarly with WESD.

Each district pledged to contribute $10,000 or more for teacher stipends. In fact, Mesa and GUHSD contributed considerably more than this, whereas WESD contributed considerably less. Each district applied for and was awarded an Arizona K-12 Center "Fast Track" grant of $10,000, since AzSTEP was a pre-approved provider of this Modeling Workshop.

Evidence of school district needs:Mesa Public Schools: Mesa is the third-largest city in Arizona. It has experienced

explosive growth in recent years. With 74,100 students in more than 90 schools and alternative K-12 programs, Mesa Public Schools is the largest district in the state of Arizona. It is now an urban district, with sections of the city inhabited almost exclusively by ethnic, mostly disadvantaged families. Hispanic and American Indian youths constitute almost a third of the student population, and 45% of Mesa’s students qualify for free or reduced lunch subsidies.

District data indicate that ethnic children in Mesa’s economically disadvantaged regions perform poorer on measures of educational achievement. As of September 2002, 18 of Mesa’s 35 Title I K-12 schools failed to make Annual Yearly Progress (AYP) for two years or more, subject to state intervention according to new regulations in President Bush’s No Child Left Behind legislation. These data underscore the profound increase in diversity, and the accompanying needs, of many students and schools in the district. If unchecked, this situation will result in significant numbers of students failing to meet minimum competencies, in turn qualifying Mesa as a “failing” district according to the new law. Thus Mesa’s school community is motivated to find ways to help the district’s lowest-achieving students.

Similar data and needs exist in WESD, GUHSD, and especially in GESD, where 65% of students are on the free lunch program.

2

Our telephone interviews in spring 2002 with science and mathematics teachers in all thirteen Mesa junior high schools revealed that few mathematics and science teachers were coordinating their courses. Moreover, junior high schools have been experiencing rapid turnover of science teachers, with the result that a preponderance of teachers are inexperienced in use of technology in instruction.

GUHSD: Our district-wide survey of mathematics teachers in August 2001 revealed that only one-third of 34 ninth grade math teachers took physics in college. For the most part, articulation in mathematics and science was lacking between GUHSD, GESD and WESD.

GESD and WESD: In spring 2001, only 3% of Glendale Elementary District eighth graders passed the mathematics section of the AIMS test. Graphing calculators were not used in most middle schools, thus depriving students of an important representational tool for learning mathematics.

Peoria USD: In surveys returned to us by 42 science teachers in grades 6, 7, and 8 in Aug. 2001, the median reported number of hours per year of school inservices related to science or math content was zero! Only half of the teachers have taken a college-level course in physics or earth science, and only one-fifth had astronomy. Most high school teachers of physical science majored in life science, and some have only a minor in any science. Fewer than one-third of ninth grade science teachers have had a calculus course.

Development of workshop: Larry Dukerich, the Mesa Modeling Workshop leader, described development of the Modeling Workshop in an e-mail to Jeremy Roschelle of SRI International (Jan. 4, 2003). Dr. Roschelle is a collaborator with Professor James Kaput in the SimCalc mathematics project at the University of Massachusetts - Dartmouth. "The course development for Models of Physical Science originated years ago when physics teachers in Modeling Workshops attempted to prepare a set of remedial materials for under-prepared physics students patterned after the efforts of Jim Minstrell and Arnold Arons. Teachers found these to be of use for their ninth grade physical science courses, where, as you likelyknow, the textbooks are abysmal. David Hestenes challenged Jeff Hengesbach (now at the Webb School in Claremont, CA) and me to flesh out the course so that it would weave the threads of modeling tools, structure of matter, and energy into a coherent course that could be used to prepare students for a physics-first course in ninth grade. The design specifications also called for greater integration with the math curriculum for eighth and ninth grades. David strongly recommended that we work Sim-Calc materials into the course design. We met with you up at NAU a year ago and we went to work. The task was simply too large for two teachers working part-time to complete, but we finished the first three units well enough (we think) that the materials could be used for the Physical Science-Math Modeling Workshops that you and Jane discussed. Since our primary goal was to develop a physical science course, we chose those Sim-Calc materials that seemed most relevant to our needs in the first unit." Recruitment of teachers:

In spring 2002, each of the five school districts recruited teachers using a two-page application form. Leaders were sought: stated criteria were, for example (GUHSD): "Preference will be given to applicants showing the greatest potential for improvement of student learning,

3

implementation of technology and modeling instruction on a regular basis, and leadership at the local school for implementation of technology."

Pledged commitments were, for example (Peoria):* Are you willing to give a 1/2 hour standardized test next year as a pre-test and post-test? * Are you willing to serve as a resource in your school for science & math teachers, both on an individual basis and in conducting training sessions on classroom technology for groups?* Are you willing to be actively involved in a learning community of physical science and math teachers?

GUHSD asked principals to choose a team of a math and a science teacher from among applicants, and to try to provide common planning period. Other districts recruited too late for this.

Summer modeling workshops:Ninety-two teachers participated in four three-week Modeling Workshops in June at:

* Independence High School: 11 GESD teachers, 10 GUHSD teachers* Washington High School: 12 WESD teachers, 11 GUHSD teachers* Dobson High School: 23 Mesa teachers, 1 from Desert Eagle charter school, 1 from Gilbert HS* Peoria High School: 20 Peoria teachers, 1 from Higley HS, Morenci HS, AZ School for Arts.

All teach grades 7, 8 or 9 except for six lower-grades GESD teachers and a few sixth grade Peoria teachers. Half of the 92 teachers teach science, 35 teach mathematics, and the rest teach both or all subjects. Forty-two schools were represented in the five districts, and almost two-thirds of the schools had two or more participants. Six teachers at Cholla Middle School in WESD participated, including almost the entire science department.

All workshops except Dobson High School met for six hours daily, with a total of 90 contact hours in 13 to 15 days. Dobson had four hours each afternoon for 15 days, for most Mesa math teachers were committed to teach in mornings. The half-day workshop design was best: teachers worked effectively and were still enthusiastically learning in the third week, whereas teachers in the three all-day workshops were tiring and losing focus.

Since optional ASU graduate credit in physical science or mathematics was available, teachers were assigned homework such that the total time devoted to the course by a participant was at least 135 hours as required by the Arizona Board of Regents for a three-semester hour course.

Workshop leaders were expert high school teachers in the respective districts, one each in physics and chemistry (listed first) and mathematics.* Independence High School: Hal Eastin and Cheryl Bryant.* Washington High School: Sheila Ringhiser (with Dawn Harman) and Veronica Carlson.* Dobson High School: Larry Dukerich and Stella Ollarsaba.* Peoria High School: David Hill and Curt Aylward.Workshop leaders communicated regularly via e-mail, sharing successes, problems, and strategies.

The workshops provided teachers with education in standards-based content and instructional strategies. ASU is well-qualified for this: in 2001 an Expert Panel of the U.S. Department of Education designated the Modeling Instruction Program as one of two Exemplary K-12 science programs in the nation, out of 27 projects reviewed. Ratings were based on these

4

criteria: (l) Quality of Program, (2) Educational Significance, (3) Evidence of Effectiveness, and (4) Usefulness to Others. In 2000 a different Expert Panel of the U.S. Department of Education designated Modeling Instruction as one of the seven best K-12 educational technology programs in the nation (out of 134 applicants!). Links to panel reports are at http://modeling.asu.edu .

Participants were introduced to the Modeling Method as a systematic approach to the design of curriculum and instruction. The name Modeling Instruction expresses an emphasis on making and using conceptual models of physical phenomena as central to learning and doing science. Adoption of "models and modeling" as a unifying theme for science and mathematics education is recommended by both NSES and NCTM Standards as well as AAAS Project 2061. However, to our knowledge, no other program has implemented it so thoroughly. Modeling workshops meet or exceed many Arizona professional teaching standards, and grade eight and high school performance objectives in the science and mathematics standards. The essence of Modeling Instruction is that content is reorganized around basic models to increase its structural coherence. Student activities are organized into modeling cycles, which engage students systematically in all aspects of modeling. The teacher guides students unobtrusively through each modeling cycle, with an eye to improving the quality of student discourse by insisting on accurate use of scientific terms, on clarity and cogency of expressed ideas and arguments. After a few cycles, students know how to proceed with an investigation without prompting from the teacher. The main job of the teacher is then to supply them with more powerful modeling tools. Lecturing is restricted to scaffolding new concepts and principles on a need basis. Details of the modeling cycle are at http://modeling.asu.edu/modeling-HS.html. In Modeling Workshops, participants are supplied with a complete set of course materials (resources) for a semester, and they work through activities alternately in the roles of student or teacher.

Workshop goals for teachers were to:* improve their instructional pedagogy by incorporating the modeling cycle, inquiry methods, critical and creative thinking, cooperative learning, and effective use of technology in instruction,* understand content in structure/properties of matter, motion, energy, scientific thinking skills, and related skills in each of the six Arizona Mathematics Standards (Strands),* strengthen coordination between mathematics and physical science. Anticipated student outcomes included improved understanding in geometrical and physical properties of matter, motion, energy, graphing, and related mathematics and reasoning skills such as measurement, proportional reasoning, and relation between graphs and equations.

Thematic strands woven into this course included scientific modeling, structure of matter, energy, and use of calculators and computers as scientific tools. Mathematics instruction was integrated seamlessly throughout the entire course by an emphasis on mathematical modeling. The course included these models and modeling activities: 1. Modeling geometric properties of matter: length, area and volume 2. Modeling physical properties of matter: mass and density 3. Model of a point particle with constant velocity 4. An atomic model of solids, liquids and gases 5. Energy and the states of matter.

5

Larry Dukerich reflected on his workshop experience in his e-mail to Jeremy Roschelle of SRI International (Jan. 4, 2003): " The three-week summer workshop was an eye-opening experience for the workshop leaders as well as the participants. Participants began to realize that it was possible for junior high math and science teachers to use common vocabulary - that the math teachers could do activities and the science teachers could use TI-83+ calculators (it was as if some taboos had been lifted). The workshop leaders became more painfully aware of how fragmented our math and science curricula were. Due to the pressures placed on math teachers for their students to perform well on the state-mandated test, it's as if blinders have been placed on the majority of math teachers. For many, their strategy has been to prepare for the test rather than to try an approach that stressed a conceptual understanding rather than drill-and-kill. The workshop leaders (and the math teachers who got it) tried to convince the rest that more iterations of what hadn't worked in the past wasn't going to lead to success. Here's a sample exchange: Traditional teacher, "How are we going to do this [Sim-Calc approach to slope] AND what we usually do?" Reformed teacher, "No, you're going to do this INSTEAD of what you've been doing."

Unsolicited comments by teachers, soon after the workshop ended (via e-mail): I really enjoyed the Math & Science Modeling Workshop. Hal and Cheryl did an excellent

job!! I learned so much. I have so many new and exciting modeling lessons and materials to share with my students this upcoming school year.

Had to let you know how much I enjoyed the class. Larry and Stella did a fabulous job in every way. Thanks for getting me involved in this venture...my fear of physics has been greatly reduced and I intend to continue with the program and your suggestions.

Teachers' workshop evaluations, measurement of teacher learning by site: Workshop rating (10=superb): Peoria 9.7,WESD/GUHSD 9.3, Mesa 9, GESD/GUHSD 6.2. GESD presented difficult challenges. They had to select six teachers of grades four to six because 7th & 8th grade teachers taught summer school. This slowed the pace of the workshop, and the classroom needs of the lower grades teachers weren't met because the level of activities was too high for them. On the other hand, although the content and level were appropriate for the ninth grade teachers, the pace was too slow for them. Also, one workshop leader, a novice leader, worked diligently and tirelessly but was less than successful because of a leadership style that is more conducive to working alone rather than with a co-leader.

Physical Science Concepts Inventory (PSCI) teacher pretest & posttest mean scores:All teachers were pre-tested and post-tested on the Physical Science Concepts Inventory

(PSCI) and the Math Concepts Inventory (MCI). These instruments for students were developed specifically to align with this course. The first eight questions are in pairs, from the Classroom Test of Scientific Reasoning, a research-based test developed in the 1970’s by Anton Lawson, on which extensive student data internationally are available (some are cited below). Most others are released NAEP and TIMSS questions, and questions similar to the NCLB-required Arizona high-stakes math test for eighth and tenth grades, the AIMS test. Considerable overlap exists between the two inventories, for it is intended that the PSCI be given to science students and the MCI to mathematics students. The PSCI and MCI were last revised in June 2003, as a result of

6

these workshops; they are in .pdf at http://modeling.asu.edu/MNS/MNS.html. The password to open them can be obtained by e-mail to Jane.Jackson@asu.edu. (The most important revision is that the harder pair of the four proportional reasoning questions, #7 and 8, was deleted.)

Teachers began the workshop with poor understanding of matter and energy; and teachers with lowest pretest scores improved most, in general. In Mesa, the pretest mean was 69%, posttest mean was 80%; WESD/GUHSD: pretest 73%, posttest 81%; GESD/GUHSD: pretest 69%, posttest 77%, Peoria: pretest 66%, posttest 83%. All teachers at each site except Peoria took the posttest on the last workshop day. The Peoria posttest was given at the Saturday follow-up session in January; only six Peoria teachers' names could be matched pre- and posttest. (Appendix A shows each teacher’s scores; it is available upon request to Jane.Jackson@asu.edu).

Content subscales of PSCI (but not determined for Peoria, since the sample was small) are:* Conservation of mass (questions 1 & 2): everyone got these right on the posttest.* Conservation of volume (questions 3 & 4): Eight out of 65 (12%) got #3 and #4 wrong on the posttest. Five of the eight were from Mesa.* Proportional reasoning (questions 5-8): at each site, the mean pretest score was about 75%; mean scores improved to about 85% on posttest. Most teachers answered all four questions right on both tests, but several teachers answered all questions wrong on the pretest and improved to one right, indicating that they do not understand even the basics of proportional reasoning.* Graphing skills/motion (four questions); at each site, pretest was about 90%, posttest about 95%.* Geometrical & physical properties of matter (four questions): Mesa started lowest, at 70%; WESD/GUHSD started highest, at 90%. Mesa and GESD/GUHSD ended at about 85%. * Atomic model of matter (six questions): at each site, pretest and posttest mean scores were about 65%. However, actual teachers' scores were diverse, from none right to all right, with little improvement by anyone. * Energy and states of matter (four questions): Tests showed tremendous improvement in most Mesa teachers' scores; moderate or little improvement in individual teachers' scores at the two Glendale sites. At each site, the mean pretest score was 40%. Mesa teachers improved by far the most, to 75%; WESD/GUHSD to 50%. GESD/GUHSD didn’t include this content because of slower pace due to six elementary school teachers; their mean posttest average was 45%. Teachers' scores were diverse in the pretest at each site, from none right to three right. In December 2002, after I related the above results to Larry Dukerich, he wrote to me, "I would like to caution you against reading too much into the test results from the Physical Science Concepts Inventory. Unlike the FCI [Force Concept Inventory], which was developed over years based on interviews with students to come up with plausible distracters, this test is an amalgam of items selected ad hoc from a variety of sources with different goals in mind. This is not to say that the individual questions are not good, but that this collection of loosely related items does not constitute a coherent test design. Drawing conclusions about the relative effectiveness of the instruction in the four workshops may be inappropriate given the test limitations. Here's my analysis of the sub-groupings in the questions. 1. The questions borrowed from Lawson's Classroom Test of Scientific Reasoning (items 1-8) have been banging around for quite some time. ... The majority of the teachers already knew

7

how to do items 5-8. Those who did not, showed only slight gains primarily because there was little instruction on proportional reasoning in the workshop. 2. Items 9 - 12 [graphing/motion] were too easy for teachers of 8th and 9th grade - hence they fail to discriminate between teachers who already understood the concepts and those who learned as a result of the workshop. 3. On the positive side, I believe that questions 13-15 and 26 (geometric properties of matter) could be used to test how much of an effect the workshop had on teachers as this concept was explicitly covered in the workshop. 4. The items on the atomic model of matter are questions that probe deep misconceptions about the atomic model in much the same way as does the FCI. I am not surprised that, given the short duration of the workshop, that there was little or no gain. This should tell us more about how we have to structure the workshop in the future. 5. The relative success of the Dobson [Mesa] workshop teachers was due, in great part to the fact that we got through all three units, whereas teachers in the other workshops barely got to energy, if at all. Stella told me that I was a slave driver, as I pushed the teachers to complete the materials, knowing that the energy treatment would be novel for most of them. I have confidence in these items simply because I know how long and hard Gregg Swackhamer worked on them. So I urge caution in using these test results to draw conclusions about the workshops."

Follow-up meetings, and reflections by workshop leaders: Each district planned and funded its own follow-up events, subject to AzSTEP's request of at least two follow-up events of significant depth, preferably totaling ten hours.

GESD: In August, most GESD teachers met for six hours on a Saturday with Hal Eastin, a workshop leader, to develop lessons for their classes. In spring 2003, Renee Kopcha, the GESD curriculum coordinator, organized three afternoons of classroom teaching demonstrations followed by discussions. The workshop co-leader, Cheryl Bryant, reported that these model teaching meetings were valuable, professionally and personally. She wrote, "Some GESD teachers do amazing things with their students! Very good structure, policy, and procedures! My students can't fool me any more, when they tell me they weren't taught well at GESD. Unfortunately, GESD has huge turnovers of students, teachers, and principals; this is not conducive to student learning!" Cheryl said she's convinced that a key to improved teaching is for teachers to observe other teachers, followed by reflection and sharing of ideas. A great need exists for this in GESD, she said, for teachers are isolated, unlike at GUHSD, where a mentoring program is in place at each of the nine high schools.

GUHSD: Debi Plum, GUHSD mathematics coordinator, organized a full day of follow-up in October, including pay for substitutes for 18 teachers and five workshop leaders. Since ninth grade science is primarily earth/space science, not physical science, workshop co-leader Dawn Harman presented modeling activities in astronomy. Teachers worked on the following technology and modeling instructional techniques:

Implementation discussion – what’s going well? What are our concerns?Moon and daylight graphing activities

8

Downloading data to Graphical Analysis softwareSimCalc MathWorlds software refresherParachute activity (modeling motion using CBR, presented by co-leader Veronica

Carlson.)GUHSD teacher teams were asked to plan two lessons together, observe their team member's teaching, and return lesson planning/observation report forms to the district office.

Mesa: On Nov. 23, Mesa held the first of two Saturday morning meetings with workshop leaders Larry Dukerich and Stella Ollarsaba; ten teachers participated. Agenda:

* Midcourse correction: testimonials, difficulties* Math and energy activities* Expressions of desired future activities. Teachers proposed these: "Get together as a math/science team and focus on a concept (e.g., proportional relationships) in which you can discuss the vocabulary, topic and ways that each subject matter can augment the other." "Develop parallel worksheets from the curriculum that can be used in science/math."

Representative comments at this meeting by teachers were compiled by Larry Dukerich.* "The workshop brought about a change in my teaching style. This is shown in the increase in my students’ enthusiasm for science. They hardly ever work alone now. They are much more engaged. My students’ test scores also show improvement."* "Big whiteboards really promote interaction between the students. While one is writing, the others are helping and checking." "While working in groups, the students are debating and critiquing each other. Oral and written communication skills are being improved."* A math teacher was so enthused that she and her principal made sure that every one of the math teachers in her school had a classroom set of whiteboards. * They all felt that they knew enough about whiteboarding that they could implement this aspect of modeling in their classrooms.* "The workshop experience changed the way I look at science in general. I find myself looking for the models in the rest of the ninth grade science curriculum. It helps me realize the way students learn so that I am a more effective teacher." * Several teachers reported that they are “converting” colleagues who didn't participate.* The level of the activities was too high for seventh grade students.* Everyone expressed the concern that what was expected of them (either preparing for district tests or conforming to the existing curriculum) hindered their ability to implement as much of the materials as they would have liked. * Several teachers commented that it would have been much better if a team of math and science teachers from their school had participated in the workshop.

Peoria: Peoria funded 10 hours follow-up at $22/hour, including an eSchool using Blackboard.com and a four-hour Saturday meeting in January with workshop leaders David Hill and Curtis Aylward. (David Hill is the science facilitator for Peoria USD, in addition to teaching physics full time.) David reported that the in-person meeting, which was attended by seven participants, was more successful than eSchool, in part because eSchool is a new modality for teachers, so it was hard to get them to use it to ask questions of the instructors and to share ideas. The agenda for the meeting included issues of teachers, using technology, and changes in activities and

9

sharing of experiences. Teachers expressed frustration at the district's slowness in installing Graphical Analysis on networked computers, and science teachers were frustrated at not being given commitment for time to do science activities because of focus on the math AIMS test. On the other hand, math teachers found creative ways to use Modeling Workshop activities to teach Arizona math standards, not by rote, but by science applications. David Hill noted that the chief value of the Modeling Workshop was in process skills, rather than content. Teachers reported to him that their classroom has become more student-centered. Rather than students just doing a lab, filling out a worksheet, and turning it in, they engage in whiteboarding and Socratic dialogue -- teachers ask students more questions. Rather than the teacher giving a definition and doing a demonstration, followed by student verification, children develop concepts through experimentation and discourse/reflection. (This is not quite model development, he noted.) He said that it was hard for 7 th and 8th grade science teachers to directly implement most Modeling Workshop activities because 7th grade is earth science and 8th grade is life science. Sixth and 9th grade are physical science, so these teachers had no problem, he said. David recommended that, since fewer than half of the teachers were able to attend the well-publicized Saturday meeting, early-release time or two after-school meetings would be better.

WESD: In early August, WESD had two full days of follow-up with workshop leaders Sheila Ringhiser and Veronica Carlson to integrate workshop learning into their curriculum. Teachers wanted more follow-up sessions to strengthen their skills, but in mid-year the curriculum coordinator, Jennifer Cruz, took another job in the district. Thus follow-up sessions did not occur, to the disappointment of some teachers.

Listserv for participants:I started a listserv for all 92 participants, workshop leaders, and district coordinators. Hardly anyone ever posted. I posted occasionally on resources: how to borrow classroom sets of TI-83s from Texas Instruments, how to write a $500 grant to the Wells Fargo Teacher Partner program, recent research on 'not giving the answer' to middle school students, exemplary middle school science programs, error-laden middle school science textbooks, and the like.

Implementation: unsolicited comments by teachers (via e-mail): "I introduced whiteboarding and the TI-83 into my classroom for the first time this year. I

cannot express strongly enough what a positive impact this has had on my classroom! Wow! My students love sharing their work through whiteboarding and it has been an effective method for me to encourage more risk-free participation in cooperative groups. In fact, I keep spreading the word about how well it works and have even got my best friend whiteboarding in her English class over at Metro Tech! In addition, the TI-83 has completely captured my students' imagination."

(Received in May 2003) "I would have sworn that I didn't use much of the material I learned last summer, but then I watched my kids take that test. I had many copies so I let the kids work on the test[sheets]. They were on task during the test (I was in shock!) and then when I looked over the tests, I realized that each child had attempted every problem and that they had a method of approach for each problem. They extended graphs, drew pictures and wrote notes to themselves. I was floored:)" [Note: this experienced mathematics teacher reported

10

little implementation but had a great deal of enthusiasm for the workshop. Her students showed little gain on the Math Concepts Inventory.]

Measurement of student learning, by school district: I preface this section by quoting Larry Dukerich (e-mail to me, June 2003): "I am hesitant to draw far-reaching conclusions about the merits of individual questions (or categories of questions) based on the scores of some teachers in Mesa who frankly admitted that they were unable to implement the curriculum in any meaningful way. I cannot tell you how dismayed I was to learn (in our follow-up sessions) that the teachers could only do this or that activity here or there because they felt compelled to cover all the topics required of them in order to prepare their students for the end-of-year tests. Having spent a significant chunk of a year devising a set of activities designed to carefully build skills and a coherent view of the atomic model of matter and energy, I felt as if it had been a real waste of time. Some of these teachers may as well be a control group! So, I would do my best to compare gains to the teachers' self-rating of the degree to which they were able to implement the curriculum they learned - just as we did with the Modeling Instruction in High School Physics workshops. It seems reasonable to me that one would only find gains for students in classes in which teachers were able to implement the curriculum in a coherent way."

The highest gaining district for grades 7, 8, and 9 was Peoria (eight percentage points increase: for brevity, we call it an 8% gain in the PSCI). When disaggregated by grade, eighth graders had this same PSCI gain. A comparable gain was 9 th grade math in Mesa (seven percentage points on the MCI, which we call a 7% gain); most students tested were in alternative algebra, a two-period course for low achievers (representative of the group most in need of improved instruction!).

Specifically, actual scores and gains in scores for the 135 9th grade Mesa students who took the Math Concepts Inventory pretest and posttest are:

subcategory: pretest posttest Percentage points gainoverall 44% 51% 7%Conservation of weight, volume 57% 65% 8%Proportional reasoning 8% 15% 7%graphing skills and motion 61% 70% 9%geometrical and physical properties of matter

35% 41% 6%

I consider the increase of seven or eight percentage points to be roughly commensurate with the typical ten percentage points increase in Force Concept Inventory scores of tens of thousands of physics students in the first year of their teacher’s implementation of Modeling Instruction. The Force Concept Inventory is simply a better test than the PSCI and MCI. Modeling Instruction is a complex innovation, and my long-term study (unpublished as of 2008) of physics teachers shows that student posttest mean scores continue to rise for three or four years after the Modeling Workshop. Student gains are higher if the teacher takes a second modeling workshop. Teachers report that the first year of implementation is hard; they feel comfortable using Modeling Instruction only after three or four years. Modeling

11

Instruction is transformative, many teachers report. Satisfaction is high, and some write that it saved their careers.

Appendix B (available by e-mailing Jane.Jackson@asu.edu) is a spreadsheet of grade 7, 8, and 9 student pretest and posttest mean scores for the PSCI and MCI in 2002-03, overall and disaggregated by district and grade. All data are matched, pre- and posttest. Included are the number of students in the sample, i.e., the number who took both tests (pre- and posttest). As of 2008, we have analyzed data by gender and race/ethnicity.

Teachers' scores in the atomic nature of matter subcategory were low and showed no improvement, so we did not analyze them for students. When looking at these student data, it is important to realize:1) Some test questions needed revision or replacement to better reflect the actual content and/or process skills of the Modeling Workshop. (This was done in June 2003.)2) Workshop leaders expressed concerns like this to me, "Science teachers were frustrated at not being given commitment for time to do science activities because of focus on the math AIMS test. On the other hand, math teachers found creative ways to use Modeling Workshop activities to teach Arizona math standards, not by rote, but by science applications." 3) Most teachers could not implement some content. They expressed three reasons for this. First, in some districts, science in a certain grade was mostly life science, or earth science; and in other districts the physical science content was specified to be other than what was taught in the workshop (for example, Newton's laws in GESD, but not energy - a bad idea, because energy is more fundamental to all sciences than forces). Second, workshop leaders reported that ALL teachers "expressed the concern that what was expected of them (either preparing for district tests or conforming to the existing curriculum) hindered their ability to implement as much of the materials as they would have liked." Third: in Mesa, 7 th and 8 th grades had only one-half year of science, and they had to race through an extensive list of topics.4) Surveys returned in spring 2003 by teachers reveal great disparities in implementation of modeling methodology. 5) Since grade nine is included on the spreadsheet, Mesa Public Schools, GUHSD, and Peoria USD are expected to have higher scores than GESD or WESD, although most 9 th grade Mesa results are for alternative algebra for low-achievers.

The weakest subcategories of inventories that were analyzed are:1) Proportional reasoning: only the easier pair of questions, #5 and 6, were analyzed; these scores are typically under 10%. (Answers for the pair of #7 and 8 were random; most students guessed.) These scores are the same as those for 104 8th graders in suburban east Mesa in January1982, as reported by Prof. Anton Lawson of Arizona State University (Lawson, Anton and Bealer, Jonathan, "Cultural Diversity and Differences in Formal Reasoning Ability", Journal of Research in Science Teaching 21, p.735-743, 1984). In that study, students were told that a given quantity of water occupies four units in a wide cylindrical container and six units when poured into a narrow one. They were asked to predict how high a given quantity of water that occupies six units in the wide container would rise if poured into the narrow container, and state their reason. (This is the essence of questions #5 and 6. Anton Lawson adapted his test to multiple choice around 2000.)

12

Eighth graders in Berkeley, California scored twice as high on this task in spring 1978: 104 students averaged about 20% (reported by Anton Lawson in the article cited). Sixth grade students tested in 2001-2002 by Dr. Kalyani Raghavan for her Model Assisted Reasoning in Science (MARS) Program at the University of Pittsburgh scored about the same on these two questions: 12% on the pretest, 14% on the posttest for both the reform group and the control group (670 students in MARS, 297 in the control group) (private communication).

2) Energy and states of matter (actual scores are typically under 10%). Four research-based conceptual questions about energy were included, one of which is the following:

As water in an ice cube tray freezes,a. it absorbs energy from its surroundings.b. its surroundings absorb energy from it.c. it absorbs coldness from and releases energy to its surroundings.d. it only absorbs the coldness from its surroundings.e. it neither absorbs nor releases energy, because its temperature stays constant.

Students don't begin to understand energy. As with proportional reasoning, scores are below the random score of 20%, indicating powerful alternative conceptions. Incidentally, the five choices are expressed in the language of children who were interviewed. Middle school textbooks do a poor job of treating energy. The Modeling Instruction Program improves instruction on energy, the most fundamental concept in science. An enlightening article, “Making Work Work,” can be downloaded at http://modeling.asu.edu/modeling-HS.html, and several other resources on energy are posted on the modeling website, including a Basic Energy Concept Inventory for grades 8 to 12.

Response rates: Of the 87 teachers in the five school districts, 61 gave the PSCI or MCI pretest to students. Thirty-two submitted posttest answer sheets as well (18 in math, 15 in science, one in both math and science). Response rates for both inventories were: GESD: 7 out of 11 teachers, GUHSD: 2/22, Mesa: 10/23, Peoria: 8/20, WESD: 5/11. GUHSD's response rate was low because many math students change teachers quarterly, so it was difficult to give the posttest to the same students. In addition, GUHSD has a week-long performance based assessment near the end of the year in all subjects, plus the AIMS test, the Stanford 9, and one more test. That is too much testing, leaving little time for teaching and learning, teachers say.

Correlation of gains of individual teachers' students with self-rating of degree of implementation: No teachers reported that they implemented all content and all components of Modeling Instruction consistently, and no teachers rated their implementation of the modeling cycle as very good. Thus I could not discern strong correlations, unlike for physics. I present a few case studies. The two science teachers who had the highest student gains are a 9th grade (physical science) and a 7th grade (earth science) in Peoria. They rated their overall implementation of the modeling cycle as good or fair. As markers of implementation, both said that they regularly ask students to work in groups in class, regularly ask different groups to discuss their ideas in class (a large increase from the previous year), sometimes or frequently use hand-held whiteboards (a large increase from the previous year), sometimes lecture, and seldom or never use a textbook. The 9th

13

grade teacher's PSCI score increased by 16 percentage points, from 74% to 90%. So that teacher's knowledge improved, she implemented most components of Modeling Instruction consistently, and her students achieved considerably. A Mesa 9 th grade algebra teacher with over 2 decades of experience said that the Modeling Workshop enhanced her teaching maximally (i.e., 5, on a scale of 1 to 5). She implemented Socratic questioning strategies regularly, used whiteboards frequently, had in-depth group work sometimes, and focused student thinking on models equally with topics. Her implementation of the math part (graphing, data gathering, and motion content) was 4, on a scale of 1 to 5. Her student gains were moderate. A brand-new Mesa 7 th grade science teacher self-reported nearly as high a degree of enhancement of his teaching (4), and a similar degree of implementation of methodology, except that he used Socratic questioning only sometimes. His student gains were small; but 7 th grade science in Mesa is life science, not physical science, and it's only half a year. Two GESD teachers who team-teach math and science in 7th and 8th grade reported fair overall implementation of the modeling cycle. Their use of whiteboards increased from never to regularly, and they frequently used the modeling cycle. Their 96 students had moderate gains on the MCI.

Summary of self-reported trends among teachers in all districts this year: Large increase in use of whiteboards Some increase in student discourse (Socratic questioning) Less lecturing Less use of a standard textbook The workshop moderately enhanced their teaching (3 on a scale of 1 to 5) Little coordination of courses with colleagues No improvement in classroom technology. Typically teachers have one to three computers in

their classroom. Some have access to a computer lab (often with difficulty in scheduling). Most teachers don't have graphing calculators, with the exception of GUHSD (and they don't have TI-83+, but rather older models for which SimCalc can't be used).

I was able to provide StRUT-donated computers for a science teacher in Cholla Middle School in WESD, where six teachers participated, and I helped get TI-83s for Shepherd Junior High School in Mesa, where four teachers participated. They wrote a grant to the Mesa Educational Foundation. My $10,000 proposal to the Medtronic Foundation for TI-83s was not funded.

The degree of encouragement from school administration to implement the workshop learning as reported by teachers was vastly different in different schools. It tended to be either low or high; not moderate. Most WESD teachers reported little or no support from their school or district. I was impressed with the support at district level in Mesa, GUHSD, GESD, and Peoria.

I perceive that one factor that can reduce support in the school of any teacher, is a lack of math and science background of the administrator. Many such administrators do not understand science or math, nor the needs of teachers of these subjects, nor reforms that work.

14

To gather additional important information and insights, I conducted half-hour phone interviews with four teachers from four districts. The teachers report the following: groups of students especially well served by Modeling Instruction are lower ability students -

concrete learners; shy, bright kids, ELL (mentioned by three teachers: visual representations; input from other kids; whiteboarding helps a lot with vocabulary!); gifted students (they can use software without supervision); special needs students -- special ed (SPED: graphing skills increase their self-esteem), and girls (they enjoy science more).

Obstacles to implementation are: modeling takes time -- block schedules are preferable to traditional length classes; too much hassle to go to the computer lab to use MathWorlds or Graphical Analysis software; Peoria school district refused to install SimCalc MathWorlds on computers; lack of meter sticks and triple beam balances (in a GESD elementary school).

One teacher used SimCalc MathWorlds software for two periods, and she'll use it for a week next year in alternative algebra.

Desires for future professional development are: review strategies in modeling, questioning, and cooperative groups; review use of TI-83+ graphing calculator; review SimCalc MathWorlds software; how to use modeling instruction in reading, social studies, and civics; take three days in summer to develop lessons for one's classes and share them; specific modeling strategies in mathematics; strategies for low-level math students.

All said that it would be very valuable if they had an hour scheduled every other week to coordinate their math and science courses with colleagues in their school.

Technology desires: two would love to have a classroom set of TI-83+SE graphing calculators (one said that calculators are preferable to computers because graphing calculators force students to work, whereas students can be lazy if they're in a group that's working on a computer); the other two would love nine computers in their classroom.

Long-term outlook: Larry Dukerich wrote this to Rick Vanosdall, the science specialist in Mesa Public Schools, during his workshop in June 2002:" On the larger issue of meaningful change, I find it hard to be optimistic in the face of the obstacles we face. The history of the fragmented approach we have been using in the Mesa junior high program has shaped the way teachers view how instruction should take place. Most see this workshop as a source of good activities that they could plug in here or there to supplement or bolster what they already do. I had this vivid image of garage sale junkies picking through the pile of dishware for a piece here and there, not recognizing that a whole set was in front of them. It was with considerable difficulty that I managed to keep the look of incredulity off my face as I listened to ____ (who has been a real pillar in this workshop) tell me just how much she felt she was expected to do in 8th grade science. Force and motion, energy, circulatory system, astronomy and ecology in 18 weeks???? In her defense, she is gamely trying to "cover" the standards set forth for grades 6 - 8. It would require a major paradigm shift for teachers to consider the possibility of using these materials as a complete course. They universally express the view that they could be far more effective if they could focus on just a few topics and do them well, but just don't see that in the cards. However, expectations of performance on the current math test and future science test hangs like a sword of Damocles over their heads. I expressed the view that perhaps we don't need to worry so much about the tests; just focus on developing the skills, and the kids will do well enough. ..."

15

A second Modeling Workshop, identical in content to the 2002 one, was held in summer 2003 at Dobson High School, for 23 more Mesa participants. Apprentice leaders and mentors were chosen from the 2002 participants to establish continuity and enhance the reform. Support of principals was being enlisted by Rick Vanosdall and Sandra Nagy, the science and math specialists; however, that process ended when Rick Vanosdall left the district in 2003. His successor, a former teacher, did not have needed skills nor insight and commitment to the reform. Thus the junior high reform paused. As of fall 2008, 10 of the 40 Mesa teachers are gone. Another, Rosanne “Rosie” Magarelli, retired early due to injury, retrained by taking another physical science modeling workshop in 2008, and is mentoring new modelers in physical science and preparing to write model-based biology lessons for her community college courses, BIO 100 and BIO 181, and helping Jane write grants to hold Modeling Workshops in high school biology.

In GUHSD, Debi Plum later organized district-wide short Modeling Workshops for math teachers that were well attended, she told me. Then she moved back into the classroom; she now coordinates her math classes with the physics modeler in her school. The number of GUHSD physics and chemistry teachers who use Modeling Instruction has grown through the years.

The WESD district curriculum coordinator changed jobs, as did the GESD staff liaison. Neither had a background in science or math, as I recall. With no district level commitment, the reform died, at least collectively.

In Peoria, no further Modeling Workshops for middle school were held. The number of physics and chemistry teachers who use Modeling Instruction has grown through the years, and Gaylene Swenson, the successor to the prior science coordinator, actively supports district physics and chemistry teachers via a professional learning community.

An enthusiastic and committed district level science or math coordinator can have a powerful influence on junior high instruction (such as Rick Vanosdall and Debi Plum did), even under adverse circumstances such as a low budget. But when the person changes jobs, their work can easily die. Furthermore, knowledge of STEM at district level in elementary school districts is hit or miss. Bottom line: school district infrastructure for STEM teaching/learning reform is weak.

16