Studying the Reporting and Processing of Practice

Edit Yerushalmi and Bat Sheva EylonWeizmann Institute of Science

Safed Seminar, July 2006

Orientation

• Focus:

the professional development program

• PD program directed at: experienced teachers, high school physics

• Methodology strand: proof of concept - small scale CPD

Well Known Challenge and Solution

Cooperative-customization workshops:

To change deep-rooted practice teachers need to take an active role, tailor innovations to specific context, try out, reflect on their practice, better in a community.

[Loucks-Horsley et al, 1989, Putnam & Borko 2000]

Challenge: Even when instructors believe in an instructional goal, wish to change and direct their practice towards it,

prior beliefs might conflict with it and distort the attempt

[VanDriel, 1997, Schoenfeld, 98, Henderson, 2004].

In cooperative-customization workshops:

Reflect…

• teachers report to peers on their new practices,

• teachers cooperatively process practice in a critical manner.

Difficulties

•“Teaching events are transient, and elusive.

•Documentation is not a common teacher practice,

•and it puts a heavy load on teachers”

• " the considerations underlying teachers’ instructional decisions are often entangled and implicit, not easily accessible

• teachers are expected to “know the right answer” not to experiment and possibly fail refrain from revealing difficulties they have encountered, and criticizing the practice and ideas of others

• teachers’ professional opportunities seldom involve cooperatively designing instruction not experienced in managing such a task.[Loucks-Horsley et. al. 1998, Grossman et. al. 2000]

b) in processing practice

a) in reporting practice [Ben Peretz, 2004]

Two Questions:

• Microanalysis of reporting and processing in the context of an effective “model” customization workshop

What standard of reporting and processing of practice can we expect teachers to perform in effective customization workshops?

What kind of mentoring facilitates teachers’ performing to this standard?

Methodological approach:

• Suggesting design features of the customization workshop underlying effective reporting and processing

• Comparison of reporting and processing in two settings: with and without these design features

Context: a “Model” Customization Workshop

A Challenging goal: Customization and implementation of instruction promoting reflectivity in problem solving in the physics classroom

Participants: 8 high school teachers

Schedule: 3 days summer workshop + yearlong bi-weekly meetings (~ 3 hours), on-site / on-line

Structure: “Flagman framework” designed to support cooperative learning

Goal achieved: Another comparative study (3 yearlong workshops)

Flagman Model Microanalysis of Reporting and Processing

Before cycle (classroom)

Teachers in turn (flagman) writes

documentation draft

Weekend (e-mail) Workshop facilitator provides preliminary feedback

Monday (forum) Flagman distributes documentation

Wednesday (forum)

Peers provide feedback

Thursday (forum) Flagman formulates questions for discussion

Thursday 10 p.m. (Chat)

1 h discussion of questions

After several cycles (F2F meeting)

Workshop facilitator provides chat summary: sub-questions and alternative directions suggested, teachers choose which to execute in F2F meeting

Microanalysis of Reporting and Processing

Categories - Combination of theory driven (High order) and grounded to focus on theory based difficulties with reporting and processing.

Analysis of the evolution in (1) grounded categories; (2) Mentor role

Reporting Processingobjectmedium format

mode and objectformat

High-order categories:

Data: one online cycle of the Flagman Framework .

Analysis:

Unit - statements - Each expresses a single idea related to a specific innovation (e.g. self-evaluation of an exam solution).

a. The class: 28 students, 12th grade, good grades

b. Goals: acquiring tools to check answersc. Concerns: Should I give multiple

choice or open problems?d. Activity plan: Simple circuits…self

evaluation of an exam solution using instructor solution

e. Student materials: they’ve got the exam problem and an evaluation worksheet

f. Results: The exam was easy for some. Even weak students performed well.

Flagman - Documentation - Draft

Object: teacher actions, students’ former achievements/performance Medium: recall of classroom events Format: hierarchical, Sparse Mode & object: goals & alternatives for instructor's actionsFormat: fragment

Pro

cess

ing

R

epo

rtin

g

Workshop facilitator – Preliminary feedback

did you construct the instructor solution in a systematic way you wanted the students to follow? If yes, how?How did you respond when students had difficulties? How did high and low achievers cope? Did you discuss the activity with the students? … What did they say?

Object: Interaction of flagman and students' actions, Differentiation in students’ performance Students' attitudes Medium: Materials distributed in class, Citations from class discussion

Format: Details Mode & object: Connecting instructional design to goalsFormat: Source of organization - flagman

Pro

cess

ing

R

epo

rtin

g

Flagman - Distributed documentation

The instructor solution was constructed in a systematic manner, according to the “PS steps”

Section # points Correct? explain

Steps in solving simple circuits problems 1. If there are several batteries connected in the circuit,

you have to replace them in an equivalent one.2. …3. …

The weak students viewed it as extra work!!!

Object: Interaction of flagman and students' actions, Students' attitudes Medium: Materials distributed in class Format: Details Mode & object: Connecting instructional design to goals, post mortem concerns regarding students attitudes P

roce

ssin

g

R

ep

ort

ing

Peers - Feedback

Keren: it is not always possible to interchange with an equivalent battery, and there you have to use Kirchoff laws

Merry: Would you conclude from class experience that the “PS steps” help primarily to the average students?

Critique, revision of

class materials

Summative evaluation

Differentiating goals

Rep

ort

ing

Pro

cess

ing

Questions (flagman) and discussion (all)

Ben: is it possible to show them?Merry: Because the outcomes are not immediateKeren: They reject because they don’t understand why it’s good for themFacilitator: Anna, what do you think on Ben’s question? …Facilitator: Ben, can you answer your question?Ben: No…I can talk with them, and show them their understanding improvedAnna: Do you think it is important?

I am concerned that the weak students don’t see the activity as worthwhile, while it is, What do you think?

Dissonance between

expectations and reality

AlternativesSpeculations Agreement Forcing the

point P

roce

ssin

gP

roce

ssin

g

Choosing alternative directions

Summary: Focus on the “weak” student:

Other questions

1. The weak students were harsh when evaluating themselves (which harmed their grade) do you have an idea why?

2.…

Possible directions: 1. …

2. Develop two level problem solving

strategy more and less detailed

Reorganization, Identification of possible

alternative actions related to goals,

hierarchical organizationPro

cess

ing

Evolution

Sparsereport

Flagman – Facilitator interaction:

Differentiation Integration

Initial practice

Advanced practice

Group interaction: Differentiation

Integration

Comprehensivereport Directing

design

Questions 1

What kind of mentoring facilitates teachers’ performing to this standard?

What standard of reporting and processing of practice can we expect teachers to perform in effective customization workshops?

Microanalysis of reporting and processing in the context of an effective “model” customization workshop

Questions 2

Suggesting design features of the customization workshop underlying effective reporting and processing

Comparison of reporting and processing in two settings: with and without these design features

Suggesting design features of the customization workshop underlying effective reporting and processing

Is the design essential?Comparison of online and onsite setting

online

Splitting the cooperative work into discrete steps Participation from home Publication of textual artifacts in an intimate circle Formal rules for interaction

Flagman model was implemented online and onsite,Onsite: 1.5 h, Bi-Weekly F2F meeting at the Weizmann

onsite

X

X

X

X

XComparing the frequency with which grounded categories of reporting and processing practice appeared in an online implementation and a

face-to-face implementation of a Flagman Framework cycle ,

in order to relate teachers’ reporting and processing to the settings’ respective design principles

highly structured cooperative work

Comparison Categories

teacher actions, students’ performance, recall of

classroom events, sparse, hierarchical, goals &

alternatives for instructor's actions

Differentiation in students’

performance, attitudes, Materials distributed in class, Citations, Details,

Source of organization -

flagman

Interaction of flagman and students' actions,

Connecting instructional design to goals, post

mortem concerns

Critique, revision of class

materials, Differentiating

goalDissonance between

expectations and realityExplanations, speculationsAgreement, Suggestions

Reorganization, Identification of possible

alternative actions related to goals,

hierarchical organization

• similar types of activities related to a different topic, kinematics,

• same length of transcript - 242 statements (representative sampling of reporting & processing ~ ½ F2F)

• similar flagmen: age, academic background, experience, suburban schools, school academic level.

Online-onsite similarities

Online - 152 Face-to-face – 117

Flagman's actions 73% 63%

Peers' actions 6% 2%

Students (achievements, performance, and attitudes)

20% 39%

Recall of classroom events 63% 97%

Materials distributed in class 35%

Citations from class discussion 1%

Students' work 4% 3%

Ob

ject

Med

ium

How teachers REPORTED practice? Distribution of grounded categories

Online: detailed, organized document, structure determined by flagman.

F2F: sparse, fragmented, organization planned by the flagman distorted by questions and remarks of peersfo

rmat

more categories, materials distributed in

class received more attention.

Splitting the cooperative work into discrete steps, & publication of textual artifacts in an intimate circle contribute to elevating reporting standards.

Indeed, the participants asked to transfer meetings to online setting, they expressed that it better supports them:

“Writing the documentation made me make order in my mind”

“I felt as if I entered another teacher classroom, and could borrow a lot from him”

In accord with suggested design features

Flagman Peer Leader Flagman Peer Leader

Relating instructional design to goals

43% 28% 29% 18% 55% 27%

Suggesting alternatives for instructors' actions

19% 56% 25% 12% 66% 22%

Revising or praising materials distributed in class

94% 6%

Assessing students’ work 30% 22% 48%

Speculating on students’ mental world

13% 62% 25% 40% 23% 37%

Identifying post mortem concerns

80% 20% 6% 81% 13%

Online Face-to-face

How teachers PROCESSED practice? Distribution of grounded categories

relative share of flagman higher

relative share of flagman higher

relative share of flagman higher

relative share of

peers higher

relative share of

peers higher

relative share of

peers higher

How teachers PROCESSED practice? Distribution of grounded categories

Online Onsite

Online-144

F2F - 147

relating instructional design to goals 40% 38%

Suggesting alternatives for instructor's actions 11% 28%

Revising or praising materials distributed in class

12% 1%

Assessing students’ work 3% 16%

Speculations on students mental world 10% 24%

Identification of post mortem concerns 8% 11%

Login from home allows more thorough processing by teachers on their peers’ classroom materials.

Teachers’ commitment to follow formal rules mandates the flagman more control and allows a process where the teacher on stage goes through a reflective process, that serves all of his peers for reflection on their own experiences.

In accord with suggested design features

What kind of mentoring facilitates teachers’ performing to this standard?

Questions 2

Suggesting design features of the customization workshop underlying effective reporting and processing

Comparing the frequency with which grounded categories of reporting and processing practice appeared in an online implementation and a face-to-face implementation of a Flagman Framework cycle

Microanalysis - theory driven and grounded categories

Reporting & processing of practice evolved through differentiation and integration processes facilitated by mentoring

Summary

Comparison of the frequency categories appeared in online and onsite (with and without design features)

Design features imposing highly structured cooperative work promotes better reporting and processing of practice

Other considerations get in the way

A Challenging Goal

Instructors believe instruction should promote reflection

Students learn by reflectively attempting to work out problems, they appreciate instructional components directed to promote reflection (group work, real world problems, outline in sample solutions) Yerushalmi, Henderson, Heller, Heller, Kuo [in preperation]

Yet, they implement those sparsely.

“In the ideal world you would use problem solutions and grading of them far more for teaching than for stratifying the student population, but I think the real situation here, and probably it’s typical, is that you just don’t have the time for it”.

Comparative Study - 3 Yearlong Workshops ,

• 3 workshops, differing in mentoring approach, • Results of macro-analysis: the model workshop is better:

– Teachers performed and initiated main stages in a systematic, iterative, and cumulative customization process.

Main stages: • Analysis of pedagogical challenges and possible solutions• Planning specific activities suited to teachers classrooms• Constructing and implementing lesson plan & classroom materials• Evaluation and revision of implemented instruction

– Teachers developed variety of instructional strategies and materials reflecting changes in participants’ understanding.

{Yerushalmi & Eylon, 2004, Yerushalmi & Eylon, 2001

Guideline 2 for Supporting Cooperative Learning: Explicate intermediate expert steps

Customization stages explicit in flagman model:

Cognitive Apprenticeship

Plan Summary - bringing up and deciding on alternative directions

Construct Documentation - description of lesson plan & classroom materials

Evaluate Questions - raising concerns regarding class experience

Analyze Discussion - suggesting what underlies those concerns

Guideline 3 for Supporting Cooperative Learning: Incentives for true cooperation

Social interdependence theory [Johnson & Johnson, 2004]

Positive interdependence

Individual and group accountability

Teachers depend on each other to refine instruction: documentation provides anchor, feedback points out pro’s and cons…

Each group member held responsible for specific duties, Outcomes of FLAGMAN cycle required for next stage

Distribution of Reporting and Processing Categories

Flagman Peer Leader Flagman Peer Leader

Reporting 65% 18% 17% 72% 10% 18% Processing 36% 44% 20% 27% 45% 28%

Online Face-to-face

Total 103 statements, 242 lines 153 statements, 242 lines

Reporting 49 statements, 152 lines 61 statements, 117 lines

Processing 69 statements, 144 lines 103 statements, 147 lines