Using Formative and Summative Evaluation to Improve Field Trip

Programs

Kimberly M. Burtnyk

Amgen Center for Science Learning

California Science Center

Pre-/Post-visit classroom activities• Curriculum compatible• K8 • CA standards

Gallery worksheets (atypical)• “Chaperone Sheets”

Available free from our website

ThinkSCIENCE! PathwaysComponents

Help teachers implement curriculum-compatible field trips

Increase educational guidance for field trips

Increase interactions with staff

Help chaperones facilitate

Increase concept understanding through chaperone facilitation

ThinkSCIENCE! PathwaysGoals of the program

Formative Evaluation Spring 2002

Focus: Access, Use, Chaperone/student behavior

Summative Evaluation Timeline Spring 2003

Focus: Cognitive gains

ThinkSCIENCE! PathwaysEvaluation Plan

Recruiting• Groups with existing reservations

Questionnaires• Pathways usage issues• Field trip planning and behavior• Chaperone roles

Observations• Stopped/didn’t stop• Time spent--we want Pathways groups to stay longer• Panels Read, Discussions, Gestures

Formative Evaluation Method

Recruiting • 37 Schools recruited from March-June

Questionnaires• 82 Returned (37 Teacher 45 Chaperone)

Observations• 15 schools tagged for observation (8 PW, 7 Non-PW)• 72 Chaperones observed

Formative Evaluation Results Summary

Only 3 of 8 PW recruits used it

Reasons Technical difficulties• Materials looked too complicated and long• Confused about where to find them Unaware of the materials• Timing

Formative Evaluation Questionnaire Highlights

StoppedDid not

stopTotals

PW Users 13 4 17

Non-PW Users 22 33 55

Totals 35 37 72

Fisher’s exact p<0.012 Sig. at p<0.05

72 Chaperones observed

Formative Evaluation Observations Highlights

Stopped/Did not Stop at Capsules

Non-PW Chaperones PW ChaperonesN=55 N=17

0

10

20

30

40

50

60

70

80

Perce

nt of

Chap

erones

0 50 100 150 200 250 300 350

Time(s)

41

9

4 1 0 00

10

20

30

40

50

60

70

80

0 50 100 150 200 250 300 350

5

2

6

1 1 2

Time(s)

Mean t: 66 s Mean t: 138 s

Mann Whitney U p<0.002

Formative Evaluation Observations Highlights

Time spent at Capsules

QualityFactor

Pathways(n=13)

Non-Pathways

(n=22)Reading 20 incidents 9Discussions 12 12Gestures 18 29

Reading: Mann Whitney U: p<0.0095

Of the 35 chaperones who stopped...

Discussion: p<0.12

Gestures: p<0.64

Formative Evaluation Observations Highlights

Reading, Discussion, Gestures

Pathways usage was limited due to:• Access to computers/printers• Length and perceived complexity of the materials Communication gaps between school administrators

and teachers

BUT...it did produce some significant behavioral

changes in chaperones who used it

Formative Evaluation Conclusions

How Did Formative Evaluation Impact the Program?

In-class/On-site activities separated

Shortened• Map removed• One activity removed (Gallery incomplete)

More direct link created• quick

• easy

• obvious

Summative Evaluation Method

Recruiting• Talked directly to lead teacher• Materials sent to groups• In-class visit prior to field trip

Pre/Post Test

Questionnaires--dropped

Observations• Gestures dropped

Summative Evaluation Results Summary

298 Students Recruited• 217 had in-class visit

254 Pre- and Post-tests completed• 28 PW• 226 Non-PW

Observations• 26 chaperones observed ONLY 4 CHAPERONES USED PATHWAYS

Summative Evaluation Conclusions

Jury still out on cognitive impact of Pathways

• The numbers are too small• Will resume attempts next year

Pre/Post test shows impact of In-class Visit

• Micrometeoroid question Chi Square p<0.004 for those who had visit vs those who did not

1. Even with intensive intervention and support, teachers did not use PW

There were special circumstances in the A&S Gallery And timing, curriculum, age groups were an issue tooBUT: Are such materials worth spending $$ developing? What do teachers really want/need, if anything? We

don’t yet know.

2. Whether or not PW improves learning outcomes as the lit. suggests is still unknown

Warrants further testing and refinement of good and USEFUL field trip practices.

What did Formative and Summative Evaluation Teach us About Pathways?

3. We really need to learn more about teachers and chaperones

Can’t assume that teachers have access to net sources

Chaperones might be willing facilitators

4. We can apply statistics to tell us about the use and effectiveness of the program

Further refining statistical techniques Learning curve Exciting prospects

What did Formative and Summative Evaluation Teach us About Pathways?