Workshop: Application of New Data Tools in Evaluation of Math and Science Initiatives2007 NSF MSP Pre-Conference

Council of Chief State School Officers

Workshop Sessions

John Smithson: Use of Surveys of Enacted Curriculum to measure changeAndy Porter: Alignment Analysis as Teacher VariableNina de las Alas: State Data SystemsPete Goldschmidt: Evaluating Initiatives with Longitudinal Data

Evaluation of Professional Development in Math and Science:

What we have learned recently –2 StudiesMeasuring Quality of PDMeasuring Effects of PD on Instruction

What we need to provide to decision-makersMeasuring Change in student achievement due to PD

Key Questions about M/S Professional Development and Evaluation

AIR/Porter National Study (99-01) findings: Most programs too short, not content focusedPrograms not based on what we know from researchSome PD does change practice

How do you measure effects of PD on change in teaching – comparable, reliable, broad scale, quantitative?How can experimental designs be implemented?Can we measure effects of PD on student achievement?

Improving Evaluation of Professional Development in Math & Science Ed. NSF REC Grant (2005 to 07)

Council of Chief State School Officers

1) Analysis of the Quality of Professional Development Programs in M/S: Cross-State Study, Dec. 2006

Developed rubric to assess program quality (5 research criteria) Panels reviewed sample of 27 programs/15 states (voluntary, mostly MSP TIIB or NSF)

Programs are more content focused Active learning in most programsCoherence with standards (school curriculum?)Less school-based, but do use follow-upEvaluation designs broad – use of Tools (see list)Measuring PD effects vs. Program evaluation Problem of feedback/ application

IMPDE Rubric

Content Focus (N=25 PD Programs)

0

2

4

6

8

10

12

14

# of

Pro

gram

s

High Scores (4s or 5s)Medium Scores (3s)Low Scores (1s or 2s)No Score

Learn subject content or concepts

Learnpedagogy content knowledge

Addresses identified content needs

Active Learning (N = 25 Program s)

0

5

10

15

20

25

# of

Pro

gram

s

Demonstrateinstruction or

lead discussion

Plan lessons Coaching or

mentoring

Develop or review

assessments

Observe other teachers

Engage in a learning network

With teachers from same dept./content area

With teachers from same grade level or span

With teachers from same school

Collective Participation Among Teachers(N=25 PD Programs)

0

5

10

15

20

25#

of P

rogr

ams

YesNo Insufficient Information

Evaluation Design (N = 23 Programs)

0

5

10

15

20

25

Quality of PDActivities

TeacherContent

Knowledge

InstructionalPractices orCurriculum

StudentAchievement

OtherOutcomes

EvaluationDesign Total

# of

Pro

gram

s

Student Achievement in Evaluation

As of December 2006, for 25 PD Initiatives

20 Eval. Reports received (2004 to 2006)

12 reported Student Achievement results for Schools or Teachers

6 reported SA results for Treatment teachers vs. Control

Of 20 reports, 12 dated since January ‘06

IMPDE Website

http://www.ccsso.org/projects/improving_evaluation_of_professional_development

Website : Evaluation Tools

Evaluation Tools for Professional Development (excel based)Professional Development ActivitiesTeacher Knowledge and SkillsChange in InstructionAnalyzing Effects on Student AchievementEvaluation Design Assistance

Longitudinal Study Effects of MSP Professional Development on Improving Math & Science InstructionMSP-RETA Project

Council of Chief State School Officers American Institutes for Research Wisconsin Center for Educational Research

Project Staff Leadership

PIRolf Blank, CCSSO, rolfb@ccsso.orghttp://www.ccsso.org/projects/Surveys_of_Enacted_Curriculum/Current_Projects/

Co-PIsBea Birman & Mike Garet (AIR)Andy Porter, Vanderbilt UniversityJohn Smithson (WCER)Kwang Suk Yoon (AIR)

Study Questions / Objectives (‘02-05)

Effects of MSP PD vs. Other on Improving Instruction ?

Use of Hi Quality Survey tools to Evaluate PD Quality?

How can Survey tools be used by MSPs, Other users ?

MSP PD Study Sites

Brockport/Rochester – SUNYCleveland Municipal SDCorpus Christi – AIMS - TX A&MEl Paso – UTEP

Survey responses – 430 Year 1 (treatment, control)180 Year 3 (size by group OK)

MSP Study Logic ModelDuring MSP Program

Professional DevelopmentExperiences

TypeDuration

Collective ParticipationActive Learning

CoherenceContent Focus

Before MSP ProgramInstructional PracticeContent, Activities, &

Strategies

Controlling for

Teacher CharacteristicsBackground VariablesTarget Class Students

DiversityProgram Characteristics

Emphasis

Year 1 Year 2 Year 3Year 0

Professional Development Activity

Log (PDAL)

Survey of Enacted Curriculum (SEC)

Wave 1

Professional DevelopmentExperiences

After MSP ProgramInstructional Practices

Analyze alignment change–Instruction w/ Stands.

Survey of Enacted Curriculum (SEC)

Wave 2

Professional DevelopmentExperiences

Findings - Longitudinal Analysis

MSP Teachers (Treat.) vs. Non-M/S (Comp.)

MSP Prof Development Quality

More PD time, Greater focus on contentMath teachers: More content in PD than comp. M&S : Increase focus on standards & instructionTeacher reports on preparation

Math Teachers better prepared for challenging content M&S better Prepared to teach diverse students

Findings - Longitudinal Analysis

MSP Teachers (Treatment) vs. Non - M/S (Comp.)

Instructional practices--Math: more emph on understanding, analysis--Science: more active learning, analysis of infor.

• Content Alignment: --PD Quality related to greater content alignment --More Coherence of PD related to focus on Math content

(www.seconline.org)

Rationale for SEC

AssessmentStandards

Curriculum

• School & Class Description• Instructional Activities

• General• Problem Solving Activities

• Pairs & Small Group Work• Use of Hands-on Materials• Use of Calculators/Computers & other Ed. Tech.

• Assessment Use• Instructional Influences

• Instructional Readiness• Teacher Opinions• Professional Development

• Types• Content Focus• Active Learning• Collegial Participation• Coherence• Time Span

• Teacher Characteristics• Instructional Content

Survey Sections

English LanguageArts & Reading

The SEC Data-set

SUMMARY MEASURES

Content Practice Climate Prof. Dev./ / /

Science

InstructionalContent

Instr.Activities

Tchr.Char./Opinions/Beliefs

ProfessionalDevelopmentMathematics

Results: Tools Developed, Tested

Online, Web-based Teacher SurveySurveys of Enacted Curriculum (SEC)

Instructional content and Practices by courseAnnual PD activities, scales Measure instruction change over time Comparable data across sites, schoolsCharts, graphs, maps for use of data

www.SECsurvey.org , www.SEConline.org

Results: Tools Developed, Tested

Professional Development Activity LogWeekly teacher log of PD activities;Quality scalesAnalysis of effects of type of activity or teacher by all activities

www.pdal.net

Applications –SEC Data

Alignment analysis --instruction, standards, assessments (annual data)Date-driven instructional improvement in schools (combine curric with achieve data)Needs assessment/ EvaluationIndicators – monitoring change over time

Applications of PD Log online

Analysis of PD Quality —research based criteriaExamine how PD relates to teachers’instructionTrack teachers’ PD over timeIdentify Teachers’ PD needsTarget PD to improve instruction

Surveys of Enacted CurriculumSurveys of Enacted Curriculum

A neutral content grid

with cognitive demand

The intendedcurriculum: State content standards—What students should learn

The enactedcurriculum: What teachers teach

The learnedcurriculum: Student outcomes based on school learning

The assessedcurriculum: State (and other) assessments—tested learning

Uses a multi-dimensional language fordescribing instructional content

SEC Instructional Content

Topics

by Cognitive Demand(Expectations for Student Learning)

Categories of Cognitive Demand

TopicsMemorize Perform

ProceduresCommunicateUnderstanding

Solve Non-Routine

Problems

ConjectureGeneralize

ProveMultiple

StepEquations

Inequalities

LiteralEquations

Lines /Slope andIntercept

Operationson

Polynomials

QuadraticEquations

Content Matrix – Math / Algebra

PD SCALES from SEC Data

Content Practice ClimateProf. Dev. / //

Variable Summary Measure DescriptionPDhrsPDfreqPDactivePDcoherePDcollPDcntPDdataPDstndPDstlrn

# of PD Hours reportedFrequency of PD Activities

PD that involved active engagement of teachers

PD that is part of a coherent plan & goals

Collective participation from school/department

PD with focus on subject matter content

PD with focus on Student Data

PD with focus on Standards & Instruction

PD with focus on Student Learning

SEC Reports: Instructional Content

Fine Grain Maps

Topics

by

Cognitive Demand

SEC Reports: Instructional Content

Coarse Grain Maps

Content Areas

by

Cognitive Demand

Key Question -- SEC ToolsResearch into Practice

How can Educators obtain reliable, valid data to determine Alignment of instruction with required standards and assessments?

And

Teachers could compare what and how they teach in a specific content area with others in their school, district, state or nation?

Educators could use anonymous teacher data to start a powerful school discussion about what the teacher needs are?

What if…

You could use data on instructional quality and content to guide professional development?

You could have consistency across grade levels?

You could know how well aligned your state standards were to the state assessment?

Online Survey Administration

Approximately 60-90 minutes to completeMay be completed in multiple sittingsData is saved as each section is completed

Administrative Functions:Administration Set-upReview Registrants, Completion RatesAdministrator Report Generator

Conducting Inquiry Using SEC Data

Forms of InquiryCollaborative or Evaluative

Teacher EnrichmentSchool ImprovementProfessional Lrng. Comm.

Program EvaluationIndicator ReportingProgram Management

Conjecture/ Generalize/ ProveRecite basic mathematics factsRecall mathematics terms & definitions

Write formal or informal proofsRecognize, generate or create patterns

Perform ProceduresUse numbers to count, order, denote

Follow procedures/instructions

Reason inductively or deductivelyOrganize or display dataRead or produce graphs and tablesExecute geometric constructions

Demonstrate Understanding Solve Non-routine Problems/ of Mathematical Ideas Make Connections

Communicate mathematical ideas

Analyze data, recognize patternsExplain findings and results from data analysis strategies

Show or explain relationships between models, diagrams, and/or other representations

Apply mathematics in contexts outside of mathematics

Use representations to model mathematical ideas

Synthesize content and ideas from several sources

Develop/explain relationships between concepts

Expectations for Students in Mathematics

Determine the truth of a mathematical pattern or proposition

Apply and adapt a variety of appropriate strategies to solve non-routine problems

Find a mathematical rule to generate a pattern or number sequence

Recall formulas and computational procedures

Solve equations/formulas/routine word problems

Memorize Facts/ Definitions/ Formulas

Do computational procedures or algorithms

Make and investigate mathematical conjecturesIdentify faulty arguments or misrepresentations of data

SEC Content TopicsMath, Science, ELA

The SEC provides a neutral, research-based language to describe content of English language arts, mathematics, and science.

NCLB anyone?

Time on Topic K-8 Mathematics Topics

<none> 6 Data Analysis / Prob. / StatisticsMemorize Facts/

Definitions/ Formulas

Perform Procedures

Demonstrate Understanding of

Mathematical Ideas

601 Bar graph, histogram

602 Pie charts, circle graphs

603 Pictographs

604 Line graphs

605 Stem and Leaf plots

606 Scatter plots

607 Box plots

608 Mean, median, mode

609 Line of best fit

610 Quartiles, percentiles

611 Sampling, Sample spaces

612 Simple probability

613 Compound probability

614 Combinations and permutations

615 Summarize data in a table or graph

Expectations for Students in M

Categories of Cognitive Demand

Topics Memorize

facts, defintions

Conduct Invest/ Perf Procedures

Communicate Understanding of sci concepts

Analzye information

Apply concepts/ Connnect

Nature of Science

Meas/Calc. in Science

Life Science

Physical Science

Earth Science

Chem/Biol/ Physics

(HS)

Science content matrix

Time on Topic Middle School Science

<none> 1 Nature of Science

Memorize Facts/

Definitions/ Formulas

101

Scientific Habits of Mind (e.g. reasoning, evidence-based conclusions, skepticism)

102Scientific Method (e.g., observation, experimentation, analysis, theory development and reporting)

103 Issues of Diversity, Culture, Ethnicity, Race, Gender in Science

104 History of Scientific Innovations

105 Ethical Issues in Science <none> 2 Science and Technology

201 Design a Solution or Product, Implement a Design

202 Relationship Between Scientific Inquiry & Technological Design

203 Technological Benefits, Trade-offs & Consequences

MemorizePerform

CommunicateConjecture

Connect0

0.020.040.060.08

0.1

Operations

Number Sense

Measurement

Geometric Concepts

Algebraic Concepts

Data Analysis

Instructional Technology

MemorizePerform

CommunicateConjecture

Connect

State J Grade 8Mathematics Instruction

Content Maps

Alignment Content Analysis

Conducted with each State’s standards, assesments – middle grades

Alignment Analyses for Planning Purposes

Mapping Curriculum Materials

25 Watch the teacher demonstrate how to do a procedure or solve a problem.

26 Read about mathematics in books, magazines, or articles (not textbooks).

27 Take notes from lectures or the textbook.28 Complete computational exercises or procedures from a

textbook or a worksheet.29 Present or demonstrates solutions to a math problem to the

whole class.30 Use manipulatives (for example, geometric shapes or algebraic

tiles), measurement instruments (for example, rulers or protractors), and data collection devices (for example, surveys or probes).

31 Work individually on mathematics exercises, problems, investigations, or tasks.

32 Work in pairs or small groups on math exercises, problems, investigations, or tasks.

33 Do a mathematics activity with the class outside the l34 Use computers, calculators, or other technology to learn

mathematics.35 Maintain and reflect on a mathematics portfolio of their own

work.36 Take a quiz or test.

How much of the total mathematics instructional time do students in the target class:

Non

e

10%

or<

11-2

5%

26-5

0%

51-7

5%

>75%

37 Make educated guesses, predictions, or hypotheses.

38 Follow step-by-step directions.

39 Use science equipment or measuring tools.

40 Collect data.

41 Change a variable in an experiment to test a hypothesis.

42 Organize and display information in tables or graphs.

43 Analyze and interpret science data.

44 Design their own investigation or experiment to solve a scientific question.

45 Make observations/classifications.

When students in the target class are engaged in laboratory activities, investigations, or experiments as part of science instruction, how much time do they: 51

-75%

> 75

%

26-5

0%

Non

e

10%

or <

11-2

5%

CLASSROOM INSTRUCTIONAL PREPARATION

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

Teach mathematics to students from a variety of cultural backgroundsTeach mathematics to students who have limited English proficiency

Teach students who have a learning disability which impacts mathematics learning

Encourage participation of females in mathematics

Teach students with physical disabilities

Help students document and evaluate their own mathematics workTeach classes for students with diverse abilities

Use a variety of assessment strategies (including objective and open-ended formats)

Teach problem solving strategies

Select and/or adapt instructional materials to implement your written curriculum

Teach mathematics at your assigned level

Use/manage cooperative learning groups in mathematics

Integrate mathematics with other subjects

Provide mathematics instruction that meets mathematics standards (district, state, or national)

Teach mathematics with the use of manipulative materials, such as counting blocks, geometric shapes, and so on

For items 81-97, please indicate how well prepared you are to:Not Well Prepared

Somewhat Prepared

Well Prepared

Very Well Prepared

Show PD Quality items, scales--PDF

Completed Surveys: Yr 1, Yr 3

Year 1 Year 3 Year 1 Year 3Total # Surveys: 227 97 208 77Included for Analysis 209 97 180 77Brockport MSP

Treatment 28 22 14 8Control 19 9 17 3Total 52 31 31 11

Cleveland MSPTreatment 51 28 59 37Control 27 4 40 7Total 84 32 99 44

Corpus Christi MSPTreatment 35 17 2 0Control 15 3 26 10Total 53 20 28 10

El Paso MSPTreatment 19 12 13 6Control 15 2 9 6Total 38 14 22 12

AllTreatment 133 79 88 51Control 76 18 92 26Total 209 97 180 77

MSP-PD Survey CountsScience SurveysMathematics Surveys

Frequency of PD Activities Year 3

0.00 5.00

10.00 15.00

20.00 25.00

30.00

WrkFrq InstFrq CrsFrq PdFrq

Math. Comp. Math MSP Sci. Comp. Sci. MSP

Hours Engaged in PD Activities Year 3

0.00

20.00

40.00

60.00

80.00

100.00

120.00

140.00

WrkHrs InstHrs CrsHrs PDHrs

Math Comp. Math MSP Sci. Comp. Sci. MSP

MSP Mathematics PD Characteristics Yr. 3

0.00

1.00

2.00

3.00

Pdactiv Pdcoher Pdcoll PDcnt Pddata pdsi pdsl

Comparison MSP

PD Characteristics

0.00

1.00

2.00

3.00

Active PD Coherent PD

Mth. Comp. Yr.3Mth. Trtd. Yr.3

Sci. Comp. Yr.3Sci. Trtd. Yr.3

Collective Participation in PD

0.00

1.00

2.00

Mth. Comp. Yr.3Mth. Trtd. Yr.3

Sci. Comp. Yr.3Sci. Trtd. Yr.3

PD Focus

0.00

1.00

2.00

3.00

Content Data Stnd/Instr Stud. Lrng.

Mth. Comp. Yr.3Mth. Trtd. Yr.3

Sci. Comp. Yr.3Sci. Trtd. Yr.3

MSP Science PD Characteristics Yr. 3

0.00

1.00

2.00

3.00

Pdactiv Pdcoher Pdcoll PDcnt PDdata Pdsi PDsl

Comparison MSP Yr.3

Change in MathematicsTeacher Opinions/Beliefs (Yr.1 to Yr.3)

-1.00

-0.50

0.00

0.50

1.00

Influence of Standards

ProfessionalCollegiality

Readiness for Challenging Content

Readiness forDiverse Populaitons

Comparison Change MSP Tchr. Change

Student Mathematics Activities Yr.3

0.00

0.10

0.20

0.30

0.40

Perform Procedures

Demonstrate Understanding

Analyze Info.

Active Learning

MakeConnections

Proportion of Instr. Time

Comparison Yr.3 Treatment Yr.3

Cognitive Demand of Instructional Content

0.000.100.200.300.40

MemorizeProcedures

Demonstrate

Conjecture

Solve non-ro

utineMemorize

ProceduresCommunica

teAnalyz

eApply

Year 1 Year 3

Mathematics Science

MathematicsStudent Performance Expectations

By District

LegendMean

-1 StD +1 StD

LegendMean

-1 StD +1 StD

Perform Procedures

Memorize, Recall

Demonstrate Understanding

Conjecture, Generalize, Prove

Make Connections

0 10 20 30 40% 0 10 20 30 40%

Year 1 Year 3MSP-PD Study

District 1 (23)District 2 (28)District 3 (19)District 4 (13)

District 1 (23)District 2 (28)District 3 (19)District 4 (13)

Mathematics Content Coverage

By District

LegendMean

-1 StD +1 StD

LegendMean

-1 StD +1 StD

Measurement

Nbr. Sense, Properties, Relationships

Algebraic Concepts

Geometric Concepts

Instructional Technology

Operations

Data Analysis, Probability, Statistics

YEAR 1 YEAR 3MSP-PD Study

District 1 (7)District 2 (23)District 3 (19)District 4 (13)

District 1 (7)District 2 (23)District 3 (19)District 4 (13)

PD Quality & Content Alignment

Year 3 Alignment to Test

Year 3 Alignment to Standard

Coherent PD yr3 0.21Sig. (2-tailed) 0.049

N 88PD Cnt. Focus yr3 0.37

Sig. (2-tailed) 0.000N 86

PD Data Focus yr3 0.29 0.36Sig. (2-tailed) 0 0.001

N 92 86PD Stnd/Instr. yr3 0.24 0.40

Sig. (2-tailed) 0.022 0.000N 92 86

Pearson Correlation PD Quality to Alignment

Science Alignment to Target Standard

0.00

0.10

0.20

0.30

Year 1 Year 3

Comp MSP

Mathematics Alignment to Target Standard

0.00

0.10

0.20

0.30

Year 1 Year 3

Comp MSP

MSP RETA – Reports on line

www.MSPnet.org

www.SECsurvey.org/projects

www.SEConline.org (by MSP data)

Instruction

Standards

Assessment ProfessionalDevelopment

Standards-based Alignment among Instruction, Assessment,

and Professional Development