OSPI Science Assessment UpdateWERA, December 2010

The State of Material on 2011 Science MSP

Science Assessment OfficeElaine Wood—Secondary Specialist

elaine.wood@k12.wa.us

Kara Monroe—Elementary Specialistkara.monroe@k12.wa.us

Troi Williams—Administrative Assistanttroi.williams@k12.wa.us

Agenda2011 Measurements of Student

ProgressStandard SettingResources availableLessons learned from scoring

2011 Measurements of Student Progress

•Grades 5 and 8•First assessment of the K-12 Science Learning Standards (adopted June 2009)

Big Ideas in 2009 Science Standards

5

EALRs 1-3Crosscutting Concepts and

Abilities

EALR 4 Domains of Science

EALR 1 Systems Physical Science

…is a way of thinking that makes it possible to analyze and understand complex phenomena.

Force and MotionMatter: Properties and ChangeEnergy: Transfer, Transformation and Conservation

EALR 2 Inquiry Earth and Space Science

…is a process of asking and answering questions about the natural world that forms the bedrock of science.

Earth and SpaceEarth Systems, Structures and ProcessesEarth History

EALR 3 Application Life Science

…is about the interaction between science and technology, and how both can help solve real-world problems.

Structures & Functions of Living SystemsEcosystemsBiological Evolution

Operational ScoringOperational ScoringOperational

Range FindingCommittee

Operational Range Finding

Committee

Items available for Operational TestingItems available for

Operational TestingData ReviewCommittee

Data ReviewCommittee Pilot ScoringPilot Scoring

Pilot Range Finding

Committee

Pilot Range Finding

Committee

Scenario & Item Pilot Testing

Scenario & Item Pilot Testing

Science Assessment Development Cycle

Science Assessment Development Cycle

Content Review & Bias/Sensitivity

Committees

Content Review & Bias/Sensitivity

Committees

Scenario & Items reviewed for content by

OSPI/contractor

Scenario & Items reviewed for content by

OSPI/contractor

Scenario & Item Writing Committee

Scenario & Item Writing Committee

Develop Test and Item Specifications

Develop Test and Item Specifications

Scenarios and ItemsScenarios provide context for sets of

items.Systems, Inquiry, and Application Physical, Earth/Space, and Life Science

The items associated with a scenario cover a range of standards & range of cognitive complexity.

Items not connected to a scenario are called “Stand Alone” items, which is clearly marked for students.

From WASL to Measurements of Student Progress

Grade 5 2009WASL

2010MSP

2011MSP

Multiple Choice 21 26 22-25

Completion NA pilot only 1-4

Short Answer 8 4 4

Extended Response 0 0 0

% Points from MC/CP 57% 76% 76%

Embedded pilot items makes the total number of items 35.

From WASL to Measurements of Student Progress

Grade 8 2009WASL

2010MSP

2011MSP

Multiple Choice 28 30 26-29

Completion NA pilot only 1-4

Short Answer 7 5 5

Extended Response 2 0 0

% Points from MC/CP 56% 75% 75%

Embedded pilot items makes the total number of items 40.

Test Specifications Grade 5 & 8The test map designates the following proportions:  At least 20% Systems (with connected EALR 4

content)30% Inquiry 20% Application (which will apply EALR 4 content)At most 30% EALR 4 domain knowledge that is not

associated with a cross-cutting concepts or abilities*

*EALR 4 content may be assessed in questions that only assess EALR 4 or in questions that cut across both EALR 4 and a cross-cutting concept or ability (Systems, Inquiry, Application).

More on Science MSP

Test LengthGrade 5: Single session 75-90 minutes*Grade 8: Single session 90-110 minutes**additional time is needed for passing out materials and reading directions

New Item Type Completion: worth 1 score point

Test Window May 2 to May 19 for paper/pencil testsMay 2 to June 3 for online tests

Online Testing Voluntary in grades 5 and 8 in 2011Goal: 25%

Standard Setting: Summer 2011

Following the process used by Mathematics in grades 3-8 last year.

Science Standard Setting Plan

Date EventJune 2009 New Science standards approvedMay 2010 Spring 2010 MSP; New items pilotedAug 2010 Test Map meeting

Sept 2010 Performance Level Descriptors (PLDs) developedOct 2010 Spring 2011 tests built

Science Standard Setting Plan

Date EventJune 2009 New Science standards approvedMay 2010 Spring 2010 MSP; New items pilotedAug 2010 Test Map meeting

Sept 2010 Performance Level Descriptors (PLDs) developedOct 2010 Spring 2011 tests built

Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in Contrasting Groups Study (CGS)

Mar/Apr 2011 NTAC and SBE formally approve standard setting planApril 2011 Teachers participate in CGS by providing ratings of studentsMay 2011 Spring 2011 MSP administered

Science Standard Setting Plan

Date EventJune 2009 New Science standards approvedMay 2010 Spring 2010 MSP; New items pilotedAug 2010 Test Map meeting

Sept 2010 Performance Level Descriptors (PLDs) developedOct 2010 Spring 2011 tests built

Feb/Mar 2011 Teachers from across state trained on how to use PLDs to participate in Contrasting Groups Study (CGS)

Mar/Apr 2011 NTAC and SBE formally approve standard setting planApril 2011 Teachers participate in CGS by providing ratings of studentsMay 2011 Spring 2011 MSP administeredAug 2011 Standard Setting events:

--Grade level panels (use “Bookmark process”, start with teacher ratings)

--“Policy panel” recommendations--NTAC certifies the planned process was followed

Aug 2011 SBE Sets Achievement Standards (Reviews all recommendations)

“Contrasting Groups” method affords broader involvement in standard settingEveryone in the state who teaches science in grades 5 and 8 is invited to help set the performance standards.Performance Level Descriptors (PLDs) are developed for “Basic” and “Proficient”Web-based training on PLDs is provided to participating teachersIn April, participating teachers rate each of their students as being “Basic or below” or “At or above Proficient” Two distributions of scores are produced: Scores obtained by students rated as “Basic”, and scores for “Proficient” Where the two distributions merge represents the separation point between “Basic” and “Proficient.”

Students rated as “Basic or below” using criteria in PLD for Basic

Students judged to be at or below "Basic"

Points on MSP (simulated)0 2 10 6 84 12 14 16 18 20 22 23 24 26 30 3228 34 40 36 38

Raw score points

Number of studen

ts

Students rated as “Proficient or above” using criteria in PLD for Proficient

Students judged to be "Proficient" or above

Points on MSP (simulated)0 2 10 6 84 12 14 16 18 20 22 23 24 26 30 3228 34 40 36 38

Raw score points

Number of studen

ts

Students judged to be at or below "Basic"

Students judged to be "Proficient" or above

Points on MSP (simulated)0 2 10 6 84 12 14 16 18 20 22 23 24 26 30 3228 34 40 36 38

Intersection indicates a region for where “Basic” separates from “Proficient”

Raw score points

Number of studen

ts

PLD trainingAvailable via Moodle, February 1, 2011Learn to use evidence to evaluate 5th or 8th grade

students’ proficiency (other grade levels can participate)

Using Moodle websiteShort videos of presentationsAll documents available for downloadCan accomplish all at once, or in multiple sittings

Free clock hours (3)Encourage groups of teachers to participate

togetherTo sign up, go to link and create an account

http://moodle.ospi.k12.wa.us/course/view.php?id=19

Contrasting Groups StudyAvailable via EDS, April 11-29, 2011After completing the PLD Training, 5th or

8th grade teachers can predict their students’ performance on the Science MSP

5th and 8th grade science teachers can gain access to the EDS application by contacting their District Assessment Coordinator

Resources Available

On the OSPI website:www.k12.wa.us/Science/

OSPI Science Assessment Webpagewww.k12.wa.us

www.k12.wa.us/science

www.k12.wa.us/Science/Assessments.aspxSelect Science

Assessment from

Assessment Drop-

down

Select Assessm

ents

Science Assessment ResourcesTest and Item Specifications—more

information about:Types of scenariosTest mapItem specificationsVocabulary listsCognitive complexity

www.k12.wa.us/Science/Assessments.aspxTest and Item Specs

www.k12.wa.us/Science/TestItemSpec.aspx

Select Test and

Item Specificati

ons

Select a

Grade

Test and Item Specswww.k12.wa.us/Science/TestItemSpec.aspx

Pages 5-14

Page 22

Science Assessment ResourcesUpdates for 2011—each grade level

contains:General information on the 2011 testSample items for classroom useScoring information, including rubrics and

sample student responses

Updates for 2011www.k12.wa.us/Science/Assessments.aspx

http://www.k12.wa.us/Science/EducatorResources.aspx

Select a

Grade

Select Educat

or Resourc

es

Science Assessment ResourcesOnline Testing Website:

http://wa-online.caltesting.org/ Demo

Flash playerTutorial

CAL testing program must be installedPractice items, same as Update for 2011

documentsBoth tools help students become familiar

with testing platform, and tests the computer system

CAL online testinghttp://wa-online.caltesting.org

Select Downloads or Demos

Stand-Alone item from 8th grade tutorial

CAL online testingShort answer item from 5th grade tutorial

CAL online testingShort answer item from 5th grade tutorial

Same item scrolled down to show symbols button and symbols tool pop-up.

Science Assessment ResourcesTeacher Resource tool—tests based on

previous standards:Types of scenariosGeneric questionsItem specifications (previous standards)Cognitive complexityState-wide statistics

Teacher Resource Toolwww.k12.wa.us/Science/Assessments.aspx

www.k12.wa.us/TeacherResourceTool2009-10/default.aspx

Select a Grade

and a Year

Select Scienc

e

Select Teacher

Resource Tool

Teacher Resource Tool

Type of scenario

Generic question,

rolls over to item

specification

Percent correct or

mean score

Science Assessment ResourcesPowerful Classroom Assessments (PCAs)

—currently linked to WASL released scenarios and items:Sample itemsScoring information, including rubrics

New PCAs will be developed from future released items

Powerful Classroom Assessmentswww.k12.wa.us/Science/Assessments.aspx

http://www.k12.wa.us/Science/EducatorResources.aspx

Select PCA

Select Educat

or Resourc

es

Select grade level

http://www.k12.wa.us/Science/PCAs/default.aspx

Lessons Learned from Scoring

Transition to new standards

Grade 5 Grade 8Terms :

“system” and “subsystem”

“inputs” and “outputs”

Human needs leading to inventions

Weight as a measurement of the force of gravity

Terms “closed” and

“open” systems“balanced” and

“unbalanced” forcesRoles of engineers

and scientists in technological advances

Common misconceptions (gr. 5)When the results don’t match the prediction,

students struggle with how to report honestlyStudents think the mass of an substance

changes when the substance breaks or meltsStudents think that plants receive food from

soilStudents have misconceptions about what

makes something a producer in an ecosystem.Students think genetic characteristics are

acquired from the environment

Common misconceptions (gr. 8)Students have trouble comparing a model to

the real-world system

Students think the mass of an substance changes when the substance changes state or temperature

Students have trouble calculating speedStudents don’t understand photosynthesisStudents struggle to see the how natural

selection leads to changes in a population over time

Going well (gr. 5)Variables questions

65% to 90% of the timeStudents are including data in

conclusion items28-37% get full points Between 30% and 40% give each supporting

data point Students are attempting energy transfer

and new procedure items3.1% to 5.0% blanks

Going well (gr. 8)Variables questions

66% to 84% of the timeStudents are including data in

conclusion items54% get full points Between 55% and 58% give each supporting

data point Students are attempting new procedure

items7.9% - 10% blanks

Constructed responses (both)Students confuse different forces (gravity, push,

pull, magnetic) with forms of energy (5th: light, heat, sound, motion, electricity; 8th: light, thermal (heat), chemical, sound, kinetic, electrical)

Students need more help writing their own procedures: Changing the manipulated variable indicated in the

questionMeasuring the responding variable indicated in the

questionStudents need to be more detailed when answering

application items; more specific than “it will work” or “it must be broken, so fix it.”

Upcoming EventsPerformance Level Descriptors training

Via Moodle website (February-March 2011)

Scenario Writing Workshops5th and 8th grade (February 2011)High School (February 2011)

Contrasting Groups Study (April 2011)

Upcoming EventsOperational Rangefinding Committees

High School (Olympia, April 2011)5th and 8th grade (Cincinnati, Ohio, May

2011)

Pilot Rangefinding Committees (July 2011)

Standard Setting (August 2011)

Opportunities for InvolvementPEPPER email list

PreSALTers Enthusiastically Promoting Powerful Educational Resources

Get updates about science happeningsSign up today with your e-mail address, or e-

mail us and ask to be a PEPPERSALT list

Science Assessment Leadership TeamAnyone can apply Respond to emails to sit on Item Development

committeesCommittees are chosen to balance state

representation and content expertise

Science Assessment OfficeElaine Wood—Secondary Specialist

elaine.wood@k12.wa.us

Kara Monroe—Elementary Specialistkara.monroe@k12.wa.us

Troi Williams—Administrative Assistanttroi.williams@k12.wa.us