Overview of Standard Setting
Leslie WilsonAssistant State Superintendent
Accountability and Assessment
August 26, 2008
Purpose of Presentation
To present the standard setting process
To provide background information to the Board in preparation for setting standards for– Modified High School Assessments– Alternative Maryland School
Assessments
Setting Performance Standards
ContentStandards
Voluntary State
Curriculum
The Tests
Expertise: Student
Knowledge/SubjectMatter
Setting Setting PerformanPerforman
ce ce StandardsStandards
Cut Scores that match students to
their appropriate
Performance Categories
The Threshold Student
A borderline or minimally qualified student– In terms of performance
The least qualified of the Proficient or Advanced students
Mod-HSA Test Performance Standards
Basic
Recommended Standards
threshold students
Proficient Advanced
Standard Setting Work Flow
Content Groups
Psychometric Council
Review & Articulation Committee
State Superintendent
State Board of Education
Set Standards
Step 1: Content Groups
Participants all have appropriate content and grade-level expertise
Include representatives from local Central Offices as well as school level staff
Participate in separate content and grade level discussions
Responsibilities of Content Group
Review Assessment
Discuss what barely proficient and advanced students know and can do
Propose cut scores and discuss
Make recommendations to Review and Articulation group
Content Group Process: Round 1
Break into small groups to encourage independent decision-making
Read, discuss ordered-item booklets
Individuals vote (privately) for a cut score
Small group discussion of votes
The Process: Round 2
Individuals vote (privately) for a cut score
Small group discussion of votes
Examine, discuss impact data
The Process: Round 3
Small groups converge
Individuals vote (privately) for a cut score
Discussion
Individuals vote (privately) for a cut score
The median cut score is the committee’s recommendation.
Bookmark Standard-Setting Procedure
Test items students must know and be able to do to be considered Proficient
Test items students must
know and be able to do to be considered Advanced
A
P
Ordered Item
Booklet
1
23
45
67
8
9
10
1112
13
14
15
Standard-Setting Process
Individual Analysis
Items most
students get
correct
Items fewer
students get
correct
Content every student should know
Review test; vote
Round 1
Discuss data and distribution
Median
Standard-Setting Process
Round 2
View impact data and discuss
VoteMedian
Standard-Setting Process
Small groups join; vote
Round 3
Median cut score is committee’s recommendation
Median
Step 2: Psychometric Council
Nationally-recognized experts
Trained in measurement and psychometrics
Independent
Familiar with Maryland tests
Similar to Technical Advisory Panels in other states
Psychometric Council Responsibilities
Reviews work of all groups
Ensures quality controls were followed
Ensures standard-setting process technically sound
Does not make judgments on whether standard is “good” or “bad”
Forwards comments and recommendations to Review & Articulation Committee
Step 3: Review and Articulation
Superintendents
Assistant Superintendents
Local Accountability Coordinators
Advocates for students with special needs (special education and ELL)
Parents
Local Boards of Education
Teachers Union representatives
Business Community members
MSDE
Review & Articulation Committee
Responsibilities include:
– Reviewing work of all groups and Psychometric Council
– Ensuring rigor is equivalent among grades
– Reviewing articulation among grades
– Forwarding recommendations to the State Superintendent
Step 4: State Superintendent of Schools
Reviews work of all groups, Psychometric Council, and Review & Articulation Committee
Makes recommendation to the State Board of Education
Step 5: State Board Approval
Reviews proposed standards
Approves standards for implementation
Next Steps
1. Review and approve proposed standards to be used to determine proficient and advanced performance on the modified HSAs. The scores will be used to determine AYP for Maryland high schools next year.
2. Review and approve proposed standards for the Alt-MSA. Science will be reported this fall for the first time.
Overview of Standard Setting
Questions and Discussion