Upload
zaide
View
36
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Vertically Articulated Performance Standards. An Exploratory Examination of their Interpretability Steve Ferrara, Gary Phillips, Paul Williams, and Shannon Mahoney American Institutes for Research October 19, 2006. Overview for this talk. Background and purpose of the study - PowerPoint PPT Presentation
Citation preview
Vertically Articulated Performance Standards
An Exploratory Examination of their Interpretability
Steve Ferrara, Gary Phillips, Paul Williams, and Shannon Mahoney
American Institutes for ResearchOctober 19, 2006
Vertically Articulated Standards (VAS) 2
Overview for this talk
Background and purpose of the study Vertically articulated standards
Definition, procedures The assessment program This study
Procedures Results, interpretation, implications
Conclusion
Vertically Articulated Standards (VAS) 3
Purposes of the study
Evaluate a system of articulated standards that we created for a state content area assessment
Examine definitions of Proficient performance and achievement growth as defined by test items around the Proficient cut scores
Consider implications for the panelist’s judgmental task in selecting cut scores and articulating standards
Vertically Articulated Standards (VAS) 4
Vertically articulated performance standards
Coherent system of performance standards When performance standards are articulated,
they form a system of standards in an orderly progression across grades in the same content area and across content areas
An orderly progression of standards is indicated by the percentages of examinees at/above each performance level in each grade
Vertically Articulated Standards (VAS) 5
Vertically Articulated Performance Standards
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
3 4 5 6 7 8
Grade
Per
cen
tag
e o
f S
tud
ents
At
or
Ab
ove
Basic
Proficient
Accelerated
Advanced
Vertically Articulated Standards (VAS) 6
Vertically Disarticulated Performance Standards
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
3 4 5 6 7 8
Grade
Per
cen
tag
e o
f S
tud
ents
At
or
Ab
ove
Basic
Proficient
Accelerated
Advanced
Vertically Articulated Standards (VAS) 7
Importance of vertically articulated standards
Discrepancies can cause confusion in the general public and media and enable misinterpretations of standards and performance Misinterpreting cross-grade differences in
performance standards as achievement differences
Mistargeting resources to a grade level or content area
Blaming or rewarding teachers
Vertically Articulated Standards (VAS) 8
The assessment program
Typical state assessment program (e.g., MC and CR items) Ambitions well beyond NCLB requirements (e.g.,
K-2 diagnostic assessments, other content areas, accountability system, growth modeling)
Reading and math, grades K-8 plus high school
Science, social studies, and writing in an elementary and middle school grade and at high school level
Vertically Articulated Standards (VAS) 9
The assessment program (cont.) Within grade assessments aligned with grade level
content standards Within grade scaling Vertical articulation of standards across the within grade
scales Vertical scale for future score reporting (not
considered in standard setting) Linking items included from the lower adjacent grade only
(“down linking”) Previous year, math: Higher linking error when include
upward linking items Joint calibration, grades 3-8; also chain linking, other
tryouts
Vertically Articulated Standards (VAS) 10
Setting cut scores within grades
Mathematics grades 3-8 Separate 3-5 and 6-8 panels Reading grades 4-5 and 6-8
Bookmark method “Place your bookmark on the page that
(approximately) two-thirds of those students who are just barely Proficient would be able to answer successfully”
RP criterion Just Barely Proficient students
Vertically Articulated Standards (VAS) 11
Proficiency level descriptions (PLDs)
In this program, the PLDs define Proficient performance explicitly in terms of growth from the previous grade
Vertically Articulated Standards (VAS) 12
Grade 3 Mathematics PLD Students performing at the Proficient level (1) show adequate
progress by using grade 3 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) use place value concepts, apply basic measurement and geometry concepts to describe attributes of shapes or objects, and interpret graphs. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language.
Vertically Articulated Standards (VAS) 13
Grade 4 Mathematics PLD Students performing at the Proficient level (1) show adequate
progress by using grade 4 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving whole numbers, decimals and simple fractions; describe perimeter and area; compare geometric figures; write an equation to describe a situation; and describe data. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language.
Vertically Articulated Standards (VAS) 14
Grade 5 Mathematics PLD Students performing at the Proficient level (1) show adequate
progress by using grade 5 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving sums and differences of fractions or decimals, measuring angles, describing the probability of events, and converting units in the same measurement system. They typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students use informal and some formal reasoning to evaluate and justify the reasonableness of a solution. They communicate mathematical thinking and solutions using a combination of informal and mathematical language.
Vertically Articulated Standards (VAS) 15
PLDs for Proficient
Each definition refers to growth from the previous grade
Each describes articulation of content and skill requirements across grades 3, 4, and 5
Some features are constant across these grades
Vertically Articulated Standards (VAS) 16
Achieving vertically articulated standards
Set cut scores for anchor grades in typical way
For interim grades, provided the target cut score (i.e., page number) that would provide an orderly progression of standards across grades
We asked panelists to try to make the bookmark judgment within three or so pages of the target
Vertically Articulated Standards (VAS) 17
Achieving vertically articulated standards (cont.)
After all standards were set, table leaders participated in a moderation process to achieve articulated standards Guidance: Cut scores that corresponded to
orderly percentages of students reaching Proficient (and other levels) in each grade
Within content area, then across content areas They chose not to adjust one standard
Vertically Articulated Standards (VAS) 18
This study: Analytic procedures
Identified items that define Just Barely Proficient performance
Examined the knowledge and skills they Target Require for successful responses We refer to this as Item Demands Analysis
Summarized and compared the knowledge and skill demands
Vertically Articulated Standards (VAS) 19
Analysis (cont.)
Specifically, we focused on the items that define Just Barely Proficient performance at grades 3 and 4
We examined those items and how they define: Just Barely Proficient performance for standard
setting panelists Growth in achievement
Vertically Articulated Standards (VAS) 20
Analysis (cont.)
We used mathematics in grades 3 and 4 from a state assessment program
We focused on the standard for Proficient performance, established using the Bookmark procedure
We focused on the concept of the Just Barely Proficient student because this is the concept that panelists use to make the cut score judgment
Vertically Articulated Standards (VAS) 21
Analysis (cont.)
Items: RP 67 locations on the vertically linked scale
Just Barely Proficient: Items at the cut score, three items below, three items above
Vertically Articulated Standards (VAS) 22
Example: grade 3 mathematics
Item ID RP67Stan-dard Load Item description CAK CAS
O3MO4016 -0.764 NS LowDivision - missing
numberPK: Division fact UA 11h
O3MO894 -0.758 ME ModMeasurement -
customaryCU: Measurement
unitsUA 11h
O3MO3414 -0.656 GS Mod Geometry # 3 - obtuse PK: acute angleUA 11b,
11h
O3MO4018 -0.576 NS Mod Estimation - subtractionCU: Reasonable
estimateAE 12a
O3MO567 -0.572 NS LowIdentify - mixed
numbersPK: Mixed number (2
1/4) as visual rep'nUA 11b,
11h
O3MO968 -0.520 PA LowPatterns - identifying
numbersRecog: subtraction
patternACH 13c
O3MO8225 -0.487 NS High Packages of buns
PK: Multiplication facts for 48;calculate correct cost; select correct bills and coins
AE 12a
Vertically Articulated Standards (VAS) 23
Item demands analysis
Targeted content standards Identified in the state content standards and test
blueprints (see App. A) Our judgments about:
Reading load Content area knowledge demands Content area skill demands/question types
Vertically Articulated Standards (VAS) 24
Definitions: Reading load
High Lots of text, complex text, interrelatedness of
elements in the text, complex format Moderate
Multiple steps, short phrases, scaffolded format Low
Small amount of text, mostly mathematics, how to proceed and respond is explicit or obvious
Vertically Articulated Standards (VAS) 25
Definitions: Content area knowledge
Prior knowledge Supply or apply an arithmetic fact or taught
algorithm Conceptual understanding
Use or explain a mathematical concept Recognition
Recognize and complete a numerical or other pattern, recognize a visual representation of a mathematical concept or procedure
Vertically Articulated Standards (VAS) 26
Definitions: Content area skills
Use/apply E.g., visual information given with the item
Answer and explain E.g., defend the answer given
Analyze, categorize, hypothesize E.g., observe and describe patterns in data,
procedures, or results Empirically supported (Ferrara, Duncan, et
al., 2004); adapted from science
Vertically Articulated Standards (VAS) 27
Selected results: Grades 3 and 4
Grade 3 JBP Student Grade 4 JBP StudentJBP Student at the Grade 34
Cut Score
Standard NS=4ME=1PA=1GS=1DA=0
NS=2ME=1PA=1GS=2DA=1
NS=2ME=2PA=1GS=0DA=2
Region of RP 67 locations of items
-0.76, -0.49Difference=.27
-0.13, +0.26Difference=.39
-0.78, -0.33Difference=.45
Scale location of cut score
-Easy (CS=-0.58)-CS at 29 of 52 (56%)
-Moderate (CS=-0.01)-CS at 15 of 52 (29%)
-Easy (CS=-0.45)-CA at 5 of 52 (10%)
Item types 5 MC, 1 SA 1 of 2, 1 SA 2 of 4 4 MC, 1 SA 1 of 2, 1 SA 2 of 2, 1 SA 1 of 4
5 MC, 2 SA 1 of 2
Content area knowledge requirements
PK=3CU=2Recog=2
PK=3CU=1Recog=3
PK=5CU=Recog=1
Content area skill requirements
UA=4AE=2ACH=1
UA=5AE=2 (1 UA and AE)ACH=0
UA=5AE=2ACH=0
Reading and language load
Low=3Mod=3High=1
Low=1Mod=3High=3
Low=3Mod=3High=1
Vertically Articulated Standards (VAS) 28
Selected results: Grades 3 and 4 cut scores
Standards targeted More number sense at grade 3, more geometry at
grade 4 Scale location of cut scores and JBP items
CS grade 3 = -0.58, CS grade 4 = -0.01 Locations grade 3 = (-0.76, -0.49), grade 4 =
(-0.13, +0.26) Item types
Similar
Vertically Articulated Standards (VAS) 29
Selected results: Grades 3 and 4 (cont.)
Content area skills Similar
Reading load Grade 3 low-moderate Grade 4 moderate-high
Vertically Articulated Standards (VAS) 30
Content area knowledge requirements
Grade 3 Grade 4
Prior knowledge
Use division and multiplication facts; calculate cost, select bills and coins (2)
Calculate statistics, use to support conclusion; calculate change, select bills and coins (3)
Conceptual understanding
Measurement units for liquid, reasonable estimate (2)
Parallel lines (1)
Recognition
Subtraction pattern, pizzas as mixed number, acute angle (3)
Addition pattern, symmetric rotation, calculate area by counting squares (3)
Vertically Articulated Standards (VAS) 31
Summary of results: Grades 3 and 4
Item demands on JBP students similar for the grade 3 and 4 cut scores except in three areas Location of cut score on vertical scale Content area knowledge requirements Reading load
Item demands for grade 3grade 4 scale Similar to grade 3 except more emphasis on prior
knowledge requirements
Vertically Articulated Standards (VAS) 32
Interpretation of results: Three areas
System of articulated standards Definitions/interpretations of what it means to be
Proficient and what is growth in achievement Implications for setting and articulating standards
We are interested in practical steps for setting performance standards that: Are vertically articulated Enable valid interpretations about what students know and
can do as they progress through school
Vertically Articulated Standards (VAS) 33
Interpretation: System of articulated standards
Articulating standards across grades using statistical considerations can enable meaningful interpretations from a content area demands point of view
Content knowledge, reading load, and difficulty demands increased in meaningful ways across grades 3 and 4 for this mathematics assessment
That increase is small and consistent with the grade level means
Vertically Articulated Standards (VAS) 34
Student proficiency estimates, vertically linked theta scale
Grade Mean SD
3 0.00 0.97
4 0.58 0.96
5 1.00 1.00
6 1.27 0.88
7 1.52 0.91
8 1.66 0.94
Vertically Articulated Standards (VAS) 35
System of articulated standards (cont.)
A concern: The articulation worked out well, but it did not happen completely by design
Content standards, PLDs, and performance standards are articulated by design
Item locations are not articulated by design Item writers aim at difficulty targets but cannot/do not control
empirical item difficulty A point being made often: Scientifically principled test design and
development and item difficulty modeling E.g., Ferrara & DeMauro, 2006; Gorin, 2006; Mislevy & Haertel
(2006)
Vertically Articulated Standards (VAS) 36
Interpretations of Proficient and growth in achievement
Articulating standards in a meaningful way enables inferences about growth in achievement; however,
Typical PLDs illuminate what that growth entails only in a very general way “Students performing at the Proficient level show
adequate progress by using grade 3 concepts and skills to solve familiar problems. …”
Vertically Articulated Standards (VAS) 37
Interpretations of Proficient and growth in achievement (cont.)
Meaningful inferences about grade-to-grade growth in achievement require that four elements are articulated: Content standards that are the basis for test
design and development PLDs that are the performance standards The cut scores that effect those standards The items around a cut score that panelists use to
define the cut scores
Vertically Articulated Standards (VAS) 38
Implications for setting and articulating standards
In item mapping procedures, panelists consider item demands to locate their cut scores
They may or may not consider whether the items they examine will result in articulated cut scores Even when we give them articulation information It seems likely that they do not
Consider using item demands summary tables as input to: The standard setting process Subsequent moderation processes to articulate standards
Vertically Articulated Standards (VAS) 39
Conclusion
Exploratory examination; we need to look at additional grades and content areas
It seems unlikely that things will work out as fortunately in all grades and content areas
Vertically Articulated Standards (VAS) 40
ReferencesFerrara, S., & DeMauro, G. E. (2006). Standardized assessment of individual achievement in K-12. In
R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger.
Ferrara, S., Duncan, T. G., Freed, R., Velez-Paschke, A., McGivern, J., Mushlin, S., Mattessich, A., Rogers, A., & Westphalen, K. (2004). Examining test score validity by examining item construct validity: Preliminary analysis of evidence of the alignment of targeted and observed content, skills, and cognitive processes in a middle school science assessment. Paper presented at the annual meeting of the American Educational Research Association, San Diego.
Gorin, J. (2006 in press). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25 (4).
Mislevy, R. J., & Haertel, G. D. (2006 in press). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25 (4).
Vertically Articulated Standards (VAS) 41
Appendix A: Content Standards
Number, Number Sense and Operations Standard
Measurement Standard Geometry and Spatial Sense Standard Patterns, Functions and Algebra Standard Data Analysis and Probability Standard Mathematical Process Standard