Understanding Test Blueprints and Their Impact on Student Growth

Understanding Test Blueprints and Their Impact on Student Growth Holly Lavender and Cathryn Shaw

Learning Targets

1.  Understand the definition and purpose of test blueprints

2.  Know how to build a test blueprint

3.  Understand how assessment blueprints connect to other initiatives

The Purpose of Test Blueprints

1

The Purpose of Test Blueprints •  Define the parameters of an assessment before you

begin the creation process. •  Review a locally- or vendor-developed assessment for

alignment to learning expectations, instruction, and cognitive complexity.

•  Confirm in advance the information and knowledge you are planning to assess, which can guide instructional activities.

•  Share blueprints with students to help make learning targets clear.

A Blueprint is a TOOL to ensure: alignment of assessment to standards

- content - depth of knowledge

alignment of assessment to instruction adequate “stretch” in assessment

Building a Test Blueprint

2

Where to Begin?

The End

Design  the  blueprint    Create  the  assessment    Plan  for  Instruc7on      

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1                          

2                          

3                          

4                          

5                          

Total  Points                                  

Weight                                  

IDENTIFIED  STANDARDS  

Learning Priorities Selection

Unit Test

Quarterly Assessment

SLO Pre/Post-Assessment

Identify Learning Priorities

Longevity

Leverage

Levels

Learning Priorities

Ainsworth,  L.    (2003).  Power  standards:    Iden7fying  the  standards  that  maHer  the  most.      Englewood,  CO:    Lead  and  Learn  Press.  

Learning Priorities: Longevity

• The knowledge and skills that are important today, tomorrow, and in the future.

• Example: Reading Comprehension Longevity

Learning Priorities: Leverage

• The knowledge and skills that are used in more than one content area at a specific grade level (horizontal).

• Example: Creating and analyzing charts and graphs

Leverage

Learning Priorities: Levels • The knowledge and

skills that are key to success in subsequent grade levels or courses.

• Example: Strong understanding of fractions to prepare for work with rational numbers

Levels

Learning Priorities

•  Introduced early • Reinforced often • Multiple opportunities

for learning Learning Priorities

Identify Learning Priorities

Clear  Focus    

 Improved  Validity  

 SLO  Checklist  

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

2.  Determine  the  lesson  or  morale  of  a  story  

3.  Demonstrate  understanding  of  key  details  in  a  text  

4.  Iden5fy  differences  in  point  of  view  

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

Total  Points                                  

Weight                                  

IDENTIFIED  STANDARDS  

Alignment Content: Are the expectations consistent for both curriculum and assessments? Depth of Knowledge (Rigor): Are the cognitive demands of curricular aims and assessments the same?

– For individual items? – For the assessment as a whole?

Cognitive Complexity

• Common Core State Standards and Ohio’s New Learning Standards call for a higher level of cognitive complexity than previous assessments could measure

©2013. Battelle for Kids. All Rights Reserved.

Hess’ Cognitive Rigor Matrix

Bloom’s Taxonomy Webb’s Depth of Knowledge

DoK Level 1 Routine Thinking

Recall and Reproduction

Describe Measure Define Calculate Locate Memorize

Jen  Jones  (2013);  helloliteracy.blogspot.com  

DoK Level 2 Conceptual Thinking

Basic Application and Reasoning

Classify Categorize Summarize Graph Infer Interpret

Jen  Jones  (2013);  helloliteracy.blogspot.com  

DoK Level 3 Strategic Thinking

Higher Level Application

 Discuss Revise

Investigate Assess Critique Argue/Justify

* Usually more than one right answer Jen  Jones  (2013);  helloliteracy.blogspot.com  

DoK Level 4 Extended Thinking

Real World Application

Synthesize Create Design Plan Develop Propose

* Usually over an extended period of time Jen  Jones  (2013);  helloliteracy.blogspot.com  

Is This Item Aligned? K.G2: Correctly name shapes regardless of their orientations or overall size. (CCSS)

•  Show  me  the  triangles.  •  Show  me  the  pentagons.  •  Show  me  the  cylinders.  

Is This Item Aligned?

LS 5.2: Food webs can be used to identify the relationships among producers, consumers and decomposers in an ecosystem. (ONLS)

Is This Item Aligned?

Write the definitions for the following words:

1.  Figure ______________________

2.  Determine ______________________

3.  Abode ______________________

3.L.4a: Use sentence-level context as a clue to the meaning of a word or phrase. (CCSS)

Is This Item Aligned?

  Analyze the causes and consequences of major political, economic and social developments of the 1930’s with emphasis on the Dust Bowl. (OACS)

Is This Item Aligned?

Create a mixed media fantasy landscape that conveys a story, incorporating a well-known character from children’s fiction.

VA 6.4: Connect selected ideas, concepts and processes used in visual art with those used in other academic disciplines. VA 6.5: Use observations, life experiences and imagination as sources for visual symbols, images and creative expression.

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2                      

2.  Determine  the  lesson  or  morale  of  a  story  

       2                      

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1                      

4.  Iden5fy  differences  in  point  of  view  

       3                      

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3                      

Total  Points                                  

Weight                                  

IDENTIFIED  STANDARDS  

Determine Weight

What portion of the total test will be

assigned to each of the identified standards?

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2                      

2.  Determine  the  lesson  or  morale  of  a  story  

       2                      

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1                      

4.  Iden5fy  differences  in  point  of  view  

       3                      

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3                      

Total  Points                                  

Weight                                  

Weight    

How is Weight Determined?

Learning targets that are broad in scope

The importance of the standard

Standards that are emphasized in classroom instruction

Why should I determine weight?

Ensures adequate coverage of content

Ensures emphasis is placed where intended

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2                       5  

points    20%  

2.  Determine  the  lesson  or  morale  of  a  story  

       2                       3  

points    12%  

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1                       3  

points    12%  

4.  Iden5fy  differences  in  point  of  view  

       3                       6  

points    24%  

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3                       8  

points    32%  

Total  Points                                  

Weight                                  

Weight    

Assessment Method

Many factors can affect the test format

How Much Time is Available?

Time is a factor when it comes to preparing and scoring tests

Consider collaborating with others to write and/or evaluate test items

What is to be Measured?

Choose the most appropriate format for

measuring the cognitive objectives of the test

FIP refers to this as the “target-method match”

Which Assessment Method? Identify the eight major planets in the solar system.

Selected  Response  

Constructed  Response  

Which Assessment Method? Make a line plot to display a data set of measurements in fractions of a unit (1/2, 1/4, 1/8).

Performance  

Which Assessment Method? Write arguments to support claims with clear reasons and relevant evidence.

Constructed  Response  

Which Assessment Method? Work effectively in a group to complete a task or solve a problem for which the group is held accountable.

Performance  

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2      

SR  &  CR                   5  

points    20%  

2.  Determine  the  lesson  or  morale  of  a  story  

       2      

CR                   3  

points    12%  

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1      

SR                   3  

points    12%  

4.  Iden5fy  differences  in  point  of  view  

       3      

P                   6  

points    24%  

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3      

P                   8  

points    32%  

Total  Points                                  

Weight                                  

Method    

Stretch

Assessments should demonstrate stretch Include both high and low-achieving students

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2      

SR  &  CR                   5  

points    20%  

2.  Determine  the  lesson  or  morale  of  a  story  

       2      

CR                   3  

points    12%  

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1      

SR                   3  

points    12%  

4.  Iden5fy  differences  in  point  of  view  

       3      

P                   6  

points    24%  

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3      

P                   8  

points    32%  

Total  Points                                  

Weight                                  

Cogni5ve  Complexity  

Options to Demonstrate Stretch

Include items that assess content from adjacent grade levels

Include items that assess on-level standards at a greater or lesser depth

How Much Stretch?

At least 50% of

assessment items should be at or

above the target DOK level

Sample Test Blueprint LEARNING  PRIORITIES   ASSESSMENT  

Learning  Priori5es   DOK  Level  1-­‐4  

MethodSR,  CR,  P  

DOK  1:  Recall  or  Produce  

DOK  2:  Basic  Applica5on  

DOK  3:  Strategic  Thinking  

DOK  4:  Extended  Thinking  

Total  Points  

Weight  

1.  Describe  how  characters  respond  to  challenges  

     2      

SR  &  CR    

 2  points      

3  points           5  

points    20%  

2.  Determine  the  lesson  or  morale  of  a  story  

       2      

CR      

1  point      

2  points           3  

points    12%  

3.  Demonstrate  understanding  of  key  details  in  a  text  

   1      

SR    

 3  points               3  

points    12%  

4.  Iden5fy  differences  in  point  of  view  

       3      

P        2  points    4  points       6  

points    24%  

5.  Compare  and  contrast  two  or  more  versions  of  a  story  

       3      

P            

 8  points       8  

points    32%  

Total  Points            6  points    7  points    12  points    0  points          

Weight            24%    28%    48%    0%          

Cogni5ve  Complexity  

Rigor  

Putting It All Together

3

Remember…. We are measuring the effects of teaching

Assessment must reflect what is taught

Teaching must reflect what is to be assessed

Remember… Look for and evaluate items from banks and collaborate with others to write and evaluate high-quality items Use the data to improve instruction and Increase student learning. The quality of the data is only as good as the quality of the assessment!

education.ohio.gov

holly.lavender@mvesc.org

cathryn.shaw@fcesc.org

Social Media

@OHEducation

ohio-department-of-education

Ohio Families and Education Ohio Teachers’ Homeroom

OhioEdDept

storify.com/ohioEdDept