The Smarter Balanced Assessment Consortium Training 2-Day 1 The Digital Library Review Process Using the Quality Criteria

The Smarter Balanced Assessment Consortium

Training 2-Day 1

The Digital Library Review Process Using the Quality Criteria

Introductions and Success Stories

• Introductions by State• Share one success from Training 1• Share one lesson learned from

Training 1

Slide 2

Digital Library Resources

Slide 3

• Commissioned Professional Learning Modules• Resources for educators, students and families

• Frame Formative Assessment within a Balanced Assessment System• Articulate the Formative Assessment Process• Highlight Formative Assessment Practices and Tools

Assessment Literacy Modules

• Commissioned Professional Learning Modules• Instructional coaching for educators • Instructional materials for students

• Demonstrate/support effective implementation of the formative assessment process

• Focus on key content and practice from the Common Core State Standards for Mathematics and English Language Arts

Exemplar Instructional

Modules

• High-quality vetted instructional resources and tools for educators• High-quality vetted resources and tools for students and families

• Reflect and support the formative assessment process• Reflect and support the Common Core State Standards for Mathematics

and English Language Arts• Create Professional Learning Communities

Education Resources

Our Successes So Far

Formative Assessment Advisory Panel Developed

Quality Criteria

State Leadership Teams and State Networks of Educators

Established, Involving Around 2,000 K-16 Educators Across

the Country

SLT Training 1 and SNE Training 1 Completed

Over 450 Resources Submitted to the Digital

Library

Blueprints for Exemplar Instructional Modules and

Assessment Literacy Modules Blueprints Drafted

Matrices for the Exemplar Instructional Modules for ELA and Math and for the

Assessment Literacy Modules Drafted

Digital Library Software Versions 1.0 and 1.1

Released

What’s Coming Next

• SLT Training 2 in November and December 2013• Software Version 1.15 in February 2014Short-Term

• SLT Trainings 3 (March 2014), 4 (May 2014), and 5 (August 2014)

• Software Version 1.2 in April 2014• First set of 23 Exemplar Instructional Modules and

Assessment Literacy Modules in March 2014• Second set of Modules in May 2014 and third set of

Modules by September 2014

Long-Term

• Digital Library Open to Consortium Members in April 2014

Milestone

Training 1 Follow-up

• SLT Webpage Updates– Updated Cover Profile Guide– Tips for Identifying the License Presentation

• Role Overview in the Digital Library– What can I view as an SLT member?

• SNE Submissions– What should be submitted by the end of

November?Slide 6

SNE Individual ReportReview

Slide 7

Training 2:

The Digital Library Review Process

Using the Quality Criteria for

Professional Learning and Instructional Resources

Slide 8

SLT Training 2

Learning Goal

• SLT members will understand the Quality Criteria ratings and how to apply them consistently.

Success

Criterion

• SLT members will provide evidence of inter-rater reliability among the state SLT members and with other state SLT members in cross-state teams.

Slide 9

Formative Assessment Advisory Panel

Convened the Formative Assessment Advisory Panel to develop the Quality Criteria during three two-day meetings

Page 10

Meeting 1April 17 – 18

Meeting 2May 8 – 9

Meeting 3May 22 – 23

• Brainstormed initial Quality Criteria

• Determined structure of Quality Criteria

• Developed comprehensive list of potential criteria

• Discussed merits of checklist vs. rubric-based approach

• Tested criteria using sample resources

• Refined criteria

Synthesized feedback

Developed first draft of

Quality Criteria

Developed 2nd draft; received feedback

from panelists; developed present

draft

Slide 10

Members of the Formative Assessment Advisory Panel

1) Dr. Lynne Anderson-Inman (University of Oregon)

2) Dr. Robert Calfee (Stanford University, UC Riverside)

3) Dr. Bridget Dalton (University of Colorado)

4) Dr. Diane Heacox (St. Catherine University)

5) Dr. Joan Herman (UCLA – CRESST)

6) Dr. John Hill (Purdue University)

7) Dr. Yvette Jackson (National Urban Alliance for Effective Education)

8) Dr. Henry Kepner (University of Wisconsin - Milwaukee)

9) Dr. Katherine McKnight (National Louis University)

10) Valerie L. Mills (National Council of Supervisors of Mathematics)

11) Dr. James Popham (UCLA)

12) Dr. Lucinda Soltero-Gonzalez (University of Colorado-Boulder)

Page 11

Twelve experts in the CCSS for ELA, the CCSS for Mathematics, the formative assessment process, adult learning, online professional learning, diverse learners, and urban and rural education comprised the Formative Assessment Advisory Panel.

Slide 11

Parts of the Quality Criteria

Cover Profile

Gatekeeping Criteria

Quality Criteria for Professional Learning Resources and Quality Criteria for Instructional Resources

Governance Criteria

Slide 12

N/A

Purposes of the Quality Criteria

Page 13

Ensure that all resources in the Digital Library are

of the highest quality, regardless of source

Ensure that all resources reflect the Smarter Balanced vision of effective formative

assessment practices

Provide consistency in the review process for all

resources

Ensure that the resources in the Digital Library reflect the intent

of the CCSS

Resource Posting Work Flow

Page 14

Step 1:Resource Submitted

Step 3:Quality Criteria Applied

Step 4:Decision

Step 2:Gatekeeping

Criteria Applied

SNE1

SNE2

SNE3

SNE1

Posted

Sent to SLT

Returned to Submitter

Cover Profile

A Close Reading of the Quality Criteria

Slide 15

• Individually Read and Annotate the Quality Criteria– Pages 2-5: Quality Criteria for Professional Learning Resources

– Pages 6-9: Quality Criteria for Instructional Resources

• Think About and Discuss with Your Table Group– What are the differences between the two sets of criteria?

– How might these criteria help focus the review process?

– Where will your state need more clarity and information for Training 2?

• Whole Group Discussion

Additional Resources

SLT Webpage: http://www.amplify.com/smarter-balanced-slt

– Quality Criteria FAQ– UDL– Professional Learning

Slide 16

http://www.amplify.com/smarter-balanced-slt



Break and Login Distribution

Slide 17

Accessing the Digital Library in the Training Environment

Page 18

1. Open your Internet browser (Google Chrome typically works best, though Safari, Firefox, and Internet Explorer 9+ are also options).

2. Type the web address: http://training.smarterbalancedlibrary.org

3. Type your assigned email address: slt#@smarterbalancedlibrary.org

4. Type the password: password

5. Click “Log into the Digital Library”

http://training.smarterbalancedlibrary.org/

Common IR Review

• Read the Cover Profile• Read the PDF Materials• Watch the Video (whole group)• Individually Rate the Resource• Discuss Ratings with Your State

Team• Determine Ratings, Comments,

Posting Recommendation, Rationale as a Team

Slide 19

Common IR ReviewDebrief

• On your poster paper:– Resource Title– Criteria Ratings– Posting Rec– Rationale

• Discuss with the whole group:– Talk us through your team’s process when

discussing and deciding on the ratings.

Slide 20

Reflection

1.What criteria will need more discussion/clarification with your SNE members?

2.What are the areas of confusion about the Quality Criteria for Instructional Resources?

3.What recommendations do you have for technology improvements?

Slide 21

Cross-State Teams

• Review the Cover Profile• Review Materials• Vet the Resource Against the Quality

Criteria

Slide 22

Slide 23

• On your poster paper:– Resource Title– Criteria Ratings– Posting Rec– Rationale

• Discuss with the whole group:What were notable differences to apply to SNE trainings?

Cross-State Debrief

SLT Training CyclesSNE Payment

Last Day of Training Cycle

PCG Sends “SNE Member Payment Spreadsheet” to SLT Lead

SLT Returns Spreadsheet to PCG

SNE Payment Due

Training 1 October 30, 2013

November 18, 2013December 16, 2013

November 26, 2013December 23, 2013

December 20, 2013January 24, 2014

Training 2December 31, 2013

January 20, 2014February 24, 2014

January 28, 2014March 4, 2014

February 21, 2014March 28, 2014

Training 3April 30, 2014

May 19, 2014 May 27, 2014 June 20, 2014

Training 4June 30, 2014

July 24, 2014 July 19, 2014 August 22, 2014

Training 5Sept. 30, 2014

October 20, 2014 October 28, 2014 November 21, 2014

Slide 24

The Smarter Balanced Assessment Consortium

Training 2-Day 2

The Digital Library Review Process Using the Quality Criteria

Day 1 Recap & Your Feedback

You all are amazing!!

Thank you for your honest conversations and collaboration.

We want to address feedback from Tickets Out the Door.

Slide 26

Follow up from Tickets out the Door

• Resource Review Queue– Content Expertise Selection in Profile (demo)– Ability to tag ELA Literacy Standards for Social

Studies, Science, Technical Subjects

• Use of Training Environment and Resources– SLT Webpage and Jump Drive

• Descriptive Feedback– SLT Arbitration Process Exercise

• Geographic Location in the Cover Profile is an optional field

Slide 27

Questions

1. How do SLTs best instruct SNEs to use the listed criteria to make a holistic judgment?

2. Can SNEs access the training site? When? Who provides the access information? How can we provide resources to practice?

3. If one area shows no evidence, is that grounds for not recommending a resource? Slide 28

Questions4. Questions about Quality Criteria

A. Instructional--#1 i. SBAC Claims, Targets—if this criteria

remains, we need to do more training with SLTs, these do not apply as easily to K-2 or to Foundational Skills at all. Could this tag be made after a resource is accepted?

ii. Does precursor/successor fit here? iii. Are there criteria from #3 that fit better in

#1.B. Instructional--#6-Can an NA rating be

included for this section of criteria for resources that may not benefit tremendously from incorporating technology?

5.What algorithms will result in the SLT receiving the resource for arbitration?

Slide 29

Copyright

• www.amplify.com/smarter-balanced-slt

• Updated Copyright Functionality in the Digital Library

• Copyright Clearance Form• Updated Copyright Information in

Cover Profile Guide• Tips for Identifying the License

Presentation

Slide 30



State share of ideas on how we can help SNE members feel more connected and maintain morale?

Slide 31

• Send Thank You Cards• Highlight Successes• Recognizing SNEs with

Principals/Providing Certificates• Reassurance of completion and

payment• Recognizing SNE First Approved

Resource• Badges

Common Professional Learning Resource Review

• Login to the training environment:– training.smarterbalancedlibrary.org

• Select Common Professional Learning Resource: Formative Assessment Learning Strategies

• Click and Review: About This Resource

Slide 32

Common Professional Learning Resource Review

• 20 minutes– Review About this Resource– Watch Video (whole group)

• 20 minutes– Review pdf files– Individually vet the resource using the

Professional Learning Quality Criteria

• 15 minutes– Discuss and compare answers with your team

• 5 minutes– Create your poster Slide 33

Debrief

What were the notable differences in reviewing a Instructional Resource versus a Professional Learning Resource?

Additional questions, comments, concerns?Tips for SNE training? Slide 34

Posting Decisions

When does an SLT have to make a resource posting decision?

Slide 35

SNE Recommendation Options

Slide 36

Do not recommend

Recommend with revisions

1.Recommend with distinction2.Recommend3.Recommend with revisions4.Do not recommend

20 SNE Recommendation Combinations

Page 37

Decision SNE Recommendation SNE Recommendation SNE RecommendationSoftware

or SLT Decision?

Post with Distinction

Recommend with Distinction



Software



Recommend Software

PostRecommend with Distinction

Recommend Recommend Software

Recommend Recommend Recommend Software

SLT Member Judgment



Recommend with Revisions

SLT



Do Not Recommend SLT


RecommendRecommend with Revisions

SLT


Recommend Do Not Recommend SLT

Recommend RecommendRecommend with Revisions

SLT

Recommend Recommend Do Not Recommend SLT

Not Post; Return to submitter


Recommend with RevisionsRecommend with Revisions

Software


Recommend with Revisions Do Not Recommend Software


Do Not Recommend Do Not Recommend Software

Recommend Recommend with RevisionsRecommend with Revisions

Software

Recommend Recommend with Revisions Do Not Recommend SoftwareRecommend Do Not Recommend Do Not Recommend Software

Recommend with Revisions Recommend with RevisionsRecommend with Revisions

Software

Recommend with Revisions Recommend with Revisions Do Not Recommend SoftwareRecommend with Revisions Do Not Recommend Do Not Recommend SoftwareDo Not Recommend Do Not Recommend Do Not Recommend Software

Please note: The decision depends on the combination of recommendations; the order of the recommendations does not matter.

Recommendation Combinations-SLT Member Decision

Page 38

An SLT member will make a posting decision given the following recommendation combinations:SNE Recommendation

SNE Recommendation

SNE Recommendation





Recommend with Distinction Do Not Recommend

Recommend with Distinction Recommend


Recommend with Distinction Recommend Do Not Recommend


Recommend Recommend Do Not Recommend

SNE Recommendation

SNE Recommendation

SNE Recommendation





Recommend with Distinction Do Not Recommend

Recommend with Distinction Recommend


Recommend with Distinction Recommend Do Not Recommend


Recommend Recommend Do Not Recommend

SLT Decision Scenario

Page 39

SLT MemberQuality Criteria Review

• Log into the “real” Digital Library

• Software Demo

• Resource Review tab

Slide 40

Training 2 SLT Arbitration Exercise Materials

• Resource• Cover Profile• Three SNE Quality Criteria Reviews

– Recommend with Distinction– Recommend– Recommend with Revisions

• SLT Arbitration Form

Page 41

When does an SLT have to make a resource posting decision?

• Find someone you haven’t spoken to from a different table

• Share your table’s recommendation and rationale.

• Whole Group Share– What did you hear that was interesting?– Any new thoughts?

Slide 42

State Team PlanningShare any good ideas on the collaboration

poster.

Slide 43

Evaluation

https://www.surveymonkey.com/s/SLTTraining2

Email with the link by 12:00pm

Slide 44




Documents

The Smarter Balanced Assessment Consortium Training 2-Day 1 The Digital Library Review Process Using the Quality Criteria