The Smarter Balanced Assessment Consortium
Training 2-Day 1
The Digital Library Review Process Using the Quality Criteria
Introductions and Success Stories
• Introductions by State• Share one success from Training 1• Share one lesson learned from
Training 1
Slide 2
Digital Library Resources
Slide 3
• Commissioned Professional Learning Modules• Resources for educators, students and families
• Frame Formative Assessment within a Balanced Assessment System• Articulate the Formative Assessment Process• Highlight Formative Assessment Practices and Tools
Assessment Literacy Modules
• Commissioned Professional Learning Modules• Instructional coaching for educators • Instructional materials for students
• Demonstrate/support effective implementation of the formative assessment process
• Focus on key content and practice from the Common Core State Standards for Mathematics and English Language Arts
Exemplar Instructional
Modules
• High-quality vetted instructional resources and tools for educators• High-quality vetted resources and tools for students and families
• Reflect and support the formative assessment process• Reflect and support the Common Core State Standards for Mathematics
and English Language Arts• Create Professional Learning Communities
Education Resources
Our Successes So Far
Formative Assessment Advisory Panel Developed
Quality Criteria
State Leadership Teams and State Networks of Educators
Established, Involving Around 2,000 K-16 Educators Across
the Country
SLT Training 1 and SNE Training 1 Completed
Over 450 Resources Submitted to the Digital
Library
Blueprints for Exemplar Instructional Modules and
Assessment Literacy Modules Blueprints Drafted
Matrices for the Exemplar Instructional Modules for ELA and Math and for the
Assessment Literacy Modules Drafted
Digital Library Software Versions 1.0 and 1.1
Released
What’s Coming Next
• SLT Training 2 in November and December 2013• Software Version 1.15 in February 2014Short-Term
• SLT Trainings 3 (March 2014), 4 (May 2014), and 5 (August 2014)
• Software Version 1.2 in April 2014• First set of 23 Exemplar Instructional Modules and
Assessment Literacy Modules in March 2014• Second set of Modules in May 2014 and third set of
Modules by September 2014
Long-Term
• Digital Library Open to Consortium Members in April 2014
Milestone
Training 1 Follow-up
• SLT Webpage Updates– Updated Cover Profile Guide– Tips for Identifying the License Presentation
• Role Overview in the Digital Library– What can I view as an SLT member?
• SNE Submissions– What should be submitted by the end of
November?Slide 6
SNE Individual ReportReview
Slide 7
Training 2:
The Digital Library Review Process
Using the Quality Criteria for
Professional Learning and Instructional Resources
Slide 8
SLT Training 2
Learning Goal
• SLT members will understand the Quality Criteria ratings and how to apply them consistently.
Success
Criterion
• SLT members will provide evidence of inter-rater reliability among the state SLT members and with other state SLT members in cross-state teams.
Slide 9
Formative Assessment Advisory Panel
Convened the Formative Assessment Advisory Panel to develop the Quality Criteria during three two-day meetings
Page 10
Meeting 1April 17 – 18
Meeting 2May 8 – 9
Meeting 3May 22 – 23
• Brainstormed initial Quality Criteria
• Determined structure of Quality Criteria
• Developed comprehensive list of potential criteria
• Discussed merits of checklist vs. rubric-based approach
• Tested criteria using sample resources
• Refined criteria
Synthesized feedback
Developed first draft of
Quality Criteria
Developed 2nd draft; received feedback
from panelists; developed present
draft
Slide 10
Members of the Formative Assessment Advisory Panel
1) Dr. Lynne Anderson-Inman (University of Oregon)
2) Dr. Robert Calfee (Stanford University, UC Riverside)
3) Dr. Bridget Dalton (University of Colorado)
4) Dr. Diane Heacox (St. Catherine University)
5) Dr. Joan Herman (UCLA – CRESST)
6) Dr. John Hill (Purdue University)
7) Dr. Yvette Jackson (National Urban Alliance for Effective Education)
8) Dr. Henry Kepner (University of Wisconsin - Milwaukee)
9) Dr. Katherine McKnight (National Louis University)
10) Valerie L. Mills (National Council of Supervisors of Mathematics)
11) Dr. James Popham (UCLA)
12) Dr. Lucinda Soltero-Gonzalez (University of Colorado-Boulder)
Page 11
Twelve experts in the CCSS for ELA, the CCSS for Mathematics, the formative assessment process, adult learning, online professional learning, diverse learners, and urban and rural education comprised the Formative Assessment Advisory Panel.
Slide 11
Parts of the Quality Criteria
Cover Profile
Gatekeeping Criteria
Quality Criteria for Professional Learning Resources and Quality Criteria for Instructional Resources
Governance Criteria
Slide 12
N/A
Purposes of the Quality Criteria
Page 13
Ensure that all resources in the Digital Library are
of the highest quality, regardless of source
Ensure that all resources reflect the Smarter Balanced vision of effective formative
assessment practices
Provide consistency in the review process for all
resources
Ensure that the resources in the Digital Library reflect the intent
of the CCSS
Resource Posting Work Flow
Page 14
Step 1:Resource Submitted
Step 3:Quality Criteria Applied
Step 4:Decision
Step 2:Gatekeeping
Criteria Applied
SNE1
SNE2
SNE3
SNE1
Posted
Sent to SLT
Returned to Submitter
Cover Profile
A Close Reading of the Quality Criteria
Slide 15
• Individually Read and Annotate the Quality Criteria– Pages 2-5: Quality Criteria for Professional Learning Resources
– Pages 6-9: Quality Criteria for Instructional Resources
• Think About and Discuss with Your Table Group– What are the differences between the two sets of criteria?
– How might these criteria help focus the review process?
– Where will your state need more clarity and information for Training 2?
• Whole Group Discussion
Additional Resources
SLT Webpage: http://www.amplify.com/smarter-balanced-slt
– Quality Criteria FAQ– UDL– Professional Learning
Slide 16
Break and Login Distribution
Slide 17
Accessing the Digital Library in the Training Environment
Page 18
1. Open your Internet browser (Google Chrome typically works best, though Safari, Firefox, and Internet Explorer 9+ are also options).
2. Type the web address: http://training.smarterbalancedlibrary.org
3. Type your assigned email address: slt#@smarterbalancedlibrary.org
4. Type the password: password
5. Click “Log into the Digital Library”
Common IR Review
• Read the Cover Profile• Read the PDF Materials• Watch the Video (whole group)• Individually Rate the Resource• Discuss Ratings with Your State
Team• Determine Ratings, Comments,
Posting Recommendation, Rationale as a Team
Slide 19
Common IR ReviewDebrief
• On your poster paper:– Resource Title– Criteria Ratings– Posting Rec– Rationale
• Discuss with the whole group:– Talk us through your team’s process when
discussing and deciding on the ratings.
Slide 20
Reflection
1.What criteria will need more discussion/clarification with your SNE members?
2.What are the areas of confusion about the Quality Criteria for Instructional Resources?
3.What recommendations do you have for technology improvements?
Slide 21
Cross-State Teams
• Review the Cover Profile• Review Materials• Vet the Resource Against the Quality
Criteria
Slide 22
Slide 23
• On your poster paper:– Resource Title– Criteria Ratings– Posting Rec– Rationale
• Discuss with the whole group:What were notable differences to apply to SNE trainings?
Cross-State Debrief
SLT Training CyclesSNE Payment
Last Day of Training Cycle
PCG Sends “SNE Member Payment Spreadsheet” to SLT Lead
SLT Returns Spreadsheet to PCG
SNE Payment Due
Training 1 October 30, 2013
November 18, 2013December 16, 2013
November 26, 2013December 23, 2013
December 20, 2013January 24, 2014
Training 2December 31, 2013
January 20, 2014February 24, 2014
January 28, 2014March 4, 2014
February 21, 2014March 28, 2014
Training 3April 30, 2014
May 19, 2014 May 27, 2014 June 20, 2014
Training 4June 30, 2014
July 24, 2014 July 19, 2014 August 22, 2014
Training 5Sept. 30, 2014
October 20, 2014 October 28, 2014 November 21, 2014
Slide 24
The Smarter Balanced Assessment Consortium
Training 2-Day 2
The Digital Library Review Process Using the Quality Criteria
Day 1 Recap & Your Feedback
You all are amazing!!
Thank you for your honest conversations and collaboration.
We want to address feedback from Tickets Out the Door.
Slide 26
Follow up from Tickets out the Door
• Resource Review Queue– Content Expertise Selection in Profile (demo)– Ability to tag ELA Literacy Standards for Social
Studies, Science, Technical Subjects
• Use of Training Environment and Resources– SLT Webpage and Jump Drive
• Descriptive Feedback– SLT Arbitration Process Exercise
• Geographic Location in the Cover Profile is an optional field
Slide 27
Questions
1. How do SLTs best instruct SNEs to use the listed criteria to make a holistic judgment?
2. Can SNEs access the training site? When? Who provides the access information? How can we provide resources to practice?
3. If one area shows no evidence, is that grounds for not recommending a resource? Slide 28
Questions4. Questions about Quality Criteria
A. Instructional--#1 i. SBAC Claims, Targets—if this criteria
remains, we need to do more training with SLTs, these do not apply as easily to K-2 or to Foundational Skills at all. Could this tag be made after a resource is accepted?
ii. Does precursor/successor fit here? iii. Are there criteria from #3 that fit better in
#1.B. Instructional--#6-Can an NA rating be
included for this section of criteria for resources that may not benefit tremendously from incorporating technology?
5.What algorithms will result in the SLT receiving the resource for arbitration?
Slide 29
Copyright
• www.amplify.com/smarter-balanced-slt
• Updated Copyright Functionality in the Digital Library
• Copyright Clearance Form• Updated Copyright Information in
Cover Profile Guide• Tips for Identifying the License
Presentation
Slide 30
State share of ideas on how we can help SNE members feel more connected and maintain morale?
Slide 31
• Send Thank You Cards• Highlight Successes• Recognizing SNEs with
Principals/Providing Certificates• Reassurance of completion and
payment• Recognizing SNE First Approved
Resource• Badges
Common Professional Learning Resource Review
• Login to the training environment:– training.smarterbalancedlibrary.org
• Select Common Professional Learning Resource: Formative Assessment Learning Strategies
• Click and Review: About This Resource
Slide 32
Common Professional Learning Resource Review
• 20 minutes– Review About this Resource– Watch Video (whole group)
• 20 minutes– Review pdf files– Individually vet the resource using the
Professional Learning Quality Criteria
• 15 minutes– Discuss and compare answers with your team
• 5 minutes– Create your poster Slide 33
Debrief
What were the notable differences in reviewing a Instructional Resource versus a Professional Learning Resource?
Additional questions, comments, concerns?Tips for SNE training? Slide 34
Posting Decisions
When does an SLT have to make a resource posting decision?
Slide 35
SNE Recommendation Options
Slide 36
Do not recommend
Recommend with revisions
1.Recommend with distinction2.Recommend3.Recommend with revisions4.Do not recommend
20 SNE Recommendation Combinations
Page 37
Decision SNE Recommendation SNE Recommendation SNE RecommendationSoftware
or SLT Decision?
Post with Distinction
Recommend with Distinction
Recommend with Distinction
Recommend with Distinction
Software
Recommend with Distinction
Recommend with Distinction
Recommend Software
PostRecommend with Distinction
Recommend Recommend Software
Recommend Recommend Recommend Software
SLT Member Judgment
Recommend with Distinction
Recommend with Distinction
Recommend with Revisions
SLT
Recommend with Distinction
Recommend with Distinction
Do Not Recommend SLT
Recommend with Distinction
RecommendRecommend with Revisions
SLT
Recommend with Distinction
Recommend Do Not Recommend SLT
Recommend RecommendRecommend with Revisions
SLT
Recommend Recommend Do Not Recommend SLT
Not Post; Return to submitter
Recommend with Distinction
Recommend with RevisionsRecommend with Revisions
Software
Recommend with Distinction
Recommend with Revisions Do Not Recommend Software
Recommend with Distinction
Do Not Recommend Do Not Recommend Software
Recommend Recommend with RevisionsRecommend with Revisions
Software
Recommend Recommend with Revisions Do Not Recommend SoftwareRecommend Do Not Recommend Do Not Recommend Software
Recommend with Revisions Recommend with RevisionsRecommend with Revisions
Software
Recommend with Revisions Recommend with Revisions Do Not Recommend SoftwareRecommend with Revisions Do Not Recommend Do Not Recommend SoftwareDo Not Recommend Do Not Recommend Do Not Recommend Software
Please note: The decision depends on the combination of recommendations; the order of the recommendations does not matter.
Recommendation Combinations-SLT Member Decision
Page 38
An SLT member will make a posting decision given the following recommendation combinations:SNE Recommendation
SNE Recommendation
SNE Recommendation
Recommend with Distinction
Recommend with Distinction
Recommend with Revisions
Recommend with Distinction
Recommend with Distinction Do Not Recommend
Recommend with Distinction Recommend
Recommend with Revisions
Recommend with Distinction Recommend Do Not Recommend
Recommend RecommendRecommend with Revisions
Recommend Recommend Do Not Recommend
SNE Recommendation
SNE Recommendation
SNE Recommendation
Recommend with Distinction
Recommend with Distinction
Recommend with Revisions
Recommend with Distinction
Recommend with Distinction Do Not Recommend
Recommend with Distinction Recommend
Recommend with Revisions
Recommend with Distinction Recommend Do Not Recommend
Recommend RecommendRecommend with Revisions
Recommend Recommend Do Not Recommend
SLT Decision Scenario
Page 39
SLT MemberQuality Criteria Review
• Log into the “real” Digital Library
• Software Demo
• Resource Review tab
Slide 40
Training 2 SLT Arbitration Exercise Materials
• Resource• Cover Profile• Three SNE Quality Criteria Reviews
– Recommend with Distinction– Recommend– Recommend with Revisions
• SLT Arbitration Form
Page 41
When does an SLT have to make a resource posting decision?
• Find someone you haven’t spoken to from a different table
• Share your table’s recommendation and rationale.
• Whole Group Share– What did you hear that was interesting?– Any new thoughts?
Slide 42
State Team PlanningShare any good ideas on the collaboration
poster.
Slide 43
Evaluation
https://www.surveymonkey.com/s/SLTTraining2
Email with the link by 12:00pm
Slide 44