Embed Size (px)
Citation preview
www.caveon.com 1
Caveon Webinar Series
July 22, 2015
Walt Drane, MS DOEJamie Mulkey, Caveon Test Security
Steve Addicott, Caveon Test Security
Lessons Learned at the National Conference on Student Assessment
www.caveon.com 2
Agenda for Today
• Important themes • Keynote• Security sessions
– Walt– Jamie
• New Resources• Q&A
www.caveon.com 3
State Assessment Director Breakfast
• Wow!• State Assessments =
Politics• Common Core or No
Common Sense?• Exodus from
Consortia
www.caveon.com 4
Impact of Technology on Security• Devices• Technology
Enhanced Items• Automated
Scoring• Computer
Adaptive Testing• Others?
www.caveon.com 5
Keynote – Dr. Andreas SchleicherHow the U.S. and Other Countries Compare:
Lessons Learned from International Education Data
What is the OECD?
Thirty-four democracies work together promoting economic growth, prosperity, & sustainable development
What is PISA?
Test that evaluates 15-year-old students in reading, mathematics, and science, worldwide
www.caveon.com 6
Keynote Key Messages
• “Math skills” vs math skills
• Problem solving skills = Career success
• Great teachers require development
Developing Teaching as a profession
Recruit top candidates into the profession
Support teachers in continued
development of practice
Retain and recognise effective teachers – path for growth
Improve the societal view of teaching as a profession
Mean mathematics performance, by school location, after accounting for socio-economic status77 Capacity at the point of delivery
www.caveon.com 8
Test Security Sessions…
We noticed….there weren’t many!!
www.caveon.com 9
Essential Tips for Leveraging Your
Vendor’s Expertise to Strengthen the Security
of Statewide Assessments
www.caveon.com 10
State Assessment Needs
• Valid and reliable score results
• If results aren’t valid, then actions may be taken – or not taken – that actually harm rather than help
www.caveon.com 11
Holistic Test Security FrameworkDet
er
Detect
Decide (now and for the future)
www.caveon.com 12
Vendor’s Role in DETERRING Negative Activities?Security Audits• Use the Vendor to “Audit” the Security Audit
Process• Thinking Outside the Box
Support/Consulting on policies and procedures• Use vendor to “audit” current policies and
procedures• Student Assessment Handbook, Appendix F, and
Legislation
Invalidations as a deterrent• Use the Vendor during the appeals process
www.caveon.com 13
Vendor’s Role in DETECTING?
Data Forensics
• Systems that utilize sophisticated statistical analyses of test-response data to identify patterns indicative of test fraud (e.g. cheating and piracy).
• Erasure Analysis for paper and pencil assessments
• Analysis of test responses of on-line Assessments
www.caveon.com 14
State’s Expectation of Vendor When DECIDING?To inform our decisions, we expect from our vendors
• Statistical analyses for invalidationsoConsistently use the data you haveoPlan for future challenges (more than
data)
• Information to support go/no go decisions for investigationsoPlay to win - don't pick a fight you are
not reasonably sure you can win
• Deep-dive data analysis and interpretation
www.caveon.com 15
Test Security’s Changing Landscape in Computerized
Testing Environments
www.caveon.com 16
Security in Computerized AssessmentsWhy should we still be concerned about test security with the move to computerized assessments?
When Threats Occur
Before During After
Unauthorized access Computer hacking Keystroke logging Storing outside delivery system Testing interruptions Testing window Memorable items Student workstations Electronic devices Social media
www.caveon.com 18
Policies and Procedures
– How do state assessment programs implement computerized assessments? What policies, procedures, and practices have they put in place?
– Focus on Exam Administration:• Assessment monitoring• Proctor training• PITA Hotline• Online training modules (25% of which
deal with test security)• Resources for
staff/teachers/coordinators
www.caveon.com 19
Test Security Investigations
How can you conduct legally defensible investigations that deliver appropriate consequences?
• Having the right documentation in place
• Collecting all related information • Conducting interviews as needed• What are some of the consequences
www.caveon.com 20
New Resources Made Available at NCSA
www.caveon.com 21
New Resources Available
http://bit.ly/1CHUvsi
www.caveon.com 22
New Resources Available
http://bit.ly/1J71f8q
www.caveon.com 23
New Resources Available
http://bit.ly/1VlyARX
www.caveon.com 24
New Resources Available
http://bit.ly/1RBCKWm
www.caveon.com 25
Thank You!
Walt DraneState Assessment Director, Mississippi [email protected]
Steve AddicottVice President, [email protected]
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
Jamie MulkeyVice President, [email protected]
