Embed Size (px)
DESCRIPTION
This year's Association of Test Publishers (ATP) Innovations in Testing Conference focused more than ever on test security, and the Caveon team was there. We share with you not only the concepts which we presented, but also new things we learned at the conference. Caveon leaders John Fremer and Steve Addicott summarize the test security ideas and strategies from ATP and the things you can do to protect your high-stakes tests.
Citation preview
Caveon Webinar Series:
Ten Test Security Lessons from ATP
March 19, 2014
John Fremer, Ph.D.PresidentCaveon Consulting Services
Steve AddicottVice PresidentCaveon Test Security
Agenda for Today
• Steve– Emphasis on Test Security– Highlights from some TERRIFIC Security Sessions– Online proctoring – Next Step in Exam Security
John– Exam Security Processes and Procedures– “Telling it to the Judge”– User Agreements– ATP Security Committee Briefing– Closing Keynote
Q&A
www.caveon.com 3
Test Security is…..
www.caveon.com 4
Test Security is Front Burner
www.caveon.com 5
Test Security is Front Burner
www.caveon.com 6
Ten Years of Test Security
Star-studded cast– Schoenig, Wollack, Cizek (and Addicott )
Key trends explored
Important “Lessons”
www.caveon.com 7
Schoenig—Test Security is a Process
ACT Process:
Deter, Detect, & Decide
Supports the 3-legged stool of
www.caveon.com 8
Wollack—Data Forensics Goes Mainstream
An amazing paradigm shiftPast: DF detects cheating = public distrust in
testing
Future: DF enhances validity = increased
public trust
Industry Standards for DF• Proprietary Dissemination
• CSDPTF COTS
DF moving closer to test instance
www.caveon.com 9
Cizek—Test Security is about Validity (not cheating)
What we have said:
“a statistical procedure... can never
prove that, for example, teachers in a
school are cheating”
“such statistical criteria are never ‘proof’
of cheating”
www.caveon.com 10
Cizek—Test Security is about Validity (not cheating)
What we can say:
“There is strong reason to be concerned that
these scores may not be valid. We recommend
that additional information be gathered or that
testing be replicated before interpreting these
scores to mean...”
www.caveon.com 11
“Exam Inoculation and Other Crazy Ideas to Stop Cheaters from Passing Exams”
Burns, Van de Velde, and Maynes
• Pirates persist
• Test Design is powerful!
• Most measures are reactive, so…
• Inoculate exams against pirates
– Trojan Horse Items
– Embedded Verification Tests
– Item cloning
– DOMC
www.caveon.com 12
Online Proctoring Is Here to Stay
• ATP SC orchestrated demo-rama
• Security concerns abound
• Are they justified?
• More research required
• Test Security Standards for Online Proctoring
now available
– http://bit.ly/1ifSv0F
– Caveon.com Resources White Papers
www.caveon.com 13
Next Step in Exam Security’s Evolution
Institute for Exam Security (I4ES)
• An independent, non-profit association
by/for Test Security professionals
• www.cespcert.org
www.caveon.com 14
Fremer Areas to be Covered
Multidisciplinary Approaches Workshop
“Telling it to the Judge”Critical Importance of User
AgreementsATP Security Committee
BreakfastClosing Keynote
www.caveon.com 15
Multidisciplinary Approaches Workshop
• Program Manager, Investigator, Attorney, Test Security Guy
• Four Group Exercises• Each Dealing with Real Challenges
in Testing Programs• Common Elements across
Discussions
www.caveon.com 16
One Exercise – The Context
Limited staffLimited staff expertiseLimited resources: software,
money, timeProblemEfficiently allocate resources
to threats based on – The risk they pose– The likelihood of finding a solution
www.caveon.com 17
Activity #2
Assess the Risks!
www.caveon.com 18
The ScenarioCBT administration, 3,000 examinees.
Following the administration, you analyze your data for statistical indicators of cheating.
180 examinees were statistically flagged for similar response patterns. They did not pass at a higher rate than expected.
Of those 180, a group of 80 examinees had been flagged in previous administrations. They did not pass at a higher rate than expected.
What are your next steps in addressing this issue?
www.caveon.com 19
Common Elements Across Reports
Notify Your Boss PromptlyCollect Relevant
Information Based on Your Response Plan
Seek Appropriate Expertise– Legal – Psychometric–Messaging/Communications
www.caveon.com 20
I Told It To The Judge, But S/He Didn’t Listen
Jennifer Semko – Baker & MckenzieCamille Thompson – ACTDennis Maynes - Caveon
www.caveon.com 21
I Told It To The Judge (cont.)
• Advice re testifying– Answer only what you are asked– Don’t be a smart alec– KISS Principle
• Courts have been Supportive• Follow Your Policies and Procedures
or Stay Home
www.caveon.com 22
I Told It to The Judge - User Agreements
User Agreement is a Binding Contract
Defines RelationshipMemorializes ObligationsYou will be Judged by How
Well you Fulfilled Your Commitments
Must Behave Responsibly and Fairly
www.caveon.com 23
Poll #4
www.caveon.com 24
ATP Security Committee
High Attendance at Breakfast
Great Deal of Productive Energy
Many Projects (Examples)– Removing Copyrighted Material from
the Internet– Test Taker Rights and Responsibilities
www.caveon.com 25
Closing Keynote
Hilary MasonBig Data – Tremendously
UsefulGives Power to Many to
Influence EventsIt is the Effective Holders of
Information who Can Wield Power
Some Problems Cut Across Fields – Unwillingness to Act on Data Alone
www.caveon.com 26
Overall Impression
Sharp Increase in Awareness of Importance of Test Security to Test Program Management
Other Domains Catching up to IT in Recognition of Problems
Focus on User Agreements and Online Proctoring
Willingness to Try New Approaches
www.caveon.com 27
Questions?
Please type questions for our presenters in the GoToWebinar control panel on your screen.
www.caveon.com 28
Thank you!
John Fremer PresidentCaveon Consulting Services
Steve Addicott Vice PresidentCaveon Test Security
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
