Embed Size (px)
DESCRIPTION
Join us for some groundbreaking exam security developments Several members of the Caveon team just returned from the first annual Conference on Test Security in Iowa City, where a gathering of industry experts presented information and key developments that are essential in keeping the road to secure exams soundly paved. Caveon's Steve Addicott and John Fremer will share messages from the sessions they presented, and what they learned from others during the conference. Steve and John are industry veterans, and have decades of experience in test security. They know what security lessons are most important to testing programs today. They will be joined by industry rock star, Rachel Schoenig, Assistant Vice President and Head of Test Security at ACT, who presented several sessions on the agenda, including one focused on measuring the effectiveness of your test security program. In addition, Steve and John attended and presented at the European Association of Test Publisher's (EATP) annual conference in late September, and will share a few lessons learned from the international gathering as well. This could be the most important webinar you attend this year.
Citation preview
Caveon Webinar Series
1
Lessons Learned at
The Conference on Test Security
& the European Association of
Test Publishers Annual
ConferenceRachel SchoenigAssistant Vice President,
Head of Test Security
ACT
John Fremer
President
Caveon Consulting Services
www.caveon.com
October 16, 2014
Agenda for Today
• EATP
• Sample Session John Fremer
• COTS
• What is COTS? Rachel Schoenig
• John Fremer Takeaways
• A Test Security Framework, Rachel Schoenig
• Q&A
2www.caveon.com
Overview of EATP
3www.caveon.com
EATP
• Annual Meeting of European Testing
Professionals and Others
• 2014 Theme – “Growing Talent in Europe:
Gaining Advantage Through Assessment”
• Primary Focus is Industrial/Organizational
Psychology
• Role of Test Security is being Increasingly
Recognized
4www.caveon.com
EATP Session – Learning from the
Past
• Presenters:
– Steve Addicott – Caveon
– Ardeshir Geranpayeh – Cambridge Assessment
– John Fremer – Caveon
• Lessons drawn from Many Centuries
– “The more things change, the more they remain
the same”
5www.caveon.com
Learning from the Past (cont)
• Cheating on the Imperial China Selection Exam could lead to Execution
– This helped keep cheating to “only” about 60 per cent of participants
• In the Ancient Olympic Games, Cheating Athletes
– Fined
– Monies used to build statues of Zeus
– Each statue - offense, athlete’s name and city
6www.caveon.com
Learning from the Past (cont)
• As is the case now, cheating was discouraged
and punished
• Rules about Testing were Disseminated
• Special Efforts Devoted to Proctoring
• Some Cheating Tools seem “Modern,” e.g. a
cheating cloak containing impermissible info
• Publicity about Misbehavior Used as a
Deterrent
7www.caveon.com
Session: The Future of Test Security
8
• Presenters– Fremer and Addicott
– Phil Dickison, NCSBN
• Key issues– Technology Based Testing
– Growth in Testing International
Career Readiness
– Privacy
www.caveon.com
Security – The Validity Paradigm
The New Paradigm
Security
9www.caveon.com
The Conference on Test Security
10www.caveon.com
11
• Presenters
Wayne Camara, ACT – the New Standards
Ray Nicosia, ETS – Management of a Major
Test Security Unit
Phil Dickison, NCSBN – Focus on Test
Validity
John Fremer, Caveon – Growth of our
Profession and Industry
Plenary – How has our approach to
Test Security Evolved
www.caveon.com
COTS – Working a Case
Best Practices in Test Security investigations
• Presenters:
– A. Benjamin Mannes – ABIM
– Rachel Watkins Schoenig – ACT
– Marc Weinstein – Dilworth Paxton
– John Fremer – Caveon
12www.caveon.com
The Call….
• Parent called the State DOE and Test Publisher hotline
• 7th grade daughter said her teacher gave them answers to questions on their year-end statewide assessment tests
• Another teacher was present in the room when this occurred
The Case:
13www.caveon.com
Consider what outcome is desired:
• Ensuring score validity
• Deterring and stopping inappropriate future
behavior
• Maintaining public confidence in test sponsor
• Determining individual culpability and/or
accountability
• Taking action from employment, education,
certification and/or licensure perspective
Case – Investigative Goals?
14www.caveon.com
So You Flagged a Cheater Now
What
• Presenters
– Aimee Hobby Rhodes - CFA Institute
– Kellie Early - National Conference of Bar
Examiners
– A. Benjamin Mannes - ABIM
– Joy Matthews Lopez - NABP
• Panel Discussion of Key Questions
15www.caveon.com
Flagging a Cheater (cont)
• Importance of Very Tight Candidate
Agreements
• Collect and Deliver Information in Writing
Only
• Not Enough to have very Good Data
Forensics Information – You have to
explain it (Like “how do you get to Carnegie
Hall – Practice, Practice, Practice”)
16www.caveon.com
Certified Exam Security Professional
• Certified Exam Security Professional (CESP)
• A “certification in the making”
• Route to a Credential that recognizes Test
Security Expertise
• Series of Credentials Envisioned beginning
with “CESP Generalist”
• Curriculum Targets and Learning Materials
and References
18www.caveon.com
Session: Protecting our Tests – the
Past, Present & Future of Test Security
19
• Presenters
– Steve Addicott
– Rachel Schoenig
• Takeaways
– Privacy
– Career Readiness
– New Designs
– Test Security Training & Development
www.caveon.com
What is happening?
Testing is
GROWING…incredibly
20www.caveon.com
Why? We Have the Gold Bars
21www.caveon.com
Issues Across Industries
0
2
4
6
8
10
12
14
16
Music Sales
1999
2009
22www.caveon.com
0
200
400
600
800
1000
1200
Counterfeit & Pirated Products
2008
2015
Issues Across Industries
ICC 2011 and OECD 2009 reports
23www.caveon.com
Protecting our gold bars
24www.caveon.com
Protecting our gold bars
Reputation
25www.caveon.com
Holistic Test Security
Framework
How to Protect our Tests
26www.caveon.com
Testing Process “Silos”
Test
Development
Test
Publication
Test
Delivery
Score &
Report
Insert Test Security Here
27
Holistic Test Security Framework
Deter
DetectDecide (now and for the future)
28www.caveon.com
How to Protect our Tests
Development
Publication
AdministrationScoring
ReportingDeter
Detect
Decide
29www.caveon.com
Thank you!
30
Follow Caveon on twitter @caveon
Check out our blog www.caveon.com/blog
LinkedIn Group “Caveon Test Security”
Rachel SchoenigAssistant Vice President,
Head of Test Security
ACT
John FremerPresident, Caveon Consulting ServicesCaveon Test Security
www.caveon.com
