Upload
caveon-test-security
View
311
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Testing industry veterans John Fremer and Steve Addicott of Caveon are joined by Lou Woodruff, past president of the National College Testing Association to share their "lessons learned" from several of this summer's biggest testing conferences. For more information, please go to www.caveon.com
Citation preview
Caveon Webinar Series
“The Endless Security Summer: Testing Conference Lessons”
August 22, 2012
2
“The Endless Security Summer”
Steve Addicott• Vice President• Caveon Test Security
John Fremer, Ph.D.• President• Caveon Consulting Services
Lou Woodruff, Ph.D.• Senior Security Consultant, Caveon• Former President, National College Testing Association
3
Agenda
• Conference on Statistical Detection of Potential Test Fraud, May 23-24
• National Conference on Student Assessment, June 26-29
• International Test Commission, July 3-5• Dutch Exam Society, July 6• National College Testing Association, August 1-4• Closing Thoughts
4
Poll Question #1
5
Conference on Statistical Detection of Potential Test Fraud
• University of Kansas’ CETE; Lawrence, KS• First-ever Data Forensics Conference• Gathering of measurement experts &
researchers– Certification– Education– Vendors– Test Publishers– Academics
6
Conference on Statistical Detection of Potential Test Fraud
• Most important takeaway
• Second most important take away
Protecting our tests grows ever harder, and ever more important
THIS CONFERENCE EXISTS!!
7
National Conference on Student Assessment (NCSA)
• June 26-29 in Minneapolis, MN
• Council of Chief State School Officers (CCSSO)
• 2012 Theme: Teaching, Learning, and Assessment through the Next Decade
8
Top Three NCSA Key Takeaways
9
NCSA Key Takeaways
#3—Bombshell! CCSSO is partnering with Center for
Assessment to “re-focus” the conference– Reduced attendance?– Budgets?– Changing priorities?
10
NCSA Key Takeaways
#2—A number of dynamics in state assessments impacting security
– Teacher Accountability• Legislatures passing laws• Using existing assessments to now grade teaching performance
– Assessment Consortia
– Technology• Classrooms• Administration• Learning• Assessments
11
#1 NCSA Takeaway• Despite all these changes,
THERE WAS NOT ONE SESSION FOCUSED ON TEST SECURITY.
NOT ONE.
12
Topics That Could Have Covered…• Thoughtful piloting of security strategies
• By 2014-2015, all state assessments will be CBT– Colossal logistical challenge– Too few PCs and test centers
• Blending individual test programs – Dozens into two– Exams will be shared across states and time zones– Administration windows will last weeks– More technology, more sharing?
13
International Test Commission
• July 3 – 5, 2012
• Group representing testing organizations and national agencies world wide
• Amsterdam, The Netherlands
14
ITC – Major Takeaways
• Growing recognition throughout the world of the importance of protecting IP and preventing cheating
• Attention to standards – ATP, ITC, ANSI, Caveon
• More attention to how to use data to detect misbehavior
• “Trust but verify”
15
CITO & Caveon Session
• CITO – very major provider of testing services to Europe and many non-European countries
• Collaborating with other Dutch testing agencies to promote security standards
• Caveon – evidence of importance of security– Internationalization of testing– Challenges of technologically savvy cheats– Notorious US cases, especially Atlanta PS
16
Faking and Test Security
• Steve Addicott Contribution• Dealing with Media
– Be prepared– Draw on all parts of your organization –
communications, legal, membership, etc.– Have a spokesman– Tell the story from your perspective– Don’t expect the story to go away
17
Dutch Exam Society (NVE) Security Summit
• NVE has 400+ members from dozens of test programs in The Netherlands
• Eight leaders gathered to:– Address Test Security concerns– Consider adoption of Security Standards for
members
• Caveon invited to play a supporting role
18
Top NVE Security Summit Takeaways#3—Times are changing
– Test Security was not, but is now, a front-burner issue
#2—Security Risks are global
#1—Rising tide lifts all boats– Organizations which are considered
“competitive” can collaborate and create positive change
19
Poll Question #2
20
National College Testing Association
11th Annual Conference
Minneapolis 2012
ncta-testing.org
21
College testing administrators and staff
representing over 600 colleges
More than 50 companies that do business
in the testing field
ncta-testing.org
22
Consortium of College Testing Centers
Listing of hundreds of college testing centers
offering proctoring services for individuals
needing to take a secure, proctored examination
Access the listing for free on the NCTA website
ncta-testing.org
23
Six Sessions Devoted to Test Security:
The Keynote Address
The General Session
Four Breakout Sessions
24
Keynote Address
Mark Poole
Product Strategist and Security Expert
Pearson VUE
“The Future of Secure Testing in a Digital World”
25
Highlighted five key trends that are
shaping the next decade of testing:
o a global cheating culture
o high tech test fraud
o testing anywhere securely
o candidate authentication
o data forensics
26
“Demystifying Data Forensics: An Overview of the Logic Underlying Cheating Detection Techniques”
Jim Wollack, Ph.D.Director, Testing and Evaluation ServicesUniversity of Wisconsin, Madison
Presented the basic principles of applied data forensics in a way that everyone could understand.
Reviewed some forensics approaches to identify examinees who copy answers, had prior access to test content, or hired surrogates to test for them
Breakout Session:
27
“Gimme Shelter: Weathering the Media Storm of a Security Breach”
Steve AddicottVice President, Client ServicesCaveon Test Security
Ray NicosiaExecutive Director of the Office of Testing IntegrityEducational Testing Services (ETS)
Faisel AlamTest Security SpecialistLaw School Admission Council (LSAC)
Breakout Session:
28
This panel of security experts shared key aspects of media strategy for testing programs and techniquesfor positioning incidents to various audiences
High profile security breaches were reviewed, highlighting details of what went well and what did not
The use and effectiveness of various media tools was analyzed, and a checklist of media “Do’s and Don'ts” was presented
29
“Security Strategies for Every Stage of the Testing Process”
DeDe Hedlund Jeff PlaceCreighton University Questionmark
Looked at best practices for colleges to protect test content and minimize cheating
Demonstrated the potential for using mobile test centers with examinees testing on tablets such as the Apple iPad
Breakout Session:
30
“Innovations in Online Proctoring”
Rick Beaudry Denis GravelCEO Director, Workforce Development B Virtual Inc. School of Applied Technology
Presented a proctoring service that proctored test takers in any location via web-cam and microphone, with a proctor monitoring the session remotely
Breakout Session:
31
“Detection and Prevention of Test Fraud
The State of the Art”
John FremerPresidentCaveon Consulting Services
Lou WoodruffSenior Security ConsultantCaveon Consulting Services
General Session
32
o Perspectives on the Perpetrators
o Test Security Anatomy
o Contemporary Test Security Responses
o The Technology of Detection
o Large Scale Security Investigations
o Future Test Security Challenges for Higher Ed
o Implications for College Testing Professionals
33
Survey data on the prevalence of cheating among the nation’s youth was shared which showed an alarming level of cheating
Cheating has grown from a strategy for increasing test scores to the manifestation of a growing social norm that accepts dishonesty in many areas of life
The cheating/detection arms race will continue with advances in biometric, data forensic and test delivery techniques enhancing prevention and detection
34
Result of Using Data Forensics in a State Assessment Program
Data Forensic Analyses of school districts statewide
Decisive actions taken on the forensic results
Two years of follow-up test administration monitoring
Found a 72% reduction in test security irregularities
Reduction in "Irregularities"
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
Rep
ort 1
Rep
ort 2
Rep
ort 3
Rep
ort 4
Rep
ort 5
Rep
ort 6
Rep
ort 7
Rep
ort 8
Rep
ort 9
35
The Atlanta Public Schools
Test Security
The Perfect
Storm
36
Test Security
The Perfect
Storm
Test Security
The Perfect
Storm
Mandated assessmentstied to federal funding
Schools and teachers evaluated on test scores
Administrators feeling unjustly treated
Test security left in the hands of the teachers and the principals
37
Governor brought in the GA Bureau of Investigation
Sixty-one agents on the case with subpoena power
District Superintendent, all Deputy Superintendents, and many Principals and Teachers were removed
Who do you trust? Sometimes – no one!
Atlanta Public Schools Test Security Case
38
Take-Aways from the NCTA Conference
College testing professionals are highly aware of and strongly interested in test security
NCTA is a valuable platform for promoting security related initiatives and partnerships between colleges the testing industry
Colleges are and will remain a rich resource for implementing cheating detection and preventionstrategies
39
Questions?
40
Today’s Presenters
Steve Addicott [email protected] John Fremer [email protected]
Lou Woodruff [email protected]