Embed Size (px)
Citation preview
Caveon Webinar Series:
Lessons Learned at NCSA and ITC
July 24, 2014
David FosterCEO, Caveon Test Security
Steve AddicottVice President, Caveon Test Security
John OlsonPresident and FounderOlson Educational Measurement& Assessment Services
Agenda for Today
NCSA• John Fremer Takeaways• Security Sessions• John Olson Survey of States
ITC• Dave Foster Takeaways• Future of Test Security
Overview of NCSA
www.caveon.com 4
NCSA Lessons Learned - General
The Speed of Change in State Assessments is Breathtaking Common CoreRTTTAssessment ConsortiaTeacher Accountability
www.caveon.com 5
NCSA Lessons Learned - General
Technology-Delivered Tests are Becoming the NormGood news
– No “erasure parties”– Innovative designs and items
Bad News– New = Unfamiliar– Infrastructure challenges = long test windows
www.caveon.com 6
NCSA Lessons Learned - General
State Interest in Test Security Continues to Grow
Special TILSA SCASS workgroup on Test Security
www.caveon.com 7
NCSA Lessons Learned - General
State Interest in Test Security Continues to GrowSeveral sessions
regarding Test Security– More every year
www.caveon.com 8
NCSA Lessons Learned - General
State Interest in Test Security Continues to Grow“Balancing Test Security and Accessibility
on Next Generation Online Assessments”
“Systemizing and Improving Test Security”
“Preventing and Detecting Cheating in Statewide Assessments – How Are We Doing and What’s Next?”
www.caveon.com 9
Top Ten Security Directions
1. Take a Proactive Stance2. Comprehensive
Security Audit3. Protect Investment in
Items4. Improve LEA
Monitoring5. Improve Administration
Practices
www.caveon.com 10
Top Ten Security Directions , Cont.6. Employ Data Forensics
7. Test Security Handbook8. Follow Up on All Testing
Irregularities9. Make Validity of Test Results
the Primary Goal10. Draw on Multiple Resources
www.caveon.com 11
Test Security Standards and Resources
• Testing and Data Integrity in the Administration of Statewide Student Assessment Programs (NCME, October 2012)
• TILSA Test Guidebook: Preventing, Detecting, and Investigating Test Security Irregularities (CCSSO, May 2013)
• Handbook of Test Security (Routledge, 2013)
www.caveon.com 12
Test Security Standards and Resources, cont.• Operational Best Practices for
Statewide Large-Scale Assessment Programs (CCSSO, 2013)
• Test Security and Students with Disabilities: An Analysis of States’ 2013-2014 Test Security Policies (NCEO, June 2014)
• Revised APA/AERA/NCME Standards for Educational and Psychological Testing, (July, 2014)
www.caveon.com 13
What Are States Doing to Prepare For the Next Generation of Assessments?
Planning For 2014-2015 and Beyond John Olson
with Barry Topol (ASG), Juan DeBrot (WV),
Roger Ervin (KY), and John Weiss (PA)
National Conference on Student AssessmentNew OrleansJune 25, 2014
www.caveon.com 14
Overview• In 2012, ASG initiated a new state
assessment survey program that focused on state planning for 2014-15 when new state- and consortia-led assessments are slated to be implemented.
• In 2013, a second round of state surveys took place May-August with a total participation of 42 states, compared to 33 states in the first round.
• The increase in the number of states surveyed was driven partially by the desire of states to know more about what other states are doing to prepare for 2014-15.
www.caveon.com 15
2013 Survey ResultsThe latest survey topics included:
• plans and recent changes in state assessment programs and consortia membership
• funding and costs for current and future assessment components
• state implementation of technology • future plans for assessment sustainability • test security issues • other issues related to the assessment consortia
and plans for the future
Data on all these topics were shared by presenters in the session to a SRO crowd of >100 people.
www.caveon.com 16
Highlights of State Survey Findings• Technology implementation was still the #1 concern of states;
however states reported some encouraging progress on using technology
• Costs for the new assessments continued to be a concern to some states
• Many states said they were “fully committed” to their consortium; at the time Plan A for most was still the PARCC or SBAC assessment, Plan B usually was to continue with current state assessment, possibly revised to align with CCSS
• States said test security was becoming an increasingly important issue and concern for them, and many useful new documents and resources were mentioned, e.g. CCSSO TILSA Test Security Guidebook, NCME whitepaper on test integrity, Operational Best Practices report, etc.
Summary and Conclusions• Since the survey was conducted, anti-
Common Core backlash (national test, cost, student data privacy) now taking hold, leading to several states recently dropping out of the assessment consortia– More states could leave the consortia in the
coming months– Some test vendors are also developing common
core based assessments as an alternative to the consortia
• States want rigor AND reasonable cost• Consortia developed tests are seen as the
highest quality alternative by most states
17
Summary and Conclusions
• States appear to making good progress in moving to online assessment. The future may finally be getting here. – Almost all new assessment implementations are
online– Roughly 2/3 of surveyed states are doing
significant testing online• However, states still extremely concerned with
their ability to implement full scale online assessment– State funding for technology is not forthcoming. – Strange things seem to happen when
implementing OLA at high volumes. Recent experience in online testing is not comforting (e.g., OK, MN, IN, KY) although consortia field tests went well. 18
Summary and Conclusions• States very interested in detailed cost data for
current and future assessments. This was shared from the survey.
• States appreciate getting the data from our survey and having a broader perspective on the issues affecting many states.
• Another state assessment survey will be done by ASG in 2014.
• State assessment staff who presented in the session provided numerous examples of how they are preparing for 2014-15 and beyond. Issues discussed included CCSS, changing standards, PD for teachers, revised and/or new assessments, OLA challenges, costs, legislative decisions, state politics, and test security.
19
Lessons Learned from ITC
20www.caveon.com
A new set of international guidelines: The Security of Tests, Examinations, and Other Assessments These can be found at: http://www.intestcom.org/guidelines/index.php116 guidelines that cover
– Planning for security– Implementing security– Dealing with a security breach
Example: Security Planning Guideline #5 (of 21)Security rules should be indicated clearly in the security plan and communicated to all interested parties. Consequences for violations of those rules should be clear.
Example: Security Planning Guideline #18.g (of 73)
Proctors (invigilators) should not have an interest or stake in the test outcome. They should not be instructors or teachers for test takers nor familiar with the content covered by a test.
Example: Responding to a Breach Guideline #4 (of 22)
Scores shown to be inaccurate as a result of test fraud should be cancelled or invalidated.
ITC Keynote Address
Technology and Test Security?
Top 10 Countdown of Current Technology Use in Testing
3 of the 4 are security related
ONLINE PROCTORING#4
DATA FORENSICS#3
DISCRETE OPTION MULTIPLE CHOICE
#1
Future Technology Effects
Technology
NOWLET’S PEEK
KEEP IN MIND
TECHNOLOGY IS A TWO-EDGED SWORD
MICROCAMERAS IN CONTACT LENSES
EFFECT: BETTER CHEATING!PLAUSIBLE!2 YEARS AWAY
SECURE TEST AND ITEM DESIGNS
EFFECT: CHEATPROOF TESTSPLAUSIBLE!5 YEARS AWAY
MONITORING TECHNOLOGY
EFFECT: AUTOMATED PROCTORINGPLAUSIBLE!5 YEARS AWAY
BETTER PRINTERS
EFFECT: UNDETECTABLE FAKE ID’SPLAUSIBLE!1 YEAR AWAY
Security Investigations
• What is needed to prepare to conduct an investigation of a security incident by John Fremer
• Using statistics as part of an investigation by Ardeshir Geranpayah
• Conducting an investigation by Marc Weinstein
From John FremerThe Basic Goal of an Investigation
To establish the validity of test scores
Can they all be trusted? Which ones? Gathering details and
evidence
It’s NOT about punishing individuals
From Ardeshir Garanpayeh, University of Cambridge
“Where cheating is seen to be widespread, even honestly obtained test results may lose credibility and certificates become devalued”
“Once cheating is detected, an action has to be put in place to:
Stop fraudulent use of test resultsDeter future cheaters”
40
PROTECTTECT
INVESTIGATE ENFORCEFORCE
“Investigations are a Vital Component of an Effective Exam Integrity Program”
From Marc Weinstein of Dilworth Paxson
“Interviews Must Be Conducted In Person.”
The only hope you have of assessing the truthfulness of a statement is by looking the person in the eye and watching his or her body language.
Nonverbal communication is key.
From Marc Weinstein, cont.
www.caveon.com 42
A Decade of Test Security: The Past and the Future
• Eugene Burke, CEB/SHL • Ada Woo, NCSBN• Steve Addicott, Caveon
Validity Triangle
Content
Psyc
hom
etric
s
Security
Validity
www.caveon.com 44
Thank you!
Follow Caveon on twitter @caveonCheck out our blog www.caveon.com/blogLinkedIn Group “Caveon Test Security”
David FosterCEO, Caveon Test Security
Steve AddicottVice President, Caveon Test Security
John OlsonPresident and FounderOlson Educational Measurement& Assessment Services
