Testing is Changing - Questionmark & ProctorU Briefing

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Testing is Changing: Practical and Secure Assessment

in the 21st Century

June, 2014

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Agenda

Slide 2

8.45– 9.30 Registration9.30 – 9.45 Welcomes and introductions

9.45 – 10.55 Testing is Changing: Practical and Secure Assessment in the 21st CenturyDemonstration

10:55 – 11:15 Networking Break11.15 – 12.20 Flexibility and Security – From Test

Authoring to Delivery and ReportingDemonstration

12.20 – 12.30 Q&A


• Founded in 1988

• Offices in US, UK, Canada; Distribution partners in 27 countries

• Questionmark provides technologies and services that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.

Background

• 1,000 current enterprise customers worldwide

• Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers

Customers

About Questionmark


• Assessment Management System

• Questionmark OnDemand (SaaS/cloud)

• Questionmark Perception (installable software)

• Multilingual

• Localized for dozens of different languages and cultures

• Global 24/7 Support

Products and Services

About Questionmark


Che OsborneVP of Sales

Kerry WilliamsRegional VP Sales, Europe

Ivan ForwardAssessment Solutions

Consultant

Chloe MendoncaMarketing

Introductions: Questionmark

Slide 5


Don Kassner Matt Jaeh

Introductions:

Slide 6


How Testing is Changing

in the 21st Century

Presented by Don Kassner – President, ProctorU


33% of all job openings and nearly 50% of all new jobs created between 2008 and 2018 will require a postsecondary degree or credential.

How can organizations keep up with the demand?

An Educated Work Force

http://www.mcgraw-hillresearchfoundation.org/wp-content/uploads/the-return-on-investment-from-adult-education-and-training.pdf#page=6


70% of companies are planning to implement mobile learning in the next 2 years.

Learning in the 21st Century e-Learning - FOUR out of FIVE

companies are reducing costs, increasing the speed of engagement with learning and reducing time spent away from the business.

http://towardsmaturity.org/article/2013/01/15/towards-maturity-focus-report-mobile-learning-work/http://corporateelearningconsortium.com/wp-content/uploads/2012/12/Leadership-research.pdf

70%

http://towardsmaturity.org/article/2013/01/15/towards-maturity-focus-report-mobile-learning-work/

http://corporateelearningconsortium.com/wp-content/uploads/2012/12/Leadership-research.pdf


Certification testing is evolving from linear-exams to performance-based assessments

Employers want something that is outcome-based

Evolution of Certification


Long distance and associated travel costs

Not 24/7

Cannot meet scalable demand

Requires time away from business

Requires time-consuming post-exam review

Cannot catch integrity breaches in real-time

No live technical support

Secured Assessments

Onsite ProctoringAutomated Remote

Proctoring


Secured Assessments

Ability to address integrity breaches as they occur

Live technical support

24/7 operation

Multiple location redundancy

Convenient testing from home / workplace

Provides a human element of customer service

Live Online Proctoring


Identity Verification

Identity Verification Attendance Verification

Government-issued identification

Keystroke analysis

Public Records challenge-based quiz

Facial-recognition technology

Fingerprint scanner (session entry verification, not identity)


Password-protected exams

Secure Learning Management Systems

Multiple test banks

Limiting or eliminating “Slippery Slope” exams

Exam Structures


Tools to combat major sources of online plagiarism

Create a culture of assessment integrity among administrators and test-takers

Tighten proctoring requirements

Additional Preventative Measures


Demonstration

Slide 16


What questions do you have?

Slide 17


Break


Ensuring Assessment SecuritySecure Technology and Good Practice


Software as a Service

Installable Software

Assessment Management System


European Data Centre

Slide 21

ISO 2700 and 9001 accredited

State-of-the-art datacentre

SaaS / Rapid Deployment

24/7 Monitoring

Global 24/7 support available

99.9+% Uptime


Protection of candidate/student PII (Personally Identifiable Information)

Protection of valuable assessment content

• Average corporate test: 15,000 GBP

• Average certification test: 100,000 to 200,000 GBP

Protect integrity of test/exam results

• A lot could be on the line…

• Reputation

• Life and Limb

Key drivers for secure assessment delivery

Slide 22

Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 23

Stakes of assessmentLicensure exams

Certification exams

Summative Assessments

Placement tests

Pre-certification exams

Job Task analysis surveys

Needs analysis surveys

Post-course test

Pre-course test

Formative quizzes

Course Evaluations

Satisfaction surveys

Stak

es o

f as

sess

men

t

Mo

tiva

tio

n t

o c

om

mit

fra

ud


Pre-employment

Public

Certifications

& Licensing

Regulatory Compliance

Sales and Technical Channel

Verification

Threat Level in Higher StakesShort Term with Low Trust Relationships

Long Term with High Trust Relationships

Larg

e P

rogr

ams

Smal

l Pro

gram

Higher threats require more

“Oversight” and so cost more

Lower threats require less “Oversight”

and so cost less to administer

Slide 24


Impersonation Content Theft Cheating

Key Threats to be Addressed in High-stakes Exam Delivery

Slide 25


Tight controls over the access to content

Shuffling items and choices –limit exposure of item pool

Not exposing the scoring algorithm beyond the content repositories/databases

Securing, and only providing limited access to, the content repositories/databases

Enforceable candidate agreements

Vigorous follow up on infractions

Mitigate Leakage of Content

Slide 26


Randomization

…of choices …of questions

Slide 27

Helpful for…• Re-use of items in practice tests/quizzes• Mitigate cheating/copying from “wandering eyes”


DemonstrationSecure, Collaborative Authoring

Slide 28


Enforceable candidates agreement; formal honesty contracts

Monitoring (Proctoring/Invigilation)

Secure browsers/players on candidate devices

Mitigate Cheating

Slide 29


Questionmark SecureOne Setting – Multiple Platforms

Slide 30

Windows

Mac

iPad

Printing

Screen capture

Task switching

Control/system keys

Right-click / context menu options

View source

Accidentally exiting exam

Prevents….


Test Centers

Events (Classrooms or Conventions)

Dedicated Kiosk with multiple cameras

Remote real-time webcam

Record & Review / 360 or webcam

Un-Monitored

Options for Monitoring Tests

Slide 31


•Personable, consistent

•Monitor vulnerable to unfair influenceTest Centers

•Convenient, personable, consistent

•Monitor vulnerable to unfair influence

Events (Classrooms or Conventions)

•Monitor is less vulnerable

•New and not yet widespread

Dedicated Kiosk with multiple cameras

•Monitor is less vulnerable

•Easy to deploy for use at home

Remote Real-time webcam

•Authentication and forensic benefits

•Nothing for content theft

Record & Review360 or webcam

•Works for employees

•Nothing for content theftUn-Monitored

Monitoring Tests Securely

Slide 32

ID F

rau

d

Pro

tect

C

on

ten

t

Min

imiz

es

Ch

eat

ing

√ √ √

√ √ √

√ X

√ X X

√ √ √

√

√ √ √


Reporting and Analytics

Slide 33


Analysis

How are our Assessments doing? How are our Participants doing?

Did the learning investment impact

performance? Where are the skills and knowledge gaps in my

organization?

Are my questions and tests valid and reliable?

What did participants think of the learning

experience?

What did participants learn from the training

or course?

How do I coach participants to improve

performance?

How did one group of test takers compare to

another?


Analyze Time to Complete Exam

Slide 35

Valuable completion time analysis useful to:

• Identify potential cheating• Flag potential content theft• Determine if allotted time is

sufficient for completing the assessment


How do aggregate results compare at one center vs. another?

Analyze Results by Test Centre/Location

Valuable for spotting problems with invigilation


DemonstrationReporting and Analytics

Slide 37


Providing a Great Candidate Experience

Slide 38


Turnkey web portal solutions for awarding

bodies

Configured and branded to your organisation’s

needs

Powered by comprehensive, scalable

and secure platform

Questionmark Portals

Slide 39


An Example Portal

Slide 40


OnDemand Portals On Your Design

Your Logo and Your Color Scheme

Slide 41


Simple For Candidates To RegisterSimple registration

Slide 42


Simple To Sign InSimple Sign In

Slide 43


Menu OptionsDifferent Menu Options for Guests and Candidates

Slide 44


Highlight Key Messages

Quick links to important messages

Slide 45


News Feeds

Latest News from your news feeds

Slide 46


Your Pages

You manage your page content

Slide 47


Certificate

Slide 48


Awarding body needs to reach a wide global audience; physical test centers do not reach global audience and are too costly

Awarding body still needs exams to be monitored Candidates can take exams online anywhere in the world,

but must book an appointment with a remote monitoring service first

Remote monitoring service is responsible for Visually identifying/authenticating candidate/securing exam

environment Generating exam launch URL Monitoring exam in real-time

Customer Story: Awarding body portal

Slide 49


Candidate Portal – Candidate Process


Candidate Portal – Monitor Process

Slide 51

Registration ID

Questionmark OnDemand Tenant ID

Launch URL


Configurable, branded portal pages

Self Registration

Automated scheduling

Payment processing

Keep candidates informed

Certificate Generation

How portals benefit certification programs


What questions do you have?

Slide 53


Complete your evaluations for prize drawing!

Slide 54


www.questionmark.com/whitepapers

Recommended White Papers

Slide 55

http://www.questionmark.com/whitepapers


Keep in Touch

Slide 56

blog.questionmark.com

www.questionmark.com/newsletter

twitter.com/questionmark

linkedin.com Look for Questionmark under “Groups” and join

youtube.com/questionmarkvideo

flickr.com/questionmarkpictures

facebook.com Become a “fan” for news, updates and more

http://www.questionmark.com/newsletter


Learn more about ProctorU

Slide 57


Thank you for attending!

Slide 58