History, Current Practice and Perspectives, and Predictions for the Future of Computer Based Assessment in K-12 EducationJohn PoggioUniversity of Kansas1(Ancient) History circa the 1970s, 80s & 90sImpersonal, frightens students, computer anxiety, lack of familiarity, equivalence to P&P at best, a dramatically different experience that guarantees the two modalities are not equivalentCAT will restrict problem solving and only support solutions in a strict linear manner

However, study as early as 1972 through the turn of the century research found no differencesAdvantagesImmediacy of results

Among the most notable features associated with computerized testing is the immediacy of results. In most systems as soon as a student finishes taking the test, scores are ready for display. Delays and slowness in reporting results can be a concern of the past. AdvantagesCost efficiency

Moving tests to computer delivery will signal a reduction in costs. While upon first consideration this may not seem obvious, production of test booklets and answer sheets are extremely large expenditure items that can be set aside.AdvantagesAccuracy in data and data collectionCapturing information from students on answer sheets is among the biggest NEGATIVES of the testing process. A casual review of information gathered on answer sheets reveals the considerable number of errors that students make. Answers to questions and the information provided, such as gender, race, age, and even name are frequently miscoded and mis-marked by students. Computer-based assessment all but eliminates this error in the measurement process.AdvantagesStudent motivation

A number of researchers and field practitioners have commented on heightened level of student motivation when assessments are taken online. This in turn translates into higher performance. Whether such motivation will be seen over time is not clear. However, at this time, students appear more attentive, comfortable, and responsive to computer rather than paper and pencil assessment. AdvantagesAdaptive alternative

Moving traditional tests to computer administration has opened the door to computer adaptive testing. This approach has the advantage of reducing traditional tests by half to as much as 75% of the original length. In this methodology, while we have considerable reduction in test length, there is relatively little loss in test properties as validity and reliability (and, assessments may be fairer!)AdvantagesReduced administrator and instructor effort

As the tests will be administered online, the traditional activities by administrators and teachers receiving, unpacking, securing, counting, sorting, and distributing test booklets and answer sheets is eliminated. This management effort that requires considerable commitment of time and attention is all but eliminated when we go to computerized testing.AdvantagesMeeting the needs of special populations

Technological advances used with computerized testing signals the dawn of a new era in testing for students with special learning needs and other populations (non native speakers). Largely unstandardized accommodations of the past can now be transformed into common and fixed procedures, for example, increasing text size with the click of a button, transforming text to speech, zooming, colorizing objects, English/native language translations, ASL avatars for the hearing impaired, etc.AdvantagesReady support for data based decision making

The speed with which data can come available to educators allows rapid analysis and evaluation of performance. Whereas in the past it could take three months for data to be returned to administrators and teachers, todays computerized tests allow educators to review results and while the information is fresh and current make instructional and evaluative decisions on behalf of students.AdvantagesStore information, ready access to results, and create archives

As students take a test on a computer, the storage and retrieval of information is efficient and speedy. In the past reliance on answer sheets meant data cleanup, scanning and processing, scoring and check up, and eventual readying of information for storage and retrieval. But with computer-based testing, the storage of information is part of the process and access of that information is efficient and accurate.AdvantagesAbility to modify tests as necessary

In the past, when an error was discovered in a test booklet, the educator had little recourse but to stand by and watch the mistake time after time during administration. In today's computerized tests when an error is found it can be corrected immediately. In this way, test scores are improved for many students.AdvantagesImproved security some risks

As tests are delivered online, traditional handling of test materials is eliminated. Breaches of test security are fewer. There are threats related to hacking that are not avoided. However, up to this time there have been few reports of security issues because of computerized administration.AdvantagesIncreased opportunity for collection of supplemental information or data

Computerized testing allows for and supports the collection of additional information. Students can be quickly and easily surveyed or additional data can be gathered. Research and evaluation can have a place in assessment once more!AdvantagesEqual (..Stronger?) PerformanceResearch of the past 25 years is clear and certain: paper and pencil testing yields scores equivalent to computer-based testing. We should continue to study these effects. But, the real effect to watch is, with student gaining CBT experience, will CBT scores become stronger than P&P performance? Disadvantages:Implementation BarriersLocal resistance to change

Change can come very hard or very slow for some. This new technology brings with it a demand for new skills and understandings that must be mastered. In addition, some will see no need to change from the way that testing has been done. Careful, thoughtful steps must be planned to assist individuals make the transition. Without assistance and help, change will be very slow for a few. Individuals must be brought into the new technology and made to feel comfortable and be supported.DisadvantagesLocal capacity

A school or school district may not have the wherewithal to move to online testing. Equipment may not be current or nonexistent, or local staff may not have the expertise to venture into this arena. The inability to move forward with the innovation will in its own way limit adoption and understanding. Fear of this unknown needs your deliberate attention.DisadvantagesRisk Not changing horses

These are high stakes tests and failure may not be tolerated. Careers are on the line and shifting to a new mode of testing may not be seen as wise. Again, awareness and sensitivity are important considerations; plans should be prepared with these factors in mind.DisadvantagesMinimize or eliminate obstacles

Perceived impediments to implementation of computerized testing will add to frustration and dissatisfaction. When systems are new to the teacher and testing is pressing forward with the threat of accountability, tolerance for the unexpected will be very low. Make the systems: work, understandable and forgiving. Keys to Future SuccessInvolve the (state, district and school) technology staff

Schools must support such personnel, and they must become involved from the outset. See to it that they work closely with educators, and that educators are comfortable reaching out to them for assistance and support. Remember: involve and train all!Keys to Future SuccessTraining workshops

Plan training sessions periodically, but frequently, during the semester - as you have staff, money and time. Get teachers working at the same grade and across grades solving problems together with technology staff together! The sessions can be brief - limited to one or two hours each.What of Staffing?Change in roles and needs...

From P&P staff to...

Programmers and web designers, and

Office staff who are computer literate and savvy, ...and who have patience and understanding of others who are lost, truly lost...Keys to Future SuccessTraining materials

Development and utilize student, parent, and teacher/administrator tutorials, student training and practice tests, and implementation guides for teachers and administrators.Keys to SuccessCommunication

Frequent, brief and effective communication is essential. Share information about the program and its activities. Constant communication (weekly newsletters, announcements, sharing and postings) ...email and the web are your friends! Build a community of knowledgeable and involved educators, and share the successes and failures encountered.About Cost...Running Dual Programs is very, very expensive... about twice the cost initially!

Support of the infrastructure to do online testing must not be shortsighted... It must be planned forWhat you once spent on paper, the attention of staff, storage and mailing/ shipping, you will now spend on servers and programmers!And now, for some illustrations of the real thing.

GRE Quantitative

GRE Vebral

GRE Writing

TOEFL CBT Structure

Basic MC item.30TOEFL CBT Structure

Word selection item.31TOEFL CBT Reading

Basic passage-based MC item.32TOEFL CBT Reading

Word or phrase selection (hotspot) item.33CTB Acuity Reading Comp

CTB Acuity Formative System

CTB Acuity

ITS Reading

ITS Short Performance

AIR Math

AIR Reading

AIR Limited CR

NWEA Math

NWEA

NWEA

Pacific Metrics

Pacific Metrics Math

Pacific Metrics Reading

Pearson Math

Pearson Reading

And (of course): Computerized Assessments and Learning (CAL)

http://cal-demo.caltesting.org/ CAL Math

CAL Math

CAL Math

CAL Review

CAL Reading

CAL Reading

Computer-Based Assessment:

Future Considerations57Comparability...Research and Evaluation... Yes, absolutely...but enough is enough!

Is P&P the Gold Standard?... It must not be!

And, research advises it has not been an issue Advice: study the engine, then generalize

Tomorrows Innovations...Research directed at teacher, all student, educator, parent audiences

Constructed Response formats: yes; Adaptive: of course; but... AI/ Automated Scoring, wow!!

Speaking as the response input and,

Use the technology to link actively assessment to instructionInstruct4Success

Instruct4Success

Instruct4Success

Instruct4Success