2010 R.M. Luecht 1

Operational CBT Implementation Issues:

Making It HappenRichard M. Luecht, PhD

Educational Research MethodologyUniversity of North Carolina at

GreensboroTenth Annual Maryland Assessment Conference: COMPUTERS AND THEIR IMPACT ON STATE ASSESSMENT: RECENT HISTORY AND PREDICTIONS FOR THE FUTURE. 18-19 October, College Park MD

2010 R.M. Luecht 2

What do you get if you combine a psychometrician, a test development specialist, a computer hardware engineer, a LSI software engineer, a human factors engineer, a QC expert, and a cognitive psychologist?

A pretty useful individual to havearound if you’re implementing CBT!!!!

2010 R.M. Luecht 3

A Naïve View of Operational CBT*

* Includes linear CBT, CAT, CMT CAST and other variants

ku,,uUjk Rj:ˆImaxikiij

11

Item Selection/Test Assembly Algorithm

,:u,,ugmaxˆkkii ii

MAPuu 1111

Ability Estimation/Scoring Algorithm

Item #001Item #001

Item #001Item #001

Item Bank

Ui=010120113Response Vector

Test Delivery Network

Ethernet

Server

Data

Data Server

Data

Examinee

2010 R.M. Luecht 4

The Challenge of CBTMoving more complex data more quickly, more securely and more accurately from item generation through final scoringImmediate responsiveness where possibleRe-engineering data management and processing systems: end-to-end99.999999% accuracy and eliminating costly and error-prone human factors through automation and better QC/QA

2010 R.M. Luecht 5

Systems Impacted by Redesign and QC/QA

Item development and bankingTest assembly and compositionExaminee eligibility, registration, and scheduling, feesTest deliveryPsychometrics and post-examination processing

Item analysis, key validation and quality assuranceTest analysisFinal scoring, reporting and communication

2010 R.M. Luecht 6

Item and Item Set Repositories

2010 R.M. Luecht 7

Test Data Repositories

Test UnitRecords

Test Unit IdentificationItem List ReferenceTiming DataNavigation ControlsPresentation ScriptsTemplate Reference

ResourceVersions

Version/ReleaseTest Units ReferenceEncryption Key(s)Restructured Test Units(e.g., XML)

Item Index

Test Unit IdentificationItem Identification

Test Repository CBT Resource Files

2010 R.M. Luecht 8

Examinee Data Repositories

ExamineeRecords

Examinee IDContact InformationDemographics

ResultsRecords

Examinee IDTest IdentificationItem-Level TransactionBLOB

Test/ScoreHistory

Examinee IDTest ReferenceTesting DateScore(s)Accommodations

Examinee Repository CBT Server Repository

2010 R.M. Luecht 9

CBT: A System of Systems

2010 R.M. Luecht 10

Item-Level DataItem or exercise rendering data

Stimulus information (e.g., MCQ stem, a reading passage)Response display labels (e.g., distractors as labels for a check box control)Scripts for interactivityTemplate references

Content and other item attributesContent category codesCognitive and other secondary classificationsLinguistic features

2010 R.M. Luecht 11

Item-Level Data – cont.Statistical item data

Classical item statistics (p-values, biserial correlations, etc.)IRT statistics (1PL, 2PL, 3PL, GPCM parameter estimates)DIF statistics and other special indices

Operational dataReuse historyExposure rates and controls (for CAT)Equating status

2010 R.M. Luecht 12

Test Unit DataObject list to include (e.g., item identifies for all items in the test unit)Navigation functions, including presentation, review and sequencing rulesEmbedded adaptive mechanisms (score + selection)Timing controls and other information (e.g., how clock functions, time limit, etc.)Title and instruction screens

2010 R.M. Luecht 13

Test Unit Data – cont.Presentation template references

Helm look-and-feel (navigation style, etc.)Functions (e.g., direction of cursor movement after ◄┘ or tab is pressed)

Reference and ancillary look-up materialsCalculatorsHyperlink to other BLOBs Custom exhibits available to test takers

2010 R.M. Luecht 14

Standard Hierarchical View of a “Test Form”

Test Form A

Section IISection I Section III

Group 1 Group 2Set 1

Item1

Item2

Item3

Item5

Item6

Item7

Item4

Item8

Item9

Item10

Item12

Item11

2010 R.M. Luecht 15

Examinee DataIdentification information

Name and identification numbersPhoto, digital signature, retinal scan informationAddress and other contact information

Demographic informationEligibility-to-test information

JurisdictionEligibility periodRetest restrictions

2010 R.M. Luecht 16

Examinee Data – cont.

Scheduled test date(s)Special accommodations requiredScores and score reporting informationTesting history and exam blockingSecurity history (e.g., previous irregular behaviors, flagged for cheating, indeterminate scores, or large score gains)General correspondence

2010 R.M. Luecht 17

Interactions of Examinee and Items or Test Units

Primary informationFinal responsesCaptured actions/inactions (state and sequencing of actions)

Secondary informationCumulative elapsed time on “unit”Notes, marks or other captured during testing

2010 R.M. Luecht 18

Response Processing in CBTResponse capturing agents convert examinee responses or actions to storable data representationsExamples

item.checkbox.state (T/F) item.response.choice=“A” {A,B,C,D}item.group.unit(j).state (T/F), j=1,…6 item.response.choice=“1,4” {1,2,3,4,5,6}item.component.text(selected=position,length) item.response.text=“text”item.component.container(freeresponse.entry) item.response.text=“text”case.grid.cellRowCol(numeric.entry) case.grid.cellref.response.text=“value”

2010 R.M. Luecht 19

Raw Response Representations for Discrete

ItemsUniform CPA Examination: Item 002

Item 1

Item 2

item002.checkbox1.state="false"

item002.stemtext.text="HTML formatted text"

item002.checkbox2.state="false"

item002.checkbox3.state="true"

item002.checkbox4.state="false"

item002.checkbox5.state="false"

Convert to item002.response.choice=“3”

2010 R.M. Luecht 20

Raw Response Representations for a Performance ExerciseCPA Examination: Simulation A

Item 1

Item 2

Item 3

Case AItem 4Item 5

HelpContext Reports12-18-01 Instructions

Date Account

12-19-01

12-19-01

12-19-01

12-19-01

12-19-01

12-19-01

280-04

345-02

280-01

345-02

280-04

Debit

12501.99

Credit

C4C3C2

R7

R6

R5

R4

R3

R2

R1

C1

280-01

Convert to…caseA.tab5.sheet001.r3c4.text=“12501.99”

2010 R.M. Luecht 21

Raw Response Representations for an

EssayStore as…RichTextBox.Item001.text=“There were two important changes that characterized the industrial revolution. First, individuals migrated from rural to urban settings in order to work at new factories and in other industrial settings (geographic change). Second, companies began adopting mechanisms to facilitate mass production (changes in manufacturing procedures, away). ”

2010 R.M. Luecht 22

Entering the Psychometric Zone: Data Components of Scoring

EvaluatorsResponsesSelections, actions or inactions: item.response.state=control.state (ON or OFF)Entries: item.response.value=control.value

Answer expressions (rethinking IA is needed)

Answer keysRubrics of idealized responses or patterns of responsesFunctions of other responses

Score evaluators process the responsesScoring evaluators convert the stored responses to numerical values—e.g., f(responseij, answer keyi)xij[0,1]Raw scoring or IRT scoringaggregation and scaling of item-level numerical scores

2010 R.M. Luecht 23

Planning for Painless Data Exchanges and ConversionsSystems and subsystems need to exchange data on a regular basis, providing different views and field conversionsThe hand-off must have several fool-proof QC steps

Verification of all inputsConversion success 100% verifiedReconciliation of all results, including counts, discrepancies, missing values, etc.

2010 R.M. Luecht 24

Example of a (Partial) Examinee’s Test Results

Recordtestp>wang>marcus>>605533641>0A1CD9>93bw100175>1>>501>001>ENU>CB1_CAST105>>90>0>0>0>DTW>06/26/96>08:41:38>05:58:32>w10>2>apt 75>1000 soldiers field rd>north fayette>IN>47900>USA>1>1235552021>>NOCOMPANYNAME>0>>>>>1>1235551378>>0>0>35>>142/218/0/u>1>-1>7>CBSectionI.12>CB1>s>p>0>36>>72/108/0/u>Survey015>survey15>s>p>0>0>>0/0/0/u>CBSectionI>CAST2S1>s>p>0>28>>28/62/0/u>CBSectionII>CAST2S4>s>p>0>42>>42/48/0/u>CBSectionII>CAST2S3>s>p>0>0>>0/0/0/u>CBSectionII>CAST2S2>s>p>0>0>>0/0/0/u>Survey016>survey2>s>p>0>0>>0/0/0/u>0>372>SAFM0377>2>0>E>5>s>E>1>76>>SAEB0549>2>0>D>5>s>A>0>68>>SAFM0378>2>0>A>5>s>A>1>72>>SAAB1653>2>0>C>5>s>D>0>102>>SABA8868>2>0>B>5>s>C>0>85>>SCAA1388>2>0>E>8>s>E>1>53>>SAAA8447>2>0>D>5>s>E>0>55>>SAAB1934>2>0>A>5>s>A>1>60>>SAAB2075>2>0>E>5>s>E>1>136>>SADA7710>2>0>D>5>s>D>1>40>>SABB1040>2>0>B>5>s>E>0>46>>SCAA1396>2>0>H>10>s>A>0>93>>SACA8906>2>0>D>5>s>E>0>75>>SADA8116>2>0>C>5>s>D>0>53>>SADA8673>2>0>B>5>s>B>1>41>>SACA8626>2>0>B>5>s>D>0>48>>SAFM0374>2>0>C>5>s>D>0>80>>SABA6397>2>0>A>5>s>A>1>110>>SAAB1088>2>0>C>5>s>C>1>55>>SACA8455>2>0>D>4>s>D>1>73>>SAAB1667>2>0>C>5>s>C>1>44>>SAAJ7633>2>0>C>5>s>C>1>89>>SABA5745>2>0>D>5>s>A>0>43>>SCAA1389>2>0>B>8>s>H>0>61>>SADA8650>2>0>A>5>s>C>0>39>>SAFB0112>2>0>C>5>s>C>1>132>>SAAB2513>2>0>B>5>s>B>1>120>>SAFA9248>2>0>E>5>s>A>0>77>>SABJ1042>2>0>D>5>s>C>0>112>>SACJ5894>2>0>C>5>s>D>0>82>>SAAA0410>2>0>D>5>s>E>0>89>>SAAB1681>2>0>C>5>s>C>1>88>>SAFM0365>2>0>A>5>s>A>1>65>>SAEA8980>2>0>A>5>s>B>0>52>>

2010 R.M. Luecht 25

Assessment XML<?xml version="1.0" encoding="UTF-8"?><AssessmentResult xmlns="http://ns.hr-xml.org/2004-08-02" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ns.hr-xml.org/2004-08-02AssessmentResult.xsd">

<ClientId idOwner="Provider Inc"><IdValue name="ClientCode">OurClient-1342</IdValue>

</ClientId><ProviderId idOwner="Customer Inc">

<IdValue>ePredix</IdValue></ProviderId><ClientOrderId>

<IdValue name="PO Number">53RR20031618</IdValue><IdValue name="Department Name">Administration</IdValue>

</ClientOrderId><Results>

<Profile>Customer Service</Profile><OverallResult>

<Description>Executive Manager</Description><Score type="raw score">51</Score><Score type="percentile">65</Score><Scale>40-60</Scale>

</OverallResult><AssessmentStatus>

<Status>In Progress</Status><Details>Remains: "GAAP Basic Knowledge"</Details><StatusDate>2003-04-05</StatusDate>

</AssessmentStatus><UserArea/>

</AssessmentResult>

2010 R.M. Luecht 26

Translating XML Entities to a Data Structure

Application

ContentHandler

ErrorHandler

DTD andSchemaHandler

EnitityResolver

ExternalDataEntity

ExternalDataEntity

XMLDocument

External DTDSchema or XSL

Sheets

XML Parser StructuredData

2010 R.M. Luecht 27

Extracting Data ViewsA data view is a set of restructuring functions that produce a data set from raw data

Views begin with a queryUsually results a formatted file structureGraphing functions produce graphic data setsDatabase functions produce database record sets

Multiple views are possible for different uses (e.g., test assembly, item analysis, calibrations, scoring)Well-designed views are reusable

Standardized queries of the database(s)Each views as a template with “object” statusViews can be manipulated by changing their properties (e.g., data types, presentation formats)

2010 R.M. Luecht 28

Types of Data Files (Views)Implicit Files: File format implies a structure for the data

Flat files with fixed columns (headers optional)Comma, tab or other delimited files

Explicit Files: Variables, data types, formats and the actual data are explicitly structured

Data base files: dBASE, Oracle, Access, etc.Row-column worksheets with “variable sets” (e.g., Excel in data mode, SPSS)XML and SGML

2010 R.M. Luecht 29

Explicit Structured “List View”

2010 R.M. Luecht 30

Structure of the Flat File(A Type of “Metadata”)

Data Type Definitions

PresentationView Data

2010 R.M. Luecht 31

Relational DBM in Access

2010 R.M. Luecht 32

Relational Schema Across Databases for ATA

Item Attribute Table

PK Item ID Code

Item NameClassification SchemeAttibute Code

Classifications

PK Classification Scheme

Classification Description

Attribute Table

PK Attribute Code

Attribute DescriptionFK1 Classification Scheme

Classification Attribute Constraints

PK,FK1 Attribute Code

Classification SchemeConstraint Expression

Eligible Item Table

PK,FK1 Item ID Code

Item IDAnswer Key TypeAnswer Key Expression CountScore CategoriesItem MeanItem SDPt. BiserialD ConstantA ParameterB ParameterC ParameterWord CountReadability

2010 R.M. Luecht 33

Data File Extractions for Psychometric Processing

Item Analysis and Key Validation, Calibration and Equating

Table.Active_Test_Forms Table.Active_Test_Form_Content

Table.Link_Item_Sets

Table.Active_Item_Bank_Keyed

Table.Active_Examinee_Test_Forms

Table.Active_Examinee_Responses

2010 R.M. Luecht 34

Query Test Form and Item Databases

SELECT Item.Records IF(Query_Conditions=TRUE)Test.TestID Item.ID

TST0181 ITM020920TST0181 ITM048392TST0181 ITM020342TST0181 ITM038632TST0181 ITM023833TST0181 ITM031935TST0181 ITM035329TST0181 ITM022222TST0183 ITM020921TST0183 ITM030102TST0183 ITM028902TST0183 ITM038886TST0183 ITM024519TST0183 ITM039981TST0183 ITM040027TST0183 ITM030253

Test.TestID Item.IDTST0181 ITM020342TST0181 ITM020920TST0183 ITM020921TST0181 ITM022222TST0181 ITM023833TST0183 ITM024519TST0183 ITM028902TST0183 ITM030102TST0183 ITM030253TST0181 ITM031935TST0181 ITM035329TST0181 ITM038632TST0183 ITM038886TST0183 ITM039981TST0183 ITM040027TST0181 ITM048392

Sort Item.ID

2010 R.M. Luecht 35

The P I QuerySELECT Examinee.Records IF(Query_Conditions=TRUE)

Exam.PersonID Exam.TestID Exam.Status Exam.Date107555 TST0183 F 11-Sep-04517101 TST0181 F 11-Sep-04670048 TST0181 F 13-Sep-04758735 TST0183 F 13-Sep-04754364 TST0183 F 13-Sep-04827960 TST0183 R 13-Sep-04619834 TST0183 F 13-Sep-04615233 TST0182 R 13-Sep-04429336 TST0182 F 14-Sep-04

Exam.PersonID Exam.TestID Exam.ItemID Exam.iSequence Exam.iTime Exam.Response107555 TST0183 ITM020921 1 39 B107555 TST0183 ITM030102 2 131 D107555 TST0183 ITM028902 3 39 C107555 TST0183 ITM038886 4 61 B107555 TST0183 ITM024519 5 58 D107555 TST0183 ITM039981 6 67 A107555 TST0183 ITM040027 7 61 A107555 TST0183 ITM030253 8 34 B517101 TST0181 ITM020920 1 65 C

2010 R.M. Luecht 36

The P I QueryGENERATE.FLATFILE(Examinee.Records,Item.Records)

Examinee.PersonID ITM

0203

42

ITM

0209

20

ITM

0209

21

ITM

0222

22

ITM

0238

33

ITM

0245

19

ITM

0289

02

ITM

0301

02

ITM

0302

53

ITM

0319

35

ITM

0353

29

ITM

0386

32

ITM

0388

86

ITM

0399

81

ITM

0400

27

ITM

0483

92

107555 B D C D B B A A517101 B C A D C C B A

107555 9919911019991009517101 1190199990019991

.PersonID Items from TST0181 & TST0183

Generate“Masked” Response

File

2010 R.M. Luecht 37

Implied Flat File View of Test Data

(Person by Item Flat Files)Example 1: “raw response vectors” (input to commercial item analysis software)

00001 BDCAABCAEDACBD00002 BDBAABDAEDBCBD00003 BCCBABCAEDABBD

Example 2: “scored response vectors” (input to commercial item calibration software) 00001 11111111111011

00002 1101110111001100003 10101111111100

2010 R.M. Luecht 38

Reconciliation 101Definitions: bringing into harmony, aligning, balancingReconciliation is essential for CBT data management and quality assurance

Test forms, items, sets, examinees, and transaction output counts match input countsResults match expectations or predictions

2010 R.M. Luecht 39

Reconciliation Example (Examinee Data for

IA/Calibrations)File Reconcilation and Rectangular File Creator (R. Luecht, [c] 2009, 2010)Date/time: mm-dd-yy hh:mm:ssControl File: ControlFile.CONExaminee_Test_Form File: ActiveExamineeTestForm.txt NP = 1687

Treatment of (Score_Status=1) items: INCLUDE Items and ResponsesItem File: MasterItemFile.DAT NI = 3857, Total Read = 3857 Excluded= 0 ===========================================================Active_Examinee_Responses File=ActiveExamineeResponse.txt No. IDs (from Active_Examinee_Test_Form)= 1687 File size (examinee transactions)= 506100 No. nonblank records input = 506100 No. records with unmatched items = 0Forms = 8 1687 scored response records saved to Data-ResponseFile-Scored.RSP 1687 raw response records saved to Data-ResponseFile-Raw.RAW

ITEM LISTING and FORM ASSIGNMENTS DETECTED

ID Identifier NOpt Opts N-Count NFrm FormsItem_21801 5 ABCDE 41 1 04Item_24601 5 ABCDE 97 3 01 03 07Item_29801 5 ABCDE 97 2 02 07 :<only partial records included to conserve space>

MISMATCH SUMMARY----------------NO unmatched item IDs to FORMSNO unmatched item IDs to RESPONSE RECORDS

Form ID 01 02 03 04 05 06 07 08

01:SampleForm001 300 0 0 37 0 37 0 0

02:SampleForm002 0 300 0 0 200 0 0 0

03:SampleForm003 0 0 300 0 0 0 200 200

04:SampleForm004 37 0 0 300 0 200 0 0

05:SampleForm005 0 200 0 0 300 0 0 0

06:SampleForm006 37 0 0 200 0 300 0 0

07:SampleForm007 0 0 200 0 0 0 300 200

08:SampleForm008 0 0 200 0 0 0 200 300

Item Counts by Form

2010 R.M. Luecht 40

Follow the Single-Source Principle

A unique master record should exist for every entity

Examinees registered/eligible to testItemsItem setsModules, testlets or groupsTest forms

Changes should be made to the master and forward-propagated for all processing

2010 R.M. Luecht 41

Example of Single Source

NO!

YES!!

2010 R.M. Luecht 42

Ignorable Missing Data?Very little data is missing completely at random, limiting the legitimate use of imputationSome preventable causes of missing data

Lost records due to crashes/transmission errorsCorrupted response capturing/recordsPurposeful omitsRunning out of time/motivation to finish

2010 R.M. Luecht 43

Challenges of Real-Time Test Assembly (CAT or LOFT)

Real-time item selection requires high bandwidth and fast servers and pre-fetch reduces precisionA “test form” does not exist until the examination is complete

QC of test forms is very difficult, except by audit sampling and careful refinement of test specifications (objective functions/constraints)QC of the data against “known” test-form entities is NOT possible

2010 R.M. Luecht 44

Thanks!Ric Luecht

rmluecht@uncg.edu