Iteman 4.3 Manual - baixardoc.com

User’s Manual for

Iteman

Classical Item and Test Analysis

Version 4.3

June 2013

Contact Information Assessment Systems Corporation

6053 Hudson Road, Suite 345

Woodbury, MN. 55125

Voice: (651) 647-9220

Fax: (651) 647-0412

www.assess.com

Bookmarks To view PDF Bookmarks for this manual, select the Bookmark tab on the left side of the Acrobat

window. The bookmark entries are hyperlinks that will take you directly to any section of the

manual that you select.

License Unless you have purchased multiple licenses for Iteman 4.3, your license is a single-user license.

Instructions for transferring your license between computers are in Appendix E.

Technical Assistance and support If you need technical assistance using Iteman 4.3, please visit the Support section of our Web site,

http://www.assess.com/xcart/pages.php?pageid=8, which contains FAQs and updated versions of

our software. Technical assistance for Iteman 4.3 is provided for as long as you maintain the

current version. Please provide us with the invoice number for your license purchase (typically 5

digits, like #22111) when you request technical assistance.

Citation Guyer, R., & Thompson, N.A., (2013). User’s Manual for Iteman 4.3. Woodbury, MN:

Assessment Systems Corporation.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any

form or by any means—electronic, mechanical, photocopying, recording, or otherwise—without

the prior written consent of the publisher.

Copyright © 2013 by Assessment Systems Corporation

All Rights Reserved

Iteman is the trademark of Assessment Systems Corporation

Table of Contents

1. Introduction ...................................................................................................................... 1

Your Iteman 4 License and Unlocking Your Copy ............................................................ 1

2. Input Files ......................................................................................................................... 4

The Data Matrix File........................................................................................................... 4

Iteman 3 Data Format ......................................................................................................... 4

Delimited Data Matrix File ................................................................................................. 5

The Item Control File ......................................................................................................... 5

3. Running the Program ...................................................................................................... 7

Using the Menus ................................................................................................................. 7

The File Menu ................................................................................................................. 7

Using the Main Program ..................................................................................................... 8

The Files Tab .................................................................................................................. 8

The Input Format Tab ..................................................................................................... 9

The Scoring Options Tab .............................................................................................. 11

The Output Options Tab ............................................................................................... 13

Using Multiple Runs Files ................................................................................................ 15

Creating a Multiple Runs File ....................................................................................... 15

Opening a Multiple Runs File ....................................................................................... 17

A Sample MRF File ...................................................................................................... 17

Running the sample files................................................................................................... 19

4. Interpreting the Output ................................................................................................. 22

Test-Level Output (Examinee Scores) .............................................................................. 22

Test-Level Output (Reliability Analysis) ......................................................................... 23

Test-Level Output (Graphics) ........................................................................................... 24

Conditional Standard Error of Measurement (CSEM) ..................................................... 24

Item-Level Output............................................................................................................. 25

What to Look For .............................................................................................................. 29

Item Difficulty .............................................................................................................. 29

The P value (Multiple Choice) ........................................................................... 29

The Item Mean (Polytomous) ............................................................................ 30

Item Correlations .......................................................................................................... 30

Multiple Choice Items ........................................................................................ 30

Polytomous Items ............................................................................................... 30

DIF Statistics ................................................................................................................. 31

Mantel-Haenszel ................................................................................................ 31

z-test Statistic ..................................................................................................... 31

p .......................................................................................................................... 32

Bias Against ....................................................................................................... 32

Option statistics ............................................................................................................. 32

Scores Output File ............................................................................................................ 32

Collusion Index (Bellezza & Bellezza, 1989) .................................................................. 33

References ........................................................................................................................... 34

Appendix A: The Iteman 3 Header ................................................................................... 35

The Control Line ............................................................................................................... 35

The Keyed Responses ....................................................................................................... 36

The Number of Alternatives ............................................................................................. 36

The Scale Inclusion Codes ................................................................................................ 36

Appendix B: Troubleshooting ........................................................................................... 37

Please check the data file format specifications ............................................................... 37

Please check the number of items or number of ID columns specified in the Iteman 3

Header ............................................................................................................................... 37

Please select an input file with an Iteman 3 Header ......................................................... 38

Valid item responses of 0 were identified. The Iteman 3 Header does not support item

responses that begin at 0. .................................................................................................. 38

At least one valid item response of 0 was identified. ....................................................... 39

At least one unidentified response character was identified and will be scored as

incorrect. ........................................................................................................................... 39

Check the data matrix file, examinee XXX did not respond to all XXX items................ 40

Appendix C: Formulas ....................................................................................................... 41

Conditional Standard Error of Measurement Formulas .................................................... 41

Livingston Classification Consistency Index ................................................................... 41

Differential Item Functioning ........................................................................................... 41

Item Collusion Index ........................................................................................................ 42

Appendix D: Program Defaults File ................................................................................. 43

Appendix E: License Transfer .......................................................................................... 46

Summary ........................................................................................................................... 46

Step 1 – Demo/Trial Program ........................................................................................... 46

Step 2 – Licensed Program ............................................................................................... 47

Step 3 – Demo/Trial Program ........................................................................................... 49

Iteman 4.3 Manual Page 1

1. Introduction

Iteman™ is a Windows® application designed to provide detailed item and test analysis reports

using classical test theory (CTT). The purpose of these reports is to help testing programs

evaluate the quality of test items by examining their psychometric characteristics.

Iteman has a friendly graphical user interface (GUI) that makes it easy to run the program, even

if you are not familiar with psychometrics. The GUI is organized into five tabs: Settings, Files,

Input Format, Scoring Options, and Output Options. These are discussed in detail in Chapter 3:

Running the Program.

Iteman 4.3 offers several substantial advantages over Iteman 3:

1. The most important advantage is the addition of graphics. It is now possible to produce

an item quantile plot for each item. Moreover, you control the number of points in the

plot.

2. Iteman 4 is able to handle pretest (trial or unscored) items—items that are not included in

the final score but for which statistics are still desired.

3. More statistics are calculated, including the alpha (KR-20) reliability coefficient with

each item deleted, several split-half reliability coefficients (both with and without

Spearman-Brown correction), conditional standard error of measurement, and subgroup P

(proportion correct) statistics for up to seven ordered groups.

4. Instead of simple ASCII text files, the output is now rich text file (RTF) format prepared

as a formal report, and also in a comma-separated value (CSV) format that is able to be

manipulated (sorted, highlighted, etc.) in spreadsheet software. It additionally produces a

CSV file of examinee scores.

5. Scaled scores and subscores can be added to the output.

6. Scores can be classified into two groups at a specified cut score, and the two groups can

use your labels.

7. Items can be analyzed relative to an external score rather than the total score on a test.

8. The maximum number of items that can be analyzed has been increased to 10,000.

9. A “batch” type of capability, using a “Multiple Runs File” has, been added to allow you

to run multiple data sets without having to use the graphic user interface for each run.

Multiple Runs files can be created outside Iteman in a text editor or interactively within

Iteman.

Your Iteman 4 License and Unlocking Your Copy Unless you have purchased a network or multiple-computer license, your license for Iteman 4 is

a single-user license. Under this license you may install Iteman 4 on two computers (e.g., a


desktop and a laptop) so long as there is no possibility that the two copies of the software will be

in use simultaneously. If you would like to use Iteman 4 on a network or by more than one user,

please contact us to arrange for the appropriate number of additional licenses.

Iteman 4 is shipped as a functionally-limited demonstration copy. It is limited to no more than 50

items and 50 examinees, but has no expiration date. We can permanently convert your demo

copy to the fully functioning software by email, phone, or fax once you have completed the

license purchase. To unlock Iteman 4, please email/phone/fax to ASC:

1. Your name and email address.

2. Your organization or affiliation.

3. Your invoice number (in the top right corner of your invoice). You should make a

record of your invoice number since you might be asked for it if you request technical

support.

4. The “unlock codes,” which are two numeric codes that are unique to the installation of

Iteman 4 on any given computer. To obtain these two codes, click on the “Unlock

Program” button when Iteman 4 starts (Figure 1.1) This license window can also be

reached by clicking on the License button and selecting “Unlock” when Iteman 4 is

running in demo mode.

Figure 1.1: Screen Visible When Iteman 4 is Locked

From the unlock screen you will need to send us the two blue Computer ID and Session ID

numbers (Figure 1.2). For your convenience we have provided a “Copy IDs to Clipboard”

button. This will copy both IDs to the Windows clipboard along with a brief message and the

email address to which to send your payment information. This can then be pasted into an email

message, filled in, and sent to [email protected]. If you have already paid for your Iteman 4

license, be sure to add your invoice number to this message.

When we receive these codes from you, we will respond with a single numeric Activation Code

(if you have purchased a permanent license) or two codes (if you have purchased an annual

subscription license) that you will need to enter into this same window from which you obtained


your Activation Codes (the red labels in Figure 1.2). Once you enter the code(s) that we send

you, your copy will be unlocked and fully functional.

Figure 1.2: The Unlock Screen

Note that if you install Iteman 4 on a second computer, you will need to repeat this process for

that computer since the unlock codes are specific to a given computer.

• Iteman 4 is permanently unlocked for academic use, but is an annual subscription for

non-academic use. The license status box (see Figure 3.1) will display the current

license status, including the number of days remaining for your subscription. As the

subscription nears the end, the background color of the box will change to alert you to the

need to renew your subscription for another year (red if you have less than 30 days

remaining, yellow if 30-90 days, and green if more than 90 days).

.


2. Input Files

Iteman 4 requires two input files: the Data Matrix File and an Item Control File. The formats for

these files are described in the following sections.

The Data Matrix File The Data Matrix File is the file that contains examinee identification (ID) and the responses to

each item. Responses can be alphabetical (A,B,C,D… or a,b,c,d…) or numerical (1,2,3,4…),

where A = a = 1, etc. Each line presents the information for one person. An example of this is

shown in Figure 2.1 for 10 items and 5 examinees. In this file, there are 9 columns of ID (the

last two are blank) and 10 columns of responses.

Figure 2.1: Example of an Input Data File (No Ignored Columns)

Additional columns can be ignored, so it is not necessary to delete any data if your data file has

information other than ID and responses. For example, your file might contain exam dates,

locations, education level, or sensitive personal data that you do not want included in the output.

In the example shown in Figure 2.2, group membership information is included in column 19.

An example of this is shown in Figure 2.2; you might want to include examinee ID numbers (the

first six columns) in your output but not names. Chapter 3: Running the Program describes

how to skip these columns.

Figure 2.2: Example of an Input Data File (Columns to Ignore)

Iteman 3 Data Format

Iteman 4 permits the analysis of a Data Matrix File in the format used with Iteman 3 (and other

programs in the Item and Test Analysis package) , which includes four header lines of control

information in the data file rather than in a separate control file. If the Iteman 3 header is

included in the Data Matrix File, then the user should specify this with the checkbox on the

“Files” tab of Iteman 4. Note that this method does not allow the use of Item IDs. See

Appendix A for a description of the Iteman 3 header.

Person1 4213323412

Person2 1213323410

Person3 3323123413

Person4 1223323414

Person5 2214323411

6153425 John Doe M 4213323412

5947824 Jane Doe F 1213323410

5976281 Jack Hall M 3323123413

1359687 Jim Hill M 1223323414

9778236 Jen Smith F 2214323411


Delimited Data Matrix File Iteman 4.3 now permits the use of a data matrix file that is delimited by either a comma or a tab.

As shown in Figure 2.3, this now permits the inclusion of variable length examinee IDs.

Chapter 3: Running the Program describes how to specify a delimited input file.

Figure 2.3: Example of a Comma-Delimited Data Matrix File

If a differential item functioning (DIF) analysis was requested the DIF group membership code

should follow the examinee ID as shown by Figure 2.4. It is important to note that the DIF

membership codes (M and F) will not be recognized if they are included as part of the examinee

ID (e.g., Person9M).

Figure 2.4: Example of a Comma-Delimited Data Matrix with DIF Codes

The Item Control File The previous version of Iteman required that the specifications for the test be provided on the

first four lines of the data file, with all the data itself moved down. Iteman 4 provides the

specifications as a separate Item Control File. This makes it easier to produce, as well as allows

for the handling of a greater amount of information. This file is tab-delimited, which means that

you can construct it in a spreadsheet program and then “Save As” a tab-delimited text file.

There are six columns of information in the control file for each item. Begin each item on a new

line:

1. Item ID: cannot contain a tab character, but length can vary across items;

2. Key(s):

• Correct answer(s) as A,B,C,D if unscored multiple-choice.

• 1 if items are scored dichotomous.

• For polytomous items: + if positively scored or – if negatively or reverse scored.

3. Number of alternatives (maximum is 15). For multiple-choice items that are scored 0

or 1, the number of options is 2.;

4. Domain or content area (unique domain label, maximum is 50);

5. Inclusion status:

• Y = Yes (included in the analysis),

Person9,4,2,1,3,3,2,3,4,1,2

Person10,1,2,1,3,3,2,3,4,1,0

Person11,3,3,2,3,1,2,3,4,1,3

Person12,1,2,2,3,3,2,3,4,1,4

Person13,2,2,1,4,3,2,3,4,1,1

Person9,M,4,2,1,3,3,2,3,4,1,2

Person10,F,1,2,1,3,3,2,3,4,1,0

Person11,M,3,3,2,3,1,2,3,4,1,3

Person12,F,1,2,2,3,3,2,3,4,1,4

Person13,F,2,2,1,4,3,2,3,4,1,1


• N = No (not included),

• P = Pretest;

6. Item type:

• M = Unscored multiple-choice items with responses that begin at 1 or A. For

scored multiple-choice items, see P below.

• R = Rating scale items: polytomous items with responses that begin at 1 or A).

• P = Items with numeric responses that begin at 0 (e.g., 0, 1, 2, 3). This includes

multiple-category partial credit items, and dichotomously scored multiple-choice

items (scored 0 or 1);

An example of the control file is shown in Figure 2.5. There are ten items, with nine multiple

choice items and one partial credit item. The first five are in Domain 1, while the latter five are

in Domain 2. The first four items in each domain are scored, while the fifth item in domain 2 is

a pretest item. The keyed answers are either 1, 2, 3, or 4 for each multiple choice item since each

item has 4 alternatives. Keys can be alphabetical or numeric. Item 7 has two keyed responses (3

and 1). For Item 7, item responses will be scored as correct if the examinee answers either 3 or

1.

If an item is polytomously scored, the key should be “+” if positively scored and “-” if negatively

(reverse) scored. Item 10 is a positively scored (+) partial credit item with item responses that

begin at 0. For item 10, the item responses will be 0, 1, 2, 3, and 4, since the item has five

options.

The control file should have as many lines as there are items in the test. The program counts the

lines of information in the control file, and that serves as the total number of items in the test.

There is a maximum of 10,000 items (lines) in Iteman 4.3.

Figure 2.5: Example of an Item Control File

Item01 1 4 Science Y M




Item05 1 4 Science P M

Item06 2 4 Reading Y M




Item10 + 5 Reading P P

Documents

Iteman 4.3 Manual - baixardoc.com