1

January 2012 Regents ScanningTraining for CFN 201 Assessment Liaisons

Version 2.02 – December 2011

22

ObjectivesThe purpose of this presentation is to educate you on the Regents scanning procedures, and empower you to turn-key the process with your school.After viewing the presentation, you will understand:

> The overall scanning process and step-by-step procedures for each stage;

> How your school should plan for Regents exam week, and special situations to prepare for;

> What documents and resources are available for training and support, including use of the demo IT environment for school training;

> Best practices for turn-keying the scanning process at schools; and> Process updates for the January 2012 administration

33

Process Changes for Jan. 2012 Regents Scanning> Key differences for January 2012 Regents Exams include:

A new exam code deck has been released in STARS student scheduling. (Slide 9)

Addition of a “Did not meet lab requirement” bubble for students who were scheduled in STARS, but failed to meet lab requirements prior to the exam, or who were mistakenly allowed to sit for the exam despite not having met lab requirements. (Slide 20)

No final score will be calculated until an answer document is fully scanned. (Slide 24)

Changes to the RGTU screen in ATS – principals can correct scanning errors on fully scanned answer documents. Any changes that result in a score moving from fail to pass will be held for DAPS approval and will lock subsequent rescanning for that document. (Slides 29-31)

Elimination of the centrally reported data validation process. (Slide 32) Addition of a misadministration field for scanned documents in RGTU to

eliminate score reporting to the transcript and report card systems (Slides 20, 21, 31)

44

SED Regents Requirement & DOE Solution

> Comprehensive English> Earth Science> Global Studies & Geography

> Integrated Algebra> Living Environment> U.S. History & Government

SED Requirement: Beginning in June 2011, the SED requires all districts to collect and submit item-level data for the following Regents exams:

DOE Solution: Leverage existing technology and systems currently used by schools to capture and report item-level data. Benefits of the scanning process include:

> Answer documents: student-specific documents are printed at school level> Scoring: teachers mark Constructive Response items only> Data capture: scan answer docs with increased accuracy> Reporting results: real-time results available in ATS after scanning > Improved Instruction: item level data easily available to schools> Audit Trail: improved audit trail will aid in special inquiries.

*RCTs will continue to use the traditional administration and scoring process and will not be scanned.

Beginning in June 2012, all ten Regents exams will be scanned, including Algebra 2/Trig, Geometry, Chemistry, and Physics.*

Pre-Planning STARS ATS &

Scanner

ATS STARS (& Beyond)

1. Schools receive training and begin preparations for Regents Scanning process.

2. Schedule students for Regents and produce exam invitations in STARS.

3. STARS data flows to ATS. Schools test scanner and generate and print pre-slugged answer docs in advance.

6. Schools scan answer docs for upload to ATS.

7. ATS instantly applies scoring key & conversion chart for scale score. Results available immediately.

8. ATS passes data nightly to STARS for report cards and transcripts, and for reporting to NYSED.

Exam Week & Scoring

5. Schools administer exams. Answer documents for walk-ins are printed on-demand. Scoring committee marks Constructive Response items and bubbles in scores on answer documents.

Attendance Scanner

Re-Scanning (Only As Necessary)

Regents Scanning from A-Z: Process Overview

5

Logistics Planning

4. Schools conduct Regents week planning activities for logistics, including assignment of roles, and performing checklist items in the week before exams are administered.

6a. Re-scan docs.

Overnight Data Transfer

Overnight Data Transfer

66

Timeline of Key Steps at School LevelDates Key Step

ASAP Assessment and/or Data/IT Liaisons: schedule and conduct training sessions as needed.

ASAP Data/IT Liaisons: consult schools to ensure appropriate staff members have access to ATS.

ASAP Schools clean/test scanner, and ensure adequate supplies (toner, paper, etc.).

Dec. 21, 2011 Schools have begun to register students for Regents in STARS.

Jan. 9, 2012 Schools begin to print answer documents in ATS

Jan. 16 – 20, 2012 Regents logistics planning & final preparation

Jan. 23 – 27, 2012 Regents administration

Jan. 24 – Feb. 3 Regents scanning window

77

Step 2: STARS> Main changes for January 2012:

New January Regents and RCT codes have been added to all high school code decks.

> Main tasks: Register students for Regents exam, and generate exam invitations.

> Best practices: Complete registration process for majority of student before printing

answer documents. Only register students expected to take exams (though when in

doubt, err in favor of registration). Use “Master Schedule” function for creating a single section, and

“Master Schedule Builder” for multiple sections. Networks monitor scheduling progress using STARS Completion

Reports and follow-up with schools requiring attention. > Middle Schools:

If a middle school wishes to offer the January Regents to accelerated students, network data points should contact their cluster academic policy/data point.

88

Step 2: STARS New Exam Code Deck> The January 2012 Regents and RCT codes for High School students are

below. Note these codes also indicate the month of exam administration.

Exam Old Code

New Code

New Course Name

Integrated Algebra MXRE MXRER ALGEBRA REGENTS JAN

Comprehensive English EXRL EXRLR ELA REGENTS JAN

Global History & Geography HXR$ HXRGR GLOBAL REGENTS JAN

US History and Government HXRA HXRUR US HIST REGENTS JAN

Living Environment SXRK SXRKR LIVINGENV REGENTS JAN

Physical Setting/ Earth Science SXRU SXRUR EARTHSCI REGENTS JAN

99

Step 3: Printing Answer Docs in ATS> There are four main options for printing answer docs:

1. Not Previously Printed (batch printing for registered students)

Prints batch of answer documents that have not been printed (RGPR)

2. Print by Student ID (for DOE walk-in student) Prints an answer sheet for any student, one at a time. This option may be used for another DOE student taking the exam at your school. (Use SBIO screen to look up student ID.) (RGPR)

3. Print by Name (non-public) (for non-DOE walk-in student) Prints an answer sheet for a non-DOE student who is taking the exam at your school. (RGPR)

4. Reprint Answer DocumentsReprint previously printed docs by student ID or answer-document page number. (RGRR)

1010

Step 3: Printing Answer Docs in ATS (RGPR)

Sorting Options

> Use RGPR screen for printing options 1, 2, and 3.

1111

Step 3: Printing Answer Docs in ATS (RGRR)> Use RGRR screen for printing option 4.

Sorting Options

1212

ATS-Generated Answer Documents Old Answer Document Regents Scannable

Individualized bar code

Student MC answers

CR teachers’ grades

Alt. language & accommodations

Pre-populated student info

Scorers’info

1313

Step 3: Printing Answer Docs in ATS> Best practices for printing answer documents

Answer documents must be printed on white paper only. Use the same stock paper as is normally used for attendance forms. Using colored paper or heavy stock may result in scanning errors.

After printing batches of answer documents, a “test page” will print. Scan the test page to confirm the ink quality for a batch. Trailing test pages are specific to each exam to better diagnose scanning issues ahead of testing period.

Do not use incompatible printers (e.g., Lexmark T630). Print answer docs only on supported printers (Lexmark T640 or later).

Finalize section/room data in STARS before printing answer docs. You may sort print jobs by teacher, section, or room number. Remind school team members on when to use RGPR versus

RGRR. Answer documents are no longer included in the Regents exam

booklet to be torn out. No generic answer documents may be used.

1414

Step 4: Logistics Planning for Regents Week> Main tasks:

Distribute test rosters and page one of answer docs to appropriate staff in advance.

Train exam proctors on new procedures and policies. Ensure adequate supply of toner, paper, pens/pencils. Establish process and task owners for dealing with walk-ins.

(See slide 17)> Best practices:

Roles and responsibilities are clear and communicated to all relevant staff before Regents Week.

Perform all checklist items.> Supporting materials: Regents Scanning – Implementation

Checklist> Support: CFN Assessment Liaison - Amy Reynolds areynolds@schools.nyc.gov 917-940-

7319, Jerry Lin jlin6@schools.nyc.gov 718-281-7561

1515

Step 5: Exam Week & Scoring> Main tasks:

Generate on-demand answer documents for “walk-in” test-takers.

Proctors emphasize proper technique - • completely fill in bubbles and erase stray marks; • remind students to sign student declaration when finished.

> Best practices: Students use PENCIL for Multiple Choice, PEN for essays,

DBQ questions, and student attestation. Teachers carefully fill in bubbles (using only pencil) when

scoring Constructive Response questions. Good preparation will lead to smooth exam administration

> Support: CFN Assessment Liaison - Amy Reynolds areynolds@schools.nyc.gov 917-940-7319, Jerry Lin jlin6@schools.nyc.gov 718-281-7561

1616

Step 5: Instructions for “Walk-Ins”Schools should make preparations and assign roles to handle walk-in students prior to Regents week.

Step DOE Student Non-DOE StudentConfirm student’s ID Check photo ID Check photo ID

School notification Check student record against notification sent from her home

school

Must present signed letter on school letterhead with exam

specified

Print answer doc in ATS “Print by Student ID”* to connect to student’s DOE record

“Print by Student ID”* if student has NYC ID from DOE summer school. “Print by Name” in other

cases.

Enter exam code Use appropriate exam code Use appropriate exam code

*If Student ID is unknown, use ATS SBIO menu to look it up.

> Inform appropriate staff how to direct walk-ins upon arrival, and appoint staff to print answer documents using ATS.

> Print answer documents by following the instructions below:

Detailed instructions on handling walk-ins is included in the Handbook and FAQ.

1717

Step 5: Exam Week & Scoring

> SED Rescoring Policy: Beginning in June 2011, schools are no longer permitted to

rescore any of the open-ended questions on any Regents exam after each question has been rated the required number of times (as specified in the rating guide), regardless of the final exam score.

Specifically, exams receiving a final score of 60-64 or 50-54 are not permitted to be re-scored.

Principals and other administrative staff in a school or district do not have the authority to set aside the scores arrived at by the teacher scoring committee and rescore student exams.

1818

Step 5: Exam Week & Scoring> Teacher scoring of their own students’ exams:

For January 2012 exams, teachers can score their own students’ exams, although NYSED requests principals should make every effort to prevent teachers from rating their own students’ responses, where possible.

Under guidance from the New York State Education Department, starting in the 2012-13 school year, teachers will no longer be able to score their own students’ responses.

Non-electronic distributed scoring pilots will take place in January and June 2012 to inform the DOE on Regents scoring approaches that meet the new NYSED requirements. Contact your Cluster Assessment Point for more information.

1919

Step 6 and 6a: Scanning Answer Documents> Main tasks:

After notification by DOE, you may use the attendance scanner to begin scanning answer documents. Notification will be communicated to CFN Assessment and Data/IT Liaisons, on the intranet Regents page, and through the ATS login screen. Documents scanned before notification will not be processed and must be rescanned.

Documents with errors in reading bar codes will be reprinted by the scanner with an error message. Documents with bubble read errors will be indicated on confirmation sheet(s) and may be investigated in ATS. Re-scan affected documents as necessary.

Ensure the scanner is clean; a dirty scanner may result in inaccurate results. See the Tips for Scanning appendix of the Regents Scanning Handbook* for instructions on cleaning.

Review REDS reports for “warnings” noted in Teacher section. Print SCRN report, which provides information on scanning status.

> Support: Data/IT Liaison Jerry Lin jlin6@schools.nyc.gov 718-281-7561

* The Handbook is available in ARIS

2020

Step 6 and 6a: Scanning Answer Documents> Main changes for January 2012:

Addition of a “Did not meet lab requirements” bubble on Earth Science and Living Environment Regents answer documents. If this bubble is filled when scanned, it will remove the score; no grade will appear on report cards or transcripts. The score will not be reported to the State. • This designation is used only for students who sat (but were ineligible)

for either exam, and should be used only on completed and scanned documents. Un-schedule students who did not sit for the exam in STARS and do not scan these documents.

Updated NYSED policy guidance - in order to preserve answer sheet quality for audit purposes, answer documents should be scanned only once except if there is missing or incorrect teacher scores on the document, or if the “did not meet lab requirements” bubble is filled.

No score will be generated until the document is completely scanned.

2121

Step 6 and 6a: Scanning Answer DocumentsCode Use Case Method of Entry Effect on exam

recordAbsent (ABS) Student is absent Scanned answer

document or RGTU Score is ABS

Did not meet lab requirements

Student mistakenly allowed to sit for exam but was not able to meet Science lab requirements in time for the exam

Re-scanned answer document

Score not reported on transcript or report card (data sent to SIF)

Invalidated (INV)

Confirmed case of student cheating

Scanned answer document (with principal initials)

Score of INV

Mis-administration(MIS)

School error (ex. missing accommodations, document was created for the wrong student)

RGTU Score not reported on transcript or report card

Do not scan completely blank (unused) answer documents

2222

Step 6 and 6a: Scanning Best Practices> When scanning 2-page documents, it is recommended to batch all

page 1 documents and scan separately from page 2 documents. > Schools may scan the first page of two-page documents

immediately, before scoring is complete for page two. > If only one page of a 2-page document is scanned, it will register in

ATS as an incomplete scan until the second page is processed. No score will be calculated until both pages of a 2-page document are scanned.

> The scanner will print a confirmation page indicating pages with scanning errors, and will reprint pages for which the bar code could not be read.

> File the scanned batches together with the DFS confirmation page. This will assist in retrieving papers as needed in the future.

> Scanning is open from 6 a.m. – 9 p.m. on weekdays, and from 8:30 a.m. – 4 p.m. on weekends. Please allow time for the whole batch to process, and do not attempt to scan large quantities of documents too close to the daily cutoff for overnight data transfer to STARS (9 p.m. on weekdays and 4 p.m. on weekends)

2323

Step 6 and 6a: Scanned Images> Scanning captures

an item-level data string as well as images of answer documents (and any ancillary re-scans) for an audit trail.

> Red bubbles indicate responses recorded by the scanner.

> Central users have access to scanned image capture.

2424

Step 7: ATS Reports: REDS (Status Report)> REDS reports (available immediately after scanning)*:

Use Document Status Report to ensure that all answer docs have been properly scanned and to flag warnings.

Use Item Report to see students’ scores of individual students. Review report for “warnings/multiples/omits” by using “Include only W/M/O” option.

Use Item Distribution Report to inform decisions on present and future instruction.

> REDS report changes:• Scores of Misadministration, Did not meet lab requirements,

Exam invalidated, and Absent will all override scaled score from responses.• Score of “INC” will be shown when only one page (of 2-page

answer doc) has been scanned.> Given the length (and width) of the REDS report, it is best viewed

when printed, as all data may not be visible on the ATS screen. Schools may download ATS reports into Excel for further analysis.

* Note: REDS reports populate in real-time. Other ATS reports (e.g., RCOS) are not real-time.

2525

Step 7: ATS Reports: REDS (Status Report)> Report uses:

Displays score, accommodations, warnings, for each student. Shows summary of documents scanned, outstanding, and warnings. Use report to track outstanding unscanned and incompletely scanned

document.

Accom.Scores Scan-specific info

Document statuses

2626

Step 7: ATS Reports: REDS (Item Report)> Report uses:

Displays line-item data for each student. Shows specific questions to review (warnings). Use to flag Teacher section questions to investigate (e.g. omits, multiples).

TEACHER SECTIONSTUDENT SECTION

Correct answers

Multiple answersIncorrect answers CR raw scores

2727

Step 7: ATS Reports: SCNS (Scan Statistics Report)> Report uses:

This report provides information on scanning status including the number of pages scanned, accepted, and rejected, and

the total number of times scanned, with and without errors.

2828

Step 7: ATS Reports: RADI (Item Dist. Report)> Report uses:

Shows school-level aggregated data for each question. Use data to flag specific subject matter and question types for instruction.

‘*’ denotes correct answer

2929

Step 7: ATS Reports: RGTU (Update Regents Answers)> The RGTU screen is used to update student and teacher responses not

accurately picked up by the scanner, including incomplete erasures, selections lightly bubbled in, or stray marks. Process:

• After a school confirms a scanner error, the school will use RGTU to update individual student or teacher section responses.

• Changes made in RGTU that do not cause a record to move from a failing score to a passing score will be processed automatically (e.g. an 80 82 or 49 52). » Requested changes that cause a record to move from a failing score to a

passing score will be held for review and approval by DAPS, and will cause the record to be locked until the document status is resolved.

• If a school attempts to rescan a locked record, the scan sheet will be successfully scanned and sent to ATS—but ATS will reject the record. The error will appear on the SCNS report, and the record will not be updated in ATS

• Principals will be able to review the status of requests in ATS. Most requests will be reviewed within 48 hours unless further information is needed.

Appeal process• Principals (and Data/IT Liaisons) will be directly notified via email if a change

requested has been denied, or if further information is needed to justify the change. Assessment and/or Data/IT Liaisons will work with DAPS to clarify any such requests.

3030

> Best practices for use of the RGTU screen: Do not attempt to use the RGTU screen on an incompletely scanned

answer document. Use RGTU for scanning errors, such as stray marks or lightly bubbled

responses. Do not erase or re-bubble and then rescan any student responses.

Under no circumstances may teachers or administrators mark, erase, or otherwise modify any part of the student section of the answer document.

If a teacher fails to properly record a score in the Teacher section, then the RGTU screen should be used to correct the error. If a Teacher section answer is omitted, the answer document should be bubbled in with the correct response and then rescanned.

In the event a document cannot be scanned, schools will need to recreate the student answer document and scan (and retain the original) in order to access the RGTU screen.

Schools may mark an exam record as misadministered on this screen. Each school must retain a written explanation of each change in RGTU for

audit and review purposes.

Step 7: ATS Reports: RGTU (Update Regents Answers)

3131

> Note corrections to Student and Teacher sections are made separately in the RGTU screen

Step 7: ATS Reports: RGTU (Update Regents Answers)

3232

Step 7: Score Verification> Main tasks:

Although optional, schools should verify a sample of scanned results for accuracy - approximately 5% of answer documents.

Pursuant to guidance from NYSED, there is no re-scoring of any scanned Regents exam.

> Best practices: Two people at school level perform data verification The Manual Scoring Data Validation Form is no longer used.

> Support: CFN Assessment Liaison

3333

Step 8: Accessing Results in STARS> Main changes from June/August 2011:

Student records with either misadministration or the “Did not meet lab requirements” answer document bubble filled in will not have scores reported to STARS, the transcript system, or SED.

> Main tasks: On the day after answer documents are scanned, results will be

available in STARS (for non walk-in students) and on transcripts (for all students), provided that scanning was completed by 9 pm the previous day.

> Best practices: Any future score changes (through re-scanning, DOE answer key

alteration, etc.) will be applied automatically. Data flows from ATS to STARS automatically; no manual data entry is

required. Results will be automatically applied to transcripts and report cards from STARS.

> Support: CFN Data & IT Liaison

3434

Turn-Key Training: CFN & School Checklist*Main Action Items Timeline

CFNs coordinate trainings with schools ASAP

Schools begin to schedule students for Regents in STARS By Dec. 21

Schools utilize demo environment to run mock exam process Dec. 12 – Jan. 6

Schools make preliminary preparations (check toner, paper supplies) Jan. 2 – 23, 2012

Schools print answer documents for majority of students By Jan. 16

Schools establish & communicate Regents week plan (roles, logistics) and notify and train relevant staff. Jan. 16 – 20

Schools administer exams, scan documents, and conduct data validation. Jan. 23 – 27

*Refer to Implementation Checklist for a complete list of tasks to be performed by schools.

3535

Turn-Keying: Demo Environment

> The demo environment is available from December 12, 2011 – January 6, 2012. After January 6, the demo period will end and all demo data will be removed.

> Detailed instructions on how to access and use the demo environment are available through a demonstration script. You will use your existing ATS username and password.

> Demo scope:

> We encourage schools to take advantage of the demo environment to gain familiarity with the new processes and to help ensure smooth Regents administration.

A demo environment is available for training purposes. Schools are encouraged to use this to prepare for Regents Week and to gain an understanding of the required processes.

DEMO

STARS(Scheduling)

ATS(Printing)

ATS(Scanning & Reporting)

STARS(Report Card &

Transcripts)

3636

On-Going Support & Resources

> Documentation and updates: ARIS: or: http://www.bit.ly/regentsscanning

> Technical support (STARS & ATS): CFN Data & IT Liaison

> Scanner and printer: DIIT Help Desk, 718-935-5100

> Overall process support: Borough Assessment Implementation Directors (BAID)

> Regents week emergency inquiries from CFNs*: Christopher Sandor (csandor2@schools.nyc.gov) Patrick Purcell (ppurcell@schools.nyc.gov)

> Kirby Fields – Office of School Support (kfields6@schools.nyc.gov)

The following resources are available to support implementation and execution of Regents scanning.

*Please do not refer school users directly to Central without first attempting to resolve the issue.

3737

Any Questions?

CFN 201 Support: CFN Assessment Liaison - Amy Reynolds

areynolds@schools.nyc.gov 917-940-7319

Data/IT Systems - Jerry Lin

jlin6@schools.nyc.gov 718-281-7561