Hawaii Statewide Hawaii Statewide Assessment Assessment Program:Program:Maintaining Test Maintaining Test Security andSecurity andReporting Testing Reporting Testing IncidentsIncidents

Fall 2008Hawaii Department of EducationSystems Accountability Office (SAO)

1

This presentation focuses on testing incidents and the procedures for reporting testing incidents and inquiries into those reports.

2

Board of Education Policy Board of Education Policy No. 2520No. 2520STATEWIDE ASSESSMENT STATEWIDE ASSESSMENT PROGRAMPROGRAM POLICYPOLICY

“Systematic school, curricular, and program improvement efforts directed toward student attainment of the Hawaii Content and Performance Standards (HCPS) requires the collection of relevant and accurate student assessment information.”

3

Maintaining Integrity of theMaintaining Integrity of theHawaii State Assessment Hawaii State Assessment

(HSA)(HSA)Criterion Referenced Test (CRT) and Norm

Referenced Test (NRT) data

Curricular, instructional, personnel, and financial decisions

Improved student achievement

Strict adherence to test administration and security procedures is required to ensure that the student assessment results are accurate.

4

HSA ProceduresHSA Procedures

All school personnel involved in the administration of the HSA are accountable for knowledge of and adherence to all test security procedures as specified for their roles.

5

HSA guidelines and procedures can be found in the Test Administrator’s Handbook (TAH), the Test Coordinator’s Handbook (TCH) and the Student Participation and Accommodations Handbook (SPAH).

Additional mandatory resources:◦HSA District Meeting sessions or school

level trainings◦DVD: Hawaii Statewide Assessment

Program

Watching the DVD and attending the mandatory training are not substitutes for reading the HSA handbooks.

6

Hawaii State Alternate Assessment Hawaii State Alternate Assessment (HSAA) &(HSAA) & Hawaiian Aligned Portfolio Hawaiian Aligned Portfolio

Assessment (HAPA)Assessment (HAPA)

HSAA procedures can be found in the:◦HSAA Rater/Coordinator Manual◦HSAA Test Coordinator’s Handbook

HAPA procedures can be found in the:◦School Administrator - Test

Administrator - Test Coordinator Handbook

7

Testing IncidentsTesting Incidents

Any event or procedure that could potentially impact the integrity of the tests, the data, and the test results before, during, and after the test administration is a testing incident.

All testing incidents must be reported immediately.

8

Types of Testing IncidentsTypes of Testing Incidents

◦Testing irregularity

◦Standardized administration irregularity

◦Test security violation9

Testing IrregularityTesting Irregularity

Is an incident that affects the student’s performance on the test, but may, or may not, compromise the test result.

Must be reported May or may not trigger an inquiryMay require the administration of the

replacement test formMay require invalidation of the student’s

score

10

Testing Irregularity Testing Irregularity ExamplesExamples

Include, but are not limited to:Electrical outage during testingFire alarm during testingMany students taken ill due to vogHurricane evacuation during testingMedical emergencies involving key

personnelDefective materialsStudent bubbled the wrong portion of

the response booklet11

Standardized Administration Standardized Administration IrregularityIrregularity

Is a change in the established test administration procedure and protocols that may, or may not, compromise the test result.

Must be reported May or may not trigger an inquiryMay affect the student’s performance on the

testMay result in the administration of the

replacement test formMay require invalidation of the student’s

score12

Standardized Administration Standardized Administration Irregularity ExamplesIrregularity Examples

Include, but are not limited to: Disallowed classroom displays containing content

information visible during test sessions (see handbooks for details)

Test related content instruction, review or practice tests are provided between testing sessions (if it implies prior knowledge of test items, this could be a test security violation)

Test administrator fails to read the directions or conduct the examples as directed

Providing inappropriate accommodations Selecting specific students to “not test” Restricting or extending time to complete the testing

sessions13

Test Security ViolationTest Security ViolationA test security violation is a serious incident

that compromises test security, data security, or both, and the integrity of the student score results.

Must be reportedWill trigger an inquiry into the incidentMay affect the student’s performance on the

testMay result in the administration of the

replacement test formMay require invalidation of the student’s scoreSuperintendent will be notified

14

Test Security Violation Test Security Violation ExamplesExamples

Include, but are not limited to:• Use of mobile phones or cameras during testing

sessions to record test items• Test items or student responses are copied,

discussed, reviewed or analyzed before, during or after the assessment is administered

• Test administrator or proctor leaves the classroom unattended or with an unqualified adult during a session (see handbooks for qualifications)

• Test booklets are assigned to test administrator or proctors for their reference and review

• Test responses are altered• Prompting (verbal or non-verbal) or giving answers

to students during testing to review specific items• Students sharing answers, creating cheat sheets or

developing group answers

15

Procedure, Procedure, Sequence and Sequence and Expectations forExpectations forReporting a Reporting a Testing IncidentTesting Incident

16

Who Reports? Who Reports? All Hawaii Department of Education

employees are required to report testing incidents to the school Administrator.

Test Coordinators and school Administrators must report any knowledge of testing incidents to:

Student Assessment Section (SAS) Administrator

Complex Area Superintendent

17

Students, Parents, and Students, Parents, and Community Members Community Members

Non-DOE personnel may report testing incidents to school personnel.

School personnel receiving such

information are responsible for reporting the information to the school Administrator.

18

Test Coordinators and Test Coordinators and School AdministratorsSchool Administrators

If you have witnessed, been informed of, or suspect the possibility of a testing incident that could potentially impact the integrity of the tests, the data, and the test results, immediately contact:

Student Assessment Section (SAS) Administrator at 808-394-1300

In no answer, call… System Planning and Improvement (SPIS)

Administrator at 808-586-3285

19

If No Answer or After If No Answer or After HoursHours

Send a LN email to Kent Hinton

(cc: Christina Tydeman and Cara Tanimura) Provide: ◦ Your name, ◦ Your school, ◦ Phone numbers with hours for contact, and◦ The nature of your call.

You will be contacted by one of the individuals within your specified hours of contact.

The SAO Director will inform the Superintendent of all reported testing incidents.

20

Besides School Besides School Personnel Reporting to Personnel Reporting to

SAS…SAS…

State level data processing can reveal potential testing incidents or school level data quality concerns that may trigger an inquiry.

21

“ “Red Flag” ExamplesRed Flag” Examples

Include, but are not limited to:• Reviews of student demographic data

– Subgroup identifiers (i.e., ethnicity, SPED, ELL, disadvantaged)

– Grade level/birth date• Analyses of testing data– Student performance trends (e.g., school level,

grade level, subgroups)– Irregular scoring patterns– Inappropriate accommodations provided– Out of grade level testing

• Missing testing materials• Altered documents• Test scores for students who have been

exempted from taking the test by parent letter

22

Testing Irregularity FormTesting Irregularity Form

All schools reporting any testing incidents must complete and submit a “Testing Irregularity Form” for each incident to the Student Assessment Section.

23

24

Information Requested When Information Requested When Reporting to SASReporting to SAS

• Role/position of individual reporting – Name is optional

• School name• Date(s) of incident(s)• Description of incident(s)–Type (e.g., cheating, missing materials, copying

items)

–Scope (e.g., individual, class, grade level, school-wide)

–By whom (e.g., student, school personnel)

25

Step 1: Did a Testing Incident Step 1: Did a Testing Incident Occur?Occur?

What happened?Who was involved?How widespread?Who initiated the incident?

26

Step 1: OutcomesStep 1: Outcomes

If no testing incident is found, no further action is needed.

If a test/administration irregularity is found, SAS shall direct school personnel with appropriate actions.

If a test security violation is suspected, the school shall be directed to take immediate action (e.g., stop all testing) and a inquiry team shall be sent to the school.

27

Step 2: If an Inquiry is Step 2: If an Inquiry is NecessaryNecessary

School Administrator and the respective Complex Area Superintendent or Charter School Executive Director will be notified.

Upon notification, School Administrator must immediately:◦Cease testing◦Provide full cooperation and assistance

to the inquiry team Private room for interviews by the inquiry team Access to all individuals involved in the incident Copies of documents upon request of SAO

28

Documents to Have Documents to Have HandyHandy

FERPA forms – with parent/guardian signature for students involved in the incident

Test Administration Training Session Attendance Verification Form

Hawaii State Assessment Security and Administration Procedures Acknowledgment Form

29

Step 2: OutcomesStep 2: Outcomes

Inquiry team is established.◦Systems Accountability Office (SAO) personnel

◦Dispatched to the school upon notification of alleged test security violation

30

Step 3: The Site VisitStep 3: The Site Visit

Upon arrival at the school, the team will:

Meet with Test Coordinator/School Administrator◦Summarize and clarify the testing

incident◦Share the scope of the inquiry◦Clarify the roles of school personnel

within the inquiry◦Discuss possible outcomes◦Review copies of all FERPA forms

Conduct interviews◦Staff may have union representation

31

Scope of the InquiryScope of the Inquiry

Who will be interviewedHow the findings will be determinedWhen and if testing will be resumedPossible consequences regarding

test security and the validity of test scores

When the report will be completed

32

At the End of the Site VisitAt the End of the Site Visit

The inquiry team will brief the Administrator and Complex Area Superintendent or Charter Schools Executive Director on the findings.

33

Components of the Inquiry Components of the Inquiry ReportReport

• School context• Summary of the inquiry methodology• Descriptions of the incident, findings,

and conclusions• Recommended actions and timeline–Test security and administration

procedures–Validity of scores No recommendations for disciplinary

actions shall be included.

34

Suppressed Test ScoresSuppressed Test Scores

In the event the inquiry team determines a student’s scores to be invalid, the scores and the related test materials will be suppressed.◦In other words, it will be as if the

student(s) did not take the test session(s).

Schools may not view any testing materials or scores that have been suppressed as a result of a testing incident inquiry.

35

Effect of Suppressed Effect of Suppressed ScoresScores

Scores from individual testing sessions may be deemed invalid resulting in partial scores in the subject area for the affected students.

If all of a student’s scores for a subject area are deemed invalid, the student will have no score and will be considered a non-participant.

36

Step 3: OutcomesStep 3: OutcomesSAO makes the final determinations of score validity.

Inquiry report is presented by SAO Director:

Complex Area Superintendent or Charter Schools Executive Director

State Superintendent.

Superintendent approves the final recommended actions for improving test security and administration procedures.

37

Possible DeterminationsPossible Determinations

1. No testing incident occurred 2. Test/administration irregularity occurred =

valid scores◦ Did not compromise test security◦ Did not compromise test results (i.e., scores)

3. Test/administration irregularity occurred = invalid scores

◦ Did not compromise test security◦ Compromised test results (i.e., scores)

4. Test security violation occurred = invalid scores◦ Compromised test security◦ Compromised test results (i.e., scores)

38

At the Superintendent’s At the Superintendent’s Discretion…Discretion…

The Superintendent may share the testing incident inquiry report with the Board of Education or the Charter Schools Review Panel.

The school may be required to cover the costs of any replacement test forms used and other related costs (e.g., re-printing of score reports and testing materials, hiring raters).

39

Step 4: Public DisclosureStep 4: Public Disclosure

Approved recommendations are implemented.

The school Administrator must notify the affected parents in writing of the test security violation and its impact on their children’s scores.

The school, in consultation with the DOE’s Communications Office, must respond to Freedom of Information requests regarding the incident.

40

Overall OutcomesOverall OutcomesImproving test security contributes to

Improved validity and accuracy of assessment scores.

Transparency of the test security inquiry process for schools.

Public accountability through full compliance with No Child Left Behind.

Increased credibility and overall confidence in the student performance results. 41

For more information For more information regarding Testing regarding Testing Incidents and Inquiries, Incidents and Inquiries, please contact:please contact:

Systems Accountability Office at 808-586-3283

42