ASSESSMENTS & ACCOUNTABILITY

Assessments and Accountability v3.0

ASSESSMENTS & ACCOUNTABILITY

Suspense Resolution and

Reports

Rules

HOLDMute yourself *6

1conversation

at a time

Silencecellphones


Address yourself when speaking

2

Who Should to Be Here

Individuals assigned toCALPADS maintenance

&Individuals responsible for

Assessment and Accountability Data


Prerequisites


Agenda

Terminology Overview

Wrap-Up


Verify Resolution

Reports& Extracts

Managing Suspense

5

Terminology



Terminology

Term Used Definition How you might know them

Program Testing Program STAR, CAHSEE, CELDT

Assessment STAR = CST, CMA, CAPA, STSCAHSEE = CAHSEECELDT = CELDT

Administration Program AdministrationIndicator for a Program Testing Cycle

STAR = AnnualCAHSEE = Jul, Oct, Nov, Dec, Feb, Mar, MayCELDT = Jul, Aug, Sep, Oct, Nov, Dec, Jan, Feb, Mar, Apr, May & Jun

Calendar Program Testing Calendar The Calendar for each Program Administered

Content Area Test Content Area (Academic Subject)

ODSR Operational Data Store Replicated - Replication (copy) of the ODS, changes in suspense are maintained in the ODSR

New

AADS Assessment/Accountability Data Storage(reports & snapshot) – used for processing reports and snapshots and holding historical assessment results

New

7


Terminology (cont)

Term Used Definition How you might know them

Pre-ID Pre-Identification process is a service using student information provided to the test vendor in advance to "pre-plug" on test answer documents or create labels

Results Record Vendor results for an SSID for a Program administration

Suspense Suspense records are records that were loaded into CALPADS for your LEA however due to a specific discrepancy they did not match with any SSID in CALPADS associated with your LEA

Academic Performance Index (API)

API reflects a school’s, an LEA’s, or a student group’s performance level, based on statewide testing results

Adequate Yearly Progress (AYP)

Statewide accountability system which requires each state to ensure that all schools and districts make Adequate Yearly Progress

Continuous Enrollment

Enrollment from Census Day thru First Day of administration date. Students who were not enrolled for more than 30 consecutive calendar days are considered not continuously enrolled8


Overview

9


A/A Purpose

Building Comprehensive

Longitudinal Data Set

Creating a complete individual student assessment

history for STAR, CAHSEE, and CELDT

10

A/A Functionality


Current Assessment and Accountability Process outside of CALPADS

Pre-Id Service

Answer Documents

Demographic Data and Continuous Enrollment

Request

Pre-ID Docs Returned

Testing Vendor

Scored Results

CDE Analysis, Measurement & Accountability Reporting

Division

AYP/API Results

Assessment Results to CDE

Pre-ID or hand-bubbled demographic completed

answer documents sent to assessment vendor for

scoring

Local Student Data (Electronic or Manual)

OptionalSTART

HERE

DataQuest/Public

Assessment Results back to LEA

Demographic Data Corrections back to Testing Vendor - Optional


2012/2013 A/A Processes

The initial phase of A/A functionality requires no change in the way LEAs currently work with assessment vendors or with CDE

Student Demographic changes will be made through the testing vendor

Testing Vendors will provide test results & reports

Pre-ID, where applicable, will continue

Testing processes will be directed by assessment program offices

API results will use the student demographic data that LEAs supply to the test vendors however, CALPADS enrollment information will be used

for the 2011 Base API and the 2012 Growth API to determine the Continuously enrolled and whether or not AEP (Alternative Education Program)

student test results will be rolled back.

AYP results will use the student demographic data that LEAs supply to the vendors as well as the continuously enrolled data supplied to the vendor –

AYP will NOT use CALPADS data for the current year

12

A/A Functionality


Current Assessment and Accountability Process includes CALPADS

Pre-Id Service

Answer Documents

Demographic Data API Continuous Enrollment

Request

Pre-ID Docs Returned

Testing Vendor

Scored Results

CDE Analysis, Measurement & Accountability

Reporting Division

AYP/API

Results

Assessment Results to CDE

Pre-ID or hand-bubbled demographic completed answer documents sent to assessment

vendor for scoring

Optional

Local Student

Data (Electronic or Manual)

START HERE

DataQuest/Public

Assessment Results back to LEA

Demographic Data Corrections back to Testing Vendor - Optional

AYP/API results imported to CALPADS

Results Records Loaded in CALPADS

Enrollment data pulled from CALPADS

SENRContinuous

CALPADS

API Continuous Enrollment

Manage Suspense

A/A Reports

Accountability Reports

Session Focus


This session will focus on

The process of loading the Results Records in CALPADS

The matching of those Results Records to records in CALPADS

The LEAs expectation of this process (Managing Suspense)

Using reports to assist with local data decisions

14


A/A Results Loading Process

Loading Assessment Data in

CALPADS

1

2

3

4

START

Depending on continual loading of data,

(Vendor updates, P1, P2, P3)

15

A/A Results Records Loading


Tentative calendar of loading the Results Records in CALPADS

P1 P2 P3 Monthly Quarterly Pre DRM Final

July

August

September

October

November

December

January

February

March

April

May

June

STAR CAHSEE CELDT

16


A/A Responsibilities

CALPADS

17

A/A Enrollment records Pulled

CALPADS Data Pull from ODS

Enrollment Data Pulled

(Census Day – First

Day of Testing)*

Enrollment Data Pulled for 11th and 12th grade

students who take CAHSEE

July-Dec

Used to Determine:

STAR DataCAHSEE

DataAPI AYP Released

2011-12 Reporting Cycle

August 15 2012 2011-12 2010-11 2012 Growth 2012 Oct 2012 2011-12 2011-12

December 20 2012 2011-12 2010-11 2012 Growth 2012 Revised - Jan 13

2011-12 2011-12

February 26 2013 2011-12 2010-11 2012 Growth 2012 Final - May 13

2011-12 2011-12


Tentative Calendar for pulling Enrollment data from CALPADS

18

Continuous Enrollment – Enrollment from Census Day thru First Day of admin date.For all students with STAR scores, students who were not enrolled for more than 30 consecutive calendar days are considered not continuously enrolled and will not be included in the API/AYP calculations.


MANAGING SUSPENSE DATA

19


1. Program Results Records are loaded into CALPADS

2. A nightly process is run which matches vendor Results Records to records in CALPADS

Linking the Results Records to CALPADS Records

3. Five attributes are used during this process matching records before “linking” results to a student.

4. Results Records that did not match or “link” are placed in suspense for LEAs to resolve.

Managing Suspense

20


Suspense records are Results Records that were loaded into CALPADS for your

LEA however due to a specific discrepancy they did not match or link with any SSID in

CALPADS associated with your LEA

What are Suspense Records?

Managing Suspense

21


Managing Suspense

Importance of Suspense Resolution:

22

Matching process

criteria


Managing Suspense

When all criteria is met the results are “matched” however as soon as one of these conditions is not met, CALPADS places the record in “suspense”.

Most current results from test vendor

Demographics in CALPADS

System generated Matching Process - Nightly

Matched No action needed

SuspenseFurther action needed

23


Managing Suspense

Date of Birth Year Mismatch

Student Date of Birth Year for SSID in Results Record <not equal to> Student Date of Birth Year in CALPADS (ODS) data that was valid at the time of testing – Birth Year does not match

Gender Mismatch

Student Gender Code for SSID in Results Record <not equal to> Student Gender Code in CALPADS (ODS) data that was valid at the time of testing – Gender does not match

Demographic Data Mismatch

No demographic data in the ODSR valid at the time of testing

Admin Date Empty

The Administration Date in the Results Record is blank

School Mismatch

If enrollment in School at the time of testing could not be validated – no SSID enrollment

LEA Unmatched

LEA manually unmatched a resolved record

Suspense Error Descriptions

24

Nightly

Process

Managing Suspense


Student Profile Data-Enrollment (SENR)-Student Information (SINF)-Student Program (SPRG)

Student Profile Data copied from….ODS to ODSR real timeODSR to AADS in …-Manage Suspense Resolution Processes (LEA & CDE)-Suspense Process (Nightly)-Snapshot Process (Data Load & CDE)

Reporting from AADS requires 1)Vendor test results and 2)CALPADS student data

Real Time

Copy of ODS and where you manage your Assessment Data (Manage Suspense)

ODSR

Where you manage your Student Profile data

ODS

Where Assessment reports are generated from

AADS

STAGING

Where all the data is loaded

Assessment Process in CALPADS

Why? - When? - How?

25

Real Time

Operational Reports

Reports used in Verifying and Managing Suspense


Managing Suspense

A user can “force” the match when there is a discrepancy in the data OR when the data has been corrected with the Vendor (but new results not yet loaded)

IMPORTANT NOTE: Once data is matched, forced or automatic, that record will not appear in suspense again, even when new data is uploaded, unless there is a change in enrollment ownership

Results from test vendor

Demographics in CALPADS

Force Matching Process – Initiated by User

SSIDBirth YearMissing AdminEnrollmentGender




2001 2002FORCE

26


Managing Suspense

Condition Cause of record in suspense Action taken to resolveResolution

Method

SSID Mismatch

Correct Results Record SSID does not match CALPADS

Correct CALPADS first then either 1 or 2

Incorrect Results Record SSIDContact vendor to correct before or after match (optional)

3 or 4

Demographics Mismatch(Birth Date or Year/Gender)

Correct Results Record demographics do not match CALPADS

Correct CALPADS first then either 1 or 2

Incorrect Results Record demographicsContact vendor to correct before or after match (optional)

3 or 4

LEA Mismatch(Enrollment Not Validated)

Incorrect CALPADS enrollment Correct CALPADS first then either 1 or 2

Incorrect SSID in Results RecordContact vendor to correct before or after match (optional)

3 or 4

SSID Administration Date Missing

Blank Administration Date in Results RecordContact Vendor to correctNote : If blank in STAR, Admin date is automatically filled in with the first day of testing window

Resolution Method

1 Manual Match Provides Manage Suspense Report real-time

2 Nightly Suspense Process System match automatically, providing report data the next day

3 Subsequent load of vendor data Allow subsequent load of vendor data to resolve automatically, providing report data at that time

4 Force Match Allows user to force match the records with discrepancies

Determine Resolution and Action to take

* Results Record are data loaded from the vendor

27


Managing Suspense

Suspense Resolution Actions

Suspense Resolution Methods

The LEA correcting CALPADS data or

The vendor provides a corrected results file after LEA corrects data on vendor site or

The LEA and/or CDE assigning the suspense record to the correct SSID

The LEA correcting CALPADS data or

The vendor provides a corrected results file after LEA corrects data on vendor site or

The LEA and/or CDE assigning the suspense record to the correct SSID

Manual match which provides reporting data to Manage Suspense immediately

or

Force match which provides reporting data to Manage Suspense immediately

or

Automated match (nightly suspense) which provides reporting data to Manage Suspense the following day

Manual match which provides reporting data to Manage Suspense immediately

or

Force match which provides reporting data to Manage Suspense immediately

or

Automated match (nightly suspense) which provides reporting data to Manage Suspense the following day

Note: Results records from vendor will be used

for AYP/API including uncorrected data

Actions and Methods for Correcting Records in Suspense

28

Managing Suspense


NO

YES

Both or 1Local

Decision

Either Option

Managing Suspense

Remember….

If Results reports are in error, changes must be made through Vendor to be reflected in AYP and API calculations

Calendar of vendor loads/updates


Results Record = Vendor results that were uploaded into CALPADS

30


Managing Suspense

Steps taken in CALPADSFor clearing Suspense Records

31

CALPADS Roles


CALPADS LEA STAR Coordinator

Assessment and Accountability

CALPADS LEA CAHSEE Coordinator

CALPADS LEA CELDT Coordinator

CALPADS LEA Accountability Coordinator

1

2

3

4

32

A/A Landing Page


CDE Only Manage Program

Administrations View Assessment Event

Status

Assessment Landing Page

33


1. Select filter for Program and Administration (subject to user’s access rights), and Academic Year

2. Select Academic Year

Assessments > Online Maintenance > Manage Suspense Data

Managing Suspense

34

1

2

Selecting report criteria and filters


Assessment > Manage Suspense Data > Search > Student List

Managing Suspense

35


Managing Suspense

Error Analysis Example #1

Condition =

Record in Suspense does not belong to your LEA

Resolution = Pass on to the CDE as “not mine”

36

Assessment > Manage Suspense Data > Search > Student List


Managing SuspenseManaging Suspense

37

Record immediately removed from list


Managing Suspense


Condition =

Results Record loaded (vendor data) is incorrect

Resolution = Force the match to current record for your LEA in CALPADS Determine locally if vendor data will be corrected outside of CALPADS

38


Managing Suspense

Select Match initiates the user-defined search for a matching CALPADS SSID

39


Managing Suspense

Enter SSID and/or demographics, using * wildcard if needed

Enter Search Criteria

40

Results Record

CALPADS Search M


Managing SuspenseReview Results: Matching results are displayed by SSID and/or Demographics, depending on the criteria entered

41

Suspense Results Record

CALPADS Search criteria

Matched criteria results in CALPADS


Managing Suspense

Choose Match: Make selection and to initiate the match

42

CALPADS Search



M


Managing Suspense

Both the suspense Results Record and the CALPADS selected match record are displayed

Match Records

43




Managing Suspense

Record Matched Successfully – Return to Student List

44

Managing Suspense


Remember what data are used on reports

The demographic data used on Reports is from the Results Record – Not the ODS

When you force the match, ask yourself what data is incorrect? Is it the results data?

Understand that you are forcing the two data sources to “link,” however if the results data is incorrect, that incorrect data will be displayed on the reports.


Managing Suspense


Condition = Record in suspense due to Calpads Data being Incorrect

Resolution = Correct data in CALPADSOption 1 force the match

Option 2 let nightly process match records once corrected

46


Managing Suspense

Select Match initiates the user-defined search for a matching CALPADS SSID

47


Managing Suspense

Enter SSID and/or demographics, using * wildcard if needed

Enter Search Criteria

48

Results Record

CALPADS Search M


Managing Suspense

Review Results: Matching results are displayed by SSID and/or Demographics, depending on the criteria entered

49


CALPADS Search


M


Managing Suspense

Correcting Data in CALPADS

50


Managing Suspense

Correcting Data in CALPADS

51


Managing Suspense

Resolving Suspense on Corrected CALPADS records

Select 1 of 2 options after correction has been made

Manually Match by selecting match, search

and find student and match records

(prior steps)

OR Nightly Auto

Matching process

Matched

52


Managing Suspense - Batch

Managing Suspense via the Batch upload process

53

The batch process is designed for LEAs with a large volume of records in Suspense

ONLY options for batch are to force the match or send to CDE for matching

For further information regarding Managing Suspense via the Batch process please submit a service request ticket


VERIFY RESOLUTION

54


Assessments > Online Maintenance > Manage Suspense Data

Verify Resolution

55


Confirm Matched Records

Verify Resolution

56


Resolve Incorrect Match - Unmatch

Verify Resolution

57


Resolve Incorrect Match - Not Mine

Verify Resolution

58


Recap Suspense Workflow

Suspense Resolution Workflow:

LEA Workflow:• Review Suspense Records• Determine best resolution and apply• Wait for data refresh and verify suspense resolved

CDE/CALPADS Process

LEA Process

Verify Data on Reports

59


Key Points in Managing Suspense

Data refresh timing depends on the method you chose when resolving the suspense record

Manual match with immediate resolution Automated match (nightly with resolution and data

refresh at completion of process) Records in Suspense that are matched manually will

be reflected in the Operational Reports immediately, all other Assessment reports, the update will not show until the following day

Reminder: When you make changes to data in the ODS, they are replicated to the ODSR in real time. However, the change will not resolve a suspense until one of the following:

You manually match the suspense data to the corrected CALPADS record or

Nightly Suspense processing clears the suspense

60


BreakQuestions

61


Assessments & Accountability Reports

& Extracts

62


A/A Reports

The following types of Assessment reports are available

REPORTby

Summary Individual Students Not

PassedStudents Not

TestedStudents Not

Included

STAR (2) (3)

LEA

School

CAHSEE (1) (2) (3)

LEA

School

CELDT (1) (2) (3)

LEA

School

AYP/API (2) (3)

LEA

School

Individual Student Assessment History (1) (3)

School

1. Currently Enrolled in LEA regardless of administered by2. Administered by LEA only3. By Subgroup

63


A/A Reports

Assessment reports use vendor score/result data for an SSID that is matched to CALPADS Student Profile data for that SSID

Remember …

64

Report content for a Program Administration can continue to change as long as suspense processing is occurring for that Program Administration and until the snapshot is frozen


A/A Reports

.

Reports can continue to change to

reflect changes in the data

65


A/A Reports

ASSESSMENT MENU

Two entry points from Assessment Menu

66


A/A Reports

xxxxxxxxx

REPORTS

67


A/A Reports

REPORTS (Cont)

CDE Only50.1 Assessment

Upload – Detail52.1 Assessment

Snapshots - Detail

68


A/A Reports

Description: Describes the content of the report

Data as of: IMPORTANT - provides as of date (loaded date)

Data Included: Testing Window, describes what data is included in report

Totals: Describes criteria if any for totals – “how” data is totaled

Links: Additional supporting report (drill down)

Glossary References: Specific terms used in report

The following information is available in the CALPADS User Manual for each report.

69


UNDERSTANDING STAR REPORTS

70


Understanding STAR Reports

Star Report ProcessStar Report Process

71

Data Load/Review Schedule

STAR July August SeptemberResults P1

LoadP2

Load

LEA Data Review LEAs can make changes in CALPADS and through

vendor.

October November DecemberResults P3 Load

LEA Data Review LEA Data Review – LEAs can make changes in CALPADS

and through vendor.

January February MarchResults P3 Load Post P3 Loads – includes LEA specific results that have been

approved by CDE through the appeals process.

LEA Data Review LEAs may make changes in

CALPADS and through vendor.

April May JuneResults

LEA Data Review


Assessment Results are loaded throughout the school year


A/A Reports

xxxxxxxxx

REPORTS

73

A/A Reports


STAR Report 20.1- LEA Summary


Results based onabove criteria

74


UNDERSTANDING CAHSEE REPORTS

75


Understanding CAHSEE Reports

CAHSEE Report ProcessCAHSEE Report Process

76


CAHSEE July August SeptemberResults Monthly

- MayQ4 - All Monthly

- JulyQ1

LEA Data Review After last ADMIN, LEAs can make corrections for all ADMINs during the year. Must be completed by Oct 8

October November DecemberResults EOY – Final -

CumulativeMonthly

- Oct

LEA Data Review Changes must be

completed by Oct 8

January February MarchResults Monthly

- NovMonthly

- DecQ2

LEA Data Review

April May JuneResults Monthly

- FebMonthly

- MarQ3

LEA Data Review




Understanding CAHSEE Reports

REPORTING TYPES

78


A/A Reports

xxxxxxxxx

REPORTS

79

A/A Reports


CAHSEE Report 21.6 – Enrollment Student List


80


UNDERSTANDING CELDT REPORTS

81


Understanding CELDT Reports

CELDT Report ProcessCELDT Report Process

82


CELDT July August September

Results

LEA Data Review

October November December

Results

LEA Data Review

January February March

Results Pre-DRM Load

LEA Data Review LEAs may make changes in DRM

April May June

Results Post-DRM Load Final

LEA Data Review




A/A Reports

xxxxxxxxx

REPORTS

84

A/A Reports


CELDT Report 22.3 – English Learner Status


85


UNDERSTANDING AYP/API REPORTS

86


Understanding AYP/API Reports

AYP/API Report ProcessAYP/API Report Process

AYP - All data are used from Vendor test resultsAYP - All data are used from Vendor test results

87


AYP/API July August SeptemberResults P1 Load

LEA Data Review LEAs can make changes in CALPADS and through vendor.

October November DecemberResults P2 - AYP/API report and data files that

incorporate P2 STAR late testing LEAs, CAHSEE data corrections, appeal and

exception decisions, ad CAPA reallocations

P3 Load

LEA Data ReviewLEAs may make changes in CALPADS and through vendor.

January February MarchResults Post P3 Load – Final AYP and Growth API that incorporate all Post-P3 changes (February)

and Final Base API (March)

LEA Data Review LEAs may make changes in

CALPADS and through vendor.

April May JuneResults

LEA Data Review




A/A Reports

REPORTS



Snapshots - Detail

89




Sample Legend





UNDERSTANDING INDIVIDUAL STUDENT

ASSESSMENT REPORTS

93


A/A Reports

REPORTS (Cont)



Snapshots - Detail

94


Understanding Individual Reports

Individual Student Report ProcessIndividual Student Report Process

95


Understanding Student Reports

REPORT 40.1 Individual Student Profile

96



REPORT 40.1 Individual Student Profile (cont)

97



REPORT 40.1 Individual Student Profile (Cont)

Entries appear only if used in report

98



VIEW STUDENT RESULTS - Online

99


UNDERSTANDING OPERATIONAL REPORTS

100


A/A Reports

REPORTS (Cont)



Snapshots - Detail

101


Verify Resolution

51.1 Assessment Suspense - Summary

Results Below

102


Verify Resolution

51.2 Assessment Suspense – Student Detail

Results Below

103


UNDERSTANDING

EXTRACTS

104


Understanding Extracts


WRAP-UP

106


Wrap-Up

It is critical that the Admin Date is correct in the Calendar file– The Admin Date is used for determining whether the student was enrolled during

the testing window Independently Reporting Charters (IRCs) testing independently need to

update with the test vendor IRC testing dependently (with the district) can submit a separate calendar

entry for their IRC, otherwise the entry for the parent LEA will be used The actual testing date is not reported in the results data for STAR and

CELDT, so CALPADS uses a window based on the first date of testing to validate enrollment

Reports display results of most recent load +/- suspense resolutions until CDE notifies the field of the final snapshot, which will freeze the data (no certification takes place)

.

KEY POINTS

107

Wrap-Up

The CDE Analysis, Measurement & Accountability Reporting Division uses results data provided by the test vendors and CALPADS enrollment and demographic data for calculating AYP/API results as follows:

– API results will use Student demographic data that LEAs supply to the test vendors CALPADS enrollment information will be used for the 2011 Base API and the

2012 Growth API to determine the Continuously enrolled and whether or not AEP (Alternative Education Program) student test results will be rolled back.

– AYP results will use Student demographic data and continuously enrolled data supplied to the

vendor – AYP will NOT use CALPADS data for the current year


KEY POINTS (cont)


Wrap-UP

Please Complete Our SurveyQuestions

109