Measuring Child Progress: Two State’s Journeys Barbara Jackson, NE Beppie Shapiro, HI Measuring...

Measuring Child Progress:Measuring Child Progress:Two State’s JourneysTwo State’s Journeys

Barbara Jackson, NEBarbara Jackson, NE

Beppie Shapiro, HIBeppie Shapiro, HI

Measuring Child and Family Outcomes

Albuquerque, NM

April 25, 2006

Format of PresentationFormat of Presentation

• Meet each stateMeet each state

• Overview of the 4 key questions to Overview of the 4 key questions to be discussedbe discussed

• For each key question:For each key question:

– Description of each state’s plansDescription of each state’s plans

– Questions and comments from Questions and comments from participantsparticipants

What Counts:

Measuring Benefits of Early Intervention in Hawai`i

Beppie Shapiro, University of Hawai`ibeppie@hawaii.edu

About Hawai`i

About 17,000 births per year

Over 7% of each birth cohort served in Part C

No majority ethnicity

4.1 % of population are immigrants, primarily Asians and Pacific Islanders

Geographic isolation

Five main islands – different access to services

Special education in Hawai`i

Part C under Department of Health

Part B under Department of Education

Outcomes process differs from C to 619

SEA = 1 LEA

Part C in Hawai`i, 2006

Broad eligibility

Three DOH agencies provide services statewide

Recent development of statewide IFSP to be used by all programs

No Part C database – 2 agencies have their own

Standardized IFSP form, process

2005: All Part C programs trained to use new statewide IFSP form– Present Levels of Development– Family-driven Outcomes* and Objectives– Services to support Objectives– Transition

What Counts?....What Counts?.... Measuring the Outcomes of Measuring the Outcomes of

Early Intervention in NebraskaEarly Intervention in Nebraska

Barbara Jackson, Munroe Meyer Institute

bjjackso@unmc.edu

About NebraskaAbout Nebraska

Birth Mandate State

Co-Leads for Part C: Health and Human Services & Education

Outcome data process will be the same across the birth through 5 age group

Nebraska (continued)Nebraska (continued)

460 School Districts

Number of children birth to 3: 1303

Number of children ages 3 and 4: 2811

Number of children 3 through 5: 4707

Key Question…………Key Question…………

1.1. How will the state determine children’s How will the state determine children’s status on each of the outcomes?status on each of the outcomes?

Assessment in Hawai`i

Part B Section 619, Hawai`i Assessment: Early Brigance

Part C: Since 2004: 3 assessment tools approved for CDE – Hawaii Development Charts (HELP), “Michigan” EIDP, or ASQ

Hawai`i’s Part C Assessment : Ages and Stages

For babies eligible due to environmental risk

Forms for specific age levelsScores indicate only “Typical”, “Monitor”, “Refer” Referred for CDE and services by program serving DDCare coordination may remain with original program

1. What Counts Design Team convenes monthly to develop/review progress

2. How can we assign a “score” to each child on each EI Goal?

What Counts: Measuring the Benefits of Early Intervention in Hawai`i:

Hawai`i’s Part C Outcomes (Goals) Measurement Process

Data Collection Intervals & Rationale– At every IFSP: initial, review, annual

–Starting when child is at least 4 months old

Hawai`i’s Part C Outcomes (Goals) Measurement Process

Assigning scores: decisions & rationale

WHO: IFSP team

Following “Present levels of Development” description

HOW: all team members reach consensus on rating using modified ECO Child Summary Reporting Form

Hawai`i’s 619 Outcomes Measurement Process

Family involved in assessment process Family provides input on written form with open-ended questionsA professional assigns rating on each goal Other providers asked for input to ratingRatings assigned at entry and annually

Nebraska’s ProcessNebraska’s Process

Child Outcome Task Force convened to guide process

Nebraska calls for Child Assessment Nebraska calls for Child Assessment that….that….

Is based on ongoing observation of children engaged in real activities, with people they know, in natural settings

Engage families and primary caregivers as active participants

Is individualized to address each child’s unique ways of learning

Reflects that development and learning are rooted in culture and supported by the family

Integrates information across settings

Nebraska’s Assessment ProcessNebraska’s Assessment Process

Which assessments?

AEPS Creative Curriculum High Scope COR for Infants/

Toddlers & Preschoolers

Why Selected?Why Selected?

Assessment approach parallels other Early Childhood Program assessment processes

Use information from multiple sources (e.g., family, providers) and multiple observations

Curriculum-based Assessment – can be used for multiple purposes

Can build the capacity of our system to support children and their families

Nebraska’s Nebraska’s Process for parent input?Process for parent input?

Parents provide input during the assessment process

How is the assessment information How is the assessment information “transformed”?“transformed”?

Working with publishers to determine feasibility of computer-based formulas

Scores will be reviewed by team to assure validity of score

Nebraska’s Assessment ProcessNebraska’s Assessment Process

Data Collection Schedule Districts will be mandated to report entry

and exit data Entry data will collected within the 45-60

days after IFSP/IEP meeting Exit data collected within 2 months of

leaving the program Districts will be encouraged follow

publishers guidelines for frequency of assessments

Key Question…………Key Question…………

2. What reporting categories has the 2. What reporting categories has the state chosen to use?state chosen to use?

Reporting Categories in Hawai`i

What Counts: Measuring Benefits of Early Intervention in Hawai`i

Reporting Categories in Hawai`i

Part C Decision & RationaleUse ECO 5 CategoriesA. maintained typical functioning

B. made progress to achieve typical functioning

C. moved nearer to typical functioning

D. progress but not enough to move nearer to typical

E. did not make progress

Meaningful, Program Improvement

Reporting Categories in Hawai`i

619– OSEP Categories

Reporting Categories in NEReporting Categories in NE

Four Categories OSEP’s 3 categories

Plus % of children who attain typical development

Key Question…………Key Question…………

3. How and in what form will data get 3. How and in what form will data get from local programs to the state?from local programs to the state?

Getting Part C Data to the StateIn Hawai`i

Simpler than most states?

More difficult than many states?

Each “Agency” will collect and summarize data from its programs

Data to explain results

Length of enrollmentAge at enrollmentICD9s/conditions

Getting Data to the State: Part C

Electronic data entry at local programTransmitted to AgencyAgency calculates number of children in each categoryAgency sends these numbers to State State creates OSEP and cross-Agency reports

Unresolved data questions

Local programs send “explanatory data” to Agency– How much of this data goes to State?

How?

How to identify potential duplicates

Getting Data to the State: 619

Teacher enters data into stand-alone EXCEL spreadsheet

Spreadsheets collected at school, sent to State

NE: Getting Data to the StateNE: Getting Data to the State

Using the publisher’s internet system- State will be the licensed manager

Link with the State Data System

Key Question…………Key Question…………

4. What will you do to maximize the 4. What will you do to maximize the reliability of the data?reliability of the data?

Training, Support to Maximize Data Reliability in Hawai`i

Training:Training:

Support:Support:

Web:Web:

7 hrs, all EI programs, by 7 hrs, all EI programs, by communitycommunity

1-2 months post training: 1-2 months post training:

On site? Conference call? On site? Conference call?

FAQs, email listservFAQs, email listserv

Maximizing Data Reliability in Hawai`i: Evidence for Rating

IFSP teams document evidence used to select rating category

Pilot showed need for training on what makes good evidence

Supervisors provide support and QA

Criteria for Evidence Criteria for Evidence (in progress)(in progress)

Different contextsDifferent contexts

Highest level of Highest level of achievementachievement

Specific to Goal AreaSpecific to Goal Area

Examining Reliability in Hawai`i

Another measurement of “how well child is doing”

– Compare ratings

Compare ratings for children with different conditions/diagnoses

Challenges for Hawai`i: Quality Assurance - Part CHow reliable are assessments?

Does every IFSP team have someone knowledgeable about typical child development?

How reliable are summary form ratings?

Reliability of Data in NE: TrainingReliability of Data in NE: Training

Intensive training on assessments throughout state with consistent trainers

Reliability of Data in NE: Reliability of Data in NE: ImplementationImplementation

Review of data at the state level

Ongoing questions/answers communication document based on local questions

Team approach to assessment process

Pilot process to determine reliability issues