1

Indicator #7: Measuring Preschool Outcomes

Pat Cameron, Department of Early Education & CareDonna Traynham, Department of Elementary and Secondary Education

October 2008

2

What We Will Cover

Why collect outcomes data? Understanding the child outcomes Assessing the accomplishment of the 3

child outcomes Introduction to the Child Outcomes

Summary Form (COSF) Collecting and reporting data using the

COSF

3

Essential Knowledge for Completing the COSF

Team members know about: The child’s functioning across settings The child’s functioning across settings

and situationsand situations Age-expected child developmentAge-expected child development Content of the 3 outcomesContent of the 3 outcomes How to use the rating scaleHow to use the rating scale

4

Why Collect Outcomes Data?

5

Public Policy Context

Age of accountability

Accountability increasingly means looking at results – not just process

Office of Special Education Programs (OSEP) is under increasing pressure to produce outcomes data on children participating in early intervention and early childhood special education programs

6

OSEP Response

Required states to submit outcomes targets and data in their State Performance Plans and Annual Performance Reports (SPP/APRs)

Funded the Early Childhood Outcomes Center to make recommendations, and to assist states in collecting, reporting and USING outcome data

7

OSEP Reporting Requirements: Child Outcomes

1. Positive social emotional skills (including positive social relationships)

2. Acquisition and use of knowledge and skills (including early language/ communication [and early literacy])

3. Use of appropriate behaviors to meet their needs

8

OSEP Reporting Categories

Percentage of children who: -Did not improve functioning; -Improved functioning, but not sufficient to

move nearer to functioning comparable to same-aged peers;

-Improved functioning to a level nearer to same-aged peers but did not reach it;

-Improved functioning to reach a level comparable to same-aged peers; or

-Maintained functioning at a level comparable to same-aged peers

9

Schedule for State reporting to OSEP

Entry/baseline data on 1,700 preschool children with disabilities in Year 1 Cohort were reported to OSEP in February, 2007

Progress data on nearly 900 preschool children exiting ECSE from Year 1 Cohort; along with entry/baseline data on 1,624 children from Year 2 Cohort was reported to OSEP in February, 2008

Progress data on Year 1 and 2 Cohorts of children exiting ECSE and entry/baseline data on Year 3 Cohort will be reported to OSEP in February, 2009

Progress data are based on the difference between each child’s status at entry and child’s status at exit

10

MA Baseline Data reported to OSEP on February 1, 2007 for Year 1 Cohort

Social-Emotional

Acquisition of skills & knowledge

Behavior to meet needs

% Age Appropriate

20 21 33

% Not Age Appropriate

80 79 67

11

Progress Data of Year 1 Cohort

Domain Category A

Category B

Category C

Category D

Category E

Positive social emotional skills

4% 25% 33% 18% 20%

Acquisition of skills and knowledge

4% 25% 32% 19% 20%

Use of behaviors to meet needs

4% 18% 29% 17% 32%

12

MA Baseline Data reported to OSEP on February 1, 2008 for Year 2 Cohort

Social-Emotional

Acquisition of skills & knowledge

Behavior to meet needs

% Age Appropriate

20 21 32

% Not Age Appropriate

80 79 68

13

Why Collect Outcomes Data?

Federal government is the driving force behind the move to collect outcomes data

However, providing data for the federal government is not the only reason to collect outcomes data

14

Why Collect Outcomes Data?

Data on outcomes are important for state and local purposesTo document program effectiveness• Support continued or increased

fundingTo improve programs• Identify strengths and weaknesses• Determine technical assistance

and/or staff development needs

15

State Design

All states are required to measure child outcomes for early childhood programs. However, the strategies chosen are based on the values held by the state about assessing young children.

16

MA Model for Indicator #7

Cohort Model – 4 cohorts with 70-90 districts each year, doubling up in year 3

Random Sample of 40 preschool students with disabilities

Baseline/entry data collection November (due to the Department in December)

Progress/exit data collection in May (due to the Department in June)

Subsequent progress /exit data collection in May 2010 and May 2011

17

Understandingthe 3 Child Outcomes

18

3 Child Outcomes: Assumptions

Children have positive social-emotional skills (including social relationships)

Children acquire and use knowledge and skills (including early language/ communication [and early literacy])

Children use appropriate behaviors to meet their needs

19

Outcomes are Functional

Functional refers to things that are meaningful to the child in the context of everyday living

Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes

They are not a single behavior, nor are they the sum of a series of discrete

behaviors

20

Outcomes are Functional

They cross domains– do not separate child development into discrete areas (communication, gross motor, etc.)

Emphasis is on how the child is able to carry out meaningful behaviors in a meaningful context

21

Children Have Positive Social Relationships

Involves:Relating with adultsRelating with other childrenFor older children- following rules

related to groups or interacting with others

Includes areas like:Attachment/separation/ autonomyExpressing emotions and feelingsLearning rules and expectationsSocial interactions and play

22

Children Acquire and Use Knowledge and Skills

Involves: Thinking, reasoning, remembering, problem-

solving Using symbols and language Understanding physical and social worlds

Includes: Early concepts – symbols, pictures, numbers,

classification, spatial relationships Imitation Object permanence Expressive language and communication Early literacy

23

Children Take Appropriate Action to Meet Their Needs

Involves: Taking care of basic needs Getting from place to place Using tools In older children, contributing to their own

health and safety Includes:

Integrating motor skills to complete tasks Self-help skills (e.g., dressing, feeding,

grooming, toileting, household responsibility) Acting on the world to get what one wants

24

Elaboration of the ECO OutcomesElaboration of the ECO Outcomes

Children have positive social relationships

Children acquire and use knowledge and skills

Children take appropriate action to meet their needs

Relation-ships with

adults

Relation-ships with

peers

Follows group rules

Masters the environ-ment

Symbol use, abstract thinking

Appliesknowledge

Listening PlayExploring

Playing

Being curious

Practicing

Touching

Attending

EngagingPersisting

Knowledge of physical world & culture

Self-care, health and safety

25

Assessing the Accomplishments

of the 3 Outcomes

26

What is Assessment?

“Early childhood assessment is a flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical, and mental health services needs of young children and their families.”

Bagnato and Neisworth, 1991Quoted in DEC Recommended Practices, 2005

27

DEC Recommended Practices for Assessment

Involves multiple sources (e.g., families, professional team members, service providers, caregivers)

Involves multiple measures (e.g., observations, criterion-curriculum-based instruments, interviews, informed clinical opinion)

28

Assessment Instruments

Potential: Assessment tools can inform us about children’s functioning in each of the 3 outcome areas

Challenge: There is no assessment tool that assesses the 3 outcomes directly

29

The Assessment Tool Lens

Each assessment tool carries its own organizing framework

Many are organized around domains But what is covered in the domains isn’t

always the same, even if the names are the same

30

Currently Available Assessment Tools

There are not right or wrong assessment tools

Key question to ask about any assessment tool:

How much and what information will How much and what information will the tool provide about the attainment the tool provide about the attainment of the 3 functional child outcomes?of the 3 functional child outcomes?

31

Using Data Collected from Assessment Tools

ECO has “crosswalked” assessment tools to the outcomes

Crosswalks show which sections of assessment are related to each outcome

The number of items addressing an outcome does not necessarily mean that the assessment captures functioning across settings

32

The High/Scope Preschool Child Observation Record (2003): Crosswalk to Child Outcomes

Outcome 1:

Positive social relationships

Outcome 2:

Knowledge and skills

Outcome 3: Action to meet needs

I. Initiative C. Initiating play [social context of play] II. Social Relations E. Relating to adults F. Relating to other children G. Resolving interpersonal conflict H. Understanding and expressing feelings V. Language & Literacy Q. Listening to and understanding speech

III. Creative Representation I. Making and building models J. Drawing and painting pictures K. Pretending V. Language & Literacy R. Using vocabulary S. Using complex patterns of speech T. Showing awareness of sounds in words U. Demonstrating knowledge about books V. Using letter names and sounds W. Reading X. Writing VI. Mathematics & Science Y. Sorting objects Z. Identifying patterns AA. Comparing properties BB. Counting CC. Identifying position and direction DD. Identifying sequence, change, and

causality EE. Identifying materials and properties FF. Identifying natural and living things

I. Initiative A. Making choices and plans B. Solving problems with materials D. Taking care of personal needs

Note: Areas that are not precursor to or components of any of the three outcomes, and therefore not included in the crosswalk, were: IV. Movement & Music:

L. Moving in various ways M. Moving with objects N. Feeling and expressing steady beat O. Moving to music P. Singing

33

Making Use of Information from Assessment Tools

Information from formal or published assessment tools can be very useful, as long as it is used in the context of achievement of the three functional outcomes

The information almost always needs to be supplemented with additional information

34

MA Direction and Decisions

Using the Child Outcome Summary Form (COSF)

Rating children annually in the fall and spring

Using information from assessment tools currently in use in local districts

Gathering data from multiple sources

35

Using the Child Outcomes Summary Form (COSF)

36

Thinking About the Achievement of Each Child Outcome

Movementaway fromage-expected

Movementtowardage-expected

Age-expectedskills and behavior

37

Helping Children Move Toward Age-expected functioning

Assumption: Children can be described with regard to how close they are to age-expected behavior in each of the 3 outcomes

By definition, most children in the general population demonstrate the outcome in an age-expected way

By providing services and supports, ECSE is trying to move children closer to age expected behavior

38

Measuring Functioning Compared to Age-Expectations

Documenting children’s movement toward age-expected development is one type of evidence that program services are effective

The Child Outcome Summary Form was designed to measure this type of progress

39

Essential Knowledge for Completing the COSF

Team members need to know: The child’s functioning across settings

and situations Age-expected child development Content of the 3 outcome areas How to use the rating scale

40

Child Outcomes Summary Form

41

Summary Ratings are Based on…

Types of Evidence Curriculum-based

assessments (e.g., Creative Curriculum)

Norm-referenced assessments (e.g.,BDI-2)

Developmental screenings (e.g., Ages and Stages)

Parent and professional observation and report

Sources of Evidence Parents and family

members Service providers Therapists Physicians Child care

providers Teachers People familiar

with the child in all the settings and situations that he/she is in

42

Group Sharing

Questions? Needed clarifications? Comments and reactions? What TA and/or other supports will you

need?

43

Contact Information Pat Cameron

Sr. Policy Specialist, Special EducationDepartment of Early Education and Care51 Sleeper StreetBoston, MA 02210617-988-7812patricia.cameron@massmail.state.ma.us

Donna TraynhamElementary School ServicesDepartment of Elementary and Secondary Education350 Main StreetMalden, MA 02148781-338-6372dtraynham@doe.mass.edu