The NC KEA Pilot - UNC Charlotte 2014 NC K-3 FAP-KEA Pilot...The NC KEA Pilot: Findings from the...

The NC KEA Pilot:

Findings from the Field Angela Ferrara and Rich Lambert

University of North Carolina at Charlotte

Background

In the fall of 2014, researchers from UNC Charlotte

conducted case studies at 8 schools piloting a new

kindergarten formative assessment known as the NC KEA.

The purpose of this research was to provide feedback to

the Department of Public Instruction regarding teacher and

administrator perspectives on professional development for

the KEA, the assessment’s content and format, the

electronic platform used during the KEA process, and how

the KEA formative assessment worked in practice. The

following presentation is a summary of these findings.

Presentation Overview

• Pilot Participation

• Feedback from the Field – Summary of KEA Pilot Findings

• The Electronic Evidences – How did Pilot Teachers use the Teaching Strategies® NC KEA System?

• Conclusions

Pilot Participants

Case Studies

8 Schools (1 per SBE region)

23 Teachers

7 Principals

4 Districts Administrators

4 Instructional Coaches

Full KEA Pilot

193 Active Pilot Teachers

51 Pilot Districts

Pilot Process • Teachers were tasked with gathering and entering ‘evidences’

of learning: • Anecdotal notes

• Photographs

• Videos

• Student work samples

• Evidences were used to denote the ‘learning status’ of students along different construct progressions: • Book and print awareness

• Following directions

• Letter naming

• Number counting

• Fine motor

• Engagement in self-selected activities

• Emotional Literacy

Data Sources

• Case Studies:

• 6 schools visited twice, 2 schools visited once

• 23 interviews (approx. 27 hours recorded)

• 17 classroom observations

• Survey:

• 72 total responses

• 52 Teachers, 16 Administrators, 4 Instructional Coaches

• 18 closed-ended questions (yes/no, Likert Scale)

• 26 open-ended questions

CLASSROOM

OBSERVATIONS

Methods

• Recorded on Observation Site Overview Form and in

descriptive field notes

• Codebook generated and all observations were coded

• Coded data analyzed to identify patterns within and

between classrooms where the KEA was observed in use

as part of normal classroom instruction and classrooms

where the use of the KEA was not evident.

Classroom Snapshot

• Average class size of 20 students (min 13, max 25)

• Average of 12 minority students

• 94% had a teaching assistant present during the entire

observation

• Classroom technology available:

• Smart board (86%)

• Desktop or laptop computer (92%)

• Tablet or other mobile device (46%)

A Spectrum of KEA Implementation

Implementing Classrooms

• Small class sizes: 14 student average

• Students easily transitioned from one

activity to another independently

• School/District had a strong

background in the use of formative

assessment

• Teachers used self-created

implementation resources to assist

KEA documentation

• Teachers worked collaboratively

• Schools had strong PLCs with a

continual focus on data driven

instruction

Non/Minimally Implementing Classrooms

• Large class sizes: 22 student average

• Students struggled to transition independently between classroom activities

• Teachers often preoccupied with behavioral interventions

• School/District did not have a strong background in the use of formative assessment

• Teachers conceptualized and implemented the KEA as a summative assessment

• Created new/additional activities to “test” each child’s ability rather than using current instruction or assessment data

Teacher Created Resources

INTERVIEWS AND

SURVEY

Interview Methods • Separate interview protocols developed for teachers and

administrators

• Questions covered the following broad topics:

• Pilot training

• Current instruction and assessment practices

• Integration of KEA into classroom practice

• Use of the KEA electronic platform

• Use of KEA data to inform instruction

• Implementation supports provided and/or needed

• Teachers interviewed as a focus group

• Administrators interviewed one-on-one

Survey Methods

• Internet based survey hosted through SurveyShare.com

• A separate set of questions was developed for teachers

and administrators/support staff

• Questions covered the following broad categories:

• Pilot training

• KEA assessment content

• Use of the KEA electronic platform

• Implementation supports provided and/or needed

• Overall pilot experience

Analysis Methods

• Interview transcripts and open-ended survey questions

were imported to NVivo 10 for grounded discourse

analysis.

• A single codebook was developed:

• 119 unique codes generated

• 3952 total references to these codes noted in the data

• Coded data was analyzed to identify themes and patterns

within the interview and survey responses.

Training Disagree Neutral Agree n

The training was well organized. 29.41% 19.12% 51.47% 68

The training was sequenced appropriately. 27.94% 20.59% 51.47% 68

The training materials provided became a good resource when implementing the pilot.

30.88% 25.00% 44.12% 68

After attending training I understood the purpose of the KEA.

23.53% 16.18% 60.29% 68

After attending training I understood the formative nature of this assessment.

19.12% 14.71% 66.18% 68

After attending training I understood the content of the progressions.

36.76% 16.18% 47.06% 68

After attending training I understood when to use a situation/task.

33.82% 20.59% 45.59% 68

After attending training I could identify current instruction or assessment practices that could act as evidences for the progressions.

27.94% 14.71% 57.35% 68

After attending training I felt confident in my ability to upload evidences to the online system.

54.42% 14.71% 30.88% 68

After attending training I understood how to pull reports from the online system.

60.30% 16.18% 23.53% 68

After attending training I felt prepared to use the evidences and progressions to drive instruction.

50.00% 22.06% 27.94% 68

After attending training I understood how the domains affected one another.

39.70% 22.06% 38.23% 68

Color Legend

Structure and Materials

KEA Philosophy and Content

Gathering Evidences

Using Online System

Application to Instruction

Training Continued

• Training should extend to additional school/district

personnel:

• Real world examples of the KEA in use and sample

student data should be incorporated into the training (39

references): • Provide preloaded documentation of various types (video,

photographs, student work samples, etc.) that teachers can

annotate, analyze, and place along the progressions.

• Provide examples (video preferred) of a teacher using the KEA

process in real time during normal classroom instruction.

Training Continued

• Greater focus is needed in the application of KEA data (i.e. how to

make meaningful planning and instructional decisions based on the

evidences and progression ratings). (39 References)

• 57% of surveyed teachers stated they could not make meaningful instructional

decisions from the evidence and progression ratings they entered.

“I’m putting all of this information in, but I’m getting nothing

out. How is this supposed to help me get my students to a

D in reading? That’s all administrators and parents care

about.” ~Pilot Teacher

NC KEA Content

• 71% of survey respondents felt the content was

developmentally appropriate for kindergarten.

“This really validates what we do and deal with

everyday…there’s so much that needs to happen before

you see a lot of academic changes. These young children

are going to be growing socially tremendously [in the

beginning of the year] and administrators need to

understand [teachers] have all this other stuff to get in

place before they can start moving academically.”

~ Pilot Teacher

NC KEA Content Continued • The NC KEA is developmentally appropriate BUT…

• 34 cross references between code “KEA

Instrument>Developmentally Appropriate” and code “KEA in

Practice>Misalignment with Current Curriculum and Assessment

Practices”

“Is this developmentally appropriate? Yes, but to be honest we don’t have the

‘freedom’ to use it. We are mandated by so many other expectations for our

children that there is no way to do the KEA the way it should be done and still

be responsible for the content we must teach and then assess them on (state

mandated summative tests by the way).” ~Pilot Teacher

“If this was 5 years ago this would have been perfect, but kindergarten is

looking more and more like first grade. So while this IS developmentally

appropriate, unfortunately it’s now more suited for PreK because kids are

expected to enter Kindergarten with most of these skills.” ~Pilot Teacher

KEA in Practice • The amount of time needed to complete the KEA was the foremost

topic in every interview and survey response:

• 133 references at code “KEA in Practice>Time Consuming”

• There were 583 individual uses of the word ‘time’!

Times when Evidences were Uploaded

In-Class 28.57%

Regular Planning Time 24.49%

Immediately

Before/After School 46.94%

Personal Time at Home 79.59%

Times when Evidences were Analyzed

In-Class 8.33%

Regular Planning Time 29.17%

Immediately

Before/After School 47.92%

Personal Time at Home 79.17%

KEA in Practice

• 8 specific examples of the KEA process used appropriately were noted in the interviews and survey responses.

“When given the opportunity to select an activity on her own, one of my students would initially wander around the room looking for her sister (a twin). When she

would see her sister playing with other children, the first twin would sit alone at her seat not choosing to be with other children. I started providing more partner games during my instruction that gave my fist twin a chance to interact with other children besides her sister. After several instances of this, she felt more comfortable joining

groups with other children and interacting.” ~Pilot Teacher

“I like the KEA assessment because it can be used with any child in kindergarten. I have an autistic, but very bright, student. He doesn’t respond to standardized tests but with the KEA I had the time and ability to observe, identify, and document many

strong skills from this child.” ~Pilot Teacher

Supports and Resources

• Requests to assist implementation:

• Additional planning time (43 references)

• An alignment guide of KEA constructs to current

district/state curriculum and assessments (49

references)

• Real world examples of data collection during class

instruction (39 references)

• Peer networking or partnership (34 references)

• Often cross-referenced with wiki or online blog (11 references)

• Dedicated technology support, at the district level

minimally (32 references)

ELECTRONIC EVIDENCES

Electronic Evidences Overview

• 193 pilot teachers created/populated classes in the

Teaching Strategies® NCKEA System

• A total of 13,422 pieces of documentation were entered

over the course of the pilot.

• More evidence was entered in September than any other

month in the pilot.

Breakdown Average

Evidences per teacher 67.79

Evidences per student 3.52

Evidences per Teacher

0

5

10

15

20

25

Emotional Fine Following Approaches Letter Number

Literacy Motor Directions to Learning Naming Counting

Evidences per Child

0.00

0.20

0.40

0.60

0.80

1.00

1.20

Emotional Fine Following Approaches Letter Number

Literacy Motor Directions to Learning Naming Counting

Further Analysis

• We are currently reviewing each piece of evidence

• Examining type and usefulness in assisting instructional

decisions

• Currently, we have reviewed 4,290 evidences

• 15 districts, 22 schools, 44 teachers

Further Analysis Continued

• Evidence is specific to a single child • Yes – 80.1%

• No – 19.9%

• Evidence contains child specific information useful for placing children on the progressions • Yes – 49.1%

• No – 50.9%

CONCLUSIONS

• Professional development needs a greater focus in areas

outside of the assessment’s core content:

• Early childhood education

• Using the electronic platform

• Qualitative data collection and its use to drive instruction

• Schools and districts vary widely in the supports needed

to effectively implement given their current capacities and

resources.

• There was a spectrum of implementation initially, but

many teachers quickly “learned by doing” and felt more

confident in their ability to effectively use the KEA toward

the end of the pilot.

• This type of formative assessment initiative will face some

opposition, even suspicion, in the current academic

climate.

CEME Technical Report

• A report summarizing these findings can be found at

http://ceme.uncc.edu/ceme-technical-reports.

• For correspondence regarding this study please email:

• Richard Lambert: rglamber@uncc.edu or,

• Angela Ferrara: aferrar2@uncc.edu