Embed Size (px)
Citation preview
July 24, 2014
QRIS National Meeting – Denver, Colorado
Leveraging Technical Assistance Data to Support QRIS
2
Overview of Session
o Introductions and Overview
o A Review of the Research
o Leveraging TA Data
o Promising State Practices & Insights
o Practices in Pennsylvania & South Carolina
o Challenges and Limitations
o Group Discussion
3
Facilitators and Panelists
o Early Learning Challenge Technical Assistance
Michelle Thomas, State Support Team
Kenley Branscome , State Support Team
o University of South Carolina Yvonne and Schuyler
Moore Child Development and Research Center
Herman Knopf, Research Director
o Pennsylvania Key
Katrina Coburn, Director of Workforce Development and
CQI Initiatives
4
Purpose of Today’s Discussion
Help states to understand the ways in which they can
leverage technical assistance data from QRIS and data
from other systems to inform the evaluation of quality
improvement strategies ands strengthen management
practices for QRIS and related systems.
5
Objectives for Today’s Discussion
During today’s discussion, states will:
Learn about promising practices
and insights from other states
Identify important data elements
and data linkages to consider
Learn about critical challenges and
limitations
Learn about tools and resources
that can assist with defining policy
questions and data elements
6
Review of the Research: Improving Outcomes
Overall, coaching is inked to:
o Improved learning environments
o Improved practices, including teacher-child interactions
o Improved developmental outcomes for children
7
Review of the Research: Effective Features
Effective features of coaching and consultation include:
o Uses a clearly specified approach or model
o Linked to other professional development
o Tends to follow a general sequence
o Tracks the progress of coaches (and teachers they serve)
o Tracks the fidelity of implementation
o Provides support to coaches and opportunities to process
the work with a peer or supervisor
8
Review of the Research: Importance of Data
Research indicates that it is important to consider:
o Dosage of TA (e.g. frequency, duration, span)
o Type of TA (e.g. consultation, coaching, mentoring, PLC)
o Purpose of TA (e.g. specific rating goal, curriculum implementation,
domain specific practices)
o Specific TA strategies used (e.g. observation, modeling, assessment,
feedback)
o Content of TA received
o Qualifications of TA provider
o Recipient of TA (e.g. lead teacher, team of teachers)
o Mode of TA (e.g. in-person, video, phone)
9
“The goal is to turn data into
information, and information
into insight.”
Carly Fiorina, Former CEO -
Hewlett-Packard Co
10
Data relevant to technical assistance may exist in multiple systems.....
Linking data from different systems can provide new information to answer essential research and management questions
Workforce
Registry
QRIS Data
System Training
Registry
Subsidy
Data
System
Licensing
Data
System
Multiple Systems May Contain Data Relevant to TA
11
Leveraging Data to Answer Essential Questions
From those systems, states can leverage data on....
Early Learning Program & Practitioner Demographics
Technical Assistance Provider Demographics and Qualifications
Quality Improvement Plans and Technial Assistance Activities
Technical Assistance Delivery Methods
To answer essential research and management questions related to QRIS and other
quality initiatives
12
o What specific features of TA are linked to
positive outcomes (e.g. Dosage, Intensity)?
o Is TA more likely to be associated with positive
outcomes if the program receiving TA is
operating at a certain level of quality
(threshold)?
o Are there particular TA approaches/methods
that are more highly correlated with positive
outcomes?
Example Questions States Are Trying to Answer
13
Example Questions States Are Trying to Answer
(Cont’d) o Are there particular qualifications for coaches that are
more highly correlated with positive outcomes?
o Are there particular TA approaches/methods that are
more successful with home vs. center programs?
o Are TA specialists responding to TA requests within the
expected response timeframe(s)?
o Is the supply of TA specialists adequate to meet the
demand for services within each operating region of
the state?
Reflection: Think about 1-2 questions your state is trying to
answer about TA being provided to programs in QRIS. Jot them
down, we will come back to them.
14
Promising State Practices
o States are using TA data to answer essential research questions related
to the impact that TA has on program quality
o States are linking their TA data to other ECE systems to reduce data
collection burden and reduce duplicative data entry
o States are using mobility technology to collect TA data to produce
efficiencies and strengthen TA data quality
15
State Insights
o Start small (if you can) and scale up only at times when the
system is stable
o Qualitative data can be useful in case-specific issues, but can be
a challenge for broader quantitative analysis
o High stakes may make trades offs necessary between collecting
data elements that are required for funding and those that
measure other important outcomes
o Engage TA consultants and researchers early and often during the
development of data systems to ensure you collect the most
essential data elements
16
References
o Aikens, N., & Akers, L. (2011). Background review of existing literature on coaching: Final report.
Los Angeles: First 5 LA. Retrieved April 16, 2012, from
http://www.first5la.org/files/07110_502.2CoachingLitRev_FINAL_07072011.pdf.
o Smith, S., Robbins, T., Schneider, W., Kreader, J., & Ong, C. (2012). Coaching and quality
assistance in quality rating improvement systems: Approaches used by TA providers to improve
quality in early care and education programs and home-based settings. New York: Columbia
University, National Center for Children in Poverty. Retrieved February 7, 2012, from
http://nccp.org/publications/pdf/text_1047.pdf.
o Tout, K. (2013). Quality Improvement Supports in QRIS. Presentation to the National Center for
Research on Early Childhood Education Spring 2013 Quality Improvement Meeting. Available at:
http://curry.virginia.edu/research/centers/castl/ncrece-spring-2013-meeting
o Tout, K., Isner, T. K., & Zaslow, M. (2011). Coaching for quality improvement: Lessons learned from
quality rating and improvement systems (QRIS). Washington, DC: Child Trends. Retrieved May 25,
2011, from http://www.childtrends.org/Files//Child_Trends-2011_04_27_FR_CoachingQuality.pdf.
o United States. Department of Education. Policy and Program Studies Service. (2011). Cross-site
evaluation of the Early Childhood Educator Professional Development Program. Washington, DC:
U.S. Department of Education, Policy and Program Studies Service. Retrieved June 2, 2011, from
http://www2.ed.gov/programs/eceducator/ecepdbrief.pdf.
17
Let’s Hear From You!
FINDING OUT WHAT WORKS
Pennsylvania
Katrina Coburn
Technical Assistance
o Providers request TA through their Regional Key, which will
support them to maintain or increase their STAR level.
o A Technical Assistance Organization and TA Consultant will be
assigned to work with the facility to develop an Action Plan.
The Action Plan will include goals and action steps the
facility will work towards in order to improve their standards
and ultimately achieve their target STAR level.
Pennsylvania STARS
20
Technical Assistance in STARS Programs
o STARS TA
o Infant/Toddler Specialist Program
o Child Care Health Consultation
o School Age Child Care TA
o School Age Child Care – After School Quality TA
o Certification Referral Technical Assistance
PELICAN Overview
Department of Public Welfare’s initiative to create a system that integrates the
Department’s child care programs under a single management information system
o All child care services
information is managed in
PELICAN
o Prior to 2011/12 STARS
Technical Assistance (TA)
Process was predominantly
paper-based
o K2Q Phase 2 initiative
incorporated the STARS TA
program into the K2Q
system
Certification
PA Pre-K Counts
Early Learning Network
Child Care Works
Keys to Quality
Child Assessment
Technical Assistance
22
o Request for TA/Program/ Facility Information
o Intake/Contact Log
o Action Plan
o Progress Interaction Log
o TA Organization and consultant
o Goals Met/Not Met
o Close-Evaluated Process
What is collected in PELICAN?
Pennsylvania Early Learning Keys to
Quality
Technical Assistance
23
STARS TA Accountability & Quality Assurance
Measurable impact on provider
80%
Qualifications, Professional Development & Professionalism
90%
Reporting Obligations
90%
Basic eligibility Knowledge of system, standards, process Required and ongoing professional development Appropriate preparation for consultation
Timeliness, accuracy of reports Records maintained for referrals Timely follow-up and completion of referral Communication with Regional Key Meets targets
Reviews plans Closed-Evaluated between April 1 and March 31 on the following factors:. Achievement of specific goals as identified in Action Plan (80%*) STAR level change (min. 33% meet)
* Up to 10% may be justifiably exempt
Pennsylvania Early Learning Keys to
Quality
24
Pennsylvania Early Learning Keys to
Quality
Keystone STARS Technical Assistance
Research Brief
(February, 2013)
o 425 providers received some form of STARS TA
o Average of 17.9 Hours Direct Service per Action Plan
o Odds of moving up a STAR level 4.3 times higher than for those providers who did not receive TA
o Most effective for Group Child Care providers
o Most effective at lower STAR Levels
o http://www.ocdelresearch.org/
Measureable Impact on Provider
25
Lessons Learned
o Professional development for all TA Consultants on
action plan and goal writing
o Strategies as a data field in PELICAN
o Considerations for specialty discipline TA types
(Business, Health and Safety) in certification
process
26
o Alignment of all TA services to new Core
Knowledge Competencies
o TAAP for ECMH Blended Model
o Technical Assistance Strategies Recommendations
o Interaction Log Standards
o Technical Assistance Sustainability Plans
What’s Next in Pennsylvania?
Pennsylvania Early Learning Keys to
Quality
EVOLUTION OF TA IN SOUTH CAROLINA:
A JOURNEY DICTATED BY PURPOSE
Herman Knopf
University of South Carolina
28
Stage 1: Began with Varied Regional Reporting
Department of Social Services
Region1
Region 2
Region 3
Region 4
29
Stage 1: Had Single Purpose and Single Consumer
Purpose
•Accountability
Consumer
•DSS
30
Stage 2: Began Using CCR&R to Collect Reports
Regions
CCR&R
DSS
31
Stage 2: Expanded Purpose of Reporting
DSS •Accountability
CCR&R
•Productivity
•Accountability
32
Stage 3: Centralized Data Entry
Regions
CCR&R Data entry
services included
DSS
33
First TA Form Had “Too Much Information”
SC#CCR&R#Network#TA#Summary# Page#1#
!
On-Site Technical Assistance Summary CCR&R
Date: TA Start Time: TA End Time:
TA specialist
Early Childhood Program: Administrator Name:
Location Address: Email: Phone:
County: Years of Experience: Director TA Yes No
Education Level: High School CDA ECE Certificate ECE Associate Degree ECE Bachelor Degree
ECE Master or Higher Other
Referral Source/
Contact Initiated By
Early Childhood Program
DSS Licensing ABC Program ABC Special Needs
PITC Network SC First Steps Other:
License Type
Licensed Registered
Approved Exempt Provisional
Pending
Type of Care
Child Care Center Family Child Care
Group Family Child Care Preschool School-Age
Head Start Specialty Center Family, Friend, Neighbor
In-Home
ABC Program
A+ A B+ B
C N/A
Consultee
Owner Director Teacher(s)
Assessment Tool
ITERS ECERS
SACERS FCCERS BAS
PAS CLASS Licensing
Level B Other:
Program Statistical Data
# of Classrooms
# Children Enrolled # Staff # Staff Master Degree
# Staff ECE BA Degree # Staff ECE AA Degree # Staff ECE Certificate
Classroom Statistical Data # Children Enrolled
# Children w/ SN: # Staff:
Target Area(s)
Activities/Curriculum
Child Development Health-Safety Nutrition
Physical Activity Soc-Emo / Behavior Language-Reasoning
Parent Involvement Program Structure
Space & Furnishing
Arrangement / Schedules Special Needs Staff Development
Business Plan/ Admin Policies & Procedures Disaster plan
Other:
Type of Technical Assistance # of Visits :
Type I-Issue Focused
(Content specific -topical, immediate or crisis
intervention
Type II-Guided
(Formal assessment-based, targeting specific key areas that improve global program quality)
Type III-Reflective (Provider identified goals and shared formal assessment)
Objectives : Technical Assistance Overall Goal
Resolve Corrective Action Maintain License in Good Standing Achieve Licensing Status Increase ABC Score above 80% Maintain ABC Rating Level Increase ABC Rating
Achieve Quality Improvement Plan Increase ERS Subscale Score Increase ERS Overall Score
Follow-Up Reason for Visit: (description of request for TA: supervision problems, ratios)
Development of Action Plan
Schedule ERS Assessment Schedule On-Site Staff Training (Target Area, Reflection) Early Learning Standard Resource Library Materials
Training Opportunities in Community Community/Curriculum Resources
Date of Next Contact: Time: Objective: Click!here!to!enter!text. Signature of Consultee: _ Date:
!!!!!!
SC#CCR&R#Network#TA#Summary# Page#2#
!
Notes
Classroom 1 Title: # of children attending: Duration of visit:
TA type: Age of Children in classroom
First Visit Infant/Toddler 0-2
Plan of Action development Preschool 3-5 Observation School Age 6-12 Mentoring Multi – age (what ages):
Assessment Training (on-site
Children w/ Special Needs yes no; if yes, # TA topic :
#1 Teacher Name: # Years of Experience: Email:
Education Level: High School CDA ECE Certificate ECE Associate Degree ECE Bachelor Degree ECE Master or Higher
#2 Teacher Name: # Years of Experience:
Email: Education Level: High School CDA ECE Certificate ECE Associate Degree !
Target Areas: Activities/Curriculum Child Development
Health-Safety Nutrition Physical Activity
Soc-Emo / Behavior Language-Reasoning Parent Involvement
Program Structure Space & Furnishing Arrangement / Schedules
Special Needs Staff Development Policies & Procedures
Disaster plan Other:
ECE Bachelor Degree ECE Master or Higher
Outcome of visit:
Do you feel that the objectives of the TA visit were sufficiently met/accomplished?
Strongly Agree Agree Disagree Strongly Disagree
Actions to be taken by Provider before next visit::
Actions to be taken/resources to be provided by CCR&R TAP before next visit:
!
Technical!Assistant!!!!!!!!!!!!!!!!!!!!!!!!!!!!Teacher!!! !!!!!!!!!!!!!Date!!!!!!!! !!!!!
!
!
!
34
Stage 3: Expanded Purpose of Reporting
DSS • Accountability
CCR&R
• Productivity
• Accountability
• Research
35
Stage 4 – Balance of Information
Regions
CCR&R
36
Stage 4 – Regional Offices Began Using the Data
DSS • Accountability
CCR&R
• Productivity
• Accountability
• Research
Regions • Productivity
37
Began Discussing Internal Reports
o Had rich discussion on:
The interpretation on elements of the report
Better efficiency
Better measurement of effectiveness
Better reflection of the complexity in the field
– Specifically intensity of services
– Duration of services
38
Consumers and Purpose Changed Over Time
Consumer
DSS
CCR&R
Regions
Purpose
Accountability
Productivity
Research
39
Measurement Issues We Grappled With
o Successful completion of the task – activities list
o Content of TA (appropriateness of action plan, time spent at the
CCP, etc.)
Vs.
o Intensity
Vs.
o Duration
Vs.
o Increase in quality level
Vs.
o System change - maintenance of behavioral change
40
Who is the right evaluator for TA?
o TAP
Vs.
o CCP
Vs.
o CCR&R Director
41
Revised Data System
41
42
Program Information
42
43
Gathering Context
43
44
Employees
44
45
Facility TA History
45
46
Action Plan- Sessions
46
47
Activities
47
48
Plan Closeout
48
49
Lessons Learned in South Carolina
o Importance of engaging key stakeholders – including TA specialists
and evaluators - early and throughout the design and
implementation process
o Ability to import data into tables in SAS supports more robust
analytic capabilities
o Importance of attending to the usefulness of all data you are
attempting to collect
o Leveraging data from other systems reduces burden of data
collection
50
What’s Next in South Carolina?
o Continued testing and refinement of TA data system
o System wide implementation
o Connect TA data with Child Care Training Registry data
o Use data to better understand the form and function of Technical
Assistance
51
Questions for Pennsylvania and South Carolina?
52
Leveraging TA Data: Challenges and Limitations
o Balancing the value of TA data with the burden of data collection
o Maintaining data quality throughout the data lifecycle
o Distinguishing between research questions that can be answered
with administrative data and those that may require supplementary
data from scientifically drawn samples or other sources
o Keeping up with the data needed for new policy directions and new
initiatives
53
Leveraging TA Data: Key Data Elements
Early Learning Program and Practitioner Demographics
Provider Name & ID
Program Type
Practitioner Name & ID
Capacity
Subsidy Participation
Accreditation
Curriculum Used
Screenings & Assessment
Quality Initiatives
QRIS Level
TA Provider Demographics and
Qualifications
Agency Name
Agency ID
TA Consultant Name
TA Consultant ID
TA Consultant ID
TA Consultant Education
TA Consultant Credentials
Registry Status
TA Activities and Quality Improvement Plan Goals
TA Activity Number
TA Domain
Activity Open/Close Date
TA Agency Assigned
TA Consultant Assigned
QI Plan
QI Plan Status
QI Plan Goals
TA Methods and Strategies
TA Contact Date
TA Contact Type
TA Intervention Type
TA Intervention Duration
Outcome of TA
Indirect TA Duration
Classroom(s)
Age Group(s)
Note: Data elements may be collected in QRIS data system or linked to from other data systems.
54
Resources to Keep in Mind
• Existing Resources:
– INQUIRE QRIS Data Tool Kit
– INQUIRE QRIS Evaluation Tool Kit
• Coming Soon:
– QRIS Compendium Update
– ELC TA’s Brief on Leveraging TA Data To Support QRIS
55
Group Discussion and Sharing
56
Let’s Stay in Touch….
o Feel free to reach out to us after
today:
Michelle Thomas
Kenley Branscome
o Go to www.elcta.org to learn more
about ELC TA
“Data that is loved tends to
survive.”
Kurt Bollacker, Data Scientist –
Freebase/Infochimp
