Quarterly Progress Report on ROA Implementation - …centerforchildwelfare.fmhi.usf.edu/qa/QA_Docs/June 2016 Quarterly...Quarterly Progress Report on ROA Implementation - June 2016

FLORIDA DEPARTMENT OF CHILDREN AND FAMILIES QUARTERLY PROGRESS REPORT ON ROA IMPLEMENTATION - JUNE 2016

DELIVERABLE 2B

Date: 08/23/2016 Version: 3.00

Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page i

TABLE OF CONTENTS

1.1 ROA Program Plan Implementation Timeline and Initiatives ........................................ 7 1.2 ROA Program Implementation Team ......................................................................... 10 1.3 ROA Program Governance ........................................................................................ 14

1.3.1 Roles and Responsibilities .................................................................................. 16

2.1 Status of ROA Program Implementation (SFY 2015-16) ............................................ 17 2.2 ROA Program Implementation Risks .......................................................................... 30 2.3 ROA Program Implementation Issues ........................................................................ 30 2.4 Looking Forward – Planned SFY 2016-17 ROA Implementation Activities ................ 31

TABLE OF EXHIBITS

Exhibit 1: ROA Program Initiatives ............................................................................................ 10 Exhibit 2: DCF, OCW and PQMU Organization ........................................................................ 12 Exhibit 3: ROA Program Implementation Governance ............................................................. 15 Exhibit 4: ROA Program Governance Roles and Responsibilities ............................................ 16 Exhibit 5: Status of ROA Program Implementation Initiatives ................................................... 28 Exhibit 6: ROA Program Implementation Risks ........................................................................ 30 Exhibit 7: ROA Program Implementation Issues ....................................................................... 31 Exhibit 8: ROA Program Implementation Activities (SFY 2016-17) .......................................... 33

Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page ii

Revision History

DATE AUTHOR VERSION CHANGE REFERENCE

06/22/2016 North Highland .001 Prepared initial draft

06/30/2016 North Highland 1.00 Updated report to prepare for submission to DCF for review

07/27/2016 North Highland 1.01 Updated report based on DCF feedback and submitted for informal review

07/28/2016 North Highland 2.00 Updated report based on DCF feedback and prepared for submission to DCF for review

08/05/2016 North Highland 2.01 Updated report based on DCF review feedback

08/15/2016 North Highland 2.02 Updated report based on DCF feedback and prepared draft for DCF review

08/23/2016 North Highland 3.00 Updated report based on DCF feedback and prepared for DCF review

Quality Review

NAME ROLE DATE Tina Worley NH Client Lead / Project Manager 06/28/2016 Tina Worley NH Client Lead / Project Manager 06/30/2016 Tina Worley NH Client Lead / Project Manager 08/01/2016

Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 1

EXECUTIVE SUMMARY

Creating positive change for Florida’s children and families is only possible when all the organizations involved with child welfare recognize their individual and collective roles in enhancing the safety, permanency, and well-being of those served. In Florida, the key child welfare stakeholders and partners include the Department of Children and Families (DCF, Department), Community-Based Care (CBC) lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. Collectively, these stakeholders represent Florida’s Child Welfare Community. The unique partnerships within the Child Welfare Community create opportunities for long-term improvement by bringing together many perspectives and experiences with a singular focus on improving the lives and safety of each child in Florida.

The Florida Legislature passed Florida Statute (F.S.) 409.997 in 2014. The actions of the Legislature allowed the creation of a platform for extensive advancement of the child welfare system through establishment of the Results-Oriented Accountability Program (ROA Program, Program). The goal of the ROA Program is to develop an integrated, research-informed framework designed to inform communities, the child welfare system, and legislators on essential elements of child protection (Chapters 20, 39, and 409, F.S.).

This document is the Progress Report for the implementation of the ROA Program for State Fiscal Year (SFY) 2015-16, reflecting status as of June 2016. The Progress Report references F.S. 409.997 and the implementation initiatives and roadmap outlined in the ROA Program Plan created by the Department in February 2015, to assess the progress of the Program implementation. The report also discusses circumstances resulting in deviation from the implementation roadmap and how guidance from Department Secretary Mike Carrol and the ROA Governance Committee establishes the strategic direction and focus for the ROA Program implementation efforts on an annual basis.

The status reported in this document is based primarily on discussions with Office of Child Welfare (OCW) management and staff, participation in ROA Program meetings and workgroups, and a review of relevant documentation and includes the following sections:

Program Background: This section includes an overview of the ROA Program background and purpose, implementation team, and governance.

Program Implementation Status: This section includes information detailing the status of the Program implementation to date, including implementation activities, risks, and issues. A summary of the ROA Program objectives and major Program implementation activities planned for the upcoming state fiscal year is also provided.

Recommendations: This section presents recommendations for the Department’s consideration related to the implementation activities planned for the upcoming state fiscal year.


Key Accomplishments

Over the past year the Department has made a number of significant accomplishments with respect to the ROA Program implementation, as highlighted below. Additional detail on these accomplishments and other implementation activities is provided in Section 2 of this report.

A Child Welfare Performance and Quality Management Unit (PQMU) was created within the OCW and charged with the responsibility for planning and managing the ROA Program implementation. A ROA Project Manager has been identified to serve as a dedicated resource for the ROA Program implementation and will be moving into this role in early SFY 2016-17.

The PQMU expended over $100,000 in ROA-related hardware and software during SFY 2015-16. As part of this investment, the PQMU secured licenses for and implemented Tableau data visualization software and SPSS predictive analytics software. Basic training on Tableau and SPSS was conducted for PQMU staff, and advanced training in both applications is planned for SFY 2016-17. Additionally, three PQMU Performance Management Unit (PMU) staff obtained their Six Sigma Yellow Belt and Green Belt certifications in SFY 2015-16.

Twenty-nine interactive dashboards for priority measures were developed in Tableau for child welfare stakeholders. A subset of those dashboards for CBC Scorecard measures was released in May 2016 to Program stakeholders and the general public via the DCF website and Tableau Public. The PQMU also developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

The ROA Program Technical Advisory Panel, which was established to serve in an advisory capacity during the development of the ROA Program Plan, was remobilized to continue to perform in this role throughout the Program implementation. A Technical Advisory Panel Charter was created describing the role and responsibilities of the panel.

The ROA Program Governance Committee was established, and members were appointed in March 2016. A Governance Committee Charter was developed describing the role and responsibilities of the committee. The initial meeting of the committee was held on April 7, 2016. During this meeting the Governance Committee Charter, ROA Program implementation status, and plan for the upcoming SFY were presented and discussed. An introduction to Tableau and demonstration of the dashboards were presented to the committee. The feedback received from the ROA Governance Committee on the status of the Program implementation was very favorable, and the next meeting is planned for July 2016.

Subject matter experts from ROA stakeholder groups statewide collaborated to develop detailed process diagrams and narratives for the five activities that drive the ROA cycle of accountability: Outcomes Monitoring, Data Analysis, Research Review, Evaluation, and Quality Improvement (QI). Process documentation for the identification


and validation of ROA outcome measures was also created. The process documentation describes the activities required to complete the ROA Program processes, the resources responsible for executing those activities, as well as critical hand-off points between the processes.

Additional external resources were secured through competitive procurement to support Program implementation efforts in the areas of advanced data analytics and data strategy and governance. Under the PQMU’s direction, a Florida Safe Families Network (FSFN) Business User Data Dictionary and Data Governance and Strategy were developed. The FSFN Business User Data Dictionary identifies, documents, and analyzes FSFN data critical to supporting ROA measures and reporting and advanced analytics initiatives. The Data Governance and Strategy document provides an overarching data strategy linked to business strategy, data governance standards, data standardization and stewardship responsibilities, standard procedures for data acquisition, and master data management (MDM) procedures and constructs. Additionally, an advanced analytics project was conducted to study re-maltreatment and re-reporting to better understand risk factors influencing outcomes for children and to identify needs for policy and practice changes. A key finding from this analysis was breaking the cycle of maltreatment requires policies and practices designed to counter perpetrator recidivism over a period of five to ten years or more.

As a result of monitoring out-of-home care (OOHC) data and identifying an increase in the number of removals and placement of children in OOHC, the Department partnered with Casey Family Programs and The Ounce of Prevention Fund of Florida to conduct an analysis of this trend, utilizing data analysis techniques such as multi-variate analysis. The OOHC Analysis Project, which is representative of the outcomes monitoring and data analysis stages of the ROA Program’s cycle of accountability, was completed in February 2016 with a final report providing countermeasures and recommendations.

The FSFN System Adoption Project was initiated which involves a collaboration between the Department and each CBC lead agency to identify CBC information and technology requirements and develop an individualized System Adoption Plan that achieves full adoption of FSFN while supporting CBC business processes. This project will help to identify training and policy needs and ensure complete, accurate, and consistent data resides in FSFN for the ROA Program and other DCF initiatives.

The Department engaged the Florida Institute for Child Welfare (FICW) to conduct a feasibility study to inform the design of a planned study to assess the inter-rater reliability and validity of new family assessment tools. The feasibility study is scheduled to be completed in December 2016.

Key Challenges

Key challenges the Department has faced in the initial year of the ROA Program implementation include:

Annual funding for the ROA Program implementation for SFY 2015-16 was very limited in comparison to the estimated funding requirements described in the ROA Program


Plan. This constraint impacted the OCW’s ability to complete all activities outlined for SFY 2015-16 in the ROA Program Plan while maintaining business as usual. As a result, OCW resources were allocated between the ROA Program implementation activities and day-to-day operational responsibilities and demands.

The ROA Program Plan recommended a much larger number of dedicated resources to perform implementation activities, including DCF, FICW, and contractor resources. The shift in available resources likely will impact not only the duration of the implementation plan, but also the overall scope and approach. As a result, the implementation roadmap and the actual implementation efforts are not aligned, which makes planning for the multi-year implementation difficult.

Recommendations for SFY 2016-17

For SFY 2016-17, under the guidance and leadership of the DCF Secretary and ROA Governance Committee, the OCW has established six objectives for the ROA Program which serve to focus implementation activities. The PQMU has identified the tasks required to achieve these objectives which form the foundation of the ROA Program SFY 2016-17 implementation schedule.

Based on the PQMU’s planned implementation activities for SFY 2016-17, North Highland has developed recommendations for the Department’s consideration, which align with three of the ROA Program objectives and are centered on continuing to build the infrastructure (people, processes, and technology) necessary for the Program’s success. These recommendations are summarized below and presented in more detail in Section 3 of this report.

Objective 3: Operationalize Year One foundational components of OCW Data Governance and Master Data Management Strategy

Recommendation: North Highland recommends the Department build executive management and program area leadership buy-in and support for formal data governance through establishing a:

Well-executed communications strategy

Strong business case

Comprehensive framework and charter documents

Once the objectives of data governance are agreed upon and approved, the Department should designate data governance roles and responsibilities, document policies and procedures under which data governance will operate, define the supporting IT and business processes, and evaluate the necessary tools.

Recommendation: North Highland recommends the OCW develop a person-centric view of data, with the child data as a sub-set. This effort builds on language in the ROA Program Plan specific to creating a data viewpoint from the perspective of a child in care and should be coordinated with the Department’s Client Data Link project


already underway. Planning, processes, procedures, and tool evaluation specific to MDM should leverage the foundation built for data governance.

Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views

Recommendation: North Highland recommends the Department conduct an additional inventory and study of services and practice data maintained at each CBC, as noted in the ROA Program Plan. Based on the results, a plan and roadmap can be developed to integrate and assimilate this information to enrich FSFN.

Recommendation: North Highland recommends the Department leverage an algorithm already being used for an existing Master Person Index (MPI) effort and create a process to apply the algorithm to FSFN at the source to begin consolidating the multiple person identities correctly. In the longer term, the Department may consider a more sophisticated commercial solution for potential improvement.

Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the State

Recommendation: In addition to the ROA communication strategy to be executed in SFY 2016-17, North Highland recommends the PQMU create an Organizational Change Management (OCM) Plan for the ROA Program implementation and begin to establish a change network across the Child Welfare Community stakeholders impacted by the initial phase of the Program deployment.

Recommendation: North Highland recommends the OCW designate a point person for all OCW advanced analytics projects and initiatives. Furthermore, it is recommended the OCW be the business owner of child welfare analytics for the State to facilitate opportunities for collaboration while ensuring consistency in approach and execution.


SECTION 1 PROGRAM BACKGROUND

Improving outcomes for Florida’s vulnerable children and families is dependent on the concept the entire community is responsible for child safety, permanency, and well-being. Florida’s child welfare system consists of key stakeholders and partners that include the Department, CBC lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. These stakeholders, which comprise the state’s Child Welfare Community, each play an important role in and have a shared accountability for the safety, permanency, and well-being of Florida’s children.

With the support of state policymakers and legislators, DCF and its partners are implementing the ROA Program to deliver greater innovation and reform. The Program will address existing and future challenges and drive positive change in Florida’s child welfare system. To do this, the Program will establish a robust and continuous cycle of monitoring, data analysis, research, evaluation, and quality improvement to further advance the system’s efforts to improve outcomes, identify new programs and services impacting those outcomes, and enable research and evidence-informed practice, policy, and decision-making.

In 2014, the Florida Legislature passed F.S. 409.997, directing the Department to contract with a qualified consultant or organization to develop a plan for the implementation of the ROA Program. As stated in F.S. 409.997, the purpose of the ROA Program is to monitor and measure:

the use of resources

the quality and amount of services provided

child and family outcomes

The Program conceived by the Legislature and whose design and implementation is laid out in detail in the ROA Program Plan submitted by the Department in February 2015 is focused on improving the outcomes of the children and families served by the OCW through evidence-informed interventions, practices, and policies. The Program is based on the “cycle of accountability” framework presented by Mark Testa and John Poertner in their book, Fostering Accountability, Using Evidence to Guide and Improve Child Welfare Policy. The cycle of accountability relies on operationalizing five key activities to further advance the efforts of state child welfare systems to evaluate performance on outcomes, identify new or promising interventions and strategies, review the validity of programs, and conduct continuous quality improvement to ensure the system as a whole is learning and moving toward the accomplishment of goals which positively impact children and their families.


The five activities included in the cycle of accountability are:

1. Outcomes Monitoring, activities to define, validate, implement, and monitor outcome measures throughout the Child Welfare Community

2. Data Analysis, approaches and procedures to critically study performance results to determine if variances discovered are in fact issues which should be explored further

3. Research Review, a series of activities to gather and validate evidence to support the development and implementation of interventions to address areas for improvement

4. Evaluation, activities to assess promising interventions for children and families to determine if deployment to a larger population is warranted

5. Quality Improvement, a series of actions to implement interventions across new domains, or to challenge, change, and test new assumptions about the underlying goals supporting the child welfare practice model

The ROA creates accountability and transparency by incorporating processes and tools for the dissemination of performance results to the Child Welfare Community and general public. The goal of Program accountability and transparency is to present accurate and timely performance information, along with an analysis of factors influencing trends, enabling stakeholders to view a true picture of the system and potential needs for improvement.

1.1 ROA PROGRAM PLAN IMPLEMENTATION TIMELINE AND INITIATIVES

The ROA Program Plan provides a five-year roadmap of initiatives for the implementation and ongoing operation of the Program. Initial implementation activities occur over two years to establish the ROA Program, including an initial set of measures and the associated reporting and monitoring for key stakeholder groups. After the initial implementation, there are three years of ongoing implementation activities to continue to incorporate new data, measures, reporting, and monitoring for additional stakeholder groups. In total, the Program Plan identified 15 implementation initiatives and their associated activities, which are summarized below in Exhibit 1: ROA Program Initiatives.

NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES PLANNED DURATION

1 Results-Oriented Accountability Implementation Project Team

This initiative creates the management structure and processes required to manage and oversee Program implementation.

Establish ROA Implementation Project Team

Create program management processes

Create governance processes and procedures

Develop implementation master schedule

Engage project teams and execute prioritized initiatives

Manage and respond to risks affecting program implementation

Initial Implementation: 07/2015-12/2015 Ongoing Implementation: 01/2016-06/2020



2 Measure Development and Validation

This initiative develops reliable and valid outcome measures, sets outcome measure baselines and targets, and identifies outcome measure drivers.

Identify ROA measures Confirm measure validity and identify

unintended consequences through research, data analysis, and logic models

Develop calculation algorithms Set performance baselines and

targets Pilot measures and make necessary

adjustments based on results Update relevant documentation


3 Master Data Management

This initiative develops and implements effective governance processes and procedures to enable data collaboration, quality, and standardization while safe guarding confidentiality.

Define MDM procedures and documentation

Establish data sharing agreements with stakeholders

Coordinate updates to data sharing agreements as needed

Implement MDM across the Child Welfare Community


4 Data Lab and Tools

This initiative establishes the ROA analytics hardware and software environment.

Install and configure analytics software

Test software installation and configuration

Conduct knowledge transfer

Implementation: 01/2016-06/2016

5 Data System Updates for Initial Measurement Gaps

This initiative resolves gaps between the selected ROA outcome measures and FSFN data required to calculate the measures.

Resolve gaps between initial set of ROA outcome measures and the availability of FSFN data required to calculate the measures


6 Accountability Reports

This initiative supports the development of ROA reports for Program stakeholders, creating transparency and accountability by facilitating timely and meaningful access to Program results.

Develop report requirements Design reports for stakeholder review

and approval Develop report extracts, data

transformation, and presentation mechanisms

Test reports Develop and conduct end user

training Deploy reports Conduct ongoing annual updates

Implementation: 10/2015-03/2016 Ongoing Implementation: 07/2016-06/2020

7 Quality Assurance (QA) / Compliance Resource Analysis

This initiative assesses and provides recommendations on the current utilization of internal and external QA/QI resources and the QI resources required to support the ROA Program.

Identify internal and external units conducting QA/QI and contract compliance activities

Assess capacity to conduct case reviews and other QI activities

Assess external capacity/cost to conduct contract compliance reviews

Develop QI staffing recommendations




8 Quality Improvement Organization

This initiative includes assessing the QI needs of the ROA Program, securing the staff to meet the requirements, and developing and delivering a QI focused training for managers, staff, and other stakeholders to instill and reinforce a “learning organization” mindset.

Assess QI needs associated with the Program

Develop/Modify a QI Program Plan and procedures to include ROA functions

Develop a QI Staffing Plan Obtain QI staff to support Program

operations, research, evaluation, and QI functions

Develop and deliver training on the QI/Learning Organization for managers, line staff, and stakeholders


9 Results-Oriented Accountability Reporting System

This initiative establishes the operational reporting system to monitor and improve results and accountability across Child Welfare Community.

Incorporate stakeholder data and screens into the ROA Reporting System

Complete typical development life cycle


10 Child Welfare Community Data

This initiative develops non-FSFN data interfaces to incorporate quality data from the Child Welfare Community into the ROA reporting system and addresses data quality issues in FSFN.

Complete FSFN System Adoption initiative

Map data sources to target fields Provide data details for data sharing

agreements Develop interfaces and data

extraction and transformation tools Configure data cleansing procedures

and tools


11 Institutional Review Policy (IRB) Updates

This initiative provides an IRB policy and process for the ROA Program which meets federal and other requirements.

Facilitate the review of IRB processes employed by FICW and affiliated institutions

Determine the need for updates to FICW or DCF IRB policy to ensure all federal and other requirements are met

Develop Program-specific procedures for engaging the IRB process


12 Research Standards

This initiative develops ROA Program research standards, a “Levels of Evidence” construct, methodologies, protocols, and tools to ensure research reviews are conducted consistently and produce reliable results.

Research and develop levels of evidence construct specific to Florida and child welfare

Research, test, and implement a meta-analysis protocol for research reviews requiring meta-analysis of large numbers of target studies

Develop a process for estimating research review budgets




13 Pilot Study Standards

This initiative develops pilot procedures to determine all of the critical elements, approvals, and considerations to be addressed prior to implementing a pilot.

Develop pilot study procedures Assess and update CBC contracts to

allow for pilots Create standards for project

management of pilot studies Create protocols for staffing pilots


14 Research and Evidence-Informed Practice Training Development

This initiative includes conducting a training needs assessment and developing research and evidence-informed practice training to drive a culture shift within the system to that of a “learning organization” in which data is used in decision-making.

Conduct a training needs assessment

Design a training strategy Develop training materials Implement training materials through

FICW Conduct training evaluation to

determine effectiveness Integrate course into new hire

curriculum


15 Results Oriented Accountability – FICW Support

This initiative represents FICW activities required to support the ROA Program, including providing a ROA center of learning, acting as ROA advisors, leading research and evaluation pilots, and optimizing the ROA implementation.

Coordinate database administration Advise ROA Program Implementation

Team Perform data analysis Perform data extracts,

transformation, and loads Perform program research Develop ROA policy, strategic

design, and thought leadership Support ROA implementation and

training


Exhibit 1: ROA Program Initiatives

The ROA Program Plan estimated the cost for the first year of the Program implementation (SFY 2015-16) between $8.5 and $10.6 million and recommended 46 to 55 full time equivalents (FTEs) working across the Department, FICW, contracted providers, and external vendors. However, the Department has received $500,000 in annual funding to implement the Program, an amount which represents only a small fraction of the original cost estimates.

Each year, with guidance from the DCF Secretary and ROA Governance Committee, the OCW establishes a set of objectives for the ROA Program which serve to focus the PQMU’s implementation efforts based on the original ROA Program initiatives. The PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives.

1.2 ROA PROGRAM IMPLEMENTATION TEAM

The Department’s OCW is primarily responsible for the implementation and operation of the ROA Program. The OCW is committed to the safety, well-being, and timely permanency of


Florida’s children and families and is responsible for a wide range of services, including assistance to help families stay intact or be reunified, out-of-home care, adoption, and independent living skills for foster care youth transitioning to adulthood. To do this, the OCW works with six DCF Region Offices, 17 CBC lead agencies, and six Sheriff’s Offices to execute policy and practice for child protective investigations and case management services.

The ROA Program Plan recommended a reorganization of the OCW to create a structure to better support the implementation and operation of the Program. In 2015, the OCW established the PQMU based on the Program Plan’s recommended design to oversee the implementation and ongoing operation of the Program. The PQMU is led by the PQMU Director who reports directly to the Assistant Secretary for Child Welfare. The PQMU includes three sub-units:

Data and Reporting Unit (DRU): This unit supports the Department’s internal and external reporting needs, including developing and producing federal reports, management reports, operations reports, ad hoc reports, and performance dashboards, as well as assisting the Department in responding to media, legislature, and general public requests for information. The DRU also provides data sets and SQL queries to the Performance Management Unit to be used for data analysis.

Performance Management Unit: This unit monitors key performance indicators, produces performance summaries, and disseminates information through publications and presentations. These products contain initial stages of data analysis in support of Root Cause Analysis, a critical part of the ROA Program’s cycle of accountability that ties data reporting, monitoring, and analysis together with continuous quality improvement. The PMU also supports data-driven management through the provision of ad hoc data analysis services and support of research and evaluation projects.

Quality Improvement / Quality Assurance (QI/QA) Unit: This unit performs ongoing case reviews of children served by DCF, including subjects of child protective investigations or children who are/were recently in out-of-home or in-home care. The unit also interviews parties involved in the child’s case to understand how day-to-day practice influences child and family functioning and outcomes. Quality management (QM) data is regularly reported to DCF leadership, Region Offices, and CBC lead agencies to ensure operations are engaged in activities to improve practice.

Exhibit 2: DCF, OCW and PQMU Organization depicts the placement of the PQMU in the Department’s organizational structure and highlights the detailed organization of the PQMU.

Quarterly Progress Report on ROA Implementation - June 2016

Florida Department of Children and Families - Office of Child Welfare Page 12

Exhibit 2: DCF, OCW and PQMU Organization

The People

Governor

Secretary

Children’s Legal Services

Director

Chief of Staff

Planning Director

Communications Director

Inspector General

Assistant Secretary for

Operations

Regional Managing Director

Northeast Region

Regional Managing Director Northwest Region


Administration

Child Welfare Operations

Director

Child Welfare Planning & Strategic Projects Director

General Counsel


Southern Region

Regional Managing Director Suncoast Region

Adult Protective Services Director

Regional Managing DirectorCentral Region


Southeast Region

Refugee Services

Director

Human Resources Director

Contract Services Director

Chief Information Officer

General Services Director

Chief Financial Officer

Child Welfare Practice

Director

Legislative Affairs Director


Economic Self-Sufficiency

Economic Self-Sufficiency

Director

Office of Public Benefits

Integrity Director

Deputy Secretary

Assistant Secretary for Sub. Abuse & MH

Sub. Abuse & MH

Director

Homelessness Director

State MH Treatment Facilities Director

Florida State Hospital

Administrator

NE FL St. Hosp. & N FL Eval.

Trmt. Ctr. Administrator

Sexually Violent Predator ProgramDirector

Child Welfare Performance & Quality Management Director

Assistant Secretary for Child

Welfare

OCW

Child Welfare Performance and Quality Management Unit

Child Welfare Performance and Quality Management Unit

Director

Data and ReportingManager

Finance / Independent Living Reporting

Ongoing Services / Special Projects

Federal Reporting

Hotline Investigations / Reporting

Performance ManagementManager

Title IV-E Stipend Specialist

Hotline QA

QA Specialist / System Admin

Federal Liaison

Hotline /DV /IL /HT

Investigations / Data Analytics / FSFN Adoption

Investigations / Child Fatalities

Case Management / CCL

Special Analyses / All Areas

QI/QA Manager


Due to limited funding, the OCW implemented the ROA Program Plan’s recommended functional organizational changes with the exception of the ROA Program Implementation and enhanced training functions described below.

The temporary ROA Program Implementation function was to include approximately seven program management staff responsible for and dedicated to the development of the ROA Program and its related processes, including the management of initiatives within the Department and across key stakeholders (See Initiative 1). Because the ROA Program Implementation function was not included and staffed as part of the PQMU’s initial organizational structure, the workload originally planned for this unit was reallocated across the other three areas of the PQMU. During the first year of implementation, the PQMU Director and the three PQMU managers formed the core ROA Program Implementation Team.

The enhanced training function proposed in the ROA Program Plan was to be responsible for developing longer-term system-wide training strategies, coordinating with partners, such as FICW, to develop and deliver Program training across the Child Welfare Community, identifying training needs resulting from ROA Program activities, and identifying new and effective ways to enhance the skills of all ROA Program stakeholders (see Initiative 14).

The ROA Program is intended to be a Department-University partnership; therefore, FICW plays a key role in the Program implementation, particularly in the areas of research, advanced data analysis, and predictive analytics. FICW was established in 2014 through F.S. 1004.615 as a consortium of public and private universities offering degrees in social work with the purpose of advancing the well-being of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development. FICW’s Director serves as a member of the ROA Governance Committee, ROA Technical Advisory Panel, and ROA Program Implementation Team. While FICW has been actively involved in many Program implementation activities to date, similar to the Department, resource constraints in SFY 2015-16 limited their ability to execute the implementation activities originally intended for FICW in the ROA Program Plan.

In addition to the OCW and FICW, the PQMU routinely calls on subject matter experts from key stakeholder groups, including the DCF Region Offices, CBC lead agencies, community service providers, and advocates, to perform activities in support of the Program implementation. For example, many of these stakeholders participated on workgroups to lead the design and documentation of the ROA Program processes.

In SFY 2016-17, the PQMU Director will appoint a ROA Project Manager as a dedicated resource to drive implementation efforts with both internal and external stakeholders and perform the activities described in Initiative 1. The ROA Project Manager has been identified and will transition into this new role in early SFY 2016-17. In addition to the ROA Project Manager, a dedicated FICW resource with advanced research and data analysis skills will be collocated with the PQMU in SFY 2016-17 to assist with a number of the ROA implementation initiatives.


1.3 ROA PROGRAM GOVERNANCE

The ROA Program governance structure provides a framework for how decisions will be made about the Program’s implementation. Well-defined governance is critical to the Program’s success in that it establishes the roles and responsibilities of all parties involved in the implementation and determines, in advance, clear decision-making processes and authority.

The ROA Program governance structure includes the following components:

ROA Program Implementation Team, which performs the activities necessary to implement the ROA Program and is currently comprised of the PQMU managers and staff and representatives from the OCW, DCF Region Offices, CBC lead agencies, community providers, university schools of social work (FICW and University of South Florida (USF)), and child welfare advocates and subject matter experts

PQMU Director, who oversees, coordinates, and manages the day-to-day activities of the ROA Program Implementation Team with the support of a ROA Program Implementation function

ROA Program Implementation function, which will be led by the ROA Project Manager to assist the PQMU Director with planning, coordinating, and managing ROA Program implementation activities

FICW, which was established by F.S. 1004.615 for the purpose advancing the well-being of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development, and as described in the ROA Program Plan, plays an integral role in advising the Department on the planning and implementation of the ROA Program

ROA Technical Advisory Panel, which was established by F.S. 409.997 to advise the Department during the planning and implementation of the ROA Program, and consistent with F.S. 409.997, includes representatives from FICW, the Senate, the House of Representatives, CBC lead agencies, family representatives, child welfare advocates, CBC providers, other contracted providers, community alliances, and family representatives, as well as the Department and Children’s Legal Services (CLS)

ROA Governance Committee, which serves as the decision-making body for the implementation and operation of the ROA Program and, as recommended in the ROA Program Plan, is currently comprised of representatives from the Department, Sheriff’s Offices, CBC lead agencies, FICW, child welfare advocates, CLS, and the court system

The ROA Program governance structure is depicted graphically in Exhibit 3: ROA Program Implementation Governance.


Exhibit 3: ROA Program Implementation Governance


1.3.1 ROLES AND RESPONSIBILITIES

The roles and responsibilities of the key ROA Program governance structure are described in Exhibit 4: ROA Program Governance Roles and Responsibilities.

ROLE RESPONSIBILITIES

ROA Governance Committee

Represents the Child Welfare Community stakeholders Provides leadership and serves as the decision-making body for the implementation of

the Program Sets Program strategies and priorities Provides authority for use of resources to implement defined strategies

PQMU Director

Oversees the Program implementation, including but not limited to: staffing, assignments, stakeholder communications, schedule management, status reporting, risk and issue management and mitigation/resolution

Coordinates Program implementation activities across Program stakeholder groups Provides overall leadership and guidance in support of the Program implementation Secures funding and resources for the Program implementation

ROA Program Implementation (Project Management)

To be led by a dedicated ROA Project Manager beginning in SFY 2016-17 Supports ROA Program Sponsor/Director Establishes and maintains the Program implementation schedule Implements and manages project management processes Assists in coordination and support of implementation activities

ROA Implementation Team

Performs activities to implement the ROA Program according to the Program implementation schedule

Reports progress to the ROA Program Director

FICW Advises the ROA Program Director and ROA Implementation Team on implementation

activities and resources Recommends improvements to proposed Program processes and procedures to ensure

the implemented Program is based on best practice and meets goals and objectives

ROA Technical Advisory Panel

Represents the Child Welfare Community stakeholders Provides technical assistance and guidance throughout the ROA Program

implementation Assists in the establishment and assessment of ROA Program implementation

benchmarks and milestones Monitors progress toward the achievement of ROA Program implementation

benchmarks and milestones

Exhibit 4: ROA Program Governance Roles and Responsibilities

In November 2015, the PQMU reconvened the ROA Technical Advisory Panel to advise on Program implementation activities. In March 2016, the PQMU established the members of the ROA Governance Committee and held an initial meeting date on April 7, 2016.


SECTION 2 PROGRAM IMPLEMENTATION STATUS

Each year, the DCF Secretary, ROA Governance Committee, and the OCW establish a set of objectives for the ROA Program which serves to guide the PQMU’s implementation efforts based on the original ROA Program initiatives and available resources. As detailed below in Section 2.1, the PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives. As described in Section 1.1, many of the implementation initiatives require ongoing effort over the course of the multi-year Program implementation. In SFY 2015-16 activities focused on building the infrastructure necessary to support further implementation tasks as well as the successful operation of the ROA Program. In the upcoming year, the Department’s work to establish these foundational elements will continue. Section 2.4 presents an overview of the SFY 2016-17 ROA Program objectives and planned implementation activities.

2.1 STATUS OF ROA PROGRAM IMPLEMENTATION (SFY 2015-16)

The current status of the ROA implementation initiatives is presented in Exhibit 5: Status of ROA Program Implementation Initiatives.

The information provided includes:

Status:

› Not Yet Started: Work on the activity has not started

› In Progress: Work on the activity has started but is not complete

› Complete: Work on the activity is complete

› Ongoing: Work on the activity has started and will continue for the duration of the implementation

Actual Completion: For completed activities, the date activity was completed

Estimated Completion: For activities not yet started, in progress, or ongoing, the estimated completion date

Comments: Additional information regarding the status of the implementation activity


INITIATIVE/ACTIVITY STATUS ACTUAL

COMPLETION ESTIMATED

COMPLETION COMMENTS

Initiative 1: ROA Program Implementation Project Team: In SFY 2015-16, these activities were supported by North Highland under the ROA Program and Data Analytics contract.

Establish Results-Oriented Accountability Implementation Project Team

Complete July 2016 n/a

In SFY 2015-16, the OCW was reorganized to create the PQMU, referencing the organization structure described in the ROA Program Plan. However, the PQMU structure does not include the recommended ROA Implementation Project Team with dedicated project management resources.

In SFY 2016-17, a dedicated ROA Project Manager position will be established within the PQMU to lead the planning and execution of ROA Program implementation activities.

Create Program Management Processes Complete June 2016 n/a

In SFY 2015-16, key project management plans/processes/tools were developed for schedule, communications, and risk management.

The ROA Project Manager will be responsible for executing and maintaining the project management plans/processes/tools for the duration of the ROA Program implementation.

Create Program Governance Processes / Procedures

Complete April 2016 n/a

In SFY 2015-16, the PQMU established ROA Program governance processes and procedures. The ROA Technical Advisory Panel was reconvened in November 2015 and met in November 2015, February 2016, March 2016, and June 2016. The Technical Advisory Panel will continue to meet at least quarterly for the duration of the Program implementation. The ROA Governance Committee membership has been established, and the first meeting was held on April 7, 2016. The ROA Governance Committee will continue to meet quarterly over the life of the Program. Pursuant to Section 409.997(2)(g), F.S., the Department must submit an annual ROA performance report to the Governor and Florida Legislature. The report summarizes performance across key safety, permanency, and well-being indicators. The OCW has submitted the annual report for the past two years, and the report covering the period of SFY 2015-16 is currently being prepared for submission on or before October 1, 2016.

The ROA Project Manager will be responsible for executing and maintaining ROA Program governance processes and procedures for the duration of the ROA Program implementation.




COMPLETION COMMENTS

Develop Implementation Master Schedule Ongoing n/a SFY 2019-20

This is an ongoing annual task completed by the ROA Project Manager for the duration of the ROA Program implementation.

The Master Implementation Schedule for SFY 2015-16 was completed in April 2016, and a draft schedule has been prepared for SFY 2016-17 based on the ROA Program objectives. As additional task-level detail becomes known for SFY 2016-17, the ROA Project Manager will update the schedule accordingly.

Engage Project Teams and Execute Prioritized Initiatives

Ongoing n/a SFY 2019-20

In SFY 2015-16, five workgroups were established to design and document ROA Program processes to prepare for the implementation of the processes in SFY 2016-17. The PQMU engaged both internal departmental stakeholders as well as external Child Welfare Community stakeholders and advisors to serve on the workgroups. The workgroups conducted regular meetings from late February through May. The final draft ROA process documentation was presented to the ROA Technical Advisory Panel on June 28, 2016.

Manage and Respond to Risks Affecting Program Implementation


In SFY 2015-16, ROA Program implementation risks were managed by the North Highland Project Manager. In SFY 2016-17, ROA Program implementation risks will be managed by the PQMU’s ROA Project Manager.

Initiative 2: Measure Development and Validation




COMPLETION COMMENTS

Identify ROA Measures Ongoing n/a SFY 2019-20

In SFY 2014-15, a proposed set of initial measures for the ROA Program was provided in the ROA Program Plan. These measures were developed by a workgroup of Child Welfare Community stakeholders and primarily apply to the Department, CBC lead agencies and providers, and Sheriff’s Offices. The measures include several of the existing federal outcome measures, CBC Scorecard measures, CPI Scorecard measures, and Priority of Effort (POE) measures.

In SFY 2015-16, DCF and other key ROA Program stakeholders developed and documented a process to identify and validate ROA Program outcome measures. In SFY 2016-17, an Outcome Measures Workgroup comprised of representatives from the Child Welfare Community stakeholder groups will be established to oversee this process and the development of a validation methodology. The Outcome Measures Workgroup will review the proposed set of initial measures included in the ROA Program Plan and make updates as necessary to finalize the initial set of measures.

Once the initial set of measures is finalized, validated, and deployed, additional measures will be identified and validated for other Child Welfare Community stakeholder groups, such as the judicial system, state partner agencies, etc., throughout the Program implementation.

Confirm Measure Validity and Identify and Address Unintended Consequences Through Research, Data Analysis, and Logic Models

Not Yet Started n/a SFY 2019-20

Develop Calculation Algorithms


Set Performance Baselines and Targets


Pilot Measures and Make Necessary Adjustments Based on Results





COMPLETION COMMENTS

Update Relevant Documentation


Initiative 3: Master Data Management: In SFY 2015-16, these activities were completed by North Highland under the ROA Program and Data Analytics contract.

Define Master Data Management Procedures and Documentation

In Progress n/a SFY 2016-17

In SFY 2015-16, North Highland developed a Data Governance and Strategy report for the OCW which included a current state assessment and the development of a future state, recommended model, and an implementation roadmap. The Data Governance and Strategy report was approved on July 21, 2016.

Establish Data Sharing Agreements with Stakeholder Communities

Ongoing n/a SFY 2019-20 In SFY 2015-16, DOH IRB approval was received (January 2016) and Vital Statistics and Healthy Families Data was provided for advanced analytics (February 2016).

Coordinate Updates to Data Sharing Agreements, As Needed


Implement Master Data Management Across the Child Welfare Community


Initiative 4: Establishment of Data Lab and Tools

Install and Configure Analytics Software TBD n/a TBD

This initiative was not completed in SFY 2015-16 as planned. It remains in the planning phase due to limited funding and DCF/FICW capacity to establish a data lab at this time. The PQMU is currently in the process of exploring other data lab solutions, such as a third party data lab. If it is decided to use a third party data lab, the tasks to implement Initiative 4 should be revised accordingly.

Note: In SFY 2015-16, the OCW invested over $70,000 in IBM SPSS software to provide PQMU performance analysts with an advanced analytics tool. The staff received basic training on this application and additional training is planned for SFY 2016-17.

Test Software Installation and Configuration TBD n/a TBD




COMPLETION COMMENTS

Conduct Knowledge Transfer TBD n/a TBD

Initiative 5: Data System Updates for Initial Measurement Gaps

Resolve Gaps Between the Initial Set of ROA Outcome Measures and the Availability of FSFN Data Required to Calculate the Measures


Initiative 5 is closely related to Initiative 2. Necessary data system updates to address identified gaps for the initial set of ROA measures will be identified by the PQMU and evaluated by the ROA Outcome Measures Workgroup in SFY 2016-17, as the initial set of ROA Program measures is selected. Activities to address data system updates are included in the ROA Outcome Measure Identification and Validation process documentation.

Initiative 6: Accountability Reports

Develop Report Requirements Ongoing n/a SFY 2019-20

Activities to develop and enhance ROA Program reporting will continue throughout the Program implementation as new measures are identified, validated, and incorporated for ROA Program stakeholder groups.

In SFY 2015-16, dashboards and reports were created for existing measures, such as federal outcome measures and CBC Scorecard measures.

Design Reports for Stakeholder Review and Approval

Ongoing n/a SFY 2019-20 See comment above.

Develop Report Extracts, Data Transformation, and Presentation Mechanisms


Test Reports Ongoing n/a SFY 2019-20 See comment above.

Develop and Conduct End User Training





COMPLETION COMMENTS

Deploy Reports Ongoing n/a SFY 2019-20

In SFY 2015-16, dashboards and reports were created for many existing measures, including the federal outcome measures and CBC Scorecard measures. The CBC Scorecard and federal outcome measure dashboards were deployed to Program stakeholders and the general public via the DCF website and Tableau Public in May 2016 and July 2016, respectively. An additional 24 dashboards were developed and published to the DCF intranet for internal use.

Additionally, in SFY 2015-16, the PQMU developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

Conduct Ongoing Annual Updates (Assessment and Update of ROA Dashboard Reports)


Initiative 7: Quality Assurance/Compliance Resource Analysis

Identify Internal and External Units Conducting QA/QI and Contract Compliance Activities (Audits)


Assess Capacity to Conduct Case Reviews and Other QI Activities


Assess External Capacity/Cost to Conduct Contract Compliance Reviews (Audits)





COMPLETION COMMENTS

Develop Recommendations Regarding QI Staffing


Initiative 8: Quality Improvement Organization

Assess QI Needs Associated with the Program


Develop / Modify QI Program Plan and Procedures, To Include ROA Functions


Develop QI Staffing Plan Not Yet Started n/a SFY 2017-18

Obtain QI Staff to Support Program Operations, Research, Evaluation, and QI Functions


Develop and Deliver QI/Learning Organization Training for Managers, Line Staff, and Stakeholders


Initiative 9: Results-Oriented Accountability Reporting System




COMPLETION COMMENTS

Incorporate Stakeholder Data and Screens Into ROA Reporting System


Activities to incorporate new data and report screens into the ROA Reporting System will continue throughout the Program implementation as new measures are identified, validated, and incorporated for ROA Program stakeholder groups.

In SFY 2015-16, DCF identified Tableau as its Department-wide dashboard and reporting tool and has purchased the software licensing and support and established a Data Reporting and Publishing Forum to guide and share information about the use of Tableau across the Department. Tableau is being used by the DRU to create dashboards for existing measures, such as the federal outcome measures and CBC Scorecard measures. At this time, the PQMU plans to use the DCF website and Tableau Public as the ROA Reporting System.

In addition to the dashboards created, the PQMU also developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

Complete Typical Development Life Cycle Ongoing n/a SFY 2019-20

CBC Scorecard and federal outcome measure dashboards were deployed to Program stakeholders and the general public via the DCF website and Tableau Public in May 2016 and July 2016, respectively. An additional 24 dashboards were developed and published to the DCF intranet for internal use.

Initiative 10: Child Welfare Community Data

Complete FSFN System Adoption Initiative In Progress n/a SFY 2016-17

In SFY 2015-16, the FSFN System Adoption project team conducted 14 CBC site visits. The remaining four site visits are scheduled for early SFY 2016-17. The team worked closely with the CBCs to assess their FSFN utilization and develop plans to address identified gaps. This work and the implementation of the plans will continue into SFY 2016-17.




COMPLETION COMMENTS

Map Data Sources to ROA Target Fields


Activities to acquire and incorporate new data for ROA Program reporting, monitoring, and analysis will continue throughout the Program implementation as new measures are identified for ROA Program stakeholder groups. Additionally, assessing and enhancing the quality of Program data will be an ongoing task.

Provide Data Details for Data Sharing Agreements

Not Yet Started n/a SFY 2019-20 See comment above.

Develop Interfaces and Data Extraction and Transformation Tools


Configure Data Cleansing Procedures and Tools


Initiative 11: Institutional Review Policy Update Facilitate Review of IRB Processes Employed By FICW and Affiliated Institutions

Not Yet Started n/a TBD The Department should confirm whether this activity is required for

ROA Program implementation.

Determine the Need for Updates to DCF or FICW Policy To Ensure All Federal and Other Requirements Are Met


In May 2016, DCF transitioned the responsibility for the IRB process from the Division of Substance Abuse and Mental Health to the OCW. The OCW has begun to review the Department’s existing IRB processes, procedures, and requirements and will update as necessary to support the ROA Program. It is likely DCF will utilize an external IRB for implementation activities requiring IRB review and approval in SFY 2016-17.

Develop Program-Specific Procedures for Engaging the IRB Process


In SFY 2015-16, DCF and FICW designed a preliminary high-level IRB sub-process. Once DCF has finalized its IRB process, procedures, and requirements, the process documentation should be updated as necessary.

Initiative 12: Research Standards

Research and Develop Levels of Evidence Construct Specific to Florida and Child Welfare


In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes, including preliminary research review standards. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.




COMPLETION COMMENTS

Research, Test, and Implement a Meta-Analysis Protocol for Research Reviews Requiring Meta-Analysis of Large Numbers of Target Studies


Develop a Process for Estimating Research Budgets


Initiative 13: Pilot Study Standards

Develop Pilot Study Procedures In Progress n/a SFY 2016-17

In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes, including preliminary pilot study procedures. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.

Assess and Update CBC Contracts to Allow for Pilots


Create Standards for Project Management of Pilot Studies


In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes. Preliminary pilot study procedures are incorporated in this process documentation, including project management of pilot studies. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.

Create Protocols for Staffing Pilots

Not Yet Started SFY 2016-17

Initiative 14: Research and Evidence Informed Practice Training Development

Conduct a Training Needs Assessment


Design Training / Strategy





COMPLETION COMMENTS

Develop Training Materials


Implement Training Materials Through FICW


Conduct Training Evaluation to Determine Effectiveness


Integrate Course Into New Employee Curriculum


Initiative 15: Results-Oriented Accountability – FICW Support

Coordinate Database Administration

Not Yet Started n/a TBD

This activity is related to Initiative 4. If the decision is made to use a third party data lab, the tasks to implement Initiative 15 should be revised accordingly.

Perform Data Analysis Not Yet Started n/a TBD See comment above.

Perform Data Extracts, Transformation, and Loads

Not Yet Started n/a TBD See comment above.

Perform Program Research


Advise ROA Program Implementation Team Ongoing n/a SFY 2019-20

In SFY 2015-16, FICW’s Director served as a member of the ROA Governance Committee, ROA Technical Advisory Panel, and ROA Program Implementation Team. In SFY 2016-17, a dedicated FICW resource with advanced research and data analysis skills will be collocated with the PQMU to assist with a number of the ROA implementation initiatives.

Develop ROA Policy, Strategic Design, and Thought Leadership


Support ROA Implementation and Training


Exhibit 5: Status of ROA Program Implementation Initiatives


Schedule delays or uncertainties which impacted the SFY 2015-16 ROA Program implementation activities are discussed below:

The implementation schedule did not include time and resources for the validation of CBC Scorecard measures which were initially identified for deployment as ROA Program measures in SFY 2015-16, along with the federal outcome measures. While the federal measures do not require validation, validation is necessary for the CBC Scorecard measures if they are ultimately selected as ROA measures.

In order to establish the ROA Program measures (Initiative 2: Measure Development and Validation), the PQMU will form an Outcome Measures Workgroup in SFY 2016-17, including representatives from key stakeholder groups. Over the next fiscal year, the Outcome Measures Workgroup will be charged with evaluating and identifying an initial set of ROA measures. As a first step, the Outcome Measures Workgroup will review the measures originally proposed in the ROA Program Plan and determine whether to include these measures, and if so, which measures require validation. Depending on the measure, the validation process could require significant time and/or resources which will need to be factored into the SFY 2016-17 implementation schedule.

The development of ROA dashboards identified for deployment in SFY 2015-16 (Initiative 6: Accountability Reports) was delayed due to hardware resource constraints, a limited number of PQMU staff with the Tableau software skill set(s) required to develop the dashboards, and the time needed for additional staff to develop and effectively apply this new skill set through formal and on-the-job training. In SFY 2016-17, advanced Tableau training for PQMU staff is planned, and additional PQMU staff are being assigned to assist in the development of the dashboards to prevent delays.

The initiative to establish data lab services for the ROA Program is being revisited and has not been finalized to date. A data lab is an entity consisting of a properly supported data environment (physical or virtual), an analytical environment which includes basic and advanced analytical software, a team of one or more business and advanced data analysts, and appropriate business processes to receive and execute requests for data analysis. Its primary function is to explore data using quantitative methodologies to understand the data and answer business, practice, and research questions. A data lab may be an internal entity or an external entity, and may be permanent or project-by-project (the latter generally implies contracting the function to a third party).

The ROA Program Plan includes initiatives (Initiative 4: Establishment of Data Lab and Tools and Initiative 15: ROA – FICW Support) and estimated costs to establish and operate a data lab within FICW to support the Program. However, due to funding constraints, a data lab could not be stood up as planned. A delay in establishing data lab services for the Program could impact activities to validate ROA Program measures and perform advanced data analysis in support of the ROA Program in SFY 2016-17. In order to address this potential issue, the PQMU is working to identify alternative solutions involving partnering with organizations with existing data labs. In March 2016, representatives from the PQMU and FICW conducted a site visit and


discussions with USF to further explore these options and minimize the impact to the implementation timeline. In addition, the OCW expended over $70,000 on IBM SPSS software to provide PQMU performance analysts with the tools needed to support advanced analytics. The staff received basic training on this application and additional training is planned for SFY 2016-17.

2.2 ROA PROGRAM IMPLEMENTATION RISKS

A risk is a potential situation or event which could have an adverse impact on the success of the ROA Program implementation. Risks should be identified and mitigated to the greatest extent possible to keep the Program implementation on track. Exhibit 6: ROA Program Implementation Risks identifies two risks which could adversely affect the success of the ROA Program implementation.

NO. RISK DESCRIPTION

1 SFY 2016-17 funding received for the ROA Program implementation is $500,000. This constraint could impact the quality and timeliness of the implementation objectives and activities currently planned for SFY 2016-17.

2 In addition to the ROA Program, the PQMU supports many other data, data analysis, and reporting needs of the OCW, including providing operational data, media requests, and ad hoc reporting. In the course of performing these ongoing duties, PQMU staff routinely calculate and report on a large number of outcome and process measures. A key principle of the ROA Program is to utilize a set of outcome measures that contains “as few measures as possible” (Testa and Poertner). To ensure the Program is implemented according to its intended design, the PQMU plans to establish an Outcome Measures Workgroup in SFY 2016-17 to evaluate and identify a set of ROA measures which align closely with the outcomes of safety, permanency, and well-being. Some of these measures may be existing measures, while others may be new measures. The establishment of Outcome Measures Workgroup to oversee this process will mitigate the risk of too many measures being used for the ROA Program and will help to ensure outcome monitoring will be manageable and Program stakeholders will be able to effectively assess their performance.

Exhibit 6: ROA Program Implementation Risks

2.3 ROA PROGRAM IMPLEMENTATION ISSUES

An issue is an existing situation or event negatively impacting the quality, resources, and/or timeliness of the ROA Program implementation. Issues must be identified and addressed to keep the Program implementation on track. Exhibit 7: ROA Program Implementation Issues below identifies two ROA Program implementation issues.


NO. ISSUE DESCRIPTION

1 The Department receives annual funding of $500,000 to support ROA Program implementation activities. The initial year of Program implementation described in the ROA Program Plan included between 46 and 55 FTEs, including resources from DCF, FICW, CBC lead agencies, contracted providers, and external vendors involved in implementation activities. Currently, it is estimated only about five to ten percent of the FTEs described in the ROA Program Plan were allocated to the implementation activities for SFY 2015-16. Due to these constraints, the reorganization of the OCW to create the PQMU did not provide for the recommended temporary ROA Program Implementation Team in the organization structure. As a result, in SFY 2015-16 no OCW resources were dedicated solely to the ROA Program implementation, and the work planned for this organizational unit was redistributed to the PQMU managers and staff. Although the PMQU managers and staff are the core members of the ROA Program Implementation Team, they have very limited availability to perform ROA Program implementation activities as most of their time is consumed with day-to-day job responsibilities. In response to this issue, the PQMU Director has identified a full-time ROA Project Manager to oversee the planning and execution of ROA Program implementation activities. Also, FICW has secured a resource with advanced data analysis and research skills to be collocated with the PQMU. While the number of FTEs available to the Department for implementation activities remains less than originally planned, these resources will greatly enhance the ROA implementation efforts in the coming year.

2 The ROA Program implementation has been impacted by funding and resource limitations. As a result, the alignment of the ROA Program Plan and the actual implementation efforts need to be reviewed and realigned as necessary to ensure the quality of the Program implementation is not affected and there is a clear path to follow moving forward.

Exhibit 7: ROA Program Implementation Issues

2.4 LOOKING FORWARD – PLANNED SFY 2016-17 ROA IMPLEMENTATION ACTIVITIES

Under the leadership and guidance of the DCF Secretary and ROA Governance Committee, the OCW has identified the ROA Program objectives and implementation activities for SFY 2016-17. These tasks represent a plan to continue to move forward with the Program implementation in an effective and efficient manner, with a particular focus on the following six objectives:

Objective 1: Partner with FICW and the National Capacity Building Center to develop an evaluation design establishing successful implementation criteria, implementation measures, and overall effectiveness for the ROA Program

Objective 2: Deploy 37 child welfare metrics (outcomes and drivers) to the public facing dashboard environment

Objective 3: Operationalize Year 1 foundational components of the OCW Data Governance and Master Data Management Strategy using a data analysis project as proof of concept

Objective 4: Improve performance of Child and Family Services Review (CFSR) Program outcomes (data measures/systematic factors/file reviews) to meet federal targets utilizing newly developed ROA processes



Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the state

Exhibit 8: ROA Program Implementation Activities (SFY 2016-17) outlines the planned activities and ROA Program Plan initiative(s) associated with each ROA Program objective.

PLANNED ACTIVITIES ROA PROGRAM PLAN INITIATIVE

Objective 1: Partner with FICW and the National Capacity Building Center to develop an evaluation design establishing successful implementation criteria, implementation measures, and overall effectiveness for the ROA Program

Develop ROA evaluation design Gather and analyze evaluation data Produce annual report

Initiative 1: ROA Implementation Project Team Initiative 13: Pilot Study Standards Initiative 15: FICW Support

Objective 2: Deploy 37 child welfare metrics (outcomes and drivers) to the public facing dashboard environment

Establish the average build cycle for each dashboard

Develop a dashboard build schedule Establish the statewide Outcome Measures

Workgroup Finalize the outcome validation methodology Validate five ROA outcome measures utilizing the

validation methodology selected

Initiative 1: ROA Implementation Project Team Initiative 2: Measure Development and Validation Initiative 5: Data System Updates for Initial

Measurement Gaps Initiative 6: Accountability Reports Initiative 9: ROA Reporting System


Procure vendor contract (Objective 2 and 5) Document initial DG/MDM OCW process Evaluate DG/MDM processes utilized in pilot

analytics project to refine and finalize DG/MDM processes to be integrated into formal ROA process guide and process maps

Initiative 1: ROA Implementation Project Team Initiative 3: Master Data Management

Objective 4: Improve performance of CFSR outcomes (data measures/systematic factors/file reviews) to meet federal targets utilizing newly developed ROA processes Complete data analysis of three of the seven

federal measures that will be included in the federal Program Improvement Plan (PIP)

Complete research review phase as outlined in the ROA processes to include feasibility study and evaluation design

Present final results/recommendations to ROA Governance Committee for intervention implementation

Initiative 1: ROA Implementation Project Team Initiative 12: Research Standards Initiative 13: Pilot Study Standards

Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views Finalize research questions Gather data and complete data analysis Validate outcomes/drivers

Initiative 1: ROA Implementation Project Team Initiative 2: Measure Development and Validation Initiative 5: Data System Updates for Initial

Measurement Gaps


PLANNED ACTIVITIES ROA PROGRAM PLAN INITIATIVE

Design individual CBC dashboard views for development

Produce final report

Initiative 6: Accountability Reports Initiative 9: ROA Reporting System Initiative 10: Child Welfare Community Data

Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the state Establish Statewide Analytics Committee and hold

kick-off meeting Inventory/track statewide analytics projects Train members on IRB changes, process, and

requirements

Initiative 1: ROA Implementation Project Team Initiative 8: Quality Improvement Organization

Exhibit 8: ROA Program Implementation Activities (SFY 2016-17)


SECTION 3 RECOMMENDATIONS

In SFY 2015-16, DCF made strong progress towards the multi-year ROA Program implementation. The Department plans to continue to work diligently on the Program’s implementation in SFY 2016-17, focusing on developing metrics and reporting, strengthening its analytical capabilities, improving data quality, implementing data governance and management strategies, and increasing stakeholder awareness and understanding of ROA and the cycle of accountability. Below, North Highland provides recommendations for the Department’s consideration to support its efforts to achieve the SFY 2016-17 ROA Program objectives and successfully complete planned implementation activities.

Note: Additional detail regarding recommendations for data governance, master data management, and advanced analytics can be found in the following North Highland deliverables:

Deliverable 4: Data Governance and Strategy

Deliverable 6: Updated FSFN Dictionary

Deliverable 7: Foundations for Analytics


Observation: The Advanced Data Analytics project is a broad initiative to expand systematically the use of data analytics and predictive analysis techniques to improve outcomes for children and families. The ROA Program is focused on the research, development, implementation, and ongoing measurement of the success of interventions or programs affecting the safety, permanency, and well-being of children and families. Both are complementary efforts supporting the same objectives. OCW has an increasing dependency on the quality of internal data and the ability to access external data in a timely manner and ascertain the quality of the data once it is received. A foundational strategy of data governance, data management, and data acquisition processes will increase efficiency and reduce expenses for data-related efforts.

Recommendation: A number of data governance projects/initiatives could start and potentially be completed in SFY 2016-17, depending on the capacity of available resources. The majority of this effort should focus on planning and applying a proof of concept approach to the data governance and management strategies introduced in SFY 2015-16. To derive the most benefit, the OCW should collaborate with DCF executive management and leadership from the other program areas, such as Economic Self-Sufficiency and Substance Abuse and Mental Health. Some projects are already underway (for example, the Client Data Link System) that would benefit from a formal data governance structure and organization.


North Highland recommends the Department build support for formal data governance with executive management and program area leadership through establishing a:

Well-executed communications strategy

Strong business case

Comprehensive framework and charter documents

The goal of the planning effort is to obtain agreement and approval on the needs and objectives of data governance. Once the objectives of data governance are approved, the Department should define and designate the roles and responsibilities within data governance, including:

Cross-functional Executive Steering Committee

Data Governance Board that defines plans to execute the strategy

Program / Functional area Data Stewards, responsible for the day-to-day work

The next step would be to document the policies and procedures under which data governance will operate and to define the supporting IT and business processes. The business processes will most likely require organizational changes and transformation to implement. With these key planning tasks complete, the Department will be ready to select and implement the tools and infrastructure necessary to support data governance and data management.

Observation: Two of the guiding principles of the ROA Program are include:

Establishing a collaborative, statewide Child Welfare Community accountable for the safety, permanency, and well-being outcomes of Florida’s children

Translating data collection in the Child Welfare Community to meaningful and useful information to enable outcome-focused decision-making

The Department seeks to evolve Florida’s Child Welfare System into a more child and family-focused, outcome-driven system. To accomplish this, the scope of data collection and modeling must be extended to capture the interdependencies between the various stakeholders and to raise awareness and accountability of those interdependencies across the Child Welfare Community. A MDM strategy and execution is required to support these principles and objectives.

Recommendation: As with data governance, the work that can be accomplished for MDM in SFY 2016-17 depends on the capacity of available resources. As originally introduced in the ROA Program Plan, North Highland recommends the OCW build a person-centric view of data (Person Golden Record), with the child data as a sub-set. This effort should be coordinated with the Department’s Client Data Link project already underway. A child-focused data model could then be built from the Person


Golden Record. All circumstances influencing a child’s welfare should be included, both people as well as the potential legal and healthcare influences.

MDM planning, justification, and tool evaluation should be built upon the foundation laid for data governance and data management. Processes and procedures specific to MDM can be developed and tools can be evaluated that add capability to the existing data governance / data management foundation.

Policies and procedures should be documented, clearly describing the planned operational aspects of MDM. The goal is to accomplish the same tasks recommended to operationalize data governance, including additional MDM-specific needs, which primarily consist of identifying the System of Record (SOR) for each master data domain. The SOR is the most reliable source for a given data element or piece of information.

The remaining MDM tasks may be beyond OCW’s resource capacity for SFY 2016-17; however, they are as follows:

Implement a child-centric data model in the MDM environment as a pilot project. Again, this child-centric data model should include all people and legal and healthcare circumstances influencing the child’s welfare. The pilot project should be defined and executed under a small-scale highly controlled scenario to validate functionality. The parameters of the pilot project should allow for easy and obvious identification of problems to make process revisions as quick as possible.

Establish additional data relationships once the data model, business rules, and automated processes are stable, making the model the basis of future data analytics and research supporting the ROA Program.

Ideally, the data model should mature beyond the OCW’s and Department’s use alone. The longer-term vision is for all state agencies affecting child welfare to utilize the person-centric data model to share data to enable a “single view of the truth” concerning the child.


Observation: Currently, only very limited and general information about services and practices is captured in FSFN. It is understood the data captured is not comprehensive and / or exists in an unstructured format (free-form text, documents and images, etc.) which is difficult to leverage. While information about out-of-home care is more consistently available, details of in-home care are scarce.

Recommendation: Two of the primary objectives of the FSFN System Adoption project are to understand how CBCs are currently using FSFN and to education them on the proper use of the system. It is anticipated some improvement in services and practice data will occur as a result of this project. In addition to this effort, North Highland


recommends the Department conduct an additional inventory and study of services and practice data maintained at each CBC if needed, as each CBC stores and manages this information differently. Based on the results, a plan and roadmap can be developed to integrate and assimilate this information to enrich FSFN.

Observation: The Advanced Data Analytics project conducted in SFY 2014-15 preliminarily quantified the issue of a single person having multiple entries, or person records, which are not properly linked in FSFN. As a result, a caseworker or hotline personnel may not see the complete history of a child and his / her environment in the course of performing their job activities. The Advanced Data Analytics project conducted in SFY 2015-16, which builds on the analysis and findings from the previous year, has estimated that a typical case of three to four participants has more than a 50 percent chance of missing history related to the case in question due to this fragmentation of information pertaining to the same person. For example, by not being able to identify all cases related to a perpetrator in the case at hand, a case worker would not see all of the relevant history for the perpetrator because this individual may have multiple person records.

Recommendation: The Department’s current effort around developing a MPI appears to be more advanced than originally thought. Given this, North Highland recommends the Department leverage the algorithm being used for MPI and create a process to apply the same algorithm to FSFN at the source to begin consolidating the multiple identities correctly. In the longer term, the Department may consider a more sophisticated commercial solution for potential improvement.

Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the State

Observation: Two of the guiding principles of the ROA Program are to:

Establish a collaborative, statewide Child Welfare Community accountable for the safety, permanency, and well-being outcomes of Florida’s children

Translate data collection in the Child Welfare Community to meaningful and useful information to enable outcome-focused decision-making

The ROA Program involves a significant cultural change across the Child Welfare Community. Buy-in and acceptance of ROA principles are critical for the Program’s success as it is incrementally deployed across stakeholder groups. In SFY 2015-16, the PQMU developed a ROA Communication Plan which includes a number of communication events to increase stakeholder awareness and understanding of the ROA Program. The ROA Project Manager will continue to assess stakeholder information needs annually, or more frequently if needed, and update the ROA Communication Plan as appropriate.

Recommendation: In addition to the Department’s communication strategy, effective change management will help to ensure the success of the ROA Program. North


Highland recommends the PQMU create an Organizational Change Management Plan for the ROA implementation and begin to establish a change network across the stakeholder groups which will be impacted by the initial phase of the Program deployment (Region Offices, CBC lead agencies, CBC contracted providers, and Sheriff’s Offices). These change agents can assist the PQMU in preparing their organizations for change, guiding, and managing the adoption of a new culture of evidence-informed decision-making, mitigating change resistance, and sustaining the change.

Observation: In SFY 2015-16, the Department developed detailed ROA process documentation. The Data Analysis process documentation includes a process for ROA-related data analysis as well as a data analysis process to support the operational needs of the OCW. PMU personnel are being trained on Six Sigma and other statistical methods to assist in conducting initial data analysis. However, the OCW is otherwise resource-constrained to develop an in-house advanced analytics team, and the use of an external lab to execute advanced analytics is being evaluated.

There are also a number of organizations throughout the State independently conducting analytical projects and initiatives. In addition to FICW and the Florida Center for Child Welfare, some of the CBCs have internal and/or external analytical capabilities and have conducted analytical projects. Currently, these external analytical activities are not transparent to OCW, and it is unknown the extent to which standards are being followed or duplication of effort is occurring.

Recommendation: To maximize analytical leverage, the OCW must establish a solid bridge between the business need for advanced analytics and those who execute advanced analytics. However, this condition does not require the skill set to execute advanced analytics reside within the OCW. As such, North Highland recommends the PMU designate a proficient business user of advanced analytics outputs to serve as a point person for OCW advanced analytics projects and initiatives. Furthermore, it is recommended that OCW be the business owner of child welfare analytics for the State to facilitate opportunities for collaboration while ensuring consistency in approach and execution. The OCW can start by building a network of collaboration and sharing for child welfare analytics, followed by the implementation of statewide IRB standards for child welfare analytics, which can then be expanded to a statewide analytical governance for child welfare.

Documents

Quarterly Progress Report on ROA Implementation - …centerforchildwelfare.fmhi.usf.edu/qa/QA_Docs/June 2016 Quarterly...Quarterly Progress Report on ROA Implementation - June 2016