of 41/41
FLORIDA DEPARTMENT OF CHILDREN AND FAMILIES QUARTERLY PROGRESS REPORT ON ROA IMPLEMENTATION - JUNE 2016 DELIVERABLE 2B Date: 08/23/2016 Version: 3.00

Quarterly Progress Report on ROA Implementation - …centerforchildwelfare.fmhi.usf.edu/qa/QA_Docs/June 2016 Quarterly...Quarterly Progress Report on ROA Implementation - June 2016

  • View
    214

  • Download
    2

Embed Size (px)

Text of Quarterly Progress Report on ROA Implementation -...

  • FLORIDA DEPARTMENT OF CHILDREN AND FAMILIES QUARTERLY PROGRESS REPORT ON ROA IMPLEMENTATION - JUNE 2016

    DELIVERABLE 2B

    Date: 08/23/2016 Version: 3.00

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page i

    TABLE OF CONTENTS

    1.1 ROA Program Plan Implementation Timeline and Initiatives ........................................ 7 1.2 ROA Program Implementation Team ......................................................................... 10 1.3 ROA Program Governance ........................................................................................ 14

    1.3.1 Roles and Responsibilities .................................................................................. 16

    2.1 Status of ROA Program Implementation (SFY 2015-16) ............................................ 17 2.2 ROA Program Implementation Risks .......................................................................... 30 2.3 ROA Program Implementation Issues ........................................................................ 30 2.4 Looking Forward Planned SFY 2016-17 ROA Implementation Activities ................ 31

    TABLE OF EXHIBITS

    Exhibit 1: ROA Program Initiatives ............................................................................................ 10 Exhibit 2: DCF, OCW and PQMU Organization ........................................................................ 12 Exhibit 3: ROA Program Implementation Governance ............................................................. 15 Exhibit 4: ROA Program Governance Roles and Responsibilities ............................................ 16 Exhibit 5: Status of ROA Program Implementation Initiatives ................................................... 28 Exhibit 6: ROA Program Implementation Risks ........................................................................ 30 Exhibit 7: ROA Program Implementation Issues ....................................................................... 31 Exhibit 8: ROA Program Implementation Activities (SFY 2016-17) .......................................... 33

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page ii

    Revision History

    DATE AUTHOR VERSION CHANGE REFERENCE

    06/22/2016 North Highland .001 Prepared initial draft

    06/30/2016 North Highland 1.00 Updated report to prepare for submission to DCF for review

    07/27/2016 North Highland 1.01 Updated report based on DCF feedback and submitted for informal review

    07/28/2016 North Highland 2.00 Updated report based on DCF feedback and prepared for submission to DCF for review 08/05/2016 North Highland 2.01 Updated report based on DCF review feedback

    08/15/2016 North Highland 2.02 Updated report based on DCF feedback and prepared draft for DCF review

    08/23/2016 North Highland 3.00 Updated report based on DCF feedback and prepared for DCF review

    Quality Review

    NAME ROLE DATE Tina Worley NH Client Lead / Project Manager 06/28/2016 Tina Worley NH Client Lead / Project Manager 06/30/2016 Tina Worley NH Client Lead / Project Manager 08/01/2016

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 1

    EXECUTIVE SUMMARY

    Creating positive change for Floridas children and families is only possible when all the organizations involved with child welfare recognize their individual and collective roles in enhancing the safety, permanency, and well-being of those served. In Florida, the key child welfare stakeholders and partners include the Department of Children and Families (DCF, Department), Community-Based Care (CBC) lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. Collectively, these stakeholders represent Floridas Child Welfare Community. The unique partnerships within the Child Welfare Community create opportunities for long-term improvement by bringing together many perspectives and experiences with a singular focus on improving the lives and safety of each child in Florida.

    The Florida Legislature passed Florida Statute (F.S.) 409.997 in 2014. The actions of the Legislature allowed the creation of a platform for extensive advancement of the child welfare system through establishment of the Results-Oriented Accountability Program (ROA Program, Program). The goal of the ROA Program is to develop an integrated, research-informed framework designed to inform communities, the child welfare system, and legislators on essential elements of child protection (Chapters 20, 39, and 409, F.S.).

    This document is the Progress Report for the implementation of the ROA Program for State Fiscal Year (SFY) 2015-16, reflecting status as of June 2016. The Progress Report references F.S. 409.997 and the implementation initiatives and roadmap outlined in the ROA Program Plan created by the Department in February 2015, to assess the progress of the Program implementation. The report also discusses circumstances resulting in deviation from the implementation roadmap and how guidance from Department Secretary Mike Carrol and the ROA Governance Committee establishes the strategic direction and focus for the ROA Program implementation efforts on an annual basis.

    The status reported in this document is based primarily on discussions with Office of Child Welfare (OCW) management and staff, participation in ROA Program meetings and workgroups, and a review of relevant documentation and includes the following sections:

    Program Background: This section includes an overview of the ROA Program background and purpose, implementation team, and governance.

    Program Implementation Status: This section includes information detailing the status of the Program implementation to date, including implementation activities, risks, and issues. A summary of the ROA Program objectives and major Program implementation activities planned for the upcoming state fiscal year is also provided.

    Recommendations: This section presents recommendations for the Departments consideration related to the implementation activities planned for the upcoming state fiscal year.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 2

    Key Accomplishments

    Over the past year the Department has made a number of significant accomplishments with respect to the ROA Program implementation, as highlighted below. Additional detail on these accomplishments and other implementation activities is provided in Section 2 of this report.

    A Child Welfare Performance and Quality Management Unit (PQMU) was created within the OCW and charged with the responsibility for planning and managing the ROA Program implementation. A ROA Project Manager has been identified to serve as a dedicated resource for the ROA Program implementation and will be moving into this role in early SFY 2016-17.

    The PQMU expended over $100,000 in ROA-related hardware and software during SFY 2015-16. As part of this investment, the PQMU secured licenses for and implemented Tableau data visualization software and SPSS predictive analytics software. Basic training on Tableau and SPSS was conducted for PQMU staff, and advanced training in both applications is planned for SFY 2016-17. Additionally, three PQMU Performance Management Unit (PMU) staff obtained their Six Sigma Yellow Belt and Green Belt certifications in SFY 2015-16.

    Twenty-nine interactive dashboards for priority measures were developed in Tableau for child welfare stakeholders. A subset of those dashboards for CBC Scorecard measures was released in May 2016 to Program stakeholders and the general public via the DCF website and Tableau Public. The PQMU also developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

    The ROA Program Technical Advisory Panel, which was established to serve in an advisory capacity during the development of the ROA Program Plan, was remobilized to continue to perform in this role throughout the Program implementation. A Technical Advisory Panel Charter was created describing the role and responsibilities of the panel.

    The ROA Program Governance Committee was established, and members were appointed in March 2016. A Governance Committee Charter was developed describing the role and responsibilities of the committee. The initial meeting of the committee was held on April 7, 2016. During this meeting the Governance Committee Charter, ROA Program implementation status, and plan for the upcoming SFY were presented and discussed. An introduction to Tableau and demonstration of the dashboards were presented to the committee. The feedback received from the ROA Governance Committee on the status of the Program implementation was very favorable, and the next meeting is planned for July 2016.

    Subject matter experts from ROA stakeholder groups statewide collaborated to develop detailed process diagrams and narratives for the five activities that drive the ROA cycle of accountability: Outcomes Monitoring, Data Analysis, Research Review, Evaluation, and Quality Improvement (QI). Process documentation for the identification

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 3

    and validation of ROA outcome measures was also created. The process documentation describes the activities required to complete the ROA Program processes, the resources responsible for executing those activities, as well as critical hand-off points between the processes.

    Additional external resources were secured through competitive procurement to support Program implementation efforts in the areas of advanced data analytics and data strategy and governance. Under the PQMUs direction, a Florida Safe Families Network (FSFN) Business User Data Dictionary and Data Governance and Strategy were developed. The FSFN Business User Data Dictionary identifies, documents, and analyzes FSFN data critical to supporting ROA measures and reporting and advanced analytics initiatives. The Data Governance and Strategy document provides an overarching data strategy linked to business strategy, data governance standards, data standardization and stewardship responsibilities, standard procedures for data acquisition, and master data management (MDM) procedures and constructs. Additionally, an advanced analytics project was conducted to study re-maltreatment and re-reporting to better understand risk factors influencing outcomes for children and to identify needs for policy and practice changes. A key finding from this analysis was breaking the cycle of maltreatment requires policies and practices designed to counter perpetrator recidivism over a period of five to ten years or more.

    As a result of monitoring out-of-home care (OOHC) data and identifying an increase in the number of removals and placement of children in OOHC, the Department partnered with Casey Family Programs and The Ounce of Prevention Fund of Florida to conduct an analysis of this trend, utilizing data analysis techniques such as multi-variate analysis. The OOHC Analysis Project, which is representative of the outcomes monitoring and data analysis stages of the ROA Programs cycle of accountability, was completed in February 2016 with a final report providing countermeasures and recommendations.

    The FSFN System Adoption Project was initiated which involves a collaboration between the Department and each CBC lead agency to identify CBC information and technology requirements and develop an individualized System Adoption Plan that achieves full adoption of FSFN while supporting CBC business processes. This project will help to identify training and policy needs and ensure complete, accurate, and consistent data resides in FSFN for the ROA Program and other DCF initiatives.

    The Department engaged the Florida Institute for Child Welfare (FICW) to conduct a feasibility study to inform the design of a planned study to assess the inter-rater reliability and validity of new family assessment tools. The feasibility study is scheduled to be completed in December 2016.

    Key Challenges

    Key challenges the Department has faced in the initial year of the ROA Program implementation include:

    Annual funding for the ROA Program implementation for SFY 2015-16 was very limited in comparison to the estimated funding requirements described in the ROA Program

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 4

    Plan. This constraint impacted the OCWs ability to complete all activities outlined for SFY 2015-16 in the ROA Program Plan while maintaining business as usual. As a result, OCW resources were allocated between the ROA Program implementation activities and day-to-day operational responsibilities and demands.

    The ROA Program Plan recommended a much larger number of dedicated resources to perform implementation activities, including DCF, FICW, and contractor resources. The shift in available resources likely will impact not only the duration of the implementation plan, but also the overall scope and approach. As a result, the implementation roadmap and the actual implementation efforts are not aligned, which makes planning for the multi-year implementation difficult.

    Recommendations for SFY 2016-17

    For SFY 2016-17, under the guidance and leadership of the DCF Secretary and ROA Governance Committee, the OCW has established six objectives for the ROA Program which serve to focus implementation activities. The PQMU has identified the tasks required to achieve these objectives which form the foundation of the ROA Program SFY 2016-17 implementation schedule.

    Based on the PQMUs planned implementation activities for SFY 2016-17, North Highland has developed recommendations for the Departments consideration, which align with three of the ROA Program objectives and are centered on continuing to build the infrastructure (people, processes, and technology) necessary for the Programs success. These recommendations are summarized below and presented in more detail in Section 3 of this report.

    Objective 3: Operationalize Year One foundational components of OCW Data Governance and Master Data Management Strategy

    Recommendation: North Highland recommends the Department build executive management and program area leadership buy-in and support for formal data governance through establishing a:

    Well-executed communications strategy

    Strong business case

    Comprehensive framework and charter documents

    Once the objectives of data governance are agreed upon and approved, the Department should designate data governance roles and responsibilities, document policies and procedures under which data governance will operate, define the supporting IT and business processes, and evaluate the necessary tools.

    Recommendation: North Highland recommends the OCW develop a person-centric view of data, with the child data as a sub-set. This effort builds on language in the ROA Program Plan specific to creating a data viewpoint from the perspective of a child in care and should be coordinated with the Departments Client Data Link project

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 5

    already underway. Planning, processes, procedures, and tool evaluation specific to MDM should leverage the foundation built for data governance.

    Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views

    Recommendation: North Highland recommends the Department conduct an additional inventory and study of services and practice data maintained at each CBC, as noted in the ROA Program Plan. Based on the results, a plan and roadmap can be developed to integrate and assimilate this information to enrich FSFN.

    Recommendation: North Highland recommends the Department leverage an algorithm already being used for an existing Master Person Index (MPI) effort and create a process to apply the algorithm to FSFN at the source to begin consolidating the multiple person identities correctly. In the longer term, the Department may consider a more sophisticated commercial solution for potential improvement.

    Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the State

    Recommendation: In addition to the ROA communication strategy to be executed in SFY 2016-17, North Highland recommends the PQMU create an Organizational Change Management (OCM) Plan for the ROA Program implementation and begin to establish a change network across the Child Welfare Community stakeholders impacted by the initial phase of the Program deployment.

    Recommendation: North Highland recommends the OCW designate a point person for all OCW advanced analytics projects and initiatives. Furthermore, it is recommended the OCW be the business owner of child welfare analytics for the State to facilitate opportunities for collaboration while ensuring consistency in approach and execution.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 6

    SECTION 1 PROGRAM BACKGROUND

    Improving outcomes for Floridas vulnerable children and families is dependent on the concept the entire community is responsible for child safety, permanency, and well-being. Floridas child welfare system consists of key stakeholders and partners that include the Department, CBC lead agencies, contracted community providers, other state agencies, tribes, and the legal and judicial systems. These stakeholders, which comprise the states Child Welfare Community, each play an important role in and have a shared accountability for the safety, permanency, and well-being of Floridas children.

    With the support of state policymakers and legislators, DCF and its partners are implementing the ROA Program to deliver greater innovation and reform. The Program will address existing and future challenges and drive positive change in Floridas child welfare system. To do this, the Program will establish a robust and continuous cycle of monitoring, data analysis, research, evaluation, and quality improvement to further advance the systems efforts to improve outcomes, identify new programs and services impacting those outcomes, and enable research and evidence-informed practice, policy, and decision-making.

    In 2014, the Florida Legislature passed F.S. 409.997, directing the Department to contract with a qualified consultant or organization to develop a plan for the implementation of the ROA Program. As stated in F.S. 409.997, the purpose of the ROA Program is to monitor and measure:

    the use of resources

    the quality and amount of services provided

    child and family outcomes

    The Program conceived by the Legislature and whose design and implementation is laid out in detail in the ROA Program Plan submitted by the Department in February 2015 is focused on improving the outcomes of the children and families served by the OCW through evidence-informed interventions, practices, and policies. The Program is based on the cycle of accountability framework presented by Mark Testa and John Poertner in their book, Fostering Accountability, Using Evidence to Guide and Improve Child Welfare Policy. The cycle of accountability relies on operationalizing five key activities to further advance the efforts of state child welfare systems to evaluate performance on outcomes, identify new or promising interventions and strategies, review the validity of programs, and conduct continuous quality improvement to ensure the system as a whole is learning and moving toward the accomplishment of goals which positively impact children and their families.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 7

    The five activities included in the cycle of accountability are:

    1. Outcomes Monitoring, activities to define, validate, implement, and monitor outcome measures throughout the Child Welfare Community

    2. Data Analysis, approaches and procedures to critically study performance results to determine if variances discovered are in fact issues which should be explored further

    3. Research Review, a series of activities to gather and validate evidence to support the development and implementation of interventions to address areas for improvement

    4. Evaluation, activities to assess promising interventions for children and families to determine if deployment to a larger population is warranted

    5. Quality Improvement, a series of actions to implement interventions across new domains, or to challenge, change, and test new assumptions about the underlying goals supporting the child welfare practice model

    The ROA creates accountability and transparency by incorporating processes and tools for the dissemination of performance results to the Child Welfare Community and general public. The goal of Program accountability and transparency is to present accurate and timely performance information, along with an analysis of factors influencing trends, enabling stakeholders to view a true picture of the system and potential needs for improvement.

    1.1 ROA PROGRAM PLAN IMPLEMENTATION TIMELINE AND INITIATIVES

    The ROA Program Plan provides a five-year roadmap of initiatives for the implementation and ongoing operation of the Program. Initial implementation activities occur over two years to establish the ROA Program, including an initial set of measures and the associated reporting and monitoring for key stakeholder groups. After the initial implementation, there are three years of ongoing implementation activities to continue to incorporate new data, measures, reporting, and monitoring for additional stakeholder groups. In total, the Program Plan identified 15 implementation initiatives and their associated activities, which are summarized below in Exhibit 1: ROA Program Initiatives.

    NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES PLANNED DURATION 1 Results-Oriented

    Accountability Implementation Project Team

    This initiative creates the management structure and processes required to manage and oversee Program implementation.

    Establish ROA Implementation Project Team

    Create program management processes

    Create governance processes and procedures

    Develop implementation master schedule

    Engage project teams and execute prioritized initiatives

    Manage and respond to risks affecting program implementation

    Initial Implementation: 07/2015-12/2015 Ongoing Implementation: 01/2016-06/2020

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 8

    NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES PLANNED DURATION 2 Measure

    Development and Validation

    This initiative develops reliable and valid outcome measures, sets outcome measure baselines and targets, and identifies outcome measure drivers.

    Identify ROA measures Confirm measure validity and identify

    unintended consequences through research, data analysis, and logic models

    Develop calculation algorithms Set performance baselines and

    targets Pilot measures and make necessary

    adjustments based on results Update relevant documentation

    Initial Implementation: 01/2016-03/2016 Ongoing Implementation: 04/2016-06/2020

    3 Master Data Management

    This initiative develops and implements effective governance processes and procedures to enable data collaboration, quality, and standardization while safe guarding confidentiality.

    Define MDM procedures and documentation

    Establish data sharing agreements with stakeholders

    Coordinate updates to data sharing agreements as needed

    Implement MDM across the Child Welfare Community

    Initial Implementation: 01/2016-06/2016 Ongoing Implementation: 07/2016-06/2020

    4 Data Lab and Tools

    This initiative establishes the ROA analytics hardware and software environment.

    Install and configure analytics software

    Test software installation and configuration

    Conduct knowledge transfer

    Implementation: 01/2016-06/2016

    5 Data System Updates for Initial Measurement Gaps

    This initiative resolves gaps between the selected ROA outcome measures and FSFN data required to calculate the measures.

    Resolve gaps between initial set of ROA outcome measures and the availability of FSFN data required to calculate the measures

    Implementation: 07/2016-06/2017

    6 Accountability Reports

    This initiative supports the development of ROA reports for Program stakeholders, creating transparency and accountability by facilitating timely and meaningful access to Program results.

    Develop report requirements Design reports for stakeholder review

    and approval Develop report extracts, data

    transformation, and presentation mechanisms

    Test reports Develop and conduct end user

    training Deploy reports Conduct ongoing annual updates

    Implementation: 10/2015-03/2016 Ongoing Implementation: 07/2016-06/2020

    7 Quality Assurance (QA) / Compliance Resource Analysis

    This initiative assesses and provides recommendations on the current utilization of internal and external QA/QI resources and the QI resources required to support the ROA Program.

    Identify internal and external units conducting QA/QI and contract compliance activities

    Assess capacity to conduct case reviews and other QI activities

    Assess external capacity/cost to conduct contract compliance reviews

    Develop QI staffing recommendations

    Implementation: 10/2015-01/2016

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 9

    NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES PLANNED DURATION 8 Quality

    Improvement Organization

    This initiative includes assessing the QI needs of the ROA Program, securing the staff to meet the requirements, and developing and delivering a QI focused training for managers, staff, and other stakeholders to instill and reinforce a learning organization mindset.

    Assess QI needs associated with the Program

    Develop/Modify a QI Program Plan and procedures to include ROA functions

    Develop a QI Staffing Plan Obtain QI staff to support Program

    operations, research, evaluation, and QI functions

    Develop and deliver training on the QI/Learning Organization for managers, line staff, and stakeholders

    Implementation: 01/2016-06/2016

    9 Results-Oriented Accountability Reporting System

    This initiative establishes the operational reporting system to monitor and improve results and accountability across Child Welfare Community.

    Incorporate stakeholder data and screens into the ROA Reporting System

    Complete typical development life cycle

    Initial Implementation: 01/2016-03/2016 Ongoing Implementation: 04/2016-06/2020

    10 Child Welfare Community Data

    This initiative develops non-FSFN data interfaces to incorporate quality data from the Child Welfare Community into the ROA reporting system and addresses data quality issues in FSFN.

    Complete FSFN System Adoption initiative

    Map data sources to target fields Provide data details for data sharing

    agreements Develop interfaces and data

    extraction and transformation tools Configure data cleansing procedures

    and tools

    Initial Implementation: 01/2016-03/2016 Ongoing Implementation: 04/2016-06/2020

    11 Institutional Review Policy (IRB) Updates

    This initiative provides an IRB policy and process for the ROA Program which meets federal and other requirements.

    Facilitate the review of IRB processes employed by FICW and affiliated institutions

    Determine the need for updates to FICW or DCF IRB policy to ensure all federal and other requirements are met

    Develop Program-specific procedures for engaging the IRB process

    Implementation: 04/2016-05/2016

    12 Research Standards

    This initiative develops ROA Program research standards, a Levels of Evidence construct, methodologies, protocols, and tools to ensure research reviews are conducted consistently and produce reliable results.

    Research and develop levels of evidence construct specific to Florida and child welfare

    Research, test, and implement a meta-analysis protocol for research reviews requiring meta-analysis of large numbers of target studies

    Develop a process for estimating research review budgets

    Implementation: 04/2016-06/2016

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 10

    NO. INITIATIVE DESCRIPTION IMPLEMENTATION ACTIVITIES PLANNED DURATION 13 Pilot Study

    Standards

    This initiative develops pilot procedures to determine all of the critical elements, approvals, and considerations to be addressed prior to implementing a pilot.

    Develop pilot study procedures Assess and update CBC contracts to

    allow for pilots Create standards for project

    management of pilot studies Create protocols for staffing pilots

    Implementation: 04/2016-08/2016

    14 Research and Evidence-Informed Practice Training Development

    This initiative includes conducting a training needs assessment and developing research and evidence-informed practice training to drive a culture shift within the system to that of a learning organization in which data is used in decision-making.

    Conduct a training needs assessment

    Design a training strategy Develop training materials Implement training materials through

    FICW Conduct training evaluation to

    determine effectiveness Integrate course into new hire

    curriculum

    Implementation: 07/2016-11/2016

    15 Results Oriented Accountability FICW Support

    This initiative represents FICW activities required to support the ROA Program, including providing a ROA center of learning, acting as ROA advisors, leading research and evaluation pilots, and optimizing the ROA implementation.

    Coordinate database administration Advise ROA Program Implementation

    Team Perform data analysis Perform data extracts,

    transformation, and loads Perform program research Develop ROA policy, strategic

    design, and thought leadership Support ROA implementation and

    training

    Implementation: 07/2016-06/2020

    Exhibit 1: ROA Program Initiatives

    The ROA Program Plan estimated the cost for the first year of the Program implementation (SFY 2015-16) between $8.5 and $10.6 million and recommended 46 to 55 full time equivalents (FTEs) working across the Department, FICW, contracted providers, and external vendors. However, the Department has received $500,000 in annual funding to implement the Program, an amount which represents only a small fraction of the original cost estimates.

    Each year, with guidance from the DCF Secretary and ROA Governance Committee, the OCW establishes a set of objectives for the ROA Program which serve to focus the PQMUs implementation efforts based on the original ROA Program initiatives. The PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives.

    1.2 ROA PROGRAM IMPLEMENTATION TEAM

    The Departments OCW is primarily responsible for the implementation and operation of the ROA Program. The OCW is committed to the safety, well-being, and timely permanency of

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 11

    Floridas children and families and is responsible for a wide range of services, including assistance to help families stay intact or be reunified, out-of-home care, adoption, and independent living skills for foster care youth transitioning to adulthood. To do this, the OCW works with six DCF Region Offices, 17 CBC lead agencies, and six Sheriffs Offices to execute policy and practice for child protective investigations and case management services.

    The ROA Program Plan recommended a reorganization of the OCW to create a structure to better support the implementation and operation of the Program. In 2015, the OCW established the PQMU based on the Program Plans recommended design to oversee the implementation and ongoing operation of the Program. The PQMU is led by the PQMU Director who reports directly to the Assistant Secretary for Child Welfare. The PQMU includes three sub-units:

    Data and Reporting Unit (DRU): This unit supports the Departments internal and external reporting needs, including developing and producing federal reports, management reports, operations reports, ad hoc reports, and performance dashboards, as well as assisting the Department in responding to media, legislature, and general public requests for information. The DRU also provides data sets and SQL queries to the Performance Management Unit to be used for data analysis.

    Performance Management Unit: This unit monitors key performance indicators, produces performance summaries, and disseminates information through publications and presentations. These products contain initial stages of data analysis in support of Root Cause Analysis, a critical part of the ROA Programs cycle of accountability that ties data reporting, monitoring, and analysis together with continuous quality improvement. The PMU also supports data-driven management through the provision of ad hoc data analysis services and support of research and evaluation projects.

    Quality Improvement / Quality Assurance (QI/QA) Unit: This unit performs ongoing case reviews of children served by DCF, including subjects of child protective investigations or children who are/were recently in out-of-home or in-home care. The unit also interviews parties involved in the childs case to understand how day-to-day practice influences child and family functioning and outcomes. Quality management (QM) data is regularly reported to DCF leadership, Region Offices, and CBC lead agencies to ensure operations are engaged in activities to improve practice.

    Exhibit 2: DCF, OCW and PQMU Organization depicts the placement of the PQMU in the Departments organizational structure and highlights the detailed organization of the PQMU.

  • Quarterly Progress Report on ROA Implementation - June 2016

    Florida Department of Children and Families - Office of Child Welfare Page 12

    Exhibit 2: DCF, OCW and PQMU Organization

    The People

    Governor

    Secretary

    Childrens Legal Services

    Director

    Chief of Staff

    Planning Director

    Communications Director

    Inspector General

    Assistant Secretary for

    Operations

    Regional Managing Director

    Northeast Region

    Regional Managing Director Northwest Region

    Assistant Secretary for

    Administration

    Child Welfare Operations

    Director

    Child Welfare Planning & Strategic Projects Director

    General Counsel

    Regional Managing Director

    Southern Region

    Regional Managing Director Suncoast Region

    Adult Protective Services Director

    Regional Managing DirectorCentral Region

    Regional Managing Director

    Southeast Region

    Refugee Services

    Director

    Human Resources Director

    Contract Services Director

    Chief Information Officer

    General Services Director

    Chief Financial Officer

    Child Welfare Practice

    Director

    Legislative Affairs Director

    Assistant Secretary for

    Economic Self-Sufficiency

    Economic Self-Sufficiency

    Director

    Office of Public Benefits

    Integrity Director

    Deputy Secretary

    Assistant Secretary for Sub. Abuse & MH

    Sub. Abuse & MH

    Director

    Homelessness Director

    State MH Treatment Facilities Director

    Florida State Hospital

    Administrator

    NE FL St. Hosp. & N FL Eval.

    Trmt. Ctr. Administrator

    Sexually Violent Predator ProgramDirector

    Child Welfare Performance & Quality Management Director

    Assistant Secretary for Child

    Welfare

    OCW

    Child Welfare Performance and Quality Management Unit

    Child Welfare Performance and Quality Management Unit

    Director

    Data and ReportingManager

    Finance / Independent Living Reporting

    Ongoing Services / Special Projects

    Federal Reporting

    Hotline Investigations / Reporting

    Performance ManagementManager

    Title IV-E Stipend Specialist

    Hotline QA

    QA Specialist / System Admin

    Federal Liaison

    Hotline /DV /IL /HT

    Investigations / Data Analytics / FSFN Adoption

    Investigations / Child Fatalities

    Case Management / CCL

    Special Analyses / All Areas

    QI/QA Manager

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 13

    Due to limited funding, the OCW implemented the ROA Program Plans recommended functional organizational changes with the exception of the ROA Program Implementation and enhanced training functions described below.

    The temporary ROA Program Implementation function was to include approximately seven program management staff responsible for and dedicated to the development of the ROA Program and its related processes, including the management of initiatives within the Department and across key stakeholders (See Initiative 1). Because the ROA Program Implementation function was not included and staffed as part of the PQMUs initial organizational structure, the workload originally planned for this unit was reallocated across the other three areas of the PQMU. During the first year of implementation, the PQMU Director and the three PQMU managers formed the core ROA Program Implementation Team.

    The enhanced training function proposed in the ROA Program Plan was to be responsible for developing longer-term system-wide training strategies, coordinating with partners, such as FICW, to develop and deliver Program training across the Child Welfare Community, identifying training needs resulting from ROA Program activities, and identifying new and effective ways to enhance the skills of all ROA Program stakeholders (see Initiative 14).

    The ROA Program is intended to be a Department-University partnership; therefore, FICW plays a key role in the Program implementation, particularly in the areas of research, advanced data analysis, and predictive analytics. FICW was established in 2014 through F.S. 1004.615 as a consortium of public and private universities offering degrees in social work with the purpose of advancing the well-being of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development. FICWs Director serves as a member of the ROA Governance Committee, ROA Technical Advisory Panel, and ROA Program Implementation Team. While FICW has been actively involved in many Program implementation activities to date, similar to the Department, resource constraints in SFY 2015-16 limited their ability to execute the implementation activities originally intended for FICW in the ROA Program Plan.

    In addition to the OCW and FICW, the PQMU routinely calls on subject matter experts from key stakeholder groups, including the DCF Region Offices, CBC lead agencies, community service providers, and advocates, to perform activities in support of the Program implementation. For example, many of these stakeholders participated on workgroups to lead the design and documentation of the ROA Program processes.

    In SFY 2016-17, the PQMU Director will appoint a ROA Project Manager as a dedicated resource to drive implementation efforts with both internal and external stakeholders and perform the activities described in Initiative 1. The ROA Project Manager has been identified and will transition into this new role in early SFY 2016-17. In addition to the ROA Project Manager, a dedicated FICW resource with advanced research and data analysis skills will be collocated with the PQMU in SFY 2016-17 to assist with a number of the ROA implementation initiatives.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 14

    1.3 ROA PROGRAM GOVERNANCE

    The ROA Program governance structure provides a framework for how decisions will be made about the Programs implementation. Well-defined governance is critical to the Programs success in that it establishes the roles and responsibilities of all parties involved in the implementation and determines, in advance, clear decision-making processes and authority.

    The ROA Program governance structure includes the following components:

    ROA Program Implementation Team, which performs the activities necessary to implement the ROA Program and is currently comprised of the PQMU managers and staff and representatives from the OCW, DCF Region Offices, CBC lead agencies, community providers, university schools of social work (FICW and University of South Florida (USF)), and child welfare advocates and subject matter experts

    PQMU Director, who oversees, coordinates, and manages the day-to-day activities of the ROA Program Implementation Team with the support of a ROA Program Implementation function

    ROA Program Implementation function, which will be led by the ROA Project Manager to assist the PQMU Director with planning, coordinating, and managing ROA Program implementation activities

    FICW, which was established by F.S. 1004.615 for the purpose advancing the well-being of children and families by improving the performance of child protection and child welfare services through research, policy analysis, evaluation, and leadership development, and as described in the ROA Program Plan, plays an integral role in advising the Department on the planning and implementation of the ROA Program

    ROA Technical Advisory Panel, which was established by F.S. 409.997 to advise the Department during the planning and implementation of the ROA Program, and consistent with F.S. 409.997, includes representatives from FICW, the Senate, the House of Representatives, CBC lead agencies, family representatives, child welfare advocates, CBC providers, other contracted providers, community alliances, and family representatives, as well as the Department and Childrens Legal Services (CLS)

    ROA Governance Committee, which serves as the decision-making body for the implementation and operation of the ROA Program and, as recommended in the ROA Program Plan, is currently comprised of representatives from the Department, Sheriffs Offices, CBC lead agencies, FICW, child welfare advocates, CLS, and the court system

    The ROA Program governance structure is depicted graphically in Exhibit 3: ROA Program Implementation Governance.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 15

    Exhibit 3: ROA Program Implementation Governance

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 16

    1.3.1 ROLES AND RESPONSIBILITIES

    The roles and responsibilities of the key ROA Program governance structure are described in Exhibit 4: ROA Program Governance Roles and Responsibilities.

    ROLE RESPONSIBILITIES

    ROA Governance Committee

    Represents the Child Welfare Community stakeholders Provides leadership and serves as the decision-making body for the implementation of

    the Program Sets Program strategies and priorities Provides authority for use of resources to implement defined strategies

    PQMU Director

    Oversees the Program implementation, including but not limited to: staffing, assignments, stakeholder communications, schedule management, status reporting, risk and issue management and mitigation/resolution

    Coordinates Program implementation activities across Program stakeholder groups Provides overall leadership and guidance in support of the Program implementation Secures funding and resources for the Program implementation

    ROA Program Implementation (Project Management)

    To be led by a dedicated ROA Project Manager beginning in SFY 2016-17 Supports ROA Program Sponsor/Director Establishes and maintains the Program implementation schedule Implements and manages project management processes Assists in coordination and support of implementation activities

    ROA Implementation Team

    Performs activities to implement the ROA Program according to the Program implementation schedule

    Reports progress to the ROA Program Director

    FICW Advises the ROA Program Director and ROA Implementation Team on implementation

    activities and resources Recommends improvements to proposed Program processes and procedures to ensure

    the implemented Program is based on best practice and meets goals and objectives

    ROA Technical Advisory Panel

    Represents the Child Welfare Community stakeholders Provides technical assistance and guidance throughout the ROA Program

    implementation Assists in the establishment and assessment of ROA Program implementation

    benchmarks and milestones Monitors progress toward the achievement of ROA Program implementation

    benchmarks and milestones

    Exhibit 4: ROA Program Governance Roles and Responsibilities

    In November 2015, the PQMU reconvened the ROA Technical Advisory Panel to advise on Program implementation activities. In March 2016, the PQMU established the members of the ROA Governance Committee and held an initial meeting date on April 7, 2016.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 17

    SECTION 2 PROGRAM IMPLEMENTATION STATUS

    Each year, the DCF Secretary, ROA Governance Committee, and the OCW establish a set of objectives for the ROA Program which serves to guide the PQMUs implementation efforts based on the original ROA Program initiatives and available resources. As detailed below in Section 2.1, the PQMU has worked diligently over the past year towards completing the planned SFY 2015-16 Program implementation activities which spanned across ten of the fifteen ROA Program initiatives. As described in Section 1.1, many of the implementation initiatives require ongoing effort over the course of the multi-year Program implementation. In SFY 2015-16 activities focused on building the infrastructure necessary to support further implementation tasks as well as the successful operation of the ROA Program. In the upcoming year, the Departments work to establish these foundational elements will continue. Section 2.4 presents an overview of the SFY 2016-17 ROA Program objectives and planned implementation activities.

    2.1 STATUS OF ROA PROGRAM IMPLEMENTATION (SFY 2015-16)

    The current status of the ROA implementation initiatives is presented in Exhibit 5: Status of ROA Program Implementation Initiatives.

    The information provided includes:

    Status: Not Yet Started: Work on the activity has not started

    In Progress: Work on the activity has started but is not complete

    Complete: Work on the activity is complete

    Ongoing: Work on the activity has started and will continue for the duration of the implementation

    Actual Completion: For completed activities, the date activity was completed Estimated Completion: For activities not yet started, in progress, or ongoing, the

    estimated completion date

    Comments: Additional information regarding the status of the implementation activity

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 18

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Initiative 1: ROA Program Implementation Project Team: In SFY 2015-16, these activities were supported by North Highland under the ROA Program and Data Analytics contract.

    Establish Results-Oriented Accountability Implementation Project Team

    Complete July 2016 n/a

    In SFY 2015-16, the OCW was reorganized to create the PQMU, referencing the organization structure described in the ROA Program Plan. However, the PQMU structure does not include the recommended ROA Implementation Project Team with dedicated project management resources.

    In SFY 2016-17, a dedicated ROA Project Manager position will be established within the PQMU to lead the planning and execution of ROA Program implementation activities.

    Create Program Management Processes Complete June 2016 n/a

    In SFY 2015-16, key project management plans/processes/tools were developed for schedule, communications, and risk management.

    The ROA Project Manager will be responsible for executing and maintaining the project management plans/processes/tools for the duration of the ROA Program implementation.

    Create Program Governance Processes / Procedures

    Complete April 2016 n/a

    In SFY 2015-16, the PQMU established ROA Program governance processes and procedures. The ROA Technical Advisory Panel was reconvened in November 2015 and met in November 2015, February 2016, March 2016, and June 2016. The Technical Advisory Panel will continue to meet at least quarterly for the duration of the Program implementation. The ROA Governance Committee membership has been established, and the first meeting was held on April 7, 2016. The ROA Governance Committee will continue to meet quarterly over the life of the Program. Pursuant to Section 409.997(2)(g), F.S., the Department must submit an annual ROA performance report to the Governor and Florida Legislature. The report summarizes performance across key safety, permanency, and well-being indicators. The OCW has submitted the annual report for the past two years, and the report covering the period of SFY 2015-16 is currently being prepared for submission on or before October 1, 2016.

    The ROA Project Manager will be responsible for executing and maintaining ROA Program governance processes and procedures for the duration of the ROA Program implementation.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 19

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Develop Implementation Master Schedule Ongoing n/a SFY 2019-20

    This is an ongoing annual task completed by the ROA Project Manager for the duration of the ROA Program implementation.

    The Master Implementation Schedule for SFY 2015-16 was completed in April 2016, and a draft schedule has been prepared for SFY 2016-17 based on the ROA Program objectives. As additional task-level detail becomes known for SFY 2016-17, the ROA Project Manager will update the schedule accordingly.

    Engage Project Teams and Execute Prioritized Initiatives

    Ongoing n/a SFY 2019-20

    In SFY 2015-16, five workgroups were established to design and document ROA Program processes to prepare for the implementation of the processes in SFY 2016-17. The PQMU engaged both internal departmental stakeholders as well as external Child Welfare Community stakeholders and advisors to serve on the workgroups. The workgroups conducted regular meetings from late February through May. The final draft ROA process documentation was presented to the ROA Technical Advisory Panel on June 28, 2016.

    Manage and Respond to Risks Affecting Program Implementation

    Ongoing n/a SFY 2019-20

    In SFY 2015-16, ROA Program implementation risks were managed by the North Highland Project Manager. In SFY 2016-17, ROA Program implementation risks will be managed by the PQMUs ROA Project Manager.

    Initiative 2: Measure Development and Validation

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 20

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Identify ROA Measures Ongoing n/a SFY 2019-20

    In SFY 2014-15, a proposed set of initial measures for the ROA Program was provided in the ROA Program Plan. These measures were developed by a workgroup of Child Welfare Community stakeholders and primarily apply to the Department, CBC lead agencies and providers, and Sheriffs Offices. The measures include several of the existing federal outcome measures, CBC Scorecard measures, CPI Scorecard measures, and Priority of Effort (POE) measures.

    In SFY 2015-16, DCF and other key ROA Program stakeholders developed and documented a process to identify and validate ROA Program outcome measures. In SFY 2016-17, an Outcome Measures Workgroup comprised of representatives from the Child Welfare Community stakeholder groups will be established to oversee this process and the development of a validation methodology. The Outcome Measures Workgroup will review the proposed set of initial measures included in the ROA Program Plan and make updates as necessary to finalize the initial set of measures.

    Once the initial set of measures is finalized, validated, and deployed, additional measures will be identified and validated for other Child Welfare Community stakeholder groups, such as the judicial system, state partner agencies, etc., throughout the Program implementation.

    Confirm Measure Validity and Identify and Address Unintended Consequences Through Research, Data Analysis, and Logic Models

    Not Yet Started n/a SFY 2019-20

    Develop Calculation Algorithms

    Not Yet Started n/a SFY 2019-20

    Set Performance Baselines and Targets

    Not Yet Started n/a SFY 2019-20

    Pilot Measures and Make Necessary Adjustments Based on Results

    Not Yet Started n/a SFY 2019-20

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 21

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Update Relevant Documentation

    Not Yet Started n/a SFY 2019-20

    Initiative 3: Master Data Management: In SFY 2015-16, these activities were completed by North Highland under the ROA Program and Data Analytics contract.

    Define Master Data Management Procedures and Documentation

    In Progress n/a SFY 2016-17

    In SFY 2015-16, North Highland developed a Data Governance and Strategy report for the OCW which included a current state assessment and the development of a future state, recommended model, and an implementation roadmap. The Data Governance and Strategy report was approved on July 21, 2016.

    Establish Data Sharing Agreements with Stakeholder Communities

    Ongoing n/a SFY 2019-20 In SFY 2015-16, DOH IRB approval was received (January 2016) and Vital Statistics and Healthy Families Data was provided for advanced analytics (February 2016).

    Coordinate Updates to Data Sharing Agreements, As Needed

    Not Yet Started n/a SFY 2019-20

    Implement Master Data Management Across the Child Welfare Community

    Not Yet Started n/a SFY 2019-20

    Initiative 4: Establishment of Data Lab and Tools

    Install and Configure Analytics Software TBD n/a TBD

    This initiative was not completed in SFY 2015-16 as planned. It remains in the planning phase due to limited funding and DCF/FICW capacity to establish a data lab at this time. The PQMU is currently in the process of exploring other data lab solutions, such as a third party data lab. If it is decided to use a third party data lab, the tasks to implement Initiative 4 should be revised accordingly.

    Note: In SFY 2015-16, the OCW invested over $70,000 in IBM SPSS software to provide PQMU performance analysts with an advanced analytics tool. The staff received basic training on this application and additional training is planned for SFY 2016-17.

    Test Software Installation and Configuration TBD n/a TBD

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 22

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Conduct Knowledge Transfer TBD n/a TBD

    Initiative 5: Data System Updates for Initial Measurement Gaps

    Resolve Gaps Between the Initial Set of ROA Outcome Measures and the Availability of FSFN Data Required to Calculate the Measures

    Not Yet Started n/a SFY 2017-18

    Initiative 5 is closely related to Initiative 2. Necessary data system updates to address identified gaps for the initial set of ROA measures will be identified by the PQMU and evaluated by the ROA Outcome Measures Workgroup in SFY 2016-17, as the initial set of ROA Program measures is selected. Activities to address data system updates are included in the ROA Outcome Measure Identification and Validation process documentation.

    Initiative 6: Accountability Reports

    Develop Report Requirements Ongoing n/a SFY 2019-20

    Activities to develop and enhance ROA Program reporting will continue throughout the Program implementation as new measures are identified, validated, and incorporated for ROA Program stakeholder groups.

    In SFY 2015-16, dashboards and reports were created for existing measures, such as federal outcome measures and CBC Scorecard measures.

    Design Reports for Stakeholder Review and Approval

    Ongoing n/a SFY 2019-20 See comment above.

    Develop Report Extracts, Data Transformation, and Presentation Mechanisms

    Ongoing n/a SFY 2019-20 See comment above.

    Test Reports Ongoing n/a SFY 2019-20 See comment above.

    Develop and Conduct End User Training

    Not Yet Started n/a SFY 2019-20

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 23

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Deploy Reports Ongoing n/a SFY 2019-20

    In SFY 2015-16, dashboards and reports were created for many existing measures, including the federal outcome measures and CBC Scorecard measures. The CBC Scorecard and federal outcome measure dashboards were deployed to Program stakeholders and the general public via the DCF website and Tableau Public in May 2016 and July 2016, respectively. An additional 24 dashboards were developed and published to the DCF intranet for internal use.

    Additionally, in SFY 2015-16, the PQMU developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

    Conduct Ongoing Annual Updates (Assessment and Update of ROA Dashboard Reports)

    Not Yet Started n/a SFY 2019-20

    Initiative 7: Quality Assurance/Compliance Resource Analysis

    Identify Internal and External Units Conducting QA/QI and Contract Compliance Activities (Audits)

    Not Yet Started n/a SFY 2017-18

    Assess Capacity to Conduct Case Reviews and Other QI Activities

    Not Yet Started n/a SFY 2017-18

    Assess External Capacity/Cost to Conduct Contract Compliance Reviews (Audits)

    Not Yet Started n/a SFY 2017-18

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 24

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Develop Recommendations Regarding QI Staffing

    Not Yet Started n/a SFY 2017-18

    Initiative 8: Quality Improvement Organization

    Assess QI Needs Associated with the Program

    Not Yet Started n/a SFY 2017-18

    Develop / Modify QI Program Plan and Procedures, To Include ROA Functions

    Not Yet Started n/a SFY 2017-18

    Develop QI Staffing Plan Not Yet Started n/a SFY 2017-18

    Obtain QI Staff to Support Program Operations, Research, Evaluation, and QI Functions

    Not Yet Started n/a SFY 2017-18

    Develop and Deliver QI/Learning Organization Training for Managers, Line Staff, and Stakeholders

    Not Yet Started n/a SFY 2017-18

    Initiative 9: Results-Oriented Accountability Reporting System

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 25

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Incorporate Stakeholder Data and Screens Into ROA Reporting System

    Ongoing n/a SFY 2019-20

    Activities to incorporate new data and report screens into the ROA Reporting System will continue throughout the Program implementation as new measures are identified, validated, and incorporated for ROA Program stakeholder groups.

    In SFY 2015-16, DCF identified Tableau as its Department-wide dashboard and reporting tool and has purchased the software licensing and support and established a Data Reporting and Publishing Forum to guide and share information about the use of Tableau across the Department. Tableau is being used by the DRU to create dashboards for existing measures, such as the federal outcome measures and CBC Scorecard measures. At this time, the PQMU plans to use the DCF website and Tableau Public as the ROA Reporting System.

    In addition to the dashboards created, the PQMU also developed a Child Welfare Key Indicators Report which is distributed monthly to child welfare partners. The report presents a number of measures the Department has identified as critical to assessing the overall health of the child welfare system. The PQMU established a monthly process to review the Key Indicators Report with DCF management to discuss trends and prioritize items for deeper analysis and further action.

    Complete Typical Development Life Cycle Ongoing n/a SFY 2019-20

    CBC Scorecard and federal outcome measure dashboards were deployed to Program stakeholders and the general public via the DCF website and Tableau Public in May 2016 and July 2016, respectively. An additional 24 dashboards were developed and published to the DCF intranet for internal use.

    Initiative 10: Child Welfare Community Data

    Complete FSFN System Adoption Initiative In Progress n/a SFY 2016-17

    In SFY 2015-16, the FSFN System Adoption project team conducted 14 CBC site visits. The remaining four site visits are scheduled for early SFY 2016-17. The team worked closely with the CBCs to assess their FSFN utilization and develop plans to address identified gaps. This work and the implementation of the plans will continue into SFY 2016-17.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 26

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Map Data Sources to ROA Target Fields

    Not Yet Started n/a SFY 2019-20

    Activities to acquire and incorporate new data for ROA Program reporting, monitoring, and analysis will continue throughout the Program implementation as new measures are identified for ROA Program stakeholder groups. Additionally, assessing and enhancing the quality of Program data will be an ongoing task.

    Provide Data Details for Data Sharing Agreements

    Not Yet Started n/a SFY 2019-20 See comment above.

    Develop Interfaces and Data Extraction and Transformation Tools

    Not Yet Started n/a SFY 2019-20 See comment above.

    Configure Data Cleansing Procedures and Tools

    Not Yet Started n/a SFY 2019-20 See comment above.

    Initiative 11: Institutional Review Policy Update Facilitate Review of IRB Processes Employed By FICW and Affiliated Institutions

    Not Yet Started n/a TBD

    The Department should confirm whether this activity is required for ROA Program implementation.

    Determine the Need for Updates to DCF or FICW Policy To Ensure All Federal and Other Requirements Are Met

    In Progress n/a SFY 2016-17

    In May 2016, DCF transitioned the responsibility for the IRB process from the Division of Substance Abuse and Mental Health to the OCW. The OCW has begun to review the Departments existing IRB processes, procedures, and requirements and will update as necessary to support the ROA Program. It is likely DCF will utilize an external IRB for implementation activities requiring IRB review and approval in SFY 2016-17.

    Develop Program-Specific Procedures for Engaging the IRB Process

    In Progress n/a SFY 2016-17

    In SFY 2015-16, DCF and FICW designed a preliminary high-level IRB sub-process. Once DCF has finalized its IRB process, procedures, and requirements, the process documentation should be updated as necessary.

    Initiative 12: Research Standards

    Research and Develop Levels of Evidence Construct Specific to Florida and Child Welfare

    In Progress n/a SFY 2016-17

    In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes, including preliminary research review standards. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 27

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Research, Test, and Implement a Meta-Analysis Protocol for Research Reviews Requiring Meta-Analysis of Large Numbers of Target Studies

    Not Yet Started n/a SFY 2016-17

    Develop a Process for Estimating Research Budgets

    Not Yet Started n/a SFY 2016-17

    Initiative 13: Pilot Study Standards

    Develop Pilot Study Procedures In Progress n/a SFY 2016-17

    In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes, including preliminary pilot study procedures. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.

    Assess and Update CBC Contracts to Allow for Pilots

    Not Yet Started n/a SFY 2017-18

    Create Standards for Project Management of Pilot Studies

    In Progress n/a SFY 2016-17

    In SFY 2015-16, ROA Process workgroups were established to design and document ROA program processes. Preliminary pilot study procedures are incorporated in this process documentation, including project management of pilot studies. This activity should be revisited at a later stage of the implementation to review the preliminary documentation and update the pilot study process, as necessary.

    Create Protocols for Staffing Pilots

    Not Yet Started SFY 2016-17

    Initiative 14: Research and Evidence Informed Practice Training Development

    Conduct a Training Needs Assessment

    Not Yet Started n/a SFY 2017-18

    Design Training / Strategy

    Not Yet Started n/a SFY 2017-18

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 28

    INITIATIVE/ACTIVITY STATUS ACTUAL

    COMPLETION ESTIMATED

    COMPLETION COMMENTS

    Develop Training Materials

    Not Yet Started n/a SFY 2017-18

    Implement Training Materials Through FICW

    Not Yet Started n/a SFY 2017-18

    Conduct Training Evaluation to Determine Effectiveness

    Not Yet Started n/a SFY 2017-18

    Integrate Course Into New Employee Curriculum

    Not Yet Started n/a SFY 2017-18

    Initiative 15: Results-Oriented Accountability FICW Support

    Coordinate Database Administration

    Not Yet Started n/a TBD

    This activity is related to Initiative 4. If the decision is made to use a third party data lab, the tasks to implement Initiative 15 should be revised accordingly.

    Perform Data Analysis Not Yet Started n/a TBD See comment above.

    Perform Data Extracts, Transformation, and Loads

    Not Yet Started n/a TBD See comment above.

    Perform Program Research

    Not Yet Started n/a SFY 2019-20

    Advise ROA Program Implementation Team Ongoing n/a SFY 2019-20

    In SFY 2015-16, FICWs Director served as a member of the ROA Governance Committee, ROA Technical Advisory Panel, and ROA Program Implementation Team. In SFY 2016-17, a dedicated FICW resource with advanced research and data analysis skills will be collocated with the PQMU to assist with a number of the ROA implementation initiatives.

    Develop ROA Policy, Strategic Design, and Thought Leadership

    Ongoing n/a SFY 2019-20 See comment above.

    Support ROA Implementation and Training

    Ongoing n/a SFY 2019-20 See comment above.

    Exhibit 5: Status of ROA Program Implementation Initiatives

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 29

    Schedule delays or uncertainties which impacted the SFY 2015-16 ROA Program implementation activities are discussed below:

    The implementation schedule did not include time and resources for the validation of CBC Scorecard measures which were initially identified for deployment as ROA Program measures in SFY 2015-16, along with the federal outcome measures. While the federal measures do not require validation, validation is necessary for the CBC Scorecard measures if they are ultimately selected as ROA measures.

    In order to establish the ROA Program measures (Initiative 2: Measure Development and Validation), the PQMU will form an Outcome Measures Workgroup in SFY 2016-17, including representatives from key stakeholder groups. Over the next fiscal year, the Outcome Measures Workgroup will be charged with evaluating and identifying an initial set of ROA measures. As a first step, the Outcome Measures Workgroup will review the measures originally proposed in the ROA Program Plan and determine whether to include these measures, and if so, which measures require validation. Depending on the measure, the validation process could require significant time and/or resources which will need to be factored into the SFY 2016-17 implementation schedule.

    The development of ROA dashboards identified for deployment in SFY 2015-16 (Initiative 6: Accountability Reports) was delayed due to hardware resource constraints, a limited number of PQMU staff with the Tableau software skill set(s) required to develop the dashboards, and the time needed for additional staff to develop and effectively apply this new skill set through formal and on-the-job training. In SFY 2016-17, advanced Tableau training for PQMU staff is planned, and additional PQMU staff are being assigned to assist in the development of the dashboards to prevent delays.

    The initiative to establish data lab services for the ROA Program is being revisited and has not been finalized to date. A data lab is an entity consisting of a properly supported data environment (physical or virtual), an analytical environment which includes basic and advanced analytical software, a team of one or more business and advanced data analysts, and appropriate business processes to receive and execute requests for data analysis. Its primary function is to explore data using quantitative methodologies to understand the data and answer business, practice, and research questions. A data lab may be an internal entity or an external entity, and may be permanent or project-by-project (the latter generally implies contracting the function to a third party).

    The ROA Program Plan includes initiatives (Initiative 4: Establishment of Data Lab and Tools and Initiative 15: ROA FICW Support) and estimated costs to establish and operate a data lab within FICW to support the Program. However, due to funding constraints, a data lab could not be stood up as planned. A delay in establishing data lab services for the Program could impact activities to validate ROA Program measures and perform advanced data analysis in support of the ROA Program in SFY 2016-17. In order to address this potential issue, the PQMU is working to identify alternative solutions involving partnering with organizations with existing data labs. In March 2016, representatives from the PQMU and FICW conducted a site visit and

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 30

    discussions with USF to further explore these options and minimize the impact to the implementation timeline. In addition, the OCW expended over $70,000 on IBM SPSS software to provide PQMU performance analysts with the tools needed to support advanced analytics. The staff received basic training on this application and additional training is planned for SFY 2016-17.

    2.2 ROA PROGRAM IMPLEMENTATION RISKS

    A risk is a potential situation or event which could have an adverse impact on the success of the ROA Program implementation. Risks should be identified and mitigated to the greatest extent possible to keep the Program implementation on track. Exhibit 6: ROA Program Implementation Risks identifies two risks which could adversely affect the success of the ROA Program implementation.

    NO. RISK DESCRIPTION

    1 SFY 2016-17 funding received for the ROA Program implementation is $500,000. This constraint could impact the quality and timeliness of the implementation objectives and activities currently planned for SFY 2016-17.

    2 In addition to the ROA Program, the PQMU supports many other data, data analysis, and reporting needs of the OCW, including providing operational data, media requests, and ad hoc reporting. In the course of performing these ongoing duties, PQMU staff routinely calculate and report on a large number of outcome and process measures. A key principle of the ROA Program is to utilize a set of outcome measures that contains as few measures as possible (Testa and Poertner). To ensure the Program is implemented according to its intended design, the PQMU plans to establish an Outcome Measures Workgroup in SFY 2016-17 to evaluate and identify a set of ROA measures which align closely with the outcomes of safety, permanency, and well-being. Some of these measures may be existing measures, while others may be new measures. The establishment of Outcome Measures Workgroup to oversee this process will mitigate the risk of too many measures being used for the ROA Program and will help to ensure outcome monitoring will be manageable and Program stakeholders will be able to effectively assess their performance.

    Exhibit 6: ROA Program Implementation Risks

    2.3 ROA PROGRAM IMPLEMENTATION ISSUES

    An issue is an existing situation or event negatively impacting the quality, resources, and/or timeliness of the ROA Program implementation. Issues must be identified and addressed to keep the Program implementation on track. Exhibit 7: ROA Program Implementation Issues below identifies two ROA Program implementation issues.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 31

    NO. ISSUE DESCRIPTION

    1 The Department receives annual funding of $500,000 to support ROA Program implementation activities. The initial year of Program implementation described in the ROA Program Plan included between 46 and 55 FTEs, including resources from DCF, FICW, CBC lead agencies, contracted providers, and external vendors involved in implementation activities. Currently, it is estimated only about five to ten percent of the FTEs described in the ROA Program Plan were allocated to the implementation activities for SFY 2015-16. Due to these constraints, the reorganization of the OCW to create the PQMU did not provide for the recommended temporary ROA Program Implementation Team in the organization structure. As a result, in SFY 2015-16 no OCW resources were dedicated solely to the ROA Program implementation, and the work planned for this organizational unit was redistributed to the PQMU managers and staff. Although the PMQU managers and staff are the core members of the ROA Program Implementation Team, they have very limited availability to perform ROA Program implementation activities as most of their time is consumed with day-to-day job responsibilities. In response to this issue, the PQMU Director has identified a full-time ROA Project Manager to oversee the planning and execution of ROA Program implementation activities. Also, FICW has secured a resource with advanced data analysis and research skills to be collocated with the PQMU. While the number of FTEs available to the Department for implementation activities remains less than originally planned, these resources will greatly enhance the ROA implementation efforts in the coming year.

    2 The ROA Program implementation has been impacted by funding and resource limitations. As a result, the alignment of the ROA Program Plan and the actual implementation efforts need to be reviewed and realigned as necessary to ensure the quality of the Program implementation is not affected and there is a clear path to follow moving forward.

    Exhibit 7: ROA Program Implementation Issues

    2.4 LOOKING FORWARD PLANNED SFY 2016-17 ROA IMPLEMENTATION ACTIVITIES

    Under the leadership and guidance of the DCF Secretary and ROA Governance Committee, the OCW has identified the ROA Program objectives and implementation activities for SFY 2016-17. These tasks represent a plan to continue to move forward with the Program implementation in an effective and efficient manner, with a particular focus on the following six objectives:

    Objective 1: Partner with FICW and the National Capacity Building Center to develop an evaluation design establishing successful implementation criteria, implementation measures, and overall effectiveness for the ROA Program

    Objective 2: Deploy 37 child welfare metrics (outcomes and drivers) to the public facing dashboard environment

    Objective 3: Operationalize Year 1 foundational components of the OCW Data Governance and Master Data Management Strategy using a data analysis project as proof of concept

    Objective 4: Improve performance of Child and Family Services Review (CFSR) Program outcomes (data measures/systematic factors/file reviews) to meet federal targets utilizing newly developed ROA processes

    Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 32

    Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the state

    Exhibit 8: ROA Program Implementation Activities (SFY 2016-17) outlines the planned activities and ROA Program Plan initiative(s) associated with each ROA Program objective.

    PLANNED ACTIVITIES ROA PROGRAM PLAN INITIATIVE

    Objective 1: Partner with FICW and the National Capacity Building Center to develop an evaluation design establishing successful implementation criteria, implementation measures, and overall effectiveness for the ROA Program

    Develop ROA evaluation design Gather and analyze evaluation data Produce annual report

    Initiative 1: ROA Implementation Project Team Initiative 13: Pilot Study Standards Initiative 15: FICW Support

    Objective 2: Deploy 37 child welfare metrics (outcomes and drivers) to the public facing dashboard environment

    Establish the average build cycle for each dashboard

    Develop a dashboard build schedule Establish the statewide Outcome Measures

    Workgroup Finalize the outcome validation methodology Validate five ROA outcome measures utilizing the

    validation methodology selected

    Initiative 1: ROA Implementation Project Team Initiative 2: Measure Development and Validation Initiative 5: Data System Updates for Initial

    Measurement Gaps Initiative 6: Accountability Reports Initiative 9: ROA Reporting System

    Objective 3: Operationalize Year 1 foundational components of the OCW Data Governance and Master Data Management Strategy using a data analysis project as proof of concept

    Procure vendor contract (Objective 2 and 5) Document initial DG/MDM OCW process Evaluate DG/MDM processes utilized in pilot

    analytics project to refine and finalize DG/MDM processes to be integrated into formal ROA process guide and process maps

    Initiative 1: ROA Implementation Project Team Initiative 3: Master Data Management

    Objective 4: Improve performance of CFSR outcomes (data measures/systematic factors/file reviews) to meet federal targets utilizing newly developed ROA processes Complete data analysis of three of the seven

    federal measures that will be included in the federal Program Improvement Plan (PIP)

    Complete research review phase as outlined in the ROA processes to include feasibility study and evaluation design

    Present final results/recommendations to ROA Governance Committee for intervention implementation

    Initiative 1: ROA Implementation Project Team Initiative 12: Research Standards Initiative 13: Pilot Study Standards

    Objective 5: Identify and validate key drivers to service delivery, financial stability, and outcomes to be monitored monthly utilizing individual CBC dashboard views Finalize research questions Gather data and complete data analysis Validate outcomes/drivers

    Initiative 1: ROA Implementation Project Team Initiative 2: Measure Development and Validation Initiative 5: Data System Updates for Initial

    Measurement Gaps

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 33

    PLANNED ACTIVITIES ROA PROGRAM PLAN INITIATIVE

    Design individual CBC dashboard views for development

    Produce final report

    Initiative 6: Accountability Reports Initiative 9: ROA Reporting System Initiative 10: Child Welfare Community Data

    Objective 6: Develop a statewide data analytics network that inventories, shares, and leverages insights from the various data analytics projects underway in the state Establish Statewide Analytics Committee and hold

    kick-off meeting Inventory/track statewide analytics projects Train members on IRB changes, process, and

    requirements

    Initiative 1: ROA Implementation Project Team Initiative 8: Quality Improvement Organization

    Exhibit 8: ROA Program Implementation Activities (SFY 2016-17)

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 34

    SECTION 3 RECOMMENDATIONS

    In SFY 2015-16, DCF made strong progress towards the multi-year ROA Program implementation. The Department plans to continue to work diligently on the Programs implementation in SFY 2016-17, focusing on developing metrics and reporting, strengthening its analytical capabilities, improving data quality, implementing data governance and management strategies, and increasing stakeholder awareness and understanding of ROA and the cycle of accountability. Below, North Highland provides recommendations for the Departments consideration to support its efforts to achieve the SFY 2016-17 ROA Program objectives and successfully complete planned implementation activities.

    Note: Additional detail regarding recommendations for data governance, master data management, and advanced analytics can be found in the following North Highland deliverables:

    Deliverable 4: Data Governance and Strategy

    Deliverable 6: Updated FSFN Dictionary

    Deliverable 7: Foundations for Analytics

    Objective 3: Operationalize Year 1 foundational components of the OCW Data Governance and Master Data Management Strategy using a data analysis project as proof of concept

    Observation: The Advanced Data Analytics project is a broad initiative to expand systematically the use of data analytics and predictive analysis techniques to improve outcomes for children and families. The ROA Program is focused on the research, development, implementation, and ongoing measurement of the success of interventions or programs affecting the safety, permanency, and well-being of children and families. Both are complementary efforts supporting the same objectives. OCW has an increasing dependency on the quality of internal data and the ability to access external data in a timely manner and ascertain the quality of the data once it is received. A foundational strategy of data governance, data management, and data acquisition processes will increase efficiency and reduce expenses for data-related efforts.

    Recommendation: A number of data governance projects/initiatives could start and potentially be completed in SFY 2016-17, depending on the capacity of available resources. The majority of this effort should focus on planning and applying a proof of concept approach to the data governance and management strategies introduced in SFY 2015-16. To derive the most benefit, the OCW should collaborate with DCF executive management and leadership from the other program areas, such as Economic Self-Sufficiency and Substance Abuse and Mental Health. Some projects are already underway (for example, the Client Data Link System) that would benefit from a formal data governance structure and organization.

  • Quarterly Progress Report on ROA Implementation - June 2016 Florida Department of Children and Families - Office of Child Welfare Page 35

    North Highland recommends the Department build support for formal data governance with executive management and program area leadership through establishing a:

    Well-executed communications str