Greater Columbia Behavioral Health
Division of Behavioral Health and Recovery External Quality Review Report
November 2013
Contract #0835-33311 Presented by
Acumentra Health 2020 SW Fourth Avenue, Suite 520 Portland, Oregon 97201-4960 Phone 503-279-0100 Fax 503-279-0190 DBHR-EQR-WA-13-9
Greater Columbia Behavioral Health
2013 External Quality Review Report
Performance Improvement Project Validation
Information Systems Capabilities Assessment
Encounter Data Validation
Clinical Record Review
November 2013
Presented to the Division of Behavioral Health and Recovery
Acumentra Health prepared this report under contract with the Division of Behavioral Health
and Recovery (Contract No. 0835-33311).
Director, State and Private Services .........Michael Cooper, RN, MN
Account Manager .....................................Jody Carson, RN, MSW, CPHQ
Project Manager–Monitoring ...................Laureen Oskochil, MPH, CHC, CPHQ
Project Manager–Validation ....................Amy Pfleiger, CISA
Project Coordinator ..................................Ricci Rimpau, RN, CHC, CPHQ
Mental Health QI Specialist .....................Nancy Siegel, PA-C, MPH
Mental Health QI Specialist .....................Erin Schwartz, PhD, LPC
Research Analyst .....................................Sara Hallvik, MPH
Information Systems Analyst ...................Christine Ryan
Writer/Editor ............................................Greg Martin
Production Assistant ................................Ellen Gehringer
2013 Greater Columbia Behavioral Health EQR Report: Table of Contents
i Acumentra Health November 2013
TABLE OF CONTENTS
Executive Summary .....................................................................................................................................1
Introduction ..................................................................................................................................................5
EQR activities ........................................................................................................................................5
Performance Improvement Project (PIP) Validation ...................................................................................6
PIP review procedures ...........................................................................................................................6
Review results for nonclinical PIP: Lowered Inpatient Readmission Rates in a High Risk
Population through the Development of Enhanced Communication with Inpatient Providers .............8
Review results for clinical PIP: Lowered PRISM Scores in a High Medical Risk Psychiatric
Inpatient Population Through the Integration of Health Information into Mental Health Records ....14
Compliance Review Follow-Up ................................................................................................................21
Information Systems Capabilities Assessment ..........................................................................................23
Summary of review results ..................................................................................................................24
ISCA Section 1: Data Processing Procedures and Personnel ..............................................................25
ISCA Section 2: Data Acquisition Capabilities ...................................................................................29
Encounter Data Validation .........................................................................................................................32
Greater Columbia Behavioral Health’s EDV procedure .....................................................................32
Validation results .................................................................................................................................34
Discussion and recommendations ........................................................................................................38
Clinical Record Review .............................................................................................................................39
Review results ......................................................................................................................................39
Discussion and recommendations ........................................................................................................39
Appendix A. PIP Validation and Scoring Protocol ................................................................................ A-1
Appendix B. ISCA Methodology ............................................................................................................B-1
Appendix C. Encounter Data Validation Procedures ..............................................................................C-1
2013 Greater Columbia Behavioral Health EQR Report: Index of Tables and Figures
ii Acumentra Health November 2013
INDEX OF TABLES AND FIGURES
Table 1. Standards for PIP validation ......................................................................................................7
Table 2. PIP scoring ranges......................................................................................................................7
Table 3. Results of compliance review follow-up for Greater Columbia Behavioral Health ................22
Table 4. Scoring scheme for ISCA standards ........................................................................................23
Table 5. Weighted average scores and ratings on ISCA sections ..........................................................24
Table 6. Results of 2013 electronic data checks ....................................................................................35
Table 7. Results of encounter data validation for Greater Columbia Behavioral Health ......................36
Table 8. Results of Golden Thread analysis for Greater Columbia Behavioral Health .........................37
Table 9. Results of clinical record review for Greater Columbia Behavioral Health ............................41
Figure 1. Summary results of Greater Columbia Behavioral Health PIP validation ................................2
Figure 2. Summary results of Greater Columbia Behavioral Health ISCA review ..................................3
Figure 3. Validation scores by standard for nonclinical PIP, Lowered Inpatient Readmission Rates
in a High Risk Population through the Development of Enhanced Communication with
Inpatient Providers .....................................................................................................................8
Figure 4. Validation scores by standard for clinical PIP, Lowered PRISM Scores in a High Medical
Risk Psychiatric Inpatient Population Through the Integration of Health Information into
Mental Health Records ............................................................................................................14
2013 Greater Columbia Behavioral Health EQR Report: Executive Summary
1 Acumentra Health November 2013
EXECUTIVE SUMMARY
The Division of Behavioral Health and Recovery
(DBHR) contracts with Acumentra Health to
perform an annual external quality review (EQR)
of managed mental health services for Medicaid
enrollees in Washington, in fulfillment of federal
requirements under 42 CFR §438.350.
This report summarizes the 2013 review of
Greater Columbia Behavioral Health (GCBH),
one of 11 regional support networks (RSNs) with
which DBHR contracts to deliver managed mental
health services. This year’s review includes:
evaluation of the RSN’s performance
improvement projects (PIPs)
follow-up review of the RSN’s compliance
with federal and state regulations and
contract provisions governing managed
care operations
an Information Systems Capabilities
Assessment (ISCA)
encounter data validation and a review of
clinical records
The reviews rate GCBH’s overall performance in
2012, identify strengths and opportunities for
improvement, and offer specific recommendations
to address deficiencies. The results summarized
below are presented in more detail in the main
body of the report.
PIP evaluation results
Because RSNs begin their PIPs at different times,
and because PIPs are typically multi-year projects,
the studies may be in different stages at the time
of the EQR evaluation. As ongoing projects, the
PIPs may not meet all standards the first year, but
a PIP is expected to achieve better scores as the
project progresses, eventually reaching full
compliance.
Acumentra Health reviewed one nonclinical and
one clinical PIP conducted by GCBH:
1. Nonclinical—Lowered Inpatient
Readmission Rates in a High Risk
Population through the Development of
Enhanced Communication with
Inpatient Providers: Feedback from local
stakeholders revealed concerns about
inpatient recidivism among children
enrolled with GCBH. Between October
2011, when GCBH established its in-house
Authorization Center, and December 2012,
GCBH calculated a 90-day readmission
rate of 28% for children. For this first-year
PIP, GCBH intends to implement the use of
the Child Inpatient Admission Review
questionnaire for each child authorized for
an inpatient stay. The purpose of the
questionnaire is to facilitate enhanced
communication between GCBH’s in-house
Authorization Center staff and inpatient
providers’ utilization personnel, with the
ultimate goal of reducing the readmission
rate for children.
2. Clinical—Lowered PRISM Scores in a
High Medical Risk Psychiatric
Inpatient Population Through the
Integration of Health Information into
Mental Health Records: This first-year
PIP focuses on integrating physical health
information into mental health treatment
plans as a means of addressing the under-
treatment of physical health needs. The
RSN provided no evidence as to the nature
and extent of this problem at the local
Medicaid level, and thus could not discuss
how its intervention strategy addresses
identified barriers or is expected to
improve the study indicator (risk scores
for mental health enrollees). It is also not
clear whether the selected indicator is the
most appropriate measure of integration of
physical health information into mental
health treatment plans.
2013 Greater Columbia Behavioral Health EQR Report: Executive Summary
2 Acumentra Health November 2013
53
43
0
30
60
90
Nonclinical Clinical
Sco
re
Figure 1. Summary results of Greater Columbia Behavioral Health PIP validation.
As shown in Figure 1, the nonclinical PIP scored
53 on a scale of 90, and the clinical PIP scored 43
out of 90, both earning a Partially Met rating.
GCBH presented little data to support the design
and implementation of the nonclinical PIP. The
RSN needs to support the selection of the study
topic and present more information regarding the
codes used to pull administrative data, procedures
for data validation, why GCBH chose the specific
intervention, and a plan for tracking and
monitoring the intervention.
With regard to the clinical PIP, GCBH did not
fully document its reasons for selecting the study
topic, indicator, and intervention. In its next
iteration, GCBH needs to focus on key PIP
concepts—relevance of the topic to the local
Medicaid population, validity of the study
indicator, and barrier analysis to support the
selection of the intervention.
Compliance follow-up results
In 2013, Acumentra Health reviewed GCBH’s
response to the specific findings of the 2012
compliance review for which DBHR required the
RSN to perform corrective action. The 2012
review addressed compliance with federal and
state standards in eight separate areas.
The follow-up review found that GCBH had
addressed all four findings from the 2012 review.
Notably, the RSN completed a new quality
management plan, rewrote several policies,
formalized its credentialing/recredentialing
process, and began using computerized auditing
tools to establish baselines and monitor for over-
and underutilization of services.
ISCA results
Acumentra Health examined GCBH’s
information systems and data processing and
reporting procedures to determine the extent to
which they supported the production of valid and
reliable state performance measures and the
capacity to manage care of RSN enrollees.
The ISCA found that GCBH partially met the
federal standards related to data processing
procedures and personnel, and fully met the
standards related to data acquisition capabilities.
The review revealed deficiencies with regard to
data security and eligibility verification.
Figure 2 depicts GCBH’s scores on individual
ISCA subsections. For additional detail,
including strengths and recommendations for
improvement, see the report section beginning
on page 23.
Substantially met
Fully met
Partially met
Not met
Minimally met
2013 Greater Columbia Behavioral Health EQR Report: Executive Summary
3 Acumentra Health November 2013
2.32.2
2.6
1.5
2.9 2.7
2.22.3
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Se
cti
on
sc
ore
Figure 2. Summary results of Greater Columbia Behavioral Health ISCA review.
Encounter data validation results
DBHR requires each RSN to conduct an annual
encounter data validation (EDV) to determine the
accuracy of encounter data submitted by network
providers. As an independent check of the RSNs’
results, Acumentra Health audited and verified
the EDV process for each RSN in 2013.
Electronic checks of the data elements making up
GCBH’s total outpatient and demographic data
generally found complete data in mandatory
fields, but 13.2% of records contained out-of-
range ethnicity values, and a small percentage of
records had a duplicate claim ID.
Acumentra Health’s review of 394 outpatient
encounter records revealed that 79.6% had a
service code that matched the service described in
the chart note. Provider type, service date and
location, and procedure code matched in 80% to
93% of encounters. Considering demographic
data, the chart information matched the state data
in 99.1% of records for first and last name, and
matched in all records for date of birth. Matching
rates were lower for ethnicity and language.
Golden Thread analysis
In conjunction with the EDV, Acumentra Health
reviewed clinical records to determine providers’
adherence to the “Golden Thread” of clinical
documentation, tying together the mental health
diagnosis, treatment plan, and progress notes.
Reviewers examined whether the assessment in
the enrollee’s chart substantiated the diagnosis,
whether the treatment plan was consistent with the
diagnosis, and whether progress notes addressed
goals and interventions in the treatment plan.
Assessments. Most mental health assessments
were well written for both children and adults.
They included documentation of the presenting
problem, medical necessity for treatment, clinical
formulation for diagnosis, and services.
Treatment plans. Although the majority of the
children’s treatment plans included interventions
and goals consistent with issues identified in the
assessment, only 67% of the adult treatment plans
included interventions, services, and measurable
goals that were consistent with the assessment.
Not met
Partially met
Fully met
2013 Greater Columbia Behavioral Health EQR Report: Executive Summary
4 Acumentra Health November 2013
GCBH needs to provide guidance to the
agencies to ensure that treatment plans
specify measurable goals, expected
outcomes, service(s) indicated, and
specific interventions consistent with the
enrollee’s assessment.
Progress notes. The progress notes for children
were consistent and well written. They typically
addressed interventions identified in the treatment
plan and the child’s progress toward meeting the
stated goals. However, as with the treatment
plans, the adult progress notes showed room for
improvement, as they did not consistently address
interventions identified in the treatment plan and
progress toward meeting the stated goals.
GCBH needs to provide direction to the
agencies to ensure that progress notes
clearly document the goals, treatment
interventions, enrollee’s response to the
interventions, and progress toward
meeting the goals.
Clinical record review results
Also in conjunction with the EDV, Acumentra
Health reviewed clinical records at four outpatient
provider agencies to assess compliance with the
mental healthcare criteria defined by DBHR. This
study focused on the degree to which the RSN’s
system of care adhered to the principles of the
Children’s Mental Health Redesign, including the
principles defined in the interim settlement of T.R.
v. Dreyfus, regarding uniform screening and
assessment of children with serious emotional
disturbances.
Assessments: Clinicians at GCBH’s provider
agencies did a good job of documenting the
child’s home environment and systems of support.
The percentage of applicable assessments that
addressed developmental and sensory impairment,
cultural and language issues, and justifying the
diagnosis all fell within the acceptable range.
However, nearly one-third of the assessments were
more than one year old.
Acumentra Health recommends updating
enrollee assessments at least annually to
document changes in the enrollee’s
functioning and life circumstances.
Treatment plans: Most treatment plans reflected
information included in the assessments. The
majority of treatment goals were based on the
children’s strengths. Most records documented
involvement of family members and support
systems in treatment. However, only 64% of the
treatment plans incorporated coordination with
other agencies into the treatment objectives.
GCBH should ensure that agencies
work with their clinicians to coordinate
care with other agencies involved in the
child’s life.
Nearly two-thirds of the treatment plans did not
include a multi-disciplinary team-based approach
to treatment.
GCBH needs to offer guidance to the
agencies to ensure that children’s
treatment includes a team-based
approach when necessary.
Progress notes: All progress notes demonstrated
that the child received unconditional treatment.
The majority of progress notes documented use of
strength-based interventions and activities as
identified in the treatment plans. Team-based
services and coordination with other agencies
were missing from many progress notes. Only
55% of progress notes indicated that services were
team-based and that care was coordinated with
other agencies and systems.
GCBH needs to ensure that services for
children are team-based and that care
coordination occurs. These services
must be documented in the progress
notes to reflect a multi-disciplinary
team-based approach.
2013 Greater Columbia Behavioral Health EQR Report: Introduction
5 Acumentra Health November 2013
INTRODUCTION
This report summarizes the results of the 2013
review of GCBH, a mental health RSN that serves
Medicaid recipients. Acumentra Health performed
the review in its capacity as DBHR’s External
Quality Review Organization (EQRO).
Currently, DBHR contracts with 11 RSNs to
deliver mental health services for Medicaid
enrollees through managed care. The RSNs, in
turn, contract with provider groups, including
community mental health programs and private
nonprofit agencies and hospitals, to deliver
treatment services. The RSNs must ensure that
services are delivered in a manner that complies
with legal, contractual, and regulatory standards
for effective care.
GCBH, headquartered in Kennewick, is a
government consortium providing public mental
health services for 10 counties and the Yakama
Nation in south central Washington. A citizen’s
advisory board advises the GCBH board of
directors, reviews and provides comments and/or
recommendations on plans and policies, and
serves on RSN workgroups and committees.
During 2012, GCBH had about 190,000 enrollees
in its service area.
EQR activities
42 CFR §438.358 specifies three mandatory
activities that the EQR must address in a manner
consistent with protocols established by the
Centers for Medicare & Medicaid Services
(CMS):
a review every three years of health plan
compliance with federal and state
regulations and contract provisions
regarding access to care, structure and
operation, and quality measurement and
improvement
annual validation of PIPs, a required
element of health plans’ quality
improvement (QI) programs
annual validation of performance
measures reported by plans or calculated
by the state, including an ISCA
Acumentra Health conducted the compliance
review for each RSN during 2011–2012, and
conducted the ISCA for each RSN and for
DBHR in 2013. In addition, Acumentra Health
conducted the PIP validation, an encounter data
validation, and a clinical record review for each
RSN in 2013. Together, these activities
addressed the following questions:
1. Does the RSN meet CMS regulatory
requirements?
2. Does the RSN meet the requirements of its
contract with the state?
3. Does the RSN monitor and oversee
contracted providers in their performance
of any delegated activities to ensure
regulatory and contractual compliance?
4. Does the RSN conduct the two required
PIPs, and are they valid?
5. Does the RSN’s information technology
infrastructure support the production and
reporting of valid and reliable performance
measures?
Review procedures for each activity were adapted
from the applicable CMS protocol:
EQR Protocol 1: Assessment of
Compliance with Medicaid Managed Care
Regulations. Version 2.0, September 2012
EQR Protocol 3: Validating Performance
Improvement Projects (PIPs). Version 2.0,
September 2012
Appendix V: Information Systems
Capabilities Assessment. September 2012
The scoring plan for each activity was adapted
from CMS guidelines, using a DBHR-approved
weighting system developed by Acumentra
Health.
2013 Greater Columbia Behavioral Health EQR Report: PIP Overview
6 Acumentra Health November 2013
PERFORMANCE IMPROVEMENT
PROJECT VALIDATION
Under 42 CFR §438.240(d), a managed care
organization that serves Medicaid enrollees must
have an ongoing program of PIPs that focus on
improving clinical care and nonclinical aspects of
service delivery. The PIPs enable the organization
to assess and improve the processes and, in turn,
the outcomes of care.
PIPs are validated each year as part of the EQR to
ensure that the projects are designed, conducted,
and reported according to accepted methods,
establishing confidence in the reported results.
The PIPs must include:
measurement of performance using
objective quality indicators
implementation of system interventions to
improve quality
evaluation and initiation of the
interventions
planning and initiation of activities for
increasing or sustaining improvement
Through repeated measurement of the selected
quality indicators, a PIP is expected to
demonstrate meaningful change in performance
relative to the performance observed during
baseline measurement.
Acumentra Health has validated the Washington
RSNs’ PIPs each year since 2008. Most RSNs
have carried their individual PIP topics forward
for at least several years, enabling Acumentra
Health to evaluate their progress toward achieving
sustained improvement.
Because RSNs begin their PIPs at different times,
the studies may be in different stages at the time
of the EQR evaluation. Some may be underway
but not yet complete; others may have progressed
to collecting baseline and remeasurement data;
still others may have progressed to multiple
remeasurements. The stage of the PIP at review
determines the level of analysis that Acumentra
Health applies.
PIP review procedures
Through document review and onsite interviews,
Acumentra Health reviews PIPs for these elements:
a written project plan with a study design,
an analysis plan, and a summary of results
a clear, concise statement of the topic being
studied, the specific questions the study is
designed to address, and the quantifiable
indicators that will answer those questions
a clear statement of the improvement
strategies, their impact on the study
question, and how that impact will be
assessed and measured
evidence that the intervention services and
materials are culturally and linguistically
appropriate, per the 2012 CMS protocol
an analysis plan that addresses project
objectives, defines indicators clearly,
specifies the population being studied,
identifies data sources and/or the data
collection procedure, and discusses the
methods for analyzing the data and
performing statistical tests
if applicable, a sampling methodology that
yields a representative sample
if the data collection involves clinical chart
review, a check on inter-rater reliability
use of validation procedures at the point of
data entry and within the database to verify
that data used for population-based analysis
are complete and accurate
a summary of results covering all data
collection and analysis, explaining
limitations in the data and methodologies
and discussing whether the intervention(s)
resulted in improvements
PIP scoring
The PIP scoring methodology, adapted from the
CMS protocol for this activity and approved by
DBHR, involves rating the RSN’s performance
on as many as 10 standards, listed in Table 1.
Appendix A defines in detail the specific criteria
used to evaluate performance.
2013 Greater Columbia Behavioral Health EQR Report: PIP Overview
7 Acumentra Health November 2013
Each standard has a potential score of 100 points
for full compliance. The total points earned for
each standard are weighted and combined to
determine an overall PIP score. The overall score
is weighted 90% for demonstrable improvement in
the first year (Standards 1–8) and 10% for
sustained improvement in later years (Standards
9–10). Thus, for a PIP that has completed one
remeasurement, the maximum overall project score
is 90 points. (Note: In years before 2012, the
maximum score for these PIPs was 80 points.) If
the PIP has progressed to at least a second
remeasurement, enabling reviewers to assess
sustained improvement, the maximum overall
project score is 100 points.
Table 1. Standards for PIP validation.
Demonstrable improvement
1 Selected study topic is relevant and prioritized
2 Study question is clearly defined
3 Study indicator is objective and measurable
4 Study population is clearly defined and, if a sample is used, appropriate methodology is used
5 Data collection process ensures valid and reliable data
6 Improvement strategy is designed to change performance based on the quality indicator
7 Data are analyzed and results interpreted according to generally accepted methods
8 Reported improvement represents “real” change
Sustained improvement
9 The RSN has documented additional or ongoing interventions or modifications
10 The RSN has sustained the documented improvement
Table 2 shows the compliance ratings and
associated scoring ranges for PIPs graded on the
90-point and 100-point scales. Appendix A
presents a sample scoring worksheet. Note: these
compliance rating ranges for the overall PIP score
are different from the ranges used in assessing
compliance for individual PIP standards; see
Appendix A.
Per the approved protocol, Acumentra Health
scores all PIPs according to the same criteria,
regardless of the stage of completion. As ongoing
multi-year QI projects, the PIPs may not meet all
standards the first year, but a PIP is expected to
achieve better scores as project activities progress,
eventually reaching full compliance.
Table 2. PIP scoring ranges.
Compliance rating Description
100-point scale
90-point scale
Fully met Meets or exceeds all requirements 80–100 72–90
Substantially met Meets essential requirements, has minor deficiencies 60–79 54–71
Partially met Meets essential requirements in most, but not all, areas 40–59 36–53
Minimally met Marginally meets requirements 20–39 18–35
Not met Does not meet essential requirements 0–19 0–17
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
8 Acumentra Health November 2013
90
100
95
60 60
35
0 00
20
40
60
80
100
1 2 3 4 5 6 7 8
Score
Standard
Review results for nonclinical PIP: Lowered Inpatient Readmission Rates in a High Risk Population through the Development of Enhanced Communication with Inpatient Providers
In response to the statewide mandate for RSNs to
provide services and treatment to children within
their communities, GCBH sought feedback from
local stakeholders about potential areas for
improvement related to children’s mental health.
The stakeholders expressed specific concerns
about inpatient recidivism. Between October
2011, when GCBH established its in-house
Authorization Center, and December 2012,
GCBH calculated a 90-day readmission rate of
28% for children. GCBH cited research studies
showing that effective discharge planning and the
establishment of better collaborative relationships
between managed care and inpatient providers are
critical in reducing hospital readmissions.
For this first-year PIP, GCBH intends to implement
the use of the Child Inpatient Admission Review
questionnaire for each child authorized for an
inpatient stay. The purpose of the questionnaire is
to facilitate enhanced communication between
GCBH’s in-house Authorization Center staff and
inpatient providers’ utilization personnel, with the
ultimate goal of reducing the readmission rate for
children.
GCBH presented little data to support the design
and implementation of this PIP. The RSN needs to
present data to support the selection of the study
topic, and more information regarding the codes
used to pull administrative data, procedures for
data validation, why GCBH chose the specific
intervention, and a plan for tracking and
monitoring the intervention.
The overall weighted score for this PIP is 53,
based on a scale of 90, resulting in a compliance
rating of Partially Met.
Figure 3 shows the score for each of the eight
validation standards reviewed for this project.
Figure 3. Validation scores by standard for nonclinical PIP, Lowered Inpatient Readmission Rates in a High Risk Population through the Development of Enhanced Communication with Inpatient Providers.
Substantially met
Fully met
Partially met
Not met
Minimally met
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
9 Acumentra Health November 2013
Standard 1: Study Topic Score: 90 (Substantially met)
To meet Standard 1, the RSN needs to establish
the importance of the study topic in general and
present local data to demonstrate that the topic
applies to a large or high-risk portion of the
Medicaid population; and demonstrate that a
systematic selection and prioritization process
was used in choosing the topic.
GCBH cited several reasons for selecting this
topic, including: (a) a contractual requirement for
each RSN to conduct at least one PIP focused on
children, with the goal of providing services and
treatment within community settings; (b) concerns
expressed by stakeholders with regard to inpatient
readmissions for children; and (c) 112 inpatient
readmissions within 90 days of discharge out of
401 admissions that occurred between October
2011 and December 2012, resulting in a 28%
readmission rate for that period. GCBH did not
break down these readmissions among acute care
settings, the state hospital, and Children’s Long-
Term Inpatient Program (CLIP) facilities, nor did
the RSN present other data that would enable a
comparison with the local readmission rate.
GCBH cited studies showing that “effective
discharge planning is critical to preventing
unnecessary inpatient readmissions” and that
“improved relations between managed care and
inpatient providers support better discharge
outcomes.”
Before opening its in-house Authorization Center
in October 2011, GCBH lacked access to the data
necessary to determine whether the RSN was
adhering to its philosophy that “inpatient services
should not be used when there are viable
alternatives that address a consumer’s needs.” The
RSN and affiliated stakeholders elected to pursue
this study topic based on concerns about hospital
readmissions for children, improved access to data
related to hospitalizations, and the ability of
GCBH’s Authorization Center staff to engage
directly with inpatient providers.
GCBH stated that readmission rates indicate the
“quality or adequacy of treatment services that the
individual received during the previous
hospitalization.” For this project, GCBH plans to
implement the use of a questionnaire intended to
facilitate enhanced communication between the
Authorization Center staff and inpatient providers.
GCBH reported that “studies suggest that benefits
to this approach include higher individual client
outcomes, identification of types of common
debilitating gaps in services, identification of
trends, and the opportunity to increase system-
wide awareness.”
To fully meet this standard, GCBH needs to
provide additional data about the use of inpatient
facilities included in the RSN’s readmission rate,
and demonstrate that hospital readmissions for
children is a significant area of need.
Standard 2: Study Question Score: 100 (Fully met)
To meet Standard 2, the RSN needs to present a
study question that provides a clear framework
for data collection, analysis, and interpretation.
The study question should refer to the proposed
intervention, a study population (denominator), a
measure (numerator), a metric (e.g., average,
percentage), and a direction of desired change.
GCBH presented this study question:
“Does enhanced communication with inpatient
providers, via the implementation of a child
inpatient admission questionnaire by GCBH
Authorization Center staff at the time of
authorization/admission, decrease the proportion
of children readmitted to community inpatient
services within 90 days of his/her inpatient
discharge?”
The question establishes a clear framework for
analysis, including an intervention (enhanced
communication), a study population (children
discharged from community inpatient services), a
measure (readmitted within 90 days), a metric
(proportion), and a direction of change (decrease).
GCBH fully meets this standard.
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
10 Acumentra Health November 2013
Standard 3: Study Indicator Score: 95 (Substantially met)
To meet Standard 3, the RSN needs to define the
measure (numerator) and study population
(denominator); define key terms; and discuss the
basis for adopting the indicator as a valid proxy
for enrollee outcomes, satisfaction, or quality of
care.
GCBH defined the elements of the study indicator
as follows.
Denominator: The number of children in a given
measurement period who were discharged from
inpatient care in a community setting, and who
were Medicaid-eligible at the time the inpatient
episode of care commenced.
Numerator: The number of children in the
denominator who were readmitted to a
community inpatient setting between the 2nd
and
90th
calendar day following discharge from a prior
episode of inpatient care.
GCBH defined the following key terms.
Authorized: Includes only children approved by
a staff member of the GCBH Authorization
Center to receive inpatient services.
Children: Individuals 20 years or younger at the
time of admission.
Episode of inpatient care: A span of days during
which the enrollee received inpatient services in a
single inpatient setting or in multiple settings
where admission to a new setting occurred within
1 calendar day of discharge from the previous
one. Defining readmissions occurring within 1
calendar day as “transfers” is a common practice
when calculating readmission rates.
Community inpatient setting: Any inpatient
entity, including evaluation and treatment (E&T)
facilities, that provides inpatient mental health
services. Includes out-of-RSN and out-of-state
providers and excludes settings to which
admission is not authorized by the GCBH
Authorization Center staff (e.g., state hospitals,
CLIP facilities).
Discharged: The end of an episode of inpatient
care.
Medicaid eligibility: Eligibility status determined
by GCBH Authorization Center staff at the time
of admission on the basis of information available
to them via Provider One.
Readmission: A new episode of inpatient care
that occurs when a patient is readmitted to a
psychiatric inpatient facility within 90 days of a
previous discharge.
GCBH noted that its definition of Medicaid
eligibility could lead to excluding children from
the study indicator who are eligible for Medicaid
at the time of admission, but not at the time of
readmission. However, GCBH stated that this
would be a rare occurrence.
In defining readmission, GCBH did not indicate
that the admission needed to occur between the
2nd
and 90th
day following a previous discharge to
be included in the study indicator.
GCBH reported that it selected the study
indicator, readmission to an inpatient setting
within 90 days of discharge, because “numerous
studies suggest that largest risk for readmission is
within the first 30 days of discharge and remains
high for 3 months.”
GCBH stated its belief that readmission rates
indicate not only the quality of medical treatment
received during an inpatient stay, but also the
quality of discharge planning. The RSN believes
that the use of the Child Inpatient Admission
Review questionnaire by the Authorization Center
will enhance communication between the RSN
and inpatient providers. GCBH anticipates that
enhanced communication will help identify
“needs, gaps, and risk factors for readmissions,”
which they believe will lead to improved
discharge planning for consumers and, ultimately,
fewer readmissions.
To fully meet this standard, GCBH should
provide a definition of readmission consistent
with the description of the study numerator.
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
11 Acumentra Health November 2013
Standard 4: Denominator (Study Population) Data Collection Score: 60 (Partially met)
To meet Standard 4, the RSN needs to list all
inclusion and exclusion criteria for the study
population; document all data sources, including
fields, codes, and calculations; and describe data
validation procedures. If a sample is selected, the
RSN needs to describe the sampling methods.
GCBH identified inclusion criteria for the study
population, specifying data sources, tables, fields,
and calculations. The data elements include
Medicaid eligibility, age, inpatient provider,
inpatient admission, inpatient discharge, and
authorization. All data are available through
GCBH’s Consumer Information System (CIS),
which consolidates enrollee information including
Medicaid eligibility and age, which are available
from Provider One. GCBH did not report specific
codes that would indicate an inpatient admission
related to mental health.
GCBH excluded individuals from the study
population who were older than 20 years at the
time of admission to an inpatient psychiatric
setting. The study also excludes enrollees who are
admitted to CLIP or state hospital facilities,
because a different authorization process is used
for those admissions. GCBH did not discuss what
effect, if any, excluding these enrollees might
have in terms of accurately assessing the RSN’s
readmission rate for children.
GCBH provided no information related to data
validation procedures for the study population.
The report directs the reader to “see the table at
the end of the document for specifics concerning
the data elements,” but no information related to
data validation was included there.
To fully meet this standard, GCBH needs to
report applicable codes for inpatient admissions
related to mental health, discuss the exclusion of
enrollees who are admitted to state hospital and
CLIP facilities, and provide information about
data validation procedures for the study
population.
Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan Score: 60 (Partially met)
To meet Standard 5, the RSN needs to list all
inclusion and exclusion criteria for the numerator
(what is being measured); document all data
sources, including fields, codes, and calculations;
describe data validation procedures; and present
a clear data analysis plan, including time frames
for the measurement and intervention periods,
and an appropriate statistical test to measure
differences between the baseline and
remeasurement periods.
Readmissions to inpatient care between the 2nd
and
90th
days following discharge from a previous
inpatient admission are the only inclusion criterion
for the numerator. Admission and discharge dates
obtained through CIS are the only data collected.
GCBH did not report the mental health codes that
would indicate a psychiatric inpatient admission.
An admission date on the same day or 1 day
following a discharge date is considered a transfer
and a continuation of care. Transfers are not
included in the numerator.
GCBH did not account for the possibility that an
enrollee who had a previous admission to a
community inpatient hospital setting could be
readmitted to a different type of facility (e.g.,
CLIP) within 90 days of discharge, but would not
be counted in the readmission rate. Conversely, an
enrollee could have been discharged from a CLIP
facility or the state hospital and then been
readmitted to a community inpatient setting
within 90 days and would also not be counted in
the readmission rate.
GCBH provided no information related to data
validation procedures for the data elements
included in the numerator for this PIP.
GCBH reported the following study timeline:
Baseline: July 1, 2012–September 30, 2013
Intervention: October 1, 2013–September 30,
2014
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
12 Acumentra Health November 2013
First remeasurement: October 1, 2013–
December 31, 2014
GCBH did not provide dates for the second
remeasurement period.
To identify statistical significance, GCBH plans
to use the chi-square test for independent groups
at the p≤.05 level of probability.
To fully meet this standard, GCBH needs to
report the relevant mental health codes for
inpatient admission criteria; discuss the effect of
excluding CLIP and state hospital admissions
from the readmission rate calculation; provide
information related to data validation procedures;
and provide dates for the second remeasurement
period.
Standard 6: Study Intervention Score: 35 (Minimally met)
To meet Standard 6, the RSN needs to select an
improvement strategy that will affect a wide range
of enrollees or a high-risk enrollee population,
and that is reasonably expected to result in
measurable improvement. The RSN needs to
discuss the basis for adopting the intervention;
document the implementation of the intervention,
including dates and locations of principal
activities; discuss cultural competence; and track
how effectively the intervention was implemented.
For this PIP, GCBH plans to implement the use of
a questionnaire to help facilitate communication
between Authorization Center staff and inpatient
providers, from the point of authorization through
discharge, for all children eligible for inclusion in
the study population. GCBH stated that it will use
the questionnaire to “identify a wide variety of
factors related to clinical outcomes.” Following
discharge, questionnaires will be given to the
GCBH quality manager who will track
readmissions and document trends.
Although the primary responsibility of GCBH
Authorization Center staff is to determine whether
admission or continuation of inpatient care
criteria have been met, GCBH believes that staff
are in a position to work collaboratively with
inpatient providers. This process is expected to
aid in discharge planning and, ultimately, have a
positive effect on clinical outcomes. GCBH
described Authorization Center staff as “well-
qualified clinicians who have a wealth of
knowledge about available resources,” as well as
information about children’s previous contacts
with the mental health system. GCBH believes
that the implementation of the questionnaire will
enable exploration of gaps or concerns, as well as
the availability or previous use of specific
services following discharge.
GCBH provided no information related to the
validity, reliability, or previous use of the
questionnaire to justify the expectation of
improvement based on its use. GCBH did not
report whether Authorization Center staff would
receive training related to the appropriate use of
the questionnaire. It is also not clear whether
inpatient providers, who are more directly
involved with discharge planning, are aware
that Authorization Center staff will use the
questionnaire or if they know the intent of the
questionnaire.
With regard to tracking and monitoring the
intervention, GCBH reported that the questionnaire
will have check boxes next to each item and ways
for Authorization Center staff to indicate if
categories are not applicable. The questionnaire
will also include dates of discharge and enrollee
identifiers “to allow the Quality Manager to track
and monitor data.” GCBH did not indicate whether
it intends to track completion of the questionnaire
for each child eligible for inclusion in the study
population. The RSN also did not report the
anticipated frequency of its tracking and
monitoring activities.
GCBH reported that the questionnaire includes an
item that allows for the exploration of cultural
factors that may affect treatment.
To fully meet this standard, GCBH needs to
provide more information about the questionnaire,
any training required to use it, and why it use is
expected to improve readmission rates. GCBH
also needs to indicate whether inpatient providers,
2013 Greater Columbia Behavioral Health EQR Report: Nonclinical PIP Validation
13 Acumentra Health November 2013
who are expected to help complete the
questionnaires, are aware of the plan to use the
questionnaire and the intent behind its use.
Finally, GCBH needs to provide more
information about its plan for tracking and
monitoring the implementation of its intervention,
particularly in terms of frequency, and whether
the RSN plans to track whether the questionnaire
is completed for each eligible child.
Standard 7: Study Results Score: 0 (Not met)
To meet Standard 7, the RSN needs to present
results according to the data analysis plan,
including the study indicator, the original data
used to compute the indicator, and a statistical
test to measure differences between the baseline
and remeasurement periods; and discuss how the
intervention influenced the results.
GCBH did not report on this standard because the
study has not advanced to this stage at the time of
the PIP review.
To fully meet this standard, GCBH needs to
present data for the numerator and denominator
for the baseline and remeasurement period,
present the results of statistical analyses
comparing the baseline and first remeasurement
using the appropriate test statistic and probability
level, and discuss how the intervention influenced
the study results.
Standard 8: Interpretation of Study Results Score: 0 (Not met)
To meet Standard 8, the RSN needs to assess
whether any reported improvement is “real” by
documenting that baseline and remeasurement
data were collected using the same methods and
are comparable; discuss the statistical and
clinical significance of the study results; address
barriers to improvement and lessons learned
during the PIP process; and identify confounding
factors that may have affected the results.
GCBH did not report on this standard because the
study has not advanced to this stage at the time of
the PIP review.
To fully meet this standard, GCBH needs to
interpret the study results in terms of statistical
and clinical significance, evaluate the methods
used, and discuss confounding factors.
Standard 9: Study Modifications After the First Remeasurement Score: n.a.
To meet Standard 9, the RSN needs to document
modifications to the intervention, or added
interventions, planned or implemented after the
first remeasurement period; and discuss changes
in other aspects of the PIP based on lessons
learned from data analysis or barrier analysis.
Acumentra Health did not score this standard, as
the study has not progressed to a second
remeasurement.
Standard 10: Overall Study Results Score: n.a.
To meet Standard 10, the RSN needs to report
complete study results for two or more
measurement periods, including the study
indicator, original data used to compute the
indicator, and a statistical test of group
differences; and interpret the statistical and
clinical significance of the overall results, discuss
lessons learned, and determine if goals were met
and sustained improvement was achieved.
Acumentra Health did not score this standard, as
the study has not progressed to a second
remeasurement.
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
14 Acumentra Health November 2013
60
80
5560 70
45
0 00
20
40
60
80
100
1 2 3 4 5 6 7 8
Sco
re
Standard
Review results for clinical PIP: Lowered PRISM Scores in a High Medical Risk Psychiatric Inpatient Population Through the Integration of Health Information into Mental Health Records
This first-year PIP focuses on integrating physical
health information into mental health treatment
plans. GCBH cited national research findings that
under-treatment of physical health needs, as a
result of “siloed” information systems, adversely
affects mental health conditions. However, the
RSN provided no evidence as to the nature and
extent of this problem at the local Medicaid level.
Because GCBH did not analyze the gap in service,
the RSN could not discuss how its intervention
strategy addresses identified barriers or is
expected to improve the study indicator. It is also
not clear whether the selected indicator is the
most appropriate measure of integration of
physical health information into mental health
treatment plans.
During the onsite interview, GCBH discussed its
reasons for selecting the study topic, indicator,
and intervention, but the RSN did not fully
document these explanations in its report. Though
engaged and interested in learning to apply quality
improvement processes, GCBH seems hindered in
its efforts by factors such as staffing resources,
time, and provider cooperation.
Acumentra Health recognizes the challenges
facing GCBH, and encourages the RSN to focus
on key PIP concepts—relevance of the topic to
the local Medicaid population, validity of the
study indicator, and barrier analysis to support
intervention selection—in its next iteration.
The overall weighted score for this PIP is 43,
based on a scale of 90, resulting in a compliance
rating of Partially Met.
Figure 4 shows the score for each of the validation
standards reviewed for this clinical project.
Figure 4. Validation scores by standard for clinical PIP, Lowered PRISM Scores in a High Medical Risk Psychiatric Inpatient Population Through the Integration of Health Information into Mental Health Records.
Substantially met
Fully met
Partially met
Not met
Minimally met
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
15 Acumentra Health November 2013
Standard 1: Study Topic Score: 60 (Partially met)
To meet Standard 1, the RSN needs to establish
the importance of the study topic in general and
present local data to demonstrate that the topic
applies to a large or high-risk portion of the
Medicaid population; and demonstrate that a
systematic selection and prioritization process
was used in choosing the topic.
This PIP, new for 2013, focuses on the integration
of physical and behavioral health, particularly the
incorporation of physical health information into
mental health records for the purpose of care
coordination. GCBH cited several research studies
that demonstrated a link between the lack of
integration of physical health issues into mental
health treatment and increased morbidity and
mortality in the mentally ill population. The RSN
also cited anecdotal reports that mental health
providers are often uninformed as to their
patients’ medical conditions. GCBH sees this PIP
as an opportunity to “facilitate holistic treatment
of the consumer.”
On a statewide level, GCBH observed that
Washington DSHS has targeted the issue of
integration through the implementation of the
Predictive Risk Intelligence System (PRISM).
PRISM is a decision support tool “intended for
use by health care providers to identify consumers
most in need of comprehensive care coordination
based upon a risk score.”
On a local level, GCBH cited “shared voices of
concern about recidivism and its relationship to
under-addressed co-occurring medical
conditions.” GCBH discussed the role of its
Authorization Center staff in coordinating care
with inpatient staff and outpatient providers.
These statements, however, do not support the
proposed study topic and intervention.
To demonstrate that the study topic is relevant to
the local Medicaid population, GCBH should first
clarify whether the gap in service is due to the
availability of physical health information to
mental health providers and/or to the utilization of
that information by mental health providers in
mental health treatment planning. GCBH then
should provide evidence to support the study topic
selection.(e.g., data on the number of enrollees in
its population with chronic medical conditions,
evidence that information on these enrollees’
physical health is not being made available to
mental health providers, discussion of current
utilization of physical health information by
mental health providers, and an explanation of
how missing physical health information has had
a deleterious impact on local enrollee health
outcomes, status, or satisfaction).
In its discussion, GCBH used “care coordination”
and “integration” to define the study problem and
the RSN’s vision. GCBH described the link
between the two concepts as follows: “Effective
coordination of care is at the heart of matching
services and information to individual needs at the
right time and in the right place.” Again, GCBH
needs to discuss the current status of coordination
of services and information at the RSN’s provider
agencies and how coordination is falling short of
the RSN’s vision.
GCBH stated that discussions about the selection
of a PIP topic had occurred during meetings of its
Clinical Directors Committee, Quality
Management Oversight Committee, Board of
Directors, and Regional Advisory Board, which
includes enrollees and their family members. The
RSN did not discuss how it identified the study
topic or why it prioritized this topic over others.
GCBH noted that the topic of integration of
physical and mental health was aligned with state
contractual guidelines on integration and with the
RSN’s own practice guidelines for recovery.
To meet this standard fully, GCBH needs to
clarify the problem it plans to address, provide
data and other evidence demonstrating that an
opportunity for improvement exists for the local
Medicaid population; and describe the PIP
selection and prioritization in more detail.
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
16 Acumentra Health November 2013
Standard 2: Study Question Score: 80 (Substantially met)
To meet Standard 2, the RSN needs to present a
study question that provides a clear framework
for data collection, analysis, and interpretation.
The study question should refer to the proposed
intervention, a study population (denominator), a
measure (numerator), a metric (e.g., average,
percentage), and a direction of desired change.
GCBH presented this study question:
“Will the training of outpatient providers and the
integration of physical healthcare information into
mental health records lower PRISM scores for
high medical risk psychiatric inpatient
population?”
The study question includes an intervention
(training and integration of physical health
information), a study population (high medical
risk psychiatric inpatients) a numerator (PRISM
scores), and a direction of change (lower). To
compare results across measurement periods, the
question needs to incorporate a metric associated
with the numerator, e.g., average. Also, GCBH
should clarify that the outpatient providers are
mental health service providers.
The appropriateness of the numerator is discussed
under Standard 3.
To meet this standard fully, GCBH needs to
include a metric in the study question and clarify
the term “outpatient provider.”
Standard 3: Study Indicator Score: 55 (Partially met)
To meet Standard 3, the RSN needs to define the
measure (numerator) and study population
(denominator); define key terms; and discuss the
basis for adopting the indicator as a valid proxy
for enrollee outcomes, satisfaction, or quality of
care.
GCBH identified the following indicator
elements.
Denominator or study population:
Medicaid eligible as determined by the
Authorization Center
18 years or older at the time of admission
(to an inpatient facility)
discharged from a community-based
inpatient psychiatric facility into a GCBH
outpatient program
had health information sent to enrollee’s
mental health outpatient team
had at least one outpatient appointment
high medical risk as evidenced by a
PRISM score ≥ 0.5
Numerator: PRISM scores for enrollees in the
denominator in each measurement period.
GCBH defined the terms adult, Authorization
Center, discharged, and Medicaid eligible.
Definitions of other key elements included:
Continuous enrollment: Enrolled in outpatient
services in each measurement period. The RSN
did not discuss any allowable gaps in enrollment.
GCBH provided this definition under numerator
inclusion criteria, but the criterion actually applies
to the study population.
Outpatient visit: At least one routine visit
following initial intake. In the onsite PIP
interview, GCBH explained that the enrollee’s
treatment plan (and incorporation of the PRISM
data) would be reviewed at a routine visit
following hospitalization. It is not clear that a
single outpatient visit allows enough time for a
mental health provider to incorporate physical
health needs into mental health treatment goals
and treatment (thereby reducing the PRISM
score). GCBH’s report should document its
rationale for this inclusion criterion.
Health information: A PDF printout of the
PRISM report submitted via the GCBH provider
portal.
PRISM: Predictive Risk Intelligence System. A
decision support tool used to support care
management interventions for high-risk Medicaid
enrollees.
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
17 Acumentra Health November 2013
PRISM score: Medical cost risk score calculated
from information from medical, social service,
behavioral health, and long term care systems. A
PRISM score is calculated “for each Medicaid
individual who has received medical and/or
psychiatric treatment and/or medications (and for
which P1 received a bill) within the last 24
months.” GCBH should discuss how often a
PRISM score is recalculated.
High medical risk: In the onsite PIP interview,
GCBH indicated that it would use PRISM scores
to identify “high risk medical” enrollees. The
RSN should document its reason for using a
PRISM score of ≥ 0.5 to indicate “high medical
risk,” and provide evidence that a high PRISM
score is always correlated with high medical risk,
i.e., it is not possible for an enrollee to have a
high PRISM score without having serious
physical health needs.
Community-based inpatient psychiatric
setting: GCBH should indicate the number of
facilities involved in the project.
The definition for outpatient team is missing from
the report.
In justifying its selection of the study indicator,
GCBH noted that PRISM reports and scores are
easily available to RSN staff and can be shared
confidentially with outpatient providers. GCBH
described PRISM as a decision support tool used
to identify those enrollees most in need of care
coordination and then to develop appropriate
interventions. In both its report and in the onsite
review, GCBH emphasized the feasibility of using
this study indicator. However, the RSN did not
provide evidence that PRISM scores are a valid
tool for assessing the effectiveness of care
coordination in the form integrating physical
health needs into mental health treatment plans,
over time. Since GCBH stated that the PRISM
score is calculated using data from multiple
systems over a two-year period, it is not clear that
a reduction in the PRISM score could necessarily
be attributed to the PIP intervention.
If GCBH cannot successfully address the multiple
challenges surrounding the use of average PRISM
scores as an indicator, the RSN should consider
selecting a different measure (e.g., the percent of
enrollees or high medical risk enrollees who have
mental health treatment goals that are consistent
with their PRISM scores).
To meet this standard fully, GCBH needs to
define all key indicator elements and provide
evidence for the validity of its study indicator, or
select an appropriate study indicator.
Standard 4: Denominator (Study Population) Data Collection Score: 60 (Partially met)
To meet Standard 4, the RSN needs to list all
inclusion and exclusion criteria for the study
population; document all data sources, including
fields, codes, and calculations; and describe data
validation procedures. If a sample is selected, the
RSN needs to describe the sampling methods.
GCBH described its study population as a pilot
study using a convenience sample. Instead of
covering the entire adult Medicaid population,
GCBH is targeting enrollees who have been
discharged from an inpatient facility. The RSN
stated that its authorization staff can identify high
medical risk enrollees through the in-house CIS,
and has easy access to the PRISM system. In the
onsite PIP interview, GCBH explained in detail the
reasons why it chose to focus on inpatient and high
medical risk enrollees. The RSN should document
this rationale in its discussion about sampling, and
discuss the possible effect a convenience sample
will have on study results and the generalizability
of the results to the larger RSN population.
Data on study population age, Medicaid
eligibility, and PRISM scores are available from
the Provider One database. Discharges from an
inpatient setting are available from the GCBH
Authorization Center and CIS database. GCBH
did not provide details about data source, codes,
and calculations for continuous enrollment criteria
and study population criteria listed under
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
18 Acumentra Health November 2013
Standard 3 (at least one outpatient appointment,
and dissemination of health information to
outpatient team).
The RSN listed no study denominator exclusions.
The RSN should consider whether exclusion
criteria may include situations in which enrollees
are not available for mental health treatment (e.g.,
incarceration, admission to a long-term care
facility, etc.).
The RSN’s quality manager will track the data on
an Excel spreadsheet. With regard to ensuring that
the study population was captured accurately,
GCBH stated that it used Provider One to validate
Medicaid eligibility data. The RSN did not
describe validation procedures for any other data
elements, including how it will validate correct
data entry of the enrollee demographics and
PRISM scores into the Excel spreadsheet and
receipt of health information by the mental health
provider.
To meet this standard fully, GCBH needs to
ensure that inclusion criteria for the study
denominator are consistent between standards;
provide the source, calculations, and codes for all
inclusion criteria; describe validation procedures
for every data element; and provide more details
on the rationale for using a convenience study
sample.
Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan Score: 70 (Partially met)
To meet Standard 5, the RSN needs to list all
inclusion and exclusion criteria for the numerator
(what is being measured); document all data
sources, including fields, codes, and calculations;
describe data validation procedures; and present
a clear data analysis plan, including time frames
for the measurement and intervention periods,
and an appropriate statistical test to measure
differences between the baseline and
remeasurement periods.
The numerator for this study is represented by
PRISM scores on an individual enrollee’s PRISM
report generated from the website. There are no
true exclusion criteria.
GCBH authorization staff will send a PDF copy
of the PRISM report and scores to the quality
manager when an enrollee is discharged from an
inpatient facility. The quality manager will track
the baseline PRISM scores on an Excel
spreadsheet, and will “track and monitor” PRISM
scores for each remeasurement period. GCBH did
not discuss whether authorization staff would also
be responsible for generating remeasurement
PRISM scores.
Regarding validation procedures, GCBH noted:
“PRISM score accepted as valid.” Even though
the PRISM score is considered valid as reported,
GCBH needs to discuss how it will ensure that the
data entry into its own tracking system is accurate.
GCBH provided the following study timeline.
Baseline: January 1–June 30, 2014
Intervention: January 1–June 30, 2014
First remeasurement: July 1–December 31, 2014
Second remeasurement: January 1–June 30, 2015
GCBH plans to follow a single cohort of enrollees
who meet the inclusion criteria and are discharged
from an inpatient facility from January through
June 2014. Remeasurement will occur every six
months for each study enrollee according to their
discharge dates in the baseline measurement
period. GCBH did not describe an acceptable
window for the measurement periods, i.e., “5–7
months” for “6 months.” The intervention will
begin after an enrollee’s baseline PRISM report is
sent to the enrollee’s outpatient provider after
discharge. In this case, “baseline” is “the first
measure” measurement period.
GCBH will conduct comparisons with a paired
t-test for the first remeasurement and a repeated
measure ANOVA for the second remeasurement
(comparing three measurement periods), with a
probability level of p<.05.
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
19 Acumentra Health November 2013
To meet this standard fully, GCBH needs to
describe how PRISM scores will be collected and
disseminated for the remeasurement periods,
describe the acceptable window for each of its
measurement periods, and discuss how it will
ensure that data entry of PRISM scores is
accurate.
Standard 6: Study Intervention Score: 45 (Minimally met)
To meet Standard 6, the RSN needs to select an
improvement strategy that will affect a wide range
of enrollees or a high-risk enrollee population,
and that is reasonably expected to result in
measurable improvement. The RSN needs to
discuss the basis for adopting the intervention;
document the implementation of the intervention,
including dates and locations of principal
activities; discuss cultural competence; and track
how effectively the intervention was implemented.
In the study question (Standard 2), GCBH
described the intervention as provider training and
the “integration of physical healthcare
information into mental health records.” Under
Standard 6, GCBH described its intervention
strategy as the dissemination of a PDF summary
of the PRISM report on study enrollees by
Authorization Center staff to the outpatient
mental health provider at the time of enrollee
discharge from an inpatient facility. In addition to
disseminating the PRISM report summary, GCBH
plans to conduct a 10- to 15-minute web training
for provider staff who will receive the document.
The training will focus on the ways the health
information in the PRISM report “can inform
mental health treatment planning.”
The assumption is that the provider, having
received both the information and the training,
will integrate an enrollee’s physical health issues
into the mental health treatment plan. However, in
the PIP interview, GCBH reported difficulty in
directing provider actions. Therefore, GCBH
should discuss how it will engage provider staff in
participating in training and changing behavior
(developing new treatment goals).
GCBH needs to provide more details about its
intervention, e.g., the type and presentation of
information in the PRISM report summary; the
allowable time frame for disseminating the report
to the provider; the number and roles of the staff
who will be trained; how newly hired staff would
receive the training; and whether or not the
training would be repeated. Training is expected
to begin in October and continue through
December 2013, followed by implementation of
the intervention in January 2014.
GCBH described the role of its Authorization
Center and discussed the feasibility of
implementing this intervention due to the
availability of qualified staff and their ability to
access PRISM reports. Although the intervention
may be feasible, GCBH still needs to discuss the
barriers upon which it based its intervention.
What barriers do mental health providers face in
integrating physical health information into
mental health treatment plans? How does the
receipt of information and a 15–minute training
address those barriers? GCBH also needs to
discuss why the intervention can be expected to
improve the study indicator; for example, because
it is an evidence-based practice or because its
successful implementation is assured by complete
support of all stakeholders.
In discussing its tracking and monitoring plan,
GCBH described the supervision of the
Authorization Center staff and stated that the
quality manager would track and monitor study
data. While it is important to establish that
qualified and supervised staff will be
disseminating information, GCBH needs to focus
on how it will track whether or not the
information (PRISM report) was disseminated
and received as planned, whether or not mental
health providers watched the training, and
whether or not mental health providers included
physical health issues into the mental health
treatment plans.
Regarding cultural and linguistic appropriateness,
GCBH stated that the web training will “include
and address cultural sensitivity as is it applies to
2013 Greater Columbia Behavioral Health EQR Report: Clinical PIP Validation
20 Acumentra Health November 2013
physical health integration into mental health
treatment.” The RSN should provide more details
about how the training addresses cultural issues.
Also, GCBH could discuss whether the PRISM
report takes cultural (e.g., disabilities, serious
mental illness) and linguistic (e.g., non-English
speakers) factors into account.
To meet this standard fully, GCBH needs to
thoroughly describe its intervention strategy;
explain how the intervention addresses identified
barriers and can be expected to improve the study
indicator; describe how it will track the successful
implementation of the intervention; and provide
more details on how it will ensure the intervention
is culturally and linguistically appropriate.
Standard 7: Study Results Score: 0 (Not met)
To meet Standard 7, the RSN needs to present
results according to the data analysis plan,
including the study indicator, the original data
used to compute the indicator, and a statistical
test to measure differences between the baseline
and remeasurement periods; and discuss how the
intervention influenced the results.
GCBH did not report on this standard because no
results were available at the time of the PIP
review.
To meet this standard fully, GCBH needs to report
data for the baseline and remeasurement periods,
compare the indicator between the two periods
with an appropriate statistical test, and discuss
how the intervention influenced the study results.
Standard 8: Interpretation of Study Results Score: 0 (Not met)
To meet Standard 8, the RSN needs to assess
whether any reported improvement is “real” by
documenting that baseline and remeasurement
data were collected using the same methods and
are comparable; discuss the statistical and
clinical significance of the study results; address
barriers to improvement and lessons learned
during the PIP process; and identify confounding
factors that may have affected the results.
The project has not advanced far enough for
GCBH to address this standard.
To meet this standard fully, GCBH needs to
discuss whether the intervention resulted in
statistically and clinically significant improvement;
identify confounding factors or barriers that had an
impact on study results and discuss how they were
addressed; and describe any lessons learned.
Standard 9: Study Modifications After the First Remeasurement Score: n.a.
To meet Standard 9, the RSN needs to document
modifications to the intervention, or added
interventions, planned or implemented after the
first remeasurement period; and discuss changes
in other aspects of the PIP based on lessons
learned from data analysis or barrier analysis.
Acumentra Health did not score this standard, as
the study had not progressed to a second
remeasurement.
Standard 10: Overall Study Results Score: n.a.
To meet Standard 10, the RSN needs to report
complete study results for two or more
measurement periods, including the study
indicator, original data used to compute the
indicator, and a statistical test of group
differences; and interpret the statistical and
clinical significance of the overall results, discuss
lessons learned, and determine if goals were met
and sustained improvement was achieved.
Acumentra Health did not score this standard, as
the study had not progressed to a second
remeasurement.
2013 Greater Columbia Behavioral Health EQR Report: Compliance review follow-up
21 Acumentra Health November 2013
COMPLIANCE REVIEW
FOLLOW-UP
Acumentra Health’s 2012 compliance review
addressed GCBH’s compliance with federal and
state standards related to eight major sections of
managed care operations. Each review section
contained elements corresponding to relevant
sections of 42 CFR §438, DBHR’s contract with
RSNs, the Washington Administrative Code, and
other state regulations where applicable. The
reviews followed a protocol adapted from the
CMS protocol and modified with DBHR’s
approval. The provisions of Washington’s
Medicaid waiver and the RSN contract are such
that some parts of the federal protocol do not
apply directly to RSNs.
In 2013, Acumentra Health reviewed GCBH’s
response to the specific 2012 EQR findings for
which DBHR required the RSN to perform
corrective action. Table 3 summarizes the results
of this follow-up review.
2013 Greater Columbia Behavioral Health EQR Report: Compliance review follow-up
22 Acumentra Health November 2013
Table 3. Results of compliance review follow-up for Greater Columbia Behavioral Health.
2012 findings 2012 recommendations RSN response
Delivery Network
GCBH does not track and monitor all requests for second opinions.
GCBH has corrected this finding, and current processes include tracking and monitoring. The RSN has reviewed the process with local clinical directors.
The RSN described a process that meets the requirement.
Coordination and Continuity of Care
None
Coverage and Authorization of Services
None
Provider Selection
GCBH does not have a mechanism in place to monitor the results of its credentialing and recredentialing process.
GCBH stated that it had a process to routinely track and monitor the credentialing and recredentialing process during 2012. In 2013, the RSN confirmed that this process includes routine monitoring of the Office of the Inspector General’s website to check each credentialed provider and staff person. Yearly, the RSN audits a sample of agency credentialing files onsite.
The RSN described a process that meets the requirement.
Subcontractual Relationships and Delegation
None
Practice Guidelines
GCBH does not review and update its practice guidelines to ensure that they still apply to enrollees needs and include any updated clinical recommendations.
GCBH has revamped its Quality Assurance program. The draft is to be vetted by the community advisory committee and the board. Practice guidelines have been updated and placed on the RSN’s website.
The RSN described a process that meets the requirement.
QA/PI Program
GCBH lacks mechanisms to detect, identify, and monitor for over- and underutilization of services.
GCBH has reviewed local data and performance measures to identify over/underutilization. The RSN has added three additional mechanisms for monitoring the service penetration rate, recidivism of inpatient stays for children, and information from the PRISM database to identify overutilization. This year, the RSN will determine the baseline experience.
The RSN described a process that meets the requirement.
Certification and Program Integrity
None
2013 Greater Columbia Behavioral Health EQR Report: ISCA Overview
23 Acumentra Health November 2013
INFORMATION SYSTEMS
CAPABILITIES ASSESSMENT
Acumentra Health examined GCBH’s
information systems and data processing and
reporting procedures to determine the extent to
which they supported the production of valid and
reliable state performance measures and the
capacity to manage care of RSN enrollees.
The ISCA procedures were based on the CMS
protocol for this activity, as adapted for the
Washington RSNs with DBHR’s approval. In
2013, Acumentra Health added a new review
subsection, Meaningful Use of Electronic Health
Records, per the CMS protocol published in
September 2012. Due to the timing of the CMS
waiver, this section will be reviewed but not
scored in 2013.
The 2011–2013 RSN contracts include Exhibit C
with more detailed data security requirements.
These criteria were included in the 2013 ISCA for
the first time.
Acumentra Health interviewed four contracted
provider agencies to gain understanding of the
flow of encounter data from providers to GCBH.
These interviews targeted the agencies’ data
validation processes; diagnoses and procedure
codes captured in their billing systems; handling
of Medicaid/Medicare dual enrollees; types of
encounter data forwarded to GCBH; and methods
for submitting encounter data to GCBH.
The ISCA review was organized in two main
sections—(1) Data Processing Procedures and
Personnel and (2) Data Acquisition Capabilities—
with nine subsections, each containing review
elements from the ISCA protocol. Appendix B
explains the review criteria in greater detail.
Within each subsection, Acumentra Health used
the information collected in the ISCA data
collection tool, responses to interview questions,
and results from the security walkthrough to score
the RSN’s performance on each element on a
scale from 1 to 3 (see Table 4).
After scoring the individual elements, Acumentra
Health combined the scores and calculated a
weighted average score for each subsection. The
detailed criteria for scoring are available from
Acumentra Health upon request.
Table 4. Scoring scheme for ISCA standards.
Score Rating Definition
2.6–3.0 Fully met (pass) Meets or exceeds the standard requirements.
2.0–2.5 Partially met (pass) Meets essential requirements of the standard but is deficient in some areas.
< 2.0 Not met (fail) Does not meet the essential requirements of the standard.
– N/A Not applicable.
2013 Greater Columbia Behavioral Health EQR Report: ISCA Overview
24 Acumentra Health November 2013
Summary of review results
During the review year (January–December
2012), GCBH used a Microsoft SQL database
management system to process encounter data in
house. In October 2012, GCBH replaced the 2005
version with MS SQL Server 2012. The RSN’s
contracted provider agencies used various practice
management systems to collect, process, and
submit encounter data.
At the end of 2012, GCBH began processing
outpatient and inpatient service authorizations in
house, assuming a function previously performed
by an outside contractor.
Since the previous full ISCA in 2011, GCBH has
experienced considerable turnover in IT staffing.
Standard upkeep and maintenance have stalled in
this period of transition, and issues from the
previous ISCA, IT practices, and policies have not
been addressed.
Acumentra Health’s review found that in 2012,
GCBH partially met federal standards related to
data processing procedures and personnel, and
fully met data acquisition capabilities standards.
Table 5 summarizes the ISCA section scores and
ratings.
Table 5. Weighted average scores and ratings on ISCA sections.
Review section/subsection Score Compliance rating
Section 1: Data Processing Procedures and Personnel
A. Information Systems 2.3 Partially met
B. Staffing 2.2 Partially met
C. Hardware Systems 2.6 Fully met
D. Security 1.5 Not met
Section 2: Data Acquisition Capabilities
A. Administrative Data (claims and encounter data) 2.9 Fully met
B. Enrollment Systems (Medicaid eligibility) 2.7 Fully met
C. Vendor Data Integration 2.2 Partially met
D. Provider Data (compensation and profiles) 2.3 Partially met
E. Meaningful Use of Electronic Health Records n.a. n.a.
2013 Greater Columbia Behavioral Health EQR Report: Data Processing Procedures and Personnel
25 Acumentra Health November 2013
ISCA Section 1: Data Processing Procedures and Personnel
Section 1A: Information Systems Score: 2.3 (Partially met)
This subsection reviews the RSN’s systems
development life cycle (SDLC) and supporting
environments, including database management
systems and/or billing software, programming
languages, and training for programmers.
GCBH transitioned in October 2012 from
Microsoft SQL Server 2005 to Microsoft SQL
Server 2012 for Medicaid encounter data
processing, data analysis, and reporting.
GCBH’s customized Microsoft SQL database is
maintained by the contracted consultant who
developed it. The RSN has no trained backup staff
who could take over if the consultant became
unavailable. GCBH has created documentation for
the database, but has not finished developing
operating procedures or training for staff.
GCBH has employed several IS managers since
the previous ISCA. Many IT practices, issues, and
policies have gone unaddressed during this period.
At the time of the 2013 review, the current IS
manager had been employed for about one
calendar quarter.
Recommendations
GCBH lacks a formal, written quality assurance
(QA) process for its information systems for
encounter data processing, analysis, and reporting.
GCBH stated that it has developed a QA process,
but provided no documentation.
GCBH needs to implement a formal,
written QA process for encounter data
processing, analysis, and reporting.
GCBH relies solely on a contracted consultant for
its programming. The RSN does not assign internal
staff to perform programming or to monitor the
code produced by the consultant. The absence of
backup, coupled with incomplete documentation
and training materials, creates a risk of unplanned
and extended disruptions of GCBH’s internal
processes and claims processing.
GCBH needs to develop a formal process
for monitoring of outsourced IT services.
GCBH needs to designate a backup
administrator who could keep the
customized database running if the current
consultant became unavailable.
GCBH does not have formal contracts or business
associate agreements for outsourced IT services.
GCBH needs to formalize its relationships
with providers of outsourced IT services.
Written agreements should specify the
consultant’s roles and responsibilities as
well as any deliverable requirements.
GCBH appears to lack appropriate documentation
to enable staff to understand RSN systems and
business processes.
GCBH needs to develop documentation of
RSN systems and business processes for
the benefit of staff.
GCBH’s computer programming and data report
production processes lack a formal peer review
process to validate the accuracy and completeness
of data before production.
GCBH needs to develop a formal peer
review process for internal programming
and report production.
GCBH lacks a formal SDLC and sign-off process
for code changes to RSN systems.
GCBH needs to develop a formal process
for software development changes,
including peer review, testing, and
authorization for production.
GCBH does not use a formal version control
process for report development.
GCBH should consider implementing a
formal version control process for its
reporting and archiving process.
2013 Greater Columbia Behavioral Health EQR Report: Data Processing Procedures and Personnel
26 Acumentra Health November 2013
GCBH does not use version control management
software for internal reporting. Instead, GCBH
staff renames previous versions of files or
programs, and edits new copy as needed, keeping
previous versions for reference.
GCBH should consider using version
control software to enable a more robust,
quick, and efficient process that is less
prone to error.
Section 1B: Staffing Score: 2.2 (Partially met)
This subsection assesses physical access by the
RSN’s staff to IT assets, as well as specific
training requirements for claims processing and
care authorization staff, and RSN staffing.
GCBH performs authorizations for inpatient and
outpatient mental health services through its
customized in-house CIS. A credentialed
utilization management (UM) specialist performs
all authorization decisions on the basis of Access
to Care standards and other information
submitted by the provider agency.
The review revealed a lack of clarity about team
members’ individual roles and responsibilities, as
well as about backup coverage. Processes were
informal, and team members appeared to operate
in silos, creating potential single points of failure
in their respective areas of expertise.
Staff appeared to lack understanding of many
aspects of the CIS database functionality, which
only the contracted consultant could address.
Many of the RSN’s business processes were
known only to the staff members handling those
responsibilities, and little or no documentation
appeared to exist.
Strengths
GCBH’s credentialed UM specialists
understand locally available resources.
Recommendations
Roles and responsibilities of individual GCBH
staff members appeared unclear.
GCBH needs to clearly define roles and
responsibilities of internal staff and of its
IT consultant.
GCBH needs to continue to monitor its IT
staffing to address underlying concerns,
and train existing staff to provide backup
for key activities.
Section 1C: Hardware Systems Score: 2.6 (Fully met)
This subsection assesses the RSN’s network
infrastructure and hardware systems.
GCBH’s CIS servers reside in the GCBH office
data center. These servers are under warranty until
second quarter 2014. GCBH’s previous IS
manager purchased a new server to implement a
virtualized CIS, but implementation was never
completed. GCBH plans to revisit implementation
plans for this new hardware solution.
Strengths
Servers are housed in a secure location
away from personnel who are not
authorized to have physical access.
GCBH takes advantage of redundant
software and hardware designs.
Recommendations
GCBH’s servers are approaching end of life and
need to be replaced. The new hardware purchased
nearly two years ago has gone unused and may be
nearing the end of its warranty period.
2013 Greater Columbia Behavioral Health EQR Report: Data Processing Procedures and Personnel
27 Acumentra Health November 2013
GCBH needs to pursue it plans to upgrade
to new equipment before the current
servers reach end of life.
GCBH needs to develop and implement
a plan to install or replace purchased
hardware in a manner that meets business
requirements. This hardware has a limited
warranty period and life expectancy, and
likely would need to be replaced soon after
being installed.
Section 1D: Security Score: 1.5 (Not met)
This subsection assesses the RSN’s information
systems for integrity and the ability to prevent
data loss and corruption. A security walkthrough
of the computer area and/or data center assesses
the possibility of a breach in security measures.
GCBH staff performs daily data backups to a
tape-based storage system. Encrypted backup
tapes are transported to a secure offsite location
once a week.
GCBH’s new IS manager has begun updating the
RSN’s Business Continuity/Disaster Recovery
(BC/DR) plan to meet current DBHR contract
requirements. GCBH’s provider agencies reported
that they maintain current disaster recovery plans.
Strengths
All RSN tapes containing protected health
information (PHI) are encrypted.
Findings
RSN Contract Exhibit C. GCBH’s security
policies and procedures have not been updated
recently and do not match current DBHR contract
requirements.
GCBH needs to continue updating its
security policies and procedures to align
with DBHR contract requirements, and
review the policies and procedures at least
annually.
RSN Contract Exhibit C – Data Security
Requirements. 1.b – Encrypting any data that
will be in transit outside the State Governmental
Network (SGN) or contractor’s internal network. At the time of the ISCA review, GCBH had at
least one laptop that was not encrypted. GCBH
reported that the RSN does not permit sensitive
data to be stored on a portable system until the
system has been encrypted. However, reliance on
a manual process by users to properly identify
PHI and avoid storage on unencrypted laptops
should be discouraged. GCBH reported that this
laptop has been encrypted since the site review.
The RSN is identifying encryption solutions for
all portable devices.
GCBH needs to proceed with its plans to
encrypt all laptops and portable devices to
align with DBHR contract requirements.
RSN Contract Exhibit C – Data Security
Requirements. 2.f. GCBH’s practice is to notify
DSHS within five days when an authorized
employee leaves the RSN or is terminated and thus
no longer requires SGN access. This practice does
not align with DBHR’s contractual requirement for
immediate notification.
GCBH needs to notify DSHS immediately
when an authorized employee or contractor
no longer requires access to the SGN.
GCBH’s policy to set devices to lock
automatically after 30 minutes of inactivity does
not align with DBHR contract requirements.
GCBH needs to revise its policy to set
devices to lock automatically after at most
20 minutes of inactivity.
GCBH’s provider agencies use various encryption
strategies for laptop computers, hard drives, and
other portable devices. One agency reported an
insufficient key length. Three agencies reported
that they do not encrypt backups.
2013 Greater Columbia Behavioral Health EQR Report: Data Processing Procedures and Personnel
28 Acumentra Health November 2013
GCBH should review provider agencies’
encryption strategies to ensure that they
are aligned with DBHR contract
requirements, current industry standards,
and HIPAA requirements.
Two provider agencies reported that passwords
used to connect to their networks or encounter data
systems do not require a complexity component.
One agency reported that its passwords expire
every six months.
GCBH needs to work with provider
agencies to ensure that passwords used to
connect to encounter data systems have
unique user IDs, sufficient complexity,
and forced change requirements that align
with DBHR contract requirements and
current industry standards.
RSN Contract 11.2.2: Business Continuity and
Disaster Recovery. GCBH’s BC/DR plan is not
aligned with current contractual requirements,
though the RSN has begun updating the plan.
GCBH needs to continue to align its
BC/DR plan with current contractual
requirements.
RSN Contract 11.2.2: Testing of DR/BC Plan.
GCBH has not tested its BC/DR plan recently.
GCBH needs to establish annual testing of
its BC/DR plan to meet DBHR contract
requirements.
Recommendations
GCBH transports backup tapes containing PHI to
an offsite location in a private vehicle. Although
the tapes are encrypted, this transportation method
is not an industry best practice.
GCBH has an older keypad entry system, used by
about 25 employees. All staff members use the
same entry code, which is changed at the time of
staff departures. This system cannot track individual
access, and is vulnerable to the risk of people
outside of the RSN learning the code.
GCBH should begin planning for a system
with more granular access control and
monitoring capabilities.
2013 Greater Columbia Behavioral Health EQR Report: Data Acquisition Capabilities
29 Acumentra Health November 2013
ISCA Section 2: Data Acquisition Capabilities
Section 2A: Administrative Data Score: 2.9 (Fully met)
This subsection reviews the RSN’s submission of
accurate information, process for describing
differences when verifying accuracy of submitted
claims, and data assessment and retention.
GCBH accepts encounter data from its provider
agencies in electronic format. During processing,
encounter data submissions run through an
automated, rules-based edit system to screen the
data, identify potential input errors, and ensure
compliance with the state’s Data Dictionary and
Service Encounter Reporting Instructions (SERI).
Screened encounter data submissions are
converted to the HIPAA-compliant 837 format
before being transmitted to DBHR via a secure
shell connection. As required by DBHR, GCBH
verifies and certifies batched encounter data for
accuracy and completeness before transmitting
data. GCBH’s director is responsible for ensuring
compliance with state reporting requirements.
Strengths
GCBH has been able to meet DBHR
requirements for submission of encounter
data even with staff turnover issues.
GCBH sends encounter data to the state on
a weekly basis.
Recommendations
GCBH lacks a process to link authorization data
with encounters. As a result, encounters may be
accepted and processed without valid authorization
or reauthorization.
GCBH needs to integrate its authorization
and encounter processes. This should
include clearly defining the process that is
followed if an encounter is submitted to the
RSN for a service without an authorization,
and ensuring that procedures are in place to
prevent the delivery of services that have
not been authorized.
It is unclear whether agencies are capturing and
reporting only the diagnosis assigned at the time of
the enrollee’s intake assessment or reassessment.
The diagnosis at the time of the service encounter
may not reflect the current diagnosis.
GCBH needs to develop a method to
ensure that the diagnosis being treated at
the time of service is reported on the 837.
Section 2B: Enrollment Systems Score: 2.7 (Fully met)
This subsection assesses the RSN’s systems
pertaining to Medicaid enrollment and
disenrollment, tracking claims and encounter data,
Medicaid enrollment data updates, Medicaid
enrollment code, and data verification.
GCBH relies on its provider agencies to check
enrollee eligibility at each service encounter. One
agency does not check eligibility at each service
encounter, but checks eligibility monthly. GCBH
staff verifies eligibility at the time of service
authorization or reauthorization.
GCBH loads the state eligibility data into its CIS
on a regular basis, but it is unclear how the RSN
uses the eligibility data.
Findings
RSN Contract 4.7.1: The Contractor must verify
eligibility for Title XIX prior to the provision of
non-crisis services to an enrollee. GCBH relies
on providers to verify eligibility at the time of
service, and does not perform eligibility checks on
encounter data.
2013 Greater Columbia Behavioral Health EQR Report: Data Acquisition Capabilities
30 Acumentra Health November 2013
GCBH needs to ensure that eligibility
checks occur before submitting encounter
data to the state.
GCBH needs to work with providers to
ensure that they verify enrollment at each
service encounter.
Section 2C: Vendor Data Integration Score: 2.2 (Partially met)
This subsection assesses how the RSN integrates
vendor data with administrative data for data
completeness and quality.
GCBH performs encounter data validation audits
yearly. The proportionate number and sampling of
encounters to be audited for the year is based on
the previous contract year’s ratio of services by
provider agency, program/project code, and
enrollee age category (children age 0–17 and
adults age 18+).
Strengths
GCBH’s provider agencies met contract
requirements for EDV standards.
GCBH has worked with agencies to
address issues that need correction but do
not require a formal corrective action plan.
Recommendations
One provider agency reported that some changes
to encounter data are made by IT staff, and that no
clinical review occurs to verify the completeness
and accuracy of those changes. Some data
changes, such as in progress notes, should be
made only by the rendering practitioner or by an
appropriate clinical supervisor.
GCBH needs to ensure that provider
agencies understand which data changes
need to be verified by clinical staff, and
that agencies implement an appropriate
review process for data changes.
One provider agency reported that it submits
current data on the 1st and the 15
th day of each
month. GCBH must wait until these dates to
obtain the most accurate information. Thus, when
changes to encounter data occur, GCBH’s CIS
data may not be kept current with information in
the providers’ data systems.
GCBH needs to work with provider
agencies to ensure that data changes are
captured in both the RSN and provider
agency data systems.
One provider agency reported difficulty in
interpreting the manual that is intended to help
agencies structure the data they send to GCBH.
According to this agency, the manual’s ambiguity
causes the agency to make several changes to its
systems for a single requirement or field.
GCBH needs to work with provider
agencies to develop a manual with the
details necessary to develop data files
appropriately with little intervention. This
process could include a work group to
determine the level of detail the agencies
need in order to develop these files.
2013 Greater Columbia Behavioral Health EQR Report: Information Systems Capabilities Assessment
31 Acumentra Health November 2013
Section 2D: Provider Data Score: 2.3 (Partially met)
This subsection assesses how the RSN maintains
its provider directory, as well as the RSN’s fee
schedules and contractual payment updates.
GCBH stated that it no longer monitors for
missing encounters as part of EDV activities.
RSN staff members perform limited monitoring
and reporting of the number of encounters
received or expected.
Recommendations
GCBH has an informal process for updating its
provider directories.
GCBH should formally define its process
to update provider directories, including
steps for adding providers, approval of
credentialing, and updates to the CIS to
identify that providers are now authorized
to serve enrollees.
GCBH does not appear to distinguish between
services paid for by Medicaid and those paid by
other funding streams.
GCBH needs to establish appropriate
policies and procedures to distinguish
services paid for by Medicaid vs. those
funded by other payment sources.
Section 2E: Meaningful Use of Electronic Health Records Score: n.a.
This subsection assesses how the RSN and its
contracted providers use electronic health records
(EHRs). Due to the timing of the CMS waiver,
Acumentra Health and DBHR determined that
this section would be reviewed in 2013, but no
scores will be reported this year.
Recommendations
GCBH should develop EHR-related
policies and procedures in advance of
implementation, specifying the RSN’s role
in EHR adoption, expectations during
implementation, and plans for transition
periods when data may not be available.
GCBH should consider testing with
provider data systems during provider
agency EHR implementation.
GCBH should consider monitoring data for
quality, completeness, and accuracy
throughout EHR implementation, including
a post-implementation review.
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
32 Acumentra Health November 2013
ENCOUNTER DATA VALIDATION
Medicaid encounter data must be complete and
accurate to be useful in calculating statewide
performance measures and determining managed
care capitation rates. DBHR’s contract requires
each RSN to conduct an annual encounter data
validation (EDV) to determine the accuracy of
encounter data submitted by providers.
As an independent check of the RSNs’ EDV
results, Acumentra Health audits and verifies the
EDV process for each RSN. DBHR accepts the
RSNs’ self-validation of their encounter data,
subject to audit and verification by Acumentra
Health.
For each RSN, the 2013 EDV involved:
checking each field in the state’s
outpatient electronic data set for missing
and out-of-range data and logic problems
comparing specific data fields in clinical
records of the RSN’s providers against the
state’s electronic data sets to determine
whether data submitted by the providers
were accurate and complete
As a special topic, the 2013 EDV also examined
the degree to which providers’ clinical records
demonstrated adherence to the “golden thread”
of mental health therapy:
1. Does the assessment in the clinical record
substantiate the individual’s diagnosis?
2. Are the documented goals of the treatment
plan consistent with the diagnosis?
3. Do the progress notes address the
individual’s progress toward meeting the
treatment plan goals?
Appendix C presents an overview of Acumentra
Health’s EDV procedures.
Greater Columbia Behavioral Health’s EDV procedure
According to documentation submitted for this
audit, GCBH validated encounter data submitted
by provider agencies to the RSN against the chart
notes during reviews of six provider agencies.
GCBH performs annual EDV activities at a
selection of agencies and creates agency-specific
analyses to summarize the results. The RSN
samples from all encounters that are expected to
be sent to the state. The time frame for the
sample reviewed was six months prior to the
review for each agency, so the time frame was
slightly different for each.
Acumentra Health recommends that for all EDV
activities, the RSN use encounters processed by
the state rather than encounters sent to the state.
That way, the RSN can ensure that its encounter
data are received and processed as expected, and
the RSN will have the opportunity to address any
data errors in a timely manner.
This review first assesses GCBH’s sampling
procedure, then discusses the strengths of the
RSN’s data entry tool and analytical procedures.
Sampling procedure
To evaluate the RSN’s sampling methodology,
Acumentra Health examined the overall sample
size and whether the sample was chosen in a
random process.
GCBH reviewed encounters at six provider
agencies. Six other agencies were excluded from
this annual review due to exemplary review
outcomes in last year’s review, and a seventh
agency was excluded from review due to its
departure from the RSN.
GCBH reviewed 1,126 encounters in 272 charts
for Medicaid enrollees, far more than required by
the DBHR contract.
To be selected in the review sample, enrollees
must have received at least six services within six
months and have received an intake, medication
monitoring/management, and/or crisis service.
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
33 Acumentra Health November 2013
GCBH’s quality manager selected the charts to be
reviewed in a manual random and blind fashion,
lacking information other than age, gender, and
enrollee ID. The sample was selected in
proportion to the age and gender distribution of all
enrollees served by each agency, and sample size
for each agency matched the proportion of clients
served in the RSN.
Acumentra Health recommended in 2012 that
GCBH develop a computerized approach to select
a random sample. A robust computerized random
sampling would:
1. be more efficient in choosing the sample
in a much shorter period of time
2. select chart and encounters over the entire
list of possible choices, not just up to the
point of reaching target numbers and ratios
of gender and age distribution
3. result in a different list of enrollees and
encounters each time the sampling
procedure is run on the same set of data
While the manual procedure may generate a
random sample, a computerized algorithm would
ensure that the three criteria above are met each
time the program is called. Using an appropriate
computerized approach would ensure that each
chart and encounter has an equal probability of
being chosen.
GCBH’s procedure is partially adequate for
choosing an EDV sample. The sample meets the
size requirement, but the random sampling
procedure should be based on a computerized
rather than manual approach.
Note: Following the EDV review, which was
based on a December 2012 report by the RSN,
GCBH reported that it had implemented a
computerized sampling approach in 2013.
Record review tools, process, and results
GCBH uses an Access database to record its
EDV results. The encounters selected by the
random sampling procedure are loaded into the
database, and results are collected on a single
form where the reviewer marks any item that
does not match the chart. After the review is
complete, the data are pulled out of the Access
database and into MS Excel for analysis.
Acumentra Health repeats the recommendation
from 2012 that GCBH improve its data entry tool
to perform the calculations now performed with
Excel. There appears to be a transfer of data back
and forth between Access and Excel. Data are
exported to Excel for purposes of sampling, then
the sample is moved back into Access so results
can be collected, then results are analyzed in
Excel. It would be more efficient to keep the data
in Access and analyze the results using Access
reports that can show agency-level as well as
RSN-level summary statistics.
GCBH reviewed all elements that are required to
be reviewed per the DBHR contract except
“service code agrees with treatment described.”
Acumentra Health recommends reviewing this
element in future EDV activities. GCBH also
reviewed additional demographic elements
(gender, date of birth, ethnicity, Hispanic origin,
language, and sexual orientation) and the “golden
thread” by examining whether the documentation
in the enrollee’s chart was sufficient to
demonstrate medical necessity.
One reviewer performs all EDV reviews and
analysis. However, it is unclear whether GCBH
reviews previous years’ procedures or uses an intra-
rater agreement test to ensure that the reviewer’s
scores remain consistent across agencies and across
time. If this process is not in place, Acumentra
Health recommends implementing it.
Reviewers examined records for Match, No
Match–Erroneous, No Match–Unsubstantiated,
and Insufficient Note. The last item is an RSN-
added element. No Match–Missing was not
reviewed. Acumentra Health recommends
examining charts for “missing” encounters, where
the clinical record contains evidence of a service
not represented in the electronic encounter data.
EDV results are presented at the data element
level, stratified by agency and aggregated by
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
34 Acumentra Health November 2013
error type for the RSN. None of the agencies
reviewed had an error rate that required a
corrective action plan.
GCBH’s record review procedure is adequate for
assessing the accuracy and completeness of the
EDV data.
Validation results
This report presents the EDV results in three
parts: first, the results of electronic data checks;
second, the results of comparing the clinical chart
documentation with the state’s electronic data, as
part of the onsite review; and finally, the results of
the “golden thread” analysis.
Electronic data checks
Acumentra Health analysts checked data fields in
329,813 outpatient encounters for missing and
out-of-range data and logic problems,
representing all outpatient encounters reported by
GCBH during October 2011–September 2012.
The fields examined included RSN ID, consumer
ID, agency ID, primary diagnosis, service date
and location, provider type, procedure code, claim
number, and minutes of service.
All fields were 100% complete and within
expected limits, although 150 claims (0.05%) had
a duplicate Claim ID.
Next, analysts checked the demographic data set,
examining 19,193 records. The fields examined
included RSN ID, consumer ID, first and last
names, date of birth, gender, ethnicity, Hispanic
origin, language preference, Social Security
number (SSN), and sexual orientation.
Considering mandatory fields, analysts found
2,526 records (13.2%) with out-of-range ethnicity
information, indicating that an invalid code was
submitted. Considering optional fields, analysts
found that 4,047 records (21.1%) omitted the
SSN. All other mandatory and optional fields
were 100% complete and within expected limits.
Table 6 summarizes the results of electronic data
checks for the outpatient and demographic data
sets.
Onsite review results
Acumentra Health staff audited 394 encounter
records for GCBH. The encounters were reported
in 107 charts. The data fields compared for each
encounter included procedure code, provider type,
service minutes, service date, and service location.
Acumentra Health staff reviewed the encounter
notes to verify that the procedure code accurately
described the treatment provided, and compared
electronic data from the state’s demographic data
set with the chart documentation for the 107
enrollees. Demographic fields that were compared
included first name, last name, date of birth,
ethnicity, and language.
The choices available to the audit team in
comparing electronic data with the source chart
documentation for each field were:
1. Chart matches state data
2. Data in chart missing from state data
3. Missing from both chart and state data
4. Could not locate in chart
5. Data found in chart do not match state data
Reviewers also compared encounters documented
in the clinical record with encounters in the
electronic data to identify “missing” encounters
that were documented in the clinical record but
were not sent to the state.
Table 7 shows the results of Acumentra Health’s
validation activity.
Within the demographic dataset, the chart
information matched the state data in 99.1% of
records for first and last name, and matched in
100% of records for date of birth. Ethnicity data
found in the chart matched the state data in 83.2%
of records, and language data found in the chart
matched the state data in 91.6% of records.
Of 394 encounters reviewed from the outpatient
data set, 79.6% had a service code that matched
the service described in the chart note. Provider
type, service location, procedure code, and service
date matched in 80% to 93% of encounters
reviewed, while only 38.6% of encounters had
matching data on minutes of service.
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
35 Acumentra Health November 2013
Table 6. Results of 2013 electronic data checks.
Field State standard % completea
Outpatient encounter data
RSN ID 100% complete (non-missing values), with values known to DBHR 100.0
Consumer ID 100% complete (non-missing values), with values known to DBHR 100.0
Agency ID 100% complete (non-missing values) 100.0
Primary diagnosis 100% complete (non-missing values), one diagnosis must be present 100.0
Service date 100% complete (non-missing values), must be in valid date format 100.0
Service location 100% complete (non-missing values), with values specified in data dictionary 100.0
Provider type 100% complete (non-missing values), with values specified in data dictionary 100.0
Procedure code 100% complete (non-missing values), with values specified in service instructions 100.0
Claim number 100% complete (non-missing values) 100.0
Minutes of service 100% complete for records with no per diem CPT/HCPCS codes 100.0
Demographic data
RSN ID 100% complete (non-missing values), with values known to DBHR 100.0
Consumer ID 100% complete (non-missing values) 100.0
First name 100% complete (non-missing values) 100.0
Last name 100% complete (non-missing values) 100.0
Date of birth Optional per the state’s Data Dictionary 100.0
Gender Optional per the state’s Data Dictionary 100.0
Ethnicity 100% complete (non-missing values), with values specified in data dictionary 86.8
Hispanic origin 100% complete (non-missing values) 100.0
Language preference 100% complete (non-missing values) 100.0
Social Security number Optional per the state’s Data Dictionary 78.9
Sexual orientation 100% complete (non-missing values) 100.0
aDue to rounding, some fields showing 100.0 percent completeness may have had a small number of missing data values.
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
36 Acumentra Health November 2013
There is a systematic issue converting service
minutes from the RSN to units required for
reporting to Provider One, which converts units
back to minutes in the data housed by DBHR.
Staff from DBHR are working on this issue.
In May 2012, the only month reviewed,
Acumentra Health reviewers found no “missing”
encounters.
“Golden thread” analysis
This portion of the EDV examines whether the
enrollee’s assessment substantiates the diagnosis,
whether the treatment plan is consistent with the
diagnosis, and whether progress notes address the
treatment plan.
The assessment substantiated the Category A
diagnosis in 89.5% of children’s charts and 89.7%
of adult charts. An additional 5.3% of children’s
charts and 3.4% of adults’ charts contained an
assessment that partially substantiated the
Category A diagnosis.
The Category B diagnosis, when applicable, was
substantiated in 100% of children’s charts. No
adult charts contained a Category B diagnosis.
The treatment plan was consistent with the
diagnosis and assessment in 95.1% of children’s
charts and 67.2% of adults’ charts, while the
treatment plan objectives were individualized in
95.0% of children’s charts and 82.0% of adults’
charts. The progress notes addressed interventions
identified in the treatment plan in 92.9% of
children’s charts and 78.3% of adults’ charts.
An additional 3.3% of adults’ charts partially
addressed interventions identified in the treatment
plan.
Table 8 displays the results of the golden thread
analysis for GCBH. “Not applicable” or blank
responses were removed from the table, so the
denominator for both “% Yes” and “% Partially”
includes only charts that were scored “yes,” “no,”
or “partially.”
Table 7. Results of Acumentra Health’s encounter data validation for Greater Columbia Behavioral Health.
Field Chart matches
state data
Data in chart missing from
state data
Missing from both chart and
state data
Data could not be located
in chart
Data found in chart do not
match state data
Demographic information from each clinical record reviewed (N=107)
First name 106 (99.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (0.9%)
Last name 106 (99.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 1 (0.9%)
Date of birth 107 (100.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%)
Ethnicity 89 (83.2%) 0 (0.0%) 0 (0.0%) 2 (1.9%) 16 (14.9%)
Language 98 (91.6%) 0 (0.0%) 0 (0.0%) 2 (1.9%) 7 (6.5%)
Results from multiple encounters and a mix of services (N=394)
Provider type 353 (89.6%) 0 (0.0%) 0 (0.0%) 36 (9.1%) 5 (1.3%)
Minutes of service 152 (38.6%) 2 (0.5%) 0 (0.0%) 51 (12.9%) 189 (48.0%)
Service location 354 (89.9%) 0 (0.0%) 0 (0.0%) 34 (8.6%) 6 (1.5%)
Procedure code 316 (80.2%) 0 (0.0%) 0 (0.0%) 67 (17.0%) 11 (2.8%)
Service date 365 (92.9%) 0 (0.0%) 0 (0.0%) 28 (7.1%) 0 (0.0%)
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
37 Acumentra Health November 2013
Table 8. Results of “golden thread” analysis for Greater Columbia Behavioral Health.
Assessment substantiates the diagnosis; treatment plan is consistent with the diagnosis; progress notes address the treatment plan
Number of charts reviewed (N=102)
Children Adults Total
% “Yes” % “Partially” % “Yes” % “Partially” % “Yes” % “Partially”
1. Does the assessment substantiate the Category A diagnosis (if applicable)?
34 (89.5%) 2 (5.3%) 52 (89.7%) 2 (3.4%) 86 (89.6%) 4 (4.2%)
2. Does the assessment substantiate the Category B diagnosis (if applicable)?
4 (100.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 4 (100.0%) 0 (0.0%)
3. Does the treatment plan include interventions and goals consistent with issues identified in the assessment?
39 (95.1%) 0 (0.0%) 41 (67.2%) 0 (0.0%) 80 (78.4%) 0 (0.0%)
4. Do the progress notes address interventions identified in the treatment plan and the individual’s progress toward meeting stated goals?
39 (92.9%) 0 (0.0%) 47 (78.3%) 2 (3.3%) 86 (84.3%) 2 (2.0%)
5. Are the treatment plan objectives individualized? 38 (95.0%) 0 (0.0%) 50 (82.0%) 0 (0.0%) 88 (87.1%) 0 (0.0%)
NOTE: Proportions exclude “not applicable” or missing responses, so the denominator of each item may vary.
2013 Greater Columbia Behavioral Health EQR Report: Encounter Data Validation
38 Acumentra Health November 2013
Discussion and recommendations
Acumentra Health finds GCBH’s EDV sampling
procedure partially adequate. The sample was an
appropriate size, but the randomization process
was manual, and Acumentra Health recommends
that GCBH develop a computerized algorithm to
select a random sample of encounters, for the
reasons noted above. The record review procedure
was adequate for assessing the accuracy and
completeness of the EDV data. We recommend
that GCBH develop the database data entry tool to
include reports that can show the summary
statistics presently calculated in Excel.
The quality of the electronic data file from DBHR
was generally good, except that 13.2% of records
contained out-of-range ethnicity data. Acumentra
Health recommends that GCBH determine how
150 claim IDs were duplicated so this type of
error can be prevented in the future.
Acumentra Health compared the above EDV
results with the results of the 2010 and 2008
encounter data validation. Among demographic
variables, there was an increase in the percentage
of records that matched the chart data in the date
of birth field, but a decrease in the percentage of
records that matched in the ethnicity field. There
was no change in the match rate for first and last
name, while language was not assessed in
previous years. Among the encounter data
variables, there was a decrease in the percentage
of records that matched the chart data in the
provider type, minutes of service, procedure code,
and service code matching the service described
in the chart note. There was no change in the
match rate for service location, while service date
was not assessed in previous years.
Golden Thread analysis
Assessments. Most mental health assessments
were well written for both children and adults.
They included documentation of the presenting
problem, medical necessity for treatment, clinical
formulation for diagnosis, and services.
Treatment plans. Although the majority of the
children’s treatment plans included interventions
and goals consistent with issues identified in the
assessment, only 67% of the adult treatment plans
included interventions, services, and measurable
goals that were consistent with the mental health
assessment.
GCBH needs to provide guidance to the
agencies to ensure that treatment plans
specify measurable goals, expected
outcomes, service(s) indicated, and
specific interventions consistent with the
enrollee’s assessment.
Progress notes. The progress notes for children
were consistent and well written. They typically
addressed interventions identified in the treatment
plan and the child’s progress toward meeting the
stated goals. However, as with the treatment
plans, the adult progress notes showed room for
improvement, as they did not consistently address
interventions identified in the treatment plan and
progress toward meeting the stated goals.
GCBH needs to provide direction to the
agencies to ensure that progress notes
clearly document the goals, treatment
interventions, enrollee’s response to the
interventions, and progress toward
meeting the goals.
2013 Greater Columbia Behavioral Health EQR Report: Clinical Record Review
39 Acumentra Health November 2013
CLINICAL RECORD REVIEW
In conjunction with the 2013 EDV for GCBH,
Acumentra Health reviewed clinical records at
four outpatient provider agencies to assess mental
healthcare criteria as directed by DBHR. The
clinical record study focused on the degree to
which the RSN’s system of care adhered to the
principles of the Children’s Mental Health
Redesign, including the principles of the interim
settlement of T.R. v. Dreyfus, regarding uniform
screening and assessment of children with serious
emotional disturbances.
The 43 charts reviewed for this activity were the
same as those requested for the EDV. The sample
included consumers age 4–18 served during
October 2011–September 2012, each of whom
had at least three outpatient service encounters
during the review period.
To ensure consistency in reviewing the clinical
records, Acumentra Health followed rigorous
procedures to ensure inter-rater reliability. Before
conducting the review at any RSN, Acumentra
Health trained all reviewers to use a customized
data collection tool and scoring criteria and
guidelines approved by DBHR.
The data collection tool prompted reviewers to
complete a series of questions concerning aspects
of adherence to the T.R. v. Dreyfus principles.
After examining the clinical record (chart) and
progress notes, reviewers recorded responses to
each question in the tool. Using the SAS Proc
Freq function, analysts calculated the distribution
of responses for each question.
Review results
In 71.8% of charts reviewed, an assessment had
been completed within the last year; 23.1% of
assessments had been completed between one and
three years previously, and 5.1% had been
completed three to five years previously.
Within the assessment, 93.0% of records assessed
the child’s living environment and safety needs.
The child’s support systems (95.4%) and activities
outside of the home (97.7%) were assessed in
nearly all records. The assessments usually
considered developmental, learning, or sensory
impairment (90.9%) and language needs (80.0%),
but considered cultural issues less often (63.6%).
The assessment fully justified the diagnosis in
90.7% of records and partially justified the
diagnosis in an additional 4.7% of records.
Nearly all records reviewed contained a treatment
plan that built on strengths to promote resiliency
(92.7%), contained individualized objectives
(95.0%), and included interventions and goals that
were consistent with the assessment (95.1%).
Only 36.8% of treatment plans prescribed team-
based services, and only half (51.2%) had a plan
for case closure.
Nearly two-thirds of treatment plans included
coordination with agencies and others identified
in the assessment, but only 54.5% of progress
notes documented care coordination.
Unconditional treatment was documented in all
progress notes; services were strength-based in
95.4% of records; and progress notes were fully
outcome-based in 80.0% of records, with an
additional 15.0% of records containing progress
notes that were partially outcome-based.
Discussion and recommendations
Assessments: Clinicians at GCBH’s provider
agencies did a good job of documenting the
child’s home environment and systems of support.
The percentage of applicable assessments that
addressed developmental and sensory impairment,
cultural and language issues, and justifying the
diagnosis all fell within the acceptable range.
However, nearly one-third of the assessments were
more than one year old.
Acumentra Health recommends updating
enrollee assessments at least annually to
document changes in the enrollee’s
functioning and life circumstances.
2013 Greater Columbia Behavioral Health EQR Report: Clinical Record Review
40 Acumentra Health November 2013
Treatment plans: Most treatment plans reflected
information included in the assessments. The
majority of treatment goals were based on the
children’s strengths. Most records documented
involvement of family members and support
systems in treatment. However, only 64%of the
treatment plans incorporated coordination with
other agencies into the treatment objectives.
GCBH should ensure that agencies
work with their clinicians to coordinate
care with other agencies involved in the
child’s life.
Nearly two-thirds of the treatment plans did not
include a multi-disciplinary team-based approach
to treatment.
GCBH needs to offer guidance to the
agencies to ensure that children’s
treatment includes a team-based
approach when necessary.
Progress notes: All progress notes demonstrated
that the child received unconditional treatment.
The majority of progress notes documented use of
strength-based interventions and activities as
identified in the treatment plans. Team-based
services and coordination with other agencies
were missing from many progress notes. Only
55% of progress notes indicated that services were
team-based and that care was coordinated with
other agencies and systems.
GCBH needs to ensure that services for
children are team-based and that care
coordination occurs. These services
must be documented in the progress
notes to reflect a multi-disciplinary
team-based approach.
2013 Greater Columbia Behavioral Health EQR Report: Clinical Record Review
41 Acumentra Health November 2013
Table 9. Results of clinical record review for Greater Columbia Behavioral Health.
Assessment, treatment plan, and progress notes indicate adherence to principles of Children’s Mental Health Redesign
Number of charts reviewed = 43
Assessment % Yes % No
1. Is there a completed assessment within the last year? 71.8 28.2
Assessment includes: % Yes % Partially
1. Living environment and safety needs 93.0
2. Documentation of current living situation
Home (parental) 76.2
Therapeutic foster home 2.4
Other friend/family home 21.4
3. Child’s/family’s natural systems of support 95.4
4. Development, learning, or sensory impairment 90.9
5. Cultural issues that may affect treatment 63.6
6. Language needs taken into consideration 80.0
7. Child/family involvement in activities outside of the home 97.7
8. Justification of diagnosis 90.7 4.7
Treatment plan includes: % Yes % Partially
1. Activities and interventions that build on strengths to promote resiliency 92.7
2. Treatment plan objectives are individualized 95.0
3. Documentation showing family/guardian participation in developing the treatment plan 82.1 2.6
4. Coordination with agencies and collaboration with others identified in assessment 63.6
5. Interventions and goals consistent with issues identified in assessment 95.1
6. Team-based services 36.8
7. Case closure 51.2
Progress notes include: % Yes % Partially
1. Interventions identified in the treatment plan and progress toward meeting stated goals 92.9
2. Unconditional treatment 100.0
3. Documentation that services delivered are strength-based 95.4
4. Progress notes care coordination with agencies and systems 54.5
5. Outcome-based progress notes 80.0 15.0
NOTE: Proportions exclude “not applicable” or missing responses, so the denominator of each item may vary.
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-1 Acumentra Health November 2013
APPENDIX A: PIP VALIDATION AND SCORING PROTOCOL
All managed care organizations that serve Medicaid enrollees must conduct two performance
improvement projects (PIPs) each year aimed at improving enrollee health outcomes or processes of
care. The PIPs are validated each year by external quality review to ensure that they are designed,
conducted, and reported according to sound methods.
Acumentra Health’s PIP validation protocol is based on the PIP validation protocol published by the
Centers for Medicare & Medicaid Services (CMS). The most recent version of the CMS protocol,
published in September 2012, added a requirement related to cultural competency. For 2013, Acumentra
Health added a criterion to the DBHR PIP Review Tool to address this requirement:
6.4 Discuss how the intervention services and materials are culturally and linguistically
appropriate.
Other changes in the 2012 CMS protocol reflect a new emphasis on certain aspects of the study design.
Acumentra Health will modify the PIP Review Tool for 2014 to conform to the new areas of emphasis
in the CMS protocol.
The 10 validation standards adapted by Acumentra Health from the CMS protocol define critical
elements in a PIP study design. Specific criteria for each standard are listed on the following pages.
Standard 1. Study Topic
Standard 2. Study Question
Standard 3. Study Indicator
Standard 4. Denominator (Study Population) Data Collection
Standard 5. Numerator (What Is Being Measured) Data Collection and Analysis Plan
Standard 6. Study Intervention
Standard 7. Study Results
Standard 8. Interpretation of Study Results
Standard 9. Study Modifications After the First Remeasurement
Standard 10. Overall Study Results
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-2 Acumentra Health November 2013
Standard 1: Study Topic
To meet Standard 1, the RSN needs to establish the importance of the study topic in general and present
local data to demonstrate that the topic applies to a large or high-risk portion of the Medicaid
population; and demonstrate that a systematic selection and prioritization process was used in choosing
the topic.
Please explain:
1.1 The importance of the study topic in general.
1.2 How the study topic is relevant to your local Medicaid population.
1.3 How you identified the study topic (e.g., quality committee, focus group, grievances, QAPI
activities, other sources).
1.4 Why you prioritized this topic, including considerations of quality (e.g., high risk, prevalent
issue) and feasibility (e.g., data and resource availability).
1.5 How the study topic relates to enrollee outcomes, satisfaction, or quality of care.
Standard 2: Study Question
To meet Standard 2, the RSN needs to present a study question that provides a clear framework for data
collection, analysis, and interpretation. The study question should refer to the proposed intervention, a
study population (denominator), a measure (numerator), a metric (e.g., average, percentage), and a
direction of desired change.
2.1 Please state your study question. A complete study question includes an intervention, a study
population (denominator), what you are measuring (numerator), a metric (percent or average),
and a desired direction of change (increase or decrease). If you have more than one study
indicator, you should present a separate study question for each study indicator.
Standard 3: Study Indicator
To meet Standard 3, the RSN needs to define the measure (numerator) and study population
(denominator); define key terms; and discuss the basis for adopting the indicator as a valid proxy for
enrollee outcomes, satisfaction, or quality of care.
Please define the following elements for each study indicator:
3.1 The denominator (study population), and continuous enrollment criteria if applicable.
3.2 The numerator (what is being measured), including the event or enrollee characteristics that
qualify for the numerator.
3.3 All relevant terms, CPT codes, diagnosis codes, etc., associated with the study indicator.
3.4 Describe why you selected the study indicator. Your description should include a discussion of:
a. The validity of the study indicator (i.e., HEDIS, commonly accepted measures, research
literature, etc.).
b. How the indicator measures enrollee outcomes, satisfaction, or quality of care either directly
or indirectly through a process which is closely related to enrollee outcomes or satisfaction.
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-3 Acumentra Health November 2013
Standard 4: Denominator (Study Population) Data Collection
To meet Standard 4, the RSN needs to list all inclusion and exclusion criteria for the study population;
document all data sources, including fields, codes, and calculations; and describe data validation
procedures. If a sample is selected, the RSN needs to describe the sampling methods.
Describe your data sources.
4.1 List the inclusion criteria for the denominator (study population) and name each data element
and its source, table, field, calculation (if applicable), and relevant codes.
4.2 List all exclusion criteria for the denominator (study population) and name each data element and
its source, table, field, calculation (if applicable), and relevant codes. You do not need to list the
inverse of the inclusion criteria as exclusions.
4.3 Describe data validation procedures for each data element.
4.4 If you used a sample, describe the sampling methodology and a justification for the sample size.
Standard 5: Numerator (What Is Being Measured) Data Collection and Analysis Plan
To meet Standard 5, the RSN needs to list all inclusion and exclusion criteria for the numerator (what is
being measured); document all data sources, including fields, codes, and calculations; describe data
validation procedures; and present a clear data analysis plan, including time frames for the
measurement and intervention periods, and an appropriate statistical test to measure differences
between the baseline and remeasurement periods.
Describe your data sources.
5.1 List the study inclusion criteria for the numerator (what is being measured) and name each data
element and its source, table, field, calculation (if applicable), and relevant codes.
5.2 List all exclusion criteria for the numerator (what is being measured) and name each data
element and its source, table, field, calculation (if applicable), and relevant codes. You do not
need to list the inverse of the inclusion criteria as exclusions.
5.3 Describe data validation procedures for each data element.
5.4 Document clear study measurement periods. The baseline period should end before the start date
of the intervention. The first remeasurement period should not begin before the start date of the
intervention. The intervention and remeasurement periods may run concurrently.
5.5 Document a data analysis plan that includes an appropriate statistical test, rationale for selecting
the test, and a probability level. If you have more than one study indicator, you should document
a separate data analysis plan for each indicator.
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-4 Acumentra Health November 2013
Standard 6: Study Intervention
To meet Standard 6, the RSN needs to select an improvement strategy that will affect a wide range of
enrollees or a high-risk enrollee population, and that is reasonably expected to result in measurable
improvement. The RSN needs to discuss the basis for adopting the intervention; document the
implementation of the intervention, including dates and locations of principal activities; discuss cultural
competence; and track how effectively the intervention was implemented.
6.1 Describe the intervention strategy. Once intervention activities begin, please provide updated
details, including dates and locations.
6.2 Describe why you selected this particular intervention; for example, because it is based on
barriers identified in your system or because it is an evidence-based practice. It should be clear
how the intervention strategy is expected to improve the study indicators.
6.3 Describe how you will track the implementation of the intervention (i.e., how you will know
whether all aspects of the intervention were implemented successfully). If the intervention has
already been implemented, report on the results of your tracking.
6.4 Discuss how the intervention services and materials are culturally and linguistically appropriate.
Standard 7: Study Results
To meet Standard 7, the RSN needs to present results according to the data analysis plan, including the
study indicator, the original data used to compute the indicator, and a statistical test to measure
differences between the baseline and remeasurement periods; and discuss how the intervention
influenced the results.
7.1 Present raw data for the numerator (what you are measuring) and denominator (study population)
as well as the calculated study indicator for the baseline and first remeasurement periods.
7.2 Present the results of your statistical analysis comparing baseline data to the first remeasurement
data. Report the probability level to determine whether or not there is a statistically significant
difference.
7.3 Discuss how the intervention influenced the study results.
Standard 8: Interpretation of Study Results
To meet Standard 8, the RSN needs to assess whether any reported improvement is “real” by
documenting that baseline and remeasurement data were collected using the same methods and are
comparable; discuss the statistical and clinical significance of the study results; address barriers to
improvement and lessons learned during the PIP process; and identify confounding factors that may
have affected the results.
Discuss the following:
8.1 Whether the PIP resulted in real statistical and clinical improvement.
8.2 Any barriers to improvement or lessons learned during the PIP process.
8.3 Whether there were any changes in methodology or inconsistencies in measurement periods and,
if so, whether measurement periods are comparable.
8.4 Any confounding factors that may have affected the PIP results.
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-5 Acumentra Health November 2013
Standard 9: Study Modifications After the First Remeasurement
To meet Standard 9, the RSN needs to document modifications to the intervention, or added
interventions, planned or implemented after the first remeasurement period; and discuss changes in
other aspects of the PIP based on lessons learned from data analysis or barrier analysis.
9.1 Discuss how you addressed the identified barriers and describe any other modifications you
made to the PIP after the first remeasurement period.
Standard 10: Overall Study Results
To meet Standard 10, the RSN needs to report complete study results for two or more measurement
periods, including the study indicator, original data used to compute the indicator, and a statistical test
of group differences; and interpret the statistical and clinical significance of the overall results, discuss
lessons learned, and determine if goals were met and sustained improvement was achieved.
10.1 Present raw data for the numerator (what you are measuring) and denominator (study
population), and the calculated study indicator for the baseline and the second remeasurement.
10.2 Present the results of a statistical analysis comparing baseline data to the second remeasurement
data. Report the probability level to determine whether or not there is a statistically significant
difference.
10.3 Interpret whether the PIP resulted in sustained statistical and clinical improvement over multiple
remeasurement periods.
10.4 Draw a conclusion about whether the PIP was successful overall. Discuss lessons learned during
the PIP process, whether you met your goals for this PIP overall, and the factors that contributed
to whether the PIP achieved sustained improvement.
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-6 Acumentra Health November 2013
PIP scoring
Acumentra Health assigns a score to each PIP standard to measure compliance with federal standards.
Each standard has a potential score of 100 points, as shown in Table A-1.
Table A-1. Compliance rating for PIP standards by point range.
Rating Definition Points
Fully met Meets or exceeds the essential criteria 100
Substantially met Meets essential criteria, has minor deficiencies 75–99
Partially met Meets criteria with deficiencies in some areas 50–74
Minimally met Marginally meets criteria 25–49
Not met Does not meet essential criteria 0–24
The scores for each standard are weighted and combined to determine the overall PIP score. The
maximum overall score is 90 points for Standards 1–8, and 100 points for Standards 1–10, as shown in
Table A-2.
Table A-2. Weighting of points on PIP standards in the overall PIP score.
Standard Scoring weight
1 Study Topic 20%
2 Study Question 10%
3 Study Indicator 10%
4 Denominator (Study Population) Data Collection 10%
5 Numerator (What Is Being Measured) Data Collection and Analysis Plan 10%
6 Study Intervention 10%
7 Study Results 10%
8 Interpretation of Study Results 10%
Demonstrable Improvement Score 90%
9 Study Modifications After the First Remeasurement 5%
10 Overall Study Results 5%
Sustained Improvement Score 10%
Overall PIP Score 100%
2013 Greater Columbia Behavioral Health EQR Report: Appendix A: PIP Validation and Scoring Protocol
A-7 Acumentra Health November 2013
The overall PIP score corresponds to a compliance rating that ranges from Fully Met to Not Met.
Table A-3 shows the compliance ratings and associated scoring ranges for PIPs graded on the 90-point
and the 100-point scale.
Table A-3. Compliance rating for PIPs by overall score.
Compliance rating Description
100-point scale
90-point scale
Fully met Meets or exceeds all requirements 80–100 72–90
Substantially met Meets essential requirements, has minor deficiencies
60–79 54–71
Partially met Meets essential requirements in most, but not all areas
40–59 36–53
Minimally met Marginally meets requirements 20–39 18–35
Not met Does not meet essential requirements 0–19 0–17
Table A-4 shows an example scoring calculation for a PIP on Standards 1–8 for demonstrable
improvement, and on Standards 1–10 for sustained improvement.
Table A-4. Scoring worksheet example.
Standard Compliance rating Assigned
points Weight Overall score
1 Fully met 100 20% 20.00
2 Fully met 100 10% 10.00
3 Partially met 50 10% 5.00
4 Partially met 50 10% 5.00
5 Fully met 100 10% 10.00
6 Minimally met 25 10% 2.50
7 Partially met 50 10% 5.00
8 Partially met 50 10% 5.00
Overall score 1–8 Substantially met 62.50
9 Substantially met 75 5% 3.75
10 Partially met 50 5% 2.50
Overall score 1–10 Substantially met 68.75
2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology
B-1 Acumentra Health November 2013
APPENDIX B: ISCA METHODOLOGY
The Information Systems Capabilities Assessment (ISCA) examines each RSN’s information systems
and data processing/reporting procedures to determine the extent to which they support the production of
valid and reliable state performance measures and the capacity to manage care of RSN enrollees.
The ISCA protocol for the Washington RSNs is adapted from the protocol published by the Centers for
Medicare & Medicaid Services (CMS). The review process consists of four phases:
Phase 1: Collect standard information about RSN information systems. The RSN completes the
ISCA data collection tool (ISCA-T) provided by Acumentra Health before the onsite review.
Acumentra Health also asks the RSN to submit other relevant documents at this time.
Phase 2: Acumentra Health reviews the completed ISCA-T and accompanying documents. Where
an answer seems incomplete or indicates an inadequate process, Acumentra Health marks that
section for follow-up.
Phase 3: Data center security walkthrough and in-depth interviews with knowledgeable RSN staff.
Provider agency interviews, also performed at this time, ask about each agency’s information
systems, encounter/claims processing, and handling of enrollment data.
Phase 4: Post-onsite analysis of findings about the RSN’s information systems and the implications
of the findings regarding:
1. completeness and accuracy of claims and encounter data collected and submitted to DBHR
2. the RSN’s capacity to conduct quality assurance/performance improvement initiatives
3. the RSN’s capacity to oversee and manage the delivery of health care to its enrollees
The following pages discuss the specific criteria for assessing compliance with each review standard.
Section 1: Data Processing Procedures and Personnel
Section 1A: Information Systems
This section provides a detailed review of the RSN’s systems development life cycle (SDLC) and
supporting environments, including database management systems and/or billing software,
programming languages, and training for programmers.
A data storage and processing system that facilitates valid and reliable performance measurement would
have the following characteristics:
flexible data structures
no degradation of processing with increased data volume
adequate programming staff
reasonable processing and coding time
ease of interoperability with other database systems
data security via user authentication and permission levels
data locking capability
proactive response to changes in encounter and enrollment criteria
adherence to the federally required format for electronic submission of encounter data
2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology
B-2 Acumentra Health November 2013
To ensure accurate and complete performance measure calculation, best practices in computer
programming include:
good documentation
clear, continuous communication between the client and the programmers on client information
needs (e.g., analysis needs, reports)
a quality assurance process
version control
continuous professional development of programming staff
Section 1B: Staffing
This section assesses physical access by the RSN’s staff to IT assets, as well as specific training
requirements for claims processing staff.
Best practices for sustaining quality in processing encounter data include
adequately trained staff for processing and tracking errors in encounter data submission
a comprehensive, documented formal training process for new hires and experienced
professionals
refresher courses for staff when updates occur and when new systems are implemented
established and monitored productivity goals
low staff turnover
Section 1C: Hardware Systems
This section assesses the RSN’s network infrastructure and hardware systems.
Best practices for sustaining quality hardware systems include
infrastructural support that includes maintenance and timely replacement of computer equipment
and software, disaster recovery procedures, adequate training of support staff, and a secure
computing environment
redundancy or duplication of critical components of a hardware system with the intention of
increasing reliability of the system, usually in the case of a backup or fail-safe
Section 1D: Security
This section assesses the RSN’s information systems for integrity and the ability to prevent data loss and
corruption. A security walkthrough of the computer area and/or data center assesses the possibility of a
breach in security measures.
Best practices for securing data are summarized below.
A well-run security management program includes IT governance, risk assessment, policy
development, policy dissemination, and monitoring. Each of these activities should flow into the
next to ensure that policies remain current and that important risks are addressed.
Computer systems and terminals should be protected from unauthorized access through use of a
password system and security screens. Passwords should be changed frequently and reset
whenever an employee terminates.
2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology
B-3 Acumentra Health November 2013
Paper-based claims and encounters should be in locked storage facilities when not in use.
Data transferred between systems/locations should be encrypted.
A comprehensive backup plan includes scheduling, rotation, verification, retention, and storage
of backups to provide additional security in the event of a system crash or compromised integrity
of the data. Managers responsible for processing claims and encounter data must be
knowledgeable of their backup schedules and of retention of backups to ensure data integrity.
To ensure integrity, backups should be verified periodically by performing a “restore” and
comparing the results. Ideally, annual backups would be kept for seven years or more in an
offsite climate-controlled facility.
Databases and database updates should include transaction management, commits, and rollbacks.
Transaction management is useful when making multiple changes in the database to ensure that
all changes work without errors before finalizing the changes. A database commit is a command
for committing a permanent change or update to the database. A rollback is a method for
tracking changes before they have been physically committed to disk. This prevents corruption
of the database during a sudden crash or some other unintentional intervention.
Formal controls in the form of batch control sheets or assignment of a batch control number
should be used to ensure a full accounting of all claims received.
Section 11.2 of DBHR’s 2011–2013 RSN contract presents requirements related to Business Continuity
and Disaster Recovery (BC/DR). The contractor must certify annually that a BC/DR plan is in place for
both the contractor and subcontractors. The certification must indicate that the plans are up to date and
that the system and data backup and recovery procedures have been tested. The plan must address:
a mission or scope statement
an appointed IS disaster recovery staff
provisions for backup of key personnel, identified emergency procedures, visibly listed
emergency telephone numbers
procedures for allowing effective communication with hardware and software vendors
confirmation of updated system and operations documentation, process for frequent backup of
systems and data
offsite storage of system and data backups, ability to recover data and systems from backup files
designated recovery options that may include use of a hot or cold site
evidence that disaster recovery tests or drills have been performed
Exhibit C of the 2011–2013 RSN contract presents detailed requirements for data security, including:
1. data protection during electronic transport, including via email and the public Internet
2. safeguarding access to data stored on hard media (hard disk drives, network server disks, and
optical discs), on paper, or on portable devices or media, and access to data used interactively
over the State Governmental Network
3. segregation of DSHS data from non-DSHS data to ensure that all DSHS data can be identified
for return or destruction, and to aid in determining whether DSHS data has or may have been
compromised in the event of a security breach
4. data disposition (return to DSHS or destruction) when the contracted work has been completed
or when data no longer needed
5. notification of DSHS in the event of compromise or potential compromise of DSHS shared data
6. sharing of DSHS data with subcontractors
2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology
B-4 Acumentra Health November 2013
Section 2: Data Acquisition Capabilities
Section 2A: Administrative Data
This section provides a detailed review of the RSN’s submission of accurate information, process for
describing differences when verifying accuracy of submitted claims, and data assessment and retention.
To ensure the validity and timeliness of the encounter and claims data used in calculating performance
measures, it is important to have documented standards, a formal quality assurance of input data sources
and transactional systems, and readily available historical data.
Best practices include:
automated edit and validity checks of procedure and diagnosis code fields, timely filing,
eligibility verification, authorization, referral management, and a process to remove duplicate
claims and encounters
a documented formal procedure for rectifying encounter data submitted with one or more
required fields missing, incomplete, or invalid. Ideally, the data processor would not alter the
data until receiving written notification via a paper claim or from the provider.
periodic audits of randomly selected records conducted internally and externally by an outside
vendor to ensure data integrity and validity. Audits are critical after major system upgrades or
code changes.
multiple diagnosis codes and procedure codes for each encounter record, distinguishing clearly
between primary and secondary diagnoses
efficient data transfer (frequent batch processing) to minimize processing lags that can affect
data completeness
Section 2B: Enrollment Systems
This section assesses the RSN’s Medicaid enrollment systems pertaining to enrollment and
disenrollment processes, tracking claims and encounter data, Medicaid enrollment data updates,
Medicaid enrollment code, and data verification.
Timely and accurate eligibility data are paramount in providing high-quality care and for monitoring
services reported in utilization reports.
Best practices are summarized below.
Access to up-to-date eligibility data should be easy and fast.
Enrollment data should be updated daily or in real time.
The enrollment system should be capable of tracking an enrollee’s entire history within the RSN,
further enhancing the accuracy of the data.
Section 2C: Vendor Data Integrity
This section assesses how the RSN integrates vendor data with administrative data for completeness of
data and quality of data.
An ideal vendor data integration system includes:
converting data, including code sets, for compatibility with the state’s data systems
2013 Greater Columbia Behavioral Health EQR Report: Appendix B: ISCA Methodology
B-5 Acumentra Health November 2013
receiving only member-level data, as opposed to aggregate data
incorporating other data (e.g., dental care, primary care manager, history of care) to provide a
more complete picture of a member’s care
ensuring consistency in the data for required fields, including multiple diagnosis and procedures
codes
Section 2D: Provider Data
This section assesses how the RSN maintains its provider directory, as well as the RSN’s fee schedules
and contractual payment updates.
An RSN designs its provider compensation structure to balance contractual expectations, the needs of
enrolled populations, and capitation rates set by the state. To set appropriate capitation rates, the state
relies on accurate and timely encounter data.
A good payment structure is critical to ensure reasonable and timely compensation, which encourages an
accessible, qualified community network of providers to continue to provide service to Medicaid
enrollees.
An ideal provider profile directory, which allows enrollees and staff to make informed choices, would
list all available providers, including their gender, credentials and specialties, languages spoken, whether
they use sign language, whether they offer interpretive services, and whether the office is ADA-
certified.
Section 2E: Meaningful Use of Electronic Health Records
This section requests information on how the RSN and its contracted providers use electronic health
records (EHRs), including:
any planning and/or development efforts the RSN has taken toward adopting and using a
certified EHR system
number of providers in the RSN network currently using EHRs
if providers are using EHRs, whether the RSN has sought to determine whether the technology
has been certified by the appropriate federal body
any training, education, or outreach the RSN has delivered to network providers on the
meaningful use of certified EHR technology
whether the RSN uses data from EHRs as part of its quality improvement program (e.g., to
improve the quality of services delivered or to develop PIPs)
strategies or policies the RSN has developed to encourage the adoption of EHR by providers that
are not eligible for the Medicaid Incentive Program
2013 Greater Columbia Behavioral Health EQR Report: Appendix C: Encounter Data Validation Procedures
C-1 Acumentra Health November 2013
APPENDIX C: ENCOUNTER DATA VALIDATION PROCEDURES
In validating RSN encounter data, Acumentra Health follows the steps outlined below, based on the
CMS protocol, Validating Encounter Data.
1. Review the state’s requirements for collecting, processing, and submitting encounter data, based
on specifications in the RSN contract, the state’s data dictionary, and other information furnished
by the state.
2. Review results of the previous EDV study to identify follow-up needs.
3. Review the capability of each RSN’s information system to capture accurate and complete
encounter data, drawing on findings of the ISCA review and interviews with RSN personnel.
4. Analyze electronic encounter data to establish the magnitude of missing data, types of potentially
missing data, overall data quality issues, and problems with how the RSN compiles and submits
encounters to the state. Subtasks include:
Apply general edit and consistency checks, such as verifying that critical fields contain
values that are consistent across fields.
Inspect data fields for general validity, including a review of each data element and of the
volume of data by type or place of service.
Using standard statistical procedures, analyze data to obtain a validity overview of the RSN’s
encounter data. This step involves analyzing and interpreting the data in submitted fields, the
volume and consistency of encounter data, and utilization rates, both overall and by specific
diagnosis, procedure, service, and provider types.
Compare the RSN’s encounter data with state standards and/or benchmarks.
5. If necessary, review clinical records to confirm findings of the above analysis.
Acumentra Health reviewed each RSN’s internal EDV activities in 2013, but unlike in 2012, did not
recreate the EDV calculations. Based solely on the information provided in the RSN’s EDV report,
Acumentra Health assessed whether the RSN’s EDV tool, sampling procedures, EDV process, and
subsequent results were adequate for assessing the accuracy and completeness of the EDV data.
Acumentra Health reviewed a sample of enrollee charts to ensure that the information in the encounter
data matched the information in the charts. The following data elements were scored:
first name
last name
date of birth
ethnicity
language
provider type
minutes of service
service location
service date
procedure code
whether the service code matched the chart note
2013 Greater Columbia Behavioral Health EQR Report: Appendix C: Encounter Data Validation Procedures
C-2 Acumentra Health November 2013
Scoring options included:
Match: cases where there is an exact match of all the minimum data elements for each randomly
selected sample between the subcontractor’s encounters and those in the clinical record
No Match: cases where the subcontractor’s encounters do not match the clinical records.
o Erroneous: Encounters occurred and are presented in the clinical record but contain incorrect
data or omit any of the minimum data elements.
o Unsubstantiated (not in the medical record): Encounters submitted by the subcontractor
either cannot be verified in the clinical record or are duplicated.
o Missing (not in the encounter record): Clinical record contains evidence of a service but the
service is not represented by the encounter record.
Also in 2013, Acumentra Health reviewed all sample enrollee charts for evidence of adherence to the
“Golden Thread” of therapy, evaluating whether the assessment substantiates the diagnosis, whether the
treatment plan is consistent with the diagnosis, and whether progress notes address the treatment plan.
The following Golden Thread data elements were scored:
Does the assessment substantiate the Category A diagnosis?
Does the assessment substantiate the Category B diagnosis?
Does the treatment plan include interventions and goals consistent with issues identified in the
assessment?
Do the progress notes address interventions identified in the treatment plan and the individual’s
progress toward meeting stated goals?
Are treatment plan objectives individualized?