Analysis of the impact of

AHRQ exclusions on the

variation in Patient Safety

Indicator (PSI) values

by Vladimir Stevanovic

Patient Safety Subgroup meeting

Paris, 26 May 2011

Background HCQI Expert Group meeting in June 2009

• Concerns were raised that PSI data may reflect rather

coding and registration practices than actual differences

in patient safety

• Several countries expressed reservations about the

publication of PSIs in Health at a Glance 2009 due to

perceived risk of misinterpretation

• The Secretariat proposed to undertake further analysis

with the aim of validating the methodological approach

Background PSI Subgroup meeting in October 2009

• The Secretariat presented a preliminary analysis on

the impact of AHRQ exclusions by using NZ data

• The findings implied that the exclusions have

varying impact on the country results and may distort

indicator values apart from the obstetric ones

• The Secretariat proposed further analysis to be

undertaken through a voluntary subsample of countries

by replicating the same methods and using UPIs

Background HCQI Expert Group meeting in June 2010

• Ten countries participated in the replication analysis:

CAN, DNK, FIN, ISR, NOR, SNG, SPA, SWE, SWI, USA

• The results confirmed the previous findings that

differences in coding practice, admission type definition

and long LOS exclusion are likely to be the most

important driving factor behind variations

• The decision was made to collect additional data by

the means of regular HCQI 2010/11 data collection

in order to better understand the impact of exclusions

Objectives

To improve international comparability of PSIs by:

• Assessing the impact of admission type (ADMT) and

and length of stay (LOS) exclusions on the rates for - PSI 07 Catheter-related bloodstream infection,

- PSI 12 Pulmonary embolism or deep vein thrombosis,

- PSI 13 Postoperative sepsis

• Exploring whether these exclusions account for any

undesired or increased variation across countries

The scope does NOT include exclusions that are

inherent to the concept of an indicator by their nature

Methods

Catheter-Related Bloodstream Infection

Age =15 y or >15 y?

Case is

assigned to MDC 14

or the PDx is listed

in table M3?

Exclude

PDx is immunocompromised

state (list C-1 / W-1), or cancer (list C-2/W-2)?

SDx is immunocompromised

state (list C-1 / W-1), or cancer (list C-2/W-2)?

yes Exclude

Excludeyes

PDx is

identical to the

numerator definition?

no

Add case to

denominator

population

Add case to

numerator

population

SDx is

identical to the

numerator definition?

no no

Calculate mean

number of secondary

diagnoses and ALOS of

denominator population

and report (6)

Count and report

denominator

cases (7)

Count numerator cases

and report (8)

no

no

yes

All hospital discharges

of one year

yes Exclude

yes

Catheter-related bloodstram infections – pre-exclusions

All hospital discharges of one year,

age =15 years or > 15 years and

LOS < 24 hours or one day

Count and report number of

cases / admissions, ALOS, and

mean number of secondary

diagnoses (1)

PDx is

identical to the

numerator definition?

yes

Count and report number of

patients (countries with UPI

only) / discharges (2)

no

End

Catheter-related bloodstram infections – impact of length of stay (LOS) exclusions

Count numerator cases and

report (5)

LOS

is < 48 hours or

< 2 days?

yes

LOS

is < 24 hours or

< 1 day?

yes

Count denominator

cases and report (4)

Count denominator

cases and report (5)

SDx is

identical to the

numerator definition?

yes

SDx is

identical to the

numerator definition?

Count numerator cases and

report (4)

yes

yes

yes

LOS > or = 48

hours or 2 days?

yes no

Perform pre-exclusion calculations,

see seperate flowchart

Count and report

denominator cases (3)

SDx is

identical to the

numerator definition?

Count and report

numerator cases (3)yes

Pre-exclusion and

Post-exclusion stats (all discharges in a year AND

age>=15 years AND LOS >= 24h)

- the total number of discharges

- the average length of stay

- the average number of SDX

Length of stay and

Admission type exclusions

- the numerator data

- the denominator data

Additional data

Countries

• Australia

• Belgium

• Canada

• Denmark

• Finland

• France

• Iceland

• Israel

• Italy

• New Zealand

• Singapore

• Spain

• Sweden

• Switzerland

• United States

Results PSI 07 Catheter-related bloodstream infection (LOS excl)

Results PSI 07 Catheter-related bloodstream infection (LOS excl)

Spearman-rank test (no LOS excl vs <1 day, <2 days) = 0.89, 0.84 (both p<0.01)

Results PSI 12 Postoperative PE or DVT (LOS excl)

Results PSI 12 Postoperative PE or DTV (LOS excl)

Spearman-rank test (no LOS excl vs <1 day, <2 days) = 0.78,(p<0.01), 0.59 (p=0.07)

Results PSI 13 Postoperative sepsis (LOS excl)

Results PSI 12 Postoperative sepsis (LOS excl)

Spearman-rank test (no LOS excl vs <1-3 days, <4 days) = 0.93-97, 0.78

Results PSI 12 Postoperative sepsis (LOS excl)

Results PSI 13 Postoperative sepsis (LOS & ADMT excl)

Results PSI 12 Postoperative sepsis (LOS & ADMT excl)

Spearman-rank test (no ADMT vs with ADMT) = -0.19 (p=0.60)

Results PSI 12 Postoperative sepsis (ADMT distribution)

Results PSI 12 Postoperative sepsis (modified excl criteria)

Results PSI 12 Postoperative sepsis (modified excl criteria)

Publication PSI 07 Catheter-related bloodstream infection

• Min/Max ratio = 100 fold

• Variation in coding practices?

• Inflammatory conditions?

Recommendations

1. The exclusion criteria for PSI 07 Catheter-related

bloodstream infections and PSI 12 Postoperative

pulmonary embolism and deep vein thrombosis are

appropriate, hence there is no need to change the

existing LOS<2 days exclusion.

2. For PSI 13 Postoperative sepsis, exclusions of

discharges with LOS<4 days and non-elective (acute)

type admissions account for an increased variation

between countries and cause bias in the resulting

postoperative sepsis rates. It is recommended to

drop the admission type exclusion criterion from

the algorithm and to use the modified LOS<3 days

exclusion instead.

Recommendations - cont.

3. For PSI 07 Catheter-related bloodstream infection,

remaining ambiguities in the definition make this

indicator not fit for reporting at this moment.

4. The following indicators seem to be robust enough and

are therefore recommended for publication in

Health at a Glance 2011:

- PSI 05 Foreign body left in during procedure

- PSI 12 Post-operative pulm. embolism or deep vein thrombosis

- PSI 13 Post-operative sepsis

- PSI 15 Accidental puncture or laceration

- PSI 18 Obstetric trauma - vaginal delivery with instrument

- PSI 19 Obstetric trauma - vaginal delivery without instrument

Optimal system requirements

• Unique patient identifier (UPI)

• Near miss and adverse events register

• Present on admission (POA) flag

• Standardised registration/coding practice

• Mapping between classification systems

• Calculation method/algorithm adjusted for

international comparison - exclusions inherent to the concept of PSI

- adjustments for the effect of confounders

• Comment on findings from the analysis,

• Decide on whether to change the algorithm used for

the calculation of PSI13 Postoperative sepsis as

recommended by this report,

• Decide on whether to adjust patient safety indicator

rates by the mean number of secondary diagnoses

among patients at risk as (previously presented by Saskia Drösler

and Patrick Romano and discussed in 2009/10),

• Make recommendations for the continuation of the

development work on PSIs (agenda item 4).

• Make recommendations on which patient safety

indicators should be published in the chapter on

Quality of Care in Health at a Glance 2011 (agenda item 5).

Members of the Patient Safety Subgroup are invited to