Applicability of the AGREE II Instrument in Evaluating the Development Process and Quality of Current National Academy of Clinical Biochemistry Guidelines

A.C. Don-Wauchope, J.L. Sievenpiper, S.A. Hill, and A. Iorio

October 2012

www.clinchem.org/cgi/content/article/58/10/1426.full

© Copyright 2012 by the American Association for Clinical Chemistry

© Copyright 2009 by the American Association for Clinical Chemistry

BackgroundBackgroundPractice guidelines are everywhere in healthcare

They have been present for decades but have been more prevalent in recent times for the following factors:

>Assessment and accountability Variability, cost, and utilization

Clinical practice should be guided by the best available evidence This is a daunting task considering the number of journals and the volume of studies

See accompanying Editorial by Bossuyt PMM

© Copyright 2009 by the American Association for Clinical Chemistry

Background (continued)Background (continued)

Guideline methodology This has improved over time and now there are a

number of standardized methodologies available for people who write guidelines

Applicability to Laboratory Medicine Additional information may need to be considered, in

relation to laboratory tests

© Copyright 2009 by the American Association for Clinical Chemistry

QuestionQuestion

Are guidelines including laboratory tests important for clinical laboratories?

© Copyright 2009 by the American Association for Clinical Chemistry

Evaluation of GuidelinesEvaluation of Guidelines

AGREE collaboration International collaboration that published an appraisal

instrument initially in 2003 and a revised version in 2009

Application to Laboratory Medicine The instrument has been used to evaluate guidelines

with laboratory medicine content but no evaluation of the National Academy of Clinical Biochemistry Laboratory Medicine Practice Guidelines has yet been performed

The 2009 version has not been validated for use in Laboratory Medicine Practice Guidelines

© Copyright 2009 by the American Association for Clinical Chemistry

ObjectivesObjectives

Demonstrate the applicability of the AGREE II instrument to Laboratory Medicine Practice Guidelines (LMPGs)

Report the quality of the LMPGs Make recommendations for improving future LMPGs

© Copyright 2009 by the American Association for Clinical Chemistry

AGREE II instrumentAGREE II instrument6 domains and an overall score

Scope and Purpose Stakeholder Involvement Rigor of development Clarity of presentation Applicability Editorial independence

23 key items (see Table 1*) Scored with a 7-point scale The largest number of key items is in the rigor of development

section. This is important because the methodology for the guideline development process is critical to the quality of the guideline.

*Clin Chem 2012; 58:1426-37AGREE Trust: www.agreetrust.org

© Copyright 2009 by the American Association for Clinical Chemistry

MethodsMethods

Guideline selection National Academy of Clinical Biochemistry website

Guideline appraisal AGREE II instrument Training of appraisers using on-line training from AGREE trust website Random allocation of LMPGs Independent appraisal with team discussion if scores varied by ≥ 4

Data analysis As described in AGREE II instrument Weighted (Quadratic) kappa for appraiser agreement Mann-Whitney test to compare guidelines before or after 2009

© Copyright 2009 by the American Association for Clinical Chemistry

QuestionQuestion

Are these methods reproducible and appropriate for the objectives of the study?

© Copyright 2009 by the American Association for Clinical Chemistry

ResultsResultsGuideline selection

National Academy of Clinical Biochemistry website had 11 guidelines posted in December 2011

Currently, there are 8 active guidelines and 6 older guidelines that are archived but still available

Recommendations For The Use Of Laboratory Tests To Support Poisoned Patients Who Present To The

Emergency Department

Emerging Biomarkers For Primary Prevention Of Cardiovascular Disease And Stroke.

Maternal-Fetal Risk Assessment And Reference Values In Pregnancy.

Use Of Tumor Markers In Testicular, Prostate, Colorectal, Breast, And Ovarian Cancers

Evidence-Based Practice For Point-Of-Care Testing. Laboratory Analysis and Application of Pharmacogenetics to Clinical Practice.

Biomarkers of Acute Coronary Syndromes and Heart Failure.

Use of Tumor Markers in Liver, Bladder, Cervical, and Gastric Cancers.

Use of Tumor Markers in Clinical Practice: Quality Requirement.

Guidelines and Recommendations for Laboratory Analysis in the Diagnosis and Management of Diabetes Mellitus.

Follow-up Testing for Metabolic Diseases Identified by Expanded Newborn Screening Using Tandem Mass Spectrometry

© Copyright 2009 by the American Association for Clinical Chemistry© Copyright 2009 by the American Association for Clinical Chemistry

Table 2. Table 2. Assessment of the NACB LMPGs with the AGREE II InstrumentAssessment of the NACB LMPGs with the AGREE II Instrument

© Copyright 2009 by the American Association for Clinical Chemistry

Figure 2Figure 2Counts of individual scale (1-7) scores for Domain 3 (Rigor of Development) showing key items 7-14. The y axis is set to 10 individual counts (max 22 counts). Each item was scored twice for the eleven LMPGs.

© Copyright 2009 by the American Association for Clinical Chemistry

Results (continued)Results (continued)

Guideline appraisal AGREE II instrument was successfully applied Only 1 item (#16: the different options for management of the

condition or health issue are clearly presented) was not applicable in 1 LMPG (Tumor Marker Quality Requirements)

The overall quality of the LMPGs was generally poor - 5 had overall scores of > 50%- the median score was 42%- the range was 8%-92%

Sensitivity analysis There was no significant change between the 2 groups of LMPGs. There was a suggestion of improving scores.

© Copyright 2009 by the American Association for Clinical Chemistry

QuestionQuestion

Are these results important for adapting the LMPGs to your clinical laboratory?

© Copyright 2009 by the American Association for Clinical Chemistry

DiscussionDiscussion

Domain 3 (Rigor of development) 3 key items (of 8) were very poorly addressed The other key items had variable results Domain 3 is crucial to help the reader assess the validity

of the LMPG

Poorly addressed key items: Use of systematic methods to find evidence Description of the methods used to evaluate and select

evidence Description of an explicit plan to update the guideline

© Copyright 2009 by the American Association for Clinical Chemistry

DiscussionDiscussionUtility of the AGREE II instrument

The AGREE II instrument was useful in appraising the 11 NACB LMPGs

The domains and item questions were applicable and appropriate

Only the 1 item (#16) raised some concerns and the AGREE instrument does allow for this and the item can be excluded for scoring purposes

AGREE II is applicable to LMPGs 4 guidelines considered AGREE II in their planning The most recent (Diabetes) scored highest and had followed a

more robust methodology, including considering the AGREE criteria

© Copyright 2009 by the American Association for Clinical Chemistry

RecommendationsRecommendations

1. Specific description of the population2. Specific description of who has contributed to the

guideline3. Present methodology in an explicit manner4. Strategy for updating the guideline must be

described5. Factors that implicate resource use should be

described6. The choice of question must be considered

carefully for each guideline

© Copyright 2009 by the American Association for Clinical Chemistry

Criticism of the AGREE instrumentCriticism of the AGREE instrument

AGREE relies on the report which may not reflect the full process

AGREE evaluates the process of producing the report and not the content of the report

AGREE definition of quality is many-layered and requires appraiser judgment

See accompanying Editorial by Bossuyt PMM

© Copyright 2009 by the American Association for Clinical Chemistry

ConclusionConclusion

Guidelines are important to Laboratory Medicine

The AGREE II instrument is useful for evaluating LMPGs

There are a number of areas that could be improved in the development and reporting of LMPGs

© Copyright 2009 by the American Association for Clinical Chemistry

Thank you for participating in this month’sClinical Chemistry Journal Club.

Additional Journal Clubs are available atwww.clinchem.org

Follow us