Nathalie Dourdin PhD, Monika Wagner PhD, Peter Melnyk PhD and Donna Rindress PhD BioMedCom Consultants inc., Montréal, QC, Canada

CONCLUSION

Despite a wealth of guidelines, evidence for their impact on the quality of published health economic studies is mixed.

Some reviews of health economic evaluations found evidence that the quality of studies is improving.

Other reviews found no or only modest improvement over time and sub-optimal adherence to guidelines.

Outcome measures for quality of HE studies varied significantly across all these studies and were usually qualitative in nature.

FUTURE DIRECTIONS

We need standardized guidelines to promote more consistent level of quality of HE publications.

This is not sufficient, use and reinforcement by journals publishing HE evaluations (for both reviewers and authors) is needed.

We also need more standardized and quantitative measures of quality.

METHODSLiterature searches

Full-text published HE guidelines and studies assessing the quality of HE evaluations were collected from available public domain sources, including PubMed.

The PubMed database was searched from inception using the search terms “health economic guidelines, quality assessment, cost-benefit analysis/standards, cost-effectiveness analysis/standards”

Limits: Meta-Analysis, Practice Guideline, Review, Consensus Development Conference, NIH, Guideline

English, French, German studies selected

Abstracts from recent conferences of ISPOR, SMDM, HTAi, ASHE and iHEA were searched for any relevant abstracts and presentations

Inclusion criteria: reviews of economic evaluations

Data extraction

Information from retrieved QA studies was systematically extracted using a data extraction template

Data extracted included HE study inclusion/exclusion criteria, type of quality assessment tool(s) used, as well as quality assessment results.

BACKGROUND & OBJECTIVE

Health economic (HE) evaluations are intended to provide decisionmakers with information on the comparative efficiency of medical technologies.

The quality of an HE study is critical to the useful application of its findings.

There are a number of different guidelines and checklists available in the public domain on the conduct and reporting of HE and outcomes research.

Adoption of guidelines should help ensure a high degree of rigour and standardization among HE evaluations.

There are currently 74 guidelines for the conduct and reporting of HE evaluations (published and public domain).

The objective of this work is to review studies assessing the quality of published HE evaluations (QA studies) and to investigate whether quality has improved over time with the use of guidelines.

RESULTS

68 QA studies met the inclusion criteria; 50 published after 2000

Studies used various tools for assessing quality as well as different ways of reporting the quality assessment

Given this heterogeneous approach to QA, some issues may not have been identified in specific assessments

Of the studies published after 2000, 39 different approaches were used to measure quality

The most commonly reported quality criteria with issues cited by QA studies were:

Perspective

Sensitivity analyses/uncertainty

Discounting

Types of costs included

Source of funding/conflict of interest

Among 68 studies included, only 15 reported on the change in quality of HE evaluations following guideline publication.

11 found evidence that the quality of studies is improving

4 found no improvement over time and sub-optimal adherence to guidelines

Can decisionmakers rely on health economic evaluations? A review of studies assessing the quality of published health economic evaluations

REFERENCES

1. Campbell H, Briggs A, Buxton M, Kim L, Thompson S. J Health Serv Res Policy. 2007;12(1):11-7. 2. Campbell JD, Spackman DE, Sullivan SD. Allergy. 2008;63(12):1581-92. 3. Colmenero F, Sullivan SD, Palmer JA, Brauer CA, Bungay K, Watkins J, et al. Am J Manag Care. 2007;13(7):401-7. 4. Greenberg D, Earle C, Fang CH, Eldar-Lissai A, Neumann PJ. J Natl Cancer Inst. 2010;102(2):82-8.5. Jefferson T, Demicheli V, Vale L. JAMA. 2002;287(21): 2809-12. 6. Kaplan RM, Groessl EJ. J Consult Clin Psychol. 2002;70(3):482-93. 7. Manuel MR, Chen LM, Caughey AB, Subak LL. Gynecol Oncol. 2004;93(1):1-8. 8. Neumann PJ, Greenberg D, Olchanski NV, Stone PW, Rosen AB. Value Health. 2005;8(1):3-9. 9. Neumann PJ, Fang CH, Cohen JT. Pharmacoeconomics. 2009;27(10):861-72. 10. Otero HJ, Rybicki FJ, Greenberg D, Neumann PJ. Radiology. 2008;249(3):917-25. 11. Phillips KA, Chen JL. Am J Prev Med. 2002;22(2):98-105. 12. Schwappach DL, Boluarte TA, Suhrcke M. Cost Eff Resour Alloc. 2007;5:5. 13. Schwappach DL, Boluarte TA. BMC Health Serv Res. 2007;7:7. 14. Spiegel BM, Targownik LE, Kanwal F, Derosa V, Dulai GS, Gralnek IM, et al. Gastroenterology. 2004;127(2): 403-11. 15. Stone PW, Schackman BR, Neukermans CP, Olchanski N, Greenberg D, Rosen AB, et al. Lancet Infect Dis. 2005;5(6):383-91.

Issues pertaining to the quality of published HE evaluations were identified by determining how many QA studies reported a given issue as a problem in 20% or more of the HE evaluations they reviewed.

When a study mentioned a quality criterion as an issue, the study was included in the percentage calculation regardless of the reason for mention (e.g. not clear, not stated, not justified, etc.,).

The proportions of QA studies that identified a specific quality issue were then calculated.

Quality issues were grouped in clusters as follows:

Outcome measures, valuation of health outcomes

Types of costs and resources included and year of costing

Methodologies for modeling, validation of model, assumptions

Sensitivity analyses and CEAC (cost effectiveness acceptability curve)

Incremental analyses, calculation errors, wrongful use of efficacy/effectiveness

Potential biases and limitations

Sources of funding and disclosure of potential conflicts of interest

1 In the literature, quality assessment has been completed using various tools and approaches

2 A few issues identified by QA studies stood out

“studies published after 1996 … were of significantly higher quality than those

published before 1996 (P < .001)” Spiegel

2004“the fraction (of studies explicitly

reporting perspective) significantly increased

over time (41% in 1990-98 vs 62% in

1999–2004, P< .001) “ Schwappach 2007

improvement

“the past reviewers advice remains

germane tothe present economic

evaluations ” Campbell 2008

no improvement

“modest improvements in quality of economic evaluations appear to

have taken place in the last decade” Jefferson

2002

“significant improvement over time in mean total score for

adherence to methodological

principles (logistic regression P =.01)”

Manuel 2004 “this fraction (of studies reporting

perspective) increased only slightly over time”

Schwappach 2007

“in general, adherence to recommended

methods has improved over time among

pharmaceutical CUAs” Neumann 2009

“although we found evidence that the Panel

Reporthad an impact, almost

1/3 of Report Citers didnot follow the

recommendation to use a 3% discount”

Philips 2002

“adherence to methodological and

reportingpractices in published

CUAs is improving, although many

studies still omit basic elements. “Neumann

2005

“average quality score increased from 4.09 ±1.24 between 1985

and 1995 to 4.26 ±1.09 between 1996 and 2005 (P=.8)” Otero

2008

“we found no evidence of improvement in the

quality ofeconomic analyses in dossiers over time. “

Colmenero 2007

“we did not find major improvement in the

cost-utility analyses done since the PHS Panel

recommendations” Stone 2005

“no improvement in the standard of

reporting over time was

apparent..” Campbell 2007

“in general, adherence to recommended

methods for conducting and

reporting CEA results …was high and has

somewhat improved over time” Greenberg

2010

“in general, the studies have improved steadily

over time” Kaplan 2002

Quality of HE evaluations over time

3 Has quality of HE evaluations improved over time?

QHES instrument

BMJ guidelines

US Panel on Cost-Effectiveness in Health and MedicineCanadian guidelines

AMCP guidelines

Various checklists and criteria

BMJ guidelines

US Panel on Cost-Effectiveness in Health and Medicine

Canadian guidelines

Various checklists and criteria

Up to 2000 After 2000

Approaches to measure quality

49% 49%

31%

22%18%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Perspective Sensitivity analyses/uncertainty

Discounting Types of costs included Source of funding/conflict of

interest

Perc

ent o

f QA

stud

ies r

epor

ting

issu

e

Criteria identified by QA studies

Table 1. Specific issues relating to the principal criteria identifiedCriterion Issue

Perspective Not stated, stated but not taken, not societal when should be

Sensitivity analyses/uncertainty Not used, not taken into consideration, incomplete, range used

not justified

Discounting Not used at all, wrong % used according to guidelines, choice

not justified

Types of costs included Not adequate, indirect costs not included

Source of funding/conflict of interest Not reported

BioMedCom, Pure and Applied Intelligence BioMedCom, L’intelligence en action