Using Information Technology To Improve the … 4 Using Information Technology To Improve the Quality of Health Care | 125 2. Background, including demographic traits (age, sex, ethnicity,

Using InformationTechnology To Improve

the Quality ofHealth Care

he quality of health care is ultimately judged by the impactof specific health services on the patient’s health status.1

Improving quality involves identifying and using healthservices that, when properly executed, produce the great-

est improvement in health status. The most direct contributionthat information technology can make to improving the quality ofhealth care is to provide the clinician with better informationabout the patient and health problem at hand, and alternative testsand treatments for that problem, preferably at the point of care.This would enable clinicians to choose more effective servicesmore quickly2 and help them avoid potentially tragic errors.3

This chapter discusses the potential for advanced informationtechnologies to improve the quality of health care—as indicated

1 Institute of Medicine, Medicare: A Strategy for Quality Assurance, K.N. Lohr (ed.)

(Washington, DC: National Academy Press, 1990), pp. 20-25.

2 K.L. Coltin and D.B. Aronow, “Quality Assurance and Quality Improvement in theInformation Age,” Putting Research to Work in Quality Improvement and Quality Assur-ance, M.L. Grady, J. Bernstein, and S. Robinson (eds.), Agency for Health Care Policy andResearch, Pub. No. 93-0034 (Rockville, MD: July 1993), pp. 51-54; S.D. Horn and D.S.P.Hopkins, “Introduction,” Clinical Practice Improvement: A New Technology for Devel-oping Cost-Effective Quality Health Care, S.D. Horn and D.S.P. Hopkins (eds.)(NewYork, NY: Faulkner & Gray, 1994), pp. 1-5.

3 L.L. Leape, “Error in Medicine,” Journal of the American Medical Association, vol.272, No. 23, Dec. 21, 1994, pp. 1851-1857; L.L. Leape et al., “Systems Analysis of Ad-verse Drug Events,”Journal of the American Medical Association, vol. 274, No.1, July 5,1995, pp. 35-43; D.W. Bates et al., “Incidence of Adverse Drug Events and Potential Ad-verse Drug Events: Implications for Prevention,” Journal of the American MedicalAssociation, vol. 274, No.1, July 5, 1995, pp. 29-34.

| 123

124 | Bringing Health Care Online: The Role of Information Technologies

by the effectiveness of clinical decisionmak-ing4—and the potential role of the federal govern-ment in that process. The most relevanttechnologies5 include:

� electronic patient records,� structured data entry,� advanced human-computer interface technolo-

gies,� portable computers,� automated capture of data from diagnostic and

monitoring equipment,� relational databases with online query (key-

word search and retrieval),� knowledge-based computing, and� computer networks.

This chapter first reviews the clinical decisionsupport approach to improving health care, andthe ways in which information technology couldenhance clinical decisionmaking. It then ex-amines the performance assessment approach toimproving health care, which involves evaluatingspecific health services, providers,6 and insuranceplans.7 Ways in which some of the problems con-fronting both approaches might be resolved by us-ing information technology are explored. Thechapter concludes with a discussion of policy is-sues and options regarding potential governmen-tal roles in those developments.

CLINICAL DECISION SUPPORTClinical decision support can be broadly and sim-ply defined as the use of information to help a cli-nician diagnose and/or treat a patient’s healthproblem. Two kinds of information are involved:

1) information about the patient; and 2) informa-tion about the kind of health problem afflicting thepatient and alternative tests and treatments for it.Clinical decision support is by no means a newphenomenon—such information traditionally hasbeen available from several sources. However,those sources have limitations that often diminishtheir reliability or their accessibility at the point ofcare.

The time pressures of clinical practice do notallow clinicians to study the patient’s entire healthhistory or review the latest clinical knowledge onevery nonroutine health problem they encounter.Consequently, one major goal of clinical decisionsupport is to locate needed information and makeit available to the clinician in readily usable format the point of care as quickly as possible, and in amanner that minimally interferes with the careprocess. Moreover, the potentially severe conse-quences of incorrect clinical decisions for both thepatient and the clinician require that the informa-tion retrieved be as accurate as possible.

❚ Limitations of Traditional InformationSources

Information About the PatientThe specific kinds of information about the pa-tient that are useful in clinical decisionmaking fallinto two broad categories:

1. Health problems, both current (signs andsymptoms, physical findings, diagnostic testresults, functional status, etc.) and previous(medical history, including previous servicesfor each health problem); and

4 This approach focuses on the effectiveness of the health services delivered by providers and insurance plans (see footnotes 6 and 7). Therole of the patient in clinical decisionmaking and self-care, and ways in which information technology can enhance that role, were discussed inchapter 1.

5 See chapter 2 for details on these technologies.

6 Throughout this chapter, the term providers includes both individual clinicians and institutional providers such as hospitals. Cliniciansinclude physicians and other licensed practitioners, such as nurses. Physicians include allopathic medical doctors, osteopaths, chiropractors,podiatrists, etc. In discussing clinicians who diagnose and treat health problems, this report includes nurse practitioners, physician assistants,and other physician extenders who are licensed to prescribe medications.

7 The term health insurance plan here includes traditional indemnity plans and managed care organizations (health maintenance organiza-

tions, preferred provider organizations, etc.).

Chapter 4 Using Information Technology To Improve the Quality of Health Care | 125

2. Background, including demographic traits(age, sex, ethnicity, and socioeconomic status),risky behaviors (substance abuse and hazard-ous occupations, sports, hobbies, or sexualpractices), exposures (occupational and environ-mental hazards), allergies, and family history.

Information about the patient traditionally hasbeen drawn from the paper-based patient recordand direct clinical examination of the patient.Briefly, the major difficulties with the paper-based patient record8 include:

� indecipherable content,� lack of comprehensiveness,� lack of completeness,� inaccuracy,� inaccessibility,� lack of uniformity and standards,� slow and cumbersome transmission,� lack of security, and� sheer physical volume.

These problems make it difficult to quickly lo-cate accurate and readily usable information aboutthe patient at the point of care.

Problems with the clinical examination9 in-clude:

� unsystematic methods in obtaining thepatient’s health history,

� unsystematic methods in observing thepatient’s signs and symptoms,

� faulty reasoning and inference in using thecollected information, and

� the amount of time required to obtain andrecord all of this information.

These drawbacks jeopardize the completenessand accuracy of new information about the pa-tient’s current health problems.

Information About the Health ProblemThe most efficient source of information about aspecific health problem is the clinician’s ownknowledge and experience with similar cases.Such information can usually be retrieved almostinstantaneously from the clinician’s memory andcan be readily applied to a health problem in termsthat the clinician understands. Indeed, between 80and 90 percent of clinical actions are based onsuch information.10 However, it is impossible forclinicians to remember all available informationabout all of the health problems they are likely toencounter, or all of the alternative tests and treat-ments for those problems. Even experts on a givenhealth problem are likely to have only selected in-formation on that problem—information that maybe unsystematic, unrepresentative, and biased.Most clinicians need to consult other sources ofclinical knowledge and experience, at least on oc-casion.

A clinician may seek the advice of other clini-cians and researchers who have special knowl-edge or experience regarding the health problemat hand. However, the patients usually seen by theconsultant may differ from the referring clini-cian’s patients in important ways. In addition, anyindividual clinician’s patients may not be typicalof all patients with the health problem at hand, andthe consultant’s knowledge and experience couldalso be highly selective. Finally, consultants sim-

8 Institute of Medicine, The Computer-Based Patient Record: An Essential Technology for Health Care, R.S. Dick and E.B. Steen (eds.)(Washington, DC: National Academy Press, 1991), pp. 12-19; P.C. Tang, D. Fafchamps, and E.H. Shortliffe, “Traditional Medical Records as aSource of Clinical Data in the Outpatient Setting,” Proceedings of the Eighteenth Annual Symposium on Computer Applications in MedicalCare, J.G. Ozbolt (ed.) (Philadelphia, PA: Hanley & Belfus, 1994), pp. 575-579; J.C. Wyatt, “Clinical Data Systems, Part 1: Data and MedicalRecords,” Lancet, vol. 344, No. 8936, Dec. 3, 1994, pp. 1543-1547. For further discussion of problems with paper-based patient records, seechapter 2.

9 See C. Selby et al., “Set Up and Run an Objective Structured Clinical Exam,” British Medical Journal, vol. 310, No. 6988, May 6, 1995, pp.

1187-1190.

10 B.C. James, “Advances in Computer-Based Patient Records for Health Services Research,” presentation at the 12th Annual Meeting of

the Association for Health Services Research, Chicago, IL, June 4-6, 1995.


ply may not be available when needed in urgentcases.

A related source of clinical guidance for practi-tioners is legal standards of care, which specifythe levels of care provided by the majority of phy-sicians in particular clinical situations. Thesestandards are determined by the courts, largely onthe basis of testimony by expert witnesses duringmalpractice lawsuits. The widely conflictingopinions expressed by different experts in manysuch contexts illustrate the extent to which legalstandards of care can be vague, inconsistent, andincomplete.11 Extensive variation in practice pat-terns among clinicians within and across localitieshas been thoroughly documented. This variationreflects, in part, the lack of consensus regardingthe most effective ways to treat most health prob-lems.12

To extend the individual knowledge and expe-rience of individual clinicians, institutional pro-viders or multifacility enterprises with largenumbers of patients sometimes conduct localclinical research on those patients over time. Suchefforts can generate information that is useful inthose providers’ own clinical decisionmaking, aswell as for publication. (Providers with fewer pa-tients often conduct such research as well, but thenumber of cases may be too small to support sta-tistically reliable comparison among treatmentgroups.) However, local research may be usefulonly in that setting if the institution’s patients orpractice patterns are atypical. In any case, exten-sive local research is not that common, evenamong large institutions.

Information published in printed clinical liter-ature (reference books, textbooks, research stud-ies, and professional periodicals) is anotherwell-established source of information for clinicaldecisionmaking. However, it can be difficult to lo-cate such information quickly because of inade-quate indexing and the problem of keeping papermaterials organized. In addition, a considerableamount of time can elapse before new informationgets published; and once published, it quickly be-comes outdated. Maintaining large amounts ofprinted information in accurate, up-to-date, andreadily accessible form can be expensive.

As with clinician knowledge and experience,clinical literature also may sometimes harborbiases resulting from the use of unsystematicmethods in generating the information. Even peer-reviewed research literature is hampered by publi-cation bias stemming from the preference ofauthors, journal editors, and reviewers for statisti-cally significant results supporting specific hy-potheses, particularly if those results areperceived as being important.13 To be consideredworthy of publication, articles whose results failto support an hypothesis must strongly challengewidely held theories and assumptions. On the oth-er hand, published research findings are oftenwidely and uncritically accepted without carefulconsideration of the soundness of the methodolo-gies used.

Despite decades of clinical and epidemiologic-al research, systematic evidence is still lacking re-garding “what works” in diagnosing, treating, and

11 U.S. Congress, Office of Technology Assessment, Defensive Medicine and Medical Malpractice, OTA-H-602 (Washington, DC: U.S.

Government Printing Office, July 1994), pp. 30-31, 164.

12 H. Krakauer et al., “The Systematic Assessment of Variations in Medical Practices and Their Outcomes,” Public Health Reports, vol. 110,No. 1, Jan.-Feb., 1995, pp. 2-12; U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work: Searching forEvidence, OTA-H-608 (Washington, DC: U.S. Government Printing Office, Sept. 1994), pp. 26-34.

13 K. Dickersin and Y.I. Min, “Publication Bias: The Problem That Won’t Go Away,” Doing More Harm Than Good: The Evaluation ofHealth Care Interventions, K.S. Warren and F. Mosteller (eds.) (New York, NY: New York Academy of Sciences, 1993); P.J. Easterbrook et al.,“Publication Bias in Clinical Research,” Lancet, vol. 337, No. 8746, Apr. 13, 1991, pp. 867-872.


preventing most health problems—much lesswhich methods are most cost-effective.14 In addi-tion, even the evidence that does exist is not al-ways put to use in clinical practice. To rephrase anearlier statement, only between 10 and 20 percentof clinical actions are based on published scientif-ic research.15

❚ Practice Guidelines and ProtocolsEven when solid experiential or research-basedevidence is available, human beings are inherentlyfallible processors of that information.16 They cantrack no more than four variables simultaneously,compared with the hundreds of variables thatcharacterize even a single health condition.17 It isimpossible for clinicians to remember all avail-able information about all of the health problemsthey are likely to encounter, or all of the alterna-tive tests and treatments for those problems, or allrelevant characteristics and histories of all of theirpatients. Moreover, recall is biased toward thingsthat are considered to be more important, that tendto confirm one’s prejudices, and that are more re-cently experienced.18 Indeed, one researcher hasreferred to clinical decision support as “uncertain-ty management.”19

In an effort to reduce this uncertainty, clinicalpractice guidelines20 have been developed overthe past few decades by numerous medical spe-cialty societies, insurance companies, utilizationreview organizations, managed care organiza-tions, and government agencies. Guidelines focuson a given health problem or procedure, and areusually developed through a group consensusprocess among selected clinical experts on thatproblem or procedure. The intent is to providebroad parameters within which clinicians contin-ue to exercise judgment, rather than to dictate ex-act steps to follow.21 Figure 4-1 reproduces analgorithm that depicts a clinical practice guidelinefor management of patients with heart failure, de-veloped by the federal government’s Agency forHealth Care Policy and Research (AHCPR). Evenwhen practice guidelines are available, however,evidence suggests that clinicians often forget tofollow them, or deviate from them without clearcause, especially in high-stress situations.22 Re-search also shows that it is difficult to change cli-nician behavior simply by providing them withinformation, even in the form of guidelines.23

Formal clinical protocols are more rigorousmodels of the process of care for a given

14 U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12; P.J. Neumann and M.

Johannesson, “From Principle to Public Policy: Using Cost-Effectiveness Analysis,” Health Affairs, vol. 13, No. 3, summer 1994, pp. 206-214.

15 James, op. cit., footnote 10.16 C.J. McDonald, “Protocol-Based Computer Reminders, The Quality of Care and the Non-Perfectability of Man,” New England Journal

of Medicine, vol. 295, No. 24, Dec. 9, 1976, pp. 1351-1355; Leape, op. cit., footnote 3.

17 A.H. Morris, “Protocol Management of Adult Respiratory Distress Syndrome,” New Horizons, vol. 1, No. 4, Nov. 1993, p. 594.18 Ibid., p. 593.19 E.H. Shortliffe, “Medical Informatics and Clinical Decision Making: The Science and the Pragmatics,” Medical Decision Making, vol.

11, No. 4, Oct.-Dec. Supplement, 1991, pp. S2-S4.

20 Clinical practice guidelines are sometimes called practice parameters or some variation of clinical paths or critical pathways—terminol-

ogy adapted from critical path analysis in manufacturing and other industries.

21 U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12, ch. 7.

22 E. Coiera, “Medical Informatics,” British Medical Journal, vol. 310, No. 6991, May 27, 1995, p. 1383.23 T.H. Lee. et al., “Failure of Information as an Intervention To Modify Clinical Management,” Annals of Internal Medicine, vol. 122, No. 6,

Mar. 15, 1995, p. 436; U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12, ch. 8.


4-1:

Alg

orith

m D

epic

ting

a C

linic

al P

ract

ice

Gui

delin

e fo

r Eva

luat

ion

and

Car

e of

Pat

ient

s w

ith H

eart

Failu

re

�� !�,.��%'/,!��/% !'%)!��)!'��'%)%��'��,��.%�!��/% !'%)!��*��/��*��2��*�&2

��

��

��

��

0%''!�� !+�,.(!).�*"��!�'.$��) ��/(�)��!,0%�!-��/�'%��!�'.$��!,0%�!��#!)�1�"*,��!�'.$��,!��*'%�1��) ��!-!�,�$��/)!��++��2��

Figu

re


health problem. They are composed of highly spe-cific steps and decision parameters regardingdiagnosis, treatment, or prevention of a problem.The inputs and outputs of a given step can be ei-ther deterministic (involving a fixed value or ac-tion) or probabilistic (involving a range ofpossible values or actions). Clinicians may stillexercise judgment and override any particular stepin a protocol, but having a clear sequence of spe-cific steps to follow can help ensure that none willbe inadvertently forgotten or altered.24

Many clinicians view practice guidelines andprotocols skeptically as being “cookbook medi-cine,”25 concocted largely by clinically unin-formed researchers and bureaucrats. Some arealso concerned that guidelines may be usedagainst clinicians in malpractice suits,26 althoughevidence indicates that they are used by bothplaintiffs’ and defendants’ attorneys.27 Other cli-nicians criticize guidelines that are based more onjudgmental consensus than on scientific evidence.These guidelines are seen as being vague and sub-jective, lacking in specificity and testability, andbased on incomplete and inaccurate informa-tion—drawbacks that make it difficult to derivecase-specific advice.28

For example, nurses at LDS Hospital in SaltLake City, Utah, found that the AHCPR guidelinefor treating pressure ulcers was too vague to use inclinical practice. Most importantly, the guidelinedid not specify the treatment options for variouscombinations of scores on six components of a

measure of risk for developing pressure ulcers. Ateam of nurses, physicians, and researchers con-verted the guideline into a more formal clinicalprotocol by developing exact specifications forthose treatment options through an iterative groupconsensus process and monitoring of patient out-comes.29 This illustrates how local research canbe used to inform the development and refinementof clinical practice guidelines and protocols. Italso emphasizes the need for careful testing andscreening of these kinds of clinical decision sup-port.

❚ Potential Contributions of InformationTechnology

The basic question in this area is whether ad-vanced information technologies can 1) improvethe accuracy of the information needed in clinicaldecisionmaking, 2) reduce the amount of time re-quired to retrieve that information, and 3) makethat information accessible at the point of care.This section highlights some of the potential con-tributions these technologies can make to clinicaldecision support. A later section summarizessome of the limited and mixed evidence bearingon these questions.

Entering and Retrieving Patient InformationThe key technology for improving patient in-formation is the electronic patient record thatstores comprehensive information on the patient

24 Coiera, op. cit., footnote 22.25 W.W. Parmley, “Clinical Practice Guidelines: Does the Cookbook Have Enough Recipes?” Journal of the American Medical Associa-

tion, vol. 272, No. 17, Nov. 2, 1994, pp. 1374-1375.

26 F. Bazzoli, “Computerized Records Will Play a Key Role in the Implementation of Clinical Guidelines,” Health Data Management, Feb-

ruary 1995, p. 32.

27 A.L. Hyams et al., “Practice Guidelines and Malpractice Litigation: A Two-Way Street,” Annals of Internal Medicine, vol. 122, No. 6,

Mar. 15, 1995, pp. 450-455.

28 C.J. McDonald and J.M. Overhage, “Guidelines You Can Follow and Can Trust: An Ideal and an Example,” Journal of the American

Medical Association, vol. 271, No. 11, Mar. 16, 1994, pp. 872-873.

29 S. D. Horn, C. Ashton, and D.M. Tracy, “Prevention and Treatment of Pressure Ulcers by Protocol,” Clinical Practice Improvement: ANew Technology for Developing Cost-Effective Quality Health Care, S.D. Horn and D.S.P. Hopkins (eds.)(New York, NY: Faulkner & Gray,1994), pp. 253-262; “LDS Nurses Reduce Pressure Ulcer Incidence with Retooled Guidelines,” Report on Medical Guidelines & OutcomesResearch, Feb. 23, 1995, pp. 10-11.


from a variety of sources (clinic, laboratory, phar-macy, etc.).30 Other technologies for handling pa-tient information operate in conjunction with theelectronic patient record. Overall, these technolo-gies could permit faster, easier, and more accuratecollection of information about the patient.31

Clinical examination results can be entered by cli-nicians at or near the point of care, particularlywith the aid of portable computers. Structureddata entry, such as on-screen forms and menus andprepared blocks of text, can encourage completedata collection and reduce keying errors, particu-larly when pen-based computing is used ratherthan keyboards. Automatic date- and time-stamp-ing of entries facilitates documentation and track-ing of patient care and outcomes over time.

Some patient data can be captured directly fromdiagnostic and monitoring equipment, bypassinghuman data entry altogether. Radiographicimages, full-motion videos, and sound recordingscan be digitized, stored, and transmitted electroni-cally, often with resolution approaching that ofanalog technologies. Patient background in-formation and risk factors32 can be entered intocomputers by patients themselves, again with theaid of structured data entry and advanced human-computer interface technologies. One example ofsuch a system is HealthQuiz33 (see appendix C).Basic demographic traits can be obtained fromother computer databases (e.g., insurance eligibil-ity files) through computer networks, again by-passing human data entry.

Using relational databases with online query,information technologies can also permit faster,easier, and better targeted search and retrieval ofpreviously collected information about the pa-

tient—even at the point of care. Portable comput-ers and advanced human-computer interfacetechnologies can also be helpful here. Electronicstorage of digitized radiographic images, full-mo-tion videos, and sound recordings can make themeasier to locate, although retrieving them can beslow if the computers, telecommunications equip-ment, or transmission lines used have insufficientcapacity. Increasingly powerful and flexiblegraphics software and higher resolution displayscan offer flexibility in the ways information is or-ganized and displayed to suit the individual needsof clinicians.

Retrieving Information About the HealthProblemComputer and telecommunications networks, inconjunction with online query, portable comput-ers, and advanced human-computer interfaces,can make information about various health prob-lems more readily accessible from either local orremote knowledge bases. Many research librariesprovide online access to their computerized cata-logs (e.g., the Library of Congress’s SCORPIO)and bibliographic databases (e.g., The NationalLibrary of Medicine’s MEDLINE) that can bequeried online. Documents can be ordered elec-tronically (even during an online literature search)from one of the more than 4,000 member librariesof the National Network of Libraries of Medicine.Documents can be shipped in hard-copy form ortransmitted electronically via the Internet or fax.Unfortunately, these databases do not cover all ofthe clinical literature, and it can be difficult toidentify all studies of a certain kind, such as ran-domized controlled trials.34

30 Although the computer-based patient record is usually conceptualized as being a centralized repository, in reality different components of

the record may be stored in separate but seamlessly linked computer systems.

31 J.C. Wyatt, “Clinical Data Systems, Part 2: Components and Techniques,” Lancet, vol. 344, No. 8937, Dec. 10, 1994, pp. 1609-1614.32 Risk factors are key health problems and background characteristics that can affect the patient’s outcome, independent of the specific

kinds of services received.

33 “ ‘HealthQuiz’ Makes Preventive Care Guidelines Easy To Apply,” Report on Medical Guidelines & Outcomes Research, Jan. 26, 1995,

pp. 5-6.

34 U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12, p. 81.


The National Cancer Institute (NCI) maintainsa Physician Data Query (PDQ) system that pro-vides online information via the Internet (Cancer-Net) and by fax (CancerFax) regarding variouscancers, ongoing clinical trials, and individualsand organizations involved in cancer care. TheUniversity of Pennsylvania also maintains a mul-timedia cancer information resource on the Inter-net called OncoLink. The Centers for DiseaseControl and Prevention provide online access tothe full text of Morbidity and Mortality WeeklyReport, and has recently launched an online jour-nal called Emerging Infectious Diseases. Severalbiomedical journals are also available online.35

Some periodicals, and even complete booksand reports, are becoming available on CD-ROMsthat can be purchased or obtained through manylibraries. (In their current form, however, CD-ROMs cannot be updated, and must be replaced asknowledge changes.) Both CD-ROMs and the In-ternet permit inclusion of graphics, videos, andsound in textual documents. This helps offset thecomplaint that it is not only less pleasant to readdocuments on a video screen than on paper, but ac-tually slower.36

Information technology is also making practiceguidelines more readily accessible. The NationalLibrary of Medicine (NLM) offers online accessto practice guidelines developed by AHCPR, andNCI’s PDQ system includes information on can-cer treatment protocols. Private organizationssuch as the American Medical Association are

also distributing their practice guidelines on CD-ROMs and computer diskettes.

In recent years, an international movementamong researchers and clinicians has developedan approach to clinical problem-solving called ev-idence-based medicine.37 It involves systematicsearching and critical appraisal of the research lit-erature to identify findings that can be applied to aclearly defined clinical problem. This approachgoes beyond the narrative review articles occa-sionally published in leading clinical journals. Itemploys systematic review of the literature, inwhich specific items of information are extractedfrom each work and compared across works, us-ing structured methods. The most sophisticatedform of systematic review is meta-analysis, orquantitative synthesis of the statistical results of anumber of studies on a given topic.38 Special jour-nals have been established to summarize and eval-uate the vast literature on selected healthproblems.39 The Cochrane Collaboration, an in-ternational network of researchers, distributesresults of systematic reviews of randomized con-trolled trials—or the most reliable evidence fromother sources—on selected health problems (be-ginning with pregnancy and childbirth) in un-copyrighted form via the Internet, as well as oncomputer diskettes and CD-ROMs.40 However, itis unclear how these results will get incorporatedsystematically into clinical practice guidelinesand protocols.41

35 H.F. Judson, “Structural Transformations of the Sciences and the End of Peer Review,” Journal of the American Medical Association, vol.

272, No. 2, July 13, 1994, p. 93.

36 J.C. Wyatt, op. cit., footnote 31, p. 1613.37 W. Rosenberg and A. Donald, “Evidence Based Medicine: An Approach to Clinical Problem-Solving,” British Medical Journal, vol. 310,

No. 6987, Apr. 29, 1995, pp. 1122-1126.

38 U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12, pp. 59-65.39 F. Davidoff et al., “Evidence Based Medicine: A New Journal To Help Doctors Identify the Information They Need,” British Medical

Journal, vol. 310, No. 6987, Apr. 29, 1995, pp. 1085-1086.

40 Cochrane Collaboration, “The Cochrane Collaboration: Preparing, Maintaining, and Disseminating Systematic Reviews of the Effects of

Health Care,” Oxford, England, brochure, n.d.

41 J.C. Wyatt, “Clinical Data Systems, Part 3: Development and Evaluation,” Lancet, vol. 344, No. 8938, Dec. 17, 1994, p. 1687.


The prospect of making information such as re-search results readily and inexpensively availablefor online query through the Internet has spawnedvisions of electronic (online) publishing.42 Notonly are certain peer-reviewed journals alreadyavailable online, but in some disciplines, such asphysics, preprints containing preliminary resultsare often distributed over the Internet prior toprinted publication.43 At first glance, this mightappear to reduce the problem of publication bias.However, in addition to the fact that much ofthat bias rests with authors themselves, there areseveral other concerns about this prospect. Mostimportant is the absence of online screeningmechanisms to replace the process of scientificpeer review that seeks to ensure the quality of pub-lished research.44 Without such mechanisms forscreening documents for scientific rigor as well asrelevance to one’s interests,45 the increasing prob-lem of information overload could worsen. More-over, public access to unrefereed preprints ofmedical research could lead some people to mis-use medications.46 On the other hand, online ac-cess to the full text of commercially publishedbooks and journal articles is likely to remain lim-ited until electronic subscription and paymentmechanisms become established and issues re-

garding intellectual property rights and electroniccopying are resolved.47

Computer-Based Clinical Decision SupportSystemsIncreasingly, the traditional sources of clinical de-cision support are being supplemented by clinicalinformation systems, mainly at large academicmedical centers. The most rudimentary of theseare library systems or simple data systems48 thatmerely display information about the patient and/or the health problem to the clinician without of-fering advice based on analysis of thatinformation. However, some clinical informationsystems contain expert systems or knowledge-based systems that do offer advice to the clinicianregarding diagnosis, testing, or treatment.49 Thegoal of either simple or knowledge-based decisionsupport systems is to provide more complete andaccurate information more quickly to the clini-cian—preferably at the point of care—thereby im-proving clinical decisionmaking in terms ofpatient outcome measures. These benefits to theclinician presumably outweigh the added burdenof more extensive data collection and entry. Clini-cal information systems may also contain otherapplications besides decision support, such as on-

42 R.E. LaPorte et al., “The Death of Biomedical Journals,” British Medical Journal, vol. 310, No. 6991, 1387-1390, May 27, 1995, pp.

1085-1086; Judson, op. cit., footnote 35.

43 J.P. Kassirer and M. Angell, “The Internet and the Journal,” Journal of the American Medical Association, vol. 332, No. 25, June 22,

1995, p 1709.

44 Both Judson and LaPorte et al, op. cit., footnote 42, propose an online peer review system in which all readers of a document wouldcomment on it. LaPorte et al. go further in proposing that readers give each document ratings in various categories. These comments and sum-mary ratings would subsequently be attached to the document for other readers to use in screening. One potential problem with this scenario isthat readers willing to take time to evaluate all documents that they read might well be a small, self-selected, hence unrepresentative group; andthere would be no way to ensure that they were qualified to evaluate the document. Kassirer and Angell discuss the perils of “majority rule”compared to peer review. Op. cit., footnote 43.

45 LaPorte et al. suggest that software agents could be used to select only documents that meet certain user-specified content criteria. Op. cit.,

footnote 42.

46 Kassirer and Angell, op. cit., footnote 43.47 U.S. Congress, Office of Technology Assessment, Information Security and Privacy in Network Environments, OTA-TCT-606 (Wash-

ington, DC: U.S. Government Printing Office, September 1994), pp. 96-110.

48 Wyatt, op. cit., footnote 41.49 Wyatt refers to these as advisory systems. Ibid.


line order entry, that allows the clinician to submitorders for tests and treatments (including pharma-ceuticals).

A knowledge-based system designed for clini-cal use, sometimes called a clinical decision sup-port system (CDSS), usually involves three basiccomponents:50

1. Data on the patient being diagnosed or treatedare either entered into the system manually,captured automatically from diagnostic ormonitoring equipment, or drawn from an elec-tronic patient record.

2. A knowledge base contains rules and decisionalgorithms that incorporate knowledge andjudgment about the health problem at hand andalternative tests and treatments for it, mainly inthe form of “if-then” statements, such as “if thepatient’s potassium is less than 3.0 mEq/dl andthe patient is on digoxin, then warn the clini-cian to consider potassium supplementa-tion.”51

3. An inference engine combines informationfrom both the patient data and the knowledgebase to perform specified tasks, outlined in ap-pendix C.

Some CDSSs—usually those developed more re-cently—employ probabilistic and adaptive ap-proaches, such as fuzzy logic, Bayesian networks,or neural networks. Others—usually those devel-oped earlier—employ rule-based systems, deci-sion trees, and other deterministic methods,although probabilistic decision nodes are some-times employed.52

Many of the major applications of CDSSs wereimplemented over the past 15 to 20 years in twopioneer systems:

� the Health Evaluation through Logical Proc-essing (HELP) system developed by Inter-mountain Health Care (IHC) and its flagshipinstitution, LDS Hospital and the University ofUtah in Salt Lake City;53 and

� the Regenstrief Medical Record System(RMRS), developed by the Regenstrief Insti-tute and Indiana University, initially at WishardMemorial Hospital in Indianapolis.54

Components of both of these systems are mar-keted commercially: HELP by the 3M Co., withabout five installations outside of Utah; andRMRS by Shared Medical Systems, Inc. (SMS),with about 10 installations outside of Indiana.Several other CDSSs, or some of their particularapplications, are also commercially available.However, most are implemented by clinical re-searchers in the form of highly specialized, local-ized, and experimental systems that vary widelyin their levels of development.

Computer-Based Clinical ProtocolsThe most advanced CDSSs integrate several ofthe applications outlined in appendix C into for-mal clinical protocols. Again, some are based ondeterministic models, while others employ proba-bilistic and adaptive approaches. Converting aclinical protocol into computer-based algorithmsforces the developer to use unambiguous ter-minology, examine the logic of all linkages

50 See D.P. Connelly and S.T. Bennett, “Expert Systems and the Clinical Laboratory Information System,” Clinics in Laboratory Medicine,

vol. 11, No. 1, March 1991, p. 136.

51 R.F. Gibson and B. Middleton, “Health Care Information Management Systems To Support CQI,” Clinical Practice Improvement: A NewTechnology for Developing Cost-Effective Quality Health Care, S.D. Horn and D.S.P. Hopkins (eds.)(New York, NY: Faulkner & Gray, 1994),p. 109.

52 R.A. Miller, “Medical Diagnositc Decision Support Systems—Past, Present, and Future,” Journal of the American Medical Informatics

Association, vol. 1, No. 1, Jan./Feb. 1994, pp. 11-16.

53 See chapter 2 and G.J. Kuperman, R.M. Gardner, and T.A. Pryor, HELP: A Dynamic Hospital Information System, Computers and Medi-

cine Series (New York, NY: Springer-Verlag, 1991).

54 See C.J. McDonald et al., “The Regenstrief Medical Record System: 20 Years of Experience in Hospitals, Clinics, and Neighborhood

Health Centers,” M.D. Computing, vol. 9, No. 4, July/August 1992, pp. 206-217.


among steps, and—in deterministic models—specify exact parameters. It also facilitates refine-ment and updating of the protocol over time,based on any of the traditional sources of clinicaldecision support outlined earlier, plus feedbackfrom clinicians who use the protocol—particular-ly the reasons they document for overriding itsrecommendations—and from local research onpatient outcomes.

Researchers at Intermountain Health Care55

have developed an approach to quality improve-ment, called clinical practice improvement, thatessentially combines computer-based protocols,local research, and the principles of continuousquality improvement (CQI).56 A protocol is devel-oped for a selected health problem (e.g., acute re-spiratory distress syndrome) based on review ofrelevant literature, clinician judgment, and retro-spective analysis of data from the electronic pa-tient record system. The protocol is refinedthrough discussion and consensus among clini-cians, and serves to guide diagnosis and treatmentfor the selected health problem. In addition, ran-domized controlled trials of various alternativediagnostic and therapeutic procedures for thatproblem are conducted, and the protocol is furtherrefined in light of the results of those trials.

Computer-based clinical protocols may alsoprove valuable in a more indirect way. The full po-tential of CDSSs is constrained by the limitationsof electronic storage devices. Storing complete,full-text information on all possible health prob-

lems and alternative tests and treatments for themin a manner that permits rapid retrieval of that in-formation at the point of care is prohibitively ex-pensive. However, by distilling selected elementsof full-text information on a particular healthproblem and its alternative tests and treatmentsinto explicit steps, criteria, and parameters, clini-cal protocols can greatly reduce storage require-ments.

Other Potential Benefits of InformationTechnologyBoth clinical protocol development and local re-search can benefit from advanced informationtechnologies. Most patients receive care frommore than one provider, and within a given pro-vider organization there are usually several sepa-rate information systems, often one for eachdepartment (inpatient, laboratory, pharmacy,etc.). Electronic patient record systems and com-puter networks within and across provider orga-nizations can facilitate the tracking of all care andoutcomes of individual patients over time. Thesesystems make it easier and more efficient to linkthe separate records for a given patient across alldepartments and providers, particularly if a com-mon, unique patient identifier is used in all re-cords. The value of assembling patient data acrossseveral departments is illustrated by local researchthat used the HELP system at LDS Hospital toidentify specific causes of adverse drug events57

and hospital-acquired infections.58 Computer net-

55 S.D. Horn and D.S.P. Hopkins (eds.), Clinical Practice Improvement: A New Technology for Developing Cost-Effective Quality Health

Care (New York, NY: Faulkner & Gray, 1994).

56 CQI (also known as total quality management, or TQM) was originally developed in the field of manufacturing and was subsequentlyadapted to health care. See W.E. Deming, Common Causes and Special Causes of Improvement: Stable System, Out of Crisis (Cambridge, MA:MIT Center for Advanced Engineering Study, 1986); D.M. Berwick, “Continuous Improvement as an Ideal in Health Care,” New England Jour-nal of Medicine, vol. 320, No. 1, Jan. 5, 1989, pp. 53-56; Institute of Medicine, Medicare: A Strategy for Quality Assurance, K.N. Lohr (ed.)(Washington, DC: National Academy Press, 1990), pp. 58-64; S.B. Kritchevsky and B.P. Simmons, “Continuous Quality Improvement: Con-cepts and Applications for Physician Care,” Journal of the American Medical Association, vol. 266, No. 13, Oct. 2, 1991, pp. 1817-1823; Leape,op. cit., footnote 3; Leape et al., op. cit., footnote 3.

57 R.S. Evans et al., “Preventing Adverse Drug Events in Hospitalized Patients,” The Annals of Pharmacotherapy, vol. 28, No. 4, April 1994,

pp. 523-527.

58 D.C. Classen et al., “Prophylactic Antibiotics Used To Prevent Surgical Wound Infections,” Horn and Hopkins, op. cit., footnote 55, pp.

217-221.


works across provider organizations could alsopermit wider and faster dissemination of clinicalprotocols and the results of generalizable local re-search, particularly to remote sites.

An indirect but important way for advanced in-formation technologies to enhance the quality ofhealth care could be improving the outcomes dataused in research on the effectiveness of specifichealth services. Electronic patient records, struc-tured data entry, advanced human-computer inter-faces, portable computers, and automated datacapture from diagnostic and monitoring equip-ment could make the collection of patient data notonly faster and easier, but also more complete andaccurate. This could permit more valid and reli-able measurement of patient risk factors, clinicalprocesses, and outcomes. Records or results forpatients with a given health problem but treated indifferent ways could be pooled across providers,creating very large databases for assessing the ef-fectiveness of specific health services. This wouldrequire using health problems, process and out-come measures, and analytical methodologiesthat were as similar as possible across providers.Research based on these improved data could en-hance the medical knowledge on which clinicaldecision support is based.

From the perspective of physicians, one directbenefit of using advanced information technologyin medical practice recently became readily appar-ent: Two malpractice insurance companies beganoffering reduced premiums to physicians who usespecific commercial electronic patient record sys-tems.59 This development mainly reflects the im-proved patient information and documentation ofcare that electronic patient records offer, com-pared with paper-based records.

❚ Continuing Problems in ClinicalDecision Support

Technology DevelopmentAs impressive as their applications are, the useful-ness of clinical decision support systems can stillbe hampered by incomplete, inaccurate, or inac-cessible information—problems that advancedinformation technologies could help overcome.As discussed in chapter 2, however, the capabili-ties of many of the information technologiesemployed in CDSSs remain limited and theircosts remain high, posing substantial barriers totheir widespread use. Several technological ad-vances are needed for faster, easier, and more ac-curate collection, entry, and retrieval of patientinformation, and more readily accessible informa-tion about the health problem. The needed ad-vances include:

� advanced human-computer interface technolo-gies, particularly voice recognition, for easierand possibly hands-free input and retrieval ofinformation;

� more extensive use of structured data entry,such as on-screen forms and menus and pre-pared blocks of text, to ensure complete datacollection and reduce keying errors;

� smaller, more portable computers that can linkinto larger computer networks, either throughwireless technologies or docking stations;

� improved wireless technologies that minimizesuch limitations as the restricted range andplacement of infrared technologies that useline-of-sight transmission, or the electromag-netic interference generated by radio-wavetransmission;

59 “Malpractice Insurers Offer Discounts for Doctors Using Electronic Records,” Health Data Management, February 1995, p. 14.


� more efficient methods of filtering and summa-rizing the enormous quantities of data captureddirectly from diagnostic and monitoring equip-ment, such as focusing on abnormal data valuesand trends;

� higher capacity and more flexible electronicstorage devices, such as updatable CD-ROMs;

� higher resolution computer displays;� more powerful and flexible graphics software;� improved technologies for capturing and stor-

ing digitized radiographic images, full-motionvideos, and sound recordings, and faster meth-ods of retrieving such information;

� faster and more flexible methods of onlinequery using relational databases;

� higher capacity telecommunications equip-ment and transmission lines; and

� more complete coverage of the research litera-ture by online bibliographic databases.

As one researcher put it:

[Clinicians] need a system that is easy to use:computer terminals must be ubiquitous, systemresponse must be immediate (not seconds), nec-essary data should always be on-line, accessible,and confidential, and very little training shouldbe required.60

In addition, system down time must be at an ab-solute minimum, and data should be retained foras long as possible without diminishing system re-sponse times. Systems that meet all the needs ofclinicians may have to be developed in-houserather than adapted from commercial products.61

Standards DevelopmentThe issues regarding standards development dis-cussed in chapters 2 and 3 apply here as well.However, some additional aspects of these topicsand issues that apply to clinical decision supportrequire further discussion.

Messaging standardsAt first glance, it might appear that the develop-ment of messaging standards for electronic ex-change of information among disparate computersystems is less important in clinical decision sup-port than in other health care applications of ad-vanced information technologies, such aselectronic claims payment. Clinical decision sup-port is inherently localized—that is, specific to in-dividual providers—whereas electronic commerceinvolves transactions among providers and be-tween providers and other parties. Nevertheless,most patients receive care from more than oneprovider and from several departments within agiven organization. Thus, messaging standardsand common, unique patient and provider identi-fiers could facilitate patient record linkagethrough computer networks. (At the same time,standards for protecting patient and provider pri-vacy would need to be developed and enforced—see chapter 3.) Such standards could alsoencourage wider and faster dissemination of clini-cal protocols and local research results, and couldenable providers with different types of computersystems to access various central repositories ofmedical knowledge.

Clinical information contentIn theory, clinical decision support could alsobenefit from further development of standards forclinical information content—mainly commonmedical nomenclatures and uniform coding sys-tems for diagnoses, procedures, and test results—to help ensure that all needed information ispresent and accurate. Some analysts believe thedevelopment of a universal clinical nomenclatureand coding system is critical to the effective use ofinformation technology to improve the quality ofhealth care.62 However, developing a truly univer-

60 Wyatt, op. cit., footnote 41, p. 1682.

61 Ibid., p. 1685.62 M. Ackerman et al., “Standards for Medical Identifiers, Codes, and Messages Needed To Create an Efficient Computer-Stored Medical

Record,” Journal of the American Medical Informatics Association, vol. 1, No. 1, January./February 1994, pp. 1-7.


sal system is a difficult task, given the wide varia-tion in existing systems and the intensity ofinstitutional commitment to those systems. In-deed, some analysts question whether a truly uni-versal system can ever be developed, contendingthat “terminology evolves in a context of use” thatcannot be supplanted.63 Instead, “vocabulariesneed to be constructed in a manner that preservesthe context of each discipline and ensures transla-tion between disciplines.”64 An alternative tocompiling enumerative systems that attempt tolist all possible terms in advance is a composition-al approach. Such systems use basic terms asbuilding blocks that can be combined in variousways to form higher level terms tailored to partic-ular applications and specialties.65

NLM has already made some progress in thisarea through the ongoing development of its Uni-fied Medical Language System (UMLS), which isbeing tested at about 500 sites.66 (Institutions andindividuals receive the software free of charge inexchange for testing it and commenting on it.) De-spite its name, UMLS is not in itself a unified clin-ical language; rather, it is more a means oftranslating among disparate clinical nomencla-tures. Its major purpose is to facilitate the retrievaland integration of biomedical information fromdisparate machine-readable sources by mappingand interpreting over 200,000 specific conceptsacross different classification systems, codingsystems, and controlled vocabularies (in which

only one term denotes each concept). The centralcomponent of the UMLS is a Metathesaurus thatessentially links synonyms from disparate vocab-ularies to a common term.

A separately developed Unified Nursing Lan-guage System (UNLS) is being incorporated intothe UMLS.67 A related NLM project is the Inte-grated Academic Information Management Sys-tem, which provides grants to academic medicalcenters for investigating communications and in-formation processing technologies. Thesetechnologies are designed to facilitate exchangeand interpretation of data among different com-puter systems, with the ultimate goal of develop-ing integrated health care information systems.68

One such effort is the Arden Syntax, a languagefor encoding and sharing medical knowledgebased largely on the HELP and RMRS systemsdescribed earlier.69 The syntax is organized intoseparate Medical Logic Modules (MLMs) thatcontain sufficient logic to make a single medicaldecision, running automatically in conjunctionwith a program known as an event monitor. Thesyntax offers the ability to query clinical data-bases, many of which have been found to be com-patible with the syntax. Six institutions areactively using the syntax, and three others are re-viewing it. MLMs have been used to generatealerts, interpretations, diagnoses, screening forclinical research, quality assurance functions, andadministrative support. However, they have not

63 Coiera, op. cit., footnote 22, p. 1384.64 P.F. Brennan, “On the Relevance of Discipline to Informatics,” Journal of the American Medical Informatics Association, vol. 1, No. 2,

Mar./Apr. 1994, p. 200.

65 Coiera, op. cit., footnote 22, p. 1385.66 D.A. Lindberg, B.L. Humphreys, and A.T. McCray, “The Unified Medical Language System,” Methods of Information in Medicine, vol.

32, No. 4, August 1993, pp. 281-291; A.T. McCray, Chief, Cognitive Sciences Branch, National Library of Medicine, personal communication,June 8, 1995.

67 K.A. McCormick et al., “Toward Standard Classification Schemes for Nursing Language: Recommendations of the American NursesAssociation Steering Committee on Databases To Support Clinical Nursing Practice,” Journal of the American Medical Informatics Associa-tion, vol. 1, No. 6, November/December 1994, pp. 421-427.

68 Coltin and Aronow, op. cit., footnote 2.69 Columbia-Presbyterian Medical Center, Arden Syntax home page on the World Wide Web <URL:http://www.cpmc.columbia.edu/ar-

den/>, 1995; T.A. Pryor and G. Hripcsak, “The Arden Syntax for Medical Logic Modules,” International Journal of Clinical Monitoring andComputing, vol. 10, No. 4, November 1993, pp. 215-224.


been fully validated for clinical use, and not all ofthe ones developed are in active use. Nonetheless,the Arden Syntax has been adopted as a standardby the American Society for Testing and Materials(ASTM document E 1460).

Effectiveness and Safety of Clinical DecisionSupport Systems

EffectivenessEvidence regarding the effectiveness of CDSSs inimproving clinical processes and patient outcom-es is limited and mixed.70 One review of the de-velopment and evaluation of clinical data systemsfocused on “simple data systems” that do not offerclinical advice, but rather simply display informa-tion on the patient and/or the health problem.71 Itconcluded that these systems improve some clini-cal processes (accuracy of predictions of patientprogress, number or types of diagnostic tests or-dered, and completeness of data collection), butthat there is little evidence of improvement in pa-tient outcomes. Regarding data accuracy, the re-viewer noted:

If the system is not interactive, if data are col-lected largely for billing or administrative pur-poses, if coding staff are poorly instructed andtrained, and if clinicians are unaware of the sys-tem and do not monitor the data, inaccuracy willbe the rule. Reasons for data inaccuracy areoften organisational, not technical.72

Data completeness is improved when the sys-tem prompts the clinician or clerk for specific data

items. However, it is less clear whether paymentincentives improve data completeness, or whetherdata are more complete when a clinician entersthem directly rather than using encounter forms. Itis also unclear whether using computers saves cli-nician time; but even if it doesn’t, it will likely im-prove the quality of data.73

A recent review of systematic studies of the im-pact of CDSSs on clinician behavior and patientoutcomes found generally positive effects on cli-nician behavior, although this effect varied ac-cording to the type of application performed bythe CDSS.74 Three of four studies of CDSSs fordetermining the dose of toxic drugs reportedstatistically significant improvements in achiev-ing therapeutic levels. Four of six studies of pre-ventive care reminder systems found significantincreases in the performance of specific immu-nizations or screening tests. Seven of nine studiesof the impact of CDSSs on active medical care(e.g., adherence to a protocol for management ofhypertension) found significant positive effects.On the other hand, only one of five studies of com-puter-aided diagnosis found a significant im-provement in diagnostic accuracy. Moreover,only three of ten studies of the impact of CDSSson patient outcomes found significant effects fa-voring the use of a CDSS.

More recent studies also demonstrate themixed potential of CDSSs. One study found thatcomputer-based alerts of rising creatinine levels inhospitalized patients enabled clinicians to changemedications or dosages earlier, thereby decreas-

70 Wyatt, op. cit., footnote 41; M.E. Johnston et al., “Effects of Computer-based Clinical Decision Support Systems on Clinician Perfor-mance and Patient Outcome: A Critical Appraisal of Research,” Annals of Internal Medicine, vol. 120, No. 2, Jan. 15, 1994, pp. 135-142; D.M.Rind et al., “Effect of Computer-Based Alerts on the Treatment and Outcomes of Hospitalized Patients,” Archives of Internal Medicine, vol. 154,No. 13, July 11, 1994, pp. 1511-1517; E.S. Berner et al., “Performance of Four Computer-Based Diagnostic Systems,” New England Journal ofMedicine, vol. 330, No. 25, June 23, 1994, pp. 1792-1796; W.A. Knaus et al., “The SUPPORT Prognostic Model: Objective Estimates of Surviv-al for Seriously Ill Hospitalized Adults,” Annals of Internal Medicine, vol. 122, No. 3, Feb. 1, 1995, pp. 191-203.

71 Wyatt, op. cit., footnote 41, p. 1686.72 Ibid., p. 1684.73 Wyatt, op. cit., footnote 41, pp. 1684-1686.74 Johnston et al., op. cit., footnote 70.


ing the risk of serious renal impairment by morethan half.75 The SUPPORT prognostic model (seeappendix C) predicted survival as well as did agroup of clinicians. However, incorporating theclinicians’ subjective estimates as predictors inthe model improved both its predictive accuracyand its ability to identify patients with high proba-bilities of survival or death.76

Using data describing 105 actual cases that dif-fered in their degree of diagnostic difficulty,another study evaluated the performance of fourgeneral diagnostic CDSSs: Dxplain, Iliad, Medi-tel, and QMR (see appendix C). The performanceof these systems on several measures of diagnos-tic accuracy was compared to diagnoses deter-mined by group consensus among 10 clinicalexperts. No single system consistently scored bet-ter than the others on all performance measures. Amajority of the diagnoses that the systems listedwere correct (or closely related to the correct diag-nosis), but the correct diagnosis usually did notappear in the top five diagnoses listed by the sys-tems. Moreover, far less than a majority of thediagnoses they listed were considered relevant.On average, they listed less than half of the diag-noses identified by the expert clinicians, but theylisted about two additional relevant diagnoses notoriginally identified by the clinicians. These re-sults emphasize the potential usefulness ofCDSSs in reminding clinicians of overlooked al-ternatives, but also the importance of clinician ex-perience and judgment in interpreting andfiltering information.77

SafetyCDSSs, particularly computer-based clinical pro-tocols, may reduce inappropriate practice varia-tions and improve patient outcomes. Yet it ispossible that the most successful CDSSs could be-come viewed as rigid sets of rules for diagnosingand/or treating particular health problems. Clini-cians might then become overly dependent onthem, adhering to the recommended steps withoutquestion or independent investigation, and allow-ing their own knowledge, skill, and judgment toerode as a result.78 Alternatively, systems thatprovide too many simultaneous streams of in-formation could cause information overload,prompting clinicians either to focus on certainitems and neglect other important tasks, or to shunall such information .79

Any of these developments could adversely af-fect the quality of patient care and undermine theinterpersonal aspects of patient care (the “qualityof caring”).80 Indeed, there are indications that pa-tients find clinicians less communicative whenusing computers to enter patient data. Cliniciansthemselves mainly fear that computers mightthreaten patient and provider privacy, create legalor ethical problems, increase government controlof health care, or rely on out-of-date knowledge.81

CDSSs are only as good as the medical knowl-edge on which they are based. Due to methodolog-ical errors in the research underlying a CDSS or tosubstantive misinterpretation of research results, aCDSS may contain incorrect parameters or deci-sion criteria or may overlook crucial steps in the

75 Rind et al., op. cit., footnote 70.

76 Knaus et al., op. cit., footnote 70.77 Berner et al., op. cit., footnote 70.78 D. DeMoro, Director, Health Care Professions Council, Service Employees International Union Local 250, Oakland, CA, personal com-

munication, Mar. 29, 1995.

79 D.M. Rind, R. Davis, and C. Safran, “Designing Studies of Computer-Based Alerts and Reminders,” M.D. Computing, vol. 12, No. 2,

Mar.-Apr. 1995, p. 125.

80 DeMoro, op. cit., footnote 78.81 Wyatt, op. cit., footnote 41, p. 1684


�� !��

��

diagnosis or treatment of a given health problem.It could thus mislead a clinician into making deci-sions that harm patients. One observer points outthat the vast amounts of information that comput-ers can process at lightning speed can make itvirtually impossible for humans to verify that theresults are correct. He recommends that:

� clinicians have substantial input into the designand development of a clinical information sys-tem,

� the limitations of the system be clearly spelledout to the user, and

� the system itself be designed to explain to theuser exactly what it is doing as it is beingused.82

Assessing Clinical Decision Support SystemsSome analysts have called for rigorous evaluationof the effectiveness and safety of clinical informa-tion systems.83 However, there seems to be littlesentiment for mandatory testing and certificationof such systems by government authorities. Re-gardless, CDSSs should be used with caution, andthey should be carefully assessed regarding their

82 M.F. Smith, “Are Clinical Information Systems Safe? Clinicians Should Give More Attention to Possible Failures in Their Computer

Systems,” British Medical Journal, vol. 308, No. 6929, Mar. 5, 1994, p. 612.

83 Wyatt, op. cit., footnote 41; Rind, Davis, and Safran, op. cit., footnote 79; R. Wall, “Computer Rx: More Harm Than Good?” Journal ofMedical Systems, vol. 15, Nos. 5/6, December 1991, pp. 321-334; L.I. Iezzoni, “ ‘Black Box’ Medical Information Systems: A TechnologyNeeding Assessment,” Journal of the American Medical Association, vol. 265, No. 22, June 12, 1991, pp. 3006-3007.


effectiveness and safety—by their developers andusers, and perhaps by payers and accrediting bod-ies, if not by the government. As one researcherput it:

Clinicians should try to judge the claims ofthese newcomers in the same cautious way thatthey would examine claims about a new drug.84

As pointed out in chapter 2, the U.S. Food andDrug Administration (FDA) already regulatesmedical software as medical devices. Currentpolicy85 exempts from regulation any softwarethat is either:

1. general in purpose (e.g., database managementsystems or library systems for storing, retriev-ing, or disseminating health care information);

2. used in education or nonclinical research, oronly in the practice of the provider (practitioneror institution) that developed it (i.e., withoutbeing disseminated further); or

3. a knowledge-based decision support systemthat is “intended to involve competent humanintervention before any impact on humanhealth occurs (e.g., where clinical judgmentand experience can be used to check and inter-pret a system’s output).”86

In its definition of research software, the FDAintends to include software that is distributed freeof charge in source-code form so that it can be ex-amined by other researchers.87 (Commercial soft-ware must be distributed as object code that is

designed to be read only by particular computers,and is thus very difficult for humans to alter.)

In the case of “home-grown” software that isnot distributed beyond the originating institution,and if the institution conducts research using fed-eral government funds, then the use of such soft-ware on human subjects is regulated by the localInstitutional Review Board.88 There are apparent-ly no restrictions on the development and use ofhome-grown software in institutions that do notconduct federally sponsored research, or amongpractitioners in private practice. Yet these sys-tems, too, could mislead clinicians into makingdecisions that might harm patients. The issue iswhether the FDA should review and/or repeal anyof the exemptions listed above.

It can be argued that regulation of clinical deci-sionmaking is not within the FDA’s purview, andthat other public and private sector mechanisms,such as the malpractice system and managed care,can adequately perform that function. Further, theeffectiveness and safety of clinical informationsystems could be assessed by private sector orga-nizations, such as payers or professional societies.On the other hand, those organizations may not becapable of performing such assessments, or ofconducting research on the best methods of doingso.

Assessment of CDSSs can include randomizedcontrolled trials in which the health outcomes ofpatients treated with the aid of a CDSS are

84 Coiera, op. cit., footnote 22, p. 1381.85 This policy is based on a draft statement published on Nov. 13, 1989 (see footnote 86) that has yet to be formally implemented. The FDA

has used this draft statement as a basis for determining the applicability of the medical device regulations to specific software products on acase-by-case basis. The agency is in the process of developing formal regulations in this area. H. Rudolph, Acting Director, Office of Scienceand Technology, Center for Devices and Radiological Health, U.S. Food and Drug Administration, personal communication, June 30, 1995.

86 U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, Office ofDevice Evaluation, “Reviewer Guidance for Computer Controlled Medical Devices Undergoing 510(k) Review” (Rockville, MD: Aug. 29,1991), pp. 37-40.

87 This new criterion for identifying research software was adopted partly in response to the case involving a radiotherapy dosing product

(see chapter 2, footnote 47). Rudolph, op. cit., footnote 85.

88 Ibid. Each provider institution that conducts research using federal government funds is required to establish an Institutional Review

Board, largely to ensure that required procedures regarding treatment of human subjects—mainly informed consent—are followed.


compared with those treated by conventionalmethods.89 However, numerous complicationscan hamper such trials.90 In particular, imple-menting a CDSS may engender other changes inpractice patterns (e.g., teamwork, consultation,training, and altered role relationships) that aremore directly responsible for any observedchanges in patient outcomes than is the CDSS it-self.

Moreover, random assignment of patients orclinical staff to comparison groups may not be fea-sible here.91 If only patients are randomized intothe comparison groups (with the CDSS beingused in one group and not in the other), then clini-cal staff may carry over CDSS-induced changes inpractice patterns from the treatment group to thecontrol group. Yet randomizing staff into the com-parison groups can disrupt teamwork and alienateone staff group or the other (e.g., new burdens forthe treatment-group staff or feelings of exclusionfor the control-group staff). Another approach isto randomize entire staff teams or departments, al-though such clustering requires much larger sam-ples to maintain precision of estimates. Thisapproach is similar to the method of the firms trialin clinical research in which patients are randomlyassigned to similar (parallel) providers who usedifferent treatments, rather than to differentgroups that receive different treatments from thesame provider.92

At a minimum, CDSSs appear to help preventclinicians from neglecting or altering basic stepsin specific processes of care. However, it will be along time before CDSSs cover every contingencyin those processes, even for highly specific healthproblems. Despite the vagaries of clinician experi-ence, memory, and judgment, these will continue

to be essential elements of clinical decisionmak-ing. As randomized controlled trials and otherforms of effectiveness research increase knowl-edge regarding which health services truly “work”for a given health problem, marked deviationsfrom established standards of practice will be-come less justifiable. However, there will contin-ue to be room for variation in the judgmentalapplication of those standards to individual pa-tients in particular settings and locations. CDSSsmust continue to be viewed as aids to clinician ex-perience and judgment, rather than as substitutesfor them; and clinicians must retain the ability tooverride the recommendations of CDSSs. At thesame time, clinicians should also be required todocument the reasons for those decisions so thatthe CDSSs can be improved over time.

PERFORMANCE ASSESSMENT

❚ Comparison to Clinical DecisionSupport

A less direct approach to improving the quality ofhealth care is assessing the performance of provid-ers and health insurance plans.93 This approachseeks to:

� evaluate the performance of providers or plansin delivering health services to patients,

� give providers or plans feedback on their per-formance to help them improve, and

� give performance information to payers, pur-chasers, and consumers to help them select pro-viders and plans.

Performance measures can focus on several as-pects of patient care. Two of the more importantones are: 1) the use of specific services that areconsidered to be appropriate for a given health

89 Johnston et al., op. cit., footnote 70; Smith, op. cit., footnote 82; Wall, op. cit., footnote 83; J. Wyatt and D. Spiegelhalter, “Evaluating

Medical Expert Systems: What To Test and How?” Medical Informatics, vol. 15, No. 3, July-Sept. 1990, pp. 205-217.

90 Wyatt, op. cit., footnote 41; Rind, Davis, and Safran, op. cit., footnote 79.91 Ibid.

92 U.S. Congress, Office of Technology Assessment, Identifying Health Technologies That Work, op. cit., footnote 12, pp. 57-58.93 This approach focuses mainly on providers. Even assessing the performance of insurance plans involves, in part, assessing the perfor-

mance of the providers employed or contracted by those plans.


problem, and 2) patient outcomes of those ser-vices, usually measured by adverse events such asdeaths, complications, and readmissions. In thisrespect, the kinds of information needed to assessproviders or plans are similar to those needed forclinical decision support: detailed informationabout individual patients and their health prob-lems, and the specific health services that individ-ual providers (clinicians or institutions) use todiagnose, treat, or prevent those problems. Thekinds of technologies required to generate and uti-lize that information are also similar.

Although, in theory, they should have no bear-ing on clinical decisionmaking, certain additionalfactors (beyond those minimally needed for clini-cal decision support) may also influence clini-cians’ choices of services and affect patientoutcomes.94 These include the patient’s socioeco-nomic status, social supports (marital status, liv-ing arrangements, etc.), and type of healthinsurance (e.g., indemnity, prepaid, public, oruninsured). These factors need to be considered inassessing provider and plan performance, and per-haps in clinical decision support as well. In addi-tion, the more subjective aspects of the careprocess, such as patient satisfaction with thehealth services they receive or with various fea-tures of insurance plans, are apparently of greaterinterest in performance assessment than in clinicaldecision support—at least at present.

In examining the link between processes andoutcomes, performance assessment usually fo-cuses on adverse outcomes that result from ser-vices already rendered, thus helping to identify

processes that need correcting. In contrast, clini-cal decision support focuses on selecting servicesin advance that are likely to maximize favorableoutcomes and minimize adverse ones. In both ap-proaches, patient risk factors condition the rela-tionship between processes and outcomes.

❚ Relationship to Other Recent TrendsThe performance assessment approach to qualityimprovement fits with recent trends toward man-aged care and increased competition amongproviders and insurance plans.95 Traditional in-demnity insurance and fee-for-service reimburse-ment are seen as creating incentives for providersto overuse health services in order to maximize in-come. Thus, one goal of performance assessmentis to reduce “unnecessary” services, thereby re-straining the escalation of health care costs.

On the other hand, managed care—particularlyprepayment for health services—is seen as creat-ing incentives for providers to keep costs lowerthan the prepayment amount. One way to do this isto reduce the volume and intensity of services de-livered to patients. If this leads to underuse of ser-vices that are “necessary” for the diagnosis,treatment, or prevention of a given health prob-lem, then patients’ health status could be adverse-ly affected. Thus, another goal of performanceassessment is to monitor patient outcomes andrates of use of services that are presumed to im-prove those outcomes.

The performance assessment approach as-sumes that giving providers feedback on their per-

94 J.S. Feinstein, “The Relationship Between Socioeconomic Status and Health: A Review of the Literature,” The Milbank Quarterly, vol.71, No. 2, 1993, pp. 279-322; N.E. Adler et al., “Socioeconomic Inequalities in Health: No Easy Solution,” Journal of the American MedicalAssociation, vol. 269, No. 24, June 23/30, 1993, pp. 3140-3145; H.R. Burstin, S.R. Lipsitz, and T.A. Brennan, “Socioeconomic Status and Riskfor Substandard Medical Care,” Journal of the American Medical Association, vol. 268, No. 17, Nov. 4, 1992, pp. 2383-2387; J. Hadley, E.P.Steinberg, and J. Feder, “Comparison of Uninsured and Privately Insured Hospital Patients: Condition on Admission, Resource Use, and Out-come,” Journal of the American Medical Association, vol. 265, No. 3, Jan. 16, 1991, pp. 374-379.

95 See R. Lavizzo-Mourey, “Measuring Quality in Health Care Reform,” Journal of Health Care for the Poor and Underserved, vol. 5, No.3, 1994, pp. 202-211; J.E. Sisk and S.A. Glied, “Innovation Under Federal Health Care Reform,” Health Affairs, vol. 13, No. 3, summer 1994,pp. 82-97; Health Care Quality Alliance, Quality Considerations: An Analysis of Federal Health Care Reform Plans (Washington, DC: HealthCare Quality Alliance, July 1994); C. Anderson, “Measuring What Works in Health Care,” Science, vol. 263, No. 25, Feb. 25, 1994, pp.1080-1082.


formance in terms of patient outcomes encouragesthem to improve their processes of care by select-ing the most effective services for a given healthproblem. Identification and correction of prob-lems in production processes is one major compo-nent of CQI in manufacturing, an approach thatwas subsequently adapted to the health care indus-try. More recently, managed care organizationsand even pharmaceutical companies have soughtto adapt the CQI approach to the management ofspecific chronic, costly health problems, such asdiabetes, asthma, and high blood pressure, acrossall care settings. In part, this approach, known asdisease management, involves practice guide-lines, outcomes measurement, and feedback toproviders and insurance plans.96 At the same time,employers and health plans have sought to dealwith the rising cost of pharmaceuticals throughpharmacy benefit management, which employstechniques of disease management as well aspharmacy networks, negotiated discounts and re-bates, lists of preferred drugs, and online utiliza-tion review.97

All of these related approaches rest on the fol-lowing series of assumptions. The most effectiveservices also tend to be the most cost-effectiveones because, even if they cost more to provide,their positive impact on patient health status leadsto reduced use and cost of services in the long run.Thus, giving providers feedback on their perfor-mance both improves the quality and reduces thecost of health care. In addition, distributing per-formance information to payers, purchasers, andconsumers helps them choose providers that

employ the most cost-effective services for a giv-en health problem. Moreover, if sufficient num-bers of payers, purchasers, and consumers useonly those providers that employ the most cost-ef-fective services, then this forces all providers touse those services and to reduce the prices of thoseservices. This increased competition among pro-viders induces further improvements in the quali-ty of health care and reductions in its cost.

❚ Performance Indicator Projects (ReportCards)

In recent years, various groups have sought to de-velop summary sets of performance indicatorscommonly called report cards.98 Assessments us-ing such indicators are designed to:

1. help consumers, payers, and self-insured pur-chasers compare and select among providers;

2. help consumers and purchasers select amonginsurance plans; or

3. give performance information to accreditationbodies for providers or insurance plans.

They can also be used to provide feedback toproviders for quality improvement purposes, andto assist public policy makers in regulating plansand formulating health policy.99 In addition, pro-viders and insurance plans often tout performanceindicator projects or favorable results in their mar-keting efforts; others respond by trying to makeprocess changes that will improve their scores onperformance indicators.100 However, systematicevidence regarding the impact of performance in-

96 K. Terry, “Disease Management: Continuous Health-Care Improvement,” Business and Health, April 1995, pp. 64-72; G. Anders, “Drug

Makers Help Manage Patient Care,” Wall Street Journal, May 17, 1995, p. B1.

97 L. Etheredge, Pharmacy Benefit Management: The Right Rx? Briefing Paper, Health Insurance Project, The George Washington Univer-

sity, April 1995.

98 For a critical appraisal of performance indicator projects, see A. Epstein, “Performance Reports on Quality—Prototypes, Problems, and

Prospects,” Journal of the American Medical Association, vol. 333, No. 1, July 6, 1995, pp. 57-61.

99 U.S. Congress, General Accounting Office, Health Care Reform: “Report Cards” Are Useful but Significant Issues Need To Be Ad-dressed, GAO/HEHS-94-219 (Washington, DC: September 1994); J. Mangano, “Report Cards Come of Age,” 1995 Medical Quality Manage-ment Sourcebook, K.J. Migdail and M. Youngs (eds.) (New York, NY: Faulkner & Gray, 1994), pp. 1-21.

100 L. Oberman, “How Do Health Plans Perform?” American Medical News, Mar. 20, 1995, pp. 1, 30.


dicator projects on provider or plan behavior islacking.

Perhaps the earliest and best-known perfor-mance indicator project was the effort by HCFA toassess mortality rates among Medicare patients inevery hospital in the nation. Reports were releasedannually to the public beginning in 1986, but weresuspended in 1993 due, in part, to criticism ofHCFA’s methodology,101 particularly regardingrisk adjustment.102 As a supplement to its PeerReview Organization program of quality assur-ance, HCFA is developing a new set of perfor-mance indicators for ambulatory care, known asDeveloping and Evaluating Methods to PromoteAmbulatory Care Quality (DEMPAQ).103 Anoth-er government project is the U.S. Public HealthService’s Year 2000 Health Objectives for the Na-tion, comprised of population-based measures ofhealth promotion and disease prevention, such asinfant mortality rates.104

In the private sector, the National Committeefor Quality Assurance (NCQA) developed theHealth Plan Employer Data and Information Set(HEDIS) as part of its oversight of health insur-ance plans (largely managed care organiza-tions).105 Box 4-1 summarizes the measures usedin HEDIS. Like many performance indicator proj-ects, the HEDIS measures focus on processes of

care, that is, utilization of presumably appropriateservices among certain groups of plan members,and the accessibility or availability of those ser-vices. Of the HEDIS “quality of care” measures,only hospitalization for asthma and low birthweight represent patient outcomes. Moreover,none of the HEDIS measures is adjusted for mem-ber or patient risk factors. HCFA is in the processof adapting the HEDIS model to its Medicare andMedicaid programs.106

In 1994, NCQA conducted a one-year pilottest of 28 of the HEDIS measures using datafrom 21 health plans throughout the United States.(The pilot study also included a survey of enrolleesatisfaction with health plan performance.) TheHEDIS pilot data from each participating planwere audited for reliability and comparability byan independent firm. Each audit involved a reviewof the overall structure of the plan’s data collectionand processing procedures; a site visit to the planby an audit team; verification of the plan’s sourcecode and specifications; and validation of theplan’s measures and data. The pilot study identi-fied needs for additional quality measures in keyclinical areas (e.g., mental health), risk adjust-ment, field testing, improved standardization ofdata collection procedures, investment in en-hanced clinical information systems, refinement

101 U.S. Congress, General Accounting Office, op. cit., footnote 99; S.T. Fleming, L.L. Hicks, and R.C. Bailey, “Interpreting the Health Care

Financing Administration’s Mortality Statistics,” Medical Care, vol. 33, No. 2, February 1995, pp. 186-201.

102 Risk adjustment is statistical control of patient risk factors in the analysis of the utilization and outcomes of health services. The term alsorefers to control of financial risk factors faced by insurance companies. See L.I. Iezzoni, “Risk Adjustment for Medical Outcome Studies,”Medical Effectiveness Research Data Methods, M.L. Grady, and H.A. Schwartz (eds.), Agency for Health Care Policy and Research, AHCPRPub. No. 92-0056 (Rockville, MD: July 1992), pp. 83-97.

103 Delmarva Foundation for Medical Care, Inc., Developing and Evaluating Methods to Promote Ambulatory Care Quality, Final Report

(Washington, DC: August 1994), pp. 1-6.

104 U.S. Department of Health and Human Services, Public Health Service, Healthy People 2000: National Health Promotion and Disease

Prevention Objectives, DHHS Pub. No. (PHS) 91-50212 (Washington, DC: U.S. Government Printing Office, 1991).

105 National Committee for Quality Assurance, Health Employer Data and Information Set and Users’ Manual, Version 2.0 (Washington,

DC: 1993); National Committee for Quality Assurance, HEDIS 2.5: Updated Specifications for HEDIS 2.0 (Washington, DC: January 1995).

106 “HCFA, Outside Groups to Adapt HEDIS for Use in Medicare, Medicaid Programs,” BNA’s Health Care Policy Report, Mar. 27, 1995,

pp. 479-480.


QUALITY OF CARE

Childhood Immunization Rate: � �1/0/13)/.� /&� #(),$1%.� 6(/� (!$� 1%#%)5%$� 20%#)&)%$� )--4.)9!3)/.2� !2� /&

3(%)1� 2%#/.$� ")13($!8�

Cholesterol Screening: � �1/0/13)/.� /&� !$4,32� !'%$� �� 3/� �� 6(/� 1%#%)5%$� !� #(/,%23%1/,� 3%23� $41).'� 3(%0!23� &)5%� 8%!12�

Mammography Screening: � �1/0/13)/.� /&� 6/-%.� !'%$� �� 3/� �� 6(/� 1%#%)5%$� /.%� /1� -/1%� -!--/:'1!-2� $41).'� 3(%� 0!23� 36/� 8%!12�

Cervical Cancer Screening: � �1/0/13)/.� /&� 6/-%.� !'%$� �� 3/� �� 6(/� 1%#%)5%$� /.%� /1� -/1%� �!0� 3%232&/1� #%15)#!,� #!.#%1� $41).'� 3(%� 0!23� 3(1%%� 8%!12�

Low Birthweight: � �1/0/13)/.� /&� !,,� ,)5%� ")13(2� 3(!3� 6%1%� ,/6� ")13(6%)'(3� �4.$%1� �� '1!-2�� /1� 5%18� ,/6")13(6%)'(3� �4.$%1� �� '1!-2�� $41).'� 3(%� 0!23� 8%!1�

Prenatal Care in First Trimester: � �1/0/13)/.� /&� 6/-%.� 6)3(� /.%� /1� -/1%� ,)5%� ")13(2� $41).'� 3(%� 0!23� 8%!1

6(/� (!$� /.%� /1� -/1%� 01%.!3!,� #!1%� 5)2)32� � 3/� �� 6%%+2� 01)/1� 3/� $%,)5%18�

Asthma Inpatient Admission Rate: � �1/0/13)/.� /&� -%-"%12� !'%$� � 3/� �� /1� �� 3/� �� 6(/� (!$� /.%� /1

-/1%� ).0!3)%.3� $)2#(!1'%2� 6)3(� !� 01).#)0!,� $)!'./2)2� /&� !23(-!� 6)3().� 3(%� 0!23� 8%!1�� !,2/�� /&� 3(/2%� -%-:

"%12� 6)3(� 24#(� $)2#(!1'%2�� 3(%� 01/0/13)/.� 6(/� (!$� -/1%� 3(!.� /.%� 24#(� $)2#(!1'%�

Diabetic Retinal Exam: � �1/0/13)/.� /&� -%-"%12� !'%$� �� 3/� �� 6)3(� $)!"%3%2� �)�%�� 6(/� 6%1%� $)20%.2%$).24,).� /1� /1!,� (80/',8#%-)#2�� 6(/� 1%#%)5%$� !� 1%3).!,� /0(3(!,-/2#/0)#� %7!-).!3)/.� $41).'� 3(%� 0!23

8%!1�

Ambulatory Follow-Up After Hospitalization for Major Affective Disorder: � �1/0/13)/.� /&� -%-"%12

!'%$� �� 3/� �� 6)3(� !.� ).0!3)%.3� $)2#(!1'%� &/1� -!*/1� !&&%#3)5%� $)2/1$%1� $41).'� 3(%� 0!23� 8%!1� 6(/� (!$� /.%

/1� -/1%� !-"4,!3/18� -%.3!,� (%!,3(� %.#/4.3%12� /1� $!8�.)'(3� 31%!3-%.32� 6)3().� �� $!82� /&� $)2#(!1'%�

______________��(%� ,)23%$� -%!241%2� !1%� 42%$� ).� 3(%� 2%3� /&� 0%1&/1-!.#%�-%!241%2� &/1� (%!,3(� ).241!.#%� 0,!.2� +./6.� !2� 3(%� �%!,3(� �,!.� �-0,/8%1

�!3!� !.$� �.&/1-!3)/.� �%3� �� %12)/.� �� 6!2� $%5%,/0%$� "8� 3(%� �!3)/.!,� �/--)33%%� &/1� �4!,)38� �2241!.#%� �� !2

0!13� /&� )32� /5%12)'(3� /&� -!.!'%$� #!1%� 0,!.2� ��(%� #!3%'/1)%2� !.$� 3)3,%2� /&� 3(%� -%!241%2� !1%� $1!6.� &1/-� 3(%� �� -!.4!,��

�� !2().'3/.�� !.4!18� �� (%� $%2#1)03)/.2� /&� 3(%� -%!241%2� !1%� �� 24--!1)%2� /&� 3(%

$%3!),%$� 20%#)&)#!3)/.2� 01%2%.3%$� ).� 3(!3�-!.4!,� ��,,� -%!241%2� !1%� "!2%$� /.� 0,!.�-%-"%12� 6(/�6%1%� #/.3).4/42,8� %.1/,,%$� ).� 3(%� 0,!.

$41).'� 3(%� 20%#)&)%$� 3)-%� 0%1)/$� �� )2� !� (%!,3(� ).241!.#%� 0,!.�� !� �� )2� !� 0%12/.� 6(/� )2� %.1/,,%$� ).� 3(%� 0,!.�

��

BOX 4-1—Health Plan Performance Measures Used in the Health Plan and Employer Data and Information Set (HEDIS)1

of audit procedures, and more research on thekinds of information consumers need.107

Another performance indicator project is theIndicator Measurement System (IMSystem) de-veloped by the Joint Commission on Accredita-tion of Healthcare Organizations, as part of its“Agenda for Change” to adopt specific outcome-oriented measures to support the process of ac-

crediting hospitals and other institutionalproviders. Implementation of the IMSystem be-gan in 1994, starting with voluntary participationby hospitals that could generate the necessarydata.108 Box 4-2 summarizes the measures used inthe IMSystem, which are about equally dividedbetween process and outcome measures.

107 National Committee for Quality Assurance, Executive Summary for Report Card Pilot Project (Washington, DC: 1995), pp. 1-6.108 Joint Commission on Accreditation of Healthcare Organizations, IMSystem General Information (Oakbrook Terrace, IL: 1994), pp. 3-6.


OTHER MEASURES2

Access to Health Services: �$!� *+)*)+-%)(� )"� � .&-� '!'�!+,� 0$)� $� � )(!� )+� ')+!� *+)/% !+� /%,%-,� .+4

%(#� -$!� *�,-� -$+!!� 2!�+, � -$!� *+)*)+-%)(� )"� *+%'�+2� ��+!� *$2,%�%�(,� ��!*-%(#� � %-%)(�&� '!'�!+, � �(

�/!+�#!� 0�%-%(#� -%'!� ")+� �� *+%'�+2� ��+!� *+)/% !+� �**)%(-'!(-�

Member Satisfaction: �$!� *+)*)+-%)(� )"� '!'�!+,� 0$)� �+!� ,�-%,"%! � 0%-$� -$!� *&�( � �( � -$!� *!+�!(-�#!

)"� '!'�!+,� )"� 0$)� +�-!� -$!� *&�(� �,� #)) �� /!+2� #)) �� )+� !1�!&&!(-�

Membership: �)-�&� (.'�!+� )"� '!'�!+� 2!�+, � �( � -$!� *+)*)+-%)(� )"� '!'�!+,� 0$)� %,!(+)&&� "+)'� -$!

*&�(� �%(�&. %(#� -$),!� 0$)� %!�� 2� -2*!� )"� *&�(�� $!�&-$� '�%(-!(�(�!� )+#�(%3�-%)(�� *+!"!++! � *+)/% !+

)+#�(%3�-%)(� � )+� *)%(-� )"� ,!+/%�!�)-$!+�

Utilization: �/!+�#!� &!(#-$� )"� %(*�-%!(-� ,-�2 � -$!� (.'�!+� %(*�-%!(-� %,�$�+#!,� *!+� �� '!'�!+� 2!�+, �( � -$!� (.'�!+� )"� %(*�-%!(-� �2,� *!+� �� '!'�!+� 2!�+,�

Finance: �/!+�#!� +!/!(.!� *!+� '!'�!+� *!+� ')(-$ � -$!� *!+�!(-�#!� �$�(#!� %(� �/!+�#!� +!/!(.! � -$!&),,� +�-%)� �*!+�!(-�#!� )"� -)-�&� *+!'%.',� !/)-! � -)� !1*!(,!,� � -$!� (.'�!+� )"� 2!�+,� %(� �.,%(!,, � (!-

%(�)'!� �+!/!(.!� '%(.,� !1*!(,!,� � �( � (!-� 0)+-$� ��,,!-,� '%(.,� &%��%&%-%!,��

Health Plan Management and Activities: � �$!� *!+�!(-�#!� )"� *+%'�+2� ��+!� *$2,%�%�(,� 0$)� �+!� �)�+

�!+-%"%! � -$!� *!+�!(-�#!� )"� ,*!�%�&%,-� *$2,%�%�(,� 0$)� �+!� �)�+ � �!+-%"%! � �( � -$!� *!+�!(-�#!� )"� *+%'�4

+2� ��+!� *$2,%�%�(,� 0$)� &!"-� -$!� *&�(�

�� )(#+!,,�� ""%�!� )"� �!�$()&)#2� �,,!,,'!(-�� ,! � )(� ��-%)(�&� �)''%--!!� ")+� �.�&%-2� �,,.+�(�!��

�� ,$%(#-)(�� (.�+2� ��

______________�$!,!� �+!� !1�'*&!,� )"� '!�,.+!,� &%,-! � .( !+� ��-!#)+%!,� )-$!+� -$�(� �� %(� -$!� �� '�(.�&�

BOX 4-1—Health Plan Performance Measures Used in the Health Plan and Employer Data and Information Set (HEDIS) (Cont’d.)

The IMSystem also adjusts for patient risk fac-tors (demographic traits, complicating healthproblems, etc.) by developing an outcome predic-tion model for each performance measure. Eachmodel is based on risk factors that contribute sig-nificantly to the prediction of that performancemeasure. Using a given model, each institution’sactual sore on the performance measure iscompared to its predicted score.109 Institutionsthat score “worse” than predicted can then investi-gate the reasons behind those results. IMSystemreports are available to consumers for $30 per hos-pital.110

Performance indicator projects are also beingconducted by several managed care organizations,employer coalitions, and state governments, someusing the HEDIS model. Examples include:United HealthCare Corp. (a national managedcare organization); the Massachusetts HealthcarePurchaser Group (an employer coalition); and thestates of California, Florida, New York, and Penn-sylvania.111 Moreover, several legislative propos-als for national health reform, including theClinton Administration’s 1994 plan, have con-tained mandates for the development of such indi-

109 Ibid., pp. 16-18.110 “JCAHO Releases Data, Gets Blasted by AHA and AMA,” Business and Health, January 1995, p. 16.111 U.S. Congress, General Accounting Office, op. cit., footnote 99; S. Vibbert et al. (eds.), The Medical Outcomes & Guidelines Sourcebook

(New York, NY: Faulkner & Gray, 1994); K.J. Migdail and M. Youngs (eds.), 1995 Medical Quality Management Sourcebook (New York, NY:Faulkner & Gray, 1994).


BOX 4-2—Provider Performance Measures Used in the Indicator Measurement System (IMSystem)1

Postprocedure Complications (five indicators): � �/-.-/1(-,� -%� . 1($,10� 2,#$/&-(,&� ./-"$#2/$0� (,3-*38

(,&� ,$01'$0( � #+(,(01/ 1(-,� ,#� ,� (,. 1($,1� 01 5� 4'-� #$3$*-.� $ "'� -%� 1'$� %-**-4(,&� .-01./-"$#2/$

"-+.*(" 1(-,0� 4(1'(,� 14-� .-01./-"$#2/$� # 50�

�7"$,1/ *� ,$/3-20� 0501$+� "-+.*(" 1(-,�

�7.$/(.'$/ *� ,$2/-*-&(" *� #$%("(1�

�7 "21$� +5-" /#( *� (,% /"1(-,�

�7" /#( "� //$01�� ,#

�7(,1/ '-0.(1 *� +-/1 *(15�

C-Section: � �/-.-/1(-,� -%� #$*(3$/($0� #-,$� !5� � $0 /$ ,� 0$"1(-,�

VBAC: � �/-.-/1(-,� -%� . 1($,10� 4(1'� � '(01-/5� -%� ./$3(-20� � $0 /$ ,� 0$"1(-,� 4'-� #$*(3$/� !5� 3 &(, *� !(/1' %1$/� � $0 /$ ,� 0$"1(-,�

Low Birthweight: � �/-.-/1(-,� -%� *(3$� !(/1'0� 4(1'� � !(/1'4$(&'1� *$00� 1' ,� �� &/ +0�

Birth Complications: � �/-.-/1(-,� -%� *(3$8!-/,� (,% ,10� 4(1'� � !(/1'4$(&'1� &/$ 1$/� 1' ,� �� &/ +0� 4'-

' 3$� -,$� -/� +-/$� -%� 1'$� %-**-4(,&� "-+.*(" 1(-,0�

�7 ,� �.& /� 0"-/$� -%� *$00� 1' ,� �� 1� �� +(,21$0�

�7 #+(00(-,� 1-� 1'$� ,$-, 1 *� (,1$,0(3$� " /$� 2,(1� 4(1'(,� -,$� # 5� -%� #$*(3$/5� %-/� *-,&$/� 1' ,� �� '-2/0�

�7"*(,(" **5� .. /$,1� 0$(62/$�� -/

�70(&,(%(" ,1� !(/1'� 1/ 2+ �

Low Birthweight Complication: � �/-.-/1(-,� -%� *(3$8!-/,� (,% ,10� 4(1'� � !(/1'4$(&'1� &/$ 1$/� 1' ,� ��

&/ +0� ,#� *$00� 1' ,� �� &/ +0� 4'-� ' 3$� ,� �.& /� 0"-/$� -%� *$00� 1' ,� �� 1� �� +(,21$0�

Delayed CABG Recovery: � �-/� . 1($,10� 2,#$/&-(,&� (0-* 1$#� "-/-, /5� /1$/5� !5. 00� &/ %1� �� ./-8

"$#2/$0�� 1'$� ,2+!$/� -%� # 50� %/-+� (,(1( *� 02/&$/5� 1-� #(0"' /&$�

Timely Thrombolytic Therapy: � �-/� . 1($,10� #+(11$#� 1'/-2&'� 1'$� $+$/&$,"5� #$. /1+$,1� �� 4(1'�

./(,"(. *� #(0"' /&$� #( &,-0(0� -%� "21$� +5-" /#( *� (,% /"1(-,� �� ,#� /$"$(3(,&� 1'/-+!-*51("� 1'$/ .5�

1'$� +-2,1� -%� 1(+$� %/-+� �� //(3 *� 1-� #+(,(01/ 1(-,� -%� 1'/-+!-*51("� 1'$/ .5�

CHF Diagnostic Accuracy: � �/-.-/1(-,� -%� . 1($,10� 4(1'� � ./(,"(. *� #(0"' /&$� #( &,-0(0� -%� "-,&$01(3$

'$ /1� % (*2/$� 4'-� ' 3$� #-"2+$,1$#� $1(-*-&5� (,#(" 1(,&� 1' 1� #( &,-0(0�

Delayed PTCA Recovery: � �-/� . 1($,10� 2,#$/&-(,&� .$/"21 ,$-20� 1/ ,0*2+(, *� "-/-, /5� ,&(-.* 015

�� 1'$� ,2+!$/� -%� # 50� %/-+� ./-"$#2/$� 1-� #(0"' /&$�

CABG Mortality: � �/-.-/1(-,� -%� . 1($,10� 2,#$/&-(,&� ,� (0-* 1$#� "-/-, /5� /1$/5� !5. 00� &/ %1� 4'-� #($� (,

1'$� '-0.(1 *�

PTCA Mortality: � �/-.-/1(-,� -%� . 1($,10� 2,#$/&-(,&� �� 4'-� #($� (,� 1'$� '-0.(1 *�

AMI Mortality: � �/-.-/1(-,� -%� . 1($,10� 4(1'� � ./(,"(. *� #(0"' /&$� #( &,-0(0� -%� �� 4'-� #($� (,� 1'$� '-0.(8

1 *�

Cancer Pathology Reporting: � �/-.-/1(-,� -%� . 1($,10� 2,#$/&-(,&� /$0$"1(-,� %-/� ./(+ /5� " ,"$/� -%� 1'$

%$+ *$� !/$ 01�� *2,&�� -/� "-*-,�/$"12+� %-/� 4'-+� � 02/&(" *� . 1'-*-&5� "-,02*1 1(-,� /$.-/1� (0� ./$0$,1� (,� 1'$

+$#(" *� /$"-/#�

______________�'$� +$ 02/$0� *(01$#� /$� 20$#� (,� 1'$� 0$1� -%� .$/%-/+ ,"$� (,#(" 1-/0� %-/� '-0.(1 *0� ,#� -1'$/� (,01(121(-, *� ./-3(#$/0� ),-4,� 0� 1'$

��501$+� ��,#(" 1-/� �$ 02/$+$,1� �501$+�� '$� ��501$+� 4 0� #$3$*-.$#� !5� 1'$� �-(,1� �-++(00(-,� -,� �""/$#(1 1(-,� -%� �$ *1'" /$

�/& ,(6 1(-,0� �� %-/� 20$� (,� ��0� ./-"$#2/$0� %-/� ""/$#(1(,&� 02"'� ./-3(#$/0�� # .1$#� ,#� !!/$3( 1$#� 1'$� 1(1*$0� ,#

#$0"/(.1(-,0� -%� 1'$� +$ 02/$0� %/-+� 0.$"(%(" 1(-,0� ./$0$,1$#� (,� 1'$� �� + ,2 *�� '(" &-�� 2&�

�� ..�� 8�

��


BOX 4-2—Provider Performance Measures Used in the Indicator Measurement System (IMSystem) (Cont’d.)

Tumor Staging: � �.,-,.0',+� ,$� -�0'#+0/� 1+"#.%,'+%� .#/#!0',+� $,.� -.'*�.5� !�+!#.� ,$� 0&#� $#*�)#� .#�/0�

)1+%�� ,.� !,),+�.#!01*� 3&,� &�2#� /0�%#� ,$� 01*,.� "#/'%+�0#"� 5� �� *�+�%'+%� -&5/'!'�+�

Breast Cancer Testing: � �.,-,.0',+� ,$� $#*�)#� -�0'#+0/� 3'0&� �0�%#� �� ,.� %.#�0#.� -.'*�.5� .#�/0� !�+!#.

1+"#.%,'+%� '+'0'�)� ',-/5� ,.� .#/#!0',+� 3&,� &�2#� #/0.,%#+� .#!#-0,.� �+�)5/'/� .#/1)0/� '+� 0&#� *#"'!�)� .#7

!,."�

Lung Cancer Diagnosis/Staging: � �.,-,.0',+� ,$� -�0'#+0/� 3'0&� +,+7/*�))� !#))� -.'*�.5� )1+%� !�+!#.� 1+"#.7

%,'+%� 0&,.�!,0,*5� 3&,� &�2#� !,*-)#0#� /1.%'!�)� .#/#!0',+� ,$� 01*,.�

Colon/Rectum Cancer Preoperative Evaluation: � �.,-,.0',+� ,$� -�0'#+0/� 1+"#.%,'+%� .#/#!0',+� $,.� -.'7

*�.5� !�+!#.� ,$� 0&#� !,),+�.#!01*� 3&,/#� -.#,-#.�0'2#� #2�)1�0',+� 5� �� *�+�%'+%� -&5/'!'�+� '+!)1"#"� #47

�*'+�0',+� ,$� 0&#� #+0'.#� !,),+�

Trauma Monitoring: �.,-,.0',+� ,$� 0.�1*�� -�0'#+0/� 3'0&� /5/0,)'!� ),,"� -.#//1.#�� -1)/#� .�0#�� +"� .#/-'.�7

0,.5� .�0#� ",!1*#+0#"� ,+� �..'2�)� '+� 0&#� �� +"� �0� )#�/0� &,1.)5� $,.� 0&.##� &,1./� ,.� 1+0')� �� "'/-,/'0',+�

3&'!&#2#.� '/� #�.)'#.�

Head Trauma Monitoring: � �.,-,.0',+� ,$� 0.�1*�� -�0'#+0/� 3'0&� /#)#!0#"� '+0.�!.�+'�)� '+(1.'#/� 3&,� &�2#� �

�)�/%,3� !,*�� /!�)#� /!,.#� ",!1*#+0#"� ,+� �..'2�)� '+� 0&#� �� +"� �0� )#�/0� &,1.)5� $,.� 0&.##� &,1./� ,.� 1+0')

�� "'/-,/'0',+�� 3&'!&#2#.� '/� #�.)'#.�

Airway Management for Comatose Trauma: � �.,-,.0',+� ,$� �� !,*�0,/#� 0.�1*�� -�0'#+0/� 3'0&� /#)#!0#"

'+0.�!.�+'�) � '+(1.'#/� 3&,� �.#� "'/!&�.%#"� $.,*� 0&#� �� -.',.� 0,� #+",0.�!&#�)� '+01 �0',+� ,.� !.'!,0&5.,0,*5�

Timely CT Scans: � �,.� -�0'#+0/� 1+"#.%,'+%� !,*-10#.'6#"� 0,*,%.�-&5� �� /!�+� ,$� 0&#� &#�"�� 0&#�*,1+0� ,$� 0'*#� $.,*� #*#.%#+!5� "#-�.0*#+0� �..'2�)� 0,� '+'0'�)� �� /!�+�

Timely Neurological Procedures: �,.� -�0'#+0/� 1+"#.%,'+%� /#)#!0#"� +#1.,/1.%'!�)� -.,!#"1.#/�� 0&#

�*,1+0� ,$� 0'*#� $.,*� #*#.%#+!5� "#-�.0*#+0� �..'2�)� 0,� -.,!#"1.#�

Timely Orthopedic Procedures: � �,.� -�0'#+0/� 1+"#.%,'+%� /#)#!0#"� ,.0&,-#"'!� -.,!#"1.#/�� 0&#� �*,1+0

,$� 0'*#� $.,*� #*#.%#+!5� "#-�.0*#+0� �..'2�)� 0,� -.,!#"1.#�

Timely Abdominal Procedures: � �,.� 0.�1*�� -�0'#+0/� 1+"#.%,'+%� /#)#!0#"� � ",*'+�)� /1.%'!�)� -.,!#7

"1.#/�� 0&#� �*,1+0� ,$� 0'*#� $.,*� #*#.%#+!5� "#-�.0*#+0� �..'2�)� 0,� -.,!#"1.#�

Preventable Death from Pneumothorax/Hemothorax: � �.,-,.0',+� ,$� -�0'#+0/� 3&,� "'#� '+� 0&#� &,/-'0�)

3'0&� �� "'�%+,/'/� ,$� -+#1*,0&,.�4� ,.� &#*,0&,.�4� 3&,� "'"� +,0� 1+"#.%,� �� 0&,.�!,/0,*5� ,.� 0&,.�!,0,*5�

Preventable Death among Trauma Patients: � �.,-,.0',+� ,$� 0.�1*�� -�0'#+0/� 3'0&� �� /5/0,)'!� ),,"� -.#/7

/1.#� ,$� )#//� 0&�+� �� **� �%� 3'0&'+� 03,� &,1./� ,$� �� ..'2�)� 3&,� "'#� '+� 0&#� &,/-'0�)� 3'0&,10� 1+"#.%,'+%� �

)�-�.,0,*5� ,.� 0&,.�!,0,*5�

�� ,+%.#//�� $$'!#� ,$� �#!&+,),%5� �//#//*#+0�� /#"� ,+� �,'+0� �,**'//',+� ,+� �!!.#"'0�0',+� ,$� �#�)0&!�.#� �.%�7

+'6�0',+/�� &'!�%,�� 1%��

cators to be used in assessing all providers andinsurance plans. Private, for-profit companieshave also entered the market for performance in-formation, producing reports for sale to the gener-al public. A prominent example is a consumer

magazine called Health Pages that reports on theservices and prices of physicians, hospitals, andmanaged care plans in several cities for $3.95 perissue.112

112 K. Thomas, “Health Pages: A Unique New Health Magazine for Consumers,” Hospital Technology Scanner, January 1995, p. 5.


In January 1995, a private, for-profit data anal-ysis firm published a performance report on 10hospitals in Orange County, California, using raw,unadjusted Medicare billing data to measuremortality rates for coronary artery bypass graftsurgery. The $10 purchase price of the report waspartially subsidized by an undisclosed subscrip-tion fee from the study’s top-ranking hospital,which used the results in newspaper advertising.While this case prompted some observers to callfor regulation of performance measurement meth-ods and reporting—by the industry itself, if not bythe government—others expressed confidencethat “the market will eventually sort itself out.”113

❚ Information Technology andPerformance Assessment

Advanced information technologies could con-tribute to performance assessment in health carein two main ways. One is improving the measuresand data on which those assessments are based.The second is making the results of those assess-ments, and the measures and data on which theyare based, more readily accessible to payers, pur-chasers, consumers, and researchers.

By its very nature, performance assessment re-views past performance, and thus cannot feasiblyemploy clinical trials and other forms of prospec-tive analysis. Performance assessment thus em-ploys retrospective analysis that involves eitherprimary data collection or secondary analysis ofavailable administrative data, or both (as with theHEDIS and IMSystem measures). Primary dataare collected mainly through: 1) clinician reviewsof paper-based patient records, and 2) surveys ofpatients and providers. Administrative data in-clude hospital discharge abstracts, and health in-surance claims or encounter records andenrollment records. Each of these data sources has

certain limitations that advanced informationtechnologies might help overcome.

Given current information technologies andanalytic methods, tradeoffs exist between primaryand secondary data for assessing provider andplan performance. A balance must be soughtamong several considerations: 1) the clinical de-tail of the information that can be gathered, 2) thenumber of patients that can be included, 3) the costper unit of information gathered, and 4) theamount of time required to obtain and clean thedata. Larger numbers of patients enhance the pre-cision of statistical estimates, and clinical detail isessential in statistical control for confoundingvariables—particularly patient risk factors—thatcould affect the provider’s choice of services orthe patient’s outcome.

In general, administrative data can cover verylarge numbers of patients at very low cost to theanalyst and can be obtained relatively quickly.(The time and expense of collecting such datahave already been absorbed by administrativeprocesses.) However, they can cover only themore objective measures of care processes (e.g.,the proportion of diabetics receiving an annual ret-inal examination) and patient outcomes (e.g., theproportion of births with low birth weight). More-over, administrative data contain very little clini-cal detail to support process and outcomemeasures.114

In contrast, primary data collection can covermore subjective measures (e.g., appropriatenessof a procedure, patient satisfaction with the carereceived, patient self-perception of health statusand quality of life, etc.) as well as several of themore objective ones. Moreover, it can obtain richdetail on those measures: clinical detail, in thecase of patient record review; and perceptual/atti-tudinal detail, in the case of surveys. However,

113 “Bypass Surgery Report Ignites Uproar Among Calif. Hospitals,” Report on Medical Guidelines & Outcomes Research, Jan. 26, 1995,

pp. 1, 2, and 12.

114 J.G. Jollis et al., “Discordance of Databases Designed for Claims Payment versus Clinical Information Systems: Implications for Out-

comes Research,” Annals of Internal Medicine, vol. 119, No. 8, Oct. 15, 1993, pp. 844-850.


such data are collected at much greater cost in bothtime and money; so they are usually gathered onfar fewer patients, thereby reducing the precisionof statistical estimates. Ideally, all measureswould be obtained in complete clinical detail onvery large numbers of patients very quickly and atvery low cost. This is precisely the vision offeredby advanced information technologies.

A major limitation of readily available admin-istrative data is the absence of measures of variousconfounding factors that may affect a provider’schoice of services or a patient’s outcome, and thusdistort the true effects of the processes of care be-ing evaluated. The most important confoundingvariables are patient risk factors (demographictraits, complicating health problems, etc.). Failingto adjust adequately for such factors could mis-lead payers, purchasers, and consumers regardingprovider or plan performance,115 as illustrated bythe recent case involving a private report card onhospitals in Orange County, California.

Many of the most important patient risk factorsare best measured using detailed clinical data,such as physical findings and diagnostic test re-sults. Computerization of such clinical informa-tion should make it easier to obtain and use inperformance assessments. One approach wouldbe to require that more clinical information be in-cluded in administrative data. In recent years,payers and government agencies have mandatedincreased numbers of diagnosis and procedurecodes and other clinical data elements included inclaims and discharge abstracts. This has greatlyincreased the information burden on providers;116

yet it still does not yield the kinds of clinical detailrequired for valid performance assessment. More-

over, accuracy problems in diagnosis and proce-dure coding render those data suspect.117

The more promising approach to providingneeded clinical information is to computerize thepatient record. Rather than having clinicallytrained personnel read, interpret, and code the in-formation contained in paper-based patient re-cords, most of the relevant information could beprecoded in the electronic patient record andreadily extracted for analysis. Alternatively, un-coded information (free text) contained in theelectronic patient record could be processedthrough advanced methods of pattern recognition,such as natural language processing (see chapter2). The usefulness of these capabilities greatly de-pends on three other aspects of advanced informa-tion technologies: input, storage, and retrieval.That is, to be useful for performance assessmentpurposes, the information in the electronic patientrecord must be accurately and easily entered (pre-ferably at the point of care) and extracted (usuallyat sites other than the point of care, e.g., an ana-lyst’s office). Moreover, storage capacities mustbe adequate to handle the huge quantities of in-formation involved.

As stated earlier, computer networks couldmake it easier to track the care and outcomes of in-dividual patients by facilitating record linkageacross all providers and departments. Networkscould also make it easier to share patient data, per-formance measurement algorithms, and assess-ment results among providers, payers, purchasers,and researchers to compare the performance ofproviders or plans. Like assessing the effective-ness of specific health services, such comparisonswould require using health problems, process and

115 Epstein, op. cit., footnote 98, pp. 58, 60; S. Salem-Schatz et al., “The Case for Case-Mix Adjustment in Practice Profiling: When Good

Apples Look Bad,” Journal of the American Medical Association, vol. 272, No. 11, Sept. 21, 1994, pp. 871-874.

116 D.R. Longo et al., Inventory of External Data Demands Placed on Hospitals (Chicago, IL: The Hospital Research and Educational Trust

of the American Hospital Association, 1990).

117 See L.I. Iezzoni, Risk Adjustment for Measuring Health Care Outcomes (Ann Arbor, MI: Health Administration Press, 1994), pp.142-167; R.A. Bright, J. Avorn, and D.E. Everitt, “Medicaid Data as a Resource for Epidemiologic Studies: Strengths and Limitations,” Journalof Clinical Epidemiology, vol. 42, No. 10, 1989, pp. 941-943; J. Whittle, “Large Administrative Database Analysis,” Tools for EvaluatingHealth Technologies: Five Background Papers, U.S. Congress, Office of Technology Assessment, OTA-BP-H-142 (Washington, DC: U.S.Government Printing Office, February 1995), pp. 33-35.


outcome measures, and analytical methodologiesthat are as similar as possible across providers.These efforts would also be facilitated by messag-ing standards for electronic exchange of informa-tion among different computer systems, and bymethods of translating among disparate clinicalnomenclatures and coding systems.

CONCLUSIONS

❚ Summary of FindingsAdvanced information technologies—electronicpatient records, structured data entry, new human-computer interface technologies, portable com-puters, automated data capture, relationaldatabases with online query, knowledge-basedcomputing, and computer networks—can poten-tially improve the quality of health care. Theycould do so by enhancing clinical decision supportand by improving data for assessing the effective-ness of health services and the performance ofhealth care providers and insurance plans. Specifi-cally, they could facilitate:

� faster and easier collection and entry of in-formation about the patient’s health problemand background, with portions of that informa-tion being:

—entered by clinicians at or near the point ofcare;

—captured directly from diagnostic and moni-toring equipment (including digitized radio-graphic images, full-motion videos, andsound recordings); or

—entered by the patient prior to care;

� faster, easier, and better targeted search and re-trieval (possibly at the point of care) of:

—previously collected information about thepatient; and

—information about the kind of health problemafflicting the patient and alternative tests andtreatments for it, drawn from local or remoteknowledge bases;

� more flexible organization and display of thisinformation as appropriate for particular clini-cians;

� development of computer-based clinical proto-cols and other forms of CDSSs that apply deci-sion rules and other knowledge-basedapproaches to information about the patientand the health problem;

� more rigorous construction and analysis ofmeasures of service effectiveness and providerand plan performance; and

� more rapid and widespread dissemination ofnot only the results of these measures and localclinical research using CDSSs, but also the pa-tient data, measurement algorithms, andCDSSs on which those results are based.

Currently, empirical evidence demonstratingthe ability of these technologies to achieve thesegoals is limited, mixed, or incomplete. Moreover,concerns have been raised about possible adverseeffects on the quality of health care arising fromthese applications, including:

� incorrect parameters or criteria, or omitted oraltered steps, in clinical decision support sys-tems that could lead to inappropriate care;

� excessive reliance on clinical decision supportsystems, which could undermine the ability ofclinicians to exercise professional judgment innonroutine cases and reduce the interpersonalaspects of patient care (“ the quality of caring”);and

� the temptation to use readily available adminis-trative data for assessing the effectiveness ofspecific health services or the performance ofproviders or insurance plans. If the data are in-complete or inaccurate, the results could bemisleading.

❚ Policy OptionsThe private sector has been largely responsible forthe development and application of informationtechnologies in clinical decision support and per-formance assessment of health care providers andinsurance plans. The federal government’s rolehas mainly involved:

� developing information systems and perfor-mance measures for its own health insuranceand health care delivery programs, most nota-bly Medicare;


� funding of intramural and extramural researchand demonstration projects; and

� participating in consensus standards-develop-ment processes along with private sector orga-nizations.

All of these activities in both the private and pub-lic sectors are likely to continue, with some in-creasing and others decreasing. In an era ofbudgetary and regulatory restraints, however,major new government initiatives, such as fund-ing for technology development or mandated reg-ulation of clinical information systems, areunlikely. It can be argued that this is appropriate—in other words, that the federal government shouldnot interfere in private market decisions regardingthe selection of new technologies or their applica-tions.

On the other hand, the federal government—specifically HCFA—is responsible for ensuringthe quality of health care rendered to Medicare andMedicaid beneficiaries.118 Recent efforts to movemore beneficiaries into managed care have under-scored quality concerns, given the expectationthat capitation creates an incentive for underser-vice.119 Several policy issues regarding the poten-tial impact of information technology on thequality of care delivered to Medicare and Medic-aid beneficiaries deserve the attention of federalpolicymakers.

Effectiveness and SafetyThe foremost issue is the extent to which clinicalinformation systems actually change clinicalpractice patterns and patient outcomes, andwhether those changes are beneficial to providersand patients. Empirical research on this issue re-mains limited, mixed, or incomplete, and moresolid evidence regarding these impacts needs to beobtained. If these systems do indeed improve thequality of care, then the next set of issues can be

addressed: What are the most efficient means ofdeveloping and implementing such systems?

Much of the research supporting the develop-ment and evaluation of clinical information sys-tems (including CDSSs) has been conducted byacademic institutions and other private sector or-ganizations. Many of these projects have receivedgrant or contract funding from federal executivebranch agencies, mainly NLM and AHCPR (or itspredecessor, the National Center for Health Ser-vices Research, NCHSR). However, there hasbeen little coordination among these privately andpublicly funded projects in terms of their methodsof evaluating the effectiveness and safety of clini-cal information systems. The focus of these evalu-ations should be on the impacts of these systemson clinical practice patterns and patient outcomes.Where possible, these evaluations should be con-ducted prospectively, including randomized con-trolled trials.

Given its methodological shortcomings, asses-sing the performance of providers and insuranceplans and disseminating information regardingthat performance to various parties may prove tobe an ineffective approach to improving the quali-ty of health care.120 At present, however, there isgreat demand for performance information in boththe public and private sectors; and if such informa-tion is going to be produced and used, it should beas valid and reliable as possible. Advanced in-formation technologies—primarily electronic pa-tient records—promise to improve performanceassessment by making more information on pa-tients, providers, services, and outcomes morereadily available in a more detailed, accurate, andusable form. Most importantly, such informationcould improve methods of risk adjustment for per-formance indicators that are based on health careprocesses and outcomes. Conversely, the devel-opment of reliable and valid performance assess-

118 The state governments share responsibility for the Medicaid program with the federal government.

119 Given a fixed payment per plan member, providers may be tempted to minimize the volume and/or intensity of services rendered for

each patient.

120 Epstein, op cit., footnote 98.


ment indicators could improve the application ofinformation technology to health care by identify-ing the most important data elements to include inelectronic patient records.

Like clinical information systems, much of theresearch and development work on performanceassessment and risk adjustment has been con-ducted by private sector organizations, often withfunding from federal agencies, mainly AHCPR(or NCHSR) and HCFA. Working with privatesector organizations, HCFA has begun develop-ing the DEMPAQ indicators for ambulatory careamong Medicare beneficiaries and adapting theprivately developed HEDIS system to the Medi-caid managed care population. Nonetheless, therehas been little coordination among all of these pri-vately and publicly funded projects on perfor-mance assessment and risk adjustment, orbetween these projects and those evaluating theeffectiveness and safety of clinical informationsystems. The basic issue is whether all of thesefederal efforts should continue as they are, orwhether more or less funding and/or coordinationwould be appropriate.

OPTION 1a: �� "

��

�� !�� "

��

� develop and test the reliability and validity ofvarious methods of measuring and assessing(with risk adjustment) the performance of pro-viders and health plans;

� develop, implement, and evaluate specific sys-tems of risk-adjusted performance indicators;

� evaluate the effectiveness and safety of clinicalinformation systems, including CDSSs.

The FDA could employ the results of the evalu-ations of clinical information systems in formulat-ing regulations for that class of medical software,and HCFA could adapt the most promising perfor-mance assessment systems for use in its Medicareand Medicaid programs—as it is now doing withthe Medicaid HEDIS indicators (which are not yetrisk-adjusted). This option would maintain thecurrent approach of funding research, develop-

ment, and evaluation programs through severalgovernment agencies, with little coordinationamong them. It would thus preserve the autonomyin program direction currently enjoyed by the var-ious agencies and the consequent diversity in thetypes of programs and their results. On the otherhand, HCFA would have to: 1) wait for the neededperformance assessment systems to be developedand evaluated; and 2) use performance indicatorsthat still may not be truly appropriate for theMedicare or Medicaid populations.

OPTION 1b: ��

��

�� "

�� "

�� !��

��

This is HCFA’s current approach in developingthe DEMPAQ indicators for ambulatory careamong Medicare beneficiaries. Given that HCFAis also adapting privately developed indicators(Medicaid HEDIS), options 1a and 1b are notmutually exclusive. However, option 1a would bemore costly than option 1b because, under option1a, development and evaluation funding would bespread over a broader array of performance assess-ment systems as well as clinical information sys-tems. From another perspective, more effort couldbe concentrated on the information needs of theMedicare and Medicaid programs for a givenamount of funding. On the other hand, option 1bwould sacrifice federal direction of evaluations ofclinical information systems that could be usefulto the FDA in formulating regulations.

OPTION 1c: �� "

��

��

��

This option could be adopted regardless ofwhether option 1a, option 1b, or both were pur-sued. The designated agency—such as HCFA orAHCPR—would ensure that all federally fundedprojects employ rigorous and uniform methods toenhance the soundness and comparability of their


results. In addition, agency personnel would meetwith representatives of private sector corpora-tions, foundations, and research organizationsthat also fund or conduct such projects to discussthe most promising approaches to research, devel-opment, and evaluation. This option would re-quire only small additional costs for personnel,travel, and meetings; yet it could greatly increasethe value and timeliness of project results. On theother hand, it would diminish the autonomy inprogram direction currently enjoyed by the vari-ous agencies and the consequent diversity in thetypes of programs and their results.

OPTION 1d: ��

��

��

��

��

��

This option would capitalize on the diversearray of performance assessment methods andsystems being developed in the private sector. Itwould reduce government expenditures, depend-ing on the amount of work needed to adapt pri-vately developed performance assessmentsystems to the Medicare or Medicaid popula-tions—which in turn would depend on the initialsuitability of those systems’ indicators. However,to an even greater extent than with option 1a,HCFA would have to: 1) wait for the needed per-formance assessment systems to be developed andevaluated in the private sector; and 2) use perfor-mance indicators that still may not be trulyappropriate for the Medicare or Medicaid popula-tions.

Until more solid evidence is available regard-ing the effectiveness and safety of existing clinicalinformation systems and the reliability and validi-ty of performance assessment systems, more dras-tic action—such as mandating the testing andcertification of all such systems—is probably notjustified. Legal questions regarding who should

be held liable in situations in which such systemslead clinicians to make decisions that harm pa-tients are probably best left to the courts to re-solve.

Standards and TechnologyAssuming that clinical information systems arefound to be effective and safe in terms of their im-pacts on practice patterns and patient outcomes,the next set of issues focuses on the most efficientmeans of developing and implementing those sys-tems. Three options regarding government in-volvement in the development of standards andtechnology that were presented in chapter 2 war-rant additional emphasis here. One is continuedgovernment participation (along with private sec-tor organizations) in the voluntary, cooperative,public- private process of developing consensusstandards for electronic messaging (exchange ofinformation among disparate computer systems).The second is funding and coordinating researchto overcome specific technological barriers (e.g.,limitations of electronic storage devices). Theseactions would not only facilitate the developmentand testing of clinical information systems andperformance assessment systems, but would alsoenhance the clinical knowledge on which they arebased.

The third option concerns continuation of fund-ing for NLM to develop the Unified MedicalLanguage System (UMLS). A major problemconfronting the UMLS project is that one of themost widely used systems for classifying and cod-ing health care services, called the Physicians’Current Procedural Terminology, Fourth Edition(CPT-4), is copyrighted by the American MedicalAssociation (AMA). Thus, the more recent ver-sions of CPT-4 cannot be incorporated intoUMLS.121 Many major payers currently employCPT-4 for “professional” billing by clinicians andother noninstitutional providers and suppliers, butalso use the International Classification of Dis-

121 McCray, op. cit., footnote 66.


eases, Ninth Edition, Clinical Modification (ICD-9-CM), Volume 3 (Procedures), for billing by in-patient hospitals and other institutional providers.

For payment and other purposes, services ren-dered by a clinician in an inpatient setting must becoded using both of these systems, creating addi-tional costs for providers. For many services,however, the codes in ICD-9-CM cannot beequated (”crosswalked”) with those in CPT-4 be-cause of substantial structural differences betweenthe two coding systems.122 Moreover, bothICD-9-CM (Vol. 3) and CPT-4 have serious tech-nical limitations, such as overlapping and duplica-tive codes and inconsistent and noncurrent use ofterminology. Most importantly, neither has ade-quate room for expansion, so both are running outof codes as new services are created or differentuses of existing services are distinguished. In ad-dition, neither system provides sufficient clinicaldetail to support the creation of the kinds of data-bases required to accurately assess patient out-comes using advanced information technol-ogies.123

Citing these and other problems, the NationalCommittee on Vital and Health Statistics(NCVHS), an advisory body to the Secretary ofHealth and Human Services, has recommendedthe development of a single, unified classificationand coding system that covers all health care ser-

vices rendered by all providers in all settings, andthat can be used for multiple purposes (reimburse-ment, research, etc.).124 The NCVHS maintainedthat, although implementing such a system wouldinitially be costly (particularly in the conversionof computer systems, databases, reimbursementsystems, and documentation), it would savemoney in the long run through administrative sim-plification; more accurate coding and documenta-tion; encouragement of automation and uniformterminology, data collection, and data processing;better monitoring and detection of errors, fraud,and ineffective procedures; and reduced trainingcosts.125

Legislation that would have required the devel-opment of such a system was introduced in the103d Congress (H.R. 1255), but was tabled in fa-vor of incorporation into broader health care re-form legislation that subsequently did not pass.126

A survey of users of ICD-9-CM (Vol. 3) andCPT-4 found extensive dissatisfaction with themand widespread support for the concept of asingle, unified system. Opposition to this conceptwas expressed mainly by physicians and represen-tatives of medical organizations.127 On the otherhand, concern has been expressed about the pro-prietary nature of CPT-4 and the AMA’s role inmaintaining a system that is widely used for pub-lic purposes.128

122 For example, in CPT-4 the code for total abdominal hysterectomy (58150) includes procedures performed with or without removal ofovaries or fallopian tubes, whereas ICD-9-CM (Vol. 3) has separate codes for total abdominal hysterectomy (68.4) and removal of ovaries and/or tubes (65.3 through 65.6). Thus, the CPT-4 code cannot be used to identify patients who had undergone only a total abdominal hysterectomy(without removal of ovaries or fallopian tubes). See American Medical Association, Physicians’ Current Procedural Terminology, 1994 (Chi-cago, IL: September 1993), p. 355, and Practice Management Information Corp., International Classification of Diseases, 9th Revision, Clini-cal Modification, Fourth Edition, 1993 (Los Angeles, CA: 1993), pp. 935, 937.

123 Iezzoni, Risk Adjustment for Measuring Health Care Outcomes, op. cit., footnote 117, pp. 164-167; Whittle, op. cit., footnote 117; U.S.Department of Health and Human Services, Public Health Service, National Committee on Vital and Health Statistics, The National Committeeon Vital and Health Statistics, 1993 (Washington, DC: May 1994), pp. 8-10, 54-75.

124 U.S. Department of Health and Human Services, Public Health Service, National Committee on Vital and Health Statistics, op. cit., foot-

note 123, pp. 54-55.

125 Ibid., pp. 59-62.126 Ibid., p. 56.127 Ibid., pp. 56-58.128 Ibid., p. 60.


The NCVHS concluded that existing serviceclassification and coding systems “are structural-ly flawed and wastefully redundant,” and that nei-ther ICD-9-CM (Vol. 3) nor CPT-4 “can be ‘fixed’without a complete overhaul (that is, creating anew classification).”129 Yet in 1994, even HCFAreaffirmed its intention to continue this dual cod-ing system policy in its Medicare and Medicaidprograms, despite the substantial barriers thisposes to efficient information processing andanalysis.130 Although the agency intends to con-duct a pilot study on the feasibility of modifyingor replacing ICD-9-CM (Vol. 3), and will remainopen to ideas regarding a unified system, HCFAintends to continue its use of CPT-4 and its “coop-erative relationship with the AMA.”131

OPTION 2a: ��

��

��

��

��

Although this research could be conducted ordirected by a single agency (such as NLM, HCFA,or AHCPR), extensive involvement by and coop-eration with other agencies, private sector orga-nizations (providers, payers, research associa-tions, and particularly the AMA), and the WorldHealth Organization (WHO) would be essential.If such a classification system were developed,NLM could then incorporate it into UMLS. Thisresearch would incur modest additional cost, andwould further delay development of a unified ser-vice classification and coding system.

OPTION 2b: ��

��

��

This option would bypass research on the feasi-bility of developing such a system (option 2a).Again, the program to develop a new system couldbe conducted by one or more executive branchagencies, with extensive input from other agen-cies, private sector organizations, and WHO. Thisoption would also incur larger additional coststhan option 2a; however, it could expedite devel-opment of the new system. On the other hand, itwould be more objectionable to parties that arecommitted to the current dual coding systempolicy.

OPTION 2c: ��

��

��

��

��

In addition to HCFA, these agencies include theDepartment of Defense, the Department of Veter-ans Affairs, and the Indian Health Service. Pro-moting efficient information processing andanalysis in these programs would seem warranted,considering the government’s enormous invest-ment in them. Given the magnitude of these pro-grams in the health care marketplace, most privatepayers would probably soon adopt the new unifiedservice classification and coding system, just asthey began using the ICD-9-CM system afterHCFA implemented it. On the other hand, such a

129 Ibid., p. 54.130 U.S. Department of Health and Human Services, Public Health Service, National Committee on Vital and Health Statistics, Subcommit-

tee on Medical Classification Systems, Meeting Minutes, Washington, DC, Apr. 18, 1994, pp. 5-7, 9-10. For physician and supplier billing,HCFA actually uses its own system, called the HCFA Common Procedure Coding System (HCPCS), that incorporates CPT-4 but also containsadditional codes.

131 Ibid., p. 5. In another manifestation of this relationship, HCFA and the AMA recently formed the National Uniform Claim Committee,“designed to give physicians more of a say in the creation and implementation of standards for electronic claims processing.” This move hasbeen harshly criticized by some participants in the existing voluntary, cooperative, public-private process of consensus standards development.“Yet Another Group Prepares To Work on Claims Standards,” Health Data Management, May 1995, p. 14.


mandatory approach would probably be the mostobjectionable option to parties that are committedto the current dual coding system policy.

OPTION 2d: ��

� ��

��

Rather than mandating and/or funding the de-velopment of a unified service classification andcoding system, Congress could continue to leavethe development of such a system to the privatesector. Minimal funding could be provided for ex-isting agencies (e.g., NLM) and committees (e.g.,

NCVHS) to monitor private sector activities andto facilitate those activities—for example, bysponsoring meetings among interested parties.This option would capitalize on the existing vol-untary, cooperative, public-private process of de-veloping consensus standards. It would also be theleast objectionable option to parties that are com-mitted to the current dual coding system policy,and it would only marginally increase governmentexpenditures. Its major drawback would be thelong period of time that would probably be re-quired for the consensus standards-developmentprocess to produce the needed system.