View
219
Download
1
Category
Preview:
Citation preview
Medical Information RetrievalChallenges in a Webbed World
William Hersh, M.D.
Associate Professor and Chief
Division of Medical Informatics and Outcomes Research
Oregon Health Sciences University
hersh@ohsu.edu
Overview
Describe current information retrieval technology
Summarize IR research activities and their results
Discuss the implications of the World Wide Web (WWW) for IR
Overview of IR Process
Indexinglanguage
Indexinglanguage
QueriesQueries DocumentsDocuments
Searchengine
Searchengine
Retrieval Indexing
What is the field of IR?
Concerned with creation, storage, organization, and retrieval of computer-based information
“IR” has traditionally focused on retrieval of information from heterogeneous textual databases
Recent expansion to multimedia and integration with “traditional” databases
Why is IR pertinent to health care?
Growth of knowledge has long surpassed human memory capabilities
Clinicians have frequent and unmet information needs
Primary literature on a given topic can be scattered and hard to synthesize
Non-primary literature sources are often neither comprehensive nor systematic
Further reading
Hersh WR, Information Retrieval: A Health Care Perspective, Springer-Verlag, 1996
Hersh WR, Hickam DH, How well do physicians use electronic information retrieval systems? A framework for investigation and systematic review, Journal of the American Medical Association, 1998, 280: 1347-1352
IR state of the art
Databases Indexing Retrieval Evaluation
Databases
Bibliographic– References to journal literature– Used in initial IR systems– Most famous example is MEDLINE
» Nearly 9 million references to peer-reviewed literature dating back to 1966
» Covers about 3,000 journals, mostly English-based
» About 300,000 new references added yearly
» Maintained by National Library of Medicine
Databases (cont.)
Full-text– Journal literature has been available for over a
decade in text-only and at high cost– Last decade has seen increasing growth of CD-
ROM market– New “evidence-based” resources are becoming
available, e.g., Best Evidence, Cochrane Hypertext
– Information linked in non-linear fashion
Indexing
Two major types:– Human indexing with controlled vocabulary
» MEDLINE uses the 18,000-term Medical Subject Headings (MeSH) vocabulary
– Computer assignment of all words in record» Often a stop word list to remove common words
(e.g., the, and, which) is used
» Some systems “stem” words to root form (eg, coughs to cough)
Limitations of indexing approaches
Human indexing– Inconsistency– Inadequate indexing vocabulary
Word indexing– Synonymy - e.g., cancer and carcinoma– Polysemy - e.g., lead– Granularity - e.g., antibiotics, penicillin– Focus
Retrieval
Traditional approach: indexing terms connected by AND, OR
Most bibliographic systems allow searching on both vocabulary and text words
Proximity operators require words to be within a certain range
Some systems hide Boolean operators
Limitations of retrieval approaches
Novices confuse ANDs and ORs Complex user interfaces dissuade busy
users Returned documents displayed in arbitrary
or, at best, reverse chronological order
An alternative approach to indexing and retrieval
Called vector-space, word-statistical, automated retrieval…
Developed by Salton in 1960’s but since works best for end-users did not achieve commercial prominence until 1990’s
Based on notion of finding similarity in words between user’s query and document
Used in Knowledge Finder (Aries) and most Web search engines
Word-statistical indexing
Indexing done of all words (though nothing precludes use of MeSH or other terms)
After stop word filtering and stemming, each word in each document assigned a weight based on product of IDF*TF:– Inverse document frequency of term i
» IDFi = log(# documents/# documents with term)+1
– Term frequency of term i in document j» Tfij = log(frequency of term in document)+1
Word-statistical retrieval
Queries entered in natural language, subject to same stop list and stemming
Each document gets a score based on sum of weights for each query term in the document
Results are sorted and presented to user (relevance ranking)
This approach allows other features:
Relevance feedback– After user designates relevant documents,
query modified Query expansion
– Same but using top-ranked documents without user relevance designations
Evaluation
What questions to ask?– Is system used?– Are users satisfied?– Do they find relevant information?– Do they complete their desired task?
Most research has focused on retrieval of relevant documents
Relevance-based measures
Recall =# retrieved and relevant documents
-------------------------------------------
# relevant documents in collection Precision =
# retrieved and relevant documents
-------------------------------------------
# relevant documents in search
Comments about recall and precision
There tends to be a trade-off between the two
“Relevance” can be a slippery notion It is unclear whether they correlate with a
user’s success in using an IR system The proliferation of standard test collections
leads to a great deal of research that excludes real users
How well do users search?(Haynes et al., Annals of Internal Medicine, 1990)
Recall Precision
Novice 27% 38%
Experiencedclinician
48% 48%
Librarian 49% 57%
More searching results(Hersh et al., Bull Med Libr Assoc, 1994)
Retrieved Recall PrecisionNovice physicians Knowledge Finder 88.8 68.2 14.7Novice physicians KF top 15 14.6 31.2 24.8Librarians Full MEDLINE 18.0 37.1 36.1Librarians Text words only 17.0 31.5 31.9Exp. Physicians Full MEDLINE 10.9 26.6 34.9Exp. Physicians Text words only 14.8 30.6 31.4
Other results
Little overlap among retrieval sets– Searchers tend to find similar quantities of
disparate relevant documents Novice searchers are satisfied with results
– Adequate information or ignorant bliss?
New approaches to evaluation
Changing the research questions– How well can clinical users answer questions?
– What factors are association with success?» Demographics, experience, cognitive factors, and
searching mechanics?
Ongoing study funded by NLM Challenges
– Appropriate questions, database, sample size, etc.
IR research directions
Enhancing word-statistical approaches Linguistic approaches Enhancing conventional indexing and
retrieval
Enhancing word-statistical approaches
Passage retrieval– Giving weight to documents that have sections
mapping closely to the query Use of phrases
– High, blood, and pressure have more meaning when occurring near each other
Linguistic approaches
Syntactic approaches– Conceptual matter tends to occur in noun
phrases Semantic approaches
– Can we overcome problems of synonymy, polysemy, granularity, etc.?
Identifying semantics in documents
SAPHIRE (Hersh and Hickam, 1995)– Direct mapping of text to terms in large
controlled vocabulary (UMLS Metathesaurus)– Works best when exact terms and synonyms
present, less well when terms vague or synonyms non-standard
MEDSPACE (Schatz, 1997)– Large-scale processing to uncover underlying
related terms and literatures
Enhancing conventional systems
Better content– Evidence-based resources
» More informative abstracts, e.g., Best Evidence
» Systematic reviews, e.g., Cochrane Database of Systematic Reviews
» Critically-Appraised Topics (CATs)
Better indexing– NLM’s MedIndex system provides expert
assistance to indexers
Enhancing conventional systems (cont.)
Better retrieval– NLM’s Internet Grateful Med looks for
common searching mistakes (eg, excessive AND’s) and informs searcher
Better vocabularies– NLM’s UMLS Project adds terminology from
other vocabularies
IR and the World Wide Web
Indexing and retrieval approaches Implications for scientific publishing Implications for health care Limitations
Indexing and retrieval on the Web
Web crawlers– Index everything they find– Examples: Alta Vista, InfoSeek, Lycos– Problems: non-discriminating, word only
Filtering and/or classifying– Sites filtered and/or classified based on criteria– Examples: Yahoo, CliniWeb, OMNI– Problems: maintenance, intended audience
Implications for scientific publishing
Peer-review process– Imperfect but best means for controlling
quality in publications Responsibility
– Increased anonymity of Web enhances ability for misrepresentation
Liability– Who is liable for inaccurate information?
Implications for health care
Informativeness vs. marketing– There is potential conflict between providing
information and self-promotion Patient empowerment
– Absolutely important but much potential for damage from misinformation
Much medical informationis on the Web
“Free” information from government agencies, medical schools, and advocacy groups is easy to access and use
“Best” information from traditional medical publishers still costly and fragmented
Some well-known launching pads– Medical Matrix: www.medmatrix.org/– CliniWeb: www.ohsu.edu/cliniweb/
Limitations of the Web(Hersh, ACPJC, 1996)
Difficult to find information - a diversity of different search engines, each with its own benefits and limitations
Everyone can be a publisher - Good for democratic society, less so for scientific and professional fields
Misrepresentation and fraud - Web can amplify misinformation and allow easy fraud
Some have expressed concern about free information on Web
Silberg et al. (JAMA, 1997) suggested standards for health information on Web– Authorship - names, affiliations, and credentials
– Attribution - references, sources, and (where appropriate) copyright
– Disclosure - potential and real conflicts of interest
– Currency - dates content posted and updated
But applicability and quality of Web content is poor
Hersh, Gorman, and Sacherek, JAMA, 1998 Searched on 50 questions generated by
clinicians Less than 10% of pages relevant, none for
half of queries Low percentage of JAMA quality indicators
Final thoughts
We are on the threshold of an exciting new era in communications and information dissemination– Integrity of information and responsibility for
it must be maintained– It should augment and not substitute for human
communication
References Cited: 1. Hersh, W., Information Retrieval: A Health Care Perspective. 1996, New York: Springer-Verlag. 2. Hersh, W. and D. Hickam, How well do physicians use electronic information retrieval systems? A framework
for investigation and review of the literature. Journal of the American Medical Association, 1998. 280: 1347-1352.
3. Haynes, R., et al., Online access to MEDLINE in clinical settings. Annals of Internal Medicine, 1990. 112: 78-84.
4. Hersh, W. and D. Hickam, The use of a multi-application computer workstation in a clinical setting. Bulletin of the Medical Library Association, 1994. 82: 382-389.
5. Hersh W. and D. Hickam, Information retrieval in medicine: the SAPHIRE experience. Journal of the American Society for Information Science, 1995. 46: 743-747.
6. Schatz B., Information retrieval in digital libraries: bringing search to the net. Science, 1997. 275: 327-334. 7. Hersh, W., Evidence-based medicine and the Internet. ACP Journal Club, 1996. 5(4): A12-A14. 8. Silberg, W., G. Lundberg, and R. Musacchio, Assessing, controlling, and assuring the quality of medical
information on the Internet: caveat lector et viewor - let the reader and viewer beware. Journal of the American Medical Association, 1997. 277: 1244-1245.
9. Hersh, W., P. Gorman, and L. Sacherek, Applicability and quality of information for answering clinical questions on the Web. Journal of the American Medical Association, 1998. 280: 1307-1308.
URLs:Division of Medical Informatics & Outcomes Research: www.ohsu.edu/bicc-informatics/CliniWeb: www.ohsu.edu/cliniweb/SAPHIRE International: www.ohsu.edu/cliniweb/saphint/
Recommended