Embed Size (px)
Citation preview
Searching & screening efficiency: emerging
techniques for systematic review
Alison WeightmanCo-convenor Cochrane Information
Retrieval Methods Group
This presentation• Searching for studies– Minimise the number of databases searched?– Maximise the value of search terms → text mining?
• Screening studies– Boost relevant records to the top of the list → text mining?– Use two screens
Emphasis on text mining for search development …. and on free software
Acknowledgement:
Selected text analysis tools
Julie GlanvilleAssociate Director, York Health Economics Consortium, University
of York, UK
Stages of a systematic review
Step 1: Develop a protocolStep 2: Search for the evidenceStep 3: Screen and select studiesStep 4: Assess the evidenceStep 5: Extract the dataStep 6: Synthesise the evidenceStep 7: Present findings
Step 2: Search for the evidence – traditional steps
Manual search of…….• Multiple databases with lots of search terms
many 000s hits
Plus supplementary searching• Web sites • Reference lists • Citation tracking• Expert contact
Step 2: Search for the evidence – new approaches
• 2.1 Minimise the number of databases.... analyse previous reviews?
• 2.2 Maximise the value of search terms.... use text mining?
Morgan EBLIP 2015….. Searches for public health intervention reviews should include Medline/PubMed, reference list checking and citation tracking along with a small number of specially selected additional databases and expert contact/specialist web site searching
2.1 Minimise the number of databases?
Deriving high-quality (useful) information from unstructured
text. The text words in relevant publications are
grouped or structured to find word combinations that can be
used to locate other similar materials
2.2 Maximise the value of search terms?..consider text mining
2.2 Text mining to maximise the value of search terms
O’Mara-Eves 2014: ‘We highly recommend that reviewers use automatic term extraction to
identify additional search terms, particularly where the topic is
broad or multidisciplinary’
2.2 Text mining for search development: A worked example
A systematic review of musculoskeletal care pathways for adults with hip and knee pain at the interface between primary and secondary care
Three main concepts for search strategy:Musculoskeletal + pathway + primary-secondary interface
Most common text words PubMed PubReMiner, GoPubMed Endnote
Most common phrases Termine
Most common indexer-defined subject headings (MeSH) MeSHonDemand GoPubMed Yale MeSH analyser
Some text analysis tools
• http://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi
• Example topic1. Musculoskeletal primary care referral:
Musculoskeletal + care pathway + primary-secondary12 relevant papers. PubMed identifiers
23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909 , 22716771, 23750266, 22711378
PubMed PubReMiner
Musculoskeletal, care (text)
Musculoskeletal diseases, referral and consultation (MeSH)
Initial terms suggested by PubReMiner
Musculo-skeletal
Title/abstract words: MusculoskeletalMedical Subject Heading, MeSH: Musculoskeletal diseases/
Care Pathway
Title/abstract words: Care
Primary-Secondary care
Medical Subject Heading, MeSH: Referral and consultation/
• http://www.nactem.ac.uk/software/termine/
• Example topicMusculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary12 relevant papers. AbstractsEnter all 12 abstracts in full
BACKGROUND: Musculoskeletal condition assessment and management is increasingly delivered at the primary to secondary care interface, by inter-disciplinary triage and treat services.OBJECTIVES: This review aimed to describe Intermediate Care pathways, evaluate effectiveness, describe outcomes and identify gaps in the evidence……………
Termine combined with visual scan of abstracts
Terms scoring ≥3
Rank Term Score1conversion rate 8.752primary care 73secondary care 53musculoskeletal service 55patient satisfaction 4.3333336systematic review 46musculoskeletal disorder 48health care service 3.1699258surgery conversion rate 3.1699258uncomplicated musculoskeletal problem 3.169925
11scope physiotherapist 311treatment effectiveness 311orthopaedic consultation 311orthopaedic surgeon 3
Search v2 after Termine analysis and visual scan of abstracts
Musculo-skeletal
Title/abstract words: Musculoskeletal, orthop?edic
Medical Subject Heading, MeSH: Musculoskeletal diseases/
Care pathway
Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*)
Primary to secondary
Title/abstract words: Conversion rate, primary adj2 secondary, scope physiotherap*, orthop?edic adj (consulta* or surgeon), Refer adj3 (GP or general practi* or primary care)
Medical Subject Heading, MeSH: Referral and consultation/
• http://mesh.med.yale.edu/
• Example topicMusculoskeletal primary care referral: Musculoskeletal + care pathway + primary-secondary12 relevant papers. PubMed identifiers
23561693, 25442485, 25242531, 21056164, 25410703, 15576743, 10848384, 24337981, 25342909 , 22716771, 23750266, 22711378
Yale MeSH analyser
Musculoskeletal pain (MeSH)
Critical pathways (MeSH)
Search v3 after adding in Medical Subject Headings via Yale MeSH analyser
Musculo-skeletal
Title/abstract words: Musculoskeletal, orthop?edic
Medical Subject Headings, MeSH: Musculoskeletal diseases/, Musculoskeletal pain/
Care pathway
Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*)
Medical Subject Headings, MeSH: Critical pathways/
Primary to secondary
Title/abstract words: Conversion rate, primary adj2 secondary, scope physiotherapy*, orthop?edic adj (consulta* or surgeon), Refer adj3 (GP or general practi* or primary care)
Medical Subject Headings, MeSH: Referral and consultation/, Triage/, Ambulatory care/, Ambulatory care facilities/
Final draft strategy v4 incorporating expert view , scoping searches and tests for sensitivity/precision balance
Musculoskeletal Title/abstract words: Musculoskeletal, orthop?edic, hip, knee
Medical Subject Headings, MeSH: Musculoskeletal diseases/, Musculoskeletal pain/
Care pathway
Title/abstract words: Care adj2 (pathway* or plan* or protocol* or map*), Clinic* adj3 (standard* or pathway* or assessment* or map* or protocol*),(Critical or patient*) adj3 pathway*,(triage adj3 (service* or process* or system*)),(screening adj3 (service* or standard* or protocol*))
Medical Subject Headings, MeSH: Critical pathways/, Triage/
Primary to secondary
Title/abstract words: Conversion rate, primary adj2 secondary, extended scope adj3 physiotherapy*, orthop?edic adj (consulta* or surgeon), Refer adj6 (GP or general practi* or primary care or outpatient or consultation), assessment adj clinic*, triage, MCAT, MCASS, MCAS, CATS
Medical Subject Headings, MeSH: Referral and consultation/, Triage/, Ambulatory care/, Ambulatory care facilities/
AND
AND
OR
OR
OR
Search efficiency
• 256 hits in Ovid Medline• Circa 1/5 hits potentially relevant (select for full
text)• Finds >90 % relevant papers (development set)• Finds >70% papers (test set)• 50% of test set were not in Medline• Additional databases (including physio) and
extensive supplementary searching required• Then further test for sensitivity
Stages of a systematic review
Step 1: Develop a protocolStep 2: Search for the evidenceStep 3: Screen and select studiesStep 4: Assess the evidenceStep 5: Extract the dataStep 6: Synthesise the evidenceStep 7: Present findings
Step 3: Screen studies – traditional steps
Manual human process involving…• Knowledge of inclusion criteria– Identified in protocol
• Selection– Title and abstract– Full text
• Screen and select independently in duplicate– Reduction of error and bias
Step 3: Screen and select studies – new approaches
• 3.1 Boost relevant records to the top of the list…. Text mining?
• 3.2 Reduce screening time ….. Two screens?http://www.cs.manchester.ac.uk/our-research/research-groups/text-mining/
Weightman 2014…if the reviewer is willing to accept that some titles/abstracts may not be picked up but that all full text includes will be found, benefits may be huge and could reduce the screening burden by up to 70%. An enormous efficiency saving where a large number of results need to be screened.
3.1 Faster screening: EPPIReviewer…Free soon to Cochrane review authors
Rathbone 2015….Abstrackr has the potential to save time and reduce research waste. [Of four reviews one relevant citation was missed in each of two reviews]
3.1 Faster screening: Abstrackrhttp://abstrackr.cebm.brown.edu/
Wang 2014…..Using dual monitors when conducting SRs is associated with a significant reduction of time spent on data extraction (~24 minutes per article). No significant difference was observed in time spent on abstract screening or full text screening
3.2 Reduce screening time ….. Two screens?
http://northerncomputer.blogspot.co.uk/2011/11/many-people-spend-good-part-of-their.html
Summary:
• Choice of databases: Carefully chosen databases – and emphasise the additional searching techniques (reference lists, citation tracking, expert contact…)
• Text mining for searching and screening: Wide choice of free software
• Two screens: No need for screening but yes for data extraction….. I use for both (ed!)
References• Morgan HE et al. Systematic reviews of public health interventions to support
practice & policy: Where should you look? [Poster] Eight International Evidence Based Library and Information Practice Conference (EBLIP8), Brisbane Australia, July 2015
• O’Mara-Eves A et al. Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Research Synthesis Methods 2014; 5(1): 50-59
• Weightman A et al. Text mining for screening efficiency? Testing within a Cochrane public health review. [Poster] 23rd Cochrane Colloquium, Hyderabad India, September 2014
• Rathbone J et al. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews 2015; 4: 80
• Wang Z et al. Dual computer monitors to increase efficiency of conducting systematic reviews. Journal of Clinical Epidemiology 2014; 67(12): 1353-1357
