When do systematic reviews become unsystematic? Julie Williams

When Systematic Reviews become unsystematic.

Julie Williams

UNSW Library

What makes a review systematic (as opposed to unsystematic) is the use of an explicit and auditable protocol for review.

If what makes a review systematic is adherence to a protocol, what makes a review

unsystematic is simply that it does not adhere to a protocol.

Sandelowski, M. (2008). Reading, writing and systematic review. Journal of Advanced Nursing, 64(1), 104-110.

My Systematic Review Line-up.

The breast feeding practices of the mountain tribes of Laos: a systematic review.

Disciplines adopting the Systematic Review methodology.

Systematic Reviews in Academia

Is it appropriate;

a) to assign an individual the task of completing a systematic review as their first major postgraduate project?

b) to assign a systematic review project to an individual, when clearly the majority of systematic reviews are completed by teams.

c) Is Academia arming these researchers/students with enough knowledge and understanding of what an SR is and how to undertake such a large and involved project.

Cochrane Handbook guidelines on Searching

“ When designing a search strategy, in order to be comprehensive it is necessary to include a wide range of free-text terms for each of the concepts selected.”

“Use a wide-variety of search terms, combined with OR within each concept.”

Use “both free-text and subject headings”.

“Searches for systematic reviews aim to be as extensive as possible in order to ensure the as many as possible of the necessary and relevant studies are included in the review”

Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, England ; Hoboken, NJ: Wiley-Blackwell

If what makes a review systematic is adherence to a protocol, what makes a review

unsystematic is simply that it does not adhere to a protocol.


Examples of Search Strategy Reporting in a Journal Based Systematic Review (2)

Data Sources and Searches

We searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.

Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L. (2009). Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737.

http://www.annals.org/

Seven recommended elements of the search strategy description (Yoshi 2009)

Examples of Search Strategy Reporting in a Journal Based Systematic Review (1)

Data sources and searches

We conducted the review by using the following protocol. Two reviewers independently searched the MEDLINE, Web of Science, EMBASE, CINAHL, PsycINFO, and ERIC databases for studies published through September 2006 whose title, abstract, or keywords included reference to both false-positive results and screening mammography.

The search terms were (false positive OR abnormal OR benign) AND (breast cancer OR mammog*).

We also manually searched the reference sections of relevant papers and circulated requests for unpublished studies among colleagues and the authors of the articles we identified. We limited the searches to English-language studies.

Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510

Search Strategy Reporting: Same topic, different search strategies and reporting.

Psychological consequences of False-Positive Mammograms

– The Bond et al search strategy, is over 18 pages

– The Brewer et al search strategy is 2 lines.

– Bond et al search 15 databases

– Brewer search 6 databases including ERIC

– Bond et al identified 5058 articles

– Brewer identified 11,726 articles.

Mary Bond, Toby Pavey, Chris Cooper, Chris Hyde, Ruth Garside. A systematic review of the psychological consequences of false-positive screening mammograms. PROSPERO 2011:CRD42011001345

Brewer, N. T., Salz, T., & Lillie, S. E. (2007). Systematic review: The long-term effects of false-positive mammograms. Annals of Internal Medicine, 146(7), 502-510

“I don’t really know much about systematic reviews but that’s what I’ve got to do.”

If you are doing almost anything related to health care today, being “evidence based” is de rigeur.

Steinberg (2005)

Steinberg, E. P., & Luce, B. R. (2005). Evidence based? Caveat emptor! Health Affairs, 24(1), 80-92.

Nelson, H. D., Tyne, K., Naik, A., Bougatsos, C., Chan, B. K., & Humphrey, L.

(2009).

Screening for breast cancer: An update for the U.S. Preventive Services Task Force. Annals of Internal Medicine, 151(10), 727-737

Letters regarding Nelson et al Screening for breast cancer: An update for the U.S. Preventive Services Task Force 2009

– We believe that this statement mischaracterises the empirical literature (DeFrank & Brewer, 2010)

– “unfortunately the report drifts away from the published evidence” (DeFrank & Brewer, 2010)

– The USPSTF evaluation and supporting articles are plagued with ambiguity over the terms “screen” and “screening” when used alone and in conjunction with mammography.” (Dean, 2010)

– Unfortunately these erroneous data from the early overview were used for both the 2002 and 2009 USPSTF evaluations. (Dean, 2010)

– It is indefensible that Nelson and colleagues base their estimate of diagnosis on flawed studies when data from 600,000 randomly assigned women are available. (Jørgensen & Gøtzsche, 2010)

– It is curious that Nelson and colleagues do not quote our Cochrane review as they searched the Cochrane Library.(Jørgensen & Gøtzsche, 2010)

– The USPSTF study lacks evidence to support a reduction in mammography screening in African-American and Latino women” (Seewaldt, 2010)


Screening for breast cancer: An update for the U.S. Preventive Services Task Force

Data Sources and Searches

We searched the Cochrane Central Register of Controlled Trials and Cochrane Database of Systematic Reviews (through the fourth quarter of 2008) and MEDLINE (1 January 2001 to 1 December 2008) for relevant studies and meta-analyses (16). We also conducted secondary referencing by manually reviewing reference lists of key articles and searching citations by using Web of Science (17). Appendix Figure 2 (available at www.annals.org) shows our search results.


http://www.annals.org/

Roles for Librarians in Systematic Reviews

As search, database and information experts;

– Scope Searching

– Assist in the formulation of Search Strategies.

– Nominate suitable databases to search

– Endnote Training

– Provide Systematic Reviewers with an overview of currently available protocols or methodologies

– Assist in the reporting of and layout of the search strategy

– Provide a critiquing role on the reporting of the review (What have they included, what have they left out)

– Librarian involvement in training for potential reviewers.

– Being a member of the Review team

– Recommending against a Systematic Review where appropriate.

BibliographyBoell, S. K., & Cecez-Kecmanovic, D. (2010). Literature reviews and the hermeneutic circle. Australian Academic and Research Libraries, 41(2),

129-144.

Boell, S. K., & Cezec-Kecmanovic, D. (2011). Are systematic reviews better, less biased and of higher quality? Paper presented at the European Conference on Information Systems (ECIS). http://aisel.aisnet.org/ecis2011/223

Dean, P. B. (2010). Comments and response on the USPSTF recommendation on screening for breast cancer [4]. Annals of Internal Medicine, 152(8), 539.

DeFrank, J. T., & Brewer, N. T. (2010). The background review for the USPSTF recommendation on screening for breast cancer [2]. Annals of Internal Medicine, 152(8), 537-538.

Dijkers, M. P. J. M. (2009). The value of "traditional" reviews in the era of systematic reviewing. American Journal of Physical Medicine and Rehabilitation, 88(5), 423-430.

Ezzo, J., Bausell, B., Moerman, D. E., Berman, B., & Hadhazy, V. (2001). Reviewing the reviews: How strong is the evidence? How clear are the conclusions? International Journal of Technology Assessment in Health Care, 17(4), 457-466.

Higgins, J. P. T., Green, S., & Cochrane Collaboration. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, England ; Hoboken, NJ: Wiley-Blackwell.

Jørgensen, K. J., & Gøtzsche, P. C. (2010). The background review for the USPSTF recommendation on screening for breast cancer [3]. Annals of Internal Medicine, 152(8), 538.

Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33, 2004.

McMichael, C., Waters, E., & Volmink, J. (2005). Evidence-based public health: What does it offer developing countries? Journal of Public Health, 27(2), 215-221.

Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007). Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Med, 4(3), e78. doi: 10.1371/journal.pmed.0040078


Sampson, M., McGowan, J., Tetzlaff, J., Cogo, E., & Moher, D. (2008). No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology, 61(8), 748-754. doi: 10.1016/j.jclinepi.2007.10.009


Seewaldt, V. L. (2010). Comments and Response on the USPSTF Recommendation on Screening for Breast Cancer. Annals of Internal Medicine, 152(8), 541-542. doi: 10.1059/0003-4819-152-8-201004200-00205

Yoshii, A., Plaut, D. A., McGraw, K. A., Anderson, M. J., & Wellik, K. E. (2009). Analysis of the reporting of search strategies in Cochrane systematic reviews. [Article]. Journal of the Medical Library Association, 97(1), 21-29.

http://aisel.aisnet.org/ecis2011/223

Health & Medicine

When do systematic reviews become unsystematic? Julie Williams