Systematic Reviews of Surgical Interventions

Surg Clin N Am 86 (2006) 101–114

Systematic Reviews of SurgicalInterventions

Martin Burton, MA, DMa,*, Mike Clarke, MA, DPhilb

aDepartment of Otolaryngology-Head and Neck Surgery, University of Oxford,

The Radcliffe Infirmary, Oxford OX2 6HE, England, UKbUnited Kingdom Cochrane Centre, Middle Way, Summertown,

Oxford OX2 7LG, England, UK

All physicians are familiar with the type of general review articles foundin many medical journals. Systematic reviews are different. They applya strict, scientific methodology to the reviewing process to produce a reviewthat is comprehensive, reliable, and as free from bias as possible. As a result,systematic reviews occupy the highest position in the ‘‘levels of evidence’’ ta-bles associated with the practice of evidence-based health care.

Systematic reviews are not limited to reviews of randomized trials of theeffects of treatments, but can and do exist for other types of study also. Sys-tematic reviews can be done for topics such as causes of disease, prognosisand prognostic factors, diagnostic test accuracy, and genetics. The appropri-ate study design to include in a review depends on the question to be an-swered; however, the well-done systematic review should always be at thetop of the hierarchy, because it brings together all the relevant researchand does not selectively focus only on research with a particular result. Inthis article, the authors discuss how systematic reviews can help to answerquestions about the relative effects of treatments. As such, focus is on re-views of randomized trials and the contribution that these make to evidencebased health care.

Evidence-based medicine (EBM) has been defined as the conscientious,explicit, and judicious use of current best evidence in making decisionsabout the care of individual patients [1]. Although this term is widelyused, it is also widely misused, and poorly understood. For the present,the key phrase is ‘‘current best evidence.’’ In recent years there has beenan explosion in the quantity of published medical information. The amount

* Corresponding author.

E-mail address: [email protected] (M. Burton).

0039-6109/06/$ - see front matter � 2006 Elsevier Inc. All rights reserved.

doi:10.1016/j.suc.2005.10.005 surgical.theclinics.com

mailto:[email protected]

102 BURTON & CLARKE

of information available in journals, books, magazines, and the media ingeneral was overwhelming a decade ago. The arrival of the Internet, withthe enormous quantity of information it now contains and the ease withwhich vast amounts of material can be found, has added considerably tothis burden. Some of the information available is of high quality and is re-liable enough for decision making; unfortunately, much of what is availableis of poor quality.

As a consequence, keeping up to datedonce possible simply by readingthe national surgical journals on a regular basis and attending a few meet-ingsdis now much more difficult. The conscientious surgical practitionerhas to be able to search out the evidence relating to his or her clinical query,appraise its quality, and then synthesize the results. The skills needed to dothis successfully need to be learned and practiced, and this training processshould be undertaken with the same diligence and perseverance applied tothe acquisition of those manual skills required of the competent surgeon.

Fortunately, for many busy practitioners, an increasing number of sys-tematic reviews now exist in which other researchers have already donemuch of this work. Later in this article, the authors discuss The CochraneCollaboration and the contribution it and its members have made to helpingpeople find, understand, and use reliable evidence quickly. Even if one findsa systematic review done by someone else, however, it still needs to be ap-praised to assess its quality and its relevance to the question at hand.Thus all practitioners, not just those who want to do a review, need to befamiliar with the process and the steps needed to reduce bias. This articlesets out this process.

Systematic reviews aim to locate, appraise, and synthesize evidence fromscientific studies. To ensure that they are comprehensive, are least likely tobe prone to bias, and to ensure their reliability, they adhere to a strict scien-tific design.

Developing a focused clinical question

At the outset, the reviewer formulates a clearly focused clinical question.This may be a question about therapy, diagnosis, prevention, or harm. Theexample given here is a question relating to a therapeutic intervention, butthe reviewing process is similar when applied to other types of questions.Whatever its nature, the question must focus on the decisions faced by pa-tients and practitioners.

An example might be: ‘‘Do perioperative antibiotics help prevent postop-erative wound infections in patients undergoing appendicectomy?’’

This has the standard three-part structure often found in well thought-out clinical questions. That is: (1) how effective is A, (2) for the manage-ment of B, (3) in patients who have C? It has a meaningful outcome (theprevention of wound infection) that is relevant to both the patient anddoctor, and it is clearly focused. A less well-focused question might

103SYSTEMATIC REVIEWS OF SURGICAL INTERVENTIONS

be: ‘‘Do antibiotics help prevent postoperative infections of any sort in sur-gical patients?’’dmuch too broad. At the other extreme, questions that aretoo narrow in focus may turn out to be impossible (and irrelevant) to an-swer: ‘‘Does penicillin help prevent postoperative wound infections in youngchildren who have complicated appendicitis?’’ The key components to thefocused question in this example are

� The types of patients here, all patients, both children and adults, under-going appendicectomy.

� The types of interventions and comparators here we are looking at anyantibiotic regime versus no antibiotic, probably a placebo. An alterna-tive review might look at direct comparisons of different antibioticregimes.

� The types of outcomes we have specified wound infections, but willhave to decide how specific we want to be about the diagnosis of theseinfections. For example, we need to decide if we are going to look forstudies that insist on a microbiological diagnosis as opposed to a clinicalone.

We then also need to consider the types of study design we are going tolook for. Because randomized controlled trials (RCTs) are the least biaseddesign for assessing the relative effects of interventions, many systematic re-views of treatment determine at the outset that they will only include RCTs.Some reviews, however, do include other types of study, but they need to becautious about the potential for bias in these studiesdbiases that might leadto misleading conclusions about the true differences between the interven-tions in the review.

The quality criteria that are going to be applied to the RCTs that arefound must also be considered. Just because an RCT is published in a repu-table medical journal, it does not necessarily mean that the trial will be ofhigh quality. Before undertaking a systematic review, it is necessary to statewhich features of a trial one believes are associated with high-quality studies(and therefore increased reliability and validity of the results). There is goodevidence available on which features to investigate to allow this to be done.

Identifying relevant studies

Another important part of the process of undertaking a systematic reviewis trying to find all those RCTs that address the question of interest. Ideallyone would find all those trials that have been published and all those unpub-lished. This is important because publication bias is well-recognizedda trialis more likely to get published if it has a positive result. There is thereforea danger that bringing together the results only of published trials willskew the results in favor of demonstrating a positive treatment effect,when in reality, had all trials been combined, no effect would have beendemonstrated.

104 BURTON & CLARKE

A second important bias to be avoided is language bias; there is no gooda priori reason why a study published in a non-English language journalshould be less relevant or of poorer quality than one published in a leadingUnited Kingdom or North American journal. Clearly there may be difficul-ties in finding non-English language studies (of which more below), not tomention difficulty in translation. But these are not insurmountable, and in-ternational organizations that produce and publish high quality systematicreviews, such as The Cochrane Collaboration, are able to use the multina-tional, multilingual membership base as a tool for identifying and translat-ing such studies.

Several methods should be used to identify the appropriate RCTs. Sensi-tive search strategies for the retrieval of RCTs have been developed for usewith MEDLINE, EMBASE, and other electronic databases. The CochraneCentral Register of Controlled Trials (CENTRAL), published in the Co-chrane Library, contains the product of such searches as well as the exten-sive searching of journals and conference proceedings that is ongoing withinThe Cochrane Collaboration [2]. In addition, CENTRAL includes studiesthat have been published in languages other than English, as well as recordsfor ongoing and unpublished studies. As a result, CENTRAL is one of thebest sources for reports of RCTs, and is easily searchable.

When searching any database, be it CENTRAL or the electronic biblio-graphic databases dedicated to the health care literature in general, it is im-portant that the correct search terms are used to identify trials addressingthe topic in question. It is helpful to elicit the assistance of an informationspecialist, because this process is not always as intuitive and simple as manyphysicians might believe at first. For example, searching using the term ‘‘ap-pendectomy’’ will miss reports using the British spelling ‘‘appendicectomy,’’and those that simply mentioned the ‘‘removal of the appendix.’’

As well as searching CENTRAL and the other databases, the search forstudies should include examination of the reference lists within the identifiedstudies. One can also consider searching journals, because the indexing pro-cess used by MEDLINE and the like has not always identified RCTs in theolder paper-based publications, and many journals are not indexed in data-bases that are readily accessible. As mentioned above, a coordinated ap-proach to the searching of journals is being organized by The CochraneCollaboration to avoid duplication of effort. Details of the journals thathave been and are being hand-searched are available from The CochraneCollaboration Web site (www.cochrane.org), and reports of RCTs identifiedare included in CENTRAL. Reviewers should also check conference pro-ceedings and abstracts that might be particularly relevant. Many studies (es-pecially those with negative results) are presented at meetings but get nofurther [3]. If abstracts reporting results of these studies can be identified,the authors can be contacted and further details of the trial obtained. It isalso important to consider writing directly to experts in the chosen fieldto see if they are aware of any studies that have not been identified.

http://www.cochrane.org


Sifting and appraising initial search results

The result of a comprehensive search is likely to be a set of titles and ab-stracts, some, but not all, of which will address the topic under evaluation.Usually, reading the titles and abstracts will allow some studies to be dis-carded as inappropriate for further evaluation, leaving a smaller set to beanalyzed further. This first step is best done by more than one person [4].After it has been completed, it becomes necessary to retrieve the originalpublications and begin the process of critical appraisal.

There are two steps in the process of critically appraising the individualstudies, which can be considered as questions.

Is the study applicable to this systematic review?

Does it evaluate the intervention in question in the types of patient inquestion? Are the outcomes evaluated in the study those of interest? It isimportant to note at the outset of the systematic reviewing process thatreviewers should state which outcome measures are of primary andsecondary interest. Usually these will be things of direct relevance to pa-tients and practitioners. Although it is always possible that the reviewerwill overlook an important outcome, this is unlikely if sufficient care and at-tention has been paid at the beginning. It is more likely that a study will beidentified that looks at an outcome not deemed important by the reviewers.In the example above, an RCT of perioperative antibiotics in patients under-going appendicectomy might have used as its only outcome measure bloodlevels of an inflammatory mediator. In this type of case, the degree to whichthe measured variable can be considered an adequate surrogate measure forthe outcome in question is important. In this example, this study wouldprobably be excluded from the systematic review on the grounds that theoutcome measure used was not appropriate; however, it might be worthcontacting the authors to see if they did collect any outcomes that mightbe of relevance to the review.

At the end of this process, another subset of studies will have been ex-cluded from the review.

What is the validity of the individual study?

In essence, is this a good or a bad study? Has the study been designed andconducted in such a way that bias has been reduced to the minimum? Spe-cific biases include selection bias, performance bias, attrition bias, and detec-tion bias. Good RCTs are designed to avoid these.

Briefly, ‘‘selection bias’’ occurs if there are systematic differences betweenthe groups within a trial that might affect the prognosis or responsiveness totreatment of the patients in one of the groups. This is best prevented by ran-domization of large enough numbers of patients to each group, because thiswill both reduce the biases and the effects of chance. The allocation of

106 BURTON & CLARKE

patients to one group or the other should be concealed from the participantsin the trial and the providers of care. This ensures that a preference for oneof the treatments will not bias the entry into that group. It is the best way tominimize differences in confounding variables between the groups, and is thereason why reviews of the effects of interventions need to focus on random-ized trials.

A decision to enter the patient into the trial should be made before theprocess of randomization, and those entering the patient should be unawareof the group to which the patient will be allocated. Good methods of ran-domization include remote randomization by telephone, or the use of se-quentially numbered, sealed, opaque envelopes. The use of alternation,hospital numbers, birth date, day of entry into trial, and the like are not sat-isfactory methods because they are not always random and are open to ma-nipulation. The sequence of patients can be changed to subvert thealternation process, and knowing which treatment a patient will receivemight affect the decision on whether or not he takes part in the study.

‘‘Performance bias’’ refers to the presence of differences in the care pro-vided to the groups of patient other than the intervention being evaluated.This can be prevented by blinding care providers and participants, and stan-dardizing the care protocol. In the review in our example, this might beachieved through the use of a placebo.

‘‘Attrition bias’’ refers to any systematic differences between the groupsin the pattern of withdrawal from the study (for example, dropouts becauseof side-effects, participants who failed to comply with their treatment, orthose who crossed over into the other arm of a trial). In the latter case, insurgical trials, this may include patients randomized to the nonsurgicalarm who actually receive surgery. The most appropriate measure to over-come this problem is the use of a so-called ‘‘intention-to-treat’’ analysis,in which all patients randomized are analyzed within the group to whichthey were allocated.

Finally, if there are systematic differences in how the outcomes are as-sessed between the groups, ‘‘detection bias’’ can occur. This is minimizedby blinding the participants and outcome assessors. Again, in the reviewin the example above, the use of a placebo might help to achieve this.

Clearly, in surgical trials, it is much more difficult to avoid some of thesebiases. The cry is often heard ‘‘It’s impossible to blind in surgical trials, be-cause the surgeon always knows what operation she is doing.’’ The truth ofthe second half of this sentence is self-evident, but the first statement is nottrue at all. The patient, the practitioner enrolling the patient in the trial,those providing postoperative care, and certainly those assessing the out-comes can, in many circumstances, be blinded as to the specific interventionin a study. This is especially the case when surgical procedure A is comparedwith procedure B. Even in some trials of surgical intervention versus nonsur-gical treatment, those assessing outcome can be blinded as to whether or nota participant received the intervention.


Summarizing and using assessments of validity

Having answered these questions, there are two ways in which the valid-ity of a study can be used by the reviewer (who will have stated a prioriwhich method will be used). The first method is to only include paperswith low risk of bias in the review. This is one option, and might be appro-priate when a large number of trials have been identified. Only those trialsdesigned and conducted to the very highest standards are included in the fi-nal analysis.

The second method is to summarize the risks of bias using a simple grad-ing scale. For example, consider the specific biases and relate these to thedesign and conduct of the individual studies, and then categorize the studiesusing this scale

Grade A ¼ low risk of biasGrade B ¼ medium riskGrade C ¼ high risk of bias.

Having done this, there are several options available for incorporatingthe grades into the reviewing process.

The grading system can be used to include or exclude studies from thereview. For example, include A and B, exclude C. Alternatively, all gradesof studies can be included in the review, and the grading may allow the re-viewer to explain differences in the results when studies are compared.

More formally, the grades can be used as the basis for a sensitivity anal-ysis. For example, imagine that the results of nine studies (three grade A,three grade B and three grade C) had been combined, and that the resultsshowed that the treatment in question was effective. A sensitivity analysiswould then assess how dependent this result was on the inclusion of thepoorer quality studies. If the six studies graded B & C were excluded butthe result remained the same, one could be confident about its veracity. Ifthis exclusion resulted in a change, howeverdif the treatment was not effec-tive when one looked simply at the grade A studiesdthis would be signifi-cant. The demonstration of effectiveness of the treatment being criticallydependant on the inclusion of poorer quality studies clearly calls into ques-tion to certainty of that result.

The grades can also be used as ‘‘weights’’ in statistical analysis. Many,but not all, systematic reviews include a meta-analysis. The weight givento a particular study in that analysis usually depends on the number of par-ticipants in the trial; however, an alternative or supplementary approach isto use the grade of the study as one of the weighting factors.

Dealing with the included studies

Having been through the process of identifying and appraising individualstudies, the reviewer will now know how many could be incorporated into

108 BURTON & CLARKE

the systematic review, and must decide whether or not the results of thestudies can be mathematically combined. Thus far there has been little men-tion of statistics and meta-analysis. It is axiomatic that statistical methodsfor combining the results of studies provide a powerful tool for deriving po-tentially important and useful conclusions from data. Equally, they providea means by which errors of interpretation can be made. Having said that, itmay be neither possible nor sensible to combine data sets from individualstudies. Each study may have measured outcomes in such a different waythat the results cannot be combined. For example, two studies examiningthe effectiveness of a perioperative intervention on postoperative pain con-trol may have measured postoperative pain in different ways. One may usepain scores based on visual analog scales, and the other may use the needfor supplemental postoperative analgesia. Both are equally valid ways tomeasure pain, but the specific results cannot easily be mathematicallycombined.

Sometimes the outcome measures used by the studies are different, butanother dichotomous variable can be calculated from these data. For exam-ple, if a scoring system has been used to measure patients’ symptoms beforeand after an intervention or placebo, it may be possible to compute the‘‘proportion of patients improved’’ in each case. Even if a number of studieshave used different scoring systems, this proportion may be calculable foreach study, and as a result, the study results can be combined.

There are several benefits of doing a meta-analysis if it is proper so to do.First, to obtain a more precise estimate of the treatment effect and to bemore certain about the size of the effect. Second, there is more statisticalpower to detect small effects that may be clinically significant. Finally,meta-analysis allows evaluation of the generalizability of the results. It can-not be emphasized enough, however, that to do a meta-analysis when thereare no relevant, valid data, or when it does not make sense, is inappropriate.Furthermore, one of the important decisions that a reviewer has to make iswhether the trials are not so dissimilar that an average of their results wouldbe meaningless. This is done by considering whether there is excessive het-erogeneity in the design of the studies (including the interventions and par-ticipants studied and the outcomes measured), and statistical tests are alsoavailable to assess whether the results of a series of trials might differfrom each other by more than chance [5].

Statistical methods

There are a variety of statistical variables and methods used in system-atic reviews and meta-analyses, and these will not be discussed furtherhere; however, the fundamental principle is that participants in one trialare never directly compared with those from another trial. A statistical re-sult is calculated for each trial independently, and then these statistics arecombined [6].


The Cochrane Collaboration

As we have discussed already, systematic reviews are vital to the interpre-tation of research evidence, to its placing in context, and to its use in evi-dence-based decision-making. These reviews need to be kept up-to-datebecause the research base on which they are built rarely stands still. The Co-chrane Collaboration is the largest organization in the world engaged in theproduction and maintenance of systematic reviews. It has received world-wide support in its efforts to make systematic reviews accessible to peoplemaking decisions about health care. Cochrane reviews bring together therelevant research findings on a particular topic, synthesize this evidence,and then present it in a standard, structured way. One of their most impor-tant attributes is that they are periodically updated to take account of newstudies and other new information, to help people be confident that the sys-tematic reviews are sufficiently current to be useful in making decisionsabout health care.

TheCochraneCollaborationwas established in 1993, founded on ideas andideals that stem from earlier times. In October 1992, Iain Chalmers, KayDickersin, andThomasChalmers began an editorial in theBMJ [7]withwordsfrom the British epidemiologist, Archie Cochrane, published in 1972:

It is surely a great criticism of our profession that we have not organized

a critical summary, by specialty or subspecialty, updated periodically, ofall relevant randomized controlled trial [8].

This editorial was published at the time of the opening of the first Co-chrane Center in Oxford, England. A year after this Centre opened, the firstCochrane Colloquium was held, bringing together 77 people from 19 coun-tries, and they established The Cochrane Collaboration as an internationalorganization. There are now eleven further Cochrane Centers, in Australia,Brazil, Canada, China, Denmark, Germany, Holland, Italy, South Africa,Spain, and the United States; with branches of these centers in several othercountries.

There are currently more than 13,000 people contributing to the work ofthe Cochrane Collaboration from almost 100 countries, and this involve-ment continues to grow. The number of people involved has increased by10% to 20% year on year for each of the 6 years to 2005.

The Cochrane Collaboration has ten guiding principles:

1. Collaboration, by internally and externally fostering good communica-tions, open decision-making, and teamwork

2. Building on the enthusiasm of individuals, by involving and supportingpeople of different skills and backgrounds

3. Avoiding duplication, by good management and coordination, to max-imize economy of effort

4. Minimizing bias, through a variety of approaches such as scientificrigor, ensuring broad participation, and avoiding conflicts of interest

110 BURTON & CLARKE

5. Keeping up to date, by a commitment to ensure that Cochrane Re-views are maintained through identification and incorporation ofnew evidence

6. Striving for relevance, by promoting the assessment of health care in-terventions using outcomes that matter to people making choices inhealth care

7. Promoting access, by wide dissemination of the outputs of The Co-chrane Collaboration, taking advantage of strategic alliances, and bypromoting appropriate prices, content, and media to meet the needsof users worldwide

8. Ensuring quality by being open and responsive to criticism, applyingadvances in methodology, and developing systems for qualityimprovement

9. Continuity, by ensuring that responsibility for reviews, editorial pro-cesses, and key functions is maintained and renewed

10. Enabling wide participation in the work of The Cochrane Collabora-tion by reducing barriers to contributing and by encouraging diversity.

Preparation, maintenance, and accessibility of Cochrane reviews

The work of preparing and maintaining Cochrane reviews is done by theauthors of Cochrane reviews working with one of 50 Cochrane Collabora-tive review groups, which collectively provide a home for reviews in all as-pects of health care interventions. These groups are responsible forparticular areas of health care, usually based around a particular conditionor state of health. For example, there is a Breast Cancer Group with an ed-itorial base in Sydney, Australia and a Pregnancy and Childbirth Groupbased in Liverpool, England. There are also groups in particular specialtiessuch as ear, nose and throat (Oxford, United Kingdom) and anesthesia (Co-penhagen, Denmark). The Cochrane Collaborative review groups organizethe refereeing of the drafts for protocols for Cochrane reviews (which setout how the review will be done), and for the reviews themselves. The edi-torial teams in these groups have the ultimate decision on whether or nota Cochrane review should be published. But, unlike editors in more tradi-tional health care journals, their role is, in part, to help the authors ofCochrane reviews ensure that their review becomes good enough to bepublished; the decision that a Cochrane review will be published dependson its quality, not its findings. The main motivation for most authors of Co-chrane reviews is a desire to answer reliably a question about the relative ef-fects of interventions for people who have particular conditionsdvery fewof them receive any direct payment for this work.

The Collaborative review groups are based around the world, and somehave editorial bases in more than one country. There are also Cochranemethods groups, with expertise in relevant areas of methodology; fields,or networks, with broad areas of interest and expertise spanning the scope


of many review groups; and a consumer network helping to promote the in-terests of users of health care. The Cochrane Collaboration Steering Group,containing elected members from these different types of entity and Co-chrane centers, is responsible for setting collaboration-wide policy and, byworking with the entities, the implementation of the Collaboration’s strate-gic plan.

As noted above, The Cochrane Collaboration grew rapidly through itsfirst decade in terms of the number of people involved. Its output has alsogrown quickly. Cochrane reviews are published in full in The Cochrane Da-tabase of Systematic Reviews (CDSR), and the first issue of this in early1995 contained 36 Cochrane reviews. A decade later there were nearly2500, with published protocols for 1600 more. Various milestones havebeen passed along the way. There were 500 Cochrane reviews in 1999,and the one thousandth appeared in 2001, with 2000 published by April2004. Hundreds of newly completed reviews and protocols are added eachyear, and a few hundred existing reviews are updated so substantivelythat they can be considered to be the equivalent of new reviews.

The Cochrane Database of Systematic Reviews is available on the Inter-net and on CD-ROM as part of The Cochrane Library. This is published byJohnWiley and Sons Ltd. and is available on a subscription basis. The estab-lishment of national contracts means that The Cochrane Library is currentlyfree at the point of use to everyone in the United Kingdom and Ireland, aswell as in Australia, Denmark, Finland, Norway, and South Africa.

The output of The Cochrane Collaboration also includes the CochraneCentral Register of Controlled Trials (CENTRAL), the Cochrane Databaseof Methodology Reviews, and the Cochrane Methodology Register. All ofwhich are unique resources. In 1993, when the Collaboration was estab-lished, fewer than 20,000 reports of randomized trials could be found easilyin MEDLINE, and one of the main tasks facing the Collaboration was theneed to identify and make accessible information on reports of trials thatmight be suitable for inclusion in Cochrane reviews. It has done this throughextensive programs of the hand searching of journals (in which a journal ischecked from cover to cover to look for relevant reports) and of electronicsearching of bibliographic databases such as MEDLINE and EMBASE.Suitable records are then added to CENTRAL, with coordination by theUS Cochrane Centre in Rhode Island [2]. By 2004, CENTRAL containedrecords for more than 400,000 reports of randomized (or possibly random-ized) trials, many of which are not included in any other electronic database.The Cochrane Database of Methodology Reviews contains the full text forCochrane methodology reviews, which are systematic reviews of issues rele-vant to the conduct of reviews of health care interventions or evaluations ofhealth care more generally. In 2005, there are 11 full Cochrane methodologyreviews and published protocols for 9 more. The Cochrane MethodologyRegister, to a large extent, provides the raw material for the Cochrane meth-odology reviews. It contains more than 7000 records relating to the

112 BURTON & CLARKE

methodology of systematic reviews and other types of evaluation of healthcare, including many records relevant to trials and other evaluations ofsurgery.

Over the next few years, The Cochrane Collaboration will strive to ensurethat its work is sustainable. Even with more than 4000 Cochrane reviews al-ready underway, and results available from 2000 of these, there is still a largeamount of work to be done. It has been estimated that approximately 10,000systematic reviews are needed to cover all health care interventions that havealready been investigated in controlled trials [9], and such reviews wouldneed to be assessed and, if necessary, updated at the rate of 5000 peryear. If the growth in The Cochrane Collaboration continues at the paceof the last few years, this target will be reached within the coming 10 years;however, this will require continuing and evolving partnership and collabo-ration. The Cochrane Collaboration will need to continue to attract andsupport the wide variety of people who contribute to its work. It will alsoneed to work together with funders and with providers of health care to en-sure that the resources needed for the work grow, and that the output of thework is accessible to people making decisions about health care around theworld [10].

Systematic reviews of surgical interventions: challenges and examples

The Cochrane Database of Systematic Reviews now contains many ex-amples of systematic reviews of surgical interventions or of issues relevantto surgery. Such reviews might not be as straightforward as reviews of a ho-mogenous series of randomized trials containing similar participants, allo-cated to receive a drug at a particular dose versus a matching placebo,with the effects assessed using an unequivocal endpoint such as death; how-ever, reviews that are this straightforward are rare.

Systematic reviews of surgical interventions do present particular chal-lenges, just as randomized trials of these interventions can be challenging.Whenever a person is a key component of the intervention, as is the casewith surgical skill, there may be a learning curve; or the surgeon might bemuch more proficient at one of the interventions. Trial designs, such as ran-domizing patients to the operator rather than the operation, have been sug-gested and would be amenable to systematic reviews [11]. The difficulties of‘‘blinding’’ were mentioned above, but as discussed, mechanisms can be putin place to ensure that the person assessing outcomes does not know whatintervention a patient was allocated to. Identical wound dressings can justas easily cover a short incision as the smaller hole used for a laparascopicincision in gallbladder removal.

And even if the randomized trials are not thought to exist for some im-portant surgical questions, a systematic review will still prove beneficial byconfirming this lack of evidence, drawing attention to the gaps, and high-lighting how the necessary research might be done. All new research should


be preceded by a systematic review to ensure that the necessary research hasnot already been done and to facilitate the design of the most appropriateand feasible trial [12].

Just to provide some flavor of systematic reviews of surgical interventionsthat now exist, the authors had a browse in The Cochrane Database of Sys-tematic Reviews. It was not very difficult to find that reviews have been doneof preoperative management (bowel preparation before colorectal surgery[13]), different surgical techniques (for treating distal radial fractures [14]),surgery versus other invasive procedures (neurosurgical clipping versus en-dovascular coiling for subarachnoid hemorrhage [15]); and postoperativewound management (tap water versus other ways to cleanse wounds [16]).

Systematic reviews relevant to surgery are no less relevant than sys-tematic reviews in other areas of health care. They should be a prerequisiteof any new research, a key component in decision making, and an opportu-nity for all surgical practitioners to get involved in the conduct and interpre-tation of research.

References

[1] Sackett DL, Rosenberg WMC, Gray JAM, et al. Evidence-based medicine: what it is and

what it isn’t. BMJ 1996;312:71–2.

[2] Dickersin K,Manheimer E,Wieland S, et al. Development of the Cochrane Collaboration’s

CENTRAL register of controlled clinical trials. Eval Health Prof 2002;25:38–64.

[3] Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in ab-

stracts. The CochraneDatabaseMethodology Reviews 2005;2. Art. No.:MR000005. pub. 2.

[4] Edwards P, Clarke M, DiGuiseppi C, et al. Identification of randomized controlled trials in

systematic reviews: accuracy and reliability of screening records. StatMed 2002;21:1635–40.

[5] Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses.

BMJ 2003;327:557–60.

[6] Deeks JJ, Higgins JPT, Altman DG, editors. Analysing and presenting results. In: Higgins

JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions 4.2.4 [up-

dated March 2005]; Section 8. In: The Cochrane Library, Issue 2, 2005. Chichester (UK):

John Wiley and Sons, Ltd; 2005.

[7] Chalmers I, Dickersin K, Chalmers TC. Getting to grips with Archie Cochrane’s agenda.

BMJ 1992;305:786–8.

[8] Cochrane AL. 1931–1971: a critical review, with particular reference to the medical profes-

sion. In: Medicines for the year 2000. London: Office of Health Economics; 1979. p. 1–11.

[9] Mallett S, Clarke M. How many Cochrane reviews are needed to cover existing evidence on

the effects of healthcare interventions? Evid Based Med 2003;8:100–1.

[10] Chinnock P, Siegfried N, Clarke M. Is evidence-based medicine relevant to the developing

world? Systematic reviews have yet to achieve their potential as a resource for practitioners

in developing countries. PLoS Med 2005;2:367–9.

[11] Devereaux PJ, BhandariM, ClarkeM, et al. Need for expertise based randomised controlled

trials. BMJ 2005;330:88–91.

[12] Clarke M. Doing new research? Don’t forget the old: nobody should do a trial without re-

viewing what is known. PLoS Med 2004;1:100–2.

[13] Guenaga KF, Matos D, Castro AA, et al. Mechanical bowel preparation for elective colo-

rectal surgery. Cochrane Database Syst Rev 2005;1:CD001544.

[14] Handoll HHG, Madhok R. Surgical interventions for treating distal radial fractures in

adults. Cochrane Database Syst Rev 2003;3:CD003209.

114 BURTON & CLARKE

[15] Algra A, Brilstra EH, Clarke M, et al. Endovascular coiling versus neurosurgical clipping

for patients with aneurysmal subarachnoid haemorrhage. Cochrane Database Syst Rev

2005;3:CD003085.

[16] FernandezR,GriffithsR,Ussia C.Water for wound cleansing. CochraneDatabase Syst Rev

2002;4:CD003861.

Documents

Systematic Reviews of Surgical Interventions