Introducing systematic reviews

Introducing systematic reviewsKarin HannesCentre for Methodology of Educational Research

OutlineIntro - daily practice: evidence-based?Systematic reviewsWhat is it and why do we need them?How do they look like?Who is producing them?Where can we locate them?What is there (potential) impact?How can we contribute to them?Daily practice: Evidence-Based?1861: I firmy believe that if the whole material medica as used now, could be sunk to the bottom of the sea, it would be all the better for mankind and the worse for the fishes. (Holmes)1979: Only 10 to 20 percent of all procedures currently used in medical practice have been shown to be efficacious by controlled trial... (The US congressional office of technology assessment)

How many of our daily educational, criminological, social welfare interventions actually ARE evidence-based?Lets Raise our Hand for

3Daily practice: Evidence-Based?

80 to 90 % based on-Magic?-Intution?-Expert opinion-???

Daily practice: Evidence-Based?EBP is the integration of the best available research with clinical expertise in the context of patient characteristics, culture and preferences.American Psychological Association. (2006). APA presidential task force on evidence based practice. Washington, DC: Author.

We all want the best for our target groupWe have always been evidence-basedOur academic education guarantees a sufficient level of scientific knowledge, skills and attitudes

Half of what you learn in school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half! (Dr. Sydney Burnwell) Daily practice: What is evidence?Evidence of effectiveness: the extent to which an intervention, when used appropriately, achieves the intended effect.Evidence of feasibility: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context.Evidence of appropriateness the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given.Evidence of meaningfulness: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.6Part 1: Systematic reviewWhat is it and Why do we need THEM?

What is it?

A systematic review is a systematic identification, evaluation and synthesis of all relevant studies regarding a specific topic, based on an explicit and pre-defined methodology

If high quality research studies exist... Summarize them!This idea led Ian Chalmers to set up the Cochrane Collaboration, first initiated in Britten, but now widely spread all over the world. CC isThe very mission of the CC has been copied by a bunch of America researchers in the Campbell Collaboration, named after Donald Campbell who was a strong advocate of experimental research in education and other disciplines.Campbell focusses on domains such as.Most of you probably know what a systematic review is I have higlighted three words in this definition because I feel that they distinguish a systematic review from a literature or a narrative review that all of us probably have used in the context of thesisses in the past.

What they aim to do isNot only help decision makers to cope with the sheer volume of literature by summarising itProvide a scientific rather than subjective summarisation of literatureCan reveal new evidenceMore reliable evidence with which to aid decision making

Some of these reviews have really had a strong impact on policy or practice decision making, to give you but one famous example from the Campbell Collaboration.

The review from Petrosini evaluating the impact of scaired straight programs on children at risk for future criminal behaviour, where they are taken on a tour of adult prison facilities.It revealed that not only did these programs fail to deter crime, they actually led to more offending behaviour and actually caused harm to the children that were exposed to this intervention. It led to an increase in criminal behaviour, basically because those kids thought is was pretty cool the stories from those inmates.

8What is it?Predefined format:To help review authors be systematicTo help people reading the reviews to find information quickly

TransparancyReliabilityMEDLINE: Meta-analysis (Publication Type) OR systematic review

Explosive growth.

Why do we need them?Over two million articles are published annually in the biomedical literature in over 20,000 journals literally a small mountain of information a stack would rise 500 metres Cynthia Mulrow, in Systematic Reviews (BMJ Publishing Group, 1995)And within that stack, we would be looking for a needleReduce bias (garbage in-garbage out)Reduce random errorProvide reliable basis for making decisionsInform and influence future researchAnd indeed, there is really too much evidence, sometimes even conflicting evidence, which means that if you need an answer to a particular question you might as well be looking for a needle in a haystack really.Thats why Archie Cochrane came up with the famous quote that it is indeed a pitty that we have captured all the available information in summaries that provide us with a nice overview of evidence.11Traditional versus systematic reviewLack rigorMethodology not transparentDifferent reviewers reach different conclusionsBecome out of dateScientific rigor to minimise biasExplicit and reproducible methodologyRegularly updated (Cochrane/Campbell)Key characteristics of a SRA clearly stated set of objectives with pre-defined eligibility criteria for studies An explicit, reproducible methodology

Key characteristics of a SRA clearly stated set of objectives with pre-defined eligibility criteria for studies An explicit, reproducible methodology A well defined, systematic search that attempts to identify all studies that would meet the eligibility criteria

Key characteristics of a SRA clearly stated set of objectives with pre-defined eligibility criteria for studies An explicit, reproducible methodology A systematic search that attempts to identify all studies that would meet the eligibility criteria An assessment of the methodological quality of the findings of the included studies (assessment of risk of bias)

Key characteristics of a SRA clearly stated set of objectives with pre-defined eligibility criteria for studies An explicit, reproducible methodology A well defined, systematic search that attempts to identify all studies that would meet the eligibility criteria An assessment of the methodological quality of the findings of the included studies (assessment of risk of bias )A systematic extraction, synthesis, and presentation of the characteristics and findings of the included studies

Why do we need them?Answers a specific healthcare questionAims to identify and include all relevant trialsUses pre-planned methods and strategies to limit bias and random errorMay include META-ANALYSIS: statistical synthesis of the results of included studiesHelps practitioners and patients make decisions about appropriate health care.Provide a scientific rather than subjective summarisation of literatureCan reveal new evidenceHelp decision makers to cope with the sheer volume of literature by summarising itMore reliable evidence with which to aid decision making

A SYSTEMATIC REVIEW IS ONE THAT AIMS TO IDENTIFY AND INCLUDE ALL THE EVIDENCE FROM RELEVANT TRIALS.

IT ANSWERS A SPECIFIC QUESTIONS, FOR EXAMPLE ARE ANTIBIOTICS EFFECTIVE IN THE TREATMENT OF SORE THROATS IN CHILDREN?

ANSWERING THIS QUESTION IS LIKE PERFORMING A SCIENTIFIC INVESTIGATION, -PRE-PLANNED METHODS AND STRATEGIES TO LIMIT BIAS - CALLED SECONDARY RESEARCH AS IT IS BASED ON THE RESULTS OF OTHER PRIMARY STUDIES, THE RANDOMISED CONTROLLED TRIALS.

IF POSSIBLE, THE RESULTS OF THE ALL THE RCTS IN THE SYSTEMATIC REVIEW CAN BE COMBINED IN A STATISTICAL PROCESS CALLED META-ANALYSIS. THIS GIVES YOU OUTPUT LIKE THE COCHRANE LOGO, WHICH IS EXTREMELY USEFUL, BUT NOT AN ESSENTIAL REQUIREMENT OF A SYSTEMATIC REVIEW.

Part 2: systematic reviewshow do they look like?

What is a meta-analysis?Optional part of a systematic reviewSystematic reviewsMeta-analysesTo identify a common effect among a set of studiesTo improve precision of an estimateTo investigate whether the effect is constantTo answer controversies arising from conflicting studies or to generate new hypothesesMETA ANALYSIS MAY BE PART OF A SYSTEMATIC REVIEW.

ASK FOR E.G.S F WHEN ITS NOT APPROPRIATE TO COMBINE STUDIES IN META-ANALYSIS.

SYSTEMATIC REVIEWS MAY INCLUDED META-ANALYSES BUT META-ANALYSIS MAY BE DONE WITH OUT SYSTEMATICALLY REVIEWING THE STUDIES THERE ARE EGS OF THIS IN JOURNALS THESE THEREFORE MAY BE BIASED

IN THE US THE TERMS ARE USED INTERCHANGABLY, BUT NOT THE CASE IN THE UKMeta-analysisGives a more precise estimate of effect than when derived from the individual studies included within a review InterventionComparisonno effectSkills training versus usual curricula in the prevention of drug use in school kids.1.Summary statistic + CI for each study individually (RR, OR, RD)2.Pooled intervention effect + CI is calculated as a weighted averageMeta-analysisFacilitates investigations of the consistency of evidence across studies, and the exploration of differences across studies. InterventionStandardno effectSkills training versus usual curricula in the prevention of drug use in school kids.CC & C2: meta-analyse

Skill-based interventions prove to be an effective strategy to prevent from marihuana use in schoolkids.Part 3: systematic reviewsWho is producing them?

Combinations of data might be better than attempts to choose amongst themAstronomy, 17th century.Karl Pearson, 1904.Many of the groups are far too small to allow of any definite opinion being formed at all, having regard to the probable error involved

Glass, 1976.Primary, secondary, and meta-analysis of research24Statistical basis of meta-analysis dates back to the 17th century when in astronomy intuition and experience suggested that combinations of data might be better than attempts to choose amongst them....noticed that observations of the same object differed even when made by the same observers under similar conditions. The calculation of the mean as a more precise value than a single measurement had appeared by the end of the 17th century.Archie Cochranes challenge

It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials. (1979)Cochrane Collaboration: 1993

an international not-for-profit organisation which aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventionsCochrane CollaborationCollaborative Review GroupsFieldsThe Consumer NetworkCentresSteering GroupMethods GroupsSteering Group: Policy making body of the Cochrane Collaboration.Elected representatives of each of the Collaboration entities

Fields: Represent a population, group, or type of care that overlaps multiple Review Group area.Examples: Primary Health Care, Health Care of Older Adults, Complementary MedicineMethods Groups: Develop methods and products integral to internal functioning of the CollaborationDevelop state of the art methods for systematic reviewExamples: Statistical, Economics, Placebo Effects, Informatics, QualitativeCentres: Help organise and register review groupsFacilitate collaboration among reviewersProvide training and consultationEstablish liaisonsPromote the Cochrane CollaborationProvide unique contributionReview Groups: Focus on Treatment of disease or health problemsTo prepare and maintain systematic reviews To develop a specialised trials registerConsumer Network:Provides consumer inputHelps set prioritiesHelps with dissemination

Campbell Collaboration (C2): 2000

an international organisation which aims to help people make well-informed decisions about public policy (crime & justice, education, social welfare) by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of social and behavioural interventionsCampbell CollaborationCoordinating Groups Users GroupPartnersSteering GroupMethods GroupsUsers group: Knowledge translation to increase the impact of Campbell reviews in policy and practice arenas, and to make the information more accessibleMethods Groups:Improving the methodology of research synthesis, and disseminating guidelines for state-of-the-art review methodsCoordinating Groups:Provide editorial services and support to authors of Campbell reviews and build links with users of systematic reviews:

Crime and JusticeSocial WelfareEducation

Steering group: strategic and policy making bodyPartners: Institutes who have a formal agreement with Campbell and support the same goals.

CC & C2 Collaboration goalsTo ensure high quality, up-to-date systematic reviews are available across a broad range of topicsTo promote access to systematic reviewsTo develop an efficient, transparent organisational structure and management system for the CollaborationTo achieve sustainability of the Collaboration

PrinciplesCollaborationBuilding on the enthusiasm of individualsAvoiding duplicationMinimising biasKeeping up to dateStriving for relevancePromoting accessEnsuring qualityContinuityPart 4: systematic reviews: where can we locate them?

The Cochrane LibraryDatabase of Systematic ReviewDatabase of Abstracts of Reviews of EffectivenessControlled Trials RegisterDatabase of Methodology ReviewsMethodology RegisterHealth Technology Assessment DatabaseNHS Economic Evaluation Database

CDSR: Database of completed CC reviews as well as protocols of on-going reviews. Reviews regularly updated and amended. New material is flagged. Each review has the same structure. This is a D/B you would search for the highest level of evidence based information on the effectiveness of interventions. Has the added advantage of being full text and therefore a one stop resource. DARE: Systematic reviews from elsewhere in the literature. Structured abstracts with qualitative assessments and provisional abstracts. Covers areas not on the Cochrane Database of Systematic Reviews. Use if you cant find what you want on CDSR.CCTR: Randomised controlled trials. Many sources, including hand-searching and pharmaceutical trials. Second level evidence for decision making. Basis of systematic reviews, therefore good place to start.CDMR: A full text D/B of systematic reviews on the methodology of research published by the C C. Full text. First level information about research methods.Introduced in 2001.CMR: Information on how to conduct reviews and trials. Bibliographic references. Of use to researchers.HTA: Reports of health technology assessments. That is, how well does a drug, device or procedure work? Ongoing projects. Material from here will also be assessed for inclusion in DARE so there may be overlap.NEED: Economic evaluations of healthcare interventions. Critical abstracts.

Still a lot of work to do!

Part 5: systematic reviewsWhat is their (potential) impact?Evaluate a solutionSituation: 1 mj. people die each year due to traffic accidents

Solution: Educate the drivers

Systematic Review: 24 studies on education, no evidence that it reduces traffic accidents

Recommendations: Stimulate teleconferences and the use of safe, public transport

Driver education is big business our results show that it is also a big con (Prof. Roberts)

Identify gaps in evidence

Situation: anti-psychotics are subscribed for pregnant women with a psychotic disease

Systematic Review: No literature on adverse effects of anti-psychotics on the mother or the development of the (unborn) babyRCTs with pregnant women are considered unethicalMedicals referals are based on habits and opinions

Recommendation: Research on the effects of anti-psychotics in pregnant women is necessary

The continued use of antipsychotic drugs in women during pregnancy and lactation without sound evidence raises serious clinical and ethical concerns. (Webb)

Identify solutions that cause harmSituation: Illinois law: Chicago Public Schools mandated to identify children at-risk for future criminal behaviour.

Solution: Scaired straigth programs: take them on tours of adult prison facilities

Systematic Review: Not only does a scared straight program fail to deter crime, it actually leads to more offending behaviour.

Governments need to adopt rigorous evaluation to ensure that they are not causing more harm to the citizens they intend to protect (Mr. Petrosino).

Authors of SR can make a difference!

Anyone who wants to make a contribution...

www.cochranecollaboration.org

www.campbellcollaboration.org

Part 6: systematic reviewshow can we contribute to them?Select a topicMake a review teamRegister your title

1. Select a topic

Systematic reviews:CommissionedInvited Unsolicited Motivation to undertake a review:Resolve conflicting evidenceAddress questions when clinical practice is uncertainExplore variations in practiceConfirm appropriateness of current practiceHighlight need for future research45Commissioned: undertaken on behalf of a client under the terms and conditions of a formal contact. Will include agreed timetable, milestone products and final delivery date.Invitation: solicitation of reviews from experts on selected topics (members of Crime & Justice Coordinating Committee and advisory group identify topics for possible reviews)Unsolicited: researchers develop an idea for a review and submit it to the relevant C2 Coordinating Committee for reaction.Learn to ask questions(Treatment or Intervention):

PpopulationIinterventionCcomparisonOoutcome1. Select a topicDoes providing information (I) at school have a preventive effect on the use of drugs (O) in high school students (P) compared to sanctions when someone is caught using drugs (C) ?

1. Select a topicInformation campagnes in the prevention of drug useDoes information provision at school decreases the risk of drug use in high school students (compared to the standard approach)? Scope of the question?BROAD (LUMP)NARROW (SPLIT)AdvantagesComprehensiveGeneralizibilityEffect modifiers (why different effect between apples and oranges?)Manageable for review teamEasy to readDisadvantagesMore resourcesmixing apples and orangesEvidence sparse?Lack of generalizibilityChoice of scope biased by authors47Determining the scope of a review question is a decision dependent upon multiple factors including perspectives regarding a questions relevance and potential impact; supporting theoretical, biologic and epidemiological information; the potential generalizability and validity of answers to the questions; and available resources. 1. Select a topicMay start with a broad scope, and be divided up into narrower reviews as evidence accumulates

Cochrane Overviews of reviews (umbrella reviews)Refinement of the review questionBias (data-driven questions)Should be documented (Differences between protocol and review)Sensitivity analysis to assess impact of changes48The advent of Cochrane overview if reviews may affect scoping decisions for reviews.2. The review teamWhy more than 1 person?Detection of errors (selection of eligible studies, data extraction)Reduces risk of biasWhen more than one person / team is interested in topicWho should be in the team?Expertise in topic areaExpertise in systematic review methodologyIncorporate view of users (clinicians, consumers)Address questions that are important to peopleTake account of outcomes that are important to those affectedMake it accessible to people making decisionsReflects variability in populations, settings3. Registering a titleContact a Cochrane Review Group (CRG) or Campbell Coordinating Group (CCG)Cochrane Review Group:> 52Specific condition (eg incontinence, learning & developmental disorder group)Group of diseases (eg ear nose and throat disorders group)Organ (eg renal group, heart group)Patient (eg neonatal group, pregnancy and childbirth group, pain and palliative care)

Campbell Coordinating Group:3Crime & Justice CGEducation CGSocial welfare CGThere is a potential to co-register reviews!3. Registering a titleFill in Title Registration FormReview by CRG/CCG editorsTopic fits in with CRG/CCG?Questions specific and answerable?Overlap with ongoing review?Review team has necessary expertise?YESYESNOYES

Good luck with making the protocol!Register title with CRGWrite reviewTitle registeredNoContact other CRGContact other review teamWrite protocolProtocol publishedPeer reviewReview publishedUpdate reviewPeer reviewPeer reviewUpdate of review publishedSystematic Review SummaryEstablish objectives, selection criteriaSearch for eligible studiesApply selection criteriaAssess study quality, extract dataAnalyse results using meta-analysis, if appropriate and possiblePerform sensitivity analysesPrepare report (keep it up-to-date!)CLEARLY FORMULATED QUESTION RE ANTIBIOTIC EXAMPLE WOULD NEED TO DEFINE AN AGE RANGE AND MAYBE SPECIFY TRIAL USING ONLY ANTIBIOTIC VS. PLACEBO OR PROTOCOL

SEARCH COMPARE WITH SELECTION CRITERIA E.G. ONE TRIAL INCLUDES CHIDREN >12 VS.>16

STUDY QUALITY SEVERAL SCORING SYSTEMS EXTRACT DATA IN DUPLICATE

CANT ASLWAYS META-ANALYSE - TRIALS MAY HAVE USED DIFFERENT OUTCOMES E.G. NO. OF DAYS THROAT PAIN PERSISTED DURATION OF FEVERSENSITIVITY ANALYSIS DATA REANALYSED TIN DIFFERENT WAYS TO DETERMINE THE EFFECT ON RESULTS

Check the resource centre at the Campbell Website for specific guidance from methods groups and coordinating groups

http://www.campbellcollaboration.org/resources/research.php