Study Design, Levels of Evidence, and Systematic Reviews

Study Design, Levels of Evidence, and Systematic ReviewsDr. Hakan Çolak DDS, PhDIshik University School of Dentistry

Outline• Common study designs

• The rationale for different study designs

• The relative strengths and weaknesses of studies

• Schemes for assigning levels of evidence

• Narrative reviews

• Systematic reviews

• Developing a PICO question

• Appropriate study design is not only integral to answering clinical questions, but the design can also influence the level of evidence and the confidence that can be placed in the clinical outcomes.

• When assessing a study, the prudent clinician will consider – the appropriateness of the research methodology,

– the strength of the experimental design,

– the control of potential bias

• Understanding the strengths and weaknesses of a study within the larger context of the existing literature serves to inform the clinical decision-making process.

Types of inquiries Appropriate study designs

Descriptions of diseases Case studies or case series

Incidence and prevalence of diseases

Cross-sectional studies or surveys

Prognosis and progression of diseases

Case studies, case series, or cohort studies

Associations between diseases and risk factors

Case-control studies, cohort studies, and systematic reviews

Possible treatments Pilot studies

Treatment efficacy and effectiveness

Randomized controlled trials and systematic reviews

Oral health–related quality of life

Phenomenology or other qualitative studies, mixed method studies, and systematic reviews

Examples of questions and common study designs

• To understand the differences in research design, it is helpful to place studies into one of two categories:

• quantitative research or qualitative research.

• Quantitative research is by far the more dominant type of research in the dental literature.

Quantitative Research• generally looks at numbers of things (ie, quantifiable data),

• based on measurements,

• typically uses statistical analysis.

• divided into – observational studies

– interventional studies.

Observational studies• investigators gather information without intentionally

controlling or manipulating the environment of the subjects.

• These may be in the form of surveys,

• They could compare specific observed outcomes of interest among different groups of people from an historical perspective or prospectively.

Observational studies• Examples of observational studies:

– cross-sectional studies,

– case-control studies,

– cohort studies

– case reports

– case series

Cross-sectional studies• attempts to gather information about a population, or a

subset of a population, at one point in time.

• This basic design may be – a population survey,

– a clinical study of selected patients, or

– a record review assessing characteristics of a population group

Cross-sectional studies

Example of a cross-sectional study design. Exposure and disease status data are gathered at one point in time.

Cross-sectional studies• A cross-sectional study would be appropriate when asking a

question such as “What are the incidence and prevalence of periodontal disease?”

Cross-sectional studies• There are several important issues with this type of study:

– Careful selection of a representative sample from the population of interest is critical. If subjects are chosen based on the wrong criteria, the sample may not represent the population accurately.

– Variable response rates to surveys can introduce bias. People who do not respond to a questionnaire may be different from responders and may be more or less likely to exhibit the trait in question.

Cross-sectional studies• Consider a study designed to answer the following question:

“What is the association between tooth erosion occurrence and the consumption of acidic foods and drinks among undergraduate university students?”

• To answer this question, researchers recruited participants, completed dental examinations, and asked the participants questions about their diet and personal oral hygiene habits

Cross-sectional studies• strengths of this study

– direct approach

– The investigators asked a group of students about their food-consumption habits and attempted to correlate these data to the condition of the students’ teeth

Cross-sectional studies• Weaknesses

– Recruitment bias may influence the results.

– The students who were selected to participate may not be representative of the population they represent.

– The examiners may introduce bias.

– The examiners who completed the Basic Erosive Wear Examinations may have introduced other biases such as observer error or correlation of appearance to wear rather than nontraumatic morphologic variation.

Example Study• Weaknesses

– Inaccuracies in reporting may distort the exposure data.

– The students may not have correctly recalled their history, they may have misrepresented their habits to project a positive image, or they may have misunderstood the questions.

– The degree to which the participants gave reliable responses determines the validity of the study.

Cross-sectional studies• This study illustrates the survey or cross-sectional study

design.

• The success of using this design to answer the clinical question is determined by – how well it controls problems inherent to the design,

– how closely the selected participants reflect the population of interest

– how well the authors consider the response rate in developing their conclusions.

Case-control studies• includes both cases and matched controls to answer a clinical

question without an intervention

• This study design starts with a disease or other outcome in the “case” group and healthy individuals in the “control” group and then looks into the past to determine if exposure to something is associated with the disease in the individuals in the “case” group

Case-control studies

Example of a simple case-control study design. They are retrospective and look back in time

Case-control studies• This design is useful

– (1) for elucidating potential associations,

– (2) when an answer is needed quickly, or

– (3) for rare conditions or diseases.

Case-control studies• This study type is unlikely to prove cause and effect but may

suggest associations, which can be further investigated.

Case-control studies• retrospective nature

– they look back in time to establish a picture of events and behaviors possibly leading to the current health presentation

– it often relies on subjects’ memories of past events, which may introduce recall bias into the study

Cohort studies • If the goal is to eliminate or limit recall bias, there is another

study design in which a group of people are selected and observed for a period of time: the cohort study.

• Cohort study starts with groups or cohorts with a particular behavior, condition, or environmental exposure and observes them over time to determine if the variable causes or is protective for a disease

Convenientgroup of people

who agree toparticipate

Those withrisk factor

Develop condition

Do not develop condition

Those withoutrisk factor

Develop condition

Do not develop condition

Example of a simple cohort study design. They are prospective and look forward in time.

Cohort studies • the elimination of recall bias does not mean that all is well

– people are free agents and can change behavior over time, complicating exposure parameters.

– an apparently causative factor may just be associated with a hidden factor that is actually the cause

Cohort studies • imagine a study that

– concluded that coffee drinking was a risk factor for cancer but the investigators failed to account for the fact that in the study population more coffee drinkers were also smokers compared with non–coffee drinkers.

– In this example, it would appear that coffee consumption was responsible for higher cancer rates when it was actually the confounder of associated cigarette smoking that was the more likely associated risk factor

Case reports• observational studies

• do not involve sampling or comparisons.

• important for reporting isolated or rare cases,

• sometimes they suggest solutions.

Case reports• Strength

• ability to focus on specific findings and alert other clinicians to those findings. – Included in the report is a clear description of the individual case and

accompanying photographs, radiographs, and diagrams.

Case Series• reports two or more similar cases annotated in a review

format– has an advantage over a single case report.

• may involve retrospective mining of data from patient records.

• Because the selection of cases for inclusion in a case series is not necessarily systematic, bias may be easily introduced in these studies by simple inclusion or exclusion of “cherry picked” cases.

Interventional studies

Interventional studies• Each of the foregoing study designs has inherent weaknesses

associated primarily with the lack of control over important variables.

• When the investigator begins to manipulate the environment in an effort to strengthen the study and eliminate bias, the study is no longer considered a purely observational study and instead takes on the character of an experiment or clinical trial

Interventional studies• With patient volunteers assigned to one or more distinct and

different treatments, usually including a control, an intervention is introduced, and the results are observed and recorded.

• There are three general types of clinical trials: – noncontrolled trials,

– randomized controlled trials,

– controlled clinical trials.

Noncontrolled trials

• observe the effects of an intervention without a control or comparison group.

• used when it would be detrimental to the patients’ health to not treat them.

• weaknesses, – possible lack of masking and the potential lack of any comparison

group

Randomized controlled trials• To add more certainty to clinical trials and further control bias,

other measures are required, include – randomization of subjects,

– masking of subjects and clinicians,

– introduction of a control group for comparison.

Randomized controlled trials• A design where subjects are

randomly assigned to either a treatment or control group is referred to as a randomized controlled trial (RCT).

Randomized controlled trials

Randomized controlled trials• In terms of potential for avoiding bias and strength of design,

the RCT is superior to all other study designs

• It is only appropriate in select circumstances.

Controlled clinical trials • a controlled trial will be conducted without the subjects being

randomized into the control or treatment groups.

• has the advantage of having a comparison group,

• lack of randomization may introduce bias into the assignment of individuals to the control and treatment groups.

Qualitative Research• Some clinical questions do not lend themselves well to any of

the research methods discussed so far.

• These so-called qualitative research methods are aimed at discovering answers to questions that may give a better rendering of issues through narratives rather than numbers.

• The term qualitative is meant to distinguish this type of research from methodologies that measure quantities or assign rank numbers such as those designs already discussed (ie, quantitative research).

Qualitative Research• often used in the social sciences

• is important to patient advocacy.

• types of questions asked in qualitative research require careful investigation of the opinions and feelings of patients or subjects.

• It may seem obvious, but if we want to know what support services would be helpful for patients suffering from chronic disease, we might consider asking them and encouraging them to suggest solutions.

Qualitative Research• the most vulnerable patients may be unable to advocate well

for themselves, so special methods may be necessary to get to the truth. – Having a specially trained researcher interview subjects

– group discussion sessions whereby the dynamics of shared concerns guided by a facilitator may elucidate issues more clearly.

Qualitative Research

Qualitative research design example.

Reviews• Reviews may be simple narratives, systematic reviews,

qualitative reviews (ie, reviews of qualitative research), or even critical appraisals of systematic reviews

Narrative reviews• a report and serves a different purpose than other research

papers.

• often intended as an overview of a topic

• may include and reference studies on several related topics.

• Has ability to acquaint (inform and make them familiar to the topic) readers with broad aspects of a particular topic. – can serve as a great starting point for developing a background in the

area of interest.

Narrative reviews• Disadvantages

– less authoritative

– often reflect biases held by the authors.

– usually no predetermined systematic approach to the literature review,

– authors are free to include or exclude articles without justification.

– decisions are frequently based on or strongly influenced by the authors’ opinions

– usually opaque in terms of methodology and typically have no clear assessment of the studies they include

Systematic reviews

Systematic reviews• Systematic reviews use scientific

methodology to systematically identify all relevant literature around a clinical question, critically appraise the studies, evaluate the data, and develop conclusions

Systematic reviews• combines and evaluates several clinical studies that ask the

same or a similar question, employ equally strong controls, and are performed by different researchers in different places

Systematic reviews• Systematic means that an established and transparent

protocol is established for searching the world’s literature for papers addressing a specific clinical question and evaluating the papers for sound design and execution of the research protocol.

Systematic reviews• A high-quality systematic review is a reliable source of valid

information for clinical decision making

Systematic reviews

Systematic reviews

Steps in a systematic review

Steps in systematic review• The first is to develop a clinical question, frequently expressed

as a PICO question, – where P represents the population,

– I represents the intervention or exposure,

– C represents the comparison to the intervention or exposure,

– O represents the outcome to be evaluated.

Systematic reviews

PICO question.

Steps in systematic review• Second step

– determine inclusion/exclusion criteria.

– refers to specific characteristics that

– should be found in studies that are to be included in the systematic review (inclusion criteria)

– should be avoided (exclusion criteria).

Steps in systematic review• Second step

– Characteristics that may be considered are study design,

– length of the study,

– age of the subjects,

– comparison groups, language,

• Inclusion/exclusion criteria are set a priori (ie, before articles are identified) to minimize bias in the systematic review process.

Steps in systematic review• Third step

– develop a search strategy to identify all studies on the topic.

– at least two electronic databases should be searched for studies.

Steps in systematic review

Electronic databases

of the medical

literature

MEDLINE(or PubMed)

Embase(Europeanliterature)

Cochrane Central Register of Controlled

Trials (CENTRAL)

LILACS(a database of Spanish and Portuguese literature)CINAHL

(a database of nursing and allied

health journals)

Steps in systematic review• The search strategy may include

• hand-searching of key journals or references of included articles,

• searching for grey literature—– studies that are not published in the peer-reviewed literature.

– studies that were reported in conference proceedings, and dissertations

Steps in systematic review• Fourth step

– to screen titles and abstracts of studies that were identified through the literature search.

– Should be conducted separately by at least two individuals.

– Each individual searches the entire list independently to determine if studies identified through the search strategy meet the inclusion/exclusion criteria.

– After screening the titles and abstracts, the individuals compare their results.

– Any discrepancies are mediated by a third party. This is done to minimize bias in the systematic review process

Steps in systematic review• fifth step

– screen the full text of studies that were identified in the prior step to make a final determination of whether the studies meet the inclusion/exclusion criteria.

– Should done independently and by multiple individuals

– to minimize bias,

– a third party mediates any disagreements.

– The studies that pass this round of review are ultimately the ones included in the systematic review

Steps in systematic review• Sixth step

– critically appraise the studies.

– done to provide an understanding of each study, including its strengths and weaknesses.

– ideally carried out separately by two independent reviewers, with a third party mediating any discrepancies.


• Seventh step – data abstraction.

– involves identifying the key information about each study included in the systematic review, including the data on the main results.

– The data from all of the studies is typically collected in a data abstraction table.


• eighth step – data analysis.

– If the studies are very similar

– ie, similar population, intervention, and outcome measurement,

– the results can be statistically entered into a process called meta-analysis

– a quantitative systematic review is conducted.

– if the studies are so different

– results cannot be statistically combined,

– a qualitative systematic review is conducted.

– the authors summarize each study’s results and collectively use the results to develop an overall conclusion.


• Ninth step– develop a conclusion

– conclusion addresses the PICO question identified in step 1.

– It should take into account the

– quality,

– quantity, and

– consistency of the results


• Ninth step– The quality refers to the quality of studies included in the systematic

review.

– The quantity refers both to the number of studies included in the systematic review and to the number of subjects included in each study.

– In an ideal systematic review, one would hope for 10 studies with a total of 1,000 subjects combined among the studies. In reality, one or both of these goals is not met.


• Ninth step– The consistency refers to the consistency

of the results among the studies.

– do all the studies demonstrate the same trend that an intervention is effective, or

– do the studies have conflicting results where some demonstrate effectiveness and others demonstrate lack of effectiveness or even harm?

Levels of Evidence• known as the hierarchy of evidence,

• comprise a ranking system of clinical study designs

• typically depicted as a pyramid

Systematic reviews

Randomizedcontrolled trials

Cohort studies

Case-control studies

Case series/reports

Ideas, editorials, Expert opinion

Levels of evidence for preventive or therapeutic studies.

Levels of Evidence• a well-conceived, well-designed, and well-executed study in

one of the so-called lower-level study designs may be more legitimate than a mediocre systematic review or inadequately power

• One good sign of a strong study is how well the study handles potential bias.

Study design

Common bias Description

Case-control Recall bias Case-control studies draw their samples based on outcome or illness instead of exposure to potential conditions that may lead to disease. It is often necessary for both cases and controls to recall circumstances in the past that may be considered risk factors. We must

concede that individuals with a disease remember past events such as exposures to potential risk factors differently than do control subjects. This biases the study odds in

favor of the cases even though actual exposure may not have been statistically different.

Cohort Loss tofollow-up

In prospective studies, especially longer studies, it may become difficult to maintain communication with participants. When a sufficient number of participants cannot be

documented because of loss to follow-up, there may be insufficient numbers to achieve statistical significance in the study.

RCT Allocation biasand masking

If investigators know to which group a potential participant will be assigned, there is inadequate allocation concealment. The effect of allocation concealment and lack of

masking cannot be under-estimated in biasing a trial because expectations by participants or researchers commonly influence observations of results.

Reviews Publication bias Studies may be unavailable or unpublished, especially if they show negative results. This leads to a systematic alteration in studies represented in the literature. If this is of

sufficient magnitude, the combined results of the accumulated data will show an artificial level of significance. Conclusions drawn from the included studies would be unwarranted

given the missing studies’ potential influence on the review and meta-analysis.

Some examples of bias* from different study types

Further reading• Evidence-Based Dentistry for the Dental Hygienist by

by Julie Frantsve-Hawley

http://www.amazon.com/s/ref=dp_byline_sr_book_1?ie=UTF8&field-author=Julie+Frantsve-Hawley&search-alias=books&text=Julie+Frantsve-Hawley&sort=relevancerank

Education

Study Design, Levels of Evidence, and Systematic Reviews