An Introduction to Critical AppraisalIsla KuhnMedical Librarian

Last updated: September 2014

Learning Outcomes

By the end of this session you will:

• Understand what Critical Appraisal is• Be aware of some of the different types

of research• Be able to interpret basic statistics within

a research paper• Gain experience in critically appraising a

research paper

How do I Appraise?

• You don’t need to be a statistics expert

• Ready-made checklists help you focus on the most important aspects of the article

• Different checklists available for different types of research (RCTs, systematic reviews, case-control studies, etc).

• Checklist for Qualitative research

• Available free from CASP

http://www.casp-uk.net

Critical Appraisal Critical appraisal of any study design must assess:

ValidityWere sound scientific methods used? Chance / Bias / Confounding Factors

ResultsWhat are the results and how are they expressed?

RelevanceAre the findings generalisable – can they be

applied to settings / situations outside the research study? Do these results apply to my local context?

Event Rates

Number of people experiencing an event as a proportion of the number of people in the population

• Form the basis of other calculations Control Event Rate (CER) Experimental Event Rate (EER)

Emerg Med J 2008 25: 26-29:

Proportion with recurrent headache (whole sample) CER = 12/31 = 39% EER = 8/30 = 27%

Risk of benefit and harm

Relative Risk (RR) = compares the risk in 2 different groups of people

tells us how many times more likely it is that an event will occur in the treatment group relative to the control group EER / CER Relative Risk of 1 means the risk is the same in each group <1 = treatment reduces risk of event >1 = treatment increases risk of event

27/39 = 0.69 = treatment reduces risk of event

Risk of headache is 0.69 times lower in the treatment group than in the control group.

Risk continued

Absolute risk reduction (ARR)

Difference in risk between experimental and control groups

Risk of Event in Control Group – Risk of Event in intervention groupARR=0 Treatment has no effectARR positive – Treatment is beneficialARR negative – Treatment is harmful

39% - 27% = 12%Dexamethasone reduces the absolute risk of

recurrent headache by 12%

Relative Risk Reduction (RRR)

tells us the reduction in the rate of the outcome in the treatment group relative to that in the control group

ARR / CER Or 1 – RR

0.12 / 0.39 = 0.31 = 31%

1-0.69 = 0.31 = 31%

Dexamethasone reduces the risk of recurrent headache by 31% relative to that occurring in the control group.

Absolute Risk Reduction & Relative Risk Reduction

Results of hypothetical trial of a new drug for myocardial infarction

Numbers Needed to Treat

Measures the impact of a treatment or intervention

States how many patients need to be treated in order to prevent an event which would otherwise occur.

NNT = 10 means that 10 patients need to be treated to prevent one adverse outcome

The closer to 1 the better

Calculation: 1 / ARR (if ARR expressed as a proportion)100/ARR (if ARR expressed as a %)100/12 = 8

P=Probability A p-value is a measure of statistical significance which tells us the

probability of an event occurring due to chance alone

In simple terms, probability (p-value) can only take values between 0 and 1:

0|-----------------------|--------------------|1

Impossible…....... Absolutely certain…

If p=0.001 the likelihood of a result happening by chance is extremely low: 1 in 1000

If p=0.05 it is fairly unlikely that the result happened by chance 1 in 20

If p=0.5 it is fairly likely that the result happened by chance 1 in 2

If p=0.75 it is very likely that the result happened by chance 3 in 4

P Values

Confidence intervals:

“The recurrent headache rate in the control group was 39%(12/31, 95% CI 22% to 57%) compared with 27% (8/30, 95% CI13% to 46%) in the dexamethasone group (relative risk (RR)0.69, 95% CI 0.33 to 1.45; p=0.47)”.

Why 95%? It measures the reliability of an estimate, so if you repeated this same study 95 times you could be certain that the result would be the same every time, within that particular range i.e. 0.33 to 1.45. CI are typically recorded as 95% but when presented in graphical terms they are sometimes expressed as intervals of 50%, 95% and 99%

Confidence Intervals

An alternative way of assessing the effects of chanceThe result of the trial is a “point estimate” – if you ran

the trial again you will get a different resultThe Confidence Interval gives the range in which you

think the real answer liesThe 95% CI is the range in which we are 95% certain

that the true population value liesLook at how wide the interval is, and the values at

each end

E.g. RR = 0.69 95% CI 0.33 to 1.45

Forest Plot – Simple Example

Individual sample size

Combined Results

Confidence Interval

Line of No Effect

Best Estimate

The shorter the Confidence Interval (CI) the more confident we can be that the results are true

If the CI crosses the line of no effect, then the results of that study are not statistically significant

Favours Treatment Favours Control

Heterogeneity – what is it?

• Relevant to statistical meta-analysis, so you are more likely to come across this in a study review or systematic review – it is when multiple studies on an effect are actually measuring somewhat different effects due to differences in subject population, intervention, choice of analysis, experimental design, etc; this can cause problems in attempts to summarize the meaning of the studies.

What is df?

• Degrees of freedom – frequently expressed with the Chi² test.

• The number of independent pieces of information available for the statistician to make the calculations

What is Chi²?

• The chi-square test is used to determine whether there is a significant difference between the expected frequencies and the observed frequencies in one or more categories. Do the number of individuals or objects that fall in each category differ significantly from the number you would expect? Is this difference between the expected and observed due to sampling error, or is it a real difference?

How do I understand and interpret different statistical information?

• The short answer is, you don’t have to understand it, you only need to look at the p value

• As a general rule, remember the following:

• Statistics that describe data – percentages, mean, median, mode, standard deviation

• Statistics that test confidence – confidence intervals, p values

• Statistics that test difference – t tests and other parametric tests, Mann-Whitney and other non parametric tests, Chi² test

• Statistics that compare risk – risk and odds ratio, risk reduction and numbers needed to treat

Source: Medical and Health Science Statistics Made Easy by Michael Harris and Gordon Taylor

Conclusion

Critical Appraisal is part of Evidence Based Healthcare

It takes practice

Use CASP checklists

Depth of Appraisal is your choice

Only you can assess usefulness

Useful websites

www.healthknowledge.org.uk/interactive-learning/finding-and-appraising-the-evidence

www.thennt.com/

www.casp-uk.net/

www.wikipedia.org

http://www.nhs.uk/news/Pages/NewsIndex.aspx NHS Choices Behind the Headlines

Help!

Isla Kuhn

Medical Librarian

Medical Library

Box 111

Addenbrooke’s Hospital

email: ilk21@cam.ac.uk

twitter: @ilk21

phone: (01223 3) 36750

web: library.medschl.cam.ac.uk

Thank you.