Critical Appraisal: an introduction Quantitative Research

Ronan Hegarty Newcomb Library & Information Service

Lesson Aims • To introduce to attendees the principles of critical

appraisal

• To enable attendees to be able to critically appraise a piece of published research and judge its reliability

Learning Outcomes By the end of the session, participants will be

expected to be able to:

• Understand the principles of critical appraisal and its role in evidence based practice

• Assess the relevance of published research to their own work environment

• Understand statistical terms related to quantitative research

• Calculate ARR, RRR and NNT and be able to interpret confidence intervals and P values

What is critical appraisal?

• “It usually comes as a surprise to students to learn that

some (the purists would say up to 99% of) published articles belong in the bin and should not be used to inform practice” (Greenhalgh, 2006)

• Not all papers are equal: some are good, some are bad, most have strengths and weaknesses

• “... the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context” (Burl, 2003)

Strengths & Weaknesses of Critical Appraisal Strengths

• Supports decision-making by evaluating whether research is reliable or clinically relevant

• Systematic approach to judging the quality and relevance of research

Weaknesses

• Can be time-consuming, although becomes easier with practice

• May not provide an easy answer

• Can be dispiriting if it highlights a lack of good research in your interest area

What Critical Appraisal is NOT

• Tearing research apart!

• Trying to prove the authors wrong

• Assessment of the paper based only on its results

• A process that can only be undertaken by expert researchers and statisticians

What Critical Appraisal IS

• Assessment of research process and results

• Balanced assessment of benefits and strengths of the research against its flaws and weaknesses

Key questions in Critical Appraisal: the 3 Rs

• Rigour: how good is the study? i.e. was the research methodology robust?

• Results: what do the results show? And are they credible?

• Relevance: is it relevant locally? i.e. can the results be applied to your patients?

Getting started with Critical Appraisal

• Read the paper at least TWICE

• Start with easy questions

• Carry on even if you find flaws in the paper

• Remember CA is not an exact science – there are no right and wrong answers!

Components of a Research Paper (IMRAD)

• Introduction – why?

• Methods – how?

• Results and – what?

• Discussion - meaning

Steps in Critical Appraisal

• Step 1 – Identify the research method (Quantitative vs. Qualitative)

• Step 2 – Identify the methodology (cohort study, case study, grounded theory etc.)

• Step 3 – Checklist questions

Quantitative vs. Qualitative Research?

• “Both methods are valid if applied to appropriate research questions, and they should complement each other” (Bowling, 2002)

• “The view that the two approaches are mutually exclusive has itself become ‘unscientific’” (Greenhalgh, 2006)

Quantitative & Qualitative research

Quantitative • Uses numbers to describe

and analyse • Useful for finding precise

answers to defined questions

• Objective • Deductive reasoning • Statistical sampling

Qualitative • Uses words to describe and

analyse • Useful for finding detailed

information about people’s perceptions and attitudes

• Subjective • Inductive reasoning • Theoretical sampling

Quantitative / Qualitative examples*

Quantitative • How many parents would

consult their GP if their child had a mild temperature?

• What proportion of smokers have attempted to give up smoking

Qualitative • Why do parents worry so

much about their child’s temperature?

• What stops people giving up smoking

*(Greenhalgh, 2010)

Quan or Qual? • How effective is outpatient pulmonary rehabilitation in

patients with COPD? • Quantitative • Why might patients with COPD choose not to receive

outpatient pulmonary rehabilitation? • Qualitative • What is the optimum model for transitions to palliative

care? • Quantitative • How are transitions to palliative staff perceived by

hospital staff? • Qualitative

Examples of quantitative research

• Systematic reviews

• Randomised controlled trials (RCTs)

• Cohort studies

• Case control studies

• Case studies

Is the research question focussed?

PICO

• Patient (e.g. child) • Intervention (e.g. MMR vaccine) • Comparison (e.g. single vaccines) • Outcome (e.g. autism)

Are the results significant?

• How was data collected? • Which statistical analyses were used? • How precise are the results? • How are the results presented?

Blinding in RCTs

• Preventing those involved in a trial from knowing to which comparison group, i.e. experimental or control, a particular participant belongs.

• The risk of bias is minimised. • Participants, caregivers, outcome assessors and

analysts can all be blinded. • Blinding of certain groups is not always possible.

e.g. surgeons in surgical trials. • Single, double and triple blinding may be used.

Why does evidence from research fail to get into practice?

• 75% cannot understand the statistics

• 70% cannot critically appraise a research paper

• Using research for Practice: a UK experience of the barriers scale. Dunn, V. et al.

Statistical analysis Odds ratios, absolute and relative risks/benefits,

hazard ratios/relative hazards • The likelihood of something happening vs. the

likelihood of something not happening Number needed to treat (NNT) • The number of people you would need to treat to

see one additional occurrence of a specific beneficial outcome.

• The number of patients that need to be treated to prevent one bad outcome.

Statistical analysis

Intention to treat • Analysing people, at the end of the trial, in

the groups to which they were randomised, even if they did not receive the intended intervention.

2 methods of assessing the role of chance

• P-values (Hypothesis Testing)

– use statistical test to examine the ‘null’ hypothesis

– associated with “p values” - if p<0.05 then result is statistically significant

• Confidence Intervals (Estimation)

– estimates the range of values that is likely to include the true value

P Value

• P for probability (ranging from zero to one). • how likely is the result to have occurred by

chance? • P value of less than 0.05 means the likelihood of

results being due to chance is less than 1 in 20 = “Statistically significant”.

• P value of less than 0.01 means the likelihood of results being due to chance is less than 1 in 100 = “Highly statistically significant”.

P Values – just a first step

Source: http://theconversation.com/the-problem-with-p-values-how-significant-are-they-really-20029

Interpretation of Confidence Intervals • Confidence interval is the range within which

we have a measure of certainty that the true population value lies

OR

• The confidence interval around a result obtained from a study sample (point estimate) indicates the range of values within which there is a specific certainty (usually 95%) that the true population value for that result lies.

(MeReC Briefing 2005)

What can a CI tell us? Tells us whether the result is significant or not

• The width of the interval indicates precision. Wider

intervals suggest less precision • Shows whether the strength of the evidence is

strong or weak.

• The general confidence level is 95%. Therefore, the 95% CI is the range within which we are 95% certain that the true population value lies

Confidence Intervals

• Longer confidence interval = less confident of results – wider range.

• Shorter confidence interval = more confident – narrower range.

• Crosses line of no effect/no significance = Inconclusive results.

Odds Ratio Diagrams (Blobbograms or Forest Plots)

Odds Ratio Diagrams • Line of no effect – no difference between

treatment and control group • Result (blob) to the Left of the line of no

effect = Less of the outcome in the treatment group.

• Result to the Right of the line = More of the outcome.

• BUT - Is the outcome good or bad?

Cardiac deaths –

Less = good

Smoking cessation –

More = good

Type 1 error = concluding a relationship exists between two variables, when in fact there is no relationship, leading us to reject the null hypothesis when it is actually true

A study has avoided Type 1 error if P<0.05

Type I Error

Type II Error Type 2 error = concluding a relationship doesn’t exist

between two variables, when in fact there is a relationship, i.e. a high (poor) P value when the null hypothesis was correct

A study has avoided Type II error if Power>80%

Not about you recalculating statistics Not about you accessing raw research data Look for evidence in the study that potential errors have been

considered and managed Achieving the sample, good power, adequate P & CI values are

just an indication that SOME errors have been avoided. P-value cannot compensate for systematic error (bias) in a trial.

If the bias is large, the p-value is likely invalid and irrelevant.

It’s all about evidence...

Expressing comparisons of effectiveness: an example

• You have responsibility for your Trust budget and are looking at introducing a new analgesic. Which one would you fund?

• A: Reduces the rate of pain by 62.5% • B: Absolute risk reduction of pain of 50% • C: Decreases patients’ pain rate from 80% to 30% • D: For every 2 patients treated with the new drug, one

extra patient would be expect a reduction in pain

Answer: they are all the same!

• 200 patients enrolled in study • We give the new analgesic agent to 100 people

and find that 30 still have pain within two hours • We give a placebo tablet containing no active

drug to another 100 people and observe 80 still have pain

• From this we can calculate the Experiment Event Rate (EER) and the Control Event Rate (CER) –from which we can calculate ARR, RRR and NNT

EER and CER

• Experimental Event Rate (EER) is the proportion of patients in the experimental group in whom an event (pain) is observed

• EER = 30/100 = 30% (0.3) • (30% of people who received the drug still experienced

pain) • Control Event Rate (CER) is the proportion of patients in

the control group in whom an event (pain) is observed. • CER = 80/100 = 80% (0.8) • (80% of people who received a placebo still experienced

pain)

ARR & RRR • Absolute Risk Reduction (ARR) is the difference between

the Control Event Rate (CER) and the Experimental Event Rate

• ARR = CER – EER ARR = 80% –30% = 50% (can also be expressed as 0.5)

• The analgesic will prevent 50% of patients experiencing pain that would have otherwise occurred

• Relative Risk Reduction (RRR) = ARR/CER • RRR = 50/80 (62.5%) (or 0.625) • Analgesic reduces risk of pain in 62.5% of the

experimental group compared to the control group

NNT

• The number of people you would need to treat to see one additional occurrence of a specific beneficial outcome

• Number needed to treat (NNT) is 100/ARR (if ARR is a %)

• NNT = 100/50 = 2 • For every 2 patients treated with the new analgesia,

one extra patient would be expected to benefit • Best NNT=1, NNT less than or equal to 5 is good

Comparison of Drug to avoid Hospitalisation

Drug RRR Hospitalisation Rates

ARR NNT

Study 1 Fixitan 60% 2% 3% 33

Placebo 5%

Study 2 Correctapril 35% 26% 14% 7

Placebo 40%

RRR=ARR/CER ARR=CER-EER NNT=100/ARR

Useful resources • Glossary from Bandolier http://www.medicine.ox.ac.uk/bandolier/glossary.html

• Other useful documents from Bandolier http://www.medicine.ox.ac.uk/bandolier/booth/booths/ebmstor.html http://www.medicine.ox.ac.uk/bandolier/learnzone.html

• What is Critical Appraisal is part of a series http://www.whatisseries.co.uk/whatis/

• Checklists SIGN: http://www.sign.ac.uk/methodology/checklists.html CASP: http://www.caspinternational.org/?o=1012 CEBM: http://www.cebm.net/index.aspx?o=1913

Feedback

Thank you for attending Questions? Please complete an evaluation questionnaire

http://www.surveymonkey.com/s/newcomb2015-2016

For more information or assistance, please contact us at the Newcomb Library: Tel: 020 8510 7751 E-mail: newcomb.library@homerton.nhs.uk