Critical appraisal

Paula Funnellp.a.funnell@qmul.ac.uk

Faculty Liaison Librarian (Medicine and Dentistry)

Introduction to

Outline· Introduction to critical appraisal· Appraising quantitative research

Levels of evidence Statistics Group appraisal

· Appraising qualitative research Qualitative methodologies Group appraisal

· Running a critical appraisal workshop

Introduction

Evidence based healthcare

“When we intervene in the lives of others we should do so on the basis of the best evidence available regarding the likely consequences of that intervention”

G Macdonald, 1998

Macdonald, G. (1998). Promoting Evidence-Based Practice in Child Protection. Clinical Child Psychology and Psychiatry, 3 (1), 71–85.

Steps of evidence based healthcare

1. The patient presents with a clinical problem

2. Formulate a focused research question

3. Search for the best evidence

4. Critically appraise the evidence you find

5. Consider the evidence in the light of your expertise and decide whether to apply it or not

Why critically appraise?

To weigh up how valid and useful

the research will be

Why critically appraise?

• In order to keep up to date, clinicians would have to read 17 articles a day, 365 days a year

• Research is of variable quality• Only an estimated 1% is

judged clinically relevant• Need to find the 1%

How it works

• Involves answering a short questionnaire

• We use the CASP questionnaires at http://www.casp-uk.net/

• The questionnaires were devised by clinicians for clinicians

• Remember to focus on appraisal not criticism

Quantitative research

What is quantitative research?

• Research on something that can be accurately measured

• Tends to use large, statistically representative samples

• Uses statistical methods to analyse data

• Represents findings with numbers• Attempts to eliminate bias

Systematic reviews

• A review of all the literature on a topic• Systematically identified• Appraised• Summarised

Question 1

Did the review ask a clearly focused question?

PICO

A focused clinical question:

P

I

C

O

Population / patient / problem

Intervention

Comparison

Outcome

Question 2

Did the authors look for the right type of papers?

Levels of evidence (therapy)

Systematic reviews

RCTs

Cohort studies

Case control studies

Expert opinion

1a

1b

2

3

4

5

Case series

http://www.cebm.net/index.aspx?o=1025

Levels of evidence (therapy)

Randomised controlled trials

Outcome

Outcome

new treatment

control treatment

Question 3

Do you think the important relevant studies were included?

Publication bias

Papers with “interesting” results are more likely to be:

• Submitted and accepted for publication• Published in a major journal • Published in English• Quoted by authors• Quoted in newspapers

Question 4

Did the review’s authors do enough to assess the quality of

the included studies?

RCT checklist

Quality assessment

• Were the inclusion/exclusion criteria clear?

• Did they use any sort of scoring system?

• Were the studies for inclusion assessed by more than one reviewer?

Question 5

If the results of the review have been combined, was it reasonable

to do so?

Heterogeneity• Are the studies similar in terms of:

– Population– Intervention– Outcome– Methodology

• Are the results similar from study to study?

• Were any tests for heterogeneity carried out?

• Are there any discussions around heterogeneity?

Question 6

What are the overall results of the review?

How are the results presented?

• Number needed to treat (NNT)• Odds Ratio• Relative risk• Mean difference

Odds and risk

Odds of winning

You versus the rest

Risk of winning

You versus all the runners

10 horses running, you bet on 1 horse

1:9 1:10

Odds and risk

Odds of winning

You versus the rest

Risk of winning

You versus all the runners

10 horses running, you bet on 4 horses

4:6 4:10

Forest plots

more than 1less than 1 1

Forest plots

more than 1less than 1 1

Line of no effect

Forest plots

more than 1less than 1 1

Line of no effect

Forest plots

more than 1less than 1 1

Line of no effect

Best estimate

Forest plots

more than 1less than 1 1

Line of no effect

Best estimate

Confidence interval

Forest plots

more than 1less than 1 1

Line of no effect

Best estimate

Confidence interval

Forest plots

more than 1less than 1 1

Line of no effect

Best estimate

Pooled result

Confidence interval

Question 7

How precise are the results?

Confidence intervals

Practical exercise

P-value

Could the result have occurred by chance?

p = 0.001 (1 in 1000)

p = 0.2 (1 in 5)

A p-value of less than 0.05 (1 in 20) is considered to be statistically significant

Question 8

Can the results be applied to the local population?

Application

• Difficult to answer if you don’t have your own population

• Need to rely on experts• Think about whether the study could be

easy replicated with another population

Question 9

Were all the important outcomes considered?

Outcomes

• Do the authors address all the outcomes they set out to study?

• Are the outcomes considered from different viewpoints e.g. patient, practitioners, policy makers?

Question 10

Are the benefits worth the harms and the costs?

Benefit v harms and costs

• How significant are the benefits?

• Is there any discussion of possible side effects etc.?

• Are financial costs mentioned?

Summary

Validity

Is it trustworthy?

Results

What does it say?

Relevance

Will it help?

Qualitative research

What is qualitative research?

• Concerned with the world of human experience from the view of the participants

• Natural settings are used• A study is ‘led’ by the subjects’

experiences• Qualitative research is rigorous and

systematic• Results are not usually generalisable or

totally replicable

Components of qualitative research

Research question

Sampling process

Data collection

Data analysis

Question 1

Was there a clear statement of the aims of the research?

The research question

• Describes why the research is being carried out

• Qualitative research addresses the

questions• Are the aims of the research clear?

Question 2

Is a qualitative methodology appropriate?

Why qualitative research?

• Increases knowledge in an area that is poorly understood

• Challenges assumptions and practices• Acts as a precursor to quantitative

research• Generates new ideas

Question 3

Was the research design appropriate to address the aims of

the research?

Research methodologies

phenomenology

field research

groundedtheory

action research

ethnography

Question 4

Was the recruitment strategy appropriate to the aims of the

research?

Sampling• Sample size can be determined by

– Data analysis– Time constraints

• What population does the sample refer to?

• How was the sample selected?

Sampling

Methods of sampling:• Convenience• Purposive • Quota• Snowballing

Question 5

Was the data collected in a way that addressed the research

issue?

Data collection

• Are the methods the most suitable for the research question?

• Are the methods explicit?• Is the venue clearly described?

Question 6

Has the relationship between researcher and participants been

adequately considered?

Reflexivity and researcher bias

VenueArea being studied

Types of interview questions asked

Meaning given to data

Question 7

Have ethical issues been taken into consideration?

Ethics

Need to consider:• Consent• Confidentiality• Professional responsibility• Reporting• Ethics committee approval

Question 8

Was the data analysis sufficiently rigorous?

Data analysis

• Description of analysis• Clarity of approach• Use of all the data• Potential for bias

Question 9

Is there a clear statement of findings?

The findings

• Are the findings explicit?• Are they credible?

– respondent validation– quality of argument– would another researcher make a similar

interpretation?– are alternative interpretations explored?

Triangulation in

terv

iew

s

observationsquestionnaire

patients

professionals

care

rs

Question 10

How valuable is the research?

Value of the research

Consider your appraisal of the paper in terms of:

• Credibility• Dependability• Transferability• Confirmability

How useful is the paper to you?

Questions to ask

Running a critical appraisal workshop

Before the session

• Who is the group?• Is it qualitative, quantitative, or both?• Know what equipment and facilities are

available• Ask participants to read the article in

advance

Structuring the session

• Dependent on the group and time frame• Presentation• Encourage audience participation• Use interactive activities • Practical critical appraisal in groups• Feedback

Choosing an article• For a mixed group choose a fairly generic

article• For a discipline specific group:

– Do you want to use a discipline specific article?– Ask for topic ideas– Not a good idea to let them choose the articles

• Choose studies with both strengths and weaknesses

• Make sure it’s easily available to all the participants

Any Questions?

The end!

Paula FunnellFaculty Liaison Librarian (Medicine & Dentistry)

Queen Mary University of Londonp.a.funnell@qmul.ac.uk