How to Interpret Data: Critical appraisal

Colette SmithUCL Research Department Infection and Population Health

JUSTRI Skills Tool Kit Training12th December 2015

Putting study results into context

• Single studies cannot be interpreted in isolation – we also need to evaluate existing literature relating to the question of interest

• Basing judgment about a particular treatment (or ‘risk factor’) only on published material may lead to a biased picture because of publication bias

• Initiatives to compile information on all trials undertaken include:

• Cochrane collaboration - registers of trials and meta-analyses (www.cochrane.org)

2

Why do we need to appraise research?

• Research informs clinical care and public health policy.

• Evidence based medicine - ”Integrating individual clinical experience with best available evidence from systematic research” (Sackett).

• Vast numbers of research studies are carried out, presented and published each year.

• In order to understand whether we should apply the findings from a research study to clinical practice we need to assess its validity, quality and applicability.

3

Why do we need to appraise research? 1

• Deliberate research fraud is unlikely but research may have:

• methodological problems or limitations

• inappropriate analysis or poor presentation of results

• insufficient explanation of methods/results

• misleading interpretation or conclusions

• problems with generalisability

4

Critical appraisal

• Important to consider and be aware of all potential biases, confounders and role of chance when interpreting results of studies

• However, it is easy to find fault with all medical research studies

• No study is ever perfect – all have strengths and weaknesses

• The important thing to consider is whether the study is sufficiently well performed that you believe the results

5

Hierarchy of evidence

[Systematic review of RCTs] Randomised controlled trials (RCTs) Non-randomised intervention studies Cohort studies Case-control studies Cross-sectional studies Time series/ecological studies Expert opinion, consensus conference

6

Strongest evidence

Weakest evidence

Components of RCT research appraisal

1. What question is the research asking?

2. Are the methods valid and appropriate?

3. What are the results?

4. How should the results be interpreted?

5. How does this study fit in with existing research?

6. What are the implications of the findings for clinical practice?

7

Components of observational appraisal 1

• Although general appraisal issues same as for clinical trial, some additional appraisal issues

• Bias due to lack of randomisation:• Differences between treatment/risk factor groups may be

explained by many different factors• Have known or possible confounding factors been considered? • Are there any relevant results from RCTs?

8

Components of observational appraisal 2

• Bias due to incomplete follow-up • Observational data may be particularly prone to loss to follow-up.

• What proportion of subjects are lost to follow-up overall and in each group of interest?

• Lack of single pre-specified comparison• Often observational databases are used to test many different

research questions using a variety of different approaches• Was the primary question of interest and analysis method clearly

defined, prior to analysis?• Have all relevant analyses been reported? • Have multiple statistical tests been performed?

9

Tools for evaluating studies

• There are a number of guidelines/checklists for reporting and appraising different types of study designs:• RCTs: Consolidated Standards of reporting trials (CONSORT)

www.consort-statement.org

• Observational studies: Strengthening the reportiong of observational studies in epidemiology (STROBE) www.strobe-statement.org

• Systematic reviews: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) www.prisma-statement.org

• All: Enhancing the QUAlity and Transparency Of health Research (EQUATOR) www.equator-network.org

10

Public health priorities

• Ideally based on ‘evidence’ - meta-analyses and systematic reviews

• Considerations of efficiency, cost-effectiveness and harm• e.g. eradication of poverty for improving health?

• Must weigh up competing interests:• Evidence – Values – Resources – Perceptions • Different groups will have different priorities:• Public• Industry• Government• Scientific community

11

Summary

• Ideally much of clinical practice and health policy are based on research findings - therefore research appraisal is crucial.

• In appraising the results of clinical trials and observational studies we should consider the appropriateness of the methods and the interpretation and implications of the results.

• It is also important to put results from new studies into the context of existing research findings.

12

Socio-economic factors and late diagnosis of HIV in 2001-2015 in the ABC Hospital

Colette Smith

on behalf of ABC HIV Hospital Cohort

Disclaimer: All data are hypothetical, and do not necessarily represent real life!

13

Introduction

• Late diagnosis of HIV in the UK continues to be a major problem, with 42% diagnosed with CD4 count <350 cells/mm3 in 20131

• A timely diagnosis is important for slowing the progression of HIV36 and therefore improving prognosis37-39, reducing onwards transmission40 and reducing costs to health services41

• Little is known about the association between late diagnosis and socio-economic factors such as housing, education and employment

14

1 Yin. Public health England. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/401662/2014_PHE_HIV_annual_report_draft_Final_07-01-2015.pdf 36 Delpech. HIV med. 2013;14(S3):19-24; 37 Nakagawa. AIDS. 2012;26(3):335-43; 38 May. BMJ. 2011;343; 39 ARTCC Collaboration. Lancet. 2008;372(9635):293-9; 40 Marks. AIDS. 2006;20(10):1447-50; 41 Mukolo. AIDS and Behavior. 2013;17(1):5-30;

Methods – study design

• Patient registration forms have been used at ABC Hospital, London, UK in April 2001 which collected data on: – Presentation– HIV risk behaviour– Medical history– Demographic and socio-economic background

• 2003 met inclusion criteria: i) Attended hospital for first visit April 2001 - May 2015 ii) Diagnosed with HIV within 1 year of this first visitiii) Had a first visit form completed (22% of all new diagnoses)

• Late diagnosis - CD4<350 cells/mm3 within 12 months of diagnosis

15

Methods - explanatory variables

• Three measures of socio-economic status collected as self-reported on patient registration form:– Housing status: Home owner vs. Non-home owner– Current employment status: Employed vs. Not employed– Highest level of education: University or higher vs. Less than university

• Prevalence of late diagnosis assessed according to each socio-economic and demographic factor

• Associations between socio-economic factors and late diagnosis were assessed by logistic regression, adjusted for:– Gender/ sexual orientation (MSM; MSW; Women)– Age (continuous)

16

0

10

20

30

40

50

60

70

Perc

ent d

iagn

osed

late

(CD4

<350

)

MSM MSW WSM <30 30-40 40-50 >50 Gender/sexual orientation Age at diagnosis

Percent diagnosed late by demographic factor

• 902/2003 (45%) were diagnosed with CD4<350 cells/mm3 (29% with CD4<200 cells/mm3)

17

P = 0.0002

~ test for trend

P = 0.0513

0

10

20

30

40

50

60

Perc

ent d

iagn

osed

late

(CD4

<350

)

Yes No Yes No Yes No Home owner University education Currently employed

Percent diagnosed late by socio-economic factor

18

P = 0.041

~ test for trend

P = 0.104 P = 0.786

Demographic factors associated with late diagnosis

Unadjusted OR (95% CI)

P Adjusted* OR (95% CI)

P

Gender/ sexual orientation

MSM 1 0.0002 1 0.234MSW 3.67 (1.73, 7.79) 3.38 (0.57, 10.23) Women 3.16 (1.63, 6.15) 3.11 (0.65, 8.09)

Age <30 years 1 0.041 1 0.04930-40 years 1.18 (0.47, 2.95) 0.91 (0.35, 2.39)40-50 years 2.17 (0.89, 5.27) 1.52 (0.60, 3.87)

>50 years 2.47 (1.02, 6.72) 1.76 (0.97, 3.19) Home owner Yes 1 0.063 1 0.73 No 3.26 (0.15, 7.83) 1.00 (0.21, 4.75) Education University 1 0.104 1 0.44 Non-university 1.77 (0.88, 3.59) 1.70 (0.56, 2.94) Employment Employed 1 0.786 1 0.80

Unemployed 1.08 (0.50, 2.33) 0.90 (0.40, 2.04)

19

Women who live in east London aged 31 – 57 years

20

Adjusted* OR (95% CI)

P

Home owner Yes 1 <0.0001 No 9.73 (3.64, 27.3) Education University 1 0.0006 Non-university 5.06 (2.43, 12.11) Employment Employed 1 0.80

Unemployed 1.02 (0.19, 5.04)

Conclusion

• Low socio-economic status has no impact on late diagnosis

• Age is an important predictor of late diagnosis whereas gender/sexual orientation group is not

• It is particularly important to monitor women who live in east London who are aged 31 to 57 years, as this is a high-risk group

21