Getting and making sense of “the best available” scientific evidence

Postgraduate Course

Getting and making sense of “the best available” scientific evidence

EBMgtHelping Managers Make Better

Decisions

Postgraduate Course

Evidence is not the same as ‘proof’ or ‘hard facts’

Evidence can be

- so strong that no one doubts its correctness, or

- so weak that it is hardly convincing at all

What is evidence?

Postgraduate Course

Evidence of effect (do!)

Evidence of no effect (don’t!)

No evidence of effect (research!)

Don’t confuse

Postgraduate Course

Assignment

CAT

Critical Appraised Topic

Postgraduate Course

CAT: Critical Appraised Topic

A critically appraised topic (or CAT) is a structured, short (2

pages max) summary of evidence on a topic of interest,

usually focused around a practical problem or question. A CAT

is like a “quick and dirty” version of a systematic review,

summarizing the best available research evidence on a topic.

Usually more than one study is included in a CAT.

Postgraduate Course

CAT: structure

1) Question2) PICOC3) Background4) Search strategy5) Results6) Conclusion7) Comments8) Recommendation9) References

Postgraduate Course

Asking the right questions

Part 2

Postgraduate Course

5-step approach

Gathering Best Scientific Evidence isa 5-step approach

1. Formulate an answerable question (PICOC)

2. Search for the best available evidence3. Critical appraise the quality of the found

evidence4. Integrate the evidence with managerial

expertise and organizational concerns and apply

5. Monitor and evaluate the results

Postgraduate Course

Asking the right question?

Does team-building work?

Does leadership development training work?

Does management development improve the

performance of managers?

Does employee participation prevent

resistance to change?

Is 360 degree feedback effective?

Postgraduate Course

P = Problem or population

I = Intervention or successfactor

C = Comparison

O = Outcome

C = Context

Answerable question: PICOC

Postgraduate Course

Scenario: You are a consultant, your client is an insurance company, there are plans for a merger, you have heard that the other company

has a different culture, you want to know if this will effect the outcome

P = Organizations with a different corporate cultureI = MergerC = Organizations with a similar corporate cultureO = Long term profitability(C)) = Profit organizations, competitive market)

Answerable question: PICO(C)

Postgraduate Course

Searching evidence

The problem with finding evidence:

the abundance of literature

Postgraduate Course

Searching evidence

Article 1

Article 2

Article 3

Article 4

Article 5

Article 6

Article 7

Article 8

There are about 1350 articles published on HRM every year. For an HR

professional to keep up this means reading 3 to 4 articles every day!

(most of these publications are not valid or irrelevant)

Postgraduate Course

Searching evidence

Evidence-based searching

In a systematic and transparant way searching for the “best” evidence Part of EBMgt where decision maker is not a ‘subject matter expert’

Postgraduate Course

Searching for scientific evidence

What kind of evidence are we looking for?

1. Studies with a design that best suits the research question

2. Studies with the highest level of evidence

Postgraduate Course

Explanation

Which design for which question?

Postgraduate Course

Levels of internal validity

Postgraduate Course

Searching evidence

Where do we search?

Postgraduate Course

Peer reviewed journals

Postgraduate Course

Which journals are peer reviewed?

Postgraduate Course

Type of Information Source

Current Information Wall street Journal, Financial Times, Business week, Financieel Dagblad

Overview of a subject

General background

Academic Information

Statistical Information

Textbooks and popular books

Encyclopedias, yearbooks & book reviews

ABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science Direct

CBS Statline, Eurostat

Theories about a subject Textbooks and encyclopedias

Information sources

Company information Company Annual Reports, Datastream, Factiva.com, Amadeus

Postgraduate Course

Searching evidence

How do we search?

Search Strategy

Postgraduate Course

Why do we need a search strategy?

Promotes deeper learning about your question

Leads to better yield of quality research.

Saves time in the long run.

Source: Inky Bob, www.flickr.com, Creative Commons, April 2006.

Search strategy

Postgraduate Course

Two types of search strategies

Search strategy

Building blocks methodSnowball method

Postgraduate Course

Snowball method

Starting from one book or article, you search for other literature on the same topic.

Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article).

Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).

Postgraduate Course

Synonyms or related terms

…. …. …. ….


…. …. …. ….


…. …. …. ….

Building blocks method


…. …. …. ….

Keyword 1 Keyword 2 Keyword 3 Keyword 4

AND AND AND

OR OR OR

Postgraduate Course

P = back office employees

I = merger, integration, back office

C = status quo

O = economy of scale

C = healthcare, different organizational culture, unequal


1. Underline the keywords

2. Number the order of importance from 1-4

Postgraduate Course

P = back office employees

I = 1. merger, 3. integration, back office

C = status quo

O = 4. economy of scale

C = 5. healthcare, different 2. organizational culture, unequal


1. Underline the keywords

2. Number the order of importance

Postgraduate Course

Search terms

Operationalise your Pico elements!

O = long term profitability?

Postgraduate Course

corporate culture: organizational behavior/character, corporate identity

merger: acquisition, take-over, fusion, combination, unification

profitability: profit, advantage, return on investment, shareholder value

The keywords of your PICOC may be enough.If not, select more words by using:

Select keywords

synonyms

alternate spelling, translations

related terms / words / subjects

narrower or broader terms

http://thesaurus.reference.com/

Postgraduate Course

Search Query

1. Search with #1 PICOC term (incl. alternative terms,

synonyms, alternate spellings, truncations, etc.) in the

thesaurus, title or abstract

2. Combine the results with OR (use the history function!)

3. Search with #2 PICOC term (incl. synonyms, etc.)

4. Combine the results with OR

5. Combine the results of step 2 and 4 with AND

Postgraduate Course

• Merger

• Fusion

• Combination

• Take over

• Acquisition

• Unification

• …

1. Merger 3. Integration

• Healthcare organization

• Non profit• Not for profit

4. Health care organization

AND

Search Query: an example

I I O

OR OR

• Integration • Corporate culture

• Organizational behavior

• Organizational character

• Corporate identity

• Core beliefs

• Shared values

2. Corporate culture

C

AND

OR

Postgraduate Course

Use the history function to combine results

Search Query

Postgraduate Course

Search Query

Postgraduate Course

Boolean operators

AND = both terms (apples AND oranges)

OR = either one of these terms (apples OR oranges)

NOT = without this term (fruit NOT oranges)

NEAR = near this term (apples NEAR oranges)

* = replaces 0,1 or more characters (apple*= apple, apples, applejack, applejuice, applepie, etc.)

?= replaces 1 character (organi?ation= organisation, organization)

Postgraduate Course

orangesapples

apples AND oranges apples OR oranges

orangesapples

Boolean operators

Postgraduate Course

Justify your search strategy

Why?To help the reader of your paper:

Follow the steps of your search process Understand the end results

How? Including keywords used for the search actions Justify information sources used (literature list)

Search Strategy

Postgraduate Course

Include literature references

Why? To give other authors the credit they are due. To show that you have made use of reliable sources To show the relationship between your work and that of others. To show that you have studied the subject in depth To make it possible to check your work. To avoid committing plagiarism !!!

How? Cite & include references to acknowledge all your sources carefully. Include sufficient own / new ideas in your work. You can make use of Reference Manager or Endnote

Search Strategy

Postgraduate Course

Start up

select ‘advanced’ select ‘peer-reviewed’ select ‘ABI/INFORM Global’

Postgraduate Course

ABI / Inform

Postgraduate Course

Learning through play !Try all buttons

Make lots of mistakes

Have fun !

Go do it & report back next week.

Postgraduate Course

Levels of evidence = A hierarchical order for research designs based on their internal validity

Internal validity = Degree the results may be unbiased. Higher when conditions demonstrating causality are present (1. control over “cause”, 2. temporal order, and 3. control over or no plausible alternative explanation for findings). Careful design of primary studies promotes these three conditions but seldom eliminates them. Threats to internal validity are overcome when

accumulated studies with different designs yield comparable findings.

Levels of Scientific Evidence

Postgraduate Course


Postgraduate Course


It is shown that …

It is likely that …

Experts are of the opinion that …

There are signs that …

Postgraduate Course

But … sometimes observational studies are as good as RCT’s

Internal validity

When the size of effect is very large (swamps the bias)

Postgraduate CourseGeneralizability

Degree findings hold across populations, settings, procedures etc. (external validity).

Reasons for rejecting generalizability must be logical and evidence-based (not mere dislike of findings)

Logical threats to generalizability include: Person/Treatment interactions: e.g., incentives based on

dice throw that work for gamblers and not Baptists File draw problem: Studies only published if show

significant effects (why unpublished sources matter)

Postgraduate Course

These treatments have not been tested in RCTs: are they supported by poor evidence?

Internal validity

Heimlich manoeuvre Dehydration: drinking water

Cardiac arrest: AED

Postgraduate Course

Better than a single study: a replication study

Better than a replication study:a systematic review / meta analysis

If there were 100 studies, 99 of which gave a ‘negative’ result (where, say, the new intervention appeared to be not effective), while one had a ‘positive’ result (were the intervention appeared effective), it would obviously be a mistake to consider only the single positive study.

But ….

Postgraduate CourseResearch designs

Systematic review or meta-analysis

Randomized controlled study (experiment)

Non-randomized controlled study (quasi-experiment)

Observational research: cohort-, panel-, case-control and cross-sectional study

Before-after study (pretest – posttest design)

Qualitative research

Postgraduate Course

Systematic review

The intention behind a systematic review is to identify as fully as possible all the scientific studies of relevance to a particular subject and to assess the validity and authority of the evidence of each study separately. As the name indicates, a systematic review takes a systematic approach to identifying studies and has the methodological quality critically appraised by multiple researchers independently of each other, as a consequence of which the review is transparent and reproducible and can be monitored. The use of statistical analysis techniques in a systematic review to pool the results of the individual studies numerically in order to achieve a more accurate estimate of the effect is termed a “meta-analysis”.

Postgraduate Course

Postgraduate Course

Founded in 1993 and named after the British epidemiologist Archy Cochrane

International non-profit and independen organization

Mission: to enable people to make well-informed decisions abouth healthcare

Dedicated to making up-to-date, accurate information about the effects of healthcare readily available worldwide.

Main product: The Cochrane Database of Systematic Reviews 1995: 36 reviews 1999: 500 reviews 2001: 1000 reviews 2004: 2000 reviews + 1400 published protocols

Reviews prepared by healthcare professionals who volunteer (10.000 people worldwide)

Application of rigorous quality standards

Cochrane Collaboration

Postgraduate Course

Controlled study

In a controlled study two or more groups are compared with each other, usually comprising one group in which an intervention is carried out (experimental group) and one group where no or an alternative intervention is conducted (control group).

Postgraduate Course

In the case of randomization, the groups compared with each other are selected entirely randomly, for example by drawing lots. This means that each participant (or other unit such as a team, department or company) has an equal chance of being in the intervention or control group. In this way, the influence of any distorting factors is spread over both groups so that these groups are as comparable as possible with each other with the exception of the intervention.

Randomization

Controlled study

Postgraduate Course

Observational research

Cohort/panel study—entities followed over time

(classic longitudinal study of AT&T managers)

Case-control study—comparisons between entities

with different outcomes (Collins’ “Good to Great”)

Cross-sectional study—one-time assess’t (turnover

rates of high performing employees in 2012

Observational research refers to studies where the researcher merely observes but does not intervene, with the intention of finding correlations among the observed data (synonym: naturalistic study, non-intervention trial)

Postgraduate Course

Case-control study

Longitudinal study in which one group of people or companies with a particular outcome (for example, above-average performance) is compared subsequently (= retrospective) with a group that does not have this outcome.

Postgraduate Course

Cross-sectional study

Study in which data of a statistically significant sample of a population (managers, CEO’s, employees) is gathered at one point in time. It provides a snapshot of the current condition but does not explain cause and effect.

Cross-sectional studies include surveys

Postgraduate Course

Assessing the quality of a study

Postgraduate Course

Intermezzo

How to read a research article?

Postgraduate Course

1. Title

2. Abstract

3. Introduction

4. Background / review of literature

5. Organizational context

6. Methodology

7. Results

8. Discussion

Structure of an article

Postgraduate Course

In general

Don’t let yourself be taken in by scientific jargon and complex use of language!! Good articles are written in plain English.

Even authorative journals with a high impact factor contain bad articles and vice versa.

Focus on research question, study design and outcome.

Don’t worry about statistics!

Be critical!! Always ask yourself: does this make sense?

Postgraduate CourseMethodological pitfalls

Randomization errors Bias Confounding Reverse Causation

Postgraduate Course

Bias: distortion of the outcome due to systematic errors caused by the way the study is designed or conducted.

NB: If bias is not taken into account then any conclusions drawn may be wrong!

Bias

Postgraduate Course

1. Selection bias

2. Information (detection) bias

3. Performance bias

4. Exclusion (attrition) bias

5. Publication bias

…

…

30. …..

Forms of bias

Postgraduate CourseSelection bias

Error in the way participants in a study were selected. Means comparison groups differ in measured or unmeasured baseline characteristics.

Types of selection bias:

Sampling bias (selecting only successful departments or individuals who have committed crimes)

Participation bias (self-selection, non-response, etc.)

Postgraduate Course

Distortion of the outcome due to misinterpretation of information or systematic errors in the the measurement of research variables which leads to misclassification.

Information bias can be prevented by the use of standardized measurement instruments, hard outcome measures, validated questionnaires and objective, independent and blinded assessors.

Types of information bias:

Reporting bias (recall bias)

Observer bias (interviewer bias, halo-effect)

Information bias

Postgraduate CourseConfounding

Confounding is the idea that a 3rd variable can distort or confuse (or confound..) a relationship between two other variables.

Let’s say that a college education is strongly positively correlated to successful completion of firefighter training. Is it true that people with less than four years of college don’t make good fire fighters? Or cannot fulfill the requirements of well-trained firefighter?

Confounding could exist if the reading materials use in firefighter training are written at a much higher level than the job actually requires.

Postgraduate CourseConfounding

Postgraduate Course

Reverse causation

Postgraduate Course

Critical appraisal of studies

Appendix: Appraisal Questions

Making sense of evidence

Postgraduate CourseEffect Size

Strength or meaningfulness of relationship between two variables (cause/effect)…several metrics exist: Practical value (average $ saved, weight lost, gain in test

scores) Effect strength (standardized indicate of d difference

between treatments or r strength of relationship across multiple studies)

Judgment required: Small effects with low cost can be of practical value (e.g. can be relatively easy to create identification or ingroup/outgroup effects)

Postgraduate Course

Standard appraisal questions

1. Did the study address a clearly focused issue?

2. Is the sample size justified?

3. Is the design appropriate to the stated aims?

4. Are the measurements likely to be valid and reliable?

5. Are the statistical methods described?

6. Did untoward events occur during the study?

7. Were the basic data adequately described?

8. Do the numbers add up?

9. Was the statistical significance assessed?

10. What do the findings mean?

11. Are important effects overlooked?

12. What implications does the study have for your practice?

Postgraduate Course

Appraisal of a controlled study


2. Were subjects randomly allocated to the experimental and control group? If not, could this have introduced bias?

3. Are objective inclusion / exclusion criteria used?

4. Were groups comparable at the start of the study?

5. Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias)

6. Were outcomes assessed blind? If not, could this have introduced bias?

7. Is the size of effect practically relevant?

8. Are the conclusions applicable?

Postgraduate Course

Appraisal of a cohort / panel study


2. Was the cohort / panel recruited in an acceptable way? (selection bias)

3. Was the cohort/ panel representative of a defined population?

4. Was a control group used? Should one have been used?

5. Are objective and validated measurement methods used and were they similar in the different groups? (misclassification bias)

6. Was the follow up of cases/subjects long enough?

7. Could there be confounding?



Postgraduate Course


2. Were the cases and controls defined precisely?

3. Was the selection of cases and controls based on external, objective and validated criteria? (selection bias)

4. Are objective and validated measurement methods used and were they similar in cases and controls? (misclassification bias)

5. Did the study incorporate blinding where feasible? (halo-effect)

6. Was there data-dredging?

7. Could there be confounding?



Appraisal of a case-control study

Postgraduate Course

Assessment of a survey


2. Was the sample size justified?

3. Could the way the sample was obtained introduce (selection)bias?

4. Is the sample representative and reliable?

5. Are the measurements (questionnaires) likely to be valid and reliable?

6. Was the statistical significance assessed?

7. Are important effects overlooked?

8. Can the results be generalized?

Postgraduate Course

Don’t confuse: representativeness and reliability The number of respondents has no direct relationship with

representativeness; even a large random sample can be insufficiently

representative. However, the number of respondents does have an

impact on the reliability of the results.

Assessment of a survey

Postgraduate Course

1. Is your organization / division / population so different from those in the study that its results cannot apply?

2. How relevant is the study to what you are seeking to understand or decide?

3. What are your organization’s potential benefits and harms from the intervention?

4. Is the intervention feasible in your setting?5. What are your executive’s (or client’s) concerns,

preferences and expectations for both the outcome you are trying to prevent and the intervention you are offering?

Organization concerns

Organization Concerns

Ask yourself to what extent the evidence is applicable in your situation:

Postgraduate Course

Turning evidence into practice

Evidence based management:closing the gap between research and practice

Documents

Getting and making sense of “the best available” scientific evidence