Pit stop EBMgt

Evidence-Based Management

(Searching & Finding Scientific Evidence)

Pitstop Evidence-Based HR, VOV lerend netwerk,

Antwerpen, 11 januari 2013

Pit stop EBMgt

1. Evidence-Based Management

A (very) Short Introduction

Pit stop EBMgt

McMaster University Medical School, Canada

Medicine: Founding fathers

David Sackett Gordon Guyatt

Pit stop EBMgt

How it all started

Pit stop EBMgt

More than 1 million articles in 40,000 medical journals per

year (= 1995; now probably more than 2 million). For a

specialist to keep up this means reading 25 articles every

day (for a GP more than 100!)

Most of the new insights and treatment

methods don’t reach the target group

Problem I: too much information

Pit stop EBMgt

Problem I: too much information

HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR

manager to keep up this means reading 3 to 4 articles

every day (for a ‘general’ manager more than 50!)

Pit stop EBMgt

Problem II: persistent convictions

if you’re hyperventilating

breathe into a bag

Pit stop EBMgt

Problem III: jumping to conclusions

elderly people who have an irregular heartbeat are much more likely to die of

coronary disease

give them a drug that reduces the number of

irregular beats

Pit stop EBMgt

How 40,000 cardiologists can be wrong

In the early 1980s newly introduced anti-arrhythmic drugs were found to be highly successful at suppressing arrhythmias.

Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality.

The CAST trial revealed Excess mortality of 56/1000.

By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.

Pit stop EBMgt

David Sackett

Half of what you learn in medical school will be

shown to be either dead wrong or out-of-date

within 5 years of your graduation; the trouble is that

nobody can tell you which half.

The most important thing to learn is how to learn

on your own.

(Remember that your teachers are as full of bullshit

as your parents)

Pit stop EBMgt

Evidence-Based Practice

1991Medicine

1998Education

1999Social care

2000Nursing

2000Criminal justice

????Management?

Pit stop EBMgt

2. Evidence based management:

What is it?

Pit stop EBMgt

Evidence based practice:

Improve information to support decision making

Definition

Pit stop EBMgt

Definition

Evidence-based management means making decisions

about the management of employees, teams or

organizations through the conscientious, explicit and

judicious use of four sources of information:

1. The best available scientific evidence

2. Organizational facts, metrics and characteristics

3. Stakeholders’ values and concerns

4. Practitioner expertise and judgment

Pit stop EBMgt

Four sources

Pit stop EBMgt

Evidence-based practice:

Focuses on the decision making process

Thinks in terms of probability (instead of golden bullets).

Pit stop EBMgt

3. Evidence-based management:

Why do we need it?

Pit stop EBMgt

Four sources

Pit stop EBMgt

Trust me: 20 years of management experience!

Pit stop EBMgt

Bounded rationality

Pit stop EBMgt

Bounded rationality

System 1

Fast Intuitive, associative heuristics & biases

System 2

Slow (lazy) Deliberate, ‘reasoning’ Rational

Pit stop EBMgt

Seeing order in randomness Mental corner cutting Misinterpretation of incomplete data Halo effect False consensus effect Group think Self serving bias Sunk cost fallacy Cognitive dissonance reduction

System 1: prone to biases

Confirmation bias Authority bias Small numbers fallacy In-group bias Recall bias Anchoring bias Inaccurate covariation detection Distortions due to plausibility

Pit stop EBMgt

2. Searching & Finding

The Best Available Scientific Evidence

Pit stop EBMgt

Step 1: Formulate a focused question

Pit stop EBMgt

5-step approach

EBMgt is a 5-step approach

1. Formulate an answerable question (PICOC)

2. Search for the best available evidence

3. Critical appraise the quality of the found

evidence

4. Integrate the evidence with managerial

expertise and organizational concerns and apply

5. Monitor and evaluate the results

Pit stop EBMgt

Asking the right question?

Effect vs Non-effect

Pit stop EBMgt

Types of questions: effect

Does it work?

Does it work better than ....?

Does it have an effect on ....?

What is the success factor for ....?

What is required to make it work ...?

Will it do more good than harm?

Effect

Pit stop EBMgt

Types of questions: non-effect

Needs: What do people want or need?

Attitude: What do people think or feel?

Experience: What are peoples’ experiences?

Prevalence: How many / often do people / organizations ...?

Procedure: How can we implement ...?

Process: How does it work?

Explanation: Why / how does it work?

Economics: How much does it cost?

Pit stop EBMgt

Focused question?

Does team-building work?

What are the costs and benefits of self-steering teams?

What are the success factors for culture change?

Does management development improve the

performance of managers?

Does employee participation prevent resistance to

change?

How do employees feel about 360 degree feedback?

Pit stop EBMgt

What is a ‘team’?

What kind of teams?

In what contexts/settings?

What counts as ‘team-building’?

What does ‘work’ mean?

What outcomes are relevant?

Over what time periods?

Foreground question?

Does team-building work?

Pit stop EBMgt

P = Population

I = Intervention (or success factor)

C = Comparison

O = Outcome

C = Context

Answerable question: PICOC

Pit stop EBMgt

Example: merger

Pit stop EBMgt

Answerable question: PICOC

Imagine you are a consultant, your client is the board of

directors of a large Canadian health-care organization.

The board of directors has plans for a merger with a

smaller healthcare organization in a nearby town.

However, it’s been said that the organizational culture

differs widely between the two organizations. The board

of directors asks you if this culture-difference can impede

a successful outcome of a merger. Most of them

intuitively sense that cultural differences matter, but they

want an evidence-based advice.

Pit stop EBMgt

Answerable question: PICOC

What else would you like to know?

Pit stop EBMgt

Answerable question: PICOC

P: What kind of Population are we talking about? Middle managers,

back-office employees, medical staff, clerical staff?

O: What kind of Outcome are we aiming for? Employee productivity,

return on investment, profit margin, competitive position, innovation

power, market share, customer satisfaction?

P/C: And how is the assumed cultural difference assessed? Is it the

personal view of some managers or is it measured by a validated

instrument?

Pit stop EBMgt

According to the board the objective of the merger is to

integrate the back-office of the two organizations (ICT,

finance, purchasing, facilities, personnel administration,

etc.) in order to create economy of scale. The front

offices and primary process of the two organizations will

remain separate.

The cultural difference is not objectively assessed (it is

the perception of the senior managers of both

organizations).

Answerable question: PICOC

Pit stop EBMgt

P = back office employees in a healthcare

organisation

I = merger, integration back office

C = status quo

O = economy of scale

C = different organizational culture, unequal

Answerable question: PICOC

Pit stop EBMgt

1. Read the following five scenario’s

2. Formulate on the basis of each scenario a focused

question (use the PICOC format).

Exercise

Pit stop EBMgt

Scenario 2

“Ik werk als manager bij een grote financiële dienstverlener. Het

concern is een holdingmaatschappij met daaronder een aantal divisies

met een eigen P&L verantwoordelijkheid. Het accountmanagement van

de divisies wordt aangestuurd op basis van een eigen set kritische

prestatie indicatoren (kpi’s) die niet onderling is afgestemd.

Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen

rijden bij gezamenlijke klanten en de omzet wordt ‘weggekaapt’ bij de

ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de

Executive Board van de holding een beroep gedaan op de

divisiedirecties om meer met elkaar te gaan samenwerken.

Mijn vraag is: staat het feit dat de divisies geen gedeelde kpi’s hebben

een goede interne samenwerking in de weg of hoeft dit geen probleem

te zijn?”

Pit stop EBMgt

Scenario 3

“I am a Dutch quality manager at a large international brewery. In the past 4 years I facilitated a lean management program that is running for more than 10 years in this organization. Since a couple of years, however, we see a decline in results, such as:

The number of problem analysis and improvement teams has been increased, but the amount of breakdowns and short stops in the production lines has not decreased.

There is slow progress for the total program for the Netherlands at this moment.

According to the middle managers the improvement program does not address really important topics. I would therefore like to know how our employees feel about lean management.”

Pit stop EBMgt

1. Neem een actueel probleem / vraag / issue dat speelt in de

organisatie waar je stage hebt gelopen (of een vraagstuk

dat je persoonlijke interesse heeft).

2. Formuleer op basis daarvan een onderzoeksvraag

3. Geef aan wat voor type vraag het is (effect of non-effect)

4. Benoem de PICOC elementen.

Pit stop EBMgt

Step 2: Search for the best available

evidence

Pit stop EBMgt

5-step approach

EBMgt is a 5-step approach

1. Formulate an answerable question (PICOC)

2. Search for the best available evidence

3. Critical appraise the quality of the found evidence

4. Integrate the evidence with managerial expertise

and organizational concerns and apply

5. Monitor and evaluate the results

Pit stop EBMgt

Searching evidence

What do we search?

Pit stop EBMgt

Type of Information Source

Current Information Wall street Journal, Financial Times, Business week, Financieel Dagblad

Overview of a subject

General background

Academic Information

Statistical Information

Textbooks and popular books

Encyclopedias, yearbooks & book reviews

ABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science Direct

CBS Statline, Eurostat

Theories about a subject Textbooks and encyclopedias

Information sources

Company information Company Annual Reports, Datastream, Factiva.com, Amadeus

Pit stop EBMgt

Articles in peer reviewed journals

Postgraduate Course

Searching for evidence

DatabasesPostgraduate Course

ABI/INFORM

Business Source Elite

PsycINFO

Web of Knowledge

ERIC

Google Scholar

Postgraduate Course

Searching for evidence

Searching evidence

How do we search?

Search Strategy

Postgraduate Course

Pit stop EBMgt

Two types of search strategies

Search strategy

Building blocks methodSnowball method

Pit stop EBMgt

Snowball method

Starting from one book or article, you search for other literature on the same topic.

Snowballing to older publications by finding out which

publications were used by the author (see bibliography of

book or article).

Snowballing to more recent publications by finding out

how often that book or article has been

cited by other authors (see Web of

Knowledge or Google Scholar).

Pit stop EBMgt

Snowball method

Pit stop EBMgt

Synonyms or

related terms

• ….

• ….

• ….

• ….

Synonyms or

related terms

• ….

• ….

• ….

• ….

Synonyms or

related terms

• ….

• ….

• ….

• ….

Building blocks method

Synonyms or

related terms

• ….

• ….

• ….

• ….

Keyword 1 Keyword 2 Keyword 3 Keyword 4

AND AND AND

OR OR OR

Pit stop EBMgt

P = back office employees, health care

I = merger, integration, back office

C = status quo

O = economy of scale

C = different organizational culture, unequal

Answerable question: PICOC

1. Underline the keywords

2. Number the order of importance from 1-4

Pit stop EBMgt

P = back office employees 5. healthcare

I = 1. merger, 3. integration, back office

C = status quo

O = 4. economy of scale

C = different 2. organizational culture, unequal

Answerable question: PICOC

1. Underline the keywords

2. Number the order of importance

Pit stop EBMgt

Search terms

Operationalise your Pico elements!

O = long term profitability?

Pit stop EBMgt

Construct validity

The Ambidextrous Organization (??)

Pit stop EBMgt

corporate culture: organizational behavior/character, corporate identity

merger: acquisition, take-over, fusion, combination, unification

profitability: profit, advantage, return on investment, shareholder value

The keywords of your PICOC may be enough.If not, select more words by using:

Select keywords

synonyms

alternate spelling, translations

related terms / words / subjects

narrower or broader terms

Pit stop EBMgt

Search in the following order

1. Thesaurus (SU)

2. Title (TI)

3. Abstract (AB)

4. Alternate spellings, synonyms

Pit stop EBMgt

• Merger

• Fusion

• Combination

• Take over

• Acquisition

• Unification

• …

1. Merger 3. Integration

• Healthcare organization

• Non profit

• Not for profit

4. Health care organization

AND

Search Query: an example

I I O

OR OR

• Integration • Corporate culture

• Organizational

behavior

• Organizational

character

• Corporate

identity

• Core beliefs

• Shared values

2. Corporate culture

C

AND

OR

Pit stop EBMgt

orangesapples

apples AND oranges apples OR oranges

orangesapples

Boolean operators

Pit stop EBMgt

Start up

Pit stop EBMgt

Always search for indexed topics or thesaurus terms

Pit stop EBMgt

thesaurus terms

Pit stop EBMgt

Use the history function to combine results

Combining results

Pit stop EBMgt

Use the history function to combine results

Combining results

Pit stop EBMgt

Narrow your results

Pit stop EBMgt

Narrow your results

Pit stop EBMgt

Save your results

Pit stop EBMgt

If an article is not available in ABI/ Inform or BSE: try Google (advanced search)

Pit stop EBMgt

Summary

1. Formulate a focused question and break it down into keywords (PICOC),

2. Underline the 2 to 4 most relevant keywords,

3. Start with the most relevant keyword: is it listed in the thesaurus? If not: search in the title or abstract.

4. Repeat your search with alternative terms, synonyms or alternate spellings.

5. Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative terms) with OR (use the history function!)

6. Repeat above steps for keyword 2

7. Combine the outcome of your search with keyword 1 and 2 with AND (use the history function!)

8. Filter the articles with “studies” (subject > studies)

9. Read the titles and abstracts of your results, select the relevant articles and safe them.

10. No full text available in ABI/Inform or BS Elite? Try Google Advanced.

Pit stop EBMgt

Learning through play !

Try all buttons

Make lots of mistakes

Have fun !

Pit stop EBMgt

Assignment: search strategy

Search for research articles on teambuilding

Pit stop EBMgt

2. Search in ABI/Inform:

1. How many articles has Stephen Covey published in peer reviewed journals?

2. How many of these articles are based on scientific research?

3. Are there articles (by other authors) that are critical of Covey’s 7 Habits?

4. How many of these critical articles are based on scientific research?

Exercise: Search for evidence

Pit stop EBMgt

Evidence-based management

Step 3: Critical appraisal of studies

Pit stop EBMgt

5-step approach

EBMgt is a 5-step approach

1. Formulate an answerable question (PICOC)

2. Search for the best available evidence

3. Critically appraise the quality of the found

evidence

4. Integrate the evidence with managerial

expertise and organizational concerns and apply

5. Monitor and evaluate the results

Pit stop EBMgt

Effect vs Non-effect

Pit stop EBMgt

Internal validity

Pit stop EBMgt

internal validity = indicates to what extent the

results of the research may be biased and is thus

a comment on the degree to which alternative

explanations for the outcome found are possible.

Internal validity

Pit stop EBMgt

Levels of internal validity

Pit stop EBMgt

Which design for which question?

Explanation

Pit stop EBMgt

Best research design?

Pit stop EBMgt

Critical appraisal

Pit stop EBMgt

Critical appraisalPostgraduate Course

1. Is the study design appropriate to the stated aims?

2. Was a control group used?

3. Was a pretest used?

4. Are the measurements likely to be valid and reliable?

5. Could bias or confounding have occurred?

6. How large was the effect size?

Pit stop EBMgt

Levels of internal validity

1. Were there enough subjects in the study?

2. Was a control group used?

3. Were the subjects randomly assigned?

4. Was a pretest used?

5. Was the study started prior to the intervention or event?

6. Was the outcome measured in an objective and reliable way?

6x yes = very high (A)

5x yes = high (A)

4-3x yes = limited (B)

2x yes = low (C)

1-0x yes = very low (D)