Rapid Risk Assessment (and/or Rapid Risk Analysis) Are they possible? Villie Flari, Food and Environment Research Agency, UK [email protected]

Rapid Risk Assessment(and/or Rapid Risk Analysis)

Are they possible?

Villie Flari,

Food and Environment Research Agency, UK

[email protected] JIFSAN / Fera annual meeting, June 2011

Trigger for this talk…

• … part of my work when at CFSAN, FDA (2007 – 2009)– “Rapid Risk Assessment Framework” project: a tool to apply with CFSAN

emergency response procedures– Ambitious and quite challenging project: aimed to identify whether there is

room for improving the existent strategies, employed at the time by CFSAN/FDA during emergency events, and if yes to suggest and justify alternative approaches

• Review FDA/CFSAN processes to assess public health threats during emergency events.• Identify areas of opportunities for implementing risk assessment thought approaches

with the view to improving the current FDA/CFSAN process.• Prepare a framework document to describe the recommended process.• Develop templates for assessing information, characterizing unknown factors and for

communicating assessment results effectively to risk managers and/or risk communicators.

Project orphan..

Are any changes to current schemes and approaches truly needed?

• What is the outcome we want to estimate – which is our aim?– Preventive approaches and emergency responses for ensuring food safety

and protecting human health

• Overall, not that bad

• Apart form regulatory processes a number of initiatives and support systems nationally and internationally developed over the last 30 years - currently in place (e.g. RASFF)

2008 2008-9 2009 2010 2011

Source: Food Safety Risk Analysis: a guide for national food safety authorities, (FAO Food and Nutrition Paper 87, 2006)

Food safety challenges - constantly change…

Emerging sciences & technologies

Dynamic

Emerging risks – emerging sciences / technologies

Nano-sized materials

Synthetic Biology

Genetically Modified Organisms

?

?

??Unknown,

thus far, emerging risk(s)Tactical and strategic

problems

Designing rapid risk assessments – are they possible?• What is the outcome we want to estimate – which

is our aim?– What do we mean precisely when we term a

document as a “risk assessment” or a “rapid risk assessment”?

– How do we define the term “risk”?• Safety assessment?• Risk assessment?• A collection of (background + epi) information?

Possible scenarios

Hazard

Chemical Biological

Known Scenario 1 Scenario 3 Scenario 5

Unknown Scenario 2 Scenario 4 Scenario 6

Identifiedvehicle (s)

Non identified vehicles (s)

Hazard

Chemical Biological

Known Melamine in infant formula – 2008 (previous relevant case: Melamine in pet food – 2007)

Salmonella in peanut butter – 2009 (previous relevant case: Salmonella in Peter Pan)

Salmonella saintpaul in fresh produce – 2008/2009

Unknown Melamine in pet food - 2007

Scenario 4 Scenario 6

Identifiedvehicle (s)

Non identified vehicles (s)

Looking into historical events & following events “in vivo”

Historical events

– “Melamine in pet food” - March to May 2007

Following events “in vivo”

– “Salmonella saintpaul in fresh produce” – May to August 2008

– “Salmonella typhimurium in peanut butter” - November 2008 to January 2009

What did we learn?

Incomplete knowledge

Limited time

Many dimensions

FIRMS’PRACTICES

HistoricalData (Firm)

Kill steps

Compliance

EPIDEMIOLOGY

Number of Ill Illness onsetrange

Most illnesses> 10/01

Deaths relatedto outbreak

IllnessesOutside USA

Raw dataCase controlStudy resultsEpi curve

Source Vehicle (s)

Complaints

Questionnaires

Packing

ProcessingPlants

SOP

Production

CLINICALDATA

INVESTIGATIVE LABORATORY FINDINGS

INVESTIGATIONFINDINGS

PRODUCTSMANUFACTURED

Shelf lifeof product

Trace back1st level

Trace back2nd level

Best before dateLOT

Internationalsamples

FIRMdata

FDAdata

STATEData

FACTS

OUTBREAK INVESTIGATION

DISTRIBUTIONTRACEFORWARD

DatabasesInspection reports

BACKGROUND INFORMATION

Trace back3rd level

Distributors

TRACEFORWARD

DomesticInternational

Recall

FOOD SUPPLY CHAIN

Market share(specific to firm (s)

General infoSpecific to product

Distributors

Firm’s productTrace back

RISK ASSESSMENT

Lines of investigation of emergency events: relevant information needs R

efDo

c 11R

efDo

c 11B

ackgro

un

d / ratio

nale

Problem i.e. Risk Management question (s)See RefDoc1 and 2 for extended information

Is Risk Assessment required by RM?

Define all lines of investigationSee RefDoc11 for extended information

Evaluate dataAccept, Reject, Withhold data

See RefDoc 14 for extended information

Form hypothesisSee RefDoc 14 for extended information

Risk Managementdecision

Do not accept

hypothesis

Repeat for each line of investigation

Monitor & Organize DataSee RefDoc12 for extended information

Hypothesisaccepted

Data considered enough foraccepting hypothesis?

See RefDoc 14for extended information

Need to redefineproblem?

Risk Managementoptions

Yes

No

Yes

No

Consider all lines of investigation

NoYes

Unc

erta

inty

ana

lysi

sD

ecis

ion

mak

ing

proc

ess

See

RefD

oc 3

& 1

6 fo

r ext

ende

d in

form

ation

YesNo

Problem classifiedAs Class I Recall?

Assess problem against criteria for Class I Recall1


Define RA endpoint (s)See RefDoc 8 for extended information

Continue investigation

Introduce RA as line ofinvestigation


Risk

Com

mun

icati

on

Test hypothesisSee RefDoc 14 for extended information

EPIDEMIOLOGY

INVESTIGATIVELABORATORY

FINDINGS

INVESTIGATIONFINDINGS

FIRMS’PRACTICES

TRACEFORWARD

FOOD SUPPLY CHAIN

RISK ASSESSMENT

Risk Assessment approach can be employed to ask information about the

denominator of epidemiological and investigative laboratory results: e.g.

“During the “Salmonella typhimurium in the peanut butter” case how many

samples have they been put under test?”

Usually, a lot of attention is paid on the positive microbiological samples,

but not on the total of samples tested. The denominator could give us

information on the actual risk posed by the particular microbiological

hazard.

Risk Assessment would allow one to be able to answer

questions regarding “how risky it is to consume peanut

butter products”, e.g. “if I eat 200 crackers from 200

different packs which was the probability to have been

contaminated by Salmonella typhimurium?”

Risk Assessment would allow to address

questions regarding the effect of risk

factors if certain mitigation methods

would be followed; for example “If the

temperature during a kill step is always

above the threshold of 350 F what would

be the decrease in probable pathogen

concentration in a cookie?”Usually a presence/absence of the

microbiological hazard is given. Colony

Forming Units (CFUs) information is not given.

As a result, Dose Response information is

missing. Requesting CFUs could help risk

assessments in the future as well.

Risk Assessment’s value:

Bringing all the information together – connecting all these lines of

investigation in order to reveal patterns, hypotheses.

Gather information, direct research for the future.

RefD

oc 11

RefD

oc 11

Backg

rou

nd

/ ration

aleB

ackgro

un

d / ratio

nale

Is this what we need?

• Partially yes, but ….

Risk assessments / analyses in emergency responses: why, how?

The STEC outbreak reported from Germany is noteworthy considering its magnitude: 276 HUS cases and two deaths due to HUS reported in just a few weeks. The majority of cases reported are adult women. Usually, about 15% of children with STEC infection present with HUS, with this proportion being much lower among adults. This means that several hundred STEC cases with diarrhoea are likely to be occurring in the current outbreak.

The isolated outbreak strain STEC O104:H4 is very rare. Prior to the current outbreak, only one case has been documented in literature, and this case was a woman in Korea in 2005.The case-control study in Hamburg revealed that raw tomatoes, fresh cucumber and leafy salad are the likely vehicles of infection. Samples of fresh cucumbers taken in Hamburg tested positive for STEC, however, the exact time and place of contamination remains unclear.

It is unclear whether the results from Hamburg can be extrapolated to the whole of Germany. Furthermore, it cannot be excluded that an alternative food item is the vehicle of infection. The definite source of the infection remains to be confirmed. There is currently no indication that either the source of the outbreak or the vehicle of infection has been distributed outside of Germany.

Likely 47% sure?71% sure?

83% sure?

The question then becomes…• Whether approaches and thought processes we

develop for and apply in innovative Risk Assessments (or Analyses…), can help us overcome difficulties such as those we encountered recently in the food safety field.– Chemical: Can we identify the true hazard quickly?– Microbiological: Can we identify the true vehicle(s)/source(s) of an

outbreak quickly?– Can we foresee possible future challenges relevant to new sciences

and technologies that are being developed?– Can we produce reproducible, transparent, fully defensible

assessments / analyses that will withstand media pressure, public criticism?

What did we learn?

The work on this project highlighted the significance a number of aspects that appear to be inherent in an emergency investigations:•Unknown factors, and in particular the value of embracing uncertainty - embracing uncertainties by recognizing them, classifying them, characterizing them can only be beneficial for all involved in the investigation, the scientists, the decision makers and the public.•The art of eliciting expert judgment: subjective information plays a major part in the science on which rapid decisions are made. How do we ensure that we obtain the most informative points of view, and how do we ensure that we capture experts’ uncertainty? How do we combine different opinions, if this is deemed necessary?•Following a problem – solving approach: this type of work may be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation.


Limited time

Uncertainty in risk assessments, risk analyses in emergency events: why, how?• What is the desired outcome – which is our aim?• Rapid Risk Assessment

– Risk is the uncertainty of the outcome• Outcome(s) Risk Assessment endpoint(s)

– Which are the chances of my child dying if s/he consumes x amount of contaminated food?

– Which is the proportion of Denmark population who may suffer from severe adverse affects (i.e. >10 days of hospitalisation) if they consume x amount of contaminated food?

• Uncertainty of outcome(s)– Probability distributions - Quantitative (Hard data and/or

expert judgment) – + Qualitative (acquired via expert judgment) information

ANDYHART

What did we learn?



Limited time

Expert judgment in risk assessments in emergency responses: why, how?

• Who is to be involved?– Best expertise possible

• Lists of experts available• Calibration of experts only in retrospect (time-consuming)

– Fit for purpose method• Qualitative information?• Quantitative information?• Both?

Uncertainty of experts to be captured Trained risk analysts Table top exercises

ROGERCOOKE

What did we learn?



Limited time

Problem i.e. Risk Management question (s)See RefDoc1 and 2 for extended information

Is Risk Assessment required by RM?

Define all lines of investigationSee RefDoc11 for extended information

Evaluate dataAccept, Reject, Withhold data


Form hypothesisSee RefDoc 14 for extended information

Risk Managementdecision

Do not accept

hypothesis

Repeat for each line of investigation

Monitor & Organize DataSee RefDoc12 for extended information

Hypothesisaccepted

Data considered enough foraccepting hypothesis?

See RefDoc 14for extended information

Need to redefineproblem?

Risk Managementoptions

Yes

No

Yes

No

Consider all lines of investigation

NoYes

Unc

erta

inty

ana

lysi

sD

ecis

ion

mak

ing

proc

ess

See

RefD

oc 3

& 1

6 fo

r ext

ende

d in

form

ation

YesNo

Problem classifiedAs Class I Recall?

Assess problem against criteria for Class I Recall1


Define RA endpoint (s)See RefDoc 8 for extended information

Continue investigation

Introduce RA as line ofinvestigation


Risk

Com

mun

icati

on

Test hypothesisSee RefDoc 14 for extended information

RefD

oc 1

RefD

oc 1

Investigators, AnalystsDecision Makers,Risk Managers

1: Report trigger (s)

2: Phrase question (s) of investigation

Questions consideredunambiguous by

investigators, analysts?

Assessment of problem, i.e. risk management question (s),

begins

Yes

No

Tem

plate flo

w ch

artT

emp

late flow

chart

Backg

rou

nd

/ ration

aleB

ackgro

un

d / ratio

nale

Defining the problem - question

It is best to define the problem (i.e. risk management question (s)) precisely to ensure

that answers provided fit the purpose.

“To identify the food vehicle responsible for causing an outbreak of

Salmonella typhimurium infections with the following PulseNet

strains1:JPXX01.1818, JPXX01.1825, JPXX01.0459”.

Problem (i.e. risk management question (s)) is dynamic; thus, it may need to be

redefined in subsequent phases of the investigation.

“To a) identify distribution patterns of Peanut Corporation of America

products, b) elicit expert opinion on manufacturing practices and c) review

epidemiological data associated with Salmonella typhimurium, particularly

strains JPXX01.1818, JPXX01.1825, JPXX01.0459 in collaboration with CDC

with the view to prioritizing products according to the risk they may impose

to consumers”.

Advantages:

Diminishing ambiguity in provided answers.

Allowing investigators, analysts who participate in the emergency case to get

familiar with important information of the investigation irrespective of their role

and/or expertise and have the opportunity to look at the big picture of the

investigation.

RefD

oc 1

RefD

oc 1

Tem

plate flo

w ch

artT

emp

late flow

chart

Backg

rou

nd

/ ration

aleB

ackgro

un

d / ratio

nale

What did we learn?• Following a problem – solving approach: this type of work may

be quite different from simply following a process. Asking “what is the problem, and how the problem could be defined?” is not merely a philosophical question; instead, it is the most important question to be settled in the start of an investigation

• “Identifying, defining and then respecting the natural shape of the problem does not come naturally to agencies, since these are based on an organizational structure that was formed years ago and most of the approaches they follow are process-based” – Sparrow, 2008

• Perhaps, such an approach becomes even more challenged during an investigation of an emergency event as the organization is under extra pressure to deliver results and facilitate relevant decision making.

People

Perceptions Biases

Following a problem-solving approach

“Solving a problem simply means representing it so as to make the solution transparent”

Herbert A. Simon: The sciences of the artificial, 1996

Pathology findings:Acute/subacute kidney

failurein cats and dogs

03/15/2007

Chemistry findings:Melamine present in

contaminated wheat gluten03/25/2007

Hypothesis:More than one compounds

responsiblein association with other

agents for illnesses and deaths

Toxicology findings:Melamine not toxic

for mammals, in particular forthe recorded adverse effects

03/27/2007

Hypothesis:Melamine responsible

for illnesses and deaths

Chemistry findings:Further compounds identifiedin contaminated wheat gluten

ureidomelamine, urea,ammeline,

ammelide, cyanuric acid03/30/2007

Hypothesis:Other compound(s)

responsiblefor illnesses and deaths

Toxicology findings:There may be some additional

linksbetween melamine and uric

acid04/03/2007

Toxicology findings:No compound, of those identified

in contaminated wheat glutenis considered as toxic to mammals

04/04/2007

Melamine

Ammeline

Urea

Ammelide

Cyanuric Acid

Ureidomelamine

Pathology findings:Oral toxicology experiments indicate

formation of crystals due tocombination of chemicals, e.g.

melamine+ammeline+ammelide+cyanuric acid04/24/2007

Melamine Ammeline

Ammelide

Cyanuric Acid

Building a conceptual model of the problem: Building a conceptual model of the problem: example based on “Melamine in example based on “Melamine in pet food” emergency eventpet food” emergency event

RefD

oc 15

RefD

oc 15

Exam

ple

Exam

ple

ECRT decides that:HypothesisIs accepted

Intervention A

InterventionB

InterventionC

ECRT COLLABORATESWITH EXPERTS TO DEFINE

OPTIONS OFINTERVENTION

What if scenario

…

ECRT COLLABORATESWITH EXPERTS,

ANALYSTS, AND DECISION MAKERS

TO APPLY DECISIONMAKING TOOLS

What if scenario

What if scenarioWhat if scenario

What if scenario

What if scenario

What if scenario

What if scenario

What if scenario

What if scenario

ECRT COLLABORATESWITH EXPERTS TO DISPLAY

SENSITIVITY ANALYSIS,GATHER INFORMATION

ECRT COLLABORATESWITH EXPERTS AND

RISK MANAGERSTO DEFINE

STAKEHOLDERS,CRITERIA

Aggregation of criteria in a single function in order to be able to “compare” and prioritize between alternative scenarios. This could be performed via a number of Decision Making tools.

Stakeholders – Criteria - Units of measure for criteria Stakeholders – Criteria - Units of measure for criteria

RefD

oc 16

RefD

oc 16

Defau

lt meth

od

to h

and

le info

rmatio

nD

efault m

etho

d to

han

dle in

form

ation

Decision makingmodels

Challenges in ameliorating risk assessments / risk analyses: Strategic versus tactical• Similarities

– Encompass uncertainties– Define problem in question precisely– Employ the best experts– Provide support tools for transparent, reproducible, fully

defensible assessments

• Further challenges when dealing with emergency events– All of the above become even more crucial when dealing

with emergency investigations!

Thank you!

• CFSAN / FDA colleagues– Sherri Dennis– Elisa Elliott– Jack Guzewich– David Hattan– Faye Fenstein– Karl Klonz– Sherri McGarry

QUESTIONS??

[email protected] JIFSAN / Fera annual meeting, June 2011

Documents

Rapid Risk Assessment (and/or Rapid Risk Analysis) Are they possible? Villie Flari, Food and Environment Research Agency, UK [email protected]