The Good, The Bad, and The Ugly: How/How-not to do a Quantitative Risk Analysis

The Good, the Bad, and the Ugly: Risk Analysis of Food SafetyDoug Popkena & Tony CoxbPresented to the Nick Petry WorkshopDec 2013a. Systems View b. Cox AssociatesIntroductionQuantitative Risk Assessments provide an objective means of evaluating risks in food safety and many other areas.Ideally, by relying upon facts and data, rather than anecdotes, emotions, and assumptions, they will guide policy makers to the right decisionsIn reality, faulty methodology and hidden agendas often distort conclusions.The Ugly Ractopamine is used in a number of countries as a feed additive to stimulate the production of muscle mass, reduce fat mass, and improve feed efficiency in swine, cattle, and turkeys at doses of 520 mg/kg of feed.Many (80) countries, including Russia and the EU, currently ban the use of ractopamine as a feed additive but it is allowed in 27 others, including the U.S., Canada, Mexico, Japan, and South Korea under tight limits. The EU recently developed new safe-use standards that are somewhat tighter than those in the U.S.The U.S. and the EU would like to export more pork to Russia. The Russians would like to prevent/stall this on the grounds that foreign pork could be contaminated with unacceptably high levels of ractopamine.We were asked by the USDA/US Pork Board to evaluate the Russian risk assessment: ASSESSMENT OF THE HEALTH RISK TO THE POPULACE FROM EXPOSURE TO RACTOPAMINE FROM FOOD ANIMAL HUSBANDRYThe Russian ractopamine report argues for retaining a complete ban on animal products containing any amount of ractopamine residue.

Russian Study Cites Carcinogenic RiskBased on a study of rats fed ractopamine for 21 months at various levelsThe numbers of rats that developed uterine hyperplasia (benign growths on uterine lining) at each dosage level were observed.A dose-effect function was developed from the data and extrapolated to cancer risk of human dosages obtained from consuming pork.Problem 1: Lack of Causal LinkThe induction of mesovarian leiomyomas in mice and rats also appears to be a general feature of b-adrenergic stimulants, as shown for various other b2-agonists such as salbutamol [asthma, COPD relief] and terbutaline [anti-contraction medication]*..Ractopamine, like other b-adrenerigic sympathomimetics, is therefore not a direct carcinogen and the induction of leiomyomas is considered to be a non-genotoxic event with a threshold, similar to other toxicological end-points*What does this have to do with human cancer risks?

*JECFA. 2004. Toxicological evaluation of certain veterinary drug residues in food. WHO Food Additive Series 53. Ractopamine (Addendum). Prepared by the Sixty-second meeting of the Joint FAO/WHO Expert Committee on Food Additives (JECFA). http://whqlibdoc.who.int/publications/2004/9241660538_ractopamine.pdfProblem 2: Inappropriate Dose-Effect ModelRaw DataLinear Regression ModelDose, g/kg of body mass per day# ratsNumber of response instances (uterine hyperplasia)Probability of developinguterine hyperplasia in rats060002,000600060,0006030.0500200,00060170.2833400,00060250.4167Hint: In the U.S., the ADI (Acceptable Daily Intake) is set at 1.24 g/kg body weight/dayRussian Study Cites Cardiovascular RiskRelies upon studies from human volunteers that demonstrate heart rate increasing with dosage.Parameters from the resulting dose-effect function are fed into a system dynamics model that estimates cumulative effectsThe model output provides # cases by age cohort, assuming lifetime exposureProblem 3: Inappropriate Dose- Effect Model (again)Raw DataLinear Regression FitDose (mg)Dose (g/kg)*Ratio of Dose to U.S. ractopamine ADIChange in bpm relative to control000056753.6010133106.40152001602025333266.43040597477.650

Problem 4: Fudging the DataRefer back to previous slideNote that the regression is based on only 4 of 6 points, that is, the groups with no effect for 5 and 10 mg doses were removed from the regression.This was not justified or even mentioned in the study but we verified the regression parameters result from a fit that removes those two levels.Problem 5: Unrealistic Evaluation ScenarioThe Russian models for cancer and cardiovascular risk require an assumed input level for ractopamine consumptionThe level chosen in their study was based on the assumption that all meat products consumed by all Russians over their entire life time will contain ractopamine residue at the maximum regulatory limit (as set by the European Codex Alimentarius Commission ).Problem 6: Inconsistent LogicNote that even the extreme usage scenario (previous slide) results in an average daily dose of only 0.0371 g/kg versus the U.S. ADI of 1.24 g/kg.The U.S. ADI is based on a safety factor of 54 times below the computed No Effects Level of the human volunteer study.The Russian RA rejects using the results from the human volunteer study for computing a safety factor (not enough data, not enough time, improper methodology, etc). Note that the computed Russian average daily dose is equivalent to a safety factory of 1806, but its still not enough for them.However, their entire cardiovascular analysis is based on a dose-effect parameter they derived from that same study.

The Bad Tetracycline antibiotics are used in the treatment and prevention of bacterial infections of both human and animals. In animal husbandry, they are used as therapeutic and veterinary drugs, and also as a feed additive for disease prevention and feed efficiency.Used as a feed additive in the U.S. since the early 50s. Sub-therapeutic use banned in the EU in 2006 under the precautionary principle.We were asked by the USDA/US Pork Board to review a risk assessment performed by Kazhakstan: "Materials for Human Health Risk Assessment of Tetracycline Intake with FoodThe report argues for a lower limit on maximum tetracycline residues in food animals than other international authorities and scientists have considered prudent.

Problem 1: Lack of knowledge and effortAbout 80% of the KZ risk assessment was a virtual cut and paste from a Russian risk assessment we evaluated last year (Lesson: Know your subject).Problem 2: Association does not equal CausationNormal microflora are critical to the function of the human body.The proportion of the intestinal microflora may change under the influence of tetracycline.A number of diseases (irritable bowel syndrome, diarrhea, constipation, inflammatory bowel disease, duodenitis, food allergies, atopic dermatitis, anemia, immunodifficiency, etc.) are related to imbalance of the intestinal microflora.Therefore, tetracycline residues cause a number of human diseases.Their core logic:Problem 3: Rejecting Accepted Scientific Findings with Weak RationaleWHO recommends an ADI for tetracylines of 30 g/kg of body weight.The KZ tetracycline assessment advocates using an earlier standard of 3 g/kg of body weight.The new standard is based on the results from relatively recent in-vitro studies.As was the case with Russia, KZ bucks the international consensus by claiming that the in-vitro studies are invalid and cannot be extrapolated to humans.Problem 4: Undefined, Non-validated ModelResults presented without describing how they were producedWe know this was lifted from a Russian RA completed last yearThey are outputs of a simulation modelThe underlying system dynamics equations came from a Russian PhD thesis.

Figure 1 - The dependence of the relative abundance of the intestinal microflora (%) on the concentration of tetracyclineProblem 5: No defined hazardThe KZ report stops after presenting the simulation results.Apparently this graph of dysbiosis is all the proof that is needed.Even the Russian RA went on to tie a model of disease generation rates to the bacterial (dis)proportions, based on Russian population studies to quantify effects.Lacking a similar study for KZ, their RA merely stops and jumps right to the conclusions.The Good Cox Associates and Systems View performed a risk assessment for the USDA/Pork Board quantifying the human health risk from MRSA in pork.Human MRSA is typically spread in hospitals, and causes damaging, hard to treat infections, with a high mortality rate.A new strain of MRSA, ST398, was detected in U.S. swine and swine farm workers in 2008, and some years prior in Europe.Alarms were raised! MRSA is in our food!

KATIE COURIC RIPS INTO AG FOR OVERUSE OF ANTIBIOTICS

Katie Couric took down big livestock farming last night on CBS, with that simple, explain-it-to-me cheeriness with which she took down Sarah Palin last year.In the first of a two part series on antibiotic-use in American agriculture, Couric repeatedly linked routine livestock antibiotics to the rise of drug-resistant staph (MRSA):

AUniversity of Iowa studylast year, led by the brilliantTara C. Smith, found a new strain of MRSA in nearly three-quarters of hogs (70 percent), and nearly two-thirds of the workers (64 percent) on several farms in Iowa and Western Illinois. All of them use antibiotics, routinely. On antibiotic-free farms no MRSA was found.[Since then, high levels of ST398 has been found on numerous antibiotic-free farms]

And with this anecdote:Former hog worker, Kim Howland took CBS News inside a factory farm in Oklahoma where she worked two years ago. They administer drugs, you know, constantly, constantly, constantly, Howland said. Thats their fix for everything. She said drugs like Tylan, Keflex, and Baytril, the same classes used to treat everything from skin to respiratory infections in humans were given regularly to pigs that were not sick. Her husband contracted MRSA and almost died. [Turns out it was not ST398 see next slide]

http://fairfoodfight.com/2010/02/10/katie-couric-rips-ag-overuse-antibiotics/Public perception19Do: Take a Deep BreathHospital deaths with ST398 MRSAU.S. cases = 0Worldwide cases = 1 possible (Lozano, 2011)Hospital outbreaks of ST398 MRSA1 instance (maybe), in Netherlands (Wulf et al., 2007)Community outbreaks of ST398 MRSA0 instances reported worldwideInvasive infections with ST398 MRSAU.S. cases = 0EU cases < 10

Do: Clearly Define the Potential HazardColonization among meat handlers and swine handlers, followed by infection, is the main risk of practical concern.ST398 Colonization ST 398 Infection is hypothesized (worst-case assumption) but not observedThis project: Quantify how large these human health risks could be forPig farmersProfessional meat handlersConsumers in general public

Do: Define a stochastic causal model: (workers)Swine herd colonization rateNumber of Swine HerdsSwine workersColonized workersP(worker colonization |swine colonization)Workers Per HerdInfected workersP(infection|colonizationDo: Define a stochastic causal model(consumers)Number of US familiesColonization among Dutch meat handlersColonization among US meat handlers Colonized consumersPork attributable fractionInfectedP(infection|colonizationUS/Netherlands ratio of MRSA on porkUS/Netherlands ratio of pork processingHandler/consumer pork contact ratioPork handlings/family/yearDo: Populate model with conservative distributions based on real dataAssume transmission (arrows) may occur even if not specifically observedDevelop probability distributions from known data (scientific reports, surveillance data, etc.)Use conservative, bayesian distributions with noninformative priors, e.g. betabinomial(s+1,n-s+1), mean = (s+1)/(n+2) [Note: this is very handy when there have been no observations (s=0)]Do: Develop output distributions via simulationPerform large numbers of simulations of the stochastic modelEach run utilizes a new random value for each distribution.Final output is a probability distribution of the conservative estimate

Mean infections in U.S. = 1.00/yr(.968 pig farm workers,.024 food handlers,.008 consumers)

The Good Cox Associates and Systems View performed a risk assessment for the USDA/Pork Board quantifying the human health risk from Toxoplasma Gondii (T. gondii) in pork.Commonly found in pork until the 90s.Most infected adult humans suffer few or no detectable ill effects from toxoplasmosis (carried by ~ 9% of U.S. adults). However, infection can be deadly for AIDS patients and devastating for the unborn children of pregnant women (blindness, retardation, chronic illness)Spread via cat feces to livestock (and humans)Poultry and pork in open (e.g. free range) production systems have greatly increased risk of T. gondii infection

Do: Clearly Define the Potential HazardInfection risk is from eating raw/undercooked pork (in the case of newborns, it is transmitted from the mother).Risk is proportional to the prevalence of T. gondii in pork.Numerous studies have shown that the prevalence of T. gondii in pork is proportional to the fraction of U.S. pork not raised in confinement.Do: Define a stochastic causal modelPrevalence among hogs in confinementPrevalence among unconfined hogs Current Fraction of unconfined hogsNumber InfectedTuner Fraction of unconfined hogsPopulation SizePrevalence in PorkP(Infection|Prevalence)Do: Populate model with conservative distributions based on real dataVariety of papers and USDA studies on prevalence of T. gondii in confined and/or unconfined hog populationsMedical articles on illness, hospitalization, and mortality ratesKey research article allowed us to convert numbers of illness by group (adult, newborn) to potential QALYs lost (Quality Adjusted Life Years)Do: Perform simulations to obtain a range of possible outcomes

Outcome (Equation)Mean5%95%Total Cases (3)37,027.0821,128.7556,698.22Hospitalizations (4)1904.72962.803173.62Deaths (5)138.4071.07226.88Congenital Cases (6)40.4520.6666.55Total QALYs (7)4036.722091.796587.93

Current T. gondii Annual ImpactsIncremental QALYs Lost versus Fractional Increase in Unconfined HogsEquates to an average of 1 QALY lost per 676 hogs moved out of confinement (note approx. 6.5M hogs are consumed/year)SummaryWhile many now at least give lip service to the value of performing quantitative risk assessments, these can still be done badly.You will often see association confused with causation, even by scientists and analysts who should know the difference.Facts and data are the best counters to fear and uncertainty