Winery Laboratory Quality Control - UCANR

How to Successfully Fudge Analytical Results

copyright 2008 Patricia Howe www.patriciahowewines.com 1

Winery Laboratory Quality Control

Pat HoweFermentation Readiness:

Harvest Decisions from the Vineyard to the Winery

UC Davis, July 29, 2019



Winery Laboratory Quality Control

…or…












Fudge: Definition

• [noun] soft creamy candy

• [verb] fake or falsify; “fudge the figures"; "cook the books"; "falsify the data"Synonyms: manipulate, fake, falsify, cook, wangle, misrepresent



Disclaimer

• This talk begins with a humorous and ironical presentation of how to fudge analytical results

• I do not advocate fudging of lab results.

• The purpose of this ironical and humorous approach is to two‐fold: • To show that a typical laboratory without a quality control program will to use the same techniques that a fudger would use

• To show that a basic quality control program will provide tools to defend or substantiate both expected and unexpected analytical results

• If you are Ironically Challenged, see footnotes on each slide as a guide to know when I am being humorous.



Reasons to fudge‐ what motivate a fudger?

• Time constraints

• Inadequate physical resources

• Inadequate training

• Laziness

• Boredom

• Long scientific tradition, the “dry‐lab”

Humorous and Ironical



Keys to successful fudging

• Understand the winemaking process and why the analysis is being requested

• Understand the analytical methods and equipment used

• Understand “expected” results

• Cover your tracks

• Anticipate questions

• Divert suspicion

Successful fudging is not easy




How is the analysis being used?

• Actionable analyses

• Establishing baselines

• Required analyses

• Curiosity/research

• Longstanding tradition




If your results are questioned…

• Making up results is only half the art of fudging

• Successfully defending your fudged result is critical to your success

• Six standard strategies

How would you react if you were being called out for fudging?




Common strategies if results are questioned

• Bad Sample

• Blame the analyst

• Error in reporting

• Lousy equipment

• Lousy procedure

• Blame the previous results

Let’s go through these techniques




1: Bad Sample

• Wrong sample • Wrong wine, wrong tank, wrong barrels, etc

• Improperly taken• Tank not mixed, bad berry sampling, bottom sample instead of top, before pump over not after

• Improperly stored• Refrigerated/frozen or left in hot car

• Improperly processed• Filtered or not, decolorized or not, centrifuged or not


This is also a true issue even with valid analytical methods



2: Blame the analyst

• Works best in multiple shift environment• Incompetent graveyard shift

• Works great in high staff turnover environments (and can contribute to high staff turnover, too!)

• The new guy is incompetent.

• Backfires if you are the analyst

• Backfires if the analysts gets wind of this and waits for you after work




3: Error in reporting

• The result was good, but:• Math error

• Transcription error

• Digits reversed

• Decimal place missed


This is also a true issue even with valid analytical methods; human error can’t be underestimated



4: Lousy equipment

• The equipment is too high tech and you can’t figure it out

• Might get you a trip to the training center

• The equipment is too old and you need to get new stuff

• Works good when your budget is always being cut




5: Lousy procedure

• Has never worked and is always suspect

• Fraught with all sorts of problems• No standard reagents, the equipment is weird

• You’ve tried all sorts of improvements: • Change sample size, time of processing, temperature, new reagents, different equipment, etc

• “Everyone” has problems with it• You are constantly discussing this with other wineries and trying to solve the problem




6: It was the previous result

• The current result is correct!

• The earlier value was wrong!

• Works for baseline and addition analyses

• Works great if staff, procedures, or equipment has recently changed




Amateur Fudging

• How many of those fudging techniques sound strangely familiar?

• Are you fudging even though you are performing analysis? You may be “fudging” without even knowing that you are doing it!

• Can you prove you are not just making up a number? How can you determine if you are getting valid results?

• What is a valid analytical result?

• What is the difference between:• Fudging the results vs Running a procedure without validating and checking?

Deadly Serious



What is a valid analytical result?




What Winemakers Mean by “Good” Results

• Knowledge of situation (expectations)

• Acceptable ranges (“reasonable values”)

• Historical perspective (previous values)




What lab technicians mean by “Good” results

Winemaker doesn’t complain




How the Lab can be “Good”

• Anticipate expectations

• Only report “reasonable values”

• Check historical values

Common Winery Laboratory approach

Also the techniques used by a good fudger!




The dangers of “good” results

• There are four possible situations when evaluating analytical outcomes

• The true value of the sample could be either• Expected, within specification, routine, usual or normal

• Unexpected, out of spec, bizarre, unusual or abnormal

• The lab results could be either• The true value or

• Not the true value

Deadly Serious



Example: Bottling and Winery A

Lab Result= True Value

Lab Result= Not True Value

30 ppmRoutine True

Value

30 ppm (expected)Standard routine

5 ppm added = 35

20 ppm (unexpected) Result is questioned, and

rechecked 5 ppm added = 35

20 ppmUnusual True

Value

20 ppm (unexpected)Result is questioned

but sustained. 15 ppm added = 35

30 ppm (expected) NOT rechecked

5 ppm added=25

Deadly Serious

Bottling Tank SO2 checksUsually first result is 30 ppm and than a 5 ppm addition

for a final approval at 35 ppm



Example: Barrel Checks at Winery B

Lab Result= True Value


0.8 g/LExpected True

Value

0.8 g/L (expected)Standard routine

Barrels combined

1.4 g/L (unexpected) Result is questioned,

and rechecked Barrels combined

1.4 g/LUnusual True

Value

1.4 g/L (unexpected) Result is questioned

but sustained. Barrels

quarantined

0.8 g/L (expected) NOT rechecked

Barrels combined and infection spread

Deadly Serious

Checking barrel lots for VA prior to rack to tank and return.Last check of the lots showed range of 0.7 to 0.9 g/L VA



The Lab’s View

Lab Results = True Value


Situation ARoutine or

expected value

NO COMPLAINTS

COMPLAINTS

Situation BUnusual or problematic

value

COMPLAINTSPrevented problems

NO COMPLAINTSProblems missed

Deadly Serious



Which is the worst situation?

Lab Results = True Value


Situation ARoutine or

expected value

NO COMPLAINTSThe “perfect” lab

COMPLAINTSBad reputation for

laboratory, questioning results

Situation BUnusual or

problematic value

COMPLAINTSPrevented problemsA strong qc program needed to report and support these results-but wine is protected

NO COMPLAINTSProblems missed

If the problem is not attributed to the lab,

then NO negative feedback The lab is

still “perfect”-but wine is

compromised

Deadly Serious



Types of Errors‐the Lab’s View

• Wineries may be missing a lot of problems because their lab might be more comfortable reporting expected values

• If the only “good” result is one that is expected, why waste time running analyses?

In other words…

THE LAB MIGHT AS WELL FUDGE!




Worst Case is a “good” (expected) number…

• The worst case is when the lab gets the “expected” value when it is not true

• When the lab reports an expected value that is incorrect no extra lab work is required, and wine quality is sacrificed

• But: Many wineries train employees to only question values that are “unexpected”

All values should be questioned

All values should be substantiated

Deadly Serious



Traditional Winery Laboratory Teachings

Actual Quotes from some Wine Analysis Texts may inadvertently promote fudging philosophies:

“Finally, we frequently fail to ask ourselves, “is this value reasonable?”

“Looking at one’s analytical results from a perspective of past values and common‐sense expectations of possible ranges or magnitudes of values can save some embarrassment.”

Deadly Serious



Let’s review: How did we get here?

• An analysis technique which always gets a “good” result will cause the least amount of trouble

• Any procedure which consistently gives “good” results will rarely be questioned

• A procedure can be horribly inappropriate for the situation, but if the results are always “good” it will be used

• The results may be “good” but they may also be inaccurate and imprecise. That is, a “good” result may be “WRONG”!

How do we fix this problem?

Deadly Serious



Anti‐fudge: A basic QC program

• All procedures are written, followed, and reviewed regularly and are fit for the task.

• Written procedures include performance criteria such as precision, accuracy and upper and lower limits as found in the lab

• All employees are trained and signed off on each procedure they perform. Competence standards are part of training.

• Laboratory Control Samples are run routinely, and there is an action plan for out‐of‐control results.

• The lab participates in performance testing, either in‐house, external, or both

Deadly Serious



Anti‐fudge: A good procedure

• Is accurate and precise based on your needs.

• The procedure (even a standard method from a book) has been validated in your lab with your equipment and personnel

• Accuracy and precision are proven, tracked, and recorded

• Upper and lower levels are known

• Has an associated written procedure that is followed

• Is reviewed for performance and read routinely by employees to update and improve

Deadly Serious



Accuracy and Precision

• Is the procedure Accurate and Precise?

• How can we assure ourselves that the result is accurate and precise?

Deadly Serious



Is this Procedure Accurate?

• Accuracy is the degree of agreement of individual measurements with some true or reference value

• It is an indication of how close to the true value the result is

• Determined by running proper standards• Not determined by comparing to other labs, despite frequent occurrence…as many wineries are unable to prepare proper standards

Deadly Serious



Is this Procedure Precise?

• Precision is the degree of mutual agreement among individual measurements

• It is an indication of how close multiple results are to one another

• Terms such as replicable, repeatable, and reproducible are used in reference to precision

Deadly Serious



Sources of Variation in Types of Precision

Source of Variability

Replicable Repeatable Reproducible

Sample Same Same Same

Sub sample Same or different

Same or different

Most likely different

Analyst SameAt least one of these must be

different

Different

Apparatus Same Different

Day Same Same or different

Laboratory Same Same Different

Deadly Serious



Common Ways to Test these Variations

Replicable Repeatable ReproducibleSame person runs a duplicate sample in the same run.

A different person runs the same

sample

Another lab runs the same sample

Same person run the same sample later in the day

A different piece of equipment is

used

Different people, equipment, sub sample,

day, or lab

Minimizes the variation,

should have the “best” precision

Maximizes the variation within

a lab, more realistic

Would expect the lowest precision: this

is also sometimes called “robustness”

Deadly Serious



Laboratory Control Samples & Control Charts

• A laboratory control sample (LCS) is simply a sample which is run frequently.

• Typically at start up, at shut down, and spaced between every x number of samples.

• It must be treated exactly as a normal sample

• It may contain a known amount of substance (a reference control) or not.

Deadly Serious



Getting started with an LCS & Control Charts

• Locate wines which are analytically identical to use for your LCS

• Treat the LCS exactly as any normal wine sample

• Perform analysis and record the results • Generate about 12 or 15 points for each analyte

• Calculate mean and standard deviation for each analyte

• Plot and chart the points and note fluctuation around the mean

• Integrate the analysis of controls into your daily runs

• Rules• > 3 sigma is out of control; >2 sigma is “warning”, more complicated rules for rigorous compliance

Deadly Serious



Controls and Control Charts

• Good analysis will be randomly distributed • 68 % of results are within 1 sigma of the mean

• 95.5% are within 2 sigma

• 99.7 % are within 3 sigma

151050

14.5

14.0

13.5

13.0

Sample Number

Sam

ple

Mea

n

X-bar Chart for Alcohol

X=13.59

2.0SL=14.05

3.0SL=14.29

-2.0SL=13.13

-3.0SL=12.90

Deadly Serious

Gau

ssia

n cu

rve



Controls and Control Charts

• Good analysis will be randomly distributed • 68 % of results are within 1 sigma of the mean

• 95.5% are within 2 sigma

• 99.7 % are within 3 sigma

151050

14.5

14.0

13.5

13.0

Sample Number

Sam

ple

Mea

n

X-bar Chart for Alcohol

X=13.59

2.0SL=14.05

3.0SL=14.29

-2.0SL=13.13

-3.0SL=12.90

Deadly Serious

Gau

ssia

n cu

rve



Simple Control Charting

• Set up in Excel• Need good excel skill set

• Contact me if you want a copy of my old sheet

• Freeware out on the web

• Record on paper and physically chart‐• Quick and easy, but someone still must crunch data

Deadly Serious



Control Charting Excel

Deadly Serious



MedLabQC (freeware)

Deadly Serious



Are the results valid? Or is the lab unknowingly Fudging?

• Use a standard method and VALIDATE the method as it is used in YOUR laboratory

• Determine the optimum accuracy and precision

• Determine linearity, minimum and maximum values

• Run blanks, standards, and controls routinely• Determine the working precision (and accuracy if known)

• Empower the technicians with the means to determine when they are doing a good job

• Control charts should show variations in results appropriate for the test‐ an occasional “out of control” result and a corrective action is the sign of a healthy laboratory environment.

Deadly Serious



THANK YOU!

Questions?

[email protected]

Documents

Winery Laboratory Quality Control - UCANR