A comparison of laboratory performances

A comparison of laboratory ~

performances How to check the reliability of analytical environmental sewices

Julie €I. Einerson Phyllis C. Pei

SMdia Nmonal Loboratories Albuquerque, NM 87185

Analytical laboratory results often are used to determine whether a facility or process is in compliance with permit or regulatory conditions; therefore, credi- ble data are an essential component of the decision-making process for an environmental manager.

At Sandia National Laboratories in Albuquerqne, NM, accurate and precise data are essential for managing our envirunmental programs. Because we seek high credibility of laboratory data, a study was conducted to determine the management of measurement accuracy and precision by commercial laboratories. This article describes some procedures that will help identify those laboratories that provide precise and accurate analytical services on a con- sistent bask Ten laboratories ( A 4 were selected . .

to participate in the competency tests. Laboratory qualifications were evaluated in six areas that had preassigned weights (Table 1): precision, accuracy, quality assurance documents, ability to customize reports, turnaround time, and price. Laboratories were scored and ranked by each category. Catego- ries were then weighted and summed to produce a list of laboratories ranked in order of competence.

A set of written instructions were given to the laboratories, and written price quotes and quality assurance (QA) plans were obtained from each laboratory. The laboratories were given check samples that were spiked with six inorganic contaminants commonly found in wastewater discharges. These samples were sent along with routine wastewater samples. Preparation of standards. Standard

solutions containing loo0 ppm of a

metal were purchased. The standards were certified for use in atomic absorp tion spectroscopy and, except for arsenic, were secondary standards trace able to the National Bureau of Standards. The standards purchased were cadmium (Cd) as cadmium ni-

tassium dichromate, mercury (Hg) as mercuric chloride, arsenic (As) as arse-

trate, trivalent chromium (CI(,)) as po-

nic trioxide in nitric acid, copper (Cu) as copper oxide in nitric acid, and lead (Pb) as lead nitrate. These stock solutions were used to prepare the check samples.

Ten batches of 2-L acidified check standards were prepared. All of the check standard solutions were poured into a 20-L container and thoroughly mixed. The final pH of the solution wa$

W1393BW88NW1~1121EOl.50/0 0 1988 American Chemical Society Envimn. Sci.Technol..Voi. 22. No. 10, 1988 1121

tested to be < 2 . The final concentra- tions of the metals (true values) after dilution were As, 1.0 ppm; Cd, 0.05 ppm; Cr,,), 0.05 ppm; Cu, 5.0 ppm; Pb, 5.0 ppm; and Hg, 0.5 ppm. Deionized distilled water blanks containing 5 mL of nitric acid per 1ooO mL. were also included as samples to detect any possible contaminant in the diluent. The water blanks submitbed to the laboratories all contained less than detectable amounts of contaminants.

Analytical results One container was given to each of

the 10 laboratories to analyze six check sample metal parameters: As, Cd, Cr,,,, Cu, Pb, and Hg. The labs also were given another duplicate set of the samples to analyze for Cr,,, and F'b. Table 2 outlines the raw data reported by each laboratory.

precision. The precision error was calculated as follows:

(1) where X , and X, are replicate values and TV is the hue value.

The smaller the difference between X , and X,, the smaller the % error and the better the precision. Laboratories G and J demonstrated 100% precision in both the chromium and lead analyses. In general, the laboratories performed

[(X, - X,)/2]rrV x 100%

bet& in the analyses of chrohium than lead. Filly percent of the labs were 100% precise in the analysis of chromium, whereas only two labs were 100% precise in the analyses of lead. The % error in precision for chromium and the % error in precision for lead were then combined by using the root mean square formula to provide an overall precision rate (Table 3). The smallest error was assigned the highest rank of 1. Except for lab A, most had good precision.

Aecuraey. In order to assess the degree of agreement between the measured value and the true value, the accuracy for each lab was measured. The difference between the reported value and the true value divided by the m e value all multiplied by 100% provides the accuracy error rate

(2) where Xis the measured value and TV is the true value.

Lab A showed an error of more than 1ooO-fold for arsenic. It is interesting that one lab, Lab B, consistently erred

, on the negative side of the true value in all six parameters, whereas labs G and H consistently erred on the positive side of the m e value (except in the case of arsenic). Table 3 illustrates the average % error from the six parameters for each of the laboratories. Labs D, G, and H all demonstrated an average ac-

'.''

[(X - Tv) /Tv ] x 100%

1122 Emimn. Sci.Technoi..Vol. 22, NO. 10, 1984

curacy error rate of less than 10%. Precision and accuracy. These cate-

gories were weighted 4096, as accurate and precise numbers are essential for environmental data. AU the elements of the sampling process-from the sampling techniques to sample preserva- tions, chains of custody, sample blanks, labratory analyses, and documents- tion-contribute to the reliability of data. It must be kept in mind that if an analytical laboratory does not produce reliable results, the entire sampling ef- fort may be rendered useless. The ulti- mate cost will be much greater if re- sampling has to be conducted. More laboratory performance8 are measured in this manner now, as is evident in the EPA Contractor Laboratory Program Statement of Work. Lab A ranked tenth in precision and ninth in accuracy; Lab G ranked first in precision and second in accuracy One may conclude from this correlation that if a laboratory has poor precision in its measurement of replicate samples, then its accuracy should also be suspect.

Quality assurance. The laboratories

were requested to provide a copy of their Quality Assurance manual for r e view. The 11 criteria under evaluation were chosen for their importance in as- suring the quality of analytical results. This list is by no means comprehensive but is meant to highlight critical aspects of a QA program. The total points available per criterion ranged from 1 to 4. The highest w r e earned the highest rank. Table 3 summarizes the total scores earned by each laboratory. The evaluation criteria included the descrip- tion and frequency of blind check samples, method detection limits, method blanks, equipment calibration, matrix

degree of laboratory errors should be minimized. For example, lab G ranked second in QA and accuracy and tied for 6rst in the precision category. The expected correlation is that excellent QA plans, when executed, will result in good accuracy and precision in analyses, as seen with lab G. Lab H ranked first in the QA and accuracy but fifth in precision, and lab E ranked third in QA, but eighth in precision and sixth in accuracy.

The results here suggest that not all QA plans are practiced at the lab bench. often, for the sake of standardization, the corporate division of a laboratory

spike sampl&, check samples, replicate samples, internal audits, EPAKLP (Contractor Laboratory Program) par- ticipant, chain-of-custody use, and corrective action program.

A good quality assurance program is essential to the accuracy and precision of analytical results. If employees are expected to understand and follow specifications, it is important that a plan be documented and implemented. If a good QA plan is followed properly, the

would produce the QA plan without in- put or feedback from the analysts. The QA prccedures could be inappropriate or not practicable in the field laboratc- ries. If no one verifies that the QA plan is being implemented, then the plan be- comes a mere marketing tool; there will be no discipline or consistency in the quality of the lab’s output.

There were instances in which a lab ranked low in QA and performed well in analysis. For example, lab C ranked

Environ. Sci. Technol.. Val. 22, No 10. 1888 1123

ninth in QA but third in precision; Lab J ranked eighth in QA but tied for first in precision. Possibly these laboratories did not explain their full QA programs well, the evaluation criteria were not adequate in scoring these labs, or the analysts have good techniques in spite of the QA plans. In this latter case, some additional time spent on relining a QA plan should correct the problem and result in a better score in this evaluation.

a m a r o u n d time. The samples were shipped to the laboratories in ice chests on the same day via Federal Ex- press. Although laboratories agreed to report data within 14 days, few were able to meet the deadlime. Those labs that met the turnaround time received a rank of 1. The lab providing data latest was ranked tenth (Tables 3 and 4). We found that the two laboratories (A and B) with the best turnaround time rank ninth and tenth, respectively, in the ac-

lation between the qualii of the e

Ability to customize reports. The laboratories were requested in writing to provide the analytical data in the fol- lowing manner. (These points were specific in the written instructions eiven to

curacy category. Rushing does not pay off.

'hrnaround time is important because regulatory agencies may que sampling and rewrting within a speck .,

the 10 laboratories.) EPA-approved methods are to be used for sample analysis. The refer- ences to the methods are to accom- pany each parameter's analytical results. The analytical results will be reported with confidence limits for each parameter based on the techniques and equipment used by the laboratory.

*The dates the analyses were performed are to be reported for each parameter. All chain-of-custcdy forms are to be returned to Sandia National Labora- tories after the samples reach the laboratory destination. All sample dam- ages and abnormalities are to be noted on the forms and signed by the laboratory recipient. Regulatory agencies often require

different formats for the submission of environmental monitoring reports, and this facilitates the use of computers by the agencies to track permit compliance information. The instructions used in this evaluation were some of the requirements imposed by the city of Albuaueraue. Those labs that followed

fied-(ofiin shortj amount of time.. ~ e - lay in reporting the results is usually frowned upon by regulatory agencies. The monitoring of process wastewater, for example, requires a short turnaround time so that parameters that are out of compliance may be quickly cor- rected; violation penalties for out-of- compliice parameters axe based on per-day calculations. Also, it is difficult for the environmental manager to a p proach the production manager about a high metal value that was discharged three months ago and expect effective corrective action today. Therefore, the sooner the problem is identified, the easier it is to determine the source of the problem.

Price. All the laboratories were given a specific set of analysis parameters in writing and asked to quote a price. They were asked to analyze for 33 parameters from 11 samples; mu- tine wastewater samples and the check

all th6 ins'tructions received a score of 10 and a rank of 1, and so forth (Tables 3 and 4). If Labs G and H (both of which followed all of the instructions)

mation from h e laboratories. There- fore, the ability of the laboratories to

tomers' requirements is essential. tailor analytical reports to suit cus-

1124 Environ. Scl.Technol..Yol. 22, No. 10, 1988

samples were mixed in for analysis. There was a wide range of prices charged by laboratories for the same amount of work. The price vs. performance relationship is further com- plicated by some labs charging 150% and 2W% of the base price to compen- sate for the required turnaround time in the study

However, there is not a definite correlation between the quality of the work and the price of the work. The laboratory that ranked lowest in the precision category, lab A, was also the second- highest priced vendor. The highest priced vendor, lab C, ranked ninth in the QA category. It is interesting to note that lab J ranked first in price but eighth in QA, and lab F ranked second in price but loth in QA. This indicates that the cost of implementing a QA program is not reflected in the prices charged. Theoretically, a well-documented, well-executed QA plan should lower the long-term costs to the laboratory, and subsequently, the costs to the customer.

Oversll ranking. The ranks for each laboratory were multiplied by the weight assigned for each category. The rank x weight values were totaled for each lab and divided by 6, the number of categories. The lowest weighted average ranked first, and so forth (Table

ond, and third, respectively. Lab H ranked first in three categories, and lab G ranked first in two categories. Labs C and F, which ranked ninth and tenth in QA, respectively, tied for last in the overall ranking. No lab ranked last in more than one category. Our study con- firms that there is a wide spectrum of differences in laboratory performances and that the customer should be aware of the differences before evaluating his or her data.

Problems One interesting problem arose during

this study. When a laboratory is aware that check samples are sent to examine its performance, more. care may be

4). Labs H, G, and D ranked h t , ~ e c -

ly by the labralory,

taken to ensure the confidence of reported data. Laboratory A was given the same samples one week apart. In the first sample, the laboratory was aware of the purpose of the samples. In the second week the same sample was given to the laboratory along with other routine wastewater samples. The results are listed in Table 5.

Analysis of the first sample resulted in parameters that were very close to the true values. Without including Cu, the average accuracy error was 12%, whereas the average accuracy error for the blind check data was 270%. It ap- pears that less care or attention was given to the blind samples. The blind As value of 1200 ppm would have been a major cause of concern if a regulatory agency submitted the sample. Based on this datum, the regulatory agency could have assessed a fine or issued a compliance order to the regulated industry. (In one of the authors’ experience, this sit- uation has occurred.)

Conclusions Ours was a small-scale study of com-

mercial laboratories that provide sewices in the Albuquerque area. We were able to select at least three laboratories with standards acceptable to Sandia Na- tional Laboratories. Among our other conclusions and suggestions are the fol- lowing. Do not take reliability for granted.

One cannot blindly rely on commercial laboratories for the values of contaminants; data must be critically evaluated. Check samples must be submitted to verify the precision and accuracy of the laboratory data. Sample blanks and equipment blanks are necessary to verify the validity of the analyses. A good laboratory invites its customers to sub- mit check samples.

Establish good communications. It is important to establish specific con- tacts and communications with the commercial laboratory. Good communications will ensure that the require- ment? of each party are understood. Sample collection and preservation techniques, sample media, labeling, and custodies must be understood by the laboratory, just as internal procedures of the laboratory should be understood by the customer. It is a good idea to physically audit the laboratory and review its adherence to the written QA program, the qualifications of its staff, and recordkeeping procedures.

Continually monitor laboratory performance. A laboratory’s performance should be periodically checked and charted to ensure data reliability. Blind check samples and replicates should be submitted periodically, and the customer should inform the laboratory if it produces an unacceptable er-

ror in its accuracy or precision. Per- formance charts also may help to reveal trends or problems. The customer’s re- sponsibility is to demand quality work so that he or she can intelligently man- age his or her programs, but the laboratory has an obligation to investigate questionable data and to take the appro- priate corrective actions in response to customers’ needs. The laboratory must provide quality work to bring the customers back. Do not expect to pay more for bet-

ter performance. As the demand for laboratory analytical work increases in response to environmental legislation, laboratory capacities and turnaround times will become critical. One may tend to choose whatever laboratory may be available at the time of need. It is important, however, that laboratory products be closely scrutinized to as- sure that accurate data are provided and meaningful decisions are made. Fur- thermore, our study demonstrates that there is no relationship between the price of analyses and the reliability of the data; therefore, price is not a good indicator of performance. The quality of the results will be the final cost to the customer.

Beware of comparing data from different laboratories. Many compan- ies are instituting long-term (e.g., 30 years) groundwater-monitoring programs. If laboratory precision and accuracy are poor, one may be feeding meaningless analytical data into com- plex computer programs to perform groundwater modeling. The output will be disappointing because we have found so much variability in data from different laboratories. It is also unwise to try to compare data from different laboratories that were analyzed at different times. Laboratory personnel, equipment, and management often change, and these variables contribute to errors. Unless there is constant vigi- lance in the oversight of laboratory performance, the data may not make sense.

Maintain credibility with the laboratory. The customer must maintain credibility with the laboratory. He or she should not provide misleading information or try to “trick” the laboratory. For example, it should be stated up front that the customer plans to sub- mit 5% or 10% of quality control samples with the environmental samples. The customer must continually provide feedback to the laboratory on its performance. Check samples must be carefully prepared and verified, where possible, by another laboratory. The pursuit of true sample values should be a partnership between the laboratory and the customer.

Acknowledgment This article has been reviewed for suitahil- ity as an ES&T feature by Richard G. Zepp. EPA Environmental Research Laho- ratory. Athens , G A 30613. and by Lawrence H. Keith, Radian Corporation, Austin, TX 78720-1088.

Additional readings Gautier. M. A. et al. Quolir? A , ~ . m r m t l for

Healfh nnd En~~ironnuvzral CIwmi.cq; Los Alamos National Laboratory. Lor Alamos. NM. 1986.

Meyerhein. R . Anolwicnl and SumplinK Rrh- niquer; New Mexico Scientific Laboratory Division. Xerox.

Rice. C.; Brinkman. J . ; Muller. D . Rdiabilify of C h e m i d Ann1y.w of W<mr SnmplP,~-The Erperienrr of rhe UMTRA Projpcr. Xerox.

Taylor. J. K . Qualify A.~surorrcc of Clwmicd Meosurrme!m; Lewis: Chelsea. MI, 1987.

Handbook for Anolwical Quolit? Conrral in Wmer Wartewnrer Lahoroiorirs; U.S. Envi- ronmental Monitoring and Support Labora- tory. Office of Research and Dcvelopment. U.S. Environmental Protection Agency. Na- tional Technical Information Service; PB- 297451, 1979.

Merhm1.v fir Chemicnl Annly.~i,r of Water and Wnrrw U.S. Environmental Monitoring and Support Laboratory. Office o f Research and Dcuelopment. U.S. Environmental Pmtec- lion Agcncy. National Technical Information Service: Cincinnati, OH, 1983: PP84- 128677.

Srandmi Merhods for the fixominorion ,,f WCI. rerond Wn.rrrw,orer, 16th ed.; Amcrican Puh- lic Health Association: Washington. DC. .nec ,YO,.

Err Methods for EvaluufinR Solid Wnsrr; 3rd ed., Office o f Solid Waste and Emergency Response. U.S. Environmental Protfctioh Agency: Washington. DC, 1986.

Wofrr nnd Cwironrnmlol Zchhnolopy. Annual Book of ASTM Srandards: American Society for Testing and Materials: Philadclphia, PA. 1987: Vnl. I1.01.

Julie H. Einerson (IJ ronmenfal programs era1 Electric j oircra huquerque, NM. Sh business from fhe L Alhuquerque, and a hiolop from fhe U. Colorado in Puehlo. experience in rhe em

currrnfly r h e cnvi- irdinaror for Gen- rt,?ine group in Al- ias an M.B.A. in erriry of Phoenix, ;. in chrmismF and Prsity ($ Sorrfhern e has four ?cum of nmenralfield

Phyllis C. Pei (rJ is an environmemalproj- ecf manager for Sandra Nafionnl Dlhoraro- ries. She has on M.A. in hiod,e,nrrty from rhe Universiry <f P x m Health Science Cenfer, Dallas. and an M.B.A. in husiness and a B.A. in chemisfo, hofh from rhe Uni- versir). of Dallas. She has 1Oyrar.s of experience in the rmalwical ond mvirorrmmral fields.

Environ. Sci. Technol.. Vol. 22, NO. 10, 1988 1125

Documents

A comparison of laboratory performances