Probability of Detection AHi

7/28/2019 Probability of Detection AHi

http://slidepdf.com/reader/full/probability-of-detection-ahi 1/8

249

ASNT Fall Conference and Quality Testing Show 2009 [Columbus, OH, October 2009]: pp 249-256. © Copyright 2009, 2011, American Society for Nondestructive Testing, Columbus, OH.

Probability of Detection Analysis for Eddy Current Inspection Systems

ABSTRACT

Jennifer Herberich Brown

Pratt & Whitney Rocketdyne

M/S 731-84, P.O. Box lO9600, West Palm Beach, FL 33410-9600

(561) 796-2230; fax (860) 622-3453;[email protected]

Eddy current (EC) inspection is a non-destructive evaluation (NDE) method commonly used for flight quality

assurance of commercial, military, and rocket engine hardware. EC inspection is used to detect surface or near

surface anomalies, such as cracks, in metallic hardware by sensing changes in coil impedance. Understanding the

relationship between crack size and system response is the basis for determining the detection capability of an EC

system and establishing find/no-find decision rules. For a given EC system, the relationship between the EC

response and crack size can be evaluated by scanning a set of standard specimens with a known number and

distribution of crack sizes. The results are analyzed using the Ii versus a analysis method to quantity the relationship

between the EC response (i.e., Ii) and crack size (i.e., a) and develop a Probability of Detection (POD) curve, which

defines the crack size that can be detected with a specified level of reliability. Available POD or statistical software

packages have the ability to perform an Ii versus a analysis. However, the validity ofthe analysis, resulting POD

curve, and any parameters of interest (e.g., a90,a90 /

9S) can be significantly impacted by the choice of analysis inputs.Hence, it is critical that the practitioner have a basic understanding of the statistical methods behind the "black box"

so as to ensure a valid analysis and avoid drawing misleading conclusions. This paper provides a practical overview

of the Ii versus a methodology and illustrates how some of the analysis inputs impact an Ii versus a analysis.

BACKGROUND

Eddy current (EC) inspection is a non-destructive evaluation (NDE) method commonly used for flight quality

assurance of commercial, military, and rocket engine hardware. One application is detection of surface

near-surface anomalies, such as cracks, in metallic hardware. When the EC probe is placed on metallic hardware, it

induces an electrical current, called an eddy current, in the hardware. When the probe scans over a surface crack or

near surface crack, the induced current is altered. The change in the coil impedance is sensed by the EC instrument,

which processes the signal and displays it as a measurable quantity. The measured EC signal is expected to increase

as defect size increases. Understanding the relationship between crack size and EC signal response is the basis for

determining the detection capability of an EC system and establishing find/no-find decision rules.

Provided that key EC system parameters are well-understood and controlled, a probability of detection (POD)

demonstration test and analysis is the best available method for quantitying the relationship between crack size and

system response and determining the crack size that the EC system is capable of detecting with a specified level of

reliability. A POD demonstration test is typically conducted in an environment similar to the anticipated production

EC environment using a set of standard specimens of the same geometry and material that have a known number

and distribution of crack sizes. Data on EC response for a given crack size is generated by scanning the specimens.

The Ii versus a analysis method, which is the recommended method per MIL-HDBK-1823, is used to quantitying the

relationship between the EC response (i.e., Ii) and known crack size (i.e., a). A POD curve can then be developed

based on the established relationship to define the capability of the EC system. Several POD software packages are

available with the capability to perform an Ii versus a analysis. However, just because the POD software produces a

POD curve does not mean that the POD curve is valid. The validity of the analysis, resulting POD curve, and anyparameters of interest (e.g., a90 /95) can be significantly impacted by the choice of user-defined analysis inputs.

Hence, it is critical that the practitioner have a basic understanding of the inner workings of the "black box" and how

the analysis inputs affect the results so as to ensure a valid analysis and avoid drawing misleading conclusions.

AVERSUS A ANALYSIS INPUTS AND UNDERLYING STATISTICAL METHODSStandard inputs to Ii versus a analysis that are required by POD software include the specified Ii versus a model

formulation, the saturation threshold level, the noise threshold level, the decision threshold, and the POD



250


demonstration test data. To understand how the analysis inputs can affect the results of an aversus a analysis, it is

necessary to have a basic understanding of the underlying statistical methods on which it is based.

The aversus a analysis method involves two parts, which are executed sequentially: (1) quantifYing the relationship

aand a, and (2) developing a POD curve. Both parts are based on established statistical methods. Part 1 is based on

advanced regression techniques. Part 2 is based on normal probability theory. An overview of each part is

presented in the following sections.

Quantifying the Relationship between Ii and aThe underlying assumption in EC inspection is that EC signal response is assumed to increase with flaw size. The

relationship between EC signal response (denoted a) and known crack size (denoted a) is assumed to be best

described by a simple linear regression model.

A simple linear regression model is characterized by having a single continuous predictor variable and a single

continuous response variable, where the relationship between them is assumed to be linear. The theoretical model is

expressed as Ji i = /30 + /31 * Xi + Cj , where Xi represents the value of the predictor variable, /-li is the expected mean

response for a given value of the predictor variable, is the y-intercept, is the slope, and Ci is the error term. Ci is

included in the theoretical model to represent the naturally occurring variation around /-li for a given Xi. In linear

regression it is assumed that the error terms Ci are independent and follow a normal distribution with mean 0 and

constant variance. In other words, the observed variability in the data for a given value of the predictor variable Xi is

expected to be centered around /-li and should not increase or decrease as the value of Xi increases or decreases.

The predictive model derived from sample data is expressed as il j = bo +bl *Xj where A is the predicted mean

response for a given value of the predictor Xi, and bo and b l are estimates of and respectively. Note that the

error term Ci is not an expression in the predictive model. This is because, in theory, the value ofci is expected to be

oon average. However, Ci plays an important role in assessing model validity, which will be discussed in detail

later.

In practice, when analyzing EC inspection POD demonstration test data, the predictive model is typically expressed

as In(a) = bo+bl * In(a) rather than a bo+bl *a . This formulation of the model falls in the class of simple linear

regression models and makes practical sense since it eliminates negative values. Flaw size and measured EC signal

response are positively valued data. Using the natural log restricts both the predictor variable and the response

variable to be greater than O. Hence, predicted EC signal response will be greater than O.

When performing a simple linear regression analysis, bo and b l are derived using the method of least squares. The

method of least squares selects values for and that minimize the sum of the squared distances between each

observed value and predicted value. In order to use this method, each data point must have an exact observed value.

However, due to inherent system noise and measurement capability, the exact EC signal response for a given flaw

size may not be known. When the response value is not known exactly, it is referred to as a censored data point.

In general, there are three types of censored data: right-censored, left-censored, and interval-censored.

Right-censored and left-censored data are typically encountered in EC inspection POD demonstration test data.

Right-censoring occurs when the signal generated by a large flaw exceeds the electronic limit of the EC system. For

example, suppose that the maximum amplitude that can be reported by an EC system is 25. If the measured signal

from a large flaw exceeds 25, all that is known is that a s some value to the "right" of 25. Left-censoring occurs

when the EC system cannot distinguish the signal generated by a small flaw from EC system and /or material noise.For example, suppose that the noise threshold is 1 division. That is, any signal below 1 division is indistinguishable

from noise. If the measured signal from a small flaw falls below 1, all that is really known is that a is within the

noise, or to the "left" of 1.

Censoring can be random or predetermined (Meeker, et aI. , 1998). In this context, it is predetermined by the

limitations of the EC instrument electronics. A predetermined saturation threshold level and noise threshold level,



251


denoted a sa t and a noise respectively, are used to identify censored data. Hence, a sat and a noise are both required inputs

in an a versus a analysis. The saturation threshold a sat is used to identify right-censored data points. That is, EC

signal responses that saturate the system are treated as right-censored data in the analysis. The noise threshold an oise

is used to identify left-censored data points. That is, any measured EC signal response following below a noise is

treated as a left-censored data point in the analysis. (Note that MIL-HDBK-1823 (1999) refers to a no ise as d hor the

inspection threshold.) An illustration of right-censored (i.e., electrical saturation) and left-censored data (i.e. , noise)

is shown in Figure 1. Note that in this example a sat = 25 and an oise = 1.

Measured EC Signal Response vs. Crack SizeIllustration of Right-censoring (Electrical Saturation) and Left-censoring (Noise)

I I I I I I I I I I

30 -T-------------------·--------i----·---T--r-·-·- i i- i rr rfiHiH 25

- ---------- -------- ------- --- - ~ - - - - + - ~ ~ - ~ ~ f - ~ f - ~ ~ + HI I I I I I I I I 111111 ....

15 -+-------------------1--------1----.. - - r - + - + - r 1 - r r r r r r H + ~I I I I I I I I I IIIIII1111

Z' I I I • . I I I I I I I I I I I I IIII10 - r- -------------------r------- t - -1--,---r --r- T- -r- i i r if i ~ ' r

.. I I I I I I I I I I I I I II I II I I ' I I I I I I I IIII1 I I

&: I I I I I I I I I I I I I I18. 5 - + - - - - - - - - - - - - - - - - - - - - - - - - - - t - - - - ~ - - - + - - t _ - + - + - r _ t - r t _ t _ t _ ~ r ~ - ! - ~:n : • I I I' I I I 1' 1 I I

'" I I I Data Typei • 0 Left-censored (Noise)

iii : • • Measured Response

&l : • 0 Right-censored (Electrical Saturation)II

¢I

CD

<p

§<0

•so

8<0

oa<0

Crack Size (a )

,I III

:

I I II I I I I I

I I I II I II I I I I II I I I I I

: : :: : :s 8 8 g g £ ~ § ~o c:i 0 c:i c:i 0 oooc:ic::X::ri:::Xi::

Figure 1: Illustration of right-censored data (electrical saturation) and left-censored data (noise).

Because censored data are commonly encountered in EC POD demonstration test data, a more advanced regression

technique for estimating and is needed in order to properly handle the censored data. Rather than using the

method of least squares, the a versus a analysis uses the method of maximum likelihood to estimate the model

coefficients. The method ofmaximum likelihood selects values for boand b l by maximizing the likelihood function.

The likelihood function is analogous to a probability distribution function (MIL-HDBK-1823, 1999). However, a

probability distribution function describes how likely a data point is given the distribution parameters, whereas the

likelihood function describes how likely the parameters are given the actual observed data. In other words, the EC

signal data are considered fixed (since they are already observed), and it is the model parameters bo and b l that can

vary. The method of maximum likelihood finds the values of bo and b l for which the value of the likelihoodfunction is the largest. That is, the method of maximum likelihood searches for values ofbo and b l that are the most

consistent with the actual observed data. In general, the method of maximum likelihood is viewed as a more

versatile method for fitting a model to data since it can be applied to a wide variety of data types, including censored

data, as well as to a wide variety of statistical models (Meeker et a1., 1998). (I t should be noted that when the values

obtained for bo and b l using the method of maximum likelihood are the same as those obtained using the method of

least squares when no censored data are present (Neter et a1., 1996).)

Developing a POD CurveA POD curve defines the crack size that the EC system is capable of detecting with a specified level of reliability.

The POD curve resulting from an aversus a analysis is based on the established relationship between aand a, the

defined decision threshold, and normal probability theory.

The established relationship between a and a defines the average EC signal response for a given crack size.However, the measured EC signal response for cracks of the same size will vary due to other physical characteristics

of the flaw, such as depth, and the inherent variability in the EC inspection process. As illustrated in Figure 2, the

variation around the average EC signal response for a given crack size is assumed to be normally distributed.



252


Measured EC Signal Response with a-hat vs. a Model At

• Measured EC Signal Response

WO -- Predicted EC Response (Maximum Lik elihood)

o Censored Data

III

I I I

W - T - - - - - - - - - - - - - - - - - - - ~ - - - - - - - - - - -

o

§a

III

•

8a

soa

, I I I I I I I I I I I II

I I I I I I I IIIII II

I I I I I I I IIIII II I I I I I I I I I I I I' i!

--1---+--+-,--;'-+-1-+-++++.1- ,-H+t-I I I I I I I , ' II I I II

I I I I I I I r IIIII II

I I I r I " I I I III I I I I II I I I II

I 'I I IIIII II

! ! ! ! !!!! !I 25I IIIII III IIIII II

I IIIII III I I I I I I IIIII II

- - ~ . - - i - - i - i - ~ - i ~ - r i i i i i i M i ~

s g g ! £ ~ ~ ~ ~ci c:i c:i ci ci ci ci c:i ci cicicX::i::::i:::i:

Crack Size (a )

Figure 2: Actual measured EC signal response is assumed to be normally distributed around the average.

In order to calculate POD, a decision threshold level is a required analysis input. The decision threshold, denotedadec. represents the value of a above which the EC signal response is interpreted as a find. POD is defined as the

fraction of flaws of a nominal size that are expected to be found (MIL-HDBK-l823, 1999). Probability equates to

the area under the normal curve. Hence, POD equates to the area under the normal curve above a dec . Suppose, for

example, that any measured EC signal above the noise threshold level is considered a find. That is , a dec = anoise = 1.

Hence, POD equates to the area under the normal curve above adec =1 as illustrated in Figure 3.

100

jillc

108.VI

;;c

'"iiuII I

Measured EC Signal Response with a-hat vs. a Model At

POD = area under the normal curve

: • Measured EC Signal Response

- , -- Predicted EC Response (Maximum Lik elihood)

: 0 Censored DataI

I I I I I I I I I I I I II


I I I I I I I IIIII II I I I I I I 111111 ' 11

- -1 - - - + - - + - - t - - t - + 1- t- ++++ ,- H +t-I I I I I I I I ' r i l l III I I I I I I J IIIII III I I I r I IIIII II

I I I 'I I IIIII III I ' I I IIIII III . I I IIIII II

r + - - - - - - - - - - - - - - - - - - - - ~ - - - - - - - - - - - - ~ - - - - ~ ~ ~ ~ ~ ~ ~III

I I

I I I I I I I I IIIII III I I I I I I I 1 1 1 1 1 II


- T - - - - - - - - - - - - - - - - - - - ~ - - - - - - - - - - - ~ 11111111111111.- 1 - - - - - T - ~ - ' - T ~ - r T T T T T r M T t

I II II III

I I I I I I I I I

I (AI) I b I 1b,l lhl l )n a

"=; 0 -Ii \ )a,

IIIIII~ ~ ~ ~ ~ - + - - - - - - r - - - - - - - ~ - - - - - T - - - - - - - - ~ ~ ~ ~ T T T T ~IIIII, ,

§co S ?<l

co 0 co g g § ~ ~ ~ ~N M M0 0 0 0 0 0

a a a a a a a a ci ci ci cicicicic:i:i:i:i:O

Crack Size (a l

Figure 3: POD is the area under the normal curve above Ii dec'

When performing an a versus a analysis, POD is calculated as



253


where <I> is the cumulative standard normal distribution function, ai is the expected mean EC signal response for a

given flaw size a;, bo and b l are the estimated model coefficients, and a- is the estimated standard deviation around

the expected mean response. Since ai is a function of crack size, POD is a function of crack size. Given the model

coefficients, a-, and a dee , POD can be calculated for each flaw size and plotted to establish a POD curve as

illustrated in Figure 4.

Probability of Detection (POD) Curve

Model : In(a-hat) = bO +b l* ln(a), bO and bl estimated using Maximum Ukelihood

1.0 TTl I I I

I I I I I I I

0.9I I I I I I Ii i 'j ' ' . , . - i iI I I I I I I II I I I I I I I

0.8 - - - - - - - - - - - - - - - ~ - - - - - - - - - ~ - - - - ~ - - - ~ - - ~ - - ~ - ~ - ~ - +I I I I I I I II I I I I I I I

0.7I I I I I I I I I

- - - - - - - - - - - - - - - - - -T - - - - - - - - - - -T - - - - - -T - - - -T - - -T - -T - -T -T - , -TI I I I I I I I II I. I I I I I I I

0.6 - - - - - - - - - - - - - - - - - -+- - - - - - - - - - - + - - - - - - + - - - - + - - - + - - + - - ~ - + - ~ - +I I I I I I I I I I

Q

0 0.50-

I I I I I I I I I I

---------------+------ . + - - - - - - ~ - - - - ~ - - - + - - ~ - - ~ - ~ - ~ - +I I I I I I I I I II I I I I I I I I I

0.4I I I I I I I I I IT ·T - - - - - -T - - - -T - - -T - -T - -T -T - , -TI I I I I I I I I II I I I I I I I I I

0.3 - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - + - - - - - - + - - - - + - - - + - - + - - ~ - + - ~ - +I I I I I I I I I II I I I I I I I I I

0.2_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ L _ _ _ _ _ _ _ _ _ _ _ L _ _ _ _ _ _ ____ ___ L _ _ _ _ _ ~ _ ~ _ L

I I I I I I I I I II I I I I I I I I I

0.1I I I I I I I I I I

- - - - - - - - - - -- - - - - - - r - - - -- - - - - - - r - - - -- -T - - - -T - - - r - -T - - r -T - i - rI I I I I I I

0.0 III lJ. ! .

8ci

8ci

§ci

8 8 ~ ~ ~c:i 0 c:i

Crack Size (a )

Figure 4: Resulting POD curve.

The resulting POD curve represents the typical (or nominal) POD curve and can be used, for example, to estimate

the flaw size that can be detected with 90% probability, denoted a90. This is accomplished by finding the flaw size

that intersects the curve where POD is 0.90. (Note: In order to estimate the flaw size that can be detected with 90%

probability and 95% confidence, denoted a90 /95 , a confidence bound is put on the established aversus a relationship

which can then be translated into a confidence bound on the POD curve in a manner similar to that which was used

to derive the typical POD curve.)

POD CURVE VALIDITYJust because the POD software produces a POD curve does not mean the POD curve is valid. Validity of the POD

curve depends on the identification of censored data, selection of the decision threshold aden

and the verification ofthe underlying aversus a model assumptions.

Identifying Censored Data in an Ii versus a AnalysisCensored data are commonly encountered in EC POD demonstration test data and contain valuable information

about the capability of the inspection system. Hence, values for a sa! and a naise are required inputs in an a versus a

analysis to identity right-censored and left-censored data respectively. Not specitying values for a Sal and a naise is

equivalent to treating the censored data as known values in the analysis , which can have a significant impact on the

aand a analysis and resulting POD curve.

For the simulated EC data in Figure 1, not specitying values for a Sal and a naise treats measured EC signals responses

that fall below the noise level (i.e., a na ;se = 1) as exact values and treats measured EC signals above the electrical

saturation as equal to 25 since the EC system cannot output values past the saturation threshold level. When no

censored data are present, a simple linear regression analysis can be performed. Figure 5 illustrates the effect onboth the aversus a model and POD calculations when censored data are treated as exact known values. The dashed

line represents the model resulting from a simple linear regression analysis, and the solid line represents the model

resulting from using the method of maximum likelihood to estimate the model coefficients.



254


100

.!!.

3!c

10

/!J.

iiic

'"iiu...

Measured EC Signal Response with Model Fit

Treating Censored Data Correctly vs. Incorrectly

Variable

o Censored Data

Predicted EC Signal Response (Method of Maximum Ukelihood)--- Predicted EC Resp (Simple Unear Reg., Method of Least Squares)

• Measured EC Signal Response

, , ,, , ,

~ - - - - - - - - - - - - - - - - - - - - + - - - - - - - - + - - - - ~ - - ~ ~ ~ ~ ~ ~ ~ ~ ~ 4 + ~ 25

~ L - - - - - - - - - - - - - - - - - - T - + - + - T ; - r + + + + + + r ~

§d

Crack Size (a l

Figure 5: Effect of treating censored data as known values on the Ii versus a model and POD calculations.

In Figure 5, it is clear there is a difference between the two models. The model resulting from a simple regressionanalysis appears to over-predict the EC signal response for small flaw sizes and under-predict large flaw sizes. In

this example, when the censored data are treated as failures, they playa large role in determining the position of the

best-fit line. This is because the method of least squares attempts to find values of and I that minimize the sum

of the squared distances between each observed value and the predicted value, including the exact values used for

the censored data. The method of maximum likelihood correctly assumes the right-censored data would have been a

higher value and the left-censored data would have been a lower value, which is a more accurate representation of

the data. In general, the regression line should run through the data, passing as close as possible to all points.

Hence, accurately accounting for censored data not only produces a better model fit with respect to the exact

measured EC signal response data but also with respect to the censored data. Figure 5 also illustrates the effect on

a90. In this example, a simple linear regression analysis results in an optimistic (i.e., smaller) a90. Since the best-fit

line has a significant impact on POD calculations, the POD curve is affected as well as shown in Figure 6.

Probability of Detection (POD) CurveTreating Censored Data Correctly vs. Incorrectly

1.0,--------------=--"'T----------"'T- - : : : ; ; ; ~ " " - - - - - - " ' - - - - " " ' ' ' ' ' ' ' ' ' ' ' ' ' ' ~: J, : : :

0.9 +---- - - - - - - - - - - - - -+- - - - - - - - ~ ~ - - - - - - - - - ---+-- --.--- -+ -+ -+1 0. 9I , I I I I I I I

0.8 - ------------------t- -----7-- ------+ - - - - t --t -- -t -t -i-t0.7 - - - - - - - - - - - - - - - r - 1- 1+ - - -T - - - -T - - - r - -T - - r -T - , - r

: , : :: : : : : :0.6 ---------------t- 1 t ----+----t---t--t--j-t-j-t

Q I I I I I I I I I I

o QS - - - - - - - - - - - - - - - - - - -+- - - - - - - - - +------t----f---+--f--}-f-4-+a. I I I I I I I I I I

0.4 - ------------------ _ __ ---- - -t ------+ - - - - - - - - - ~ - - ~ - ~ - ~ - t: I : :: : : : : :

0.3 -------------------+. '-------- --+------+----+---+--f--+-f--1-+., I I I .. I I I

0.2 .+ t ·t/ 1 11-- POD (Maximum Ukelihood)

., : : 1- - - POD (Least Squares)

O.O-L____ - - - " ~ ...: ; ; , , , = = ~ = = " " " ~ ~ ~ - - - - - , - - - - - - - - ' - - - - - , r - - . - - ."" joJ

a90 i f90

0.1

8ci

§ci

8 8 ~ ~ ~c:i 0 c:i

Crack Size (a l

Figure 6: Effect on POD curve of treating censored data as known values.



255


Incorrect specification of the values for a Sal and a no;se can also result m an inaccurate representation of the

relationship between EC signal response and crack size.

Selection of the Decision Threshold

The value selected for a dec is a critical input in an a versus a analysis. A POD curve cannot be produced if a dec is not

specified. If a dec is incorrectly specified, the result could be a seriously misleading POD curve.

By definition, a dec is associated with the flaw size that can be detected with 50% probability, denoted aso. Hence

changing a dec shifts the POD curve along the x-axis. Increasing the value of a dec> for example, will shift the PODcurve to the right. Figure 7 and Figure 8 illustrate the effect of increasing a dec on the POD calculation and resulting

POD curve. (Note that the saturation threshold and noise threshold remain the same in the analysis, e.g., aSal = 25

and a no;se = 1. All that is changing is the value of a dec . )

j.eiiic8.<II

i'"iiu...

100

10

8o

Measured EC Signal Response with a-hat vs. a Model Fit

• Measured EC Signal Response-- Predicted EC Response (Maximum Likelihood)

0 Censored Data

8 eJ eJ 8 g ~ ~ ~ ~o d d d d d d d d d ddc:X:i:i:i:i±5

Crack Size (a )

Figure 7: Effect on POD of increasing the decision threshold.

Probability of Detection (POD) Curve

1.0 . ' I I I

1-- POD (Maximum Likelihood) ' . : . . . : .0.9 -1___ Increased Decision Threshold - - , --,- ---r---T---T--r-r-T-,-T

I : : : : : : :oS -- - - - - - - - - - - - - - - - - -+ ---------- - - ( - - - - t--- -1" -- -t --t- - j -t j -+

0.7

0.6 ---------------+-------

J,I

I I I I I I II I I I I I I-T - - - -T - - -T - -T- -r -T - i - T

I I I I I I II I I I I I I- - - + - - - - + - - - + - - + - - ~ - + - 4 - +

Q I : : : : : : :o 0.5 - - - - - - - - - - - - - - - - - - - t - - - - - - - - - - ~ - - - - - - + - - - - + - - - + - - + - - f _ - + - i - tQ. , : : : : : : :

Q4 - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - t - - - - t - - - t - - t - - r - 1 - 1 - +Q3 - - - - - - - - - - - - - - - - - - - ~ - - - - - - - - - - - - - - - - - 1 - - - - 1 - - - t - - ~ - - ~ - 1 - ~ - ~

I I I I I I I I

0.2 ---------------+---- I - - - - ~ - - - - ~ - - - t - - ~ - - ~ - ~ - ~ - +I ::::: :

0.1 _ I ____ ____ - - - L - - ~ - - L - + - + - .I : : :

o .o -L - --""'''''''''''''''''''''''''''''''''i'''''''''''''''':::!!!!!.!!!.. . _ - - _ . _ - - . _ - - r - - , ~ . . . , J8 8 § 8888888o o o ci ci 0 0 ci ci ci

Crack Size (a)

Figure 8: Effect on POD curve of increasing the decision threshold.



256


Verification of the Underlying aand a Model AssumptionsThe validity of the POD curve is directly related to the validity of the aversus a model. The analysis results should

not be considered valid until the following underlying model assumptions are evaluated and shown to hold true: (1)

the error terms Ei are independent and follow a normal distribution with mean 0 and constant variance, and (2) the

relationship is in fact linear. Recall that the error term Ei is not an expression in the predictive model since in theory

it is expected to be 0 on average. However, it can be estimated as the difference between the observed value and the

model prediction. This difference is more commonly referred to as the residual. In general, analysis of the residuals

should be performed after any regression analysis to verify that the underlying model assumptions hold. This can be

accomplished by a few simple plots and/or more formal statistical methods. For example, the normality assumptioncan be assessed using a histogram and/or Anderson-Darling test for normality. The POD software may produce

analysis results even if the underlying model assumptions do not hold. If the assumptions do not hold, any

conclusions drawn about the EC system capability based on the analysis are likely to be erroneous and,

consequently , misleading (MIL-HDBK-1823, 1999). If the POD software does not perform or does not have the

capability to perform a residual analysis, the practitioner should still perform a residual analysis to verify that all

underlying model assumptions hold.

The validity of the a versus a model also depends whether the maximum likelihood convergence criterion is met.

When censored data are present, the likelihood equation from which the model coefficients are derived does not

have a closed solution. Hence, an iterative numerical search procedure is used to find values of bo and b l that

maximize the likelihood. The procedure iterates until a convergence criterion is met, at which point estimates of the

model coefficients are obtained from the last iteration. If no solution exists, then the procedure will not reach

convergence. Some POD software may produce estimates of bo and b even though the convergence criterion hasnot been met. However, they are likely to be erroneous and should not be used. A POD software package should

provide some indication (though it may be subtle) as to whether the convergence criterion was met. It is the

responsibility of the practitioner to verify that convergence was achieved.

SUMMARYStandard inputs to a versus a analysis that are required by POD software include the specified a versus a model

formulation, the saturation threshold level (asol ), the noise threshold level (ana;se), the decision threshold (adee), and

the POD demonstration test data. If anyone of these inputs is incorrectly specified or if the established relationship

between aand a is invalid, then the analysis and resulting POD curve (as well as any associated confidence bound)

is invalid. POD software performs the calculations based on user-defined inputs. The practitioner is responsible for

ensuring that the analysis inputs are correctly specified and that the underlying model assumptions hold. POD

software should not be treated as a "black box". It is critical that the practitioner have a basic understanding of the

underlying statistical methods and how the analysis inputs affect the results to ensure a valid analysis and avoiddrawing seriously misleading conclusions about the inspection capability of an EC system.

ACKNOWLEDGMENTSThis paper would not have been possible without the support and encouragement ofKon Haake. Thanks also to

Islay Rodriguez and David Raulerson for taking the time to educate a statistician in the physics of eddy current

inspection and providing opportunities for hands-on experience; to Tommie Watkins for imparting his knowledge of

POD methodology; and to Chuck Annis for happily entertaining all my questions.

REFERENCES1. MIL-HDBK-1823 (30 April 1999). Non-Destructive Evaluation System Reliability Assessment.

2. MIL-HDBK-1823 (28 February 2007 DRAFT). Non-Destructive Evaluation System Reliability Assessment.

3.Meeker, W.Q. and L.A. Escobar, Statistical Methods for Reliability Data, John Wiley and Sons, NewYork. 1998.

4. Neter, Kutner, Nachtsheim, & Wasserman, Applied Linear Statistical Models, 4th Ed., The McGraw

Hill Companies, Inc. 1996.

Documents

Probability of Detection AHi