Applying Change Point Detection in Vaccine Manufacturing Hesham Fahmy, Ph.D. Merck & Co., Inc. West Point, PA 19486 [email protected] Midwest Biopharmaceutical

Applying Change Point Detection in Vaccine Manufacturing

Hesham Fahmy, Ph.D.

Merck & Co., Inc.West Point, PA 19486

[email protected]

Midwest Biopharmaceutical Statistics Workshop (MBSW) - MAY 23 - 25, 2011

2

Outline

Definitions Detection methods CUSUM and EWMA estimators

Case studies CUSUM and EWMA SSE

Conclusions

3

Methods of Detection

Visual (Simple but Subjective) Raw data; run chart CUSUM chart EWMA chart

Analytical (Complicated but Objective) Change-Point estimators; i.e. CUSUM,

EWMA Mathematical Modeling; i.e. MLE, SSE

4

Types of Variation

Common Causes – natural (random) variations that are part of a stable processMachine vibrationMachine vibrationTemperature, humidity, electrical current fluctuationsTemperature, humidity, electrical current fluctuationsSlight variation in raw materialsSlight variation in raw materials

Special Causes – unnatural (non-random) variations that are not part of a stable processBatch of defective raw materialBatch of defective raw materialFaulty set-upFaulty set-upHuman errorHuman errorIncorrect recipeIncorrect recipe

5

Cumulative Sum Control Chart CUSUM: cumulative sum of deviations from

average

A bit more difficult to set up More difficult to understand Very effective when subgroup size n=1 Very good for detecting small shifts Change-point detection capability Less sensitive to autocorrelation

10,0max ttt CKyC

0 wabove/belo deviations daccumulate tC

6

Exponentially Weighted Moving Average

EWMA: weighted average of all observations

A bit more difficult to set upVery good for detecting small shiftsChange point detection capabilityLess sensitive to autocorrelation

“EWMA gives more weight to more recent observations and less weight to old observations.”

-1t

0j

-1 01 EyE tjt

jt

CUSUM Shewhart0 1

7

Process Shifts

Step Linear Nonlinear

Others

8

Process Model

in-control state

out-of-control state

t

Change point, "Unknown"

9

Change-Point Estimation Procedures

CUSUM Change-Point Estimation Procedure (Page 1954)

HCTtiHCCt TitCUSUM ,1,....1 0 ,0:̂

EWMA Change-Point Estimation Procedure (Nishina 1992)

TTiitEWMA UCLETtiUCLEEt ,1,....,1 ,:ˆ 00

0 wabove/belo deviations daccumulate tC

10

Example:

HCTtiHCCt TitCUSUM ,1,....1 0 ,0:̂

y

8

9

10

11

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

t

Process Data

10,0max ttt CKyC C

+

0

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

t

Upper CUSUM Chart

5H

5.0K

Most Recent Reintialization at t =22

Actual Change-Point= 20

11

Example: y

8

9

10

11

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

t

Process Data

TTiitEWMA UCLETtiUCLEEt ,1,....,1 ,:ˆ 00

11 ttt EyE 2.0

1000 E

E

9.5

10.0

10.5

11.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

t

EWMA Chart

UCL=11

LCL=9

tE at t =19Most Recent time

Actual Change-Point= 20

12

Real Example

29.0

30.0

31.0

32.0

33.0

Run

Cha

rt o

f Pro

dx

25 50 75 100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

Sample

Run Chart of Prodx

13

CUSUM vs. EWMA

-10

0

10

20

30

40

50

60

70

80

90

100

110

120

130

140

Cum

ulat

ive

Sum

of P

rodx

25 50 75 100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

Sample

CUSUM of Prodx

31.06

31.07

31.08

31.09

31.10

31.11

31.12

31.13

31.14

EW

MA

of P

rodx

25 75 125

175

225

275

325

375

425

Sample

Avg=31.0870

EWMA of Prodx, Lamda=0.001

14

5.0

2.0

29.5

30.0

30.5

31.0

31.5

32.0

32.5

EW

MA

of P

rodx

25 50 75 100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

Sample

Avg=31.087

EWMA of Prodx, Lamda= 0.2

29.5

30.0

30.5

31.0

31.5

32.0

32.5

EW

MA

of P

rodx

25 50 75 100

125

150

175

200

225

250

275

300

325

350

375

400

425

450

Sample

Avg=31.087

EWMA of Prodx, Lamda= 0.5

15

Simulated Case Studies

To test different methods' (control charts/analytical) capability for identifying change-points Case # 1: small shifts/drifts Case # 2: mirror image of case # 1 Case # 3: large shifts/drifts

16

Case #1

-6

-5

-4

-3

-2

-1

0

1

2

3

4

Run C

hart

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Run Chart of Y

-6

-5

-4

-3

-2

-1

0

1

2

3

4

Run C

hart

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Run Chart of Y

17

Case # 1

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

Y

1 2 3 4 5 6

0 25 50 75 100 125 150 175 200 225 250

Obs

Individual Measurement of Y

1 & 6: In-control process; N(0,1)

2 : -ve linear trend; 0.1 sigma

3 : Step shift; N(-3,1)

4 : Step shift; N(-1.5,1)

5 : +ve linear trend; 0.1 sigma

18

-100

-75

-50

-25

0

25

50

75

100

Cum

ula

tive S

um

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

CUSUM of Y (Alpha=0.0027)

CUSUM (V-Mask)

-6

-5

-4

-3

-2

-1

0

1

2

3

4

Run

Cha

rt of

Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Run Chart of Y

Detection Criterion: Slope Change

-3

-2

-1

0

1

2

3

Run

Cha

rt o

f Y

40 42 44 46 48 50 52 54 56 58 60

Sample

Samples 40 to 60

Diagnostic Sequence Plot

19

-100

-75

-50

-25

0

25

50

75

100

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample


CUSUM; 01.0

0027.0

Exact Profile-100

-75

-50

-25

0

25

50

75

100

Cum

ula

tive S

um

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample


20

CUSUM – Target Adjusted

-300

-200

-100

0C

umul

ativ

e S

um o

f Y

0 25 50 75 100 125 150 175 200 225 250

Sample

CUSUM of Y - Target= 0

-150

-100

-50

0

50

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150

Sample


-300

-250

-200

-150

-100

Cum

ula

tive S

um

of Y

50 75 100 125 150 175 200

Sample


21

EWMA;

-5

-4

-3

-2

-1

0

1

2

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=-1.11

EWMA of Y (Lamda=0.5)

5.0

-3

-2

-1

0

1

2

3

Run

Cha

rt o

f Y

40 42 44 46 48 50 52 54 56 58 60

Sample

Samples 40 to 60

Detection Criterion: Slope Change

22

EWMA;

-1.20

-1.18

-1.16

-1.14

-1.12

-1.10

-1.08

-1.06

-1.04

-1.02

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=-1.1080

EWMA of Y (Lamda=0.001)

001.0

-100

-75

-50

-25

0

25

50

75

100

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample


CUSUM

Same Profile

23

Case # 2 “Mirror Image”

-4

-3

-2

-1

0

1

2

3

4

5

6

7

Y

1 2 3 4 5 6

0 25 50 75 100 125 150 175 200 225 250

Sample




3 : Step shift; N(3,1)

4 : Step shift; N(1.5,1)


24

0

100

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

CUSUM of Y

CUSUM Chart

-100

-75

-50

-25

0

25

50

75

100

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample


Mirror Image

Case # 2

Case # 1

25

0

50

100

150

200

250

300

350

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample



26

0.5

0.6

0.7

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=0.992

EWMA of Y-Lamda=0.01

EWMA Chart; 01.0

27

EWMA Chart; 2.0

-1

0

1

2

3

4

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=0.99

EWMA of Y - Lamda=0.2

-1

0

1

2

3

4

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

µ0=0.00

EWMA of Y - Target= 0, Lamda= 0.2

28

EWMA Chart;

-2

-1

0

1

2

3

4

5

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=0.99

EWMA of Y-Lamda=0.5

5.0

29

Case # 3

-5

-3

-1

1

3

5

7

9

11

13

15

17

Y

1 2 3 4 5 6

0 25 50 75 100 125 150 175 200 225 250

Sample







30

CUSUM

-300

-200

-100

0

100

200

300

400

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

CUSUM of Y

31


0

1000

Cum

ulat

ive

Sum

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

CUSUM of Y - Target=0

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

Cum

ulat

ive

Sum

of Y

50 75 100 125 150 175 200

Sample


-500

-400

-300

-200

-100

0

100

200

300

400

500

600

Cum

ulat

ive

Sum

of Y

-25 0 25 50 75 100 125

Sample


32

EWMA;

-2

0

2

4

6

8

10

12

14

16

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=5.45

EWMA of Y, Lamda=0.5

5.0

33

EWMA;

-2

0

2

4

6

8

10

12

14

16

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=5.45

EWMA of Y, Lamda= 0.2

2.0

34

EWMA;

-2

-1

0

1

2

3

4

5

67

8

9

10

11

12

13

14

15

EW

MA

of Y

0 25 50 75 100 125 150 175 200 225 250

Sample

Avg=5.45

EWMA of Y, Lamda=0.1

1.0

35

Detection by SSE Pick a window of about 30 points

including the “investigated point” Fit a two-phase regression using all

possible change-points & calculate the SSE

Plot possible change-points vs. their SSEs

36

-3

-2

-1

0

1

2

3

Y

20

21

22

23

24

25

26

27

28

29

30

31

32

SS

E

30 35 40 45 50 55 60 65 70

Obs

Left Scale: Y

Right Scale: SSE

35 to 65 Window

90% CI

-1

0

1

2

3

4

5

6

Y

28

30

32

34

36

38

40

42

44

SS

E

55 60 65 70 75 80 85 90 95

Obs

Left Scale: Y

Right Scale: SSE

60 to 90 Window

-1

0

1

2

3

4

5

Y

10

11

12

13

14

15

16

SS

E

105 110 115 120 125 130 135 140 145

Obs

Left Scale: Y

Right Scale: SSE

110 to 140 Window

-1

0

1

2

3

4

5

Y

105 110 115 120 125 130 135 140 145

Obs

Overlay Plot

Examples from Case # 2

37

-5

-2.5

0

2.5

5

7.5

10

Y

10

20

30

40

50

60

70

SS

E

30 35 40 45 50 55 60 65 70

Obs

Left Scale: Y

Right Scale: SSE

35 to 65 Window

2.5

5

7.5

10

12.5

15

17.5

Y

20

30

40

50

60

70

80

90

100

SS

E

55 60 65 70 75 80 85 90 95

Obs

Left Scale: Y

Right Scale: SSE

60 to 90 Window

7

8

9

10

11

12

13

14

15

Y

10

15

20

25

SS

E

105 110 115 120 125 130 135 140 145

Obs

Left Scale: Y

Right Scale: SSE

110 to 140 Window

3

4

5

6

7

8

9

10

11

Y

15

20

25

30

35

40

45

50

55

60

SS

E

155 160 165 170 175 180 185 190 195

Obs

Left Scale: Y

Right Scale: SSE

160 to 190 Window

Examples from Case # 3

38

Conclusions

Change-point problem is general and can be applied in many applications such as 4 parameter logistic regression and degradation curves.

Another application in manufacturing processes includes detection of the change-point for process variance.

It is preferred to combine both analytical and visual techniques; in addition to process expertise; to get accurate results.

39

References

• Fahmy, H.M. and Elsayed, E.A., Drift Time Detection and Adjustment Procedures for Processes Subject to Linear Trend. Int. J. Prod. Research, 2006, 3257–3278.

• Montgomery, D. C., Int. to Stat. Quality Control, 1997, (John Wiley: NY).

• Nishina, K., A comparison of control charts from the viewpoint of change-point estimation. Qual. Reliabil. Eng. Int., 1992, 8, 537–541.

• Pignatiello, J.J. Jr. and Samuel, T.R., Estimation of the change point of a normal process mean in SPC applications. J. Qual. Tech., 2001, 33, 82–95.

• Samuel, T.R., Pignatiello Jr., J.J. and Calvin, J.A., Identifying the time of a step change with X control charts. Qual. Eng., 1998, 10, 521–527.

40

Acknowledgements

Lori Pfahler Julia O’Neill Jim Lucas

Documents

Applying Change Point Detection in Vaccine Manufacturing Hesham Fahmy, Ph.D. Merck & Co., Inc. West Point, PA 19486 [email protected] Midwest Biopharmaceutical