Characterization of a Biomanufacturing Fermentation ...asa-qprc.org/2013/.../uploads/...a_Biomanufacturing_Process-QPRC2… · fitted models could capture all important effects even

Characterization of a Biomanufacturing

Fermentation Process Using Definitive

Screening Designs

Quality and Productivity Research Conference 2013

Niskayuna, NY, June 5–7, 2013

Dogan Ornek, Ph.D Philip J. Ramsey, Ph.D.

Senior Scientist North Haven Group &

Fermentation Development University of New Hampshire

Lonza Biologics Inc. Durham, NH 03824

97 South Street [email protected]

Hopkinton, MA 01748 [email protected]

[email protected]

mailto:[email protected]

mailto:[email protected]

Lonza’s Biomolecule Process Description

Flow diagram of the biomolecule production process. This study focuses on the Fermentation step.

4

Fermentation

Clean Genome E. coli

Unique Fed-batch HCD

Cell Lysis

Continuous

chemical lysis

Clarification & Concentration

Unique Filtration Train

Purification

2 column HIC based

(Lonza Proprietary)

Final

Product

Cell Harvest

Continuous

centrifugation

The Fermentation Experiments

The talk focuses on an experiment to optimize the biomolecule yield of the Fermentation step in a bio-process.

For the experiment K = 5 factors were identified:

1. pH (6.8, 7.2) = fermentation solution pH;

2. Dissolved Oxygen (%DO) (target values 20%, 40%);

3. Induction Temperature (39.5 C, 42.5 C) = Temperature at which the biomolecule production is induced in the E. Coli cells.

4. Induction OD600 (20, 40) = biomass at which the induction is initiated as measured by optical density at 600 nm.

5. Feed Rate (1.9, 3.5 mL/hr) = feed rate of a growth media containing 50% glycerol added to the fermentation solution when induction is initiated.

5

Definitive Screening Designs

A new class of screening designs have been developed by Jones and Nachtsheim (2011a, 2011b) and the authors refer to them as Definitive Screening Designs (DSD); the designs have subsequently been enhanced by Xiao (2012).

These DSDs offer an improvement over traditional screening designs in a number of ways:

For K factors DSDs require 2K+1 runs if K is even and 2K+3 if K is odd.

Main effects are orthogonal and independent of quadratic and two-way interaction effects.

No quadratic or two-way interaction effect is fully aliased.

It is possible to estimate main effects plus a combination of some two-factor interactions and some quadratic effects in a single experiment; assuming Effect Sparsity.

7


The two goals of the experiment were to characterize the fermentation step and to maximize the yield of the biomolecule.

Keep in mind that the goal was not necessarily to maximize the mass of the E. Coli community, but rather the goal is to maximize the yield of biomolecule X produced by the E. Coli.

It is possible to substantially increase the mass of a microbial community without maximizing biomolecule production.

The three responses of interest are:

1. Yield = biomolecule titer measured in units of mg/L;

2. OD600 = measure of biomass by optical density at 600 nm;

3. WCW = wet cell weight in units of g/L.

The latter two are a measure of the mass of the microbial community, while biomolecule yield is the most important response.

9


Given DSD experiments are new, it was decided to simultaneously perform a separate experiment utilizing a traditional fractional factorial design with the same factors and levels as the DSD.

The goal was to make a direct comparison of the DSD results to the results from a more traditional screening design.

The design selected was a 25-1 resolution V fractional factorial; smaller resolution screening designs are commonly used in bio-process characterization.

The fractional factorial design had 16 factorial runs plus 3 center points for a total of 19 runs.

Subsequently, the 25-1 design had to be augmented with axial runs to estimate nonlinear effects detected during analysis, which increased the total number of experimental runs to 31– a CCD essentially.

The DSD had a total of 15 runs including 4 center points.

10


Below is the 15 run DSD experiment.

11

Effect Sparsity

Before proceeding with the analysis of experimental results we need to make a quick diversion to discuss the concept of Effect Sparsity.

Effect Sparsity can be thought of as the better-known Pareto Principle applied to design of experiments.

For any given experiment the number of important effects is likely to be only some small subset, typically 20% to 30%, of the total number of potential effects.

Over decades of experimentation with physical systems at the macro level, the Effect Sparsity Principle has been well documented.

Without the Effect Sparsity Principle screening designs and to some degree design of experiments in general have little chance of successfully characterizing a physical system.

For a good discussion of Effect Sparsity see Goos and Jones (2011).

12

Effect Sparsity

The Effect Sparsity Principle does not imply that only 20% to 30% of the potential effects will have significantly small p-values.

Rather the Principle implies that only a small subset of the potential effects are sufficient to describe the behavior of the response.

Remember, statistically significant effects (based on p-values) are not necessarily important effects and vise verse.

In experiments with relatively small amounts of experimental error (small RMSE) a large number of effects appear significant even with relatively small impacts on the response.

With relatively large experimental error few or no effects appear to be significant even with apparent large impacts on the response.

For purposes of this discussion we use the term important effects.

13

Effect Sparsity

Effect Sparsity in an experiment can be thought of in two ways;

Absolute Sparsity = the number of active effects is about 20% to 30% of the total number of possible effects.

Relative Sparsity = the number of active effects is 50% (more or less) of the number of unique runs in the experiment.

If the number of active effects appears to exceed the 50% level by much, then it may become necessary to augment the DSD.

Although Effect Sparsity is established, it has some weaknesses:

It lacks a true operational definition;

It is largely anecdotal with no theoretical underpinning (center manifold reduction in nonlinear dynamic systems theory?).

Effect Sparsity is an area of statistics and applied math in need of serious research.

14

Model Building

Analyzing the DSD experimental results constitutes a special case of the supersaturated design problem.

Unlike most supersaturated design scenarios, a DSD allows the estimation of all main effects and all quadratic effects; or, a combination of main effects, quadratic effects, and two-way interactions assuming Effect Sparsity.

The DSD supports the estimation of a subset of the terms in a full quadratic model; possibly all terms could be estimated.

A full quadratic model is comprised of all main effects, all quadratic effects, all two-way interactions, and the intercept.

For K factors the number of terms N in the full quadratic model is:

15

2 1

2

K KN

Model Building

In analyzing the DSD results we use the following set of principles:

Hierarchy = lower order effects are more likely to be important than higher order effects; e.g., main effects are important more often than two-factor interactions.

Heredity = if a higher order effect is important, then the lower order parent terms of that effect are also important; e.g., if a two-factor interaction is important, then so are the two main effects involved in the interaction.

Thee model does not exist = only in simulations are there correct models (all models are wrong, some are useful, the late G.E.P. Box).

Subject matter expertise is eminent in model selection.

Parsimony = the simplest models that adequately predict the response well are considered best.

16

Model Building

In selecting models we generally have two competing issues:

Under fitting the model resulting in biased or inaccurate prediction;

Over fitting the model resulting in inflated prediction error.

Although the classic approach to the under and over fitting problem is to find a single, best compromise model, this is not necessarily an optimal strategy.

With the modern computing power and statistical algorithms available it is no longer necessary to search, by some means, to identify a best compromise model balancing under and over fitting.

Again, there is no correct model to be discovered, these only exist in simulation studies, we look for models that best describe the behavior of the physical system.

17

Model Building

Two widely accepted measures of fit for a model are:

AICc = bias corrected Akaike Information Criterion;

BIC = Bayesian Information Criterion;

We omit the mathematical details on AICc and BIC, see Burnham and Anderson (2004) or JMP 10 Help for dicussions.

Each statistic punishes under and over fitting, but in a different way so that they may not agree on the best model(s) – they often do not.

There is not agreement in the statistical community as to whether AICc or BIC criterion is preferred; it depends upon the application.

For both the AICc and BIC smaller values indicate better predictive models; the best balance of under and over fitting.

However, in near saturated cases both criteria may spike to artifically large negative values that do not indicate better models.

18

Model Building

Model Averaging is often confused with Prediction Averaging, however they are distinctly different approaches.

In Model Averaging, a large number of models are fit (e.g., All Possible Models) and the AICc or desired statistic is generated for each model.

A final or averaged model is generated by taking a weighted average of the coefficient estimates from a large subset of the fitted models.

Often 5% or so of the worst models in terms of AICc (or BIC) are deleted from the averaging.

A common weight is the exponential AICc, which has the general form (for a subset of M models to be used):

Model Averaging allows one to estimate the entire full quadratic model from the supersatured DSD.

21

( ) [ 0.5( min( ))] 1,..,i i jj

AICc Weight Exp AICc AICc j M

Model building strategies involving ensembles of models have an advantage for screening designs in that a single model is not required to resolve all important effects.

This is particularly important for DSDs given it is possible to estimate any of the terms in the full quadratic model and an ensemble of fitted models could capture all important effects even in the when Effect Sparsity does not hold for a single model.

Next, we demonstrate the model building strategies using the data from the DSD experiment on the fermentation process.

The JMP Pro® Version 10 software was used for the various analyses of the biomolecule experimental data presented in the talk.

The first step was to define a full quadratic model for K = 5 factors, the full model has 21 terms.

Model Building for the DSD Biomolecule Experiment

22

All Possible Models analysis was the performed with 11 as the largest possible model (including the intercept) and effect heredity was enforced.

For each model an AICc and BIC value was generated.

If the goal is to find a single best model or ensemble of models for Prediction Averaging, then the AICc and BIC values can be examined to select a more manageable subset of candidate models.

Given, there are a large number of models and that AICc and BIC do not necessarily indicate the same size models as best, a visualization can be useful to the analyst in selecting models for further consideration.

An approach used here is to create an overlay plot of AICc and BIC by model size.


23

From the overlay plot BIC indicates a model with 8 effects is best, while AICc suggests models in the 4 to 6 effect range.


24

Models with 4, 5, 6, 7, and 8 effects were examined further.

For sake of time within each size class of models, the model with the smallest RMSE was selected; other criteria such as PRESS could also be used.

None of the selected models exhibited significant lack of fit, however, substantial differences in Mean Square Press were evident for the models.

The 8 effect model: lowest MS Press, highest AICc, lowest BIC.


25

Model Size MS Press AICc BIC

8 18.21 199.42 151.50

6 50.81 175.01 157.68

7 62.34 185.34 155.75

5 69.06 174.46 163.42

4 70.25 174.29 168.04

Based upon MS Press and BIC the 8 term model appears to be the “best” , however for a single model Relative Effect Sparsity would not hold (too many terms) and this model may be over fit.

Rather than use the “best” model, another strategy is to use Prediction Averaging with all 5 models.

In this way no one model is required to resolve all important effects and the averaging reduces the potential effect of over fitting; if it exists.

A new Average Predictor was defined as the arithmetic average of the 5 model predictions; a simple weighting scheme.

Later the single best model and the Prediction Averaging model will be compared along with to other model building strategies.


26

Another strategy is to use Model Averaging based upon the All Possible Models analysis combined with the exponential AICc weights (other schemes could be used).

All terms in the full quadratic model are estimated from the ensemble of fitted models.


27

The Pareto Model is also based on the All Possible Models output.

Each model generated is parsed into separate model terms.

The AICc exponential weight is calculated for each model.

The weighted frequency for each possible model term is tallied.

Note, the AICc exponential weighting is needed to down weight model terms that occur frequently, but in weaker models.

Finally, a weighted Pareto Plot is created for all of the potential model effects appearing in the All Possible Models report.

The most frequent terms appearing in the Pareto Plot are then used to fit a model.

We therefore call this the Pareto Model given it is based entirely on the frequencies in the Pareto Plot – model heredity is enforced, however it occurred naturally in the analysis.


28

Below is the weighted Pareto Plot used to generate the Pareto Model.

Notice that pH only appears as a relatively minor effect.

The first 8 effects in the Plot were selected and that model was fit to the data.


29

Below is the Overlay Plot of AICc and BIC for the All Possible Models analysis for the Augmented FF (or CCD) experiment.

Model Building for the Augmented FF Experiment

30

Based on the AICc, BIC, RMSE, and MS Press, models with 6, 7, 8, 9, and 10 effects were considered for further analysis;

The 6 effects and 7 effects models exhibited significant lack of fit and were not considered further.

The model with 9 effects had the smallest MS Press and was selected as the best single model.

Subsequently the 8, 9, and 10 Effect models were used for Prediction Averaging.

Model Averaging was used to create a predictive model in the same manner as was done for the DSD experiment.

Finally, a Pareto Model was created in the same manner as was done for the DSD.


31

The Pareto Model results for to the Augmented FF experiment.

The Plot is shown to the right and once again pH appears to be a weak effect.

The first 8 Effects are used to fit a model.


32

Comparing the DSD to the Augmented FF Design

Three confirmation trials were performed about one month after both the DSD and Augmented FF experiments were completed.

The confirmation trials provide one way of comparing and contrasting the performance of the DSD and Augmented FF Design models on validation data.

The settings for the confirmation trials were determined by the scientists.

Confirmation trial settings and results are shown below.

33


The table below shows the performance of the Augmented FF design models and the DSD models based on the confirmation runs.

RASE is the root average squared error and AAE is the average absolute error; both calculated only on the confirmation trials.

The Pareto model from the DSD experiment was by far the best.

34

Design Model RASE AAE # Trials

Aug. FF Pred. Bio X n=9 81.60 81.63 3

Aug. FF Prediction Avg. 72.28 72.17 3

Aug. FF Model Avg. Pred. 49.56 42.72 3

Aug. FF Pareto Model Pred. 42.37 37.85 3

DSD Pred. Bio X n=8 18.74 14.12 3

DSD Prediction Avg. 25.71 22.57 3

DSD Model Avg. Pred. 51.84 46.31 3

DSD Pareto Model Pred. 5.59 4.88 3


The table below compares the predicted maximum yield for the equivalent models from the Aug. FF design and the DSD.

The next table below shows the predicted Yield for the 3 confirmation trials based on the Pareto Model from the Aug. FF and from the DSD.

35

Model Aug. FF DSD Difference

Best Single Model 660.57 522.68 137.89

Prediction Averaging 642.08 513.49 128.59

Model Averaging 576.95 482.31 94.64

Pareto Model 569.96 478.13 91.83

Trial Actual Aug. FF Aug. FF Diff.

DSD DSD Diff.

1 432.2 459.74 27.54 436.37 4.71

2 438.3 459.74 21.44 436.37 -1.93

3 368.0 303.44 -64.56 359.47 -8.53

Optimizing Yield for the Fermentation Step

Below are the Coefficient Estimates for the DSD Pareto Model.

This model would not have been selected based upon more traditional criteria; e.g., AICc, BIC, R2, MS Press, or p-values.

The results offer a cautionary note concerning the over-reliance on p-values as a method of model selection.

No mathematical link exists between p-values and information contained in a set of predictors as this example illustrates.

36

Optimizing Yield for the Fermentation Step

Below is the Desirability Report indicating the factor settings to achieve maximum Yield of Biomolecule – X.

The maximized average Yield is predicted to be 478.13 mg/L with a 95% confidence interval of [392.97, 563.29].

37

2011 North Haven Group 38

DSD Vs. RSM Design Sizes

Below is a plot of the required runs vs. number of factors for a CCD,

Doehlert Design, and Definitive Screening Design (DSD).

Summary and Conclusions

A study was conducted to compare the Definitive Screening Designs to more traditional designs in terms of characterizing and optimizing a fermentation step in a bio-manufacturing process.

The DSD results were as good and arguably better than the results from the much larger traditional augmented fractional factorial experiment (a CCD).

In terms of predicting the confirmation trial results, the DSD based models out performed the CCD based models.

Four modeling strategies were also evaluated: Find a best model; Prediction Averaging; Model Averaging; Pareto Plot analysis.

The best overall model strategy, based upon the prediction of the confirmation trial results, was the model derived from a Pareto Analysis of the All Possible Model results, from the DSD data; this method was also best among the CCD models.

39


This study supports the use of DSDs for characterization and optimization of bio-manufacturing processes given the DSDs are very economical as the number of factors K increases.

Study results also highlight a number of areas where statistical research and development is lacking; for example:

Effect Sparsity, needs an operational definition and theoretical justification.

Model Selection Strategy, too much focus on finding the magic statistic or method and not enough focus on “engineered strategies”; it is likely one method does not fit all scenarios.

How does one measure the information carried in set of potential predictors – p-values can and will lead to poor model selection?

Important or Active Factors – need a precise operational definition of these terms.

40


Open research questions also exist for model selection strategies.

For Prediction Averaging what criterion should be used to select models for the subset and what weighting scheme should be used?

For Model Averaging we have the same issue in terms what subset should be used and what weighting scheme should be used to average the coefficient estimates?

For the Pareto Plot method, again what is the best weighting criterion and how should one select models from the Pareto analysis?

For completeness: Adaptive LASSO, Least Angle Regression, and Reversible Jump Models (Winbugs) were also examined.

In all cases, these methods select badly under fit models; the LASSO is also very sensitive to validation method; a major issue for designed experiments.

41

References

Bickel, P. J., Ritov,Y. and Tsybakov,A. (2008) Simultaneous Analysis of Lasso and Dantzig selector. Ann. Statist.

Burnham, K P, and D R Anderson. Multimodal Inference: Understanding AIC and BIC in model selection. Sociological Methods and Research 33, Nr. 2 (2004): 261-304.

Erler, A., de Mas, N., Ramsey, P., Henderson, G. (2013). Efficient biological process characterization by definitive-screening designs: the formaldehyde treatment of a therapeutic protein as a case study. Biotechnology Letters, 35, 3.

FDA, CDER, CBER, and CVM (2011). Process Validation: General Principles and Practices. Guidance for industry.

Goos, P, and Jones, B. Optimal Design of Experiments: A Case Study Approach. John Wiley & Sons, LTD, 2011.

Hansen, B.E. (2008). Least-squares Forecast Averaging. Journal of Econometrics 146, p. 342350

42

References

Jones, B. and Nachtsheim, C. (2011a). Efficient Designs With Minimal Aliasing. Technometrics, 53, 1, 62 – 71.

Jones, B. and Nachtsheim, C. (2011b). A Class of Three-Level Designs for Definitive Screening in the Presence of Second-Order Effects. Journal of Quality Technology, 43, 1, 1 – 15.

Marley, C.J. and Woods, D.C. A Comparison of Design and Model Selection Methods for Supersaturated Experiments. Computational Statistics and Data Analysis 54 (2010) 3158 – 3167.

Xiao, L, Lin, D, and Bai, F. Constructing Definitive Screening Designs Using Conference Matrices. Journal of Quality Technology 44, Nr. 1 (2012): 2-8.

43

Documents

Characterization of a Biomanufacturing Fermentation ...asa-qprc.org/2013/.../uploads/...a_Biomanufacturing_Process-QPRC2… · fitted models could capture all important effects even