Current Challenges and the Application of Computer Experiments in Industry

William Myers

The Procter & Gamble Company

DAE 2015: Design & Analysis of Experiments Conference – Cary, NC

2

Outline

The Business Value of Computer Experiments in Industry Brief Introduction to Computer Experiments Unique Challenges in Industry

o Design o Modeling

• Industry Example

3

Competitive Advantage in Industry

In a highly competitive and complex product development environment, reliance on computer experiments is crucial - often allow faster innovation and cheaper development costs

o When a series of design decisions must be made upfront prior to the

availability of a physical prototype o Physical experimentation can often be too time consuming and too

expensive or even impossible

“explore and develop virtually and confirm physically”

4

Simulates the Performance of a Package on Drop Testing

5

Simulates the Performance of a Package on a Standard Conveying Line

Baseline (original) Option 2, best option

6

Tube & Carton Packing Line

7

Computer Experiments Computer simulations are often computationally intensive and can take

too much time for the purpose of sensitivity analysis or product design optimization - a single computer code run can take from an hour to several days

The goal of computer experiments is to find an approximate model –

metamodel approach

8

General Approach

9

Modeling Computer Experiments

The Gaussian Process (GP) (ordinary kriging) model is a popular choice for use as a metamodel for computer experiments

They are flexible – fit a wide variety of surfaces GP models interpolate the data (adopted from the spatial statistics literature) GP models treat the deterministic output response as a realization of a random

stochastic process

( ) ( )Y x Z xµ= +

2( ) ~ (0, )Z x GP Rσ

2( , ) exp ( )ij k ik jkk

R X x xθ θ = − − ∑

10

Design Challenges

Design Region Constraints

Good Projection Properties Design with Qualitative Variables

Sample Size

11

Important Design Properties for Computer Experiments

Good Space-filling Properties o Spread points evenly throughout the design region (help to bound the bias and impact

predictive accuracy of surrogate model) o Assume no prior knowledge (model form) about the underlying system o Have the ability to fit a variety of models

Large number of levels for each input factor Good Projection Properties (minimize the average reciprocal distance)

o Not all the factors will be important – Effect Sparsity o Consider the projections of the design points onto its subspaces

o “Non-collapsing” – space-filling in lower dimensional projections o Poor projection properties will potentially produce quasi-replicates or sub-optimal coverage

X2

X1

X2

X1

12

Constrained Design Space

Examples o Geometric constraints on bottle and cap/closure o Constraint with respect to packaging component or equipment o Constraint with respect to different equipment parts

Example 1

Example 2

13

Constrained Design Space

Create space-filling designs in a constrained design space o Stinstra et al. (2003). Constrained Maximin Designs for Computer Experiments. Technometrics o Draguljic et al. (2012). Non-collapsing Designs for Bounded Non-rectangular Regions. Technometrics o Lekivetz et al. (2014). Fast Flexible Space-Filling Designs for Nonrectangular Regions. Quality & Reliability Engineering

International

Software is beginning to provide options

Still needs to be more evaluation of the different approaches

14

Projection Properties

MmLHDS have problems with projection properties in 2, 3, …, p-1 dimensions

Joseph, V.R., Gul, E. and Ba, S. Maximum Projection Designs for Computer Experiments (soon to appear in Biometrika)

MmLHDS MaxPro

15

Qualitative Variables in Computer Experiments

Typically when people think of computer experiments and space-filling

designs you think of continuous variables

Very common to have qualitative variables in computer experiments – At least at P&G!

o Examples o design/shape of equipment part o presence or absence of a piece of equipment o type or shape of packaging material o type of conveyor material

16

Qualitative Variables in Computer Experiments

Sliced Space-filling Designs (Qian et al. Biometrika 2009)

Sliced Latin-hypercube Designs (Qian – JASA 2012) o A sliced Latin hypercube design is a special Latin hypercube design that can be partitioned into slices of smaller

Latin hypercube designs based on the categorical variables. o R code available

Desirable properties

SLHD should have good space-filling properties for the whole design as well as for each slice Good projection properties

17

Qualitative Variables in Computer Experiments

Ba, S., Brenneman, W. A., and Myers, W. R. Optimal Sliced Latin Hypercube Designs (soon to appear in Technometrics) o First constructs the small LHD for each slice and then arrange their levels to form an overall SLHD o General algorithm which can search for the optimal SLHD efficiently o R code available

Overall

By Slice

18

Run Size for Computer Experiment “How large of a computer experiment do I need to run?” Loeppky, J. L., Sacks, J. and Welch, W. J. (2009). Choosing the Sample Size of a

Computer Experiment: A Practical Guide. Technometrics. o “10d runs will provide reasonable prediction accuracy for tractable functions and are sufficient to diagnose more

difficult problems”

In practice, there are situations where 10d is impractical Under effect sparsity

o only a subset, d0, of the input variables are important) then a lower bound on the number runs could be implemented

o (10 x d0) We often will implement this if there are budget constraints or if the computer simulations are very time consuming o Want to ensure good projection properties

19

Modeling Issues/Challenges

Stationary GP Model is inadequate

Qualitative Variables

20

When Stationary GP Model is Inadequate

Ba, S and Joseph, V. R. (2013). Composite Gaussian Process Models for Emulating Expensive Functions. Annuals of Applied Statistics o When design is sparse the CGP is more stable and more accurately approximates complex surface o A composite of two Gaussian processes 1) models the smooth global trend 2) models local details o JMP script

GP with nugget CGP

21

Industry Example

GP (with a nugget) CGP

LOO CV Error =7086 LOO CV Error =1363

Obs

erve

d

Obs

erve

d Jackknife Predicted Jackknife Predicted

22

Modeling Challenges with Qualitative Variables

As previously pointed out it is not uncommon to have qualitative variables in computer experiments

Simple approach – t different response curves for each qualitative level Can fit separate GP model to each response surface If the response surfaces are similar, then it would be “best” to borrow information from the other surfaces

References

o Qian et al. (2008) Gaussian Process Models for Computer Experiments with Qualitative and Quantitative Factors Technometrics

o Han et al. (2009) Prediction for Computer Experiments Having Quantitative and Qualitative Input Variables Technometrics

o Zhou et al. (2011) A Simple Approach to Emulation for Computer Models with Qualitative and Quantitative Factors Technometrics

o Zhang et al. (2015) Computer Experiments with Qualitative and Quantitative Variables: A Review and Reexamination Quality Engineering –puts it directly in the hands of the practitioner

23

Industry Example – Package Design

Quality of Model Fit

Sensitivity Analysis

24

Industry Example – Package Design

Understand Interactions to Gain Better Insight

25

Final Thought

There are still many interesting challenges in the area of computer experiments

from an industry perspective – both in the area of design and modeling

Also, many research opportunities!