Algorithms and Model Spaces for Model-Robust Experiment Design fileIntroduction Work in JQT Extensions and Ongoing Work Preliminary Results from Ongoing Work Algorithms and Model Spaces

IntroductionWork in JQT

Extensions and Ongoing WorkPreliminary Results from Ongoing Work

Algorithms and Model Spaces for Model-RobustExperiment Design

Byran Smucker

Miami University, Oxford, OH

June 14, 2012IMS/ASA Spring Research Conference, Harvard

1 / 37



Outline

1 Introduction

2 Work in JQT

3 Extensions and Ongoing Work

4 Preliminary Results from Ongoing Work

2 / 37



Optimal Design

For standard least squares regression:

Cov(β̂) = σ2(X′X)−1.

Goal: Make (X′X)−1 small in some way. Examples:

Minimize |(X′X)−1| – D-optimality.

Minimize tr(X′X)−1 – A-optimality.

Minimize largest eigenvalue of (X′X)−1 – E-optimality.

3 / 37



Why Use Optimal Design?

Estimates can be made precisely

Parameter estimates;

Predictions;

The design can be tailored to the experimental scenario at hand.

Constrained design space

Mixture of continuous and categorical factors

Sample size constraints

4 / 37



A Drawback

The form of the model between response and factors must bespecified before design is constructed.

5 / 37



D-optimal vs. Model-robust

Instead of focusing on a single model (optimal design), find designthat is “good” for all models of interest, if possible.

D-optimal design, starting with arbitrary n-run design ξn:

ξnf→ X(ξn, f )

X′X−→ M(ξn, f )→ ξ∗n = argmax

ξn

|M(ξn, f )|

—–

Model-robust design with respect to a set of modelsF = (f1, f2, . . . , fr ):

ξnF→ {X1,X2, . . . ,Xr} → {M1,M2, . . . ,Mr} → ξ∗n = argmax

ξn

g [|M1|, |M2|, . . . , |Mr |]

6 / 37






ξnf→ X(ξn, f )


ξn

|M(ξn, f )|

—–



ξn

g [|M1|, |M2|, . . . , |Mr |]

7 / 37






ξnf→ X(ξn, f )


ξn

|M(ξn, f )|

—–



ξn

g [|M1|, |M2|, . . . , |Mr |]

8 / 37



Three Questions with the Set-of-Models Approach

1 What is the set of models, F?

2 What is the function, g?

3 What methods can be used to find the design?

9 / 37



Other Work

Set-of-Models Approach (not an exhaustive review)

Lauter (1974), Cook and Nachtsheim (1982), Dette (1990),Dette and Franke (2001) [continuous designs];

DuMouchel and Jones (1994); Jones et al. (2008); Jones andNachtsheim (2011) [Bayesian approach which implicitlyconsiders 2 models];

Li and Nachtsheim (2000), Loeppky et al. (2007), Jones et al.(2009), Smucker et al. (2012) [two-level designs]

Heredia-Langner et al. (2004) [genetic algorithm];

Tsai and Gilmour (2010) [approximation of As optimality]

10 / 37



Outline

1 Introduction

2 Work in JQT



11 / 37



Smucker, del Castillo, and Rosenberger (JQT, 2011)

Figure: Smucker et al. (2011), JQT

12 / 37



For JQT, Answer to Question 1

Q. What is the set of models, F?A. There were several.

1 A small set of models consisting of maximal models inspecified classes.

e.g. F = {f ′i (x), 1 ≤ i ≤ 3, x ∈ χ} where

f′

1(x) = (1, x1, x2)

f′

2(x) = (1, x1, x2, x1x2)

f′

3(x) = (1, x1, x2, x1x2, x21 , x

22 )

2 All possible submodels of the quadratic model.

3 All possible submodels of the quadratic model that respectstrong effect heredity.

13 / 37




Q. What is the function, g?A. Something that boiled down to

g(MF ) =∏F|Mf | (1)

This was motivated by a connection to multiresponse D-optimaldesign: Under certain assumptions, (1) is the D-criterion for amultiresponse experiment with responses corresponding to themodel in F .

14 / 37




Q. What method was used to find the design?A. A modified Fedorov exchange algorithm.

Proceeds run by run through a design and examines the effectof swapping out a run for one in a candidate list.

As design problems get larger, candidate list gets too large.

Coordinate exchange algorithm (Meyer and Nachtsheim 1995)solves this problem, but no published, effective algorithms formixture designs.

JMP has implemented a method to do this, I believe inventedby Chris Gotwalt and Laura Lancaster.

15 / 37



Mixture Experiment Example

Experiment reported by Frisbee and McGinity (1994).

Response: Glass transition temperature of a certain film.

Factors: A mixture of three nonionic surfactant factors.

Goal: Minimize the transition temperature.

Possible models:

f′1(x) = ({xi , i = 1, 2, 3}) (2)

f′2(x) = (f

′1, {xixj , i < j ≤ 3}) (3)

f′3(x) = (f

′2, {x1x2x3}) (4)

f′4(x) = (f

′1, {min(xi , xj), i < j ≤ 3}) (5)

f′5(x) = (f

′4, {min(x1, x2, x3)}) (6)

so that F = {f ′i (x), 1 ≤ i ≤ 5, x ∈ χ}.16 / 37



Result: Mixture Experiment with Disparate Models

What models were actually good fits for the data produced by thisexperiment?

f′fm(x) = (x1, x2, x3, x1x3, x2x3) (7)

f′rc(x) = (x1, x2, x3,min(x1, x3),min(x2, x3)) (8)

ModelDesign (2) (3) (4) (5) (6) (7) (8)Model Det. 19.81 5.91e-3 5.36e-6 0.569 2.78e-2 6.61e-2 1.46Robust Ef .745 .954 1 1 1 .812 .866Frisbee and Det. 8.25 1.22e-3 1.51e-6 .146 8.82e-3 2.30e-2 .588McGinity Ef .556 .733 .835 .797 .849 .658 .722Optimal (foreach model)

Det. 48 7.8e-3 5.36e-6 .569 2.73e-2 .188 3

Table: Objective Function Values and D-efficiencies for Example 4,n = 11, protecting against 5 Models.

17 / 37



Result: Mixture Experiment with Disparate Models

What models were actually good fits for the data produced by thisexperiment?

f′fm(x) = (x1, x2, x3, x1x3, x2x3) (7)

f′rc(x) = (x1, x2, x3,min(x1, x3),min(x2, x3)) (8)

ModelDesign (2) (3) (4) (5) (6) (7) (8)Model Det. 19.81 5.91e-3 5.36e-6 0.569 2.78e-2 6.61e-2 1.46Robust Ef .745 .954 1 1 1 .812 .866Frisbee and Det. 8.25 1.22e-3 1.51e-6 .146 8.82e-3 2.30e-2 .588McGinity Ef .556 .733 .835 .797 .849 .658 .722Optimal (foreach model)

Det. 48 7.8e-3 5.36e-6 .569 2.73e-2 .188 3

Table: Objective Function Values and D-efficiencies for Example 4,n = 11, protecting against 5 Models.

18 / 37



Designs

Figure: On the left, Frisbee and McGinity’s design; on the right, themodel-robust design

19 / 37



Larger Model Spaces

Typical scenario: The fitted model is not one of a few maximalmodels that we designed for.

So, pick a maximal models;

Design for the set of all possible submodels.

Issues:

May need to respect effect heredity.May result in model sets that are too large.

20 / 37



Three-Factor (non-mixture) Example

Design Mean StDev Min

3-MR 0.864 0.060 0.576

511-MR 0.877 0.069 0.705

Quadratic Only 0.872 0.069 0.690

Table: Measurements in terms of D-efficiencies with respect to all 511possible models.

3-MR is design robust for three models:

1 main effects only;2 main effects + two-factor interactions;3 full quadratic model.

511-MR: Design robust for all submodels of the full quadratic model.

21 / 37



Three-Factor Example, Respecting Effect Heredity

Design Mean StDev Min

3-MR 0.907 0.033 0.835

63-MR 0.907 0.034 0.833

Quadratic Only 0.881 0.068 0.740

Table: Measurements in terms of D-efficiencies with respect to all 63models, respecting effect heredity.

3-MR is design robust for three models:

1 main effects only;2 main effects + two-factor interactions;3 full quadratic model.

63-MR: Design robust for all submodels of the full quadratic modelwhich respect strong heredity.

22 / 37



Outline

1 Introduction

2 Work in JQT



23 / 37



Recall the Questions (in reverse order)

3. How to find the design?

2. What is the function, g?

1. What is the set of models, F?

24 / 37



Question 3

Instead of an exchange algorithm that requires a candidate list, Inow use coordinate exchange (Meyer and Nachtsheim 1995).

Considers exchanges coordinate by coordinate.

Faster.

No effective algorithm published that can incorporate mixtureconstraints (JMP can, however).

25 / 37



Question 2

As alternatives to g(MF ) =∏

f ∈F |Mf |, I have used

g1(MF ) =∏

f ∈F |Mf |1/pf

g2(MF ) = minf ∈F (|Mf | /|M∗f |)1/pf .

For what follows, we’ll assume g1.

26 / 37



Question 2

As alternatives to g(MF ) =∏

f ∈F |Mf |, I have used

g1(MF ) =∏

f ∈F |Mf |1/pf

g2(MF ) = minf ∈F (|Mf | /|M∗f |)1/pf .

For what follows, we’ll assume g1.

27 / 37



Question 1

Most realistic and useful model spaces have a major problem:

They are too big.

28 / 37



Question 1

Most realistic and useful model spaces have a major problem:

They are too big.

29 / 37



The Problem with Model Spaces

k p r (no heredity) r (weak heredity) r (strong heredity)

1 3 3 2 22 6 31 16 123 10 511 184 944 15 16,383 4,048 1,3365 21 1,048,575 175,208 38,618

Table: k is the number of factors; p is the number of parameters; r is thenumber of possible models is r .

30 / 37



Proposed Solution: Approximate Model Spaces

Select a small subset of the full model space.

Find a design robust for the small subset.

Evaluate this design with respect to the full model space.

31 / 37



Outline

1 Introduction

2 Work in JQT



32 / 37



Two-Factor Example: All Possible Models (No EnforcedHeredity)

Model space: All possible submodels of a maximal quadraticmodel.

D-EfficiencyModels Min Mean Time log(SP)MR-1 .743 .861 .008 34.565MR-8 .693 .874 .050 34.991MR-16 .693 .874 .110 34.991MR-31 .693 .874 .199 34.991

33 / 37



Four-Factor Example: All Possible Models (No EnforcedHierarchy)

Model space: All possible submodels of a maximal quadraticmodel.

D-EfficiencyModels Minimum Average Time log(SP)MR-1 .628 .870 .09 40895MR-16 .614 .877 1.02 41026MR-64 .614 .877 4.53 41026MR-128 .614 .877 8.80 41026MR-16383* .632 .873 1159.35 40960* Based on only two algorithm tries.

34 / 37



Work In-Progress

Applying this idea to other model spaces, e.g. model spacesfor two-level designs:

Main effects plus interaction (MEPI) (Li and Nachtsheim2000)Projective (Loeppky et al. 2007)Supersaturated (Jones et al. 2009)

35 / 37



Recap

Began with a basic approach and algorithm that relaxesoptimal design model assumption.

Impractical for large model spaces.

Discussed refinements in terms of

Model-robust criterion;Exchange algorithm;Approach to attacking large model spaces.

36 / 37



Acknowledgments

Co-authors/advisors: Enrique del Castillo and JimRosenberger.

Census Bureau for financial support.

37 / 37

Documents

Algorithms and Model Spaces for Model-Robust Experiment Design fileIntroduction Work in JQT Extensions and Ongoing Work Preliminary Results from Ongoing Work Algorithms and Model Spaces