1

Fitting Regression Models

• It is usually interesting to build a model to relate the response to process variables for prediction, process optimization, and process control

• A regression model is a mathematical model which fits to a set of sample data to approximate the exact appropriate relation

• Low-order polynomials are widely used• There is a strong “interaction” between the design of

experiments and regression analysis• Regression analysis is often applied to unplanned

experiments

2

Linear Regression Models

• In general, the response variable y may be related to k regressor variables by a multiple linear (first order) regression model

y = o + 1x1 + 2x2 + + kxk + • Models of more complex forms can be analyzed

similarly. E.g.,

y = o + 1x1 + 2x2 + 12x1x2 + =>

y = o + 1x1 + 2x2 + 3x3 + • Any regression model that is linear in the parameters

( values) is a linear regression model, regardless of the shape of the response surface

• The variables have to be quantitative

3

Model Fitting – Estimating Parameters• The method of least squares is typically used• Assuming that the error term are uncorrelated random

variables• The data can be expressed as

• The model equation is

yi = o + 1xi1 + 2xi2 + + kxik + i

• Least square method chooses the ´s so that the sum of the squares of the errors i is minimized

4

Model Fitting – Estimating Parameters• The least squares function is

• The least squares estimators ( ) must satisfy and

or

2

1 11

2

n

i

k

jijjoi

n

ii xyL

j

0

o

L

0

j

L

n

iiik

n

ik

n

iiik

n

i

n

iiko

n

iii

n

iikik

n

iii

n

i

n

iio

n

ii

n

iikk

n

ii

n

iio

yxxxxxx

yxxxxxxx

yxxxn

ikik

i

11

2

122

1

21

1

11

11

1212

1

21

11

11122

111

ˆˆˆˆ

ˆˆˆˆ

ˆˆˆˆ

1

5

Model Fitting – Estimating Parameters• Or in the matrix notation

y = X + • The least squares estimators of are

• The fitted model and residuals are

• Example 10-1:• Response: viscosity of a polymer (y)• Variables: reaction temperature (x1), catalyst feed

rate (x2)• The model

y = o + 1x1 + 2x2 +

yXXX 1

ˆ Xy yye ˆ

6

21 58.862.708.1566ˆ xxy

7

8

9

Fitting Regression Models in Designed Experiments

• Example 10-2: regression analysis of a 23 factorial design• Response: yield of a

process• Variables: temperature,

pressure, and concentration• Single replicate with 4

center points

10

Fitting Regression Models in Designed Experiments

• A main effects only model

y = o + 1x1 + 2x2 + 3x3 +

321 125.1625.10625.5000.51ˆ xxxy

11

Fitting Regression Models in Designed Experiments

• The regression coefficients are exactly one-half of the effect estimates in a 2k design

• Because the factorial designs are orthogonal designs, the off-diagonal elements in X´X are zero, or X´X is diagonal

• Regression method is useful when the experiment (or data) is not perfect

• Regression analysis of data with missing observations. Example 10-3: assuming run 8 of the observations in Example 10-2 was missing. Fit the main effect model using the remaining observations

y = o + 1x1 + 2x2 + 3x3 +

12

Example 10-3

321 125.1625.10625.5000.51ˆ xxxy

321 25.175.1075.525.51ˆ xxxy

Original model

13

Fitting Regression Models in Designed Experiments

• Regression analysis of experiments with inaccurate factor levels• Example 10-4: assuming the process variables are not at their

exact assumed values in Example 10-2. Fit the main effect model y = o + 1x1 + 2x2 + 3x3 +

14

Example 10-4

321 125.1625.10625.5000.51ˆ xxxy

321 08.117.1042.536.50ˆ xxxy

Original model

15

Fitting Regression Models in Designed Experiments

• Regression analysis can be used to de-alias interactions in a fractional factorial using fewer than a full fold-over fraction in a resolution III design

• Example 10-5: consider Example 8-1, assume effects A, B, C, D, and AB+CD were large – de-alias AB+CD using fewer than 8 additional runs. Consider the model

y = o + 1x1 + 2x2 + 3x3 + 4x4 + 12x1x2 + 34x3x4 +

16

• The X matrix for the model is

• Adding one run from the alternate fraction to the original 8 runs, the X matrix becomes

17

Hypothesis Testing in Multiple Regression

Measuring the usefulness of the model• Test for significance of regression – determine if there is a

linear relationship between the response y and a subset of the regressor variables x1, x2, , xk.

• Testing hypothesis

Ho: 1 = 2 = = k = 0

H1: j 0 for at least one j• Analysis of variance

SST = SSR + SSE

n

y

yXSS

n

ii

R

2

1ˆ

yXyySSE

18

Hypothesis Testing in Multiple Regression

• If Fo exceeds F,k,n-k-1, the null hypothesis is rejected

E

R

E

Ro MS

MS

knSS

kSSF

)1/(

/

19

T

E

T

R

SS

SS

SS

SSR 12

)1()(

)1(1

)1/(

)/(1 22 R

pn

n

nSS

pnSSR

T

Eadj

20

Hypothesis Testing in Multiple Regression• Testing individual and group of coefficients – determine if one

or a group of regressor variables should be included in the model

• Testing hypothesis (for an individual regression coefficient)

Ho: j = 0

H1: j 0

if Ho is not rejected, then xj can be deleted from the model.

• Ho is rejected if |to| > tn

jj

jo

Ct

2ˆ

ˆ

21

• The contribution of a particular variable, or a group of variables can be quantified using sums of squares

• Confidence intervals on individual regression coefficients

• Example 10-7

jjpnjjjjpnj CtCt 2,2/

2,2/ ˆˆˆˆ

22

• Prediction of new response observations

• The future observation yo at a point (xo1, xo2, ,xok) with x’o =[1, xo1, xo2, ,xok]

• Regression model diagnostics

• Testing for lack of fit

• Sections 10-7, 8

))(1(ˆ)(ˆ

))(1(ˆ)(ˆ

12,2/

12,2/

oopno

ooopno

xXXxtxy

yxXXxtxy