140425 2. Regression

7/21/2019 140425 2. Regression

1/60

MINITABUsers Guide 2

MEET MTB UGUIDE 1 SC QREFUGUIDE 2INDEXCONTENTS HOW TO USE


2Regression

Regression Overview, 2-2

Regression, 2-3

Stepwise Regression, 2-14

Best Subsets Regression, 2-20

Fitted Line Plot, 2-24

Residual Plots, 2-27

Logistic Regression Overview, 2-29

Binary Logistic Regression, 2-33

Ordinal Logistic Regression, 2-44

Nominal Logistic Regression, 2-51

See also,

Resistant Line, Chapter 8

Regression with Life Data, Chapter 16
http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://stedanal.pdf/http://stedanal.pdf/http://rsregrld.pdf/http://rsregrld.pdf/http://ug2index.pdf/http://ug2index.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://rsregrld.pdf/http://rsregrld.pdf/http://stedanal.pdf/http://stedanal.pdf/

7/21/2019 140425 2. Regression

2/60

Chapter 2 Regression Ove

2-2 MINITABUsers G



Regression OverviewRegression analysis is used to investigate and model the relationship between aresponse variable and one or more predictors. MINITABprovides various least-squareand logistic regression procedures.

Use least squares regression when your response variable is continuous. Use logistic regression when your response variable is categorical.

Both least squares and logistic regression methods estimate parameters in the model so thafit of the model is optimized. Least squares minimizes the sum of squared errors to obtainparameter estimates, whereas MINITABs logistic regression commands obtain maximumlikelihood estimates of the parameters. See Logistic Regression Overviewon page 2-29formore information about logistic regression.

Use the table below to assist in selecting a procedure:

Use to

response

type

estimati

method

Regression(page 2-3)

perform simple or multiple regression: fit a model,store regression statistics, examine residual diagnostics,generate point estimates, generate prediction andconfidence intervals, and perform lack-of-fit tests

continuous least squa

Stepwise(page 2-14)

perform stepwise, forward selection, or backwardelimination which add or remove variables from amodel in order to identify a useful subset of predictors


Best Subsets(page 2-20)

identify subsets the predictors based on the maximumR2criterion


Fitted Line Plot(page 2-24)

perform linear and polynomial regression with a singlepredictor and plot a regression line through the data;on the actual or log10scale


Residual Plots(page 2-27)

generate a set of residual plots to use for residualanalysis: normal score plot, a chart of individualresiduals, a histogram of residuals, and a plot of fitsversus residuals


Binary Logistic(page 2-33)

perform logistic regression on a response with only twopossible values, such as presence or absence

categorical maximumlikelihood

Ordinal Logistic(page 2-44)

perform logistic regression on a response with three ormore possible values that have a natural order, such asnone, mild, or severe


NominalLogistic(page 2-51)

perform logistic regression on a response with three ormore possible values that have no natural order, suchas sweet, salty, or sour

http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/

7/21/2019 140425 2. Regression

3/60

Regression Regr




RegressionYou can use Regression to perform simple and multiple regression using the metholeast squares. Use this procedure for fitting general least squares models, storingregression statistics, examining residual diagnostics, generating point estimates,generating prediction and confidence intervals, and performing lack-of-fit tests.

You can also use this command to fit polynomial regression models. However, if yowant to fit a polynomial regression model with a single predictor, you may find it madvantageous to use Fitted Line Plot(page 2-24).

Data

Enter response and predictor variables in numeric columns of equal length so that erow in your worksheet contains measurements on one observation or subject.

MINITABomits all observations that contain missing values in the response or in the

predictors, from calculations of the regression equation and the ANOVA table items

h To do a linear regression

1 Choose Stat Regression Regression.

2 In Response, enter the column containing the response (Y) variable.

3 In Predictors, enter the columns containing the predictor (X) variables.

4 If you like, use one or more of the options listed below, then click OK.

Options

Graphs subdialog box

draw five different residual plots for regular, standardized, or deleted residualsChoosing a residual typeon page 2-5. Available residual plots include a:
http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://ug2index.pdf/http://ug2index.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/

7/21/2019 140425 2. Regression

4/60

Chapter 2 Regr

2-4 MINITABUsers G



histogram.

normal probability plot.

plot of residuals versus the fitted values ( ).

plot of residuals versus data order. The row number for each data point is shoon the x-axis (for example, 1 2 3 4n).

separate plot for the residuals versus each specified column.

For a discussion, see Residual plotson page 2-6.

Results subdialog box

display the following in the Session window:

no output

the estimated regression equation, table of coefficients, s, R2, and the analysisvariance table

the default output, which includes the above output plus the sequential sumssquares and the fits and residuals of unusual observations

the default output, plus the full table of fits and residuals

Options subdialog box

perform weighted regressionsee Weighted regressionon page 2-6

exclude the intercept term from the regression by unchecking Fit InterceptseRegression through the originon page 2-7

display the variance inflation factor (VIFa measure of multicollinearity effect)associated with each predictorsee Variance inflation factoron page 2-7

display the Durbin-Watson statistic which detects autocorrelation in the residuals

see Detecting autocorrelation in residualson page 2-7

display the PRESS statistic and adjusted R-squared

perform a pure error lack-of-fit test for testing model adequacy when there arepredictor replicatessee Testing lack-of-fiton page 2-8

perform a data subsetting lack-of-fit test to test the model adequacysee Testinlack-of-fiton page 2-8

predict the response, confidence interval, and prediction interval for newobservationssee Prediction of new observationson page 2-9

Storage subdialog box

store the coefficients, fits, and regular, standardized, and deleted residualsseeChoosing a residual typeon page 2-5.

store the leverages, Cooks distances, and DFITS, for identifying outliersseeIdentifying outlierson page 2-9.

Y

7/21/2019 140425 2. Regression

5/60

Regression Regr




store the mean square error, the (XX)-1matrix, and the Rmatrix of the QRorCholesky decomposition. (The variance-covariance matrix of the coefficients isMSE*(XX)-1.) See Help for information on these matrices.

Residual analysis and regression diagnostics

Regression analysis usually does not end when a regression model has been fit. Youcan examine residual plots and other regression diagnostics to assess if the residualappear random and normally distributed. MINITABprovides a number of residual plthrough the Graphs subdialog box. Alternatively, after fits and residuals are stored, can use Stat Regression Residual Plotsto obtain four plots within a single grawindow.

MINITABalso produces regression diagnostics for identifying outliers or unusualobservations. These observations may have a significant influence upon the regressresults. See Identifying outlierson page 2-9.You might check unusual observations toif they are correct. If so, you can try to determine why they are unusual and consid

what effect they have on the regression equation. You might wish to examine howsensitive the regression results are to the outliers being present. Outliers can suggeinadequacies in the model or a need for additional information.

Choosing a residual type

You can calculate three types of residuals. Use the table below to help you choose wtype you would like to plot:

Residual type Choose when you want to Calculation

regular examine residuals in the original scale of the data response fit

standardized use a rule of thumb for identifying observations that arenot fit well by the model. A standardized residual greaterthan 2, in absolute value, might be considered to belarge. MINITABdisplays these observations in a table ofunusual observations, labeled with an R.

(residual) / (standardeviation of theresidual)

Studentized identify observations that are not fit well by the model.Removing observations can affect the variance estimateand also can affect parameter estimates. A large absoluteStudentized residual may indicate that including the

observation in the model increases the error variance orthat it has a large affect upon the parameter estimates, orboth.

(residual) / (standardeviation of theresidual). The ithstudentized residua

computed with theobservation remove

7/21/2019 140425 2. Regression

6/60

Chapter 2 Regr

2-6 MINITABUsers G



Residual plots

MINITABgenerates residual plots that you can use to examine the goodness of modeYou can choose the following residual plots:

Normal plot of residuals.The points in this plot should generally form a straightif the residuals are normally distributed. If the points on the plot depart from astraight line, the normality assumption may be invalid. To perform a statistical tesnormality, use Stat Basic Statistics Normality Test (page 1-43).

Histogram of residuals.This plot should resemble a normal (bell-shaped)distribution with a mean of zero. Substantial clusters of points away from zero mindicate that factors other than those in the model may be influencing your resu

Residuals versus fits.This plot should show a random pattern of residuals on bosides of 0. There should not be any recognizable patterns in the residual plot. Thfollowing may indicate error that is not random:

a series of increasing or decreasing points

a predominance of positive residuals, or a predominance of negative residu

patterns such as increasing residuals with increasing fits

Residuals versus order.This is a plot of all residuals in the order that the data wcollected and can be used to find non-random error, especially of time-relatedeffects.

Residuals versus other variables.This is a plot of all residuals versus anothervariable. Commonly, you might use a predictor or a variable left out of the modeand see if there is a pattern that you may wish to fit.

If certain residual values are of concern, you can brush your graph to identify thesevalues. See the BrushingGraphschapter in MINITABUsers Guide 1for more informat

Weighted regression

Weighted least squares regression is a method for dealing with observations that hanonconstant variances. If the variances are not constant, observations with

large variances should be given relatively small weights

small variances should be given relatively large weights

The usual choice of weights is the inverse of pure error variance in the response.

h To perform weighted regression

1 Choose Stat Regression Regression Options.

2 In Weights, enter the column containing the weights. The weights must be greathan or equal to zero. Click OKin each dialog box.
http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://stbasics.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://stbasics.pdf/

7/21/2019 140425 2. Regression

7/60

Regression Regr




If there are n observations in the data set, MINITABforms an n n matrix Wwith thcolumn of weights as its diagonal and zeros elsewhere. MINITABcalculates the regres

coefficients by (XWX)-1(XWY). This is equivalent to minimizing a weighted error of squares,

, where wiis the weight.

Regression through the origin

By default, the y-intercept term (also called the constant) is included in equation. TMINITABfits the model

However, if the response at X = 0 is naturally zero, a model without an intercept camake sense. If so, Uncheck Fit Intercept in the Options subdialog box, and the 0t

will be omitted. Thus, MINITABfits the model

Because it is difficult to interpret the R2when the constant is omitted, the R2is notprinted. If you wish to compare fits of models with and without intercepts, comparmean square errors and examine residual plots.

Variance inflation factor

The variance inflation factor (VIF) is used to detect whether one predictor has a stro

linear association with the remaining predictors (the presence of multicollinearityamong the predictors). VIF measures how much the variance of an estimated regrescoefficient increases if your predictors are correlated (multicollinear).

VIF = 1 indicates no relation; VIF > 1, otherwise. The largest VIF among all predictooften used as an indicator of severe multicollinearity. Montgomery and Peck [21]suggest that when VIF is greater than 5-10, then the regression coefficients are pooestimated. You should consider the options to break up the multicollinearity: collecadditional data, deleting predictors, using different predictors, or an alternative to square regression. For additional information, see [3], [21].

Detecting autocorrelation in residuals

In linear regression, it is assumed that the residuals are independent (that is, they anot autocorrelated) of each other. If the independence assumption were violated, smodel fitting results would be questionable. For example, positive correlation betw

wi y fit( )2[ ]

Y 0 1+ X1 2X2 kXk + + + +=

Y 1X1 2X2 kXk + + + +=

7/21/2019 140425 2. Regression

8/60

Chapter 2 Regr

2-8 MINITABUsers G



error terms tends to inflate the t-values for coefficients. Because of that, checking massumptions after fitting a model is an important part of regression analysis.

MINITABprovides two methods to check this assumption:

A graph of residuals versus data order (1 2 3 4n)can provide a means to visuainspect residuals for autocorrelation.

The Durbin-Watson statistic tests for the presence of autocorrelation in regressioresiduals by determining whether or not the correlation between two adjacent eterms is zero. The test is based upon an assumption that errors are generated byfirst-order autoregressive process. If there are missing observations, these are omfrom the calculations, and only the nonmissing observations are used.

To reach a conclusion from the test, you will need to compare the displayed statwith lower and upper bounds in a table. If D > upper bound, no correlation; if Dlower bound, positive correlation; if D is in between the two bounds, the test isinconclusive. For additional information, see [4], [22].

Testing lack-of-fit

MINITABprovides two lack-of-fit tests so you can determine whether or not theregression model adequately fits your data. The pure error lack-of-fit test requiresreplicates; the data subsetting lack-of-fit test does not require replicates.

Pure error lack-of-fit testIf your predictors contain replicates (repeated x valuwith one predictor or repeated combinations of x values with multiple predictorMINITABcan calculate a pure error test for lack-of-fit. The error term will bepartitioned into pure error (error within replicates) and a lack-of-fit error. The F-tcan be used to test if you have chosen an adequate regression model. For additi

information, see [9], [22], [29].

Data subsetting lack-of-fit testMINITABalso performs a lack-of-fit test that donot require replicates but involves subsetting the data, and attempts to identify nature of any lack-of-fit. This test is nonstandard, but it can provide informationabout the lack-of-fit relative to each variable. See [6] and Help for more informat

MINITABperforms 2k+1 hypothesis tests, where k is the number of predictors, anthen combines them using Bonferroni inequalities to give an overall significance lof 0.1. A message is printed out for each test for which there is evidence of lack-oFor each predictor, a curvature test and an interaction test are performed bycomparing the fit above and below the predictor mean using indicator variables

test can also be performed by fitting the model to the central portion of the dand then comparing the error sums of squares of that central data portion to theerror sums of squares of all the data.

7/21/2019 140425 2. Regression

9/60

Regression Regr




Prediction of new observations

If you have new predictor (X) values and you wish to know what the respowould be using the regression equation, then use Prediction intervals for observationsin the Options subdialog box. Enter constants or columnscontaining the new x values, one for each predictor. Columns must be of e

length. If you enter a constant and a column(s), MINITABwill assume that ywant predicted values for all combinations of constant and column values. can change the confidence level from the default 95%, and you can also stthe printed values: fits, standard errors of fits, confidence limits, and prediclimits. If you use prediction with weights, see Help for obtaining correct res

Identifying outliers

In addition to graphs, you can store three additional measures for the purpof identifying outliers, or unusual observations that can have a significantinfluence upon the regression. The measures are leverages, Cooks distance

and DFITS: Leveragesare the diagonals of the hat matrix, H= X(XX)-1X, where

the design matrix. Note that h idepends only on the predictors; it does ninvolve the response Y. Many people consider hito be large enough to mchecking if it is more than 2p/n or 3p/n, where p is the number of predic(including one for the constant). MINITABdisplays these in a table of unuobservations with high leverage. Those with leverage over 3p/n or 0.99,whichever is smallest, are marked with an X and those with leverage grethan 5p/n are marked with XX.

Cooks distancecombines leverages and Studentized residuals into oneoverall measure of how unusual the predictor values and response are fo

each observation. Large values signify unusual observations. GeometricaCooks distance is a measure of the distance between coefficients calculawith and without the ithobservation. Cook [7]and Weisberg [29]suggechecking observations with Cooks distance > F (.50, p, np), where F is value from an F-distribution.

DFITS, like Cooks distance, combines the leverage and the Studentizedresidual into one overall measure of how unusual an observation is. DFIT(also called DFFITS) is the difference between the fitted values calculatedwith and without the ithobservation, and scaled by stdev ( i). Belseley, and Welsch [3]suggest that observations with DFITS > 2 should bconsidered as unusual. See Help for more details on these measures.

e Example of performing a simple linear regression

You are a manufacturer who wishes to easily obtain a quality measure on aproduct, but the procedure is expensive. However, there is a quick-and-dirtway of doing the same thing that is much less expensive but also is slightlyprecise. You examine the relationship between the two scores to see if you

Y

p n

7/21/2019 140425 2. Regression

10/60

Chapter 2 Regr

2-10 MINITABUsers G



predict the desired score (Score2) from the score that is easy to obtain (Score1). Yoalso obtain a prediction interval for an observation with Score1 being 8.2.

1 Open the worksheet EXH_REGR.MTW.


3 In Response, enter Score2. In Predictors, enter Score1.

4 Click Options.

5 In Predict intervals for new observations, type 8.2. Click OKin each dialog bo

Sessionwindowoutput

Regression Analysis: Score2 versus Score1

The regression equation isScore2 = 1.12 + 0.218 Score1

Predictor Coef SE Coef T PConstant 1.1177 0.1093 10.23 0.000Score1 0.21767 0.01740 12.51 0.000

S = 0.1274 R-Sq = 95.7% R-Sq(adj) = 95.1%

Analysis of Variance

Source DF SS MS F PRegression 1 2.5419 2.5419 156.56 0.000Residual Error 7 0.1136 0.0162Total 8 2.6556

Unusual ObservationsObs Score1 Score2 Fit SE Fit Residual St Resid 9 7.50 2.5000 2.7502 0.0519 -0.2502 -2.15R

R denotes an observation with a large standardized residual

Predicted Values for New Observations

New Obs Fit SE Fit 95.0% CI 95.0% PI1 2.9026 0.0597 ( 2.7614, 3.0439) ( 2.5697, 3.2356)

Values of Predictors for New Observations

New Obs Score11 8.20

Interpreting the results

The regression procedure fits the model

Y 0 1X + +=

7/21/2019 140425 2. Regression

11/60

Regression Regr




where Y is the response, X is the predictor, 0and 1are the regression coefficients,is an error term having a normal distribution with mean of zero and standarddeviation . MINITABestimates 0by b0, 1by b1, and by s. The fitted equation is

where is called the predicted or fitted value. In this example, b0

is 1.12 and b1

is0.218.

Table of Coefficients.The first table in the output gives the estimated coefficients,and b1, along with their standard errors. In addition, a t-value that tests whether thnull hypothesis of the coefficient is equal to zero and the corresponding p-value isgiven. In this example, the p-values that are used to test whether the constant and sare equal to zero are printed as 0.000, because MINITABrounds these values to thredecimal points. These p-values are actually less than 0.0005. These values indicate there is sufficient evidence that the coefficients are not zero for likely Type I error ra(levels).

S = 0.1274.This is an estimate of , the estimated standard deviation about theregression line. Note that

R-Sq = 95.7%.This is R2, also called the coefficient of determination. Note that R2

Correlation (Y, )2. Also,

R2= (SS Regression) / (SS Total)

The R2value is the proportion of variability in the Y variable (in this example, Scoreaccounted for by the predictors (in this example, Score1).

R-Sq(adj) = 95.1%.This is R2adjusted for degrees of freedom. If a variable is adde

an equation, R2will get larger even if the added variable is of no real value. Tocompensate for this, MINITABalso prints R-Sq (adj), which is an approximately unbiestimate of the population R2that is calculated by the formula

converted to a percent, where p is the number of coefficients fit in the regressionequation (2 in our example). In the same notation, the usual R2is

Analysis of Variance.This table contains sums of squares (abbreviated SS). SSRegression is sometimes written SS (Regression | b0) and sometimes called SS ModeError is sometimes written as SS Residual, SSE, or RSS. MS Error is often written as MSS Total is the total sum of squares corrected for the mean. Use the analysis of varia

Y b0 b1X+=

Y

s2

MSError=

Y

R2

adj( ) 1 SS Error n p( )

SS Total n 1( )--------------------------------------------=

R2

1SS Error

SS Total--------------------=

7/21/2019 140425 2. Regression

12/60

Chapter 2 Regr

2-12 MINITABUsers G



table to assess the overall fit. The F-test is a test of the hypothesis H0: All regressioncoefficients, excepting 0, are zero.

Unusual Observations.Unusual observations are marked with an X if the predictounusual (large leverage), and they are marked with an R if the response is unusual (lstandardized residual). See Choosing a residual typeon page 2-5and Identifying outon page 2-9. The default is to print only unusual observations. You can choose to prfull table of fitted values by selecting this option in the Results subdialog box.

The Fit or fitted Y value is sometimes called predicted Y value or . SE Fit is the(estimated) standard error of the fitted value. St Resid is the standardized residual.

Predicted Values.The interval displayed under 95% CI is the confidence interval forpopulation mean of all responses (Score2) that correspond to the given value of thpredictor (Score1 = 8.2). The interval displayed under 95% PI is the prediction intefor an individual observation taken at Score1 = 8.2. The confidence interval isappropriate for the data used in the regression. If you have new observations, use tprediction interval. See Prediction of new observationson page 2-9.

Regression analysis would not be complete without examining residual patterns. Thfollowing multiple regression example and residual plots procedure provide additioinformation about regression analysis.

e Example of a multiple regression

As part of a test of solar thermal energy, you measure the total heat flux from homYou wish to examine whether total heat flux (Heatflux) can be predicted by insulatby the position of the focal points in the east, south, and north directions, and by ttime of day. Data are from [21], page 486. You found, using best subsets regressionpage 2-23, that the best two-predictor model included the variables North and Souand the best three-predictor added the variable East. You would like to evaluate the

three-predictor model using multiple regression.



3 In Response, enter Heatflux.

4 In Predictors, enter North South East. Click OK.

Y

7/21/2019 140425 2. Regression

13/60

Regression Regr




Sessionwindowoutput

Regression Analysis: HeatFlux versus North, South, East

The regression equation isHeatFlux = 389 - 24.1 North + 5.32 South + 2.12 East

Predictor Coef SE Coef T PConstant 389.17 66.09 5.89 0.000

North -24.132 1.869 -12.92 0.000South 5.3185 0.9629 5.52 0.000East 2.125 1.214 1.75 0.092

S = 8.598 R-Sq = 87.4% R-Sq(adj) = 85.9%


Source DF SS MS F PRegression 3 12833.9 4278.0 57.87 0.000Residual Error 25 1848.1 73.9Total 28 14681.9

Source DF Seq SS

North 1 10578.7South 1 2028.9East 1 226.3

Unusual ObservationsObs North HeatFlux Fit SE Fit Residual St Resid 4 17.5 230.70 210.20 5.03 20.50 2.94R22 17.6 254.50 237.16 4.24 17.34 2.32R

R denotes an observation with a large standardized residual


MINITABfits the regression model

where Y is the response, X1, X2, andX3are the predictors, 0, 1, 2, and3are theregression coefficients, and is an error term having a normal distribution with mea0 and standard deviation .

The multiple regression output is similar to the simple regression output, but it alsoincludes the sequential sums of squares. Sequential sums of squares differ fromt-statistics. T-statistics test the null hypothesis that each coefficient is zero, given thaother variables are present in the model. The sequential sums of squares are the un

sums of squares of the current variable, given the sums of squares of any previouslyentered variables.

For example, in the sequential sums of squares column of the Analysis of Variance tathe value for North (10578.7) is the sums of squares for North; the value for South(2028.9) is the unique sums of squares for South given the sums of squares for Nor

Y 0 1X1 2X2 3X3 e+ + + +=

7/21/2019 140425 2. Regression

14/60

Chapter 2 Stepwise Regr

2-14 MINITABUsers G



and the value for East (226.3) is the unique sums of squares for East given the sumssquares of North and South.

The first line in the sequential sums of squares table gives SS (b1| b0), or the reductin SS Error due to fitting the b1term (an equivalent is to use X1 as a predictor),assuming that you have already fit b0. The next line gives SS (b2| b0, b1), or thereduction in SS Error due to fitting the b2term, assuming that you have already fit terms b0and b1. The next line is SS (b3| b0, b1, b2), and so on. If you want a differsequence, say SS (b2| b0, b3), then repeat the regression procedure and enter X3 fthen X2. MINITABdoes not print p-values for the sequential sums of squares. Exceptthe last sequential sums of squares, the mean square error should not be used to tethe significance of these terms.

In this example, t-test p-values of less than 0.0005 indicate that there is significantevidence that the coefficients of variables North and South are not zero. The coefficof the variable East, however, has an t-test p-value of 0.092. If the evidence for thecoefficient not being zero appears insufficient and if it adds little to the prediction, may choose the more parsimonious model with predictors North and South. Make

decision only after examining the residuals. In the residual plots example on page 2you examine the residuals from the model with predictors North and South.(Alternatively, you could have used the graphs available in the Graphs subdialog bo

Stepwise RegressionStepwise regression removes and adds variables to the regression model for the purpof identifying a useful subset of the predictors. MINITABprovides three commonly uprocedures: standard stepwise regression (adds and removes variables), forward

selection (adds variables), and backward elimination (removes variables).

Data

Enter response and predictor variables in the worksheet in numeric columns of equlength so that each row in your worksheet contains measurements on one observaor subject. MINITABautomatically omits rows with missing values from the calculati

7/21/2019 140425 2. Regression

15/60

Stepwise Regression Regr




h To do a stepwise regression

1 Choose Stat Regression Stepwise.

2 In Response, enter the numeric column containing the response (Y) data.

3 In Predictors, enter the numeric columns containing the predictor (X) variables.


Options

Stepwise dialog box

By entering variables in Predictors to include in every model, you can designaset of predictor variables that cannot be removed from the model, even when thp-values are less than the Alpha to entervalue.

Method subdialog box

perform standard stepwise regression (adds and removes variables), forwardselection (adds variables), or backward elimination (removes variables).

when you choose the Stepwise method, you can enter a starting set of predictovariables in Enter. These variables areremoved if their p-values are greater than Alpha to entervalue. If you want keep variables in the model regardless of theirp-values, enter them in Predictors to include in every model in the main dialobox. See Stepwise regression (default)on page 2-16.

when you choose the Stepwise or Forward selection method, you can set the valu

the for entering a new variable in the model in Alpha to enter. See Stepwiseregression (default)and Forward selectionon page 2-17.

when you choose the Stepwise or Backward elimination method, you can set thevalue of for removing a variable from the model in Alpha to remove. See Stepregression (default)and Backward eliminationon page 2-17.

7/21/2019 140425 2. Regression

16/60


2-16 MINITABUsers G




display the next best alternate predictors up to the number requested. If a newpredictor is entered into the model, MINITABdisplays the predictor which was thsecond best choice, the third best choice, and so on, up to the requested numb

set the number of steps between pauses. See User interventionon page 2-17.

exclude the intercept term from the regression by unchecking Fit Intercept. SeeRegression through the originon page 2-7.

Method

MINITABprovides three commonly used procedures: standard stepwise regression (p2-16), forward selection (page 2-17), and backward elimination (page 2-17)

Stepwise regression (default)

The basic method of stepwise regression is to calculate an F-statistic for each variabthe model. Suppose the model contains X1,, Xj. Then the F-statistic for Xi is

with 1 and n j 1 degrees of freedom.

MINITABthen determines the corresponding p-value. If the p-value for any variable greater than Alpha to remove, the variable with the largest p-value is removed frothe model. The regression equation is calculated for this smaller model, the results printed, and the procedure proceeds to a new step.

If no variable can be removed, the procedure attempts to add a variable. An F-statiscalculated for each variable not yet in the model. Suppose the model, at this stagecontains X1, , Xp. Then the F-statistic for a new variable, Xj + 1is

MINITABthen determines the corresponding p-value. The variable with the smallestp-value is then added, provided its p-value is smaller than Alpha to enter. Adding variable is equivalent to choosing the variable with the largest partial correlation orchoosing the variable that most effectively reduces the error SS. The regression equais then calculated, results are displayed, and the procedure goes to a new step.

When no more variables can be entered into or removed from the model, the stepprocedure ends.

SSE X1 X i 1( ) X i 1+( ) Xj, ,,, ,[ ] SSE X1 Xj, ,[ ]MSE X1 Xj, ,[ ]

-------------------------------------------------------------------------------------------------------------------------------------------------

SSE X1 Xj, ,[ ] SSE X1 Xj Xj 1+,, ,[ ]MSE X1 Xj Xj 1+,, ,[ ]

-----------------------------------------------------------------------------------------------------------

7/21/2019 140425 2. Regression

17/60





Forward selection

This procedure adds variables to the model using the same method as the stepwiseprocedure. Once added, however, a variable is never removed. The forward selectiprocedure ends when none of the candidate variables have a p-value smaller thanAlpha to enter.

Backward elimination

This procedure starts with the model that contains all the predictors and then removariables, one at a time, using the same method as the stepwise procedure. No variahowever, can re-enter the model. The backward elimination procedure ends when nof the variables included the model have a p-value greater than Alpha to remove.

User intervention

Stepwise proceeds automatically by steps and then pauses. You can set the numbesteps between pauses in the Options subdialog box.

The number of steps can start at one with the default and maximum determined byoutput width. Set a smaller value if you wish to intervene more often. You must chEditor Enable Commands in order to intervene and use the procedure interactivIf you do not, the procedure will run to completion without pausing.

At the pause, MINITABdisplays a MORE? prompt. At this prompt, you can continue display of steps, terminate the procedure, or intervene by typing a subcommand.

To Type

display another page of steps (or untilno more predictors can enter or leavethe model)

YES

terminate the procedure NO

enter a set of variables ENTER CC

remove a set of variables REMOVE CC

force a set of variables to be in model FORCE CC

display the next best alternate predictors BEST K

set the number of steps between pauses STEPS K

change F to enter FENTER K

change F to remove FREMOVE K

change to enter AENTER K

change to remove AREMOVE K

7/21/2019 140425 2. Regression

18/60


2-18 MINITABUsers G



Use of variable selection procedures

Variable selection procedures can be a valuable tool in data analysis, particularly in early stages of building a model. At the same time, these procedures present certaidangers. Here are some considerations:

Since the procedures automatically snoop through many models, the modelselected may fit the data too well. That is, the procedure can look at manyvariables and select ones which, by pure chance, happen to fit well.

The three automatic procedures are heuristic algorithms, which often work very but which may not select the model with the highest R2value (for a given numbepredictors).

Automatic procedures cannot take into account special knowledge the analyst mhave about the data. Therefore, the model selected may not be the best from apractical point of view.

e Example of a stepwise regression

Students in an introductory statistics course participated in a simple experiment. Eastudent recorded his or her height, weight, gender, smoking preference, usual activlevel, and resting pulse. They all flipped coins, and those whose coins came up hearan in place for one minute. Afterward, the entire class recorded their pulses once m

You wish to find the best predictors for the second pulse rate.

1 Open the worksheet PULSE.MTW.

2 Pressc+Mto make the Session window active.

3 Check Editor Enable Commands.

4 Choose Stat Regression Stepwise.

5 In Response, enter Pulse2.

6 In Predictors, enterPulse1 RanWeight.

7 Click Options.

8 In Number of steps between pauses, enter 2. Click OKin each dialog box.

9 At the first More?prompt, type Yes.

10At the second More?prompt, type No.

7/21/2019 140425 2. Regression

19/60





Sessionwindowoutput

Stepwise Regression: Pulse2 versus Pulse1, Ran, ...

F-to-Enter: 4 F-to-Remove: 4

Response is Pulse2 on 6 predictors, with N = 92

Step 1 2Constant 10.28 44.48

Pulse1 0.957 0.912T-Value 7.42 9.74P-Value 0.000 0.000

Ran -19.1T-Value -9.05P-Value 0.000

S 13.5 9.82R-Sq 37.97 67.71R-Sq(adj) 37.28 66.98

C-p 103.2 13.5More? (Yes, No, Subcommand, or Help)

Step 3Constant 42.62

Pulse1 0.812T-Value 8.88P-Value 0.000

Ran -20.1T-Value -10.09

P-Value 0.000

Sex 7.8T-Value 3.74P-Value 0.000

S 9.18R-Sq 72.14R-Sq(adj) 71.19C-p 1.9More? (Yes, No, Subcommand, or Help)


This example uses six predictors. You requested that MINITABdo two steps of theautomatic stepwise procedure, display the results, and allow you to intervene.

The first page of output gives results for the first two steps. In step 1, the variablePulse1 entered the model; in step 2, the variable Ran entered. No variables wereremoved on either of the first two steps. For each model, MINITABdisplays the cons

7/21/2019 140425 2. Regression

20/60

Chapter 2 Best Subsets Regr

2-20 MINITABUsers G



term, the coefficient and its t-value for each variable in the model, S (square root oMSE), and R2.

Because you answered YES at the MORE? prompt, the automatic procedure continfor one more step, adding the variable Sex. At this point, no more variables could eor leave, so the automatic procedure stopped and again allowed you to intervene.Because you do not want to intervene, you typed NO.

The stepwise output is designed to present a concise summary of a number of fittemodels. If you want more information on any of the models, you can use the regresprocedure (page 2-3).

Best Subsets RegressionBest subsets regression generates regression models using the maximum R2criteriofirst examining all one-predictor regression models and then selecting the two mod

giving the largest R2. MINITABdisplays information on these models, examines alltwo-predictor models, selects the two models with the largest R2, and displaysinformation on these two models. This process continues until the model contains predictors.

Data

Enter response and predictor variables in the worksheet in numeric columns of equlength so that each row in your worksheet contains measurements on one unit orsubject. MINITABautomatically omits rows with missing values from all models.

When there are a large number of predictors, there are many subsets. There is a limthe number of free predictors (variables which may optionally enter the model). If yhave m free predictors and force q predictors to be in the model, then (m q) musless than or equal to 31.

In general, best subsets regression can take a long time when there are 15 or more predictors. The length of time, however, varies with the data set. If you have to anaa very large data set, consider forcing certain predictors to be in the model, therebdecreasing the number of free variables.

7/21/2019 140425 2. Regression

21/60

Best Subsets Regression Regr




h To do a best subsets regression

1 Choose Stat Regression Best Subsets.

2 In Response, enter the numeric column containing the response (Y) data.

3 In Free predictors, enter up to 31 numeric columns containing the predictor (X

variables. MINITABcan include any of these free predictors if they meet the specifcriteria (for example, greater than F to enter).


Options

Best Subsets Regression dialog box

force a set of predictors to be in the model by entering these variables in Predicin all models. These variables cannot be removed, even when their F-statistics a

less than the F to Entervalue.


display information for a specified number of free variables in the models. You cgive a minimum and maximum. For example, if you specify 3 as the minimum aas the maximum, MINITABwill display the best 3, 4, 5, and 6 variable models (thnumber does not include variables forced into the model).

display information about the best models of each variable number. You canspecify the number of best models to display. For example, if you specify 3,MINITABwill display the best, second best, and third best models for each numbevariables. You can enter a value from 1 to 5 (the default is 2).

exclude the intercept term from the regression by unchecking Fit InterceptseRegression through the originon page 2-7.

7/21/2019 140425 2. Regression

22/60

Chapter 2 Best Subsets Regr

2-22 MINITABUsers G



Using the best subsets regression procedure

The best subsets regression procedure can be used to select a group of likely modelfurther analysis. The general method is to select the smallest subset that fulfills certstatistical criteria. The reason that you would use a subset of variables rather than aset is because the subset model may actually estimate the regression coefficients an

predict future responses with smaller variance than the full model using all predicto[15].

The statistics R2, adjusted R2, Cp, and s (square root of MSE) are calculated by the bsubsets procedure and can be used as comparison criteria.

Typically, you would only consider subsets that provide the largest R2value. HowevR2always increases with the size of the subset. For example, the best 5-predictor mwill always have a higher R2than the best 4-predictor model. Therefore, R2is mostuseful when comparing models of the same size. When comparing models with thsame number of predictors, choosing the model with the highest R2is equivalent tchoosing the model with the smallest SSE.

Use adjusted R2and Cpto compare models with different numbers of predictors. Incase, choosing the model with the highest adjusted R2is equivalent to choosing thmodel with the smallest mean square error (MSE). If adjusted R2is negative (usuallywhen there is a large number of predictors and small R2) then MINITABsets the adjuR2to zero.

The Cpstatistic is given by the formula

where SSEpis SSE for the best model with p parameters (including the intercept, if

in the equation), and MSEmis the mean square error for the model with all mpredictors.

In general, look for models where Cpis small and close to p. If the model is adequa(i.e., fits the data well), then the expected value of Cpis approximately equal to p (number of parameters in the model). A small value of Cpindicates that the model irelatively precise (has small variance) in estimating the true regression coefficients apredicting future responses. This precision will not improve much by adding morepredictors. Models with considerable lack-of-fit have values of Cplarger than p. Seefor additional information on Cp.

Exercise caution when using variable selection procedures such as best subsets (and

stepwise regression). These procedures are automatic and therefore do not considepractical importance of any of the predictors. In addition, anytime you fit a model tdata, the goodness of the fit comes from two basic sources:

fitting the underlying structure of the data (a structure that will appear in other sets gathered in the same way)

fitting the peculiarities of the one particular data set you analyze

CpSSEp

MSEm

----------------- n 2p( )=

7/21/2019 140425 2. Regression

23/60

Best Subsets Regression Regr




Unfortunately, when you search through many models to find the best, as you dbest subsets regression, a good fit is often chosen largely for the second reason. Thare two ways that you can verify a model obtained by a variable selection procedur

You can

verify the model using a new set of data.

take the original data set and randomly divide it into two parts. Then use the variselection procedure on one part to select a model and verify the fit using the secpart.

e Example of best subsets regression

Total heat flux is measured as part of a solar thermal energy test. You wish to see htotal heat flux is predicted by other variables: insolation, the position of the focal poin the east, south, and north directions, and the time of day. Data are fromMontgomery and Peck [21], page 486.

1 Open the worksheet EXH_REGR.

2 Choose Stat Regression Best Subsets.


4 In Free Predictors, enterInsolation-Time. Click OK.

Sessionwindowoutput

Best Subsets Regression: HeatFlux versus Insolation, East, ...

Response is HeatFlux

Ins

o S Nl E o o Ta a u r it s t t m

Vars R-Sq R-Sq(adj) C-p S i t h h e

1 72.1 71.0 38.5 12.328 X1 39.4 37.1 112.7 18.154 X2 85.9 84.8 9.1 8.9321 X X2 82.0 80.6 17.8 10.076 X X3 87.4 85.9 7.6 8.5978 X X X3 86.5 84.9 9.7 8.9110 X X X4 89.1 87.3 5.8 8.1698 X X X X4 88.0 86.0 8.2 8.5550 X X X X5 89.9 87.7 6.0 8.0390 X X X X X


Each line of the output represents a different model. Vars is the number of variablespredictors in the model. The statistics R2, adjusted R2, Cp, and s are displayed next

7/21/2019 140425 2. Regression

24/60

Chapter 2 Fitted Lin

2-24 MINITABUsers G



and adjusted R2are converted to percentages). Predictors that are present in the mare indicated by an X.

In this example, the best one-predictor model uses North (R2 adj = 71.0) and thesecond-best one-predictor model uses Insolation (R2 adj = 37.1). Moving from the one-predictor model to the best two-predictor model increased the adjusted R2fro71.0 to 84.8. R2usually increases slightly as more predictors are added even when new predictors do not improve the model. The best two-predictor model might beconsidered as the minimum fit. The multiple regression example on page 2-12andresidual plots example on page 2-28indicate that adding the variable East does noimprove the fit of the model.

Fitted Line PlotThis procedure performs regression with linear and polynomial (second or third ord

terms, if requested, of a single predictor variable and plots a regression line throughdata, on the actual or log10scale. Polynomial regression is one method for modelincurvature in the relationship between a response variable (Y) and a predictor variab(X) by extending the simple linear regression model to include X2and X3as predic

Data

Enter your response and single predictor variables in the worksheet in numeric coluof equal length so that each row in your worksheet contains measurements on one or subject. MINITABautomatically omits rows with missing values from the calculati

h To do a fitted line plot

1 Choose Stat Regression Fitted Line Plot.

2 In Response(Y), enter the numeric column containing the response data.

3 In Predictor (X), enter the numeric column containing the predictor variable.


7/21/2019 140425 2. Regression

25/60

Fitted Line Plot Regr




Options

Fitted Line Plot dialog box

choose a linear (default), quadratic, or cubic regression model to automaticallyinclude all lower order terms. See Polynomial regression model choiceson page 2-


transform the y-variable by log10Y. You can also choose to display the y-scale in log10scale.

transform the x-variable by log10X. You can also choose to display the plot x scathe log10scale. If you use this option with polynomials of order greater than onethen the polynomial regression will be based on powers of the log10X.

display confidence bands and prediction bands about the regression line. You caalso change the confidence level from the default of 95%.

replace the default title with your own title.

Storage subdialog box

store the residuals, fits, and regression model coefficients (b0, b1, b2, up to b3dothe column, where biis the coefficient of the i

thpower of the predictor ortransformed predictor).

store the scaled residuals and scaled fits when using the y-variable transformatiolog10Y.

Polynomial regression model choices

You can fit the following linear, quadratic, or cubic regression models:

Another way of modeling curvature is to generate additional models by using the loof X and/or Y for linear, quadratic, and cubic models. In addition, taking the log10may be used to reduce right-skewness or nonconstant variance of residuals.

Model type Order Statistical model

linear first

quadratic second

cubic third

Y 0 1+ X +=

Y 0 1+ X 2X2 + +=

Y 0 1+ X 2X2 3X

3 + + +=

7/21/2019 140425 2. Regression

26/60

Chapter 2 Fitted Lin

2-26 MINITABUsers G



e Example of plotting a fitted regression line

You are studying the relationship between a particular machine setting and the amoof energy consumed. This relationship is known to have considerable curvature, anyou believe that a log transformation of the response variable will produce a moresymmetric error distribution. You choose to model the relationship between themachine setting and the amount of energy consumed with a quadratic model.


2 Choose Stat Regression Fitted Line Plot.

3 In Response (Y), enter EnergyConsumption.

4 In Predictor (X), enter MachineSetting.

5 Under Type of Regression Model, choose Quadratic.

6 Click Options. Check Logten of Y, Display logscale for Y variable, Displayconfidence bands, and Display prediction bands. Click OKin each dialog box

Sessionwindowoutput

Polynomial Regression Analysis: EnergyConsum versus MachineSetti

The regression equation islog(EnergyConsum) = 7.06962 - 0.698628 MachineSetti+ 0.0173974 MachineSetti**2

S = 0.167696 R-Sq = 93.1 % R-Sq(adj) = 91.1 %


Source DF SS MS F PRegression 2 2.65326 1.32663 47.1743 0.000Error 7 0.19685 0.02812

Total 9 2.85012

Source DF Seq SS F PLinear 1 0.03688 0.1049 0.754Quadratic 1 2.61638 93.0370 0.000

Graphwindowoutput

7/21/2019 140425 2. Regression

27/60

Residual Plots Regr





The quadratic model (p-value = 0.000, or actually p-value < 0.0005) appears to proa good fit to the data. The R2 indicates that machine setting accounts for 93.1% ofvariation in log10of the energy consumed. A visual inspection of the plot reveals ththe data are evenly spread about the regression line, implying no systematic lack-oThe lines labeled CI are the 95% confidence limits for the log10of energy consume

The lines labeled PI are the 95% prediction limits for new observations.

Residual PlotsYou can generate a set of plots to use for residual analysis by storing fits and residuusing another procedure, such as regression, and then using the Residual Plotsprocedure to produce a normal score plot, a chart of individual residuals, a histograresiduals, and a plot of fits versus residuals, all on the same graph.

Data

You must save a column of residuals and a column of fits from another MINITABprocedure. MINITABautomatically omits rows with missing values from the calculat

h To display the residual plots

1 Choose Stat Regression Residual Plots.

2 In Fits, enter the column containing stored fits.

3 In Residuals, enter the column containing the stored residuals.

4 If you like, use the option listed below, then click OK.

Options

You can replace the default title with your own title.

7/21/2019 140425 2. Regression

28/60

Chapter 2 Residua

2-28 MINITABUsers G



e Example of residual plots

You examine the residuals from the best two-predictor model of the best subsetsregression example on page 2-23. You determined in the multiple regression examon page 2-12that adding the third variable from the best three-predictor model mnot add appreciably to the fit. You now examine residual patterns from the besttwo-predictor model to further examine goodness-of-fit.

Step 1: Store the residuals and fits from a regression analysis




4 In Predictors, enter South North.

5 ClickStorage. Check Fitsand Standardized residuals.

6 Click OKin each dialog box.

Step 2: Generate the residual plots

1 Choose Stat Regression Residual Plots.

2 In Fits, enter the column containing the stored fits.

3 In Residuals, enter the column containing the stored residuals. Click OK.

Sessionwindowoutput

Residual Plots

TEST 1. One point more than 3.00 sigmas from center line.Test Failed at points: 22

TEST 2. 9 points in a row on same side of center line.Test Failed at points: 16

Graphwindowoutput

7/21/2019 140425 2. Regression

29/60

Logistic Regression Overview Regr





The residuals plots procedure generates four plots in one graph window. The normplot shows an approximately linear pattern that is consistent with a normal distributSimilarly, the histogram exhibits a pattern that is consistent with a sample from anormal distribution. However, the I Chart (a control chart of individual observationreveals that one point labeled with a 1the twenty-second valueis outside the th

sigma limits, and another point labeled with a 2 is flagged because it is the ninth inrow on the same side of the mean.

The plot of residuals versus fits shows that the fit tends to be better for higher predivalues. Investigation shows that the highest residual coincides with the highest valuthe variable East. Including East in the model and repeating the residual plots procedshowed that no points are flagged as unusual (not shown). The contribution to theby the variable East may warrant further investigation.

Logistic Regression OverviewBoth logistic regression and least squares regression investigate the relationshipbetween a response variable and one or more predictors. A practical difference betwthem is that logistic regression techniques are used with categoricalresponse variaband linear regression techniques are used with continuousresponse variables.

MINITABprovides three logistic regression procedures that you can use to assess therelationship between one or more predictor variables and a categorical responsevariable of the following types:

Both logistic and least squares regression methods estimate parameters in the model so thafit of the model is optimized. Least squares minimizes the sum of squared errors to obtainparameter estimates, whereas logistic regression obtains maximum likelihood estimates of parametersusing an iterative-reweighted least squares algorithm [19].

Tip You can identify points in the plots using the brushing capabilities. See the BrushingGraphschapter in MINITABUsers Guide 1.

Variabletype

Number ofcategories Characteristics Examples

Binary 2 two levels success, failureyes, no

Ordinal 3 or more natural orderingof the levels

none, mild, severefine, medium, coarse

Nominal 3 or more no natural orderingof the levels

blue, black, red, yellowsunny, rainy, cloudy

7/21/2019 140425 2. Regression

30/60

Chapter 2 Logistic Regression Ove

2-30 MINITABUsers G



How to specify the model terms

The logistic regression procedures can fit models with:

up to 9 factors and up to 50 covariates

crossed or nested factorssee Crossed vs. nested factorson page 3-19

covariates that are crossed with each other or with factors, or nested within facto

Model continuous predictors as covariates and categorical predictors as factors. Heare some examples. A is a factor and X is a covariate.

The model for logistic regression is a generalization of the model used in MINITABsgeneral linear model (GLM) procedure. Any model fit by GLM can also be fit by thelogistic regression procedures. For a discussion of specifying models in general, seeSpecifying the model termson page 3-20and Specifying reduced modelson page 3-2the logistic regression commands, MINITABassumes any variable in the model is acovariate unless the variable is specified as a factor. In contrast, GLM assumes that avariable in the model is a factor unless the variable is specified as a covariate.

Model restrictions

Logistic regression models in MINITABhave the restrictions as GLM models:

There must be enough data to estimate all the terms in your model, so that themodel is full rank. MINITABwill automatically determine if your model is full rank display a message. In most cases, eliminating some unimportant high orderinteractions in your model should solve your problem.

The model must be hierarchical. In a hierarchical model, if an interaction term isincluded, all lower order interactions and main effects that comprise the interactterm must appear in the model.

Model terms

A X AX fits a full model with a covariate crossed with a factor

A | X an alternative way to specify the previous model

A X XX fits a model with a covariate crossed with itself making a squared term

A X(A) fits a model with a covariate nested within a factor
http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://mmtoc.pdf/http://ug1tocnt.pdf/http://scqref13.pdf/http://ug2tocnt.pdf/http://ug2index.pdf/http://showbookmarks/http://mtb13doc.pdf/http://stanovar.pdf/http://stanovar.pdf/http://stanovar.pdf/http://stanovar.pdf/http://stanovar.pdf/http://stanovar.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://mtb13doc.pdf/http://showbookmarks/http://ug2index.pdf/http://ug1tocnt.pdf/http://mmtoc.pdf/http://ug2tocnt.pdf/http://scqref13.pdf/http://stanovar.pdf/http://stanovar.pdf/http://stanovar.pdf/

7/21/2019 140425 2. Regression

31/60

Logistic Regression Overview Regr




Reference levels for factors

MINITABneeds to assign one factor level as the reference level, meaning that theinterpretation of the estimated coefficients is relative to this level. MINITABdesignatthe reference level based on the data type:

For numeric factors, the reference level is the level with the leastnumeric value.

For date/time factors, the reference level is the level with the earliestdate/time.

For text factors, the reference level is the level that is firstin alphabetical order.

You can change the default reference level in the Options subdialog box.

For more information, Interpreting the parameter estimates relative to the event and treference levelson page 2-39.

Logistic regression creates a set of design variables for each factor in the model. If tare k levels, there will be k1 design variables and the reference level will be coded Here are two examples of the default coding scheme:

Reference event for the response variable

MINITABneeds to designate one of the response values as the reference event. MINITdefines the reference event based on the data type:

For numeric factors, the reference event is the greatestnumeric value.

For date/time factors, the reference event is the most recentdate/time.

For text factors, the reference event is the lastin alphabetical order.

You can change the default reference event in the Options subdialog box.

For more information, Interpreting the parameter estimates relative to the event and treference levelson page 2-39.

Note If you have defined a value order for a text factor, the default rule above does notapply. MINITABdesignates the first value in the defined order as the reference value. SeeOrdering Text Categoriesin the Manipulating Datachapter in MINITABUsers Guide 1.

Factor A with 4 levels

(1 2 3 4)

Factor B with 3 levels

(Temp Pressure Humidity)

reference

levelA1 A2 A3 reference

levelB1 B2

1 0 0 0 Humidity 0 0

2 1 0 0 Pressure 1 0

3 0 1 0 Temp 0 1

4 0 0 1

Note If you have defined a value order for a text factor, the default rule above does notapply. MINITABdesignates the last value in the defined order as the reference event. SeeOrdering Text Categoriesin the Manipulating Datachapter in MINITABUsers Guide 1.

7/21/2019 140425 2. Regression

32/60

Chapter 2 Logistic Regression Ove

2-32 MINITABUsers G



Worksheet structure

Data used for input to the logistic regression procedures may be arranged in twodifferent ways in your worksheet: as raw (categorical) data, or as frequency (collapsdata. For binary logistic regression, there are three additional ways to arrange the din your worksheet: as successes and trials, as successes and failures, or as failures an

trials. These ways are illustrated here for the same data.

The response entered as raw data or as frequency data

C1 C2 C3 C4

Response Factor Covar

0 1 12

1 1 12

1 1 12

.

.

....

.

.

.

1 1 12

0 2 12

1 2 12

.

.

....

.

.

.

1 2 12

.

.

....

.

.

.

C1 C2 C3 C4

Response Count Factor Cova

0 1 1 12

1 19 1 12

0 1 2 121 19 2 12

0 5 1 24

1 15 1 24

0 4 2 24

1 16 2 24

0 7 1 50

1 13 1 50

0 8 2 50

1 12 2 50

0 11 1 125

1 2 1 1250 9 2 125

1 11 2 125

0 19 1 200

1 1 1 200

0 18 2 200

1 2 2 200

Raw Data:one row for eachobservation

Frequency Data:one row for ecombination of factor and

1

19

1

19

7/21/2019 140425 2. Regression

33/60

Binary Logistic Regression Regr




The binary response entered as the number of successes, failures, or trials

Enter one row for each combination of factor and covariate.

Use caution when viewing large regression coefficients

If the absolute value of the regression coefficient is large, exercise caution in judgingp-value of the test. When the absolute regression coefficients are large, their calculastandard errors can be too large, leading you to conclude that they are not signific[13].If you have one or more large absolute regression coefficients for the factor(s) or covariate(s), the best test is to perform logistic regression both with and withoutthese terms and make a conclusion based upon the change in the log-likelihood.

If you do test the significance of model terms in this way, your test statistic will be2(log-likelihood from reduced model log-likelihood from full model). To comp

the p-value for this test, choose CalcProbability DistributionsChi-square. InDegrees of freedom, enter the model degrees of freedom from full model modedegrees of freedom from reduced model, where the model degrees of freedom arenumber of estimated coefficients. Check Input constant, and enter the test statistifrom above. Store the answer in a constant, say k1, and then calculate the p-value 1 k1 using CalcCalculator.

Binary Logistic RegressionUse binary logistic regression to perform logistic regression on a binary responsevariable. A binary variable only has two possible values, such as presence or absenceparticular disease. A model with one or more predictors is fit using aniterative-reweighted least squares algorithm to obtain maximum likelihood estimatethe parameters [19].

Successes and Trials

C1 C2 C3 C4

S T Factor Covar

19 20 1 1219 20 2 12

15 20 1 24

16 20 2 24

13 20 1 50

12 20 2 50

9 20 1 125

11 20 2 125

1 20 1 200

2 20 2 200

Successes and Failures

C1 C2 C3 C4

S F Factor Covar

19 1 1 1219 1 2 12

15 5 1 24

16 4 2 24

13 7 1 50

12 8 2 50

9 11 1 125

11 9 2 125

1 19 1 200

2 18 2 200

Failures and Trials

C1 C2 C3 C4

F T Factor Covar

1 20 1 121 20 2 12

5 20 1 24

4 20 2 24

7 20 1 50

8 20 2 50

11 20 1 125

9 20 2 125

19 20 1 200

18 20 2 200

7/21/2019 140425 2. Regression

34/60

Chapter 2 Binary Logistic Regr

2-34 MINITABUsers G



Binary logistic regression has also been used to classify observations into one of twocategories, and it may give fewer classification errors than discriminant analysis for scases [

Documents

140425 2. Regression