1 1 Slide Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple...

1 1 Slide

Multiple Regression

Multiple Regression Model Least Squares Method Multiple Coefficient of

Determination Model Assumptions Testing for Significance Using the Estimated Regression

Equation for Estimation and Prediction Qualitative Independent Variables Residual Analysis

2 2 Slide

In this chapter we continue our study of regression analysis by considering situations involving two or more independent variables.

Multiple Regression

This subject area, called multiple regression analysis, enables us to consider more factors and thus obtain better estimates than are possible with simple linear regression.

3 3 Slide

The equation that describes how the dependent variable y is related to the independent variables x1, x2, . . . xp and an error term is:

Multiple Regression Model

y = b0 + b1x1 + b2x2 + . . . + bpxp + e

where:b0, b1, b2, . . . , bp are the parameters,

e is a random variable called the error term,p is the number of independent variables

4 4 Slide

The equation that describes how the mean value of y is related to x1, x2, . . . xp is called the multiple regression equation.

Multiple Regression Equation

E(y) = 0 + 1x1 + 2x2 + . . . + pxp

5 5 Slide

A simple random sample is used to compute sample statistics b0, b1, b2, . . . , bp that are used as the point estimators of the parameters b0, b1, b2, . . . , bp.

Estimated Multiple Regression Equation

^y = b0 + b1x1 + b2x2 + . . . + bpxp

The estimated multiple regression equation is:

6 6 Slide

Estimation Process

Multiple Regression ModelE(y) = 0 + 1x1 + 2x2 +. . .+ pxp + e

Multiple Regression EquationE(y) = 0 + 1x1 + 2x2 +. . .+ pxp

Unknown parameters areb0, b1, b2, . . . , bp

Sample Data:x1 x2 . . . xp y. . . .. . . .

0 1 1 2 2ˆ ... p py b b x b x b x

Estimated MultipleRegression Equation

Sample statistics are

b0, b1, b2, . . . , bp

provide estimates ofb0, b1, b2, . . . , bp

7 7 Slide

Least Squares Method

Least Squares Criterion

2ˆmin ( )i iy ywhere:

yi = observed value of the dependent

variable for the ith observation = estimated value of the dependent

variable for the ith observationiy

8 8 Slide

The years of experience, score on the aptitudetest, and corresponding annual salary ($1000s) for a sample of 20 programmers is shown on the nextslide.

Example: Programmer Salary Survey

A software firm collected data for a sampleof 20 computer programmers. A suggestionwas made that regression analysis couldbe used to determine if salary was relatedto the years of experience and the scoreon the firm’s programmer aptitude test.

9 9 Slide

47158100166

92105684633

781008682868475808391

88737581748779947089

23.734.335.838

22.223.13033

3826.636.231.62934

30.133.928.230

Exper. Score ScoreExper.Salary Salary

10 10 Slide

Suppose we believe that salary (y) isrelated to the years of experience (x1) and the

score onthe programmer aptitude test (x2) by the

following regression model:

where y = annual salary ($1000) x1 = years of experience x2 = score on programmer aptitude test

y = 0 + 1x1 + 2x2 +

11 11 Slide

Solving for the Estimates of 0, 1, 2

Input DataLeast Squares

Output

x1 x2 y

4 78 24

7 100 43

3 89 30

ComputerPackage

for SolvingMultiple

RegressionProblems

Download ProgramerSalary.xlsx

12 12 Slide

Excel Worksheet (showing partial data)

Note: Rows 10-21 are not shown.

A B C D1 Programmer Experience (yrs) Test Score Salary ($K)2 1 4 78 24.03 2 7 100 43.04 3 1 86 23.75 4 5 82 34.36 5 8 86 35.87 6 10 84 38.08 7 0 75 22.29 8 1 80 23.1

13 13 Slide

Using Excel’s Regression Tool

Performing the Regression Analysis

Step 3 Choose Regression from the list of Analysis Tools

Step 2 In the Analysis group, click Data AnalysisStep 1 Click the Data tab on the Ribbon

14 14 Slide

Excel’s Regression Dialog Box

15 15 Slide

Excel’s Regression Equation Output

Note: Columns F-I are not shown.

A B C D E3839 Coeffic. Std. Err. t Stat P-value40 Intercept 3.17394 6.15607 0.5156 0.6127941 Experience 1.4039 0.19857 7.0702 1.9E-0642 Test Score 0.25089 0.07735 3.2433 0.0047843

16 16 Slide

Estimated Regression Equation

SALARY = 3.174 + 1.404(EXPER) + 0.251(SCORE)SALARY = 3.174 + 1.404(EXPER) + 0.251(SCORE)

Note: Predicted salary will be in thousands of dollars.

17 17 Slide

Interpreting the Coefficients

In multiple regression analysis, we interpret each

regression coefficient as follows: bi represents an estimate of the change in y corresponding to a 1-unit increase in xi when all other independent variables are held constant.

18 18 Slide

Salary is expected to increase by $1,404 for each additional year of experience (when the variablescore on programmer attitude test is held constant).

b1 = 1. 404b1 = 1. 404

19 19 Slide

Salary is expected to increase by $251 for each additional point scored on the programmer aptitudetest (when the variable years of experience is heldconstant).

b2 = 0.251b2 = 0.251

20 20 Slide

Multiple Coefficient of Determination

Relationship Among SST, SSR, SSE

where: SST = total sum of squares SSR = sum of squares due to regression SSE = sum of squares due to error

SST = SSR + SSE

2( )iy y 2ˆ( )iy y 2ˆ( )i iy y

21 21 Slide

Excel’s ANOVA Output

A B C D E F3233 ANOVA34 df SS MS F Significance F35 Regression 2 500.3285 250.1643 42.76013 2.32774E-0736 Residual 17 99.45697 5.8504137 Total 19 599.785538

SSRSST

22 22 Slide

R2 = 500.3285/599.7855 = .83418

R2 = SSR/SST

23 23 Slide

Adjusted Multiple Coefficientof Determination

Adding independent variables, even ones that are not statistically significant, causes the prediction errors to become smaller, thus reducing the sum of squares due to error, SSE.

Because SSR = SST – SSE, when SSE becomes smaller, SSR becomes larger, causing R2 = SSR/SST to increase.

The adjusted multiple coefficient of determination compensates for the number of independent variables in the model.

24 24 Slide

n pa2 21 1

( )R R

2 21 11

2 20 11 (1 .834179) .814671

20 2 1aR

where: n is the number of observations p is the number of independent variables

After adjusting the number of independent variables,we see 81.46% of variability in salary can be explainedby the regression.

25 25 Slide

Excel’s Regression Statistics

A B C23 24 SUMMARY OUTPUT2526 Regression Statistics27 Multiple R 0.91333405928 R Square 0.83417910329 Adjusted R Square 0.81467076230 Standard Error 2.41876207631 Observations 2032

𝑀𝑢𝑙𝑡𝑖𝑝𝑙𝑒𝑅=√𝑅2

𝑆𝑡𝑎𝑛𝑑𝑎𝑟𝑑 𝐸𝑟𝑟𝑜𝑟=√𝑀𝑆𝐸

26 26 Slide

The variance of , denoted by 2, is the same for all values of the independent variables. The variance of , denoted by 2, is the same for all values of the independent variables.

The error is a normally distributed random variable reflecting the deviation between the y value and the expected value of y given by 0 + 1x1 + 2x2 + . . + pxp.

Assumptions About the Error Term

The error is a random variable with mean of zero. The error is a random variable with mean of zero.

The values of are independent. The values of are independent.

27 27 Slide

In simple linear regression, the F and t tests provide the same conclusion. In simple linear regression, the F and t tests provide the same conclusion.

Testing for Significance

In multiple regression, the F and t tests have different purposes. In multiple regression, the F and t tests have different purposes.

28 28 Slide

Testing for Significance: F Test

The F test is referred to as the test for overall significance. The F test is referred to as the test for overall significance.

The F test is used to determine whether a significant relationship exists between the dependent variable and the set of all the independent variables.

29 29 Slide

A separate t test is conducted for each of the independent variables in the model. A separate t test is conducted for each of the independent variables in the model.

If the F test shows an overall significance, the t test is used to determine whether each of the individual independent variables is significant.

Testing for Significance: t Test

We refer to each of these t tests as a test for individual significance. We refer to each of these t tests as a test for individual significance.

30 30 Slide

Testing for Significance: F Test

Hypotheses

Rejection Rule

Test Statistics

H0: 1 = 2 = . . . = p = 0

Ha: One or more of the parameters

is not equal to zero.

F = MSR/MSE

Reject H0 if p-value < a or if F > F,

where F is based on an F distribution

with p d.f. in the numerator andn - p - 1 d.f. in the denominator.

p is the number of independent variables

31 31 Slide

F Test for Overall Significance

Hypotheses H0: 1 = 2 = 0

Ha: One or both of the parameters

is not equal to zero.

Rejection Rule For = .05 and d.f. = 2, 17; F.05 = 3.59

p-value approach: Reject H0 if p-value < .05

Critical value approach: Reject H0 F > 3.59

Critical value = FINV(0.05,2,17)

32 32 Slide

p-value used to test for

overall significance

p-value = FDIST(42.76013,2,17)=2.32774E-07

numerator’s DF

denominator’s DF

33 33 Slide

Test Statistics F = MSR/MSE = 250.16/5.85 = 42.76

Conclusion p-value < .05, so we can reject H0.(Also, F = 42.76 > 3.59)

The regression includes at least onesignificant independent variable.

34 34 Slide

Testing for Significance: t Test

Hypotheses

Rejection Rule

Test Statistics

where t is based on a t distribution

with n - p - 1 degrees of freedom.p is the number of independent variables.

0 : 0iH

: 0a iH

p-value approach: Reject H0 if p-value < aCritical value approach: Reject H0 if t < -tor t > t

35 35 Slide

t Test for Significanceof Individual Parameters

Hypotheses

Rejection RuleFor = .05 and d.f. = 17, t.025 = 2.11

d.f. = n-p-1 = 20-2-1 = 17

p-value approach: Reject H0 if p-value < aCritical value approach: Reject H0 if t < -tor t > t

Critical values are t.025 = TINV(0.025*2,17)=2.11 and - t.025 =-2.11

36 36 Slide

t statistic and p-value used to test for the individual significance of

“Experience”p-value for Experience = TDIST(7.07,17,2)=1.9E-06

37 37 Slide

t statistic and p-value used to test for the individual significance of “Test

Score”p-value for Test Score = TDIST(3.2433,17,2)=0.00478

38 38 Slide

1 40391986

7 07 ..

1 40391986

7 07 ..

2508907735

3 24 ..

2508907735

3 24 ..

Test Statistics

Conclusions Reject both H0: 1 = 0 and H0: 2 = 0.

Both independent variables aresignificant.

p-values p-value 1= 1.9E-06p-value 2= 0.00478

critical values t.025 = 2.11

-t.025 = -2.11

39 39 Slide

In many situations we must work with qualitative independent variables such as gender (male, female), method of payment (cash, check, credit card), etc.

For example, x2 might represent gender where x2 = 0 indicates male and x2 = 1 indicates female.

Qualitative Independent Variables

In this case, x2 is called a dummy or indicator variable. In this case, x2 is called a dummy or indicator variable.

For a qualitative independent variable with kcategories, we create k-1 dummy variables. For a qualitative independent variable with kcategories, we create k-1 dummy variables.

40 40 Slide

As an extension of the problem involving thecomputer programmer salary survey, supposethat management also believes that theannual salary is related to whether theindividual has a graduate degree incomputer science or information systems.

The years of experience, the score on the programmer

aptitude test, whether the individual has a relevant

graduate degree, and the annual salary ($1000) for each

of the sampled 20 programmers are shown on the next

slide.

Example: Programmer Salary Survey

41 41 Slide

47158100166

92105684633

781008682868475808391

88737581748779947089

23.734.335.838

22.223.13033

3826.636.231.62934

30.133.928.230

Exper. Score ScoreExper.Salary SalaryDegr.

NoYes NoYesYesYes No No NoYes

Yes NoYes No NoYes NoYes No No

42 42 Slide

y = b0 + b1x1 + b2x2 + b3x3

where:

y = annual salary ($1000) x1 = years of experience x2 = score on programmer aptitude test x3 = 0 if individual does not have a graduate degree 1 if individual does have a graduate degree

x3 is a dummy variable

Use ProgramerSalary.xlsx (Qualitative IV-2Cat tab)

43 43 Slide

Excel Formula Worksheet (showing data)

grammerExperience

(years)Test

ScoreGrad.

DegreeSalary ($000)

2 1 4 78 0 24.03 2 7 100 1 43.04 3 1 86 0 23.75 4 5 82 1 34.36 5 8 86 1 35.87 6 10 84 1 38.08 7 0 75 0 22.29 8 1 80 0 23.1

44 44 Slide

significant

not significant

��𝑎𝑙𝑎𝑟𝑦=7.94+1.14 𝐸𝑥𝑝+0.19𝑆𝑐𝑜𝑟𝑒+2.28𝐷𝑢𝑚𝑚𝑦𝐷𝑒𝑔

45 45 Slide

Categorical Independent Variables

R2 = 507.896/599.7855 = .8468

2 20 1

1 (1 .8468) .818120 3 1aR

46 46 Slide

Excel’s Regression Statistics

Categorical Independent Variables

A B C23 24 SUMMARY OUTPUT2526 Regression Statistics27 Multiple R 0.92021523928 R Square 0.84679608529 Adjusted R Square 0.81807035130 Standard Error 2.39647510131 Observations 2032

Previously,R Square = .8342

Previously,Adjusted R Square = .815

(essentially no improvement)

47 47 Slide

For example, a variable indicating level of education could be represented by x1 and x2 values as follows:

More Complex Qualitative Variables

HighestDegree x1 x2

Bachelor’s 0 0Master’s 1 0Ph.D. 0 1

48 48 Slide

More Complex Qualitative Variables

significant

not significant

49 49 Slide

Residual Analysis

For simple linear regression the residual plot against

and the residual plot against x provide the same information.

y In multiple regression analysis it is preferable

to use the residual plot against to determine if the model assumptions are satisfied.

50 50 Slide

Standardized Residual Plot Against y

Standardized residuals are frequently used in residual plots for purposes of:• Identifying outliers (typically, standardized

residuals < -2 or > +2)• Providing insight about the assumption that

the error term e has a normal distribution The computation of the standardized residuals

in multiple regression analysis is too complex to be done by hand

Excel’s Regression tool can be used

51 51 Slide

Excel Value Worksheet

A B C D2829 RESIDUAL OUTPUT3031 Observation Predicted Y Residuals Standard Residuals32 1 27.89626052 -3.89626052 -1.77170689633 2 37.95204323 5.047956775 2.29540601634 3 26.02901122 -2.32901122 -1.05904757235 4 32.11201403 2.187985973 0.99492059636 5 36.34250715 -0.54250715 -0.246688757

standardized residual > 2 outlier

Remember to check the standardized residual box in Regression tool dialog box

52 52 Slide

Excel’s Standardized Residual Plot

Standardized Residual Plot

0 10 20 30 40 50

Predicted Salary

sOutlier

1 1 Slide Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple...

Documents

1 1 Slide © 2008 Thomson South-Western. All Rights Reserved Chapter 15 Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple

Multiple Regression Chapter 1313 Multiple Regression Multiple Regression Assessing Overall Fit Assessing Overall Fit Predictor Significance Predictor

MULTIPLE REGRESSION ANALYSIS: ESTIMATION - …tastan/teaching/03 Multiple Regression... · Multiple Regression Analysis In the simple regression analysis with only one explanatory

Multiple linear regression: estimation and model fitting · simple linear regression and multiple regression Multiple Simple regression regression Solar 0.05 0.13 Wind -3.32 -5.73

Regression Analysis and Multiple Regression

Multiple Linear Regression - Analysis Made Easy · Multiple Linear Regression The MULTIPLE LINEAR REGRESSION command performs simple multiple regression using least squares. Linear

Multiple regression:

2. Korrelation, Linear Regression und multiple Regression · 2. Korrelation, Linear Regression und multiple Regression 2. Korrelation, lineare Regression und multiple Regression 2.1

1 1 Slide © 2003 Thomson/South-Western Chapter 13 Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple Coefficient of Determination

2. Korrelation, Linear Regression und multiple · PDF file2. Korrelation, Linear Regression und multiple Regression 2. Korrelation, lineare Regression und multiple Regression 2.1 Korrelation

methodenlehre ll – Multiple Regression · PDF file09.06.2009 1 methodenlehre ll – Multiple Regression • Multiple Regression Thomas Schäfer | SS 2009 1 methodenlehre ll – Multiple

Doing Multiple Regression with SPSS Multiple Regression ...math.ou.edu/~mcknight/4753/spss/SPSS9.pdf · 1 Doing Multiple Regression with SPSS Multiple Regression for Data Already

Multiple Regression Analysis Multiple Regression Model Sections 16.1 - 16.6

Multiple Regression1 Multiple Linear Regression Multiple Regression Model A regression model that contains more than one regressor variable. Multiple Linear

Multiple Linear Regression - AnalystSoft Inc. · 2019-09-24 · Multiple Linear Regression The MULTIPLE LINEAR REGRESSION command performs simple multiple regression using least squares

Multiple Regression (continued) Polynomial Regression

Multiple Regression Analysis: Estimation...Multiple Regression Analysis: Estimation I n Chapter 2, we learned how to use simple regression analysis to explain a dependent variable,

Multiple regression

1 1 Slide © 2007 Thomson South-Western. All Rights Reserved Chapter 13 Multiple Regression n Multiple Regression Model n Least Squares Method n Multiple