Embed Size (px)
Citation preview
Stat 200b. Chapter 8. Linear regression models.
T
jj
j
T
j
jjppjj
jjj
xyE
x
xxy
p
njEsx
xy
x
}{
...
iesexplanator
,...,1 ,0}{ fixed, '
8.1 Example
,1 iesexplanator two- level sea Venice
10
10
nnn x
x
x
y
y
y
.
.
.
1
.
.
.
1
1
.
.
.
8.1 Example
model regressionlinear
matrixdesign :
rowsstack :formmatrix
2
1
1
2
1
2
1
0 Xy
X
Xy
n by 1, n by 2, 2 by 1, n by 1
s'unknown in linear Note.
1by p pby n
.
.
.
... 1
.
.
.
... 1
...y
regression polynomial single 8.2 Example
1
0
1
1
11
1
1
2
210j
pp
nn
p
j
p
jpjj
xx
xx
xxx
x
X
12 8 68 10 1
.
.
.
52 15 29 1 1
60 6 26 7 1
...
Model
CaOAl 3
clinkers weight % :,...,
gram)(calories/ evolvedheat :
dataCement 8.3 Example
44110
32
41
X
jjjj xxy
O
xx
y
13 by 5
- psi 60 40, pressure tyre
-on off, dynamo
- 30" ,26" height seat
levels 2 with factors Three
hill up ride to time:
data cycling 8.4 Example
y
1by 4 4by 16
1 1 1 1
1 1 1 1
1 1 1- 1
1 1- 1 1
1 1- 1 1
1 1- 1- 1
1 1- 1- 1
1- 1 1 1
1- 1 1 1
1- 1 1- 1
1- 1 1- 1
1- 1- 1 1
1- 1- 1 1
1- 1- 1- 1
1- 1- 1- 1
3
2
1
0
X
Effect of increasing seat height is 21
Some matrix review
transpose
multiplication
inverse
derivatives
Normal linear model. Estimation
exists inverse if )(ˆ
)ˆ(
equations Normal
p1,...,r ,0ˆ)(2/)(
)()()()(
2/})(1
log{),(
),(
),(~
11
2
1
2
12
22
2
2
yXXX
0XyX
XyXy
T
T
T
j
T
jjrr
TT
j
n
jj
T
j
n
jj
T
jj
xyxSS
xySS
xynl
L
xINy
Profile log likelihood
)/()ˆ( S
/)ˆ()ˆ(ˆ
),(max)(
22
2
22
pnxy
n
ll
T
jj
p
XyXy T
Straight-line/simple regression.
2
1
0
1
1
0
11
)( )(
)(
ˆ
ˆˆ
.
.
.
1
.
.
.
1
.
.
.
xxxx
xxn
xx
xx
y
y
jj
j
nnn
XXT
)2/()ˆ()ˆ( :estimate variance
ˆ:residuals
)(
)(/1 0
0 /1)(
2
n
yxx
y
xx
n
jj
j
j
XyXy
Xye
yX
XX
T
T
1T
Fitted values.
ˆ
)(I
ˆ
:Residuals
)ˆ()ˆ()ˆ(
matrixhat ,
)(ˆ
yH
Hyy
Xye
yyyy
Hy
yXXXXXy
T
T1T
n
SS
NB. Assuming matrix inverse exists
Weighted least squares.
WyXWXX
W
XyWXy
T1T
2T
)(ˆ
)()(
][
2/}/)()(log{),(
known /var
2
22
2
j
T
jj
j
jjj
wxySS
wdiag
nl
wwy
inverse existing
Example 8.8. Cycling data
up pressure ,563.1
on dynamo ,563.1
upseat ,437.5
average overall ,19.47
)(ˆ
/16)(
16
1)by 1,16by 4,4by 1,16by (16
(8.2) Model
4
1
4
yXXX
IXX
IXX
Xy
T1T
T
T
55psi pressure dynamo, no seat,high :mefastest ti
ec2(-1.563)s change fitted :40not 55psi, pressure
c2(1.563)se change fitted :offnot on, dynamo
ec2(-5.437)s change fitted 26":not ,30"seat
563.1563.1437.519.47ˆ
1,,
321
321
xxxy
xxx
Example 8.10. Maize data.
22
0
1
0
12
012
202
101
)( 2/ˆ )2,...,2,2(
1by 1m and 1mby m
.
.
1...100
1...00 1
.
.
0...01 1
0...101
0...10 1
0...0 1 1
2}{
15,...,1
ddSdmdiag
yyd
yyE
mj
y
y
j
m
jjj
jj
jjj
jjj
XX
X
T
Likelihood quantities.
})(21
{)(
)(1
1
2
6422
2
42
2
2
2
T
jj
T
jjjr
r
jsjr
sr
xyl
xyxl
xxl
Take expected values
)/2
)(;( asymp
ˆ
ˆ
/2
)(),(
2/
),(
4
2
222
4
12
12
4
2
2
nN
nI
nI
0
0XX
0
0XX
0
0XX
1T
T
T
Normal distribution theory. Full rank case
1T2
1T
T1T
2
XXVarI
XX
yXXX
)(ˆ, if
ˆ,0 If
case normalNon
pby 1 and 1by p )ˆ)(ˆ(ˆVar
~)ˆ(
tlyindependen
))(,(~
)(ˆ
),cov( ,0
),0(~
2
22
2
2
T
T
pn
n
j
E
EE
E
SS
N
IE
IN
A useful decomposition.
22 )(
valuesexpected Taking
)ˆ()ˆ(
)ˆˆ()ˆˆ(
)()(
pEn
ee
XXee
XXXyXXXy
XyXy
T
TTT
T
TT
Confidence interval.
2/1
}
12
11
2
)(1{
ˆ
for interval Prediction
)(
pivotal ~)ˆ(
xXXx
x
XXT
TT
T
rs
pn
rr
rr
S
y
y
tS
Gauss-Markov Theorem. page 374
HHHIBBB
AHIA
AXXXAXAA
XXIA
IAX
AX
XXX
XXyvarXyE
2
n
T
T
TT1TT
T
n
p
TT
T2
noting , where0
)(
})({
)(ˆvar~
var
so
allfor ~
E Proof.
ˆvar~
var meansBest
~E means Unbiased
A somefor ~
meansLinear
(BLUE)
of estimate unbiasedlinear best theis ˆthen
ˆ
rnonsingula ,)(,)( Suppose
2
2
2
12
n
n
Ay
y
I
There is a generalized inverse variant
Example.
ˆvar~
var,ˆ
ˆ is BLUE
1by p and pby 1 ,Consider
PPPEP
P
P
Eg. Teaching methods data, p. 427
Method average
Usual 17 14 24 20 24 … 24 19.67
Praised 28 30 29 24 27 … 23 27.41
2121
2121
- BLUE, - in interested beMight
] [ BLUE], [in interested beMight
1,...,9r 1,2 t Model
yy
yy
y trttr
