Investigating the relationships between the statistical characteristics:
2
Investigating the relationship between qualitative characteristics, e.g. AB , called measurement of association
Investigating the relationship between quantitative characteristics – Regression and correlation analysis
Regression and correlation analysis:
4
examining causal dependency, exploring the relationship between
cause and effectWhen one or more effects (attributes,
independent variables) cause resulting effect – dependent variable
Y = f (X1 X2…... Xk ,Bo , B1 ,….Bp ) +e
Dependent
variable
- effect
Independent
variables
- cause
Unknown
parameters of
a functional relationship
Random,
unspecified effects
Example of false correlation
5
One of the famous spurious correlations:If the skirt lenght gets shorter, quotation of stocks gets higher
Apart from that it is not always true, it would be false, or spurious correlation.
Examples of statistical - free - dependence
6
Examination how consumption of pork depend on income, price of pig meat, beef, poultry and tradition resp. another unspecified, or random effects.
Examination of dependence of GNP on Labour and Capital...
Ivestigation if the nutrition of the population depend on the degree of economic development of the country
Opposite of the statistical dependence is the functional dependence
7
Y = f(X1 X2…... Xk ,Bo , B1 ,…., Bp)
Where the dependent variable is clearly determined by functional relationship,
Examples from physics, chemistry – this kind of relationship is not the subject of statistical investigation.
Regression and correlation analysis (RaKA)
8
Two basic task of RaKA:Regressiona) find a functional relationship by which
the dependent variable changes with the change of independent variables - find a suitable regression line (function).
b) It is also necessary to estimate the parameters of the regression function.
Correlation - to measure strength of the examined dependence (relationship).
According to the number of independent variables are distinguished:
10
Simple dependence, when we consider only one independent variable X, we investigate the relationship between Y and X.
Multiple dependence , we are considering at least two independent variables veličiny X1, X2, … Xk , for k 2
Simple regression and correlation analysis
11
Consider statistical sign X and Y which are in the population in linear relationship Y = Bo + B1 X +e
point estimate of the regression function is a straight line yj = b0 + b 1 xj + ej , with coefficients calculated from the sample data
Which method to use ???
The least square method (LSM)
12
MINyyn
jjj )( 2
1
,
1,2,..p i 0, b
)b,...b,b(F
i
p10
We get set of p+1 equation with p+1 unknown parameters => Ordinary least square method (OLS)
yj = b0 + b 1 xj + ej we can rewrite yj = yj ,
+ ej and ej = y j - yj ,
Principle of the LSM
13
(ej ) = y j - y j’
(ej )2 = (y j - y j’)2
MINyy j
n
jj )( 2,
1
Can be proved that coefficients bo , b1 , …, bp
determined by OLS are “best estimates” of parameters B 0 , B1 , …, Bp if the random error meet the assumptions:
14
E (ej ) = 0,
D (ej ) = E (ej2 ) = 2 ,
E(ej1 , ej2 ) = 0 , for each j1 j2
Verbal formulation : Random errors are required to have zero mean, constant variance and should be independent.
Coefficients of the simple regression function can be derived:
15
0)xb - b(y b
)b,b(F
0)y(y b
)b,b(F
2j1o
n
1jj
i
10
2,j
n
1jj
i
10
0)x)(xb - b(y2 b
)b,b(F
0)1)(xb - b(y2 b
)b,b(F
jj1o
n
1jj
1
10
j1o
n
1jj
0
10
After transformation we get two normal equations with two unknown parameters:
16
n
1j
2j1
n
1jj0
n
1jjj
n
1jj10
n
1jj
x .b x.b yx
x .b n.b y
The system of equation can be solved by elimination method , or by using determinants. We get the coefficients b o a b 1
The procedure for calculating the coefficients LRF
17
xj yj xjyj xj 2
x1 y1 x1y1 x12
x2 y2
xn yn jy jx jj yx 2jx
Interpretation of simple linear regression coefficients
18
bo …intercept - expected value of dependent variable if the independent variable is equal to zero
b 1 …. Regression coefficient express the change in dependent variable, if the independent variable will change by one unit.
if b1 > 0 …positive correlation (dependence)
if b1< 0 ….negative correlation (dependence)
Properties of least square method:
19
min )yy( 2n
1j
,jj
01
) y(yn
j
,jj
Regression function passes throught the coordinates a x y
When OLS can be applied?
20
If the regression function is linear
Linear in parameters (LiP)
Or we can transform regression function to be linear in parameters
Consider in which of the following regression functions can be used OLS
Some types of simple regression function:
21
j
j
xoj
bjoj
xoj
jjoj
joj
joj
bbby
xby
bby
xb xbby
xbby
xbby
21'
'
1'
221
'
1'
1'
.
.
. .
log .
/
1
Examples from micro- and macroeconomy
22
Phillips curve ????Cobb -Douglas production curveEngel curvesCurve of economic growthAny other? …...
Examining the consumption of selected commodities (depends on the level of GNP)
23
Obrázok 2. Priebeh spotreby energie živočíšneho pôvodu
0200
400600800
10001200
14001600
0 10000 20000 30000 40000
HNP v US$ na obyv. a rok
kcal
na
obyv
. a
deň
rozvinuté krajiny
rozvojové krajiny
25
Confidence interval for linear regression In addition to point estimates of parameters of linear regression functions are often calculated also interval estimates of parameters, which are called confidence intervals. Calculations of confidence intervals can be done with standard deviations of parameters and residual variance. Residual variance, if all the conditions of classical linear model are satisfied, is undistorted estimate of the stochastic parameter and is calculated according to equation
pn
yy
s
n
1j
2jj
2rez
1kp
2
26
Interval estimate of any parameters for the regression line
Assumes that if the assumptions formulated in classical linear model has variable
t distribution with n – p degrees of freedom. For the chosen confidence level 1 – is confidence interval for parameter given by relationship
jj xbby 10
ib
iii s
bt
0
1s.tbs.tbP0b000b0
27
And for parameter
Analogically is constructed confidence interval for regression line
Where is quantile of t distribution S with (for regression line n-2) degrees of freedom.
1
1s.tbs.tbP1b111b1
1s.tyYs.tyPjyjjjyj
2j
r1b )xx(
1.ss
Role of the correlation
28
Examine tightness - strength - of dependence
We use various correlation indicesShould be bounded in intervaland within that interval increased to a
higher power of dependence
29
Correlation analysis provides methods and techniques which are used for verifying of explanatory abilityof quantified regression modelsas a whole and its parts.
Verification of explanatory ability of quantified regression modelsleads to calculation of numerical characteristics,which in concentrated form describe the quality of the calculated models.
Index of correlation and index of determinationIn population Iyx estimate from sample data
is iyx est Iyx = iyx . Principle lies in the decomposition of variability of dependent variable Y
30
2j
n
1jj
2n
1jj
2n
1jj )'yy()y'y( )yy(
Total variability of dependent variable
Variability of dependent variable explained by regression function
Variability unexplained by regression function- Residual variability
31
. Its obvious that there is a relationship: T = E + U
T = Total sum of squares (of deviation)
n
jj yy
1
2
E = is explained sum of squares
n
jj yy
1
2
U = is unexplained (residual) sum of squares.
n
njjj yy 2
Index of correlation iyx
32
T
E
)(
)'(
2
1
2n
1j
yy
yy
i n
jj
j
yx
Index of determination iyx2
n
jj
n
jjj
yy
yy
T
U
T
UTi
1
2
1
2
2 11
33
Index of determination can take values from 0 to 1, when the value of the index is close to 1, the great proportion of the total variability is explained by the model and vice versa, if the index of determination is close to zero, the low proportion of the total variability is explained by the model.Index of determination is commonly used as a criterion in deciding about the shape of the regression function. However, if the regression functions has different number of parameters, it is necessary to adjust the index of determination to the corrected form:
2korI
n
jj
n
jjj
yypn
yyn
1
2
1
2
)(
)1(
1
34
n
1j
2j yy
n
njjj yy 2
1p
pn
12
p
Vs y
pn
Nsr
22
2
r
y
s
s
n
jj yy
1
2. 1n
Variability
Sum of squares
Degrees of freedom Variance
F test
Explained
Unexplained
V =
N =
F =
Total C =
35
Test criterion in the table can be used for simultaneous testing the significance of the regression model, the index of determination and also correlation index. We compare calculated value of F test and quantile of F distribution with p-1 and n-p degrees of freedom. if F regression model is insignificant, as well as the index of correlation and index of determination. if F > regression model is statistically significant as well as the index of correlation and index of determination.
pn,1pF
pn,1pF
36
For a detailed evaluation of the parameters quality of regression model is used t tests. We formulate the null hypothesis
H0 : pre i = 0, 1 H1 :0i
where we assume zero therefore insignificant effect or impact of the variable at which the parameter is. The test criterion is defined by relationship:
ib
ii s
bt
0i
37
Where is value of the parameter of regression functionand is standard error of the parameter.
We will compare calculated value of test criterion with quantile of t distribution at significance level and degrees of freedom .:- if we do not reject null hypothesis about insignificance of the parameter.- if we reject null hypothesis, and confirm statistical significance of the parameter.
ib
ibs
pn
)pn(tt
)pn(tt
38
Nonlinear regression and correlation analysis
In addition to linear regression functions, in practice
are very often used nonlinear functions, which can be used also with two or more parameters. Some non-linear regression functions can be suitably transformed to be linear in parameters, and we can then use the method of least squares.. Most often, we can transform nonlinear function with two parameters to shape:
ZU 10
39
We estimate regression function in form
jj zbbu .10 where
)(yfu )(xfz
Function is then calculated as a linear function. Not all non-linear functions can be converted in this way, only those which are linear in parameters, ie there is some form of transformation called the linearising transformation, most often it is the substitution and logarithmic transformation for example
42
Exponential function
jxj ccy 10 .
10 log.loglog cxcy jj
yu log jxz 00 log cb 11 log cb
zbbu j .10
43
power (Cobb-Douglas production function)
1.0bjj xcy
jj xbcy log.loglog 10
xz logyu log00 log cb
zbbu j .10
44
Similarly, it is possible to modify some more parametrical nonlinear functions such as.second degree parabola
2210 .. jjj xbxbby
2xz
jjj zbxbby .. 210
46
It should be noted that the transformed regression functions do not always have the same parameters as the original non-linear regression function, so it is necessary for the estimated parameters of the transformed functions to do backwards calculations of the original parameters. Thus obtained estimates of the original parameters, do not have optimal statistical properties, but are often sufficient to solve specific tasks.Some regression function can not be adjusted or transformed to functions linear in parameters. Estimates of the parameters of such functions are obtained using different approximate or iterative methods. Most of them are based on so-called gradual improvement of initial estimates, which may be eg. expert estimates, or the estimates obtained by the selected points and so on.
47
Multiple regression and correlation analysis
Suppose that the dependent variable Y and explanatory (independent) variables Xi ,i = 1, 2, ..., k
Are in linear relationship, we have already mentioned in previous sections, can be written:
),,,,,,,,( 21021 kkXXXfY Which we estimate:
),,,,,,,( 1021 kkjjjj bbbxxxfy
48
Coefficients , which are estimates of parameters , should meet the condition of the Least squares method
kbbb ,...,, 10
k ,...,, 10
n
1j
2jjk10 minyy)b,...,b,b(F
since we assume a particular shape of the regression functions, we can install it into previous relationship and look for a minimum of this function ie.:
n
1j
2kjkj110jk10 minxbxbby)b,...,b,b(F
we determine the minimum of the function similarly like in the case of a simple regression equation using partial derivatives of functions
49
0b
)b,,b,b(F
i
k10
Which leads to system of equations:
n
j
n
jkjk
n
jjj xbxbnby
1 11110 ...
n
jkjjk
n
jj
n
jj
n
jjj xxbxbxbyx
11
1
211
110
11 ....
n
jkjk
n
jkjj
n
jkj
n
jjkj xbxxbxbyx
1
2
111
110 ..
50
The solution of this system of equations will be the coefficients of linear regression equations Like for the simple linear relationship, we can calculate estimate of the parameters from the matrix equation
kbbb ,...,, 10
yXXXb TT 1)(
kb
b
b
b
b
2
1
0
knn
k
k
k
xx
xx
xx
xx
X
1
313
212
111
1
1
1
1
ny
y
y
y
2
1
The quality of a regression model can be evaluated similarly to the simple linear relationship, which we described in the previous section.
51
SUMMARY OUTPUT
Regression StatisticsMultiple R 0.809324R Square 0.655006Adjusted R Square0.647818Standard Error 175.9096Observations 50
ANOVAdf SS MS F Significance F
Regression 1 2820028 2820028 91.13269 1.13E-12Residual 48 1485322 30944.2Total 49 4305350
CoefficientsStandard Error t Stat P-value Lower 95%Upper 95%Intercept -584.881 150.0542 -3.8978 0.000301 -886.585 -283.177lnGNP 164.9321 17.27699 9.546344 1.13E-12 130.1944 199.6698
52
Important terms:
Correlation analysis – group of techniques to measure the association between two variables
Dependent variable – variable that is being predicted or estimated
Independent variable – variable that provides the basis for estimation. It is the predictor variable
Coefficient of correlation – a measure of the strength of the linear relationship
Coefficient of determination – The proportion of the total variation inthe dependent variable Y that is explained, or accounted for, by the variation in the independent variable X