1

Exploiting statistical dependencies of time serieswith hierarchical correlation reconstruction

Jarek DudaJagiellonian University, Golebia 24, 31-007 Krakow, Poland, Email: dudajar@gmail.com

Abstract—While we are usually focused on forecasting futurevalues of time series, it is often valuable to additionallypredict their entire probability distributions, e.g. to evaluaterisk, Monte Carlo simulations. On example of time seriesof ≈ 30000 Dow Jones Industrial Averages, there will bepresented application of hierarchical correlation reconstructionfor this purpose: mean-square estimating polynomial as jointdensity for (current value, context), where context is forexample a few previous values. Then substituting the currentlyobserved context and normalizing density to 1, we get predictedprobability distribution for the current value. In contrast tostandard machine learning approaches like neural networks,optimal polynomial coefficients here can be inexpensivelydirectly calculated, have controllable accuracy, are unique andindependently calculated, each has a specific cumulant-likeinterpretation, and such approximation using can approachcomplete description of any real joint distribution - providinga perfect tool to quantitatively describe and exploit statisticaldependencies in time series. There is also discussed applicationfor non-stationary time series like calculating linear time trend,or adapting coefficients to local statistical behavior.

Keywords: time series analysis, machine learning, den-sity estimation, risk evaluation, data compression, non-stationary time series, trend analysis, wallet analysis

I. INTRODUCTION

Modeling spatial or temporal statistical dependenciesbetween observed values is a difficult task required in acountless number of applications. Standard approaches likecorrelation matrix, PCA (principal component analysis) ap-proximate this behavior with multivariate gaussian distribu-tion. Further corrections can be extracted by approaches likeGMM (gaussian mixture model), KDE (kernel density esti-mation) [1] or ICA (independent component analysis) [2],but they have many weaknesses like lack error control, largefreedom of parameters and varying their number, or focusingon a specific types of distributions.

Fitting polynomial to observed data sample is universalapproach in many fields of science, can provide as closeapproximation as needed. It turns out also very advanta-geous for density estimation, including multivariate jointdistribution ([3], [4]), especially if variables are normalizedto approximately uniform distribution on [0, 1] with CDF ofapproximated distribution, to properly model tails, improveperformance and standardize coefficients.

Using orthonormal basis ρ(x) =∑f aff(x), it turns out

that mean-square (MSE, L2) optimization leads to estimatedcoefficients being just averages over the observed sample:af =

1n

∑ni=1 f(x

i). For multiple variables we can use basis

Figure 1. Top: degree m = 5 polynomials (integrating to 1) on [0, 1]range predicting probability density based on length 5 context (previous5 values) in 100 random positions of analysed sequence (normalizedDow Jones Industrial Averages): joint density for d = 1 + 5 = 6variables (current value and context) was MSE fitted as polynomial, thensubstituting the current context and normalizing to integrate to 1, we getpredicted density for the current value. We can see that some predicteddensities go below 0, what is an artifact of using polynomials, but can beinterpreted using below evaluation/calibration curves. Predicted densitiesare usually close to marked ρ = 1 uniform density (obtained if not usingcontext), but often localize improving prediction - for example they usuallyavoid extreme values beside some predictable conditions. Bottom: sortedpredicted densities for the actual current values in all 29349 situations:in ≈ 20% cases it gives worse prediction than ρ = 1 (without usingcontext), but in the remaining cases it is essentially better. The number ofcoefficients in the used basis is |B| = (m+1)d. We can see that predictiongenerally improves (higher density) with growing number of coefficients,however, beside growing computational cost, it comes with overfitting (e.g.negative density) - polynomial approaches sum of Dirac deltas in points ofthe sample .

of products of 1D orthornormal polynomials. On exampleof DJIA time series 1, with results summarized in Fig. 1, it

1Source of DJIA time series: http://www.idvbook.com/teaching-aid/data-sets/the-dow-jones-industrial-average-data-set/

arX

iv:1

807.

0411

9v3

[cs

.LG

] 1

1 Se

p 20

18

http://www.idvbook.com/teaching-aid/data-sets/the-dow-jones-industrial-average-data-set/http://www.idvbook.com/teaching-aid/data-sets/the-dow-jones-industrial-average-data-set/

2

will be used for prediction of current probability distributionbased on a few previous values.

Finally we get asymptotically complete description of sta-tistical dependencies - approaching any real joint distributionof observed variables. Coefficients can be cheaply calculatedas just averages, are unique and independently calculated,for stationary time series we can control their accuracy. Eachhas also a specific interpretation: resembling cumulants, butbeing much more convenient for reconstructing probabilitydistribution - instead of the difficult problem of moments [5],here they are just coefficients of polynomial. However,disadvantage of using polynomial as density parametrizationis that it occasionally leads to negative densities, what can beinterpreted as low positive - plot of sorted predicted densitiesof actually observed values allows for such calibration.

In the discussed here example: analysis of DJIA timeseries, we will first normalize the variables to nearly uniformprobability distribution on [0, 1]: by considering differencesof logarithms, and transforming them by CDF (cumulativedistribution function) of approximated distribution (Laplace)as shown in Fig. 2.

Then looking at d successive positions of such normalizedvariable, if uncorrelated they would come from ρ ≈ 1distribution on [0, 1]d. Its corrections as linear combinationof orthonormal basis of polynomials can be inexpensivelyand independently calculated, providing unique and asymp-totically complete description of statistical dependenciesbetween these neighboring values. Treating d − 1 of themas earlier context, substituting their values and normalizingto 1, we get predictions of probability distribution for thecurrent value as summarized in Fig. 1.

There will be also proposed handling of non-stationarytime series: by replacing af = 1n

∑ni=1 f(x

i) global averagewith local averages over past values with exponentiallydecaying weights, or using interpolation treating time asadditional dimension.

Presented approach can be naturally extended to multi-variate time series, e.g. stock prices of separate companiesto model their statistical dependencies, what is presentedin [6] on example of yield curve parameters, here therewill be presented example of modelling various statisticaldependencies of 29 of Dow Jones companies.

II. NORMALIZATION TO NEARLY UNIFORM DENSITY

We will discuss on example of Dow Jones IndustrialAverages time series {vt}t=1..n0 for n0 = 29355. Asfinancial data usually evolve in multiplicative not additivemanner, we will work with ln(vt) to make it additive.

Time series are usually normalized to allow assumptionof stationary process: such that joint probability distributiondoes not change when position is shifted. The standardapproach, especially for gaussian distribution, is to subtractmean value, then divide by the standard deviation.

However, above normalization does not exploit localdependencies between values, what we are interested in.Using experience from data compression (especially losslessimage e.g. JPEG LS [7]), we can use a predictor for the

Figure 2. Normalization of the original variable to nearly uniform on[0, 1] (marked green) used for further correlation modelling. The originalsequence {vt} of 29355 Dow Jones daily averages (over 100 years) is firstlogarithmized (top plot), then we take differences yt = ln(vt+1)− ln(vt).Sorting {yt} we get its approximated CDF, which, in contrast to standardGaussian assumption, turns out in good agreement with Laplace distribution(µ ≈ 0.00044, b ≈ 0.0072) - estimated and drawn (red) in the secondplot. The marked green next plot is the final xt = CDFLaplace(µ,b)(yt)sequence used for further correlation modeling. The bottom plot showssorted {xt} values to verify that they come from nearly uniform distribution(line) - its inaccuracy will be repaired later with fitting polynomial (Fig.4).

next value based on its local context: for example a fewprevious values (2D neighbors for image compression), orsome more complex features (e.g. using averages over time

3

windows, or dimensionality reduction methods like PCA),then model probability distribution of difference from thepredicted value (residue).

Considering simple linear predictors: vt ≈∑ki=1 biv

t−i

like in ARIMA-like models, we can use optimize {bk}parameters to minimize mean square error. For 2D imagesuch optimization leads to approximate parameters vx,y ≈0.8vx−1,y−0.3vx−1,y−1+0.2vx,y−1+0.3vx+1,y−1. For DowJones sequence such optimization leads to nearly negligibleweights for all but the previous value. Hence, for simplicitywe will just operate on

yt = ln(vt+1)− ln(vt) (1)

time series, where the number of possible indexes has beenreduced by 1 due to shift: n1 = n0 − 1.

Such sequences of differences from predictions (residues)are well known in data compression to have nearly Laplacedistribution - density:

g(y) =1

2bexp

(−|y − µ|

b

)(2)

where maximum likelihood estimation of parameters is just:µ = median of y, b = mean of |y−µ|. We can see in Fig. 2that CDF from sorted yt values has decent agreement withCDF of Laplace distribution. Otherwise, there can be usede.g. generalized normal distribution [8], also called expo-nential power distribution or generalized error distributions,which includes both gaussian and Laplace distribution. Sta-ble distributions (Levy) [9] might be also worth consideringas they include heavy tail distributions. These distributionshave also known asymmetric variants - which should beconsidered if two directions have essentially different tails.

For simplicity we use Laplace distribution here to nor-malize our variables to nearly uniform in [0, 1], what allowsto compactify the tails, improve performance and normalizefurther coefficients:

xt = G(yt) where G(y) =∫ y−∞

g(y′) dy′ (3)

is CDF of used distribution (Laplace here). We can seein Fig. 2 that this final xt sequence has nearly uniformprobability distribution. Its corrections will be includedin further estimation of polynomial as (joint) probabilitydistribution, like presented later in Fig. 4.

We will search for ρX(x) density. To remove transfor-mation (3) to get final ρY (y) density, observe that P (y′ =G−1(x) ≤ y) = P (x ≤ G(y)). Differentiating over y, weget ρY (y) = ρX(G(y)) · g(y).

III. HIERARCHICAL CORRELATION RECONSTRUCTION

After normalization we have {xt} sequence with nearlyuniform density, marked green in Fig. 2 here. Taking its dsucceeding values, if uncorrelated they would come fromnearly uniform distribution on [0, 1]d - difference fromuniform distribution describes statistical dependencies inour time series. We will use polynomial to describe them:estimate joint density for d succeeding values of x.

Figure 3. Top: the first 6 of used 1D orthonormal basis of polynomials(〈f, g〉 =

∫ 10 fg dx): j = 0 coefficient guards normalization, the remain-

ing functions integrate to 0, and their coefficients describe perturbationfrom uniform distribution. These coefficients have similar interpretation ascumulants, but are more convenient for density reconstruction. Center: 2Dproduct basis for j ∈ {0, 1, 2}. The j = 0 coordinates do not change withcorresponding perturbation. Bottom: sorted calculated coefficients (withouta000000 = 1) for DJIA sequence, m = 5 and length 5 context (d = 6)modelling. Assuming stationarity, for uniform distribution their standarddeviation would be σ ≈ 1/

√n ≈ 0.006, exceeded here more than tenfold

by many coefficients - allowing to conclude that they are essential: not justa noise.

Define xti = xt−i+1 for i = 1, . . . , d and t = 1, . . . , n,

n = n1 − d + 1. They form xt = {xti}i=1..d ∈ [0, 1]dvectors containing value with its context - we will modelprobability density of these vectors. Generally we can alsouse more sophisticated contexts, for example average of afew earlier values (e.g. (xt−5+xt−6)/2) as a single contextvalue to include correlations of longer range. Normalizationto nearly uniform density is recommended for the predictedvalues (xt1), for context values it might be better to omit it,especially when absolute values are important like for imagecompression.

Finally assume we have {xt}t=1,...,n ⊂ [0, 1]d vectorsequence of value with its context, we would like to modeldensity of such vectors as polynomial. It turns out [3]that using orthonormal basis, which for multidimensionalcase can be products of 1D orthonormal polynomials, meansquare (L2) optimization leads to extremely simple formula

4

for estimated coefficients:

ρ(x) =∑

j∈{0...m}dajfj(x) =

m∑j1...jd=0

aj fj1(x1) · . . . · fjd(xd)

with estimated coefficients: aj =1

n

n∑t=1

fj(xt) (4)

The basis used this way has |B| = (m + 1)d functions,generally it seems worth to consider different mi for sepa-rate coordinates (|B| =

∏di=1(mi+1)). Beside inexpensive

calculation, this simple approach has also very convenientproperty of coefficients being independently calculated, giv-ing each j unique value and interpretation, and controllableerror. Independence also allows for flexibility of consideredbasis - instead of using all j, we can focus on morepromising ones: with larger absolute value of coefficient,replacing negligible aj. Instead of mean square optimization,we can use often preferred: likelihood maximization [4], butit requires additional iterative optimization and introducesdependencies between coefficients.

Above fj 1D polynomials are orthonormal in [0, 1]:∫ 10fj(x)fk(x)dx = δjk, getting (rescaled Legendre): f0 =

1 and for j = 1, 2, 3, 4, 5 correspondingly:√3(2x−1),

√5(6x2−6x+1),

√7(20x3−30x2+12x−1),

3(70x4 − 140x3 + 90x2 − 20x+ 1),√11(252x5 − 630x4 + 560x3 − 210x2 + 30x− 1).

Their plots are in top of Fig. 3. f0 corresponds tonormalization. The j = 1 coefficient decides about reducingor increasing the mean - have similar interpretation asexpected value. Analogously j = 2 coefficient decidesabout focusing or spreading given variable, similarly asvariance. And so on: further aj have similar interpretationas cumulants, however, while reconstructing density frommoments is a difficult problem, presented description isdirectly coefficients of polynomial estimating the density.

For multiple variables, aj describes only correlationsbetween C = {i : ji > 0} coordinates, does not affectji = 0 coordinates, as we can see in the center of Fig.3. Each coefficient has also a specific interpretations here,for example a11 decides between increase and decrease ofsecond variable with increase of the first, a12 analogouslydecides focus or spread of the second variable.

Assuming stationary time series (fixed joint distributionin [0, 1]d), errors of such estimated coefficients come fromapproximately gaussian distribution:

ã− a ∼ N

(0,

1√n

√∫(fj − aj)2ρ dx

)(5)

For ρ = 1 the integral has value 1, getting σ = 1/√n ≈

0.006 in our case. As we can see in bottom of Fig. 3, a fewpercents of coefficients here are more that tenfold larger:can be considered as essential, not a result of noise.

Here is a list of the largest |aj| > 0.14 coefficients forDow Jones normalized series (beside a000000 = 1) in d =

6, m = 5 case. It neglects shifted sequences, for examplea200200 ≈ a020020 ≈ a002002.Positive:a200200 ≈ 0.184867 a200002 ≈ 0.183297a200020 ≈ 0.178384 a202000 ≈ 0.177606a554555 ≈ 0.176333 a220000 ≈ 0.176184a554535 ≈ 0.169778 a554355 ≈ 0.161684a545445 ≈ 0.156764 a555555 ≈ 0.149727a555355 ≈ 0.147934 a454523 ≈ 0.145962

Negative:a555552 ≈ −0.170723 a344544 ≈ −0.166773a455235 ≈ −0.156860 a342544 ≈ −0.149314a455255 ≈ −0.147201 a555451 ≈ −0.146523a555532 ≈ −0.145356 a553451 ≈ −0.143087a555352 ≈ −0.142076 a355451 ≈ −0.140343Each such unique coefficient describes a specific cor-

rection from uniform density: by aj fj1(x1) · . . . · fjd(xd).For example we can see large positive coefficients for allpairs of j = 2, what means upward directed parabola forboth variables: quantitatively describes how large changein a given day increases probability of large changes inneighboring days. Further coefficients have more complexinterpretations, for example large positive a555555 meansthat 6 large increases in a row are preferred, but 6 largedecreases are less likely. In contrast, large negative a555552means that larger change 5 days earlier reduces probabilityof 5 large increases in a row.

Having such density we can use it to predict probabilitydistribution of the current symbol basing on the context(Fig. 1): by substituting context to the polynomial andnormalizing the remaining 1D polynomial to integrate to 1.

Figure 4. Modelling probability distribution as independent variable(d = 1) using degree m polynomials: ρ(x) =

∑mj=0 ajfj(x). After

normalization with CDF of Laplace distribution, we should have ρ ≈ 1.Here we repair its inaccuracy with estimated polynomial (left column),corresponding to the plot in the bottom of Fig. 2 - the right column containsdifferences between empirical CDF and such fitted polynomial. Obviouslythis difference reduces with degree m, however, we can see that it containsa growing number (≈ m) of oscillations (Runge’s phenomenon).

5

Unfortunately such density can sometimes go below zero,what needs a separate interpretation as low positive.

IV. ADAPTIVITY FOR NON-STATIONARY TIME SERIES

We have previously assumed stationary time series: thatjoint probability distribution within length d moving timewindows is fixed, what is often only approximation forreal time series. As coefficients here are just averages overvalues: af = 1n

∑x f(x), for coefficients describing local

Figure 5. Modelling non-stationary probability distribution of values (d =1) - like in Fig. 4, but adapted to inhomogeneous behavior in time. The toptwo plots used adaptive averaging at+1f = λa

tf +(1−λ)f(x

t) for m = 4with two different learning rates λ = 0.9997 or 0.999. The bottom plothas estimated m = 9 degree polynomial for density of (t, xt) variables -in contrast to adaptive averaging, it requires already knowing the future.

behavior we can use (known in data compression) averagingwith exponentially decaying weights [4]:

at+1f = λatf + (1− λ)f(xt) ρt(x) =

∑f

atff(x) (6)

Figure 6. Top: evaluation of results of models presented in Fig. 5 - sortedpredicted densities of actual values. They are compared with stationarymodel: green line, using fixed coefficients being averages over entire timeperiod. Bottom: time dependence of first four coefficients over the time:ρt(x) =

∑j a

tjfj(x). They are constant for the stationary model (green

lines), degree 9 polynomials for interpolation (red), and noisy curves foradaptive averaging - especially the orange one for relatively low λ = 0.999.The blue curve for λ = 0.9997 is smoother, however, it is at cost of delay(shifted right) - needs more time to adapt to new behavior.

6

for some learning rate λ: close but smaller than 1 (e.g.λ = 0.999), starting for example with af (0) = 0. Itsproper choice is a difficult question: larger λ gives smootherbehavior, but needs more time to adapt (delay).

For modeling of time trends or a posteriori analysis ofhistorical data (with known future), we can alternativelyestimate polynomial for multi-dimensional variable withtime as one of coordinates, rescaled to [0, 1] range, e.g.(t/n, xt). This way we estimate behavior of each coefficientas polynomial, allowing e.g. to interpolate to real time, ortry to forecast that future trend (as low degree polynomial)will be similar as in the earlier period.

It might be tempting to use this approach also for extrap-olation to predict future trends, e.g. rescale time to [0, 1− �]range instead, and look at behavior in time 1. However,such polynomial often has some uncontrollable behavior atthe boundaries, suggesting caution while such extrapolation.Other orthonormal families (e.g. sines and cosines) havebetter boundary behavior - might be more appropriate forsuch extrapolation, however, discussed earlier modellingof joint distribution with context representing the past isgenerally a safer approach.

The last 3 figures present such analysis for discussedDJIA sequence. Figure 4 contains estimation of densityas polynomial using stationarity assumption (inaccuracy ofLaplace used in normalization). Figure 5 contains its timeevolution for non-stationary models: adaptive or interpola-tion. Figure 6 evaluates these approaches and shows timeevolution for first 4 cumulant-like coefficients.

V. MODELLING MULTIVARIATE DEPENDENCIES

Example of predicting probability distribution for multi-variate time series is presented in [6]. To continue the DowJones topic, there will be presented application of the dis-cussed methodology to better understand complex statisticaldependencies between stock prizes of 30 companies it is cal-culated from, for example for more accurate wallet analysis.The selection of these companies has changed throughoutthe history, but for the last decade there can be downloadeddaily prices from NASDAQ webpage (www.nasdaq.com) forall but DowDuPont (DWDP) - there will be used daily closevalues for 2008-08-14 to 2018-08-14 period (n0 = 2518values) for the remaining 29 companies: 3M (MMM), Amer-ican Express (AXP), Apple (AAPL), Boeing (BA), Caterpil-lar (CAT), Chevron (CVX), Cisco Systems (CSCO), Coca-Cola (KO), ExxonMobil (XOM), Goldman Sachs (GS),The Home Depot (HD), IBM (IBM), Intel (INTC), John-son&Johnson (JNJ), JPMorgan Chase (JPM), McDonald’s(MCD), Merck&Company (MRK), Microsoft (MSFT), Nike(NKE), Pfizer (PFE), Procter&Gampble (PG), Travelers(TRV), UnitedHealth Group (UNH), United Technologies(UTX), Verizon (VZ), Visa (V), Walmart (WMT), WagreensBoots Alliance (WBA) and Walt Disney (DIS).

The standard approach to quantitatively describe statisti-cal dependencies between variables is by correlation matrix- presented in Fig. 7. We can use discussed here cumulant-like coefficients for better undersetting of more complex

Figure 7. Top: 10-year evolution 2008-2018 of logarithm of stock price of29 out of 30 Dow Jones companies (without DWDP). We will work on theirdaily differences: yi. Center: eigenvectors corresponding to 5 largest eigen-values of their covariance matrix (Σij = E[(Yi − E[Yi])(Yj − E[Yj ])])is a popular tool (PCA) showing directions of largest variance, groupingdependent variables - e.g. financial companies in vector presented as theyellow plot. Bottom: correlation matrix (covariance matrix normalized tounit variance for each variable) - real non-negative, describing strength ofdependency between variables.

statistical dependencies and their time trends, each of themhaving unique specific interpretation.

The simplest application is modelling pairwise statisticaldependencies. After normalization of all variables to nearlyuniform distribution (with Laplace CDF like previously),pairwise joint distribution would be nearly uniform on [0, 1]2

if they are uncorrelated. To this ρ∅ = 1 initial densitywe add independently calculated correction as products oftwo orthonormal polynomials - coefficients of the first threefor all pairs are presented in Fig. 8 and 9. There are alsopresented linear time trends for two of them: using firstcoefficient with time rescaled to [0, 1] as additional variablelike in the previous section. The diagonal terms have nomeaning in this methodology - are filled with a constantvalue.

These additional coefficients allow for deeper understand-ing of statistical dependencies, like between growth of one

7

variable and variance of the second (12) or between theirvariances (22). Time trends (calculated for previous decade)may suggest further evolution of these dependencies. Thesecoefficients are similar to multivariate mixed cumulants, buthaving a direct translation to probability density.

Figure 7 also contains first 5 eigenvectors of covariancematrix as in standard PCA technique. They correspondto largest variance directions and often turn out to groupdependent variables. Here we usually estimate density ashigher degree polynomials (formally

∑i ji), for which there

are generalizations of eigenvector decomposition [10], butthey are more difficult to calculate and interpret. Presentedmatrices use only subset of coefficients, allowing to usestandard eigenvector decomposition, e.g. [cαβ ]vk = λkvk:

∑αβ

cαβf2(xα)f2(xβ) =

n∑k=1

λk

(∑α

vkαf2(xα)

)2

Presented matrices describe only pair-wise statistical de-pendencies, we can analogously model dependencies intriples e.g. f1(xα)f1(xβ)f1(xγ) or in larger groups, andtheir time evolution.

The discussed here approach starts with normalizing allvariables to nearly uniform distribution. Alternative ap-proach is to multiply idealized distribution (e.g. multivariateGaussian from PCA, or Laplace) by polynomial, estimatingthe coefficients [3].

VI. EXTENSIONS

The used example presented basic methodology for ed-ucative reasons, which in real models can be extended, forexample:

• Selective choice of basis: we have used complete basisof polynomials, what makes its (m+ 1)d size imprac-tically large especially for high dimensions. However,usually only a small percentage of coefficients is abovenoise - we can selectively choose and use a sparse basisof significant values instead - describing real statisticaldependencies. A simpler option is to selectively reducepolynomial degree for some of variables, or for exam-ple restrict the real degree of the polynomial:

∑i ji

instead of each coordinate.• Long-range value prediction: combination with state-

of-art prediction models exploiting long-range depen-dencies, for example using a more sophisticated (thanjust the previous value) predictor of the current value.

• Improving information content of context used forprediction: instead of using a few previous values asthe context, we can use some features e.g. describinglong-range behavior like average over a time window,or for example obtained from dimensionality reductionmethods like PCA (principal component analysis).

• Multivariate time series usually allow for much betterprediction, as presented in [6]. Using for examplemacroeconomic variables should improve prediction.

APPENDIXThis appendix contains Wolfram Mathematica source for

discussed procedures for stationary process, optimized touse built-in vector operations:im = Import["c:/djia-100.xls"];v = Log[Transpose[im[[1]]][[2, 2 ;; -1]]];Print[ListPlot[v]];n0 = Length[v];yt = Table[v[[i + 1]] - v[[i]], {i, n1 = n0 - 1}];syt = Sort[yt]; (* for approximated CDF *)mu = Median[yt]; (* Laplace estimation *)b = Mean[Abs[yt - mu]];cdfL = If[y < mu, Exp[(y-mu)/b]/2, 1-Exp[-(y-mu)/b]/2];Print["Laplace distribution: mu= ", mu, " b= ", b];Print[Show[

ListPlot[Table[{syt[[i]], (i - 0.5)/n1}, {i, n1}]],Plot[cdfL, {y, -0.1,0.1},PlotStyle -> {Thin, Red}]]];

xt = Table[cdfL /. y -> yt[[i]],{i,n1}]; (* normalized *)Print[ListPlot[Sort[xt]]]; Print[ListPlot[xt]];cl = 3; d = 1 + cl; (* dimension = 1 + context length *)m = 4; (* maximal degree of polynomial *)coefn = Power[m + 1, d]; Print[coefn, " coefficients"];p = Table[Power[x, k], {k, 0, m}];p = Simplify[Orthogonalize[p,Integrate[#1 #2,{x,0,1}]&]];Print["used orthonormal polynomials: ", p];n = n1 - cl; (* final number of data points *)(* table of contexts and their polynomials: *)ct = Transpose[Table[xt[[i + cl ;; i ;; -1]], {i, n}]];ctp = Table[

If[j==1, Power[ct,0], p[[j]] /. x -> ct], {j, m+1}];(* calculate coefficients: *)coef = Table[jt = IntegerDigits[jn, m + 1, d] + 1;

Mean[Product[ctp[[jt[[c]], c]], {c, d}]],{jn, 0, coefn - 1}];

(* find 1D polynomials for various times: *)pt = Table[0, {i, m + 1}, {i, n}];Do[jt = IntegerDigits[jn, m + 1, d] + 1;pt[[jt[[1]]]] +=coef[[jn+1]] * Product[ctp[[jt[[c]], c]],{c, 2, cl + 1}], {jn, 0, coefn - 1}];

(* probability normalization to 1: *)Do[pt[[i]] /= pt[[1]], {i, m + 1, 1, -1}];(* predicted densities for observed values: *)rho = Sum[ctp[[i, 1]] * pt[[i]], {i, m + 1}];Print[ListPlot[Sort[rho]]];(* densities in 10 random times: *)plst = RandomInteger[{1, n}, 10];pl = Table[i = plst[[k]];Sum[pt[[j, i]]*p[[j]], {j, m + 1}], {k, Length[plst]}];

Plot[pl, {x, 0, 1}, PlotRange -> {{0, 1}, {0, 5}}]

REFERENCES[1] M. Rosenblatt et al., “Remarks on some nonparametric estimates of

a density function,” The Annals of Mathematical Statistics, vol. 27,no. 3, pp. 832–837, 1956.

[2] A. Hyvärinen and E. Oja, “Independent component analysis: algo-rithms and applications,” Neural networks, vol. 13, no. 4, pp. 411–430, 2000.

[3] J. Duda, “Rapid parametric density estimation,” arXiv preprintarXiv:1702.02144, 2017.

[4] ——, “Hierarchical correlation reconstruction with missing data,”arXiv preprint arXiv:1804.06218, 2018.

[5] J. Shohat, J. Tamarkin, and A. Society, The Problem of Moments, ser.Mathematical Surveys and Monographs. American MathematicalSociety, 1943. [Online]. Available: https://books.google.pl/books?id=xGaeqjHm1okC

[6] J. Duda and M. Snarska, “Modeling joint probability distribution ofyield curve parameters,” arXiv preprint arXiv:1807.11743, 2018.

[7] M. J. Weinberger, G. Seroussi, and G. Sapiro, “The loco-i losslessimage compression algorithm: Principles and standardization intojpeg-ls,” IEEE Transactions on Image processing, vol. 9, no. 8, pp.1309–1324, 2000.

[8] M. K. Varanasi and B. Aazhang, “Parametric generalized gaussiandensity estimation,” The Journal of the Acoustical Society of America,vol. 86, no. 4, pp. 1404–1415, 1989.

[9] B. Mandelbrot, “The pareto-levy law and the distribution of income,”International Economic Review, vol. 1, no. 2, pp. 79–106, 1960.

[10] L. Qi, “Eigenvalues of a real supersymmetric tensor,” Journal ofSymbolic Computation, vol. 40, no. 6, pp. 1302–1324, 2005.

https://books.google.pl/books?id=xGaeqjHm1okChttps://books.google.pl/books?id=xGaeqjHm1okC

8

Figure 8. Top: normalization of each variable to nearly unform distributionlike in Fig. 2 using separately estimated Laplace (left) or Gauss (right)distribution for 29 companies. We can see that the later is far fromuniform distribution (line), especially the tail behavior - what means (oftendangerous) underestimation of probability of extreme events, hence Laplacedistribution is further used. Center: pairwise joint distribution for pairs ofnormalized x variables would be nearly uniform on [0, 1]2 if uncorrelated- the presented matrices contain coefficients for ten-year average off1(xiα )f1(xiβ ) correction to this uniform joint distribution for all pairs(upper) and their linear in time coefficient (lower): f1(xα)f1(xβ)f1(t/n)- linear trend over this time period, obtained by using time normalizedto [0, 1] as additional variable like in the adaptivity section. Averagecoefficients are similar as for correlation matrix - what supports theircumulant-like interpretation. Linear trends are a new information andsurprisingly turn out always negative here - we can see a general trendof companies loosing dependencies with others, possibly through somedilution of dependencies by increasing their number. Bottom: conformationof this general trend of lowering dependencies by its adaptive calculationfor some 2 chosen companies with all 28 remaining.

Figure 9. Ten-year average coefficients for two succeeding correctionsto uniform pair-wise joint distributions. Top: f1(xα)f2(xβ) coefficientsdescribing growth or reduction of variance of the second variable withgrowth of the first variable. Obtained matrix is no longer symmetric,we can clearly see blue columns of companies having lower influenceon variance of others. Center: its linear time trend - coefficients off1(xα)f2(xβ)f1(t/n). Bottom: f2(xα)f2(xβ) coefficients describingdependencies of variance - it turns out always positive here, meaning thatthe larger coefficient, the less likely that only one from the pair obtainsextreme value - increasing risk for using both. Its linear time trend isnot presented, but it turned out always negative like for f1(xα)f1(xβ),meaning weakening of dependencies. Diagonal terms are meaningless sothey are filled with 0. While these corrections are formally degree 3 or 4polynomials, restricting to only some of them allows to use eigenvectorsof above matrices.

I IntroductionII Normalization to nearly uniform densityIII Hierarchical correlation reconstructionIV Adaptivity for non-stationary time seriesV Modelling multivariate dependenciesVI ExtensionsAppendixReferences