Trend analysis: methodology

Trend analysis: methodology

Victor Shatalov

Meteorological Synthesizing Centre East

TFMM trend analysis workshop, 17-18 November 2014

Main topics

Trend analysis of annual averages of concentration/deposition fluxes

Trend analysis of monthly averages (with seasonal variations)


B[a]P concentrations in Germany

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

Trend analysis: generalities

Aim: investigation of general tendencies in time series such as:

Measured and calculated pollutant concentrations at monitoring sites

Average concentrations/deposition fluxes in EMEP countries

…Method: trend analysis – decomposition of the considered series into regular component (trend) and random component (residue)


0.0

0.2

0.4

0.6

0.8

1.0

1.2

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

TrendResidue

Residue (random component)

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3


Main steps

Detection of trend and its character: increasing decreasing mixed

Identification of trend type: linear quadratic exponential other

Quantification of trend: total reduction annual reduction magnitude of seasonal variations magnitude of random component other

Interpretation of the obtained results

Presentation by Markus Wallasch, 15 TFMM meeting, April 2014


Determination of trend existence

B[a]P measurements: SE12

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3

Decreasing pair

Increasing pair

Mann-Kendall test:Z = (number of increasing pairs) – (number of decreasing pairs) with normalization.Critical values: ± 1.44 at 85% level

± 1.65 at 90% level± 1.96 at 95% level

Z = - 1.49Decreasing trend at 85% significance level


0

0.2

0.4

0.6

0.8

1

1.2

19

90

19

92

19

94

19

96

19

98

20

00

20

02

20

04

20

06

20

08

20

10

ng

/m3

Z = - 4.05 Z = 1.8Mixed trend character:

In the period from 1990 to 2000 – statistically significant (at 95% level) decreasing trend

In the period from 2004 to 2010 – statistically significant (at 90% level) increasing trend

Typical situation for HMs and POPs


Random component

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Calculations

Linear trend

Determination of trend type: linear trend

Conc = A · Time + B + ω Calculation of A and B:regression or Sen’s slope

Z = - 3.1 decrease

Z = 3.8 increase

Residual trend exists

Criterion of the choice of trend type: Mann-Kendall test should not show statistically significant trend on all sub-periods of the time series

ω – residues (random component)



0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

Air concentrations

Trend

Criterion of non-linearityCriterion of non-linearity of the obtained trend in time:

NL = max[abs(Δi /Cichord)] · 100%

Supposed threshold value: 10%

C ichord

Δ i

i

Chord

B[a]P concentrations in Finland

0.0000.0100.020

0.0300.0400.0500.0600.070

0.0800.0900.100

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Air concentrations Trend

NL = 15.6%

Non-linear trend

B[a]P concentrations in Belgium

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3

Air concentrations Trend

NL = 8.1%

Linear trend

Fraction of non-linear trends

Heavy metals (Pb) 87%

POPs (B[a]P) 62%


Residual (random component)

-0.2

-0.1

0.0

0.1

0.2

0.3

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Determination of trend type: mono-exponential trend

Conc = A · exp(- Time / ) + ω,

– characteristic time

Z = - 3.3 Z = 3.2 decrease increase


Calculation of A and :least square method


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Calculations

Exponential trend


Random component

-0.2

-0.1

0.0

0.1

0.2

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Determination of trend type: polynomial trend

Conc = A · Time2 + B · Time + C + ω

Z = 0.5 Z = -2.3 no trend decrease


Calculation of A, B and C:least square method


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Calculations

Polynomial trend


Residual (random component)

-0.2

-0.1

0.0

0.1

0.2

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3


0

0.2

0.4

0.6

0.8

1

1.2

1.4

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Calculations

Bi-exponential trend

Determination of trend type: bi-exponential trend

Calculated byleast square method

Z = 0 Z = -1.4 no trend no trend

See [Smith, 2002]

Conc = A1 · exp(- Time /1) + A2 · exp(- Time /2)

Ai – amplitudes, i – characteristic times

No statistically significant residual trend obtained



0

0.2

0.4

0.6

0.8

1

1.2

1.4

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Calculations


Statistical significance of increasing trend

Z = 1.8

Mann-Kendall test for 2004 – 2010:

does not confirm statistically significant increasing trend

does not claim the absence of increasing trend

Confidence interval for trend slope:

[TS0 + A, TS0 + B]

TS0 – slope of calculated trend

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

[A, B] – confidence interval for slope of random component

Typical situation for B[a]P: increase in the end of the period

0

0.2

0.4

0.6

0.8

1

1.2

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

0

0.2

0.4

0.6

0.8

1

1.2

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

ng

/m3

Increase is statistically significant


Non-linear trend analysis

Conc = A1 · exp(- Year / 1) + A2 · exp(- Year / 2) + ω

Regression model, non-linear in the parameters 1 and 2

Non-linear regression models are widely investigated, for example:

Nonlinear regression, Gordon K. Smith, in Encyclopedia on Environmetrics, ISBN 0471899976, Wiley&Sons, 2002, vol 3, pp. 1405 – 1411

Estimating and Validating Nonlinear Regression Metamodels in Simulation, I. R. dos Santos and A. M. O. Porta Nova, Communications in Statistics, Simulation and Computation, 2007, vol. 36: pp. 123 – 137

Nonlinear regression, G. A. F. Seber and C. J. Wild, Wiley-Interscience, 2003



0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

Calculations


Parameters for trend characterization: reduction/growth

ΔCi

Negative values of reduction mean growth

Relative annual reductionRi = ΔCi / Ci = (1 – Ci+1 / Ci)

Total reduction per periodRtot = (Сbeg–Cend)/Cbeg=1–Cend/Cbeg

Average annual reductionRav = 1 – (Cend / Cbeg) 1/(N-1)

where N – number of years

Reduction parametersRmin = min (Ri)Rmax = max (Ri)Rav

Rtot

For the considered example:Rmin = - 6% (growth)Rmax = 15%Rav = 6%Rtot = 69%

Cbeg

Cend



0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

Calculations


Δ

Residue (random component)

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

ng

/m3

Parameters for trend characterization: random component

Parameter: standard deviation of random component normalized by trend values Frand = σ(Δ/Ctrend)

-20%

-10%

0%

10%

20%

30%

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Normalized random component

For the considered example: Frand = 11%

Frand


Seasonal variations of pollution

Seasonal variations are characteristic of heavy metals and (particularly)

for POPs

B[a]P concentrations measured at EMEP site CZ3 from 1996 to 2010. Pronounced seasonal variations are seen.

B[a]P concentrations at CZ3 site

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Pb concentrations measured at EMEP site DE7 from 1990 to 2008. Seasonal variations are also seen.

Pb concentrations at DE7 site

0

10

20

30

40

50

60

70

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

ng

/m3

TFMM trend analysis workshop, 17-18 November 2014 Possible approaches to description of seasonal

variations

t – time

– chatracteristic times,

A, B – constants,

φ – phase shifts.

Bi-exponential approximation

Mono-exponential approximation *)

Conc = A · exp(– t / + B · cos(2 · t – φ))

or

Log(Conc) = A’ – t / + B · cos(2 · t – φ)

*) Kong et al., Statistical analysis of long-term monitoring data… Environ. Sci. Techn., 10/2014

Conc = A1 · exp(– t / 1) · (1 + B1 · cos(2 · t – φ1)) + A2 · exp(– t / 2) · (1 + B2 · cos(2 · t – φ2))


Usage of higher harmonics

Trend calculated by bi-exponential approach. Possible artifact: negative trend values

Measurement data at CZ3 from 1996 to 2010

Statistical significance of second harmonic: Fisher’s test F

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3

Measurements trend standard

Possibility to avoid negative values: usage of higher harmonicsConc = Tr1 + Tr2 , Tri = Ai·exp(– t / i)·(1+Bi·cos(2·t–φi)+Ci·cos(4·t–ψi))

0

0.5

1

1.5

2

2.5

3

3.5

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Measurements two harmonics


One harmonic

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Calculations Trend Main component

Average B[a]P concentrations in Europe from 1990 to 2010 (main harmonic only)

Poor approximation for small values of concentrations

Pronounced harmonic trend with doubled frequency

Residues for one-harmonic approximationResidues, one harmonic

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3



One harmonic

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3


Significance of second harmonic is confirmed by Fisher’s test

Average B[a]P concentrations in Europe from 1990 to 2010 (main harmonic only)

Poor approximation for small values of concentrations

Trend including two harmonics

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3




Full trend

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Concentrations

Trend

Main component

Splitting trends to particular componentsExample: average B[a]P concentrations for Germany from 1990 to 2010.

Main component

0

0.2

0.4

0.6

0.8

1

1.2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3 Cmain

Ctot = Cmain + Cseas + Crand

Seasonal component

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Full

trend

CseasRelative annual reductions (as above): Rmin, Rmax, Rav, Rtot

Ctot

Random component

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Crand


Full trend

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Concentrations

Trend

Main component


Main component

0

0.2

0.4

0.6

0.8

1

1.2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3 Cmain


Seasonal component

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Full

trend

Cseas

Ctot

Random component

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Crand

Seasonal component, normalized

-100%

-80%

-60%

-40%

-20%

0%

20%

40%

60%

80%

100%

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

Normalization: Cseas/Cmain

Average value of the annual amplitude of the normalized seasonal component Fseas

Threshold value: 10%

Fraction of trends with essential seasonality

Heavy metals (Pb) 93%

POPs (B[a]P) 100%


Full trend

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Concentrations

Trend

Main component


Main component

0

0.2

0.4

0.6

0.8

1

1.2

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng/m

3 Cmain


Seasonal component

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Full

trend

Cseas

Ctot

Random component

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

ng

/m3

Crand

Random component, normalized

-100%-80%-60%-40%-20%

0%20%40%60%80%

100%

19

90

19

91

19

92

19

93

19

94

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

Normalization: Crand/Cmain

Standard deviation of normalized random component Frand


Phase shift as a fingerprint of source type

Trends for PB concentrations at CZ1

0

2

4

6

8

10

12

14

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Air

conc

entr

atio

ns, n

g/m

3

Anthropogenic

Secondary

Δφ

Difference Δφ of phase shift φ between Pb pollution at CZ1 location due to anthropogenic and secondary sources.

Phase shift can be used to determine which source type (anthropogenic or secondary) mainly contributes to the pollution at given location (in a particular country).


List of trend parametersParameters for trend characterization:

Relative reduction over the whole period (Rtot),

Relative annual reductions of contamination:

average over the period (Rav),

maximum (Rmax),

minimum (Rmin).

Relative contribution of seasonal variability (Fseas).

Relative contribution of random component (Frand).

Phase shift of maximum values of contamination with respect to the beginning of the year (φ).

Statistical tests:

Non-linearity parameter (NL) 10%

Relative contribution of seasonal variability (Fseas) 10%

Documents

Trend analysis: methodology