Collaborative studies, what have we learned?

Some practical examples

Maciej Czechowicz, Wolf-Dieter Heller Institut für Tabakforschung GmbH (IfT)

Berlin, Germany

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Overview

• Generation of TNCO data over 10 years in the European Union Collaborative study (EUCS)

• TNCO data under two smoking regimes from the ISO-WG10 Collaborative study

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Studies which provide information

• CORESTA – RAC, SPA + ASR TF

• DIN-EUCS • ACS-ASIA study • ISO-WG10

– TNCO study – TobLabNet study

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

DIN – EUCS EUropean Collaborative Study on Cigarette Smoke Analysis

• Objectives Determination of the degree of agreement in measuring NFDPM, Nicotine and

Carbon monoxide in mainstream smoke among the participants under ISO machine smoking regime.

• Organised by:

Deutsches Institut für Normung e.V. (DIN) (German Institute for Standardization)

• Test Items: 4 brands of the German market (1 mg…10 mg tar) CORESTA monitor (CM4…CM7)

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

46

53

61

70 73

65 64 61

67 64

38 41

45

52

48 46

41 38

42 41

16 19 20

23

19 18 17 16 19

17

0

10

20

30

40

50

60

70

80

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nu

mb

er EUCS Participants 2005 - 2014

smoking devices Labs Countries

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Possibilities for serial comparisons:

1. Taking the results of the whole studies per year as elements of the time series

(mean, r, R, etc…)

2. Results of these labs which participated from 2005 on as single time series per lab

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Questions to the data set: 1. Do participants improve their performance

over time?

2. Do % outliers reduce as participant performance improves?

3. Risks in removing too many or too few outliers – how to mitigate for this?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Re

pe

atab

ility

( r

) EUCS 2005 - 2014

Repeatability (r) for Coresta Monitor (NFDPM Level 14mg/cig)

Nicotine CO NFDPM

CM-4 CM-4 CM-5 CM-6 CM-6 CM-6 CM-6 CM-7 CM-7 CM-7

Question 1:

r: small improvement for nicotine 20% or higher reduction for CO and NFDPM

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0

0.25

0.5

0.75

1

1.25

1.5

1.75

2

2.25

2.5

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Re

pro

du

cib

ility

(R

) EUCS 2005 - 2014

Reproducibility (R) for Coresta Monitor (NFDPM Level 14mg/cig)

Nicotine CO NFDPM

CM-4 CM-4 CM-5 CM-6 CM-6 CM-6 CM-6 CM-7 CM-7 CM-7

Question 1:

• Robust methods – R expressed as a percentage of mean yield is relatively low • between 10 and15 %

• Some indication of overall reduction in between laboratory variability for NFDPM and CO over time

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Re

pe

atab

ility

(r)

EUCS 2005 - 2014: NFDPM

Repeatability (r) for 4 Brands (A,B,C,D) and CM=E

A* B* C* D* E*

Question 1:

r : slight reduction for B,C,D,E A (low NFDPM) shows a clear reduction of more than 30%

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0

0.25

0.5

0.75

1

1.25

1.5

1.75

2

2.25

2.5

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Re

pro

du

cib

ility

- R

EUCS 2005 - 2014: NFDPM

Reproducibility (R) for 4 Brands (A,B,C,D) and CM=E

A* B* C* D* E*

Question 1:

R: clear reduction for E small reduction for D and A

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

• Charts 10 and 11 may give an slightly biased impression, as the participating labs are changing over time.

• The following charts demonstrate a more direct picture – why?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

21

14 13

15

10

7 8

10 11

14

0

5

10

15

20

25

1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 10 x

Nu

mb

er

of

La

bs

Multiple Participation in EUCS 2005 - 2014

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

5.75 5.84

5.48

5.71

5.92

5.61 5.63 5.72 5.70

6.00

5.00

5.25

5.50

5.75

6.00

6.25

6.50

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

NF

DP

M (

mg

/cig

.)

NFDPM Sample C (NFDPM level 6mg/cig.)

14 participants continously 10 years

• The labs follow in a direct manner the overall mean of the study (blue line) • Most labs are not always higher or lower as the overall mean • Some labs are consistently on one side of the overall mean

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0.52 0.52

0.49 0.49

0.53

0.48 0.48 0.47

0.49

0.53

0.400

0.450

0.500

0.550

0.600

0.650

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

Nic

oti

ne (

mg

/cig

.)

Nicotine Sample C (NFDPM level 6mg/cig.)

14 participants continously 10 years

• Similiar to NFDPM

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

7.04 7.26

6.67

6.95

6.66 6.73 6.52

6.81

7.32

6.87

5.50

6.00

6.50

7.00

7.50

8.00

8.50

2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

CO

(m

g/c

ig.)

CO Sample C (NFDPM level 6mg/cig.)

14 participants continously 10 years

• More labs than compared to NFDPM and nicotine are continously on one side of the overall mean

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Questions to the data set:

1. Do participants improve their performance over time?

2. Do % outliers reduce as participants performance improves?

3. Risks in removing too many or too few outliers – how to mitigate for this?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0.0000.0020.0040.0060.0080.0100.0120.0140.0160.0180.0200.0220.0240.0260.0280.030

lab

inte

rnal

SD

EUCS Nicotine lab internal SD

Sample A (NFDPM level 1,0 mg/cig)

2007 2010 2013

outlier outlier

outlier

Question 1, 2:

• Distribution of lab internal SD for three timepoints of the study • SD is reduced over time • Labs that were in the normal range in 2007 were outliers in 2013

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0SD

: la

b in

tern

al s

tan

dar

d

dev

iati

on

EUCS : NFDPM lab internal SD Coresta Monitor (NFDPM level 14,0 mg/cig)

2008 2010 2013

outlier

outlier

outlier

Question 1, 2:

• In 2013 the SD‘s were very dense up to 0.35 • All labs with SD > 0.35 were outliers.

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Cochran outlier test is based on this expression :

Nicotine Sample A (NFDPM level 1 mg/cig)

2007 2010 2013

0,0062 0,0040 0,0032

• The „density“ of the data in the core of the distribution is also important for being an outlier or not

• Is the Cochran test suitable due to the assumption of normality?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Outliers in general over the 10 years for all 5 samples :

NFDPM Nicotine CO

2005 2,67% 7,89% 2,22%

2006 1,17% 3,49% 1,17%

2007 2,46% 3,85% 2,12%

2008 4,79% 3,57% 1,82%

2009 1,98% 3,38% 4,25%

2010 3,16% 1,26% 3,16%

2011 2,84% 6,31% 1,58%

2012 3,72% 1,69% 2,03%

2013 4,83% 4,83% 2,42%

An improvement of r and/or R needs not to be related with a reduction in the number of outliers

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Questions to the data set:

1. Do participants improve their performance over time?

2. Do % outliers reduce as participants performance improves?

3. Risks in removing too many or too few outliers – how to mitigate for this?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

0

5

10

15

NFDPM

Nicotine

COTPM

WaterCochran

Grubbs single & double

Classical Z-score

Robust Z-score

Cochran Grubbs single &

double Classical Z-score Robust Z-score

NFDPM 6 2 2 3

Nicotine 15 3 4 8

CO 8 0 1 5

TPM 10 2 3 5

Water 14 1 2 6

Total 53 8 12 27

Outlier overview - EUCS 2013

• Clearly more Cochran outlier than location outlier • Z scores are suitable for analysis of location outliers (similar to Grubbs) • There were clearly more outliers obtained using z-scores. • Transforming data into Z scores may have an effect on the number of outliers

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Why „produce“ the robust z scores more location outliers in EUCS?

Where Z is the classical Z-score S = the standard deviation of a sample X = each value in the data set µ = mean of all values in the data set m = median norm.IQR = normalized interquartile range

In our examples the norm.IQR is an estimator which is clearly smaller than the relevant estimators in Z or in the Grubbs test – more outliers!

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Overview

• Generation of TNCO data over 10 years in the EUCS study

• TNCO data under two smoking regime : ISO-WG10 study

– 41 smoking machines (linear and rotary)

– ISO smoking and HCI smoking

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

25

30

35

40

45

3 4 6 11 14 16 17 20 22 23 25 26 28 30 32 35 37 38 49 50 1 7 9 10 12 13 15 19 24 27 29 31 34 36 41 44 45 46 47 48 51 52

TP

M (

mg

/cig

.)

Smoking machine / participant

ISO TC 126 WG 10 - Collaborative Study 2010 Intense smoking - TPM: Sample B

Linear smoking machines Rotary smoking machines

Two clear effects dependent on machine type: • For linear machines - higher TPM values and higher variability

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Do the TPM yields follow a normal distribution? Data from WG 10 Study using the HCI smoking regime for Sample B

Normal distribution statistic:

Kolmogorov-Smirnov : 0,195 significance: 0,000 Shapiro-Wilk: 0,895 significance: 0,001

Due to the inhomogeneity caused by the two machine types there clearly exists a bimodal distribution and a significant deviation from normality - effect on use of ISO 5725?

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

WG 10 Study: TPM using the HCI smoking regime for Sample B

Normal distribution statistic:

Kolmogorov-Smirnov : 0,136 p=0,200 Shapiro-Wilk: 0,919 p= 0,107

Considering only one machine type normality can be assumed for linear and …..

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

WG 10 Study: TPM using the HCI smoking regime for Sample B

Normal distribution statistic:

Kolmogorov-Smirnov : 0,086 P= 0,200 Shapiro-Wilk: 0,967 P= 0,651

……rotary machines

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

3.0

3.5

4.0

4.5

5.0

5.5

3 4 6 11 14 16 17 20 22 23 25 26 28 30 32 35 37 38 49 50 1 7 9 10 12 13 15 19 24 27 29 31 34 36 41 44 45 46 47 48 51 52

NF

DP

M (

mg

/cig

.)

Smoking machine / participant

ISO TC 126 WG 10 - First Collaborative Study 2010 ISO smoking - NFDPM: Sample B

Linear smoking machines Rotary smoking machines

Harmonisation of yields under ISO smoking – minimal machine effects

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

WG 10 Study: TPM ISO smoking regime Sample B

Normal distribution statistic:

Kolmogorov-Smirnov : 0,104 P= 0,200 Shapiro-Wilk: 0,965 P= 0,217

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

• The smoking machine type seems to be the major contributor to the between-laboratory variability of the intense smoking regime

• In contrast to the ISO smoking regime, extreme differences

can be observed between the two smoking machine types for most of the measured parameters – this leads to problems in using ISO 5725 for statistical evaluation as we do have “stratified” data populations.

Summary from ISO WG10 study

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A

Overall Conclusions • There are improvements in r and R over time in a more or less

stable study population like the EUCS study,. • The risk of removing too many or too few outliers clearly depends

on the type of study (number of participants etc.), the structure of the data distribution and the data itself being used for outlier determination (transformed data).

• The Cochran test may be not suitable for certain data sets due to his (strong) dependency on normality

• Taking other smoke constituents into account much higher variability must be expected (see results found by the Special Analytes group).

• This is especially the case if a study like EUCS would be enlarged to other smoke constituents.

2014

_ST

51_H

elle

r.pd

fC

ongr

ess2

014

- D

ocum

ent n

ot p

eer-

revi

ewed

by

CO

RE

ST

A