1

WLTP gtr Annex 9Determination of Method Equivalency -

Progress Report

► K. Kolesa, 2.6.2015

WLTP-11 23e

2

Situation

Conclusion of meeting WLTP #10:► Proposal for determination of system equivalence will be developed based on

provisions of gtr 4 and US Regulation 1065 .► It was requested to use definitions of ISO 5725► Proposal is expected to be presented in WLTP IWG meeting #11.

Meanwhile results:

► Experts have reviewed the ISO Standard 5725. The standard is complete but very specific and complex.

► General requirements can be applied. Details need a thorough statistical background.

► Experts discovered slightly different statistical formulae in gtr 4 and US Regulation 1065.

► General approach for Annex 9 was confirmed, but further work and validation on implementation prescription is needed

K. Kolesa, May 2015

3

Definitions in ISO 5725-1

K. Kolesa, May 2015

4

Basic Recommendations in ISO 5725-6 for comparison of alternative measurement methods (I)

Requirements for international standard measurement methods:► Applicable to a wide range of levels of characteristics► Equipment shall be available on an international basis► Cost of performing the measurement shall be acceptable► The precision and trueness of the measurement methods shall be acceptable

for the users of the results.► The comparison between methods shall be made on the results of precision

experiments on reference and candidate method. Both experiments shall be conducted in accordance with ISO-5725-2

► The experimental results should be able to detect if the ratio between the precisions of the compared methods are in a specified range

► The trueness of both method shall be proven equal involving identical samples by showing that the difference in mean values is statistically insignificant (e.g. 95 % confidence level)

► The difference between the expected values of the results of the compared methods, if available, shall be in a specified range

K. Kolesa, May 2015

5

Basic Recommendations in ISO 5725-6 for comparison of alternative measurement methods (II)

► The accuracy experiment shall be conducted in accordance with ISO 5725-1► The procedures for both methods shall be documented in sufficient detail so as to avoid

misinterpretations. No modification to the procedure is permitted during the experiment.► As precision of measurement methods is affected by the matrix of test samples

comparison is best done on identical test samples.► Required number of test samples varies depending on the range of the characteristic

levels of interest and the dependency of the accuracy on the level. ► The candidate measurement method shall provide the same or better precision under

repeatable conditions

K. Kolesa, May 2015

6

Proposal for Inclusion in GTR 15 (I)

Annex 9 Determination of method equivalency1. General RequirementUpon request of the manufacturer other measurement methods may be approved by the type approval authority, if it is found that they yield equivalent results in accordance with paragraph 1.1. The equivalence of the candidate method shall be demonstrated to and accepted by the authority.

1.1 Decision on EquivalencyA candidate method shall be considered equivalent, if the accuracy and the precision is same or better than the reference method.

1.2 Determination of EquivalencyThe determination of system equivalency shall be based on a correlation study between the candidate and the reference methods. The methods to be used for correlation testing shall be subject to the approval by the authority.

The basic principle for determination of accuracy and precision of candidate and reference methods shall applied following the guidance laid down in ISO 5725 Part 6 Annex 8 “Comparison of alternative Measurement Methods”.

1.3 Implementation requirements (reserved)

K. Kolesa, May 2015

7

Implementation Approach of Heinz Steven

Determination of System equivalency: ► Perform ncand measurements with the candidate system and nref measurements with the

reference system and compare standard deviations and averages with the F-test and the t-test. ncand and nref need to be equal or higher than 7 and ideally ncand = nref.

► 1.step: Check, whether the hypothesis σcand = σref is accepted or rejected. The counterhypothesis is σcand > σref, in which case the candidate method is not equivalent.

► If scand²/sref² <= F1-α(fcand; fref), then sigmacand = sigmaref is not rejected, then step 2 has to be applied. If it is rejected, the candidate system is not equivalent. fcand = ncand – 1, fref = nref – 1, ncand and nref are the sample sizes. α = 10%.

► 2. step: Check, whether the hypothesis µcand = µref is accepted or rejected. The counterhypotheses are µcand > µref and µcand < µref.

► If Abs(xave, cand - xave, ref)*Sqrt(ncand*nref/(ncand +nref))/s <= tf; 1- α, then µcand = µref is not rejected and the candidate system is equivalent to the reference system. If it is rejected, the candidate system is not equivalent.

f = ncand + nref – 2, α = 10%.

s = Sqrt(((ncand – 1)*scand² + (nref – 1)*sref²)/(ncand + nref – 2))► This is equivalent to the text in annex 4 of GTR #4, except for the first step, where it says

in annex 4 of GTR #4, but it needs to read K. Kolesa, May 2015

2minor

2major

ss

F 2ref

2cand

ssF

8

Conclusion and Proposal for further Proceeding

► IWG WLTP is informed about the on going work of alignment of gtr 4 and 1065 provisions with SAE 5725.

► For WLTP phase 1b it is proposed to include the drafted text for Annex 9

► In WLTP phase 2 the proposed implementation prescription should be validated by an expert group with statistical background to show effects for comparison of equipment and procedures

► IWG WLTP is asked to support this two step approach and to accept proposed text for inclusion in gtr 15

K. Kolesa, May 2015