This content has been downloaded from IOPscience. Please scroll down to see the full text.

Download details:

IP Address: 134.151.40.2

This content was downloaded on 08/09/2014 at 04:26

Please note that terms and conditions apply.

Relevance of methods and standards for the assessment of measurement system

performance in a High-Value Manufacturing Industry

View the table of contents for this issue, or go to the journal homepage for more

2014 Metrologia 51 S219

(http://iopscience.iop.org/0026-1394/51/4/S219)

Home Search Collections Journals About Contact us My IOPscience

| Bureau International des Poids et Mesures Metrologia

Metrologia 51 (2014) S219–S227 doi:10.1088/0026-1394/51/4/S219

Relevance of methods and standards forthe assessment of measurement systemperformance in a High-Value ManufacturingIndustry

Pete Loftus and Seb Giudice

Rolls-Royce Plc, PO Box 31, Derby DE24 8BJ, UK

E-mail: sebastiano.giudice@rolls-royce.com

Received 7 March 2014, revised 15 May 2014Accepted for publication 28 May 2014Published 11 July 2014

AbstractMeasurements underpin the engineering decisions that allow products to be designed,manufactured, operated, and maintained. Therefore, the quality of measured data needs to besystematically assured to allow decision makers to proceed with confidence. The use ofstandards is one way of achieving this. This paper explores the relevance of internationaldocumentary standards to the assessment of measurement system capability in High ValueManufacturing (HVM) Industry. An internal measurement standard is presented whichsupplements these standards and recommendations are made for a cohesive effort todevelop the international standards to provide consistency in such industrialapplications.

Keywords: measurement uncertainty, uncertainty quantification, measurement integrity,measurement capability, standards

(Some figures may appear in colour only in the online journal)

1. Glossary

Measurement Umbrella term for Measurementquality (MQ): Traceability, Capability,

and Integrity.Measurement A set of activities, which

integrity assessment: involves evaluating the riskof not conducting asuccessful measurement. After themeasurement, the results are given aquality rating.

MSA: Measurement Systems Analysis.GR&R: Gauge Repeatability

and Reproducibility.PoD: Probability of Detection.FMEA: Failure Modes Effects Analysis.FEM: Finite Element Model.NMI: National Measurement Institute.

2. Introduction

The competitive nature of High Value Manufacturing (HVM)(Rose 2009) industries such as Aerospace and Energy requiresthat engineering limits are challenged. Challenging these lim-its can only be achieved with confidence if the measurementsystems used throughout the product life-cycle are dependable.This confidence is underpinned by Traceability, Capability, andIntegrity, and we have chosen to treat these concepts under theumbrella term of ‘Measurement Quality’ (MQ). InternationalStandards and Best Practice guidance can support this quest forMQ, but the framework of standards is not yet comprehensive.Its practical implementation in systems that can be both com-plex, and complicated, gives rise to some challenges not yetfully addressed by international standards and, in some cases,not yet soluble. For example, ISO 10012 describes the man-agement framework required to ensure that measurements per-formed in industry are valid. This requires that industry itself

0026-1394/14/040219+09$33.00 S219 © 2014 BIPM & IOP Publishing Ltd Printed in the UK

Metrologia 51 (2014) S219 P Loftus and S Giudice

provide the assurances that, in the consumer setting, are pro-vided by the state through legal metrology. However, individ-ual businesses must define the way in which they will complywith the standard and there are many implementations with dif-fering degrees of rigour. Similarly, ISO GUM (2008) describesthe use of measurement uncertainty analysis (MUA) but spe-cific measurement challenges may be beyond its scope andsome of these require the development of novel methodology.

This paper explores the relevance of measurementstandards to HVM industry. It first describes the operationalcontext in which measurement activities are conducted. Itthen evaluates the current provision of relevant standards anddemonstrates the need for their development. The paperhighlights where the provision of future standards wouldbenefit the industry.

3. Operational context

To appreciate the relevance of measurement standards toHVM industry it is necessary to consider the purpose of,and the environment wherein, measurements are conducted.Measurement data underpins the whole life-cycle of highintegrity products such as aircraft, ships, and power generatingplant. Measurement supports such functions as:

• Research, development, and testing of new technology.• Research, development, and testing of new materials.• Engine tests to confirm that new designs agree with

predictions (design validation).• Inspection to prove that a manufactured part or

product conforms to a design specification (conformityassessment).

• Controlling the product and monitoring its health whilstin service.

For each measurement activity, the ‘Quality’ of ameasurement system needs attention whether conductingroutine measurements with established systems, acquiring an‘off-the-shelf’ standard measurement system, or developing acustom measurement system.

In this paper, we use the term ‘Measurement Quality’to describe a set of concepts that provide confidence in themeasurement data used to inform engineering decisions. Itcomprises Metrological Capability, Measurement Integrity,and Measurement Traceability. The latter is better understoodthan Capability and Integrity.

Measurement Capability is the term being used torepresent those methodologies that allow the user to know ifthe Measurement System is ‘fit for purpose’. Within this groupof methodologies, it is important to make a distinction betweenthose methods which provide information associated with theresult of a measurement, and those methods which are usedto characterise the performance of the measurement system.This is considered an important distinction. MeasurementUncertainty typically describes the former, whereas examplesof the latter could be Probability of Detection (PoD), usedin Non-destructive testing practices, or Gauge Repeatabilityand Reproducibility (GR&R) often used in manufacturingenvironments. GR&R is used to estimate how much of

Measurement Quality

Measurement Traceability Assessments

Calibration

Measurement Integrity Assessment

FMEA Data flags

Measurement Capability Assessments

MUA MSA

Round Robin

PoDGR&R

Figure 1. Activities in Measurement Quality.

the observable manufacturing variability originates from theMeasurement System. However, as will be discussed, thereare opportunities to use GR&R for an uncertainty assessment.

Measurement Integrity refers to a set of activities donebefore and after the measurement is performed: Before themeasurement, the risk of using the measurement system isestimated prior to investing in its acquisition or commencingits operation. Following the measurement, checks are madeon the quality of the data that has been collected. This conceptis also discussed in greater detail later on in this paper.

The Venn diagram in figure 1 illustrates the relationshipbetween the concepts we refer to.

To show how measurements conducted in practice mightdiffer from those typically considered by the internationalstandards, and to provide an illustration for discussion, weuse an example of some measurements required to design anaero gas turbine compressor blade. This example is chosen toillustrate a typical industrial measurement problem where themeasurand is not directly accessible, and where the deliveryof the measurement result depends on multiple processesconducted on different global sites, and integration with themodelling discipline.

3.1. Vibratory stress analysis

When developing new compressor blades it is necessary toestablish that the stresses imposed on the blade by vibrationare acceptable for long-term operation. Whilst in service, theblade can vibrate in a number of different modes e.g. 1st, 2nd,3rd cantilever, 1st, 2nd, 3rd torsional and higher-order, morecomplex modes. The measurand is the stress which limitsthe life of the blade in service, i.e. the maximum principalstress in the component. This is essentially a complex exampleof conformity assessment as the maximum stress that can betolerated varies with the operating condition of the engine.More stress can be tolerated during engine acceleration anddeceleration than during continuous running that accumulatesvastly more cycle counts. Moreover, more stress can betolerated in lower frequency vibration modes for the samereason. The measurement method evolved by the industry is asfollows: Finite element analysis provides a reasonably faithfulmodel of these mode shapes. Informed by these models,locations on the blade will be chosen to bond in place one or twostrain-gauges. The location will be a compromise between the

S220

Metrologia 51 (2014) S219 P Loftus and S Giudice

Probe measures tipdisplacement

Blade motion

Strain Gauges

Location of maximum stress may benot accessible i.e. inside the blade

Shaker Table vibratesthe blade

Figure 2. Set up for vibratory stress analysis experiments.

need for measureable strain levels in all modes of interest, andthe need to maximise gauge life. The gauges will be bondedin place and the blade mounted on a shaker table to subjectthem to vibration generating the mode shapes of interest. Theset-up is shown in figure 2.

In each resonant mode, the ratio of the strain indicatedby the strain gauges to the blade tip displacement, measuredoptically, is calculated. These experiments are conducted in anenvironmentally controlled laboratory at room temperature.

The finite element model (FEM) is then used to relatethe maximum principal stress in the blade under operatingconditions of temperature and centripetal acceleration, to thetip displacement. Other elements of the system, such as thepolarization and data acquisition, are calibrated separately bydifferent groups of people; and corrections may be appliedfor differences in the operating temperature of the gauge andthe resistance of the cabling using data produced by differentpeople again. The strain-gauge output during the engine test isthen related back, with corrections applied, to the validatedFEM, and the maximum principal stress in the blade iscalculated. Therefore, the measurement system here includesthe FEM as well as the gauges, polarizing, and data acquisition.

However, before the engine test, the measurement systemshave to survive the engine build processes and engine run-ning. As temporary installations, reliability is compromisedto deliver the minimum impact to the engine operation. Itis relatively common therefore that, despite all precautions,some gauges will be damaged. Depending on priority andtimescales, not all gauges will be repaired and the overall dataquality will therefore be lower than intended by the experimentdesign.

So this environment is very different to that encounteredin a traditional scientific laboratory. In a laboratory, the personconducting the measurement might know the purpose of themeasurement, he/she might have selected the measurementinstrument, therefore he / she will have some level of familiarityand appreciation for its capability. In the example above, the

use of standards to control all of the diverse activities requiredof numerous staff in different parts of the organization becomesessential. In order to complete the conformity assessment, themeasured results, their uncertainty estimates, and an allowancefor the integrity of the measurement (e.g., if some gauges havefailed the ability to average results is compromised) needs tobe considered.

4. Governing measurement standards

A key element in instilling Measurement Quality practiceas a routine is by mandating conformity to internationalstandards. Traceability of instruments can be controlledby demonstrating conformance to ISO 17025 (2005). Thismandates the use of measurement uncertainty analysis incalibration laboratories. But, for Uncertainty and Integrityof the end-to-end measurement chain, there are less clearopportunities. For example, ISO 9001 (2008), is the universalstandard that many companies subscribe to. This states:

The organization shall establish processes to ensure thatmonitoring and measurement can be carried out and arecarried out in a manner that is consistent with the monitoringand measurement requirements.

Here ISO 9001 suggests that a measurement process andrequirements for it exist. However, this text has limited valueto the measurement community. What are the measurementrequirements? How does one show conformance to theserequirements? The statements in ISO 9001’s are open topersonal judgement and can therefore be prone to comprises.This limitation often means that this standard is of limited useto protect the quality of the measurement systems. This hasbeen the motivation for the Aerospace industry to develop theNADCAP (2013) checklists. This goes a little further than ISO9001 as its audit checklist includes, for example, the followingquestions.

7.1.2 Does the documented procedure address the useof activities to demonstrate capability of the measurementsystem?

S221

Metrologia 51 (2014) S219 P Loftus and S Giudice

Compliance Assessment Guidance: Examples couldbe AIAG Guide to Measurement System Analysis (MSA),ISO14253-2, customer defined processes such as gageRepeatability and Reproducibility (R&R), attribute agreementanalysis, etc.

What makes this checklist valuable is that it beginsto constrain the interpretation of the phrase ‘demonstratecapability’, as the guidance points towards statistical methodssuch as Measurement Systems Analysis and MeasurementUncertainty Analysis. The authors consider this a welcomeimprovement to the provision of standards, albeit confinedto conformity assessment in manufacturing. There are someexamples in the existing international standards which are quitespecific. For example, ISO 10012 (2003) specifies that:

The measurement uncertainty shall be estimated foreach measurement process covered by the measurementmanagement system (see 5.1).

Uncertainty estimations shall be recorded. The analysisof measurement uncertainties shall be completed before themetrological confirmation of the measuring equipment, andthe validation of the measurement process.

All known sources of measurement variability shall bedocumented.

Achievement of ISO 10012 represents, for many, the idealapproach to any measurement activity: Every measurementdelivered with a measurement uncertainty estimate. Thisstandard leads the accomplishment of one pillar supportingMeasurement Quality.

The above are only examples of the available measurementstandards framework but serve to illustrate the potential for alack of consistency. The following section aims to highlightthat choice of a method to assess measurement capability isoften dependent on operational context. This means therehas been limited consideration of the requirements of themeasurement.

5. Measurement quality assessment methods

Given the diverse nature of the measurement functions inHVM, there are various measurement capability assessmentmethods. Notably, Measurement Systems Analysis (MSA),MUA, Inter-laboratory comparisons (Round Robins) andPoD. We now aim to briefly describe how each of thesemay contribute to the assessment of measurement systemperformance:

5.1. Measurement systems analysis

MSA is a family of techniques used to assess the capability ofthe measurement system in the context of the manufacturingprocess performance. MSA helps make a clear distinctionbetween production variation and measurement variation. Themethod is prevailing in manufacturing environments such asthe automotive industry as its principles are supported byThe Ford Motor Company; General Motors Corporation; andChrysler Group LLC. These companies are responsible for theAutomotive Industry Action Group’s (AIAG) which publishesthe ‘Measurement System Analysis (MSA) Manual’. As

for national standards, ASTM (2011) has published a usefuldocument. These techniques aim to determine how muchof the observed manufacturing variability originates from themeasurement system.

MSA is part of the Six Sigma approach which iswidespread within many manufacturing industries. MSA’spopularity has led to the provision of easy to use applicationsand templates. Dedicated statistical software packages cannow support the user in each step of the analysis. For example,these will provide the Design of Experiments (DoE), and thedata entry template. They will then analyse the data, andpresent it in a recognized format. Such tools enable non-experts to treat the measurement system as a ‘black-box’, butstill report a level of capability.

One of the techniques, within the MSA toolkit, usedto characterize the measurement variation is called GaugeRepeatability and Reproducibility (GR&R). This focuseson understanding ‘Repeatability’ (how well measurementinstrument repeats without change to the conditions of use)and ‘Reproducibility’ (how well the measurement could bereproduced with foreseeable variations in the conditions ofuse). It makes use of DoE which leads to standard deviationscalculated using the ANOVA method. These standarddeviations are summed in quadrature (root sum squares (RSS)),resulting in a ‘combined’ standard deviation. This is multipliedby a factor to represent multiple standard deviations (usually 6).The resulting normal probability distribution, based in themeasured value and the above standard deviation, is comparedto a tolerance zone. If the width of the distribution consumestoo much of (often defined as >20%) the tolerance, then themeasurement may be deemed not fit for purpose.

The idealized intent of MSA is to capture all sourcesof variability, but it can be viewed as a snap-shot in timeof the performance of the measurement system, and can beexpensive to thoroughly implement. For these reasons, onemight omit the inclusion of stability and linearity and replacethese with a simple bias check: a comparison with an artefactor a reference measurement made with a measurement systemof better accuracy.

5.2. Measurement uncertainty analysis

The assessment of measurement system capability for producttesting has evolved differently than it has in manufacturing.Archived reports, within Rolls-Royce, show the use of MUAbefore the deployment of the ISO GUM in 1993 (ISO2008). Here references to Abernethy (1969) methods areoften cited. He was active in the application of statisticsin aerospace, specifically uncertainty and reliability methods.For example, Abernethy and Ringhiser (1985) highlight themotivations for the use of summation in quadrature. It alsodescribes how the methodology was commissioned by theInter-agency Chemical Rocket Propulsion Group (ICRPG)in 1965, who then approved Abernethy et al’s subsequentproposal (Abernethy 1969). Today, however, the ISO GUM isthe authorative text for MUA. Other standards have evolved,such as ASTM E2655 (2008) and ASME PTC 19.1 (2005) andthese appear ‘in harmony’ with, and reference the ISO GUM.

S222

Metrologia 51 (2014) S219 P Loftus and S Giudice

5.3. Other methods

Inter-laboratory comparisons, known as Round Robins, areoften used to evaluate potential measurement technology.Round Robins are often used by the NMI community as ameans of comparing measurement capabilities. In our context,the ‘round robin’ might be used to characterize the typicaluncertainty of a type of measurement system. Here equivalentspecimens will be sent to participating laboratories, who areasked to make measurements using a specific method. Theprocess is described in ISO 5725-2 (1994) and ASTM E691(2011). GR&R and the methods prescribed in ISO 5725 areoften thought to be the same (Deldossi and Zappa 2009),as ISO 5725 calculates ‘repeatability’ and ‘reproducibility’.However, in this context there is a difference with theGR&R methods described in the AIAG. The AIAG assumesthat reproducibility and repeatability are quantified usingexperiments conducted on a specific measurement system ina specific location by a group of operators all trained in thesame manner. For the methods described in ISO 5725-2, therepeatability and reproducibility are a result of completelydifferent laboratories taking part in the study. The RoundRobin is typically used to select measurement systems forthe characterization of materials. Often however the levelsof repeatability and reproducibility are unacceptably high. Sothis activity is usually used to select a specific laboratory withwhich Measurement uncertainty is applied. A Round Robincan also be a check for integrity, which is discussed later on.

In Non-destructive testing, the methods of PoD arepopular. In principle, this is similar to GR&R and capturesthe statistically accessible uncertainties in detection of defects.Like GR&R, it neglects sources of uncertainty not amenableto statistical treatment.

6. Comparisons of methods

There is reason to suggest that the choice of method mightbe dependent on convenience and the educational backgroundof those responsible rather than a meticulous study of whatis required. For example, the development of measurementsystems for engine testing tends to attract people from aphysics background comfortable with statistics Manufacturingon the other hand might attract more engineers. For manyUK engineers statistics is not part of the core undergraduateengineering curriculum. Instead, it might be an optionalmodule meaning that industrial training has to fill this gap.This often comes in the form of six sigma principles; hence,they are more likely to implement MSA rather than MUA.

This method however suffers from a drawback as it isonly a snap-shot in time of the capability of the measurementsystem and only caters for those sources of uncertaintywhich can be captured experimentally and treated statistically:Repeatability and Reproducibility. The holistic approach doesinstruct the user to consider other sources, drift, linearity,etc. . . . but incorporation of these additional sources couldrequire additional testing, which may become expensive.

On the other hand, despite the ISO GUM’s 20-yearexistence, its established heritage and its wide-ranging

applicability, there are still barriers to its widespreadimplementation outside the laboratory environment. Theperceived complexity of MUA is often cited as the causeof its lack of popularity. Existing standards and guidancedo not lend themselves to easy reading and deter peoplefrom further involvement. Given this lack of popularity,a benefit of MUA is overlooked: a good MUA requiresthat many sources of uncertainty be considered, not justrepeatability, reproducibility. The economic advantage is thatthe magnitude of these uncertainties can be estimated fromhistorical evidence, or engineering judgement (a priori). Thismight reduce the reliance on expensive experiments as withGR&R. To achieve this however, one must trust that all sourcesof uncertainty have been accounted for. This assurance iscommensurate with the experience of the person or peopleinvolved in the MUA activity. An experienced engineer mightsuggest a different set of uncertainties when compared tosomeone new to the measurement system.

Many would argue that GR&R and MUA have differentintents. For the former it is to ascertain the degree ofvariability caused by the measurement system. For thelatter, it is to determine the likely quantity values thatcould have given rise to the measured value. Despitethese differences, there are similarities that should not beoverlooked: (1) identifying the causes of variability (orsources of uncertainty), (2) combination using RSS, and (3)‘expansion’ to provide the desired coverage probability. ForGR&R this involves multiplying the combined uncertainty byintegers such as 4 or 6, assuming normality, to get the 95% or99.7% confidence intervals respectively. Whereas the GUMrecommends that this multiplier be derived using the combineddegrees of freedom, calculated using the Welch–Satterthwaiteformula. This is usually a value close to 2 or 3 to get the 95%or 99.7% confidence intervals respectively. The multiplier forGR&R is twice that used for an uncertainty analysis, as itdescribes the width of the confidence (or coverage) intervaleither side of the mean value. Whereas a reported uncertainty,describes the difference between the width of the confidenceinterval in one direction, from the mean value.

These similarities may have led to the logical progressionof blending the two methods. ISO 22514–7 (2012)demonstrates how to take data collected for GR&R, andanalysed using the ANOVA method, as estimates of therandom contributors to uncertainty. It then demonstrateshow to add other sources of systematic uncertainties. Ittherefore provides a means of using existing data, collectedto characterize the variation from the measurement system,to infer the spread of the value that could be reasonableattributed to the measurement results. This seems a logicaland justifiable instruction, given that the MUA methodologyallows for uncertainties to be estimated using methods, otherthan experimentation, such as modelling, historical evidence,and engineering judgement. Therefore it can be assumed thatthe use of GR&R data is being used in the MUA framework as a‘priori distribution’, which quantifies the random uncertainty.However, the standard claims that the method should not beused for ‘complex’ measurement systems (although a cleardefinition of this is not provided).

S223

Metrologia 51 (2014) S219 P Loftus and S Giudice

Ensure Capable Measurement Systems

Ensure Calibration Perform Measurement Process Measured Data Ensure Data Quality

Define Requirements Design SolutionEstimate Measurement

QualityValidate Measurement

Quality

Main process

Sub-process

Measurement Uncertainty and Integrity

Analysis occurs in

these steps

Figure 3. Quality Management Standards for Rolls-Royce.

Within MUA itself there is differentiation with regardsto how the method is implemented. For example, the MUAstandard for engine testing is ASME PTC 19.1. This makesthe distinction between Systematic and Random contributorsof uncertainty. With this method, systematic and randomuncertainties are reported separately and then together. Thereasons often cited for this are that the terminology has moremeaning compared to type A and B distinctions, which aresuggested by the ISO GUM. Another example can be drawnfrom the world of chemistry. Here the guide by Eurachem ispreferred (Eurachem 2012). This is a ‘top-down’ approach,which is best suited for measurement systems where there is alack of understanding of a measurement model. FurthermoreEurachem encourages the causes and effects of uncertainty tobe studied, so as not ‘double-account’ when gathering data.

We argue that many of these fragments of best practice,from guides assumed to apply to specific measurementsystems, can in fact be generalized. For example, thestudy of cause and effect, promoted by Eurachem couldbe universally adopted, as there are many other occasionswhere the measurement model might not be fully understood,especially when ‘off-the-shelf’ measurement systems are used,so a ‘top-down’ rather than ‘bottom up’ approach might bemore suited. Likewise, the limitations of the approachesdescribed in ISO 22514–7 needs to be explored. Hence, webelieve that a review of all capability assessment methodswould lead to robust guidance, which could be generalized andexploited for general measurement applications. As an interim,the authors’ company has developed an internal standard toalleviate some of the confusion with external standards. Thisis discussed in the next section.

To sum up the challenge of applying the currentinternational standard landscape for these applications, theassessment of measurement quality is more difficult to achievethan in the laboratory because of the number of people andprocess steps involved, yet the skill level in metrology of thoseresponsible is often lower than the responsible person in alaboratory. If we are to facilitate valid measurement in thesecircumstances, the standards must evolve to guide users in asystematic approach to these complex measurement systems.

7. An internal measurement standard

One way of controlling measurement activities in anorganization is to introduce internal standards that make upthe ‘Quality’ Management system. These are in essencea set of governing procedures or rules, which a companymandates in order to deliver objectives for the business suchas enhanced customer satisfaction and reduced operatingcosts. Measurement, as other activities, can be controlledwith a standard which describes the process by which allmeasurements will be conducted. The authors have beeninstrumental in creating such a procedure, summarized infigure 3. The implementation of the procedure is monitoredthrough regular audits that ensure compliance. Identified non-conformance usually results in improvement projects or, inworse cases, ceased activities until compliance to the procedurecan be demonstrated.

Figure 3 outlines the Measurement Process recentlyimplemented by Rolls-Royce.

For routine measurements on systems where capabilityhas been established, only the main process is required. Forexample, this might include the inspection of a part followinga manufacturing process. If, however, a new measurementsystem is required to meet new requirements, then the secondprocess is called for. Key to this second process is the needto estimate MQ. This is done to make informed capabilityacquisition and experiment design decisions. Estimates of MQare often based on engineering insight (type B) only. Once itcan be proven that estimates of MQ suggest that the newlydesigned measurement system could be fit for purpose, thesystem is built, or purchased, and MQ is then validated usingtype A and type B data. For the validation of the measurementsystem, two methods are permitted. The first is MSA formanufacturing, and the second is MUA for the rest of themeasurement functions.

The power of this procedure comes, not in spelling outprocess steps that may seem obvious to the metrologist, butin mandating that measurements cannot be conducted withoutall the elements of metrological control being in place andvalidated.

This high-level procedure cannot provide the detailnecessary to demonstrate the capability of a measurement

S224

Metrologia 51 (2014) S219 P Loftus and S Giudice

system. For this reason, a lower-level document provides theguidance on how to conduct an assessment of measurementsystem performance. This makes use of all of the externalguides and standards. It sets principles and presents theseindependent of the type of measurement application. Thisshould remove the need for others to seek out guides andimplement their own interpretation. Thus, the provisionof one guide provides a series of robust approaches thatensure conformance and exceed the requirements of governingstandards such as ISO 9001.

However, to further develop this lower-level document,guidance from the international measurement community isneeded. The following section suggests areas of focus neededto ensure that external standards complement modern-dayHVM measurement activities.

8. Areas where standards could further support theuse of measurement in HVM

The current provision of international standards needs toadvance to cater for future measurement needs. The authorsconsider that following areas would benefit from attention fromthe international measurement community to ensure rigor is notcompromised.

8.1. Measurement uncertainty and uncertainty quantification

‘Uncertainty quantification’ (UQ) is used to quantify theuncertainty in data derived from the use of computationalmodels. Many would argue that measurement and modellingare separate, and that the measurement community shouldfocus its efforts solely on the measurement. However, as shownin the example of vibratory stress analysis, models can form anintegral part of the measurement chain. Therefore, we believethat metrological traceability requires the uncertainty analysisof software systems and models that are used as part of themeasurement model. Guidance is therefore needed for the‘calibration’ and validation of computational models.

As we look to harmonize UQ and MUA, it is likely thatlearning might be transferred from one discipline to the other.For example, UQ categorizes uncertainties as ‘aleatory’ or‘epistemic’. The former is used to describe uncertaintieswhich are well understood and can be described by a givendistribution. Type A or B uncertainties are likely to bein the Aleatory category. The latter refers to a lack ofknowledge. MUA however focuses solely on the provision ofa statement based on the ‘available information’ (ISO GUM).Would MUAs benefit from the consideration of epistemicuncertainties? Would this approach prompt measurementpractitioners to develop better understanding of a measurementsystem? Alternatively, is it best placed in a MeasurementIntegrity Assessment as advocated later in this paper?

8.2. Back-to-back testing

Often through development of a product, back-to-back testingis conducted. For example, an engine will be put on test, andits specific fuel consumption measured. Following the test, the

engine’s turbine blades might be removed and replaced withones with a new geometry. Then the engine is tested again, inorder to establish the influence of the blade design standard onthe specific fuel consumption. In these cases, it is necessaryto assess the residual uncertainty once the biases addressed bythe experiment design are removed. These biases will varydepending on the detail of the implementation. For example,if the Test Facility instrumentation is recalibrated between thetests, an additional set of biases is introduced. Tracking therelevant and irrelevant biases presents considerable difficultywhen one considers a typical development engine test with2000 measuring instruments, which may have 20–30 sources ofuncertainty each. Uncertainties that are negligible in absolutemeasurements become non-negligible in back-to back testingand vice-versa. Challenges therefore exist in clarifying thebest approach to these problems and in designing systems tomanage the estimation of uncertainty.

8.3. Uncertainty for system design

Very often, the perceived benefits of implementing MUA arelimited to the provision of an index of quality of a measurementwhich has taken place, hence we report measurements withtheir uncertainty. However, there is also a need to be able topredict the uncertainty of the measurements to ensure that thesystems built, commissioned, and regularly used will be fit forpurpose. It is on these occasions that it is necessary to conductan estimate of future uncertainty.

Recognition of the use of MUA prior to the creation ofthe measurement system is sparse in the standards. ASMEPTC 19.1 briefly discusses this, and AIR5925 (SAE Aerospace2007) shows how conducting estimates can offer financialbenefits. Beside these two sources, there is limited informationin the public domain. This makes it difficult to produce astandard that would confirm the validity of data sources used.A way to achieve this might be to use modelling, data fromcalibrations, and historical evidence. We also need to betterdefine the rules for ‘reading-across’ known capabilities fromone measurement system to a similar one. Finally, we requireguidance on elicitation (O’Hagan and Oakley 2004) for useof type B uncertainty. The current approaches would benefitfrom enhanced structure especially in proving provenanceof estimates based on engineering judgement. Examplesof this can be found in other methodologies e.g. by ENIQ(2010). Here, the term Technical Justification (TJ) is usedto represent the body of evidence that supports motivationsfor test methodologies. TJ includes previous experience ofconducting the experiments; experiment data; mathematicalmodelling; an (O’Hagan and Oakley 2004) physical reasoning.Such approaches could eventually be synonymous with use ofdata to support type B estimates of uncertainty.

The capability assessment of a sensor network also needsfurther consideration in the standards. A network of sensorsmay be used to derive the measurand, which might notbe measurable, For example, temperatures too extreme forthermocouples. In these cases, a network of thermocouplesmight be placed in the surrounding region and the temperatureof the measurand inferred from the results. Each sensor in

S225

Metrologia 51 (2014) S219 P Loftus and S Giudice

the system might be treated as a separate measurement chainwith its own uncertainty. However, does the use of multiplesensors to infer a measurement lead to a lower uncertainty?The spatial distribution of the sensors might lead to greatervariability in the measurements, but surely multiple sensorsincrease our confidence in the results. Guidance is needed onhow to deal with inferred measurands such as the validity ofcalibrations and uncertainty statements for the whole network,not just for the individual sensors.

8.4. Sources of uncertainty

Another opportunity for improvement is the standardization ofmanufacturer’s specifications. In designing or implementingnew measurement systems, the engineer will draw upon alibrary of components. The engineer choosing componentswill make his/her decision on cost and so-called reportedperformance characteristics ‘accuracy,’ ‘linearity‘, ‘stability’etc. Unfortunately, each manufacturer will report variousvalues, which reflect the ‘capability‘ of their products.Usually these figures are chosen to show the product ina good light. Being able to request information as peran international standard would greatly decrease the degreeof effort required in quantifying uncertainties. Thesestandardized lists would require that manufacturers abide tocertain reporting requirements (i.e. set coverage probability),but they should also ensure that the main sources of uncertainty(such as precision, linearity, drift, hysteresis, resolution,calibration uncertainty and sensitivity to temperature) are listedand quantified. The provision of this information would enablecomponent libraries to be built up and uncertainties to bequantified, pre and post measurement, with less effort. Thesestandardized lists will also need to be supported by decisionbased logic rules, which guide the user towards the adequateselection of sources of uncertainty. ISO 14253 (BSI Standards2011) provides comprehensive lists of sources of uncertaintythat might affect dimensional measurement as does Eurachem(Eurachem 2012) for chemical measurements. These lists ofuncertainty are very useful for the less experienced engineer,but they there opportunities to improve these as well. MUguidance is needed to help the user choose the sources ofuncertainty.

9. Measurement integrity

MUA assumes that the measurement system used is as per theagreed design intent between the ‘provider’ of the system andthe ‘user’ of the results. Hence it assumes that the measurementsystems used fulfil the requirements of the user of the data. Yetfinite reliability of measurement equipment and the fallibilityof its operators mean that deviations from the design intentcan occur. In some cases, the harsh environments withinwhich the measurements are taken, and operational pressures todeliver data quickly and without impact to the operation, meanthat these deviations from the design intent can be relativelyfrequent. In other cases, the extreme importance of somemeasurement systems (with safety related roles and with higheconomic consequence of failure) makes it necessary to be able

Table 1. Data flagging criteria.

Flag Description

Incomplete The check has not been completedPass The system passes the checkFail The system failed the checkSuspect The data points between a ‘Pass’ and

a subsequent ‘Fail’ are called ‘Suspect’

to describe and control the probability of failure. Managingthe risk of using measurement systems, and the subsequentresult provided, is also a key requirement for ISO 10012. Thisis emphasized in the following extracts from the internationalstandard:

‘the risks and consequences of failure to comply withmetrological requirements shall be taken into account’.

The choice of elements and control limits shall becommensurate with the risk of failure to comply with specifiedrequirements.

Given this, the principles of ‘Measurement Integrity’ arebeing introduced. This refers to a set of activities donebefore and after the measurement. Before the measurement,the risk presented by the measurement system is estimatedbefore investing in its acquisition or commencing operation.Following the measurement, checks are made on the qualityof the data that has been collected.

We have defined ‘Measurement Integrity’ as:

‘The degree of belief that there are no deviations from theagreed design intent for the measurement system’

To quantify this degree of belief, we have turned toreliability methodology and developed a customized failuremodes and effects analysis (FMEA), which estimates residualrisk. This approach was defined as follows:

During the measurement system, design activity, systemchecks are incorporated to provide metrological confirmationand increase system integrity. The FMEA quantifies thelikelihood of a failure of the measurement system and itsconsequences. It also helps test the operation of these integritychecks and may highlight opportunities to improve the designwith further checks. Many engineering teams and operationsalready use risk evaluation techniques such as FMEA, or holda risk register. It is therefore envisaged that the informationrequired to evaluate risk with the measurement system wouldbe additional to these original activities. The FMEA activityleads to a recommendation of the types of checks that need tobe conducted during the measurement system installation andafter the measurement has taken place. The outcome of thesechecks results in one of four flags, shown in table 1, which aresaved with the measurement data and stay with the data fromthat point on. This ensures that future users of the data areaware of potential issues that might have arisen at the time themeasurement was taken.

By including Measurement Integrity in the internalmeasurement standards today, provides a formal and structuredframework for managing risks. This will result in additionaldata being be collected and saved, which can then be analysed

S226

Metrologia 51 (2014) S219 P Loftus and S Giudice

for future product development programmes to objectivelyquantify the risk of a measurement failure, and therefore makeinformed engineering decisions.

10. Conclusion

We have outlined some of the diverse ways in whichmeasurement supports R&D, manufacturing, and operationsin High Value Manufacturing Industries. All of these activitiesare founded on knowledge, about the products, derived frommeasurement. This knowledge is increasingly dependent,as much on computational modelling and simulation, as itis on measurement, but is increasingly critical to business.Therefore, Industrial metrologists need to deliver, in the firstcase, measurement systems with sufficient integrity for the taskat hand (making high-quality products)

This environment calls for further development of thestandards and controls applied to measurement to deliver theconfidence needed by businesses. Some of the steps taken, andplanned, towards this goal have been described and remaininggaps identified.

Specifically, we advocate:

1. A process description of measurement that can be used tomandate metrological control.

2. Recognition that confidence in a measured result depends,not only on its uncertainty, but also on its integrity.

3. Development of appropriate validated methods for theperformance assessment of: complex measurementsystems including those reliant on modelling; complexexperiments; and complex product design.

It is the authors’ judgement that the goal of confidence in allof our measurements will only be achieved comprehensivelyby the combined efforts of the international metrologycommunity and analysts, experimentalists, system designers,and manufacturing engineers in industry.

References

Abernethy R B 1969 ICRPG Handbook for Estimating theUncertainty in Measurements Made with Liquid PropellantRocket Engine Systems s.l. JANNAF (formerly ICRPG)Perforamance Standardization Working Group Reporte CPIANo.180 (AD 851 127)

Abernethy R B and Ringhiser B 1985 The History and StatisticalDevelopment of the New ASME-SAE-AIAAISO Measurementand Uncertainty Methodology (New York: AIAA)AIAA-85–1403, pp 1–9

AIAG 2012 Measurement Systems Analysis 4th edn (ChryslerGroup LLC, Ford Motor Company, General MotorsCorporation)

ASME PTC 19.1, 2005 Test Uncertainty (ASME International)ASTM E2655, 2008 Reporting Uncertainty of Test Results and Use

of the Term Measurement Uncertainty in ASTM Test Methods(ASTM International)

ASTM E2782-11, 2011 Standard Guide for Measurement SystemsAnalysis (ASTM International)

ASTM E691-11, 2011 Standard Practice for Conducting anInterlaboratory Study to Determine the Precision of a TestMethod (ASTM International)

BSI Standards, 2011 BS EN ISO 14253-2:2011 GeometricalProduct Specifications (GPS) —Inspection by Measurmeent ofWorkpieces and Measuring Equipment (London: BSI)

Deldossi L and Zappa D 2009 ISO 5725 and GUM: comparison andcomments Accred. Qual. Assur. 14 159–66

ENIQ 2010 ENIQ Recommended Practice 2, Strategy andRecommended Contents for Technical Justifications, Issue 2(Petten: Institute for Energy)

Eurachem 2012 Quantifying Uncertainty in Analytical Measurement3rd edn

ISO 10012, 2003 Measurement Management System—Requirementsfor Measurement Processes and Measuring Equipment (BritishStandards Institute)

ISO 14253-, 2011 Geometrical Produce Specifications(GPS—Inspection by Measurement of Workpieces andMeasuring Equipment. Part 2: Guidance for the Estimation ofUncertainty in GPS Measurement, in Calibration of MeasuringEquipment and in Product Verification (British StandardsInstitute)

ISO 17025, 2005 General Requirement for the Competence ofTesting and Calibration Laboratories (British StandardsInstitute)

ISO 22514–7, 2012 Statistical Methods in ProcessManagement—Capability and Performance. Part 7: Capabilityof Measurement (British Standards Institute)

ISO 5725-2, 1994 Accuracy (Trueness and Precision) ofMeasurement Methods and Results (British Standard Institute)

ISO 9001, 2008 Quality Management Systems—Requirements(British Standard Institute)

ISO VIM, 2012 International Vocabulary of Metrology—Basic andGeneral Concepts and Associated Terms (VIM) (JCGM)

ISO, 2008 Evaluation of Measurement Data—Guide to theExpression of Uncertainty in Measurement 1st edn (Geneva:ISO)

Nadcap 2013 Audit Criteria (Warrendale, PA: Performance ReviewInstitute)

O’Hagan A and Oakley J E 2004 Probability is perfect, but we can’telicit it perfectly Reliab. Eng. Syst. Saf. 85 239–48

Rose S J 2009 Creating a High-Value Economy (Royal Society forthe Encouragement of Arts, Manufactures and Commerce:Rolls Royce)

SAE Aerospace 2007 Aerospace Information Report—MeasurementUncertainty Applied to Cost-Effective Testing (Warrendale, PA:SAE)

S227