Proposal of an innovative benchmark for comparison of the performance of contactless digitizers

1

Proposal of an innovative benchmark for comparison of contactless

digitisers performances

Luca Iuliano, Paolo Minetola and Alessandro Salmi

Department of Production Systems and Business Economics,

Politecnico di Torino, C.so Duca degli Abruzzi 24,10129 Torino, Italy

E-mail: paolo.minetola@polito.it

The publisher holds the copyright of this publication. The PDF of the draft is provided here to

ensure rapid dissemination of scholarly work. It is understood that you will use this file only in a

manner consistent with the fair use provisions of the relevant copyright laws. You may not

distribute it or use it for any commercial purpose. Thanks.

Please refer to the published version and cite this work as:

Iuliano L., Minetola P., Salmi A., “Proposal of an innovative benchmark for comparison of the

performance of contactless digitisers”, Measurement Science & Technology, Institute of Physics

Publishing (IOP), 2010, Vol. 21 n. 10, pp. 1-13, ISSN: 0957-0233, DOI: 10.1088/0957-

0233/21/10/105102

Abstract

Thanks to the increasing performances of 3D optical scanners, in terms of accuracy and repeatability, Reverse

Engineering applications have extended from CAD model design or reconstruction to quality control. Today,

contactless digitizing devices constitute a good alternative to Coordinate Measuring Machines (CMMs) for the

inspection of certain parts.

The German guideline VDI/VDE 2634 is the only reference to evaluate if 3D optical measuring systems comply

with the declared or required performance specifications. Nevertheless it is difficult to compare the performance of

different scanners referring to such guideline. An adequate novel benchmark is proposed in this paper: focusing on

the inspection of production tools (moulds), the innovative test piece was designed using common geometries and

free-form surfaces. The reference part is intended to be employed for the evaluation of the performance of several

contactless digitizing devices in computer aided inspection, considering dimensional and geometrical tolerances as

well as other quantitative and qualitative criteria.

Keywords: reverse engineering, optical scanners, contactless digitizing, quality control, contactless inspection,

performance

1. Introduction

Industrial metrology of mechanical components has been facing a gradual revolution lately through the application

of optical contactless devices in Computer Aided Inspection (CAI). Recent developments in the performances of

3D optical scanners have extended the application field of Reverse Engineering (RE) techniques to quality control.

Up to some years ago, such sector was exclusive competence of CMMs.

The traditional measuring method is time-consuming and it often requires customised fixtures to support complex

parts during measurements [1]. Even if the resolution of 3D scanners is not yet comparable to the one of CMMs,

contactless digitising devices can provide fast scans with high point density (hundreds thousand or millions of

points per second) for those part whose tolerances are larger some thousandths of a millimetre. Therefore they are

preferred to pointwise measuring machines when complex free-form shapes or large parts have to be inspected [2-

2

3]. By means of contactless digitisers, products or production tools can be measured completely in short times

without the need to produce and manage customised fixtures. If needed, the component can be digitised twice, as

individual part and as part of the assembly, to analyse the deformations caused by the assemblage. Another

advantage is off-line inspection from saved scan data. If for any reason some new features have to be evaluated on

an inspected part after some time, there is no need to repeat the measurement of the piece. After loading the point

cloud of the scanned part, adding some evaluations on its geometries is rather fast and easy. The inspection of the

same features by a CMM would require fixing the part on the machine once again, aligning it and then measuring it

on-line.

While the working volume of a CMM is fixed, the one of 3D scanners can be easily adjusted by changing their

optical configuration, so these devices are much more versatile. Transportability and the ease of configuration

changes make structured light scanners shortly adjustable for the acquisition of small and large objects. Thus they

are employed in very different sectors ranging from Cultural Heritage to medical and dental applications passing

through wide consumption products, transportation industries included [4-6].

Following the industrial trend and request, Reverse Engineering software packages have been recently equipped

with new modules for quality control. Dense point cloud data can be compared to CAD models: after the alignment

of the coordinate system, the deviation of individual points is calculated and then displayed by coloured deviation

maps. Overall and local deviations can be analysed to understand their causes and geometries can be defined to

calculate geometrical tolerances (GD&T - Geometric dimensioning and tolerancing) and then take go/no-go

decisions as well [7]. Thus the new term of CAI (Computer Aided Inspection) has been recently added to Computer

Aided applications (CAx), including both Reverse Engineering software packages for inspection and those for

CMMs metrology.

The application of contactless digitisers in metrology opens up new issues and provides research topics on two

fronts. On the one hand, there is the need to assess the measurement uncertainty and the performances of this kind

of measuring machines. Different physical measuring principles are exploited by optical scanners, but the

desiderata are metrologic characteristics resulting from standard tests to be comparable. Measurement uncertainty

[8] is one of metrology key factors. On the other hand, novel procedures to manage large amount of scan data and

complex free form geometries have to be defined, because so far metrology has been mean dealing with point-wise

measures and classic geometries.

Focusing on the first aspect, there are plenty of Reverse Engineering devices for sale on the market, each one with

its own characteristics and advantages. Comparing and choosing among different systems is a quite arduous task

because at the moment there is no standard procedure internationally recognised and accepted for qualifying and

evaluating the performances of 3D scanners.

In the field of verification, the problem is more relevant, because performance verification is very important in

customer - dealer relationship (usually system performance have to be declared and guaranteed by the dealer). The

lack in international standards for metrologic performance of contactless scanners depends in part on the great

improvement these systems showed in recent times and in part on the variety of devices that may be classified as

“contactless scanner”, which makes hard to develop comprehensive methodologies for uncertainty evaluation,

performance evaluation and performance verification.

The German guideline VDI/VDE 2634 part 2 [9] defines a series of acceptance tests for 3D optical measuring

systems based on triangulation and area scanning of a single view. The VDI/VDE 2634 part 3 [10] was recently

released and refers to multiple scans. The acceptance test shall be repeated at regular intervals with the purpose of

verifying whether the measurement error lies within the limits specified by the manufacturer. In the acceptance test,

length standards and calibrated artefacts are measured.

Nevertheless the measurement uncertainty for real measurements of real parts or products is not as good as the

outcome of an acceptance test, where well defined artefacts of good quality are used. Moreover, real measurements

are influenced by a great number of external factors [11-13], so it is difficult to compare the performance of

different scanners referring to such guideline. The same unique reference part should be employed for the

comparison of the performance of different digitizing devices in order to achieve comparable results.

In the field of performance evaluation several more or less empirical approaches were proposed [14-28]. All these

procedures were defined coherently with the development of the measuring device they were applied to, so they are

highly specific for it and the related performance indicators are hard to compare.

To investigate the potential offered by non-contact quality control, a common adequate benchmark has to be

employed to correctly evaluate the performances of different scanning systems using the same judging parameters

and non-specific procedures.

3

Focusing on the inspection of production tools (moulds), an innovative benchmark is proposed in this paper. In the

design phase, particular attention was paid to the definition and the organisation of classic and free-form

geometries to guarantee the opportunity of measuring different shapes and geometrical tolerances too. The

benchmark was manufactured and then inspected by means of a CMM and a contact digitiser to measure and assess

the final real geometry of the part “as fabricated”. The result of contact measurements will be the reference for the

evaluation of the performance of several contactless scanners in multiple views CAI. Some first results of the

comparative analysis are provided for the ATOS III (Advanced TOpometric Sensor) scanner as an essay of the

forthcoming activity to be developed.

2. Benchmark design

The accuracy of digitizing systems can be easily evaluated when classic geometric entities are scanned in a single

view. Normally 3D optical scanners require multiple scans to complete the acquisition of complex parts. The

number of views is related with the presence of optical occlusions among the geometries and the ratio between the

overall dimensions or size of the part and the scan area. Several point clouds, one for each scan, have to be aligned

and merged to obtain the complete model of the part. The aligning and merging operations may introduce some

noise and errors, due to the superposition of data coming from different scans. Nevertheless, to be adequate for

quality control, non-contact scanning systems have to guarantee a good accuracy even after combining multiple

scans in one point cloud.

The proposed benchmark was designed with reference to moulds, i.e. production tools. The manufacturing

equipment has to be checked for quality, dimensions and tolerances. In fact it plays a fundamental role in the

definition of the shape and the dimensions of the manufactured part.

The benchmark overall dimensions are 200 x 210 x 80 mm (figure 1). Its size was decided to make it easy

transportable and fit into the working volume of small contact digitizing systems. Most non contact scanners, that

are widespread for RE activities into the mechanical sector of moulds and large consumption products, employ

laser triangulation or structured light techniques and have a scan area comparable to the size of the benchmark as

well. Moulds’ common geometric features and sculptured surfaces were considered in the design phase to make the

benchmark representative of a variety of production tools and wide consumption products.

Referring to quality control, the presence of simple classic geometries is imperative, since form errors and

geometrical tolerances are defined on them. Female and male mould cavities can consist of concave or convex

shapes that correspond to protuberances, boss or pockets, according to the insert side. Simple geometries were

represented in the benchmark in both concave and convex form, in order to consider all possible cases (figure 1).

a)

b)

Figure 1. Benchmark CAD Model (a) and nomenclature of benchmark features (b).

4

The following entities appear in couples, because one of them is concave (negative) and the other convex

(positive):

two quarters of semisphere (S) of radius 40 mm.

two quarters of a cylinder (C) of radius 60 mm.

two quarters of a truncated cone (TC): they connect the quarters of semisphere with the quarter of cylinder of

the same type (concave or convex). So the bottom and top circle have a radius of 60 mm and 40 mm

respectively. The cone angle measures 43.60 degrees.

Later on in the text, convex geometries are referred using the adjective “OUTER” (O), whereas the adjective

“INNER” (I) is used for concave ones to distinguish between them. So, for example, OS addresses the outer

semisphere, whereas IS refers to the inner semisphere. A complete list of benchmark features notation is available

in table 1A in Appendix A.

The reference piece is also composed by the following classic geometries:

Two coaxial cylinders (CC), each one 10 mm high. The smallest has a radius of 8 mm and lays over the biggest,

whose radius measures 16 mm.

An elliptic pocket (EP), whose minimum and maximum radii measure 15 and 25 mm respectively.

Three tilted planes (TP). The biggest (TP1) was generated tilting the free base of the two quarters of cylinder by

15 degrees. The second plane (TP2) is located near the elliptic pocket and has an inclination of about 20 degrees

to the horizontal plane. The smallest plane (TP3) is located near the outer semisphere OS (figure 1b) and has an

inclination of 10 degrees to the vertical plane.

Some planes, that are parallel (HP for horizontal plane) or orthogonal (VP for vertical plane) to the base of the

piece.

A truncated square pyramid (TSP). The edge of the top face of it measures 20 mm and the side faces are tilted to

the top one at different angles: 0°, 5°, 10° and 15°. The inclination of the side faces will be useful to investigate

whether optical digitisers allow distinguishing between those angular differences.

All these shapes have been organised, located and oriented rationally to be representative for the evaluation of

geometrical tolerances during the inspection phase.

Complex shapes are widely employed by designers to enhance the aesthetic value and ergonomics everyday

products and goods. Therefore free-form surfaces are representative of a large class of mass customisation products

and are included in manufacturing tools as well.

For this reason, some free-form surfaces have been defined and inserted on the benchmark to connect and fill the

gap between concave and convex classic geometries (figure 2a and figure 2b).

a) b)

Figure 2. First NURBS surface (a), second NURBS surface (b).

The most used free-form surface entity in 3D modelling is a NURBS (Non Uniform Rational B-Spline), whose

mathematical formula is the following:

5

)()(

)()(

),(

,0 0

,,

,0 0

,,,

vBuBw

vBuBpw

vuP

rj

n

i

m

jqiji

rj

n

i

m

jqijiji

(1)

where

Bi,q and Bj,r are the B-Spline Basis functions,

pi,j are the surface control points,

wi,j are the weights.

The exact geometry of a NURBS surface is unknown to the CAD designer. The user normally constructs free-form

entities from some boundary curves of style, exploiting modelling routines available in the CAD package, but he

does not know the values of the terms appearing in the mathematical formula of the created surface.

A way to get the surface control points pi,j and weights wi,j is to export the entity in the IGES (Initial Graphic

Exchange Specification) format [29]. An IGES interpreter has been specifically written for the IGES entity 128

(Rational B-Spline Surface) in Visual Basic language. The program reads the NURBS control points pi,j and

weights wi,j from the IGES file and computes the surface points by changing the values of the parameters (u and

v). Thus the surface is completely known: the theoretical coordinates of every single surface point can be computed

by the equation 1 and used for quality control. Nevertheless, there is no standard procedure or guideline to compute

the deviation of a NURBS surface in coordinate measuring metrology. Investigations on this issue are still in

progress [30] and a novel procedure could be still developed and applied.

3. Benchmark fabrication and inspection by CMM

The manufacturing of the benchmark was carried out at the Department of Production Systems and Business

Economics at Politecnico di Torino by means of a three axis milling machine She Hong (figure 3a) equipped with a

Fidia F0 numerical control (NC). The stock was an aluminium block measuring 200 x 210 x 100 mm.

The NC milling path was computed using high speed milling routines of Vero Visi Series software. The rough

milling operation was performed by a 12 mm roughing mill and a Z depth increment of 2 mm, leaving 0.3 mm of

residual material on the piece.

The finishing tool for the central part of the benchmark was a 10 mm ball-end mill and the Z increment was set to

0.05 mm. A 12 mm square-end mill was used for the finishing of flat areas. The fabricated piece appears as in

figure 3b.

a)

b)

Figure 3. She Hong milling machine (a) and benchmark after milling (b).

6

The benchmark was not finished manually, because hand polishing would modify the surfaces of the part of a

quantity that is not definable mathematically, introducing major deviations to the CAD model. Moreover the virtual

model of the benchmark cannot be employed as reference since it does not reflect the final real geometry of the

benchmark, because it does not into account the deviations introduced by the fabrication process.

The arithmetic average roughness Ra of the benchmark surfaces after machining was measured by an Hommel

Tester T-1000 for a standard length of 15 mm on 15 different planar surfaces. The mean value of the Ra measures

is 0.86 m with a standard deviation of 0.32 m. First of all such value is much smaller than optical scanners’

accuracy. In addition, the value of benchmark roughness is not significant for the contactless inspection activity,

since a thin layer of white opaque powder is normally sprayed on the part with prior to optical scanning. Such layer

was proven to be influence free on non contact measurements [31], in the sense that it doesn’t alter the results, but

it is useful to avoid light reflection problems typical of metallic surfaces. No matter what the colour of the part is,

the white powder also enhances the contrast of the object surfaces with respect to the optical digitizing device and

protects the surface from corrosion (although the benchmark is not kept in an aggressive environment).

After manufacturing, the benchmark was inspected by means of a CMM and a contact digitiser to measure and

assess the final real geometry of the part “as fabricated”. Measurements were performed in the metrology room at

the Department of Production Systems and Business Economics at Politecnico di Torino. The room is temperature-

controlled and it is artificially illuminated, since it has no windows on the outside of the building. For these reasons

temperature changes and natural variation in daylight cannot affect the benchmark and cannot influence the

measurements.

The geometrical entities of the benchmark were first inspected on a DEA CMM model GLOBAL Image 07.07.07

(figure 4a), that was equipped with an indexable swinging head and a touch trigger probe whose resolution is 1 µm.

The machine declared volumetric length measuring uncertainty MPEE according to ISO-10360/2 [32] is 1.5 +

L/333µm, where MPE is the acronym for Maximum Permissible Error and L is the measured length. It is important

to highlight that the accuracy of the CMM is one order of magnitude better than that of the contactless digitizing

devices whose performances will be evaluated by means of the benchmark. Therefore DEA CMM measures are

taken as reference values for the real geometry of the classic features of the benchmark “as fabricated”. Three

replications were made for each measurement and the results are listed in table 2A in Appendix A.

a) b)

Figure 4. DEA CMM model Global Image 07.07.07 (a) and Renishaw Cyclone contact scanning machine (b).

5. Benchmark contact digitizing and CAD model comparison

In order to have a reference value for the whole real geometry and not only for the classic features of the

benchmark, the part was digitised by a contact scanning device. The Renishaw Cyclone (figure 4b) machine was

selected and used for this purpose. It is supplied with a Renishaw SP620 analogue contact probe that has a

resolution of 1 m. The machine has the bridge structure typical of CMMs and it is characterised by a high

scanning rate and a high precision. Styli of different length and tip diameter can be mounted on the low force

probe.

7

The digitizing of the benchmark by Cyclone took almost five hours and it was made by two 45° crossed scans

using a probe tip of 1 mm. The dense point cloud (1,162,953 points) of Cyclone’s scan data was compared to the

original CAD model of the piece to highlight deviations that are a consequence of the manufacturing milling

operations. All point clouds processing and comparisons were made using RapidForm 2006 software by Inus

Technology. Absolute or signed deviations between scan data and the CAD surfaces are generally displayed as

coloured deviation maps by the software.

Figure 5. Cyclone scan data vs. CAD model coloured deviation map and error distribution

(mean error = 0.02 mm and standard deviation = 0.06 mm).

After the alignment of Cyclone scan data to the CAD model by defining the same Cartesian reference system (X,

Y, Z) and the coordinate origin, deviations were computed. The mean error between Cyclone’s point cloud (figure

5) and the CAD model is 0.02 mm with a standard deviation of 0.06 mm. The value of maximum and minimum

deviations is not very meaningful, since it refers to a single point or to a very small area: the distribution of the

errors becomes thinner as the deviation value grows. Highest deviations are localized on sharp corners and sharp

edges: the contact digitiser cannot detect and measures point on features that are smaller than the probe tip radius.

Cyclone’s scan data is taken as reference for the real geometry of the whole benchmark “as fabricated”.

Scanning results can be analysed to point out overall and local deviations, to measure classic geometrical entities

and to calculate geometrical tolerances.

6. Contact dimensional measures comparisons

Classic simple geometries can be measured on point cloud data by computing best-fit elements. Reverse

Engineering software reconstructs geometries that interpolate several points or regions (figure 6 and figure 7): the

algorithm of Moving Least Squares (MLS) is the most used.

8

a) b)

Figure 6. Region to be fit (a) and best fit cylinder (b).

a)

b)

Figure 7. Region to be fit (a) and best fit sphere (b).

All the classic geometrical entities that were defined during the design of the benchmark can be evaluated by best

fitting the corresponding regions on Cyclone scan data. The results are shown in table 3A in Appendix A.

The reference value of pointwise measures by CMM (table 2A) can be compared to the ones obtained by best

fitting the Cyclone scan data (table 3A). Standard deviation of scan points to the corresponding computed best fit

geometry was not included in table 3A since the value was not significant: it was lower than 0.01 mm for every

feature. As already mentioned, thirty points were measured using the DEA CMM, whereas the best-fit geometries

were computed on regions containing thousands of points.

7. Inspection of NURBS surfaces

At the moment there is no standard procedure in metrology to evaluate the deviations and form errors of a freeform

sculptured surface [29, 33-34]. Two different alignment strategies were employed for NURBS surfaces inspection.

The first procedure takes into consideration the alignment in the Cartesian reference system (X, Y, Z). The points

belonging to the two NURBS surfaces where extracted from the whole point cloud aligned for the comparisons

whose results are shown in table 3A in Appendix A. By comparing the extracted NURBS points, the deviation of

the two surfaces is computed when the all data is aligned (Whole Data Deviation) in the Cartesian reference system

(X, Y, Z).

As regards the second alignment strategy, each NURBS surface was best-fit aligned with the corresponding points

of the CAD model. In this new alignment, the deviation (Best Fit Deviation) of the two NURBS is computed

9

without taking into account the position of the whole benchmark data (disregarding the Cartesian reference

system). The mean value and the maximum of the two types of deviation are reported in table 1 for both NURBS

surfaces.

Table 1. Result of NURBS surfaces inspection by means of Cyclone device.

Deviation

Values

First NURBS surface Second NURBS surface

Whole Data

Deviation [mm]

Best Fit

Deviation [mm]

Whole Data

Deviation [mm]

Best Fit

Deviation [mm]

Maximum 0.26 0.26 0.05 0.05

Mean 0.02 < 0.01 0.02 < 0.01

St. Dev. < 0.01 < 0.01 < 0.01 < 0.01

Once again, the errors can be displayed as coloured maps to highlight the zones of the NURBS surface wherein the

milling process has introduced major deviations to the CAD model. An example is shown in figure 8 for the

deviations of the second NURBS surface in the whole data alignment (Cartesian reference system).

Figure 8. Deviation map of the second NURBS surface in the whole data alignment

(mean error = 0.02 mm and standard deviation < 0.01 mm).

8. First results of contactless inspection

DEA Global pointwise measurements and Cyclone contact scanning were used to inspect and assess the real

geometry of the benchmark “as fabricated” and they constitute the reference for the forthcoming comparative

analysis of contactless inspection by different optical scanners that is still in progress. Some results are presented

hereafter for a structured light scanner as an essay of the activity to be developed.

The first compared optical scanner is the ATOS III device produced in Germany by GOM GmbH. ATOS III (figure

9) is a general purpose structured light scanner that exploits binocular vision as it has two built-in 4 Mpixel CCD

cameras, which store images of the light fringes projected on the scanned object.

10

Figure 9. GOM ATOS III structured light scanner

The projector, placed in the centre of the sensor, projects a sequence of four interference patterns (phase-shift

technique) [35] followed by six Gray coded binary images [36-37]. The scanner is VDI/VDE 2634 certified for

inspection performances and the accuracy declared on the device data sheet is 0.02 mm. Reference points are

applied on the scanning object sticking adhesive targets (i.e. markers). Each scan takes about two seconds and

retrieves as many as four million points on the object surface. Scan area can be adjusted to the object size, from 150

x 150 mm to 2000 x 2000 mm, simply changing the projector’s lenses and the cameras’ ones. Due to the

benchmark overall dimensions, the minimum scan area was employed for contactless digitizing of the reference

part. Sixteen views were needed to completely scan the benchmark by means of the ATOS III device and the task

took a bit more than one hour.

ATOS III scan data (567,266 points) was aligned and compared to Cyclone scan data, which was assumed as

master reference for the real benchmark geometry. The mean error between ATOS III point cloud (figure 10) and

Cyclone reference is 0.02 mm with a standard deviation of 0.07 mm.

Figure 10. ATOS III scan data vs. Cyclone scan data (master reference)

(mean error = 0.02 mm and standard deviation = 0.07 mm).

The classic simple geometries of the benchmark can be measured on ATOS III scan data by computing best-fit

elements. Some results of the measurements are shown in table 2.

11

Table 2. Results of best fitting some of the benchmark classic geometries from ATOS III scan data

Feature Nr. Points

(ATOS III) Measurand Unit

Best fit value

(ATOS III)

Reference value

(DEA CMM) Difference

TSP1 955 Z mm 15.06 15.05 0.01

TSP2 403 X mm 4.99 5.00 -0.01

degrees 0.00 0.00 0.00

TSP3 465 degrees 4.95 4.98 -0.03

OS 10607

R mm 40.01 40.02 -0.01

Xc mm 70.06 70.00 0.06

Yc mm 129.98 129.95 -0.03

Zc mm -0.08 -0.05 -0.03

IC 18505

R mm 60.02 60.11 -0.09

Ya mm 59.95 59.88 0.07

Za mm 59.98 60.03 -0.05

ITC 9718 degrees 43.68 43.83 -0.15

Ya mm 59.99 60.19 -0.20

Za mm 59.99 59.96 0.03

CC1 4384

Z mm 20.04 20.03 0.01

R mm 8.07 8.08 -0.01

Xa mm 169.98 169.96 0.02

Ya mm 25.04 25.01 0.03

CC2 5726

Z mm 10.03 10.02 0.01

R mm 16.06 16.06 0.00

Xa mm 169.99 169.95 0.04

Ya mm 25.00 24.97 0.03

EP 14352 min R mm 14.91 14.92 -0.01

max R mm 24.92 24.92 0.00

HP1 29535 Z mm 0.01 0.01 0.00

HP2 2502 Z mm 60.03 60.01 0.02

HP3 5868 Z mm 20.01 20.01 0.00

HP4 7200 Z mm 15.00 15.02 -0.02

TP1 7965 degrees 15.00 14.99 0.01

TP2 8037 degrees 21.81 21.83 -0.02

VP1 1431 X mm 29.98 30.00 -0.02

VP2 670 Y mm 59.98 59.98 0.00

9. Conclusions

At the moment an international standard for evaluating the measurement uncertainty and verifying the metrologic

performance of contactless digitisers is missing. It is hard or impossible to compare the measurement accuracy and

uncertainty results achieved by different researchers with different contactless scanning devices because, specific

procedure were defined and applied.

A novel benchmark is proposed in this paper to overcome that problem. The reference part was designed with

attention to the industrial sector of production tool manufacturing (moulds and dies). Several classic features and a

couple of sculptured freeform geometry (NURBS) were rationally organised so that the part can be representative

of a wide range of products.

The comparison of contactless digitisers performance is the main activity the benchmark was designed for. The

inspection of the fabricated part is of primary importance. As a matter of fact the CAD model cannot be taken as

reference to compare the results of optical digitizing. A contact scan is needed to measure the real piece accurately

and to highlight the deviations from the CAD data, that are a consequence of the milling operations. The results of

contact scanning and CMM measurements presented in this work is taken as reference geometry of the benchmark

“as fabricated”. The results of benchmark geometries measurements by best fit of Cyclone data differ very little

from the corresponding DEA CMM measurements. Apart from the outer truncated cone, it can be notice that

12

largest difference was measured on vertical planes. As a matter of fact, because of the scan strategy set on the

Cyclone device, the contact probe tip slides on planes that are orthogonal to the base of the benchmark and to the of

the digitizing machine. The less accurate Cyclone measures of points on vertical planes are a direct consequence of

that. So, for example, the vertical lateral plane of the truncated square pyramid (TSP2) on Cyclone data has an

inclination of 2.75°, while the tilting angle measured by DEA CMM is null (0°).

As an example of the comparative activity on contactless digitizers to be further carried out, some first

measurements and comparisons were performed using the benchmark’s scan data retrieved by means of the

structured light scanner ATOS III. The overall absolute deviation of ATOS point cloud from the Cyclone reference

data is in accordance with the optical device accuracy (0.02 mm) declared by the producer. Such conclusion is also

confirmed by the results of the best fitting of the classic geometries of the benchmarks (table 2).

Several other 3D optical scanners will be employed for the scanning and the inspection of the benchmark. The

results obtained by each device will then be compared to the reference by performing a dimensional analysis like

the one presented before. With respect to previous research works available in literature, the contactless scanning

devices performance will be comparable because the values will be referred to the inspection of the same reference

part (the benchmark) without a specific procedure designed for a certain type of device (triangulation laser,

structured light, stereoscopic scanner, etc.).

Beside this activity, the comparison can be extended to RE software packages. These applications have been

readily updated with routines for quality control to satisfy requests coming from the industrial sector. Different

alignment strategies can be employed to register the scan data on the CAD model and coloured error maps may be

analysed to understand whole and local deviations. Standard measuring protocols can be defined and reports in

different formats (MS Excel, MS Word or HTML) can be created and exported. Evaluations, measures and

comparisons like the ones performed by means of Rapidform 2006, will be repeated with different CAI

applications to investigate whether they are consistent or not. CAI results are expected to depend only on the

measuring data set and not on the software routines and algorithms used for computations. In other words, the CAI

software should be influence-free on measurement result if the data set does not change.

Finally the size of the proposed benchmark was decided to fit in the scan area of contactless scanners commonly

used for Reverse Engineering and inspection of mould and die makers for wide consumption products.

Nevertheless benchmark features and geometries can be used for the comparison of contactless digitizers, other

than laser scanner and structured light devices, whose scanning area is bigger or smaller than the size of the

benchmark. For example, for small scale metrology scanners based on conoscopic holography [38] have an

accuracy of less than ten microns and a working volume of some tens of millimetres. Whereas for large scale

metrology the accuracy of laser trackers is in the order of some millimetres for distances of hundreds of meters.

For this purpose, the benchmark should be scaled and its fabrication tolerances should be consistent with the

accuracy of the tested contactless device. Furthermore the reference part should be fabricated of the proper material

to be representative of the real digitized objet the 3D scanner is employed for. As regards laser trackers, they are

largely employed in surveys of buildings, statues and structures in the field of Cultural Heritage, Architecture and

Civil Engineering. For such reason the large-scale benchmark should be fabricated and replicated in different

materials (metal, wood, concrete, etc.) to better represent a wider range of real scan object.

References

[1] Iuliano L, Minetola P and Violante M G 2007 Design and production of fixtures for free-form

components using selective laser sintering Rapid Prototyping Journal 13 30-37

[2] Gao J, Folkes J, Yilmaz O and Gindy N 2005 Investigation of a 3D non-contact measurement based blade

repair integration Aircraft Engineering and Aerospace Technology: An International Journal 77(1) 34-41.

[3] Bernard A and Veron M 2000 Visibility Theory Applied to Automatic Control of 3D complex Parts using

Plane Laser Sensor Annals of the CIRP 49 113

[4] Iuliano L and Vezzetti E 2002 Application of Reverse Engineering to Paleontology, Proceedings of

Scanning 2002 Paris - France 145-158

[5] Williams F L and Richtsmeier J T 2003 Comparison of mandibular landmarks from computed tomography

and 3D digitizer data Clinical Anatomy 26 494-500

[6] Boehler W and Marbs A 2004 3D Scanning and photogrammetry for heritage recording: a comparison,

Proceedings of Geoinformatics 2004 University of Gävle – Sweden 291-298

[7] ElMaraghy W and Rolls C 2001 Design by Quality Product Digitization Annals of the CIRP 50 93

13

[8] ISO/IEC GUIDE 99:2007(E/F): International Vocabulary of basic and general terms in Metrology (VIM)

Geneva-Switzerland 1st edition

[9] The Association of German Engineers, 2002 VDI/VDE 2634 part 2: Optical 3D-measuring systems -

Optical systems based on area scanning

[10] The Association of German Engineers, 2008 VDI/VDE 2634 part 3: Optical 3D-measuring systems -

Multiple view systems based on area scanning

[11] De Chiffre L and Hansen H N 1995 Metrological limitations of optical probing techniques for dimensional

measurements Annals of the CIRP 44 501.

[12] Beraldin J A 2008 Digital 3D Imaging and Modeling: A Metrological Approach Time Compression

Technology Magazine January/February 2008 33-35

[13] Jecić S and Drvar N 2004 3D Shape Measurement Influencing Factors Proceedings of Matest 2004

Zagreb-Croatia 109-116

[14] Lazzari A and Iuculano G 2004 Evaluation of the uncertainty of an optical machine with a vision system

for contact-less three-dimensional measurement Measurement 36(3-4):215-231

[15] Santo M D Liguori C Paolillo A and Pietrosanto A 2004 Standard uncertainty evaluation in image-based

measurements Measurement 36(3-4):347-358

[16] Tsai M J and Hung C C 2006 A Fast Evaluation Approach of Geometrical Correspondence Uncertainty for

3-D Vision Measurement System JSME International Journal Series 49(2) 527-534

[17] Dorsch R G Hausler G and Herrmann J M 1994 Laser triangulation: fundamental uncertainty in distance

measurement Appl Optic 33(7/1) 1306-1314

[18] Marinello F Bariani P Savio E Horsewell A and De Chiffre L Critical factors in SEM 3D stereo

microscopy Meas Sci Tech 19(6) 1-12

[29] Contri A Bourdet P and Lartigue C 2006 Quality of 3D digitised points obtained with non-contact optical

sensors CIRP Annals 51(1) 443-446

[20] Sansoni G Carocci M and Rodella R 2000 Calibration and performance evaluation of a 3-D imaging sensor

based on the projection of structured light IEEE T Instrum Meas 49(3) 628-636

[21] Marinello F Savio E Carmignato S and De Chiffre L 2008 Calibration artefact for the microscale with high

aspect ratio: The fiber gauge CIRP Annals 57(1) 497-500

[22] De Chiffre L Hansen H N and Morace R E 2005 Comparison of Coordinate Measuring Machines using an

Optomechanical Hole Plate CIRP Annals-Manufacturing Technology 54(1) 479-485

[23] Sansoni G and Docchio F 2005 In-field performance of an optical digitizer for the reverse engineering of

free-form surfaces The International Journal of Advanced Manufacturing Technology 26(11) 1353-1361

[24] Bariani P De Chiffre L Hansen H N and Horsewell A 2005 Investigation on the traceability of three

dimensional scanning electron microscope measurements based on the stereo-pair technique Precision

Engineeering 29(2) 219-228

[25] Beraldin J A Rioux M Cournoyer L Blais F Picard M and Pekelsky J 2007 Traceable 3D imaging

metrology Proceedings of the 8th Conference on Optical 3-D Measurement Techniques Zurich-Switzerland

[26] Vlaar S T and Van Der Zel J M 2006 Accuracy of dental digitizers International Dental Journal 56 301-

309

[27] Guidi G Remondino F Morlando G Del Mastio A Uccheddu F and Pelagotti A 2007 Performances

Evaluation Of A Low Cost Active Sensor For Cultural Heritage Documentation Proceedings of the 8th

Conference on Optical 3-D Measurement Techniques Zurich-Switzerland 59-69

[28] Mechelke K Kersten T P and Lindstaedt M 2007 Comparative investigations into the accuracy behaviour

of the new generation of terrestrial laser scanning system Proceedings of the 8th Conference on Optical 3-

D Measurement Techniques Zurich-Switzerland 319-327

[29] IGES Initial Graphics Exchange Specifications 53 ANS US PRO/ IPO-100-1996 National Institute of

Standards and Technology US Pro US http://wwwnistgov/iges

[30] Li Y and Gu P 2005 Inspection of free-form shaped parts Robotics and Computer-Integrated

Manufacturing 21(4-5) 421-430

14

[31] Iuliano L and Minetola P 2009 Enhancing moulds manufacturing by means of Reverse Engineering

International Journal, Advanced Manufacturing Technology 43 551-562

[32] ISO 10360-2:2009 Geometrical product specifications (GPS) – Acceptance and reverification tests for

coordinate measuring machines (CMM) – Part 2: CMMs used for measuring linear dimensions

International Organization for Standardization

[33] Li Y and Gu P 2005 Feature-Based Alignment and Comparison between Portion and Whole of Free-Form

Surfaces Annals of the CIRP 54 135

[34] Savio E De Chiffre L and Schmitt R 2007 Metrology of freeform shaped parts CIRP Annals 56(2) 810-835

[35] Wang Z Braynston-Cross P J and Whitehouse D J 1996 Phase difference determination by fringe pattern

matching Optics & Laser Technology 28(6) 417-422

[36] Perry K E and McKelvie J 1995 Reference phase shift determination in phase-shifting interferometry

Optics and Laser in Engineering 22 79-90

[37] Pagès J Salvi J García R and Matabosch C 2003 Overview of coded light projection techniques for

automatic 3D profiling IEEE International Conference on Robotics and Automation ICRA Taipei-Taiwan

133-138

[38] Sirat G Y Paz F Agronik G and Wilner K 2005 Conoscopic holography Advanced Topics in

Optoelectronics, Microelectronics, and Nanotechnologies II - Proceedings of the SPIE 5972 11-16

15

Appendix A

Table 1A. Notation used for benchmark features.

Notation Meaning

TSP Truncated Square Pyramid

TSP1 Top plane of the truncated square pyramid

TSP2 Lateral vertical ( = 0°) plane of the truncated square pyramid

TSP3 Lateral titlted ( = 5°) plane of the truncated square pyramid

TSP4 Lateral titlted ( = 10°) plane of the truncated square pyramid

TSP5 Lateral titlted ( = 15°) plane of the truncated square pyramid

OS Outer Sphere (convex geometry)

IS Inner Sphere (concave geometry)

OC Outer Cylinder (convex geometry)

IC Inner Cylinder (concave geometry)

OTC Outer Truncated Cone (convex geometry)

ITC Inner Truncated Cone (concave geometry)

CC1 First Coaxial Cylinder

CC2 Second Coaxial Cylinder

HP Horizontal Plane

HP1 Bottom horizontal plane of benchmark geometries

HP2 Top horizontal plane of benchmark geometries

HP3 Top horizontal plane of the elliptic pocket

HP4 Bottom horizontal plane of the elliptic pocket

EP Elliptic pocket

TP Tilted Plane

TP1 Big tilted plane

TP2 Tilted plane near the elliptic pocket

TP3 Small tilted plane

VP Vertical Plane

VP1 Vertical plane near the second NURBS surface along Y direction

VP2 Vertical plane near the inner sphere along X direction

VP3 Vertical plane near the elliptic pocket along X direction

VP4 Vertical plane near the elliptic pocket along Y direction

: tilting angle of the feature;

X: X-axis coordinate of feature position

Y: Y-axis coordinate of feature position

Z: Z-axis coordinate of feature position

R: Radius of the feature

Xc: X-axis coordinate of feature center

Yc: Y-axis coordinate of feature center

Zc: Z-axis coordinate of feature center

Xa: X-axis coordinate of feature axis

Ya: Y-axis coordinate of feature axis

Za: Z-axis coordinate of feature axis

16

Table 2A. Results of DEA CMM measurements for the benchmark classic geometries.

Feature Measurand Unit CAD model 1st Repl. 2nd Repl. 3rd Repl. Mean St. Dev.

TSP1 Z mm 15.00 15.05 15.05 15.05 15.05 0.00

TSP2 X mm 5.00 5.01 5.01 4.98 5.00 0.02

degrees 0.00 0.00 0.00 0.00 0.00 0.00

TSP3 degrees 5.00 4.98 4.98 4.98 4.98 0.00

TSP4 degrees 10.00 10.00 10.00 9.99 10.00 0.01

TSP5 degrees 15.00 14.97 14.97 14.96 14.97 0.01

OS

R mm 40.00 40.03 40.02 40.02 40.02 0.01

Xc mm 70.00 70.00 70.00 69.99 70.00 0.01

Yc mm 130.00 129.94 129.95 129.95 129.95 0.01

Zc mm 0.00 -0.05 -0.05 -0.05 -0.05 0.00

IS

R mm 40.00 40.06 40.06 40.05 40.06 0.01

Xc mm 80.00 80.02 80.02 80.01 80.02 0.01

Yc mm 60.00 59.95 59.95 59.96 59.95 0.01

Zc mm 60.00 60.02 60.01 60.01 60.01 0.01

OC

R mm 60.00 60.07 60.07 60.07 60.07 0.00

Ya mm 130.00 129.90 129.97 129.92 129.93 0.04

Za mm 0.00 -0.10 -0.10 -0.11 -0.10 0.01

IC

R mm 60.00 60.10 60.13 60.09 60.11 0.02

Ya mm 60.00 59.88 59.85 59.91 59.88 0.03

Za mm 60.00 60.03 60.04 60.01 60.03 0.02

OTC

degrees 43.60 43.46 43.39 43.42 43.42 0.04

Ya mm 130.00 129.81 129.75 129.78 129.78 0.03

Za mm 0.00 -0.22 -0.29 -0.27 -0.26 0.04

ITC

degrees 43.60 43.74 43.82 43.92 43.83 0.09

Ya mm 60.00 60.11 60.18 60.28 60.19 0.09

Za mm 60.00 60.04 59.96 59.88 59.96 0.08

CC1

Z mm 20.00 20.03 20.03 20.03 20.03 0.00

R mm 8.00 8.09 8.08 8.08 8.08 0.01

Xa mm 170.00 169.96 169.96 169.96 169.96 0.00

Ya mm 25.00 25.00 25.00 25.02 25.01 0.01

CC2

Z mm 10.00 10.02 10.02 10.02 10.02 0.00

R mm 16.00 16.06 16.06 16.06 16.06 0.00

Xa mm 170.00 169.95 169.95 169.95 169.95 0.00

Ya mm 25.00 24.97 24.97 24.98 24.97 0.01

EP min R mm 15.00 14.91 19.93 14.92 14.92 0.01

max R mm 25.00 24.91 24.92 24.92 24.92 0.01

HP1 Z mm 0.00 0.01 0.01 0.00 0.01 0.01

HP2 Z mm 60.00 60.01 60.01 60.01 60.01 0.00

HP3 Z mm 20.00 20.01 20.01 20.01 20.01 0.00

HP4 Z mm 15.00 15.02 15.02 15.01 15.02 0.01

TP1 degrees 10.00 10.04 10.05 10.06 10.05 0.01

TP2 degrees 21.80 21.83 21.83 21.83 21.83 0.00

TP3 degrees 10.00 10.04 10.05 10.06 10.05 0.01

VP1 X mm 30.00 30.00 30.00 30.00 30.00 0.00

VP2 Y mm 60.00 59.98 59.98 59.98 59.98 0.00

VP3 Y mm 10.00 10.00 10.00 10.00 10.00 0.00

VP4 X mm 10.00 10.00 10.00 10.00 10.00 0.00

17

Table 3A. Results of best fitting the benchmark classic geometries from Cyclone scan data

Feature Nr. Points

(Cyclone) Measurand Unit

Best fit value

(Cyclone)

Reference value

(DEA CMM) Difference

TSP1 5848 Z mm 15.07 15.05 0.02

TSP2 1181 X mm 5.13 5.00 0.13

degrees 2.75 0.00 2.75

TSP3 3184 degrees 4.99 4.98 0.01

TSP4 4133 degrees 9.99 10.00 -0.01

TSP5 4416 degrees 14.95 14.97 -0.02

OS 34880

R mm 39.99 40.02 -0.03

Xc mm 70.01 70.00 0.01

Yc mm 129.99 129.95 0.04

Zc mm -0.01 -0.05 0.04

IS 35798

R mm 40.02 40.06 -0.04

Xc mm 80.03 80.02 0.01

Yc mm 60.01 59.95 0.06

Zc mm 60.00 60.01 -0.01

OC 65589

R mm 60.08 60.07 0.01

Ya mm 129.93 129.93 0.00

Za mm -0.07 -0.10 0.03

IC 60008

R mm 60.07 60.11 -0.04

Ya mm 59.94 59.88 0.06

Za mm 60.03 60.03 0.00

OTC 98556 degrees 43.74 43.42 0.32

Ya mm 130.01 129.78 0.23

Za mm -0.02 -0.26 0.24

ITC 90703 degrees 43.69 43.83 -0.14

Ya mm 59.98 60.19 -0.21

Za mm 60.02 59.96 0.06

CC1 3931

Z mm 20.04 20.03 0.01

R mm 8.01 8.08 -0.07

Xa mm 169.98 169.96 0.02

Ya mm 25.03 25.01 0.02

CC2 7616

Z mm 10.04 10.02 0.02

R mm 15.97 16.06 -0.09

Xa mm 169.92 169.95 -0.03

Ya mm 24.97 24.97 0.00

EP 10523 min R mm 14.90 14.92 0.02

max R mm 24.91 24.92 -0.01

HP1 134188 Z mm 0.01 0.01 0.00

HP2 23609 Z mm 60.03 60.01 0.02

HP3 27886 Z mm 20.01 20.01 0.00

HP4 17066 Z mm 15.01 15.02 -0.01

TP1 53777 degrees 14.99 14.99 0.00

TP2 40784 degrees 21.82 21.83 -0.01

TP3 4123 degrees 10.01 10.05 -0.04

VP1 241 X mm 29.94 30.00 -0.06

VP2 205 Y mm 60.18 59.98 0.20

VP3 348 Y mm 10.04 10.00 0.04

VP4 848 X mm 10.05 10.00 0.05