Platinum Inspection

Case StudyA platinum inspection system using image processingDarien Pardinas Diaz

So, what Ill do today is to run through one of the machine inspection systems I developed while working at BEThis project was for a client who manufactures platinum to be used mainly in medical implants, such as cochlear implantsThe problem they were facing was that they had inspect manually the platinum strips they produce and assess the quality of the surface, so they wanted to automate this process and hired BE to look at the problemSo I joined BE about 2 years after this project started, but at that time the image processing algorithms that were designed didnt provide the results the client requested.So, my first assignment was to start looking at the existing algorithms and come up with new approaches that would solve this problem.11/08/20161

System concept and setupScanning setupDefect types to detectTraining and validation dataset developmentManual annotations with product managerAlgorithm developmentPrototypingDevelopment tools usedLessons learnt and recommendationsQuick Demo

Page 2Outline

11/08/20162Briefly will talk about the concept and setup of this vision system, how the platinum strips were scanned and what type of defects we were supposed to reliably detectThen Ill talk about the development of a training and validation set that was the cornerstone on assessing how well the algorithms being developed performedRight after Ill jump to the algorithms I proposed, how they were prototyped, what development tools were used Finally, Ill tell you what learned and what I would recommend for similar projectsThen, Ill quickly demo the system

Tray of up to 15 platinum strips 480mm x 23mmLine scan camera with 100pixels/mm resolution (600MB)3 Lighting arrangements targeting different defect typesScanning time less than 5 minutes

Page 3System concept and scan setup

-So the system consisted of a tray where you could lay up to 15 platinum strips of approximately 480mm by 23mm and image them with a line scan camera.-The image resolution was about 100pixels/mm, producing grayscale images that were 600MB each uncompressed.-Each strip is imaged progressively under 3 different lighting arrangement with the aim of highlighting a range of defect types with each one.-This system is capable of imaging and processing the tray in less than 5min and show the defects found.-As a note, each of this strips costs over $1000, so you can understand now why Im not bringing one of them as sample 11/08/20163

Page 4Scanning the tray

In this video you can see how the strips are scanned with an X/Y robot. The line scan camera is coupled to the Y motor encoder to trigger the acquisition of lines as the scan goes along.

11/08/20164

Page 5Sample scan image

Lighting BLighting ALighting C(Never used)

So this is what the acquired scan image looks like. On every scan, 3 different strips are imaged at once, each illuminated with a different light arrangement.Then the camera is moved to the right, one strip position at the time. For a 15 strip scan, you end up with 17 scan images.There is a pre-processing step to segment each scan image into individual strip images so that we end up with 3 strip images referring to the same strip.In the final setup, we ended up just using lighting A and B, as defects showing up on C would always show better in either A or B light.11/08/20165

ScratchesSquigglesBlisters (burst/unburst)DentsStainsPage 6Defect types to find

In this slide you can see the most common defect types that the system had to be able to detect.They include scratches, squiggles, blisters (both burst and unburst blisters), dents, stains, pits, inclusions, etc.11/08/20166

Page 7Training set development23 failed strips with a range of defect types and severities

Given the range of defect types and severities in which they could appear it becomes really important to develop a dataset as large as possible that contains a representative sample of each of the defect types to findSo for this project I developed an annotation tool that would allow us to annotate the defects we were trying to find as well as things that would look just like a defect, but werent, for example lint dust that remained on top of the strip after cleaning them with a lint-free cloth. This was really important in order to provide the machine learning algorithm good quality input.So I went onsite and spent a couple of days with the production manager imaging and annotating as many failed strips as we could (23 in total)This is a really tedious work, but the effort you put into it really have an impact in the success or failure of a projectFor example, while going through the annotation process we found that some defects showed well the image, but were hardly visible on the strips by the naked eye and vice-versa and this is because the platinum is highly reflective and the defect visibility is extremely sensitive to the illumination angle and the eye locationThe fact that the production manager was involved in this annotation process helped initiate discussions that ultimately ended up adjusting the clients expectation on the system and gave them the proper view that this project had a huge research component that translated in project risk. In order to keep things as transparent as possible, I regularly provided algorithm performance measures such as sensitivity/specificity and accuracy across the range of defects and severities those are well known metrics that apply to most detection and classification systemsWhile it was hard for them to understand what these numbers actually mean, at least gave them a sense that we were making progress while the algorithms were refined and tuned.

11/08/20167

Page 8Algorithm developmentADark blob detectionHysteresis thresholdCoarse blob selectionClassification (MLP)Line DetectionLine gauss operator with scaled imageContour selectionDefects(Squiggles, some scratches)Defects(Mainly scratches)Classification (MLP)BDetect Bright BlobsThresholdingCoarse blob selectionPaired blobs Defects(Pits and dents)Detect Dark BlobsThresholdingCoarse blob selection

Classification (MLP)Defects(Pits, Deep scratches)Defects(Squiggles, stains)Classification (MLP)Classification (MLP)Image normalisation

So to cover the broad range of defect types I designed several algorithms to extract potential defective blobs from the strip imagesFor the strip image under lighting A, I simply applied a hysteresis thresholding algorithm to extract dark blobsIn this lighting, defects such as scratches were visible but really challenging to detect, so I used a line detection operator that is really good at joining weak and broken lines togetherFor the strips image under lighting B, things were more complicated, in this case there was a number of defects that would either show as dark regions or bright regions depending on the surface relief of the strip. In this case, I had to normalise the image intensity range and they apply traditional thresholding techniques to segment potential defective blobs.Once the potential candidates are found the next step is to extract a feature vector for each of them and pass them to the classification model, which will evaluate each input and provide a score on how likely this blob is a defect based on the learning from the training stage.

11/08/20168

Page 9Dark blob detection - example

hysteresis_threshold

In this image you can see for example a well defined squiggle somewhat overlapping with a short scratch, in this case you can see how a simple hysteresis thresholding algorithm can perfectly segment the defect. Hysteresis thresholding is similar to normal thresholding but has the advantage of adding less defined pixels to the blob as long as they are over a minimum threshold intensity and certain distance of a well defined pixel (upper threshold)11/08/20169

Page 10Lines detection - scratches

line_gauss

In this image you can see how the scratch detection algorithm works.The line detector operator I used works by estimating for each pixel, if they were part of a line, in what direction that line would run and how likely is that that pixel is part of a line.Then, depending on how selective the algorithm is configured, it will output a collection of contours which you can filter out, for example, these short lines on the left are immediately ignored because their length is too short.11/08/201610

Page 11Intensity normalisation

subtract mean image

Here you can see the normalisation algorithm in action.On the left you get the original image taken under lighting B, which shows a surface illuminated unevenly and in the right side you can see the normalised image, which makes easer to parameterise a segmenting algorithm.11/08/201611

Page 12Paired blob detection dents, pits

Blob pairing algorithm

This is one of my favourite contributions to this projectIn order to detect defects like dents, I came up with a simple and effective algorithm:I found that if I pair dark and bright blobs that satisfied certain Area / Intensity and maximum separation distance, I could detect defects like dents with really good accuracy.11/08/201612

Morphological features (blob contour)Texture features (pixel intensity)areacontlengthcircularitycompactnessdist_sigmaroundnessentropyanisotropyanisometrybulkinessstructure_factorphilengthconvexitymean_graystd_grayenergycorrelationhomogeneitymean_graystd_grayenergycorrelationhomogeneitycontrastentropyanisotropy

Invetech confidential file: File name/number (use macro : alt+f8 > Run footer)Features for classificationMLP training Overfitting problemAdjusted the number of neurons in the hidden layer (30 neurons)Used PCA to reduce the number of neurons in the input layerMultiple scans of the same annotated strip evaluate repeatability

Once the potential blobs are detected you need to classify them into defect or non defect using some classification modelFor this project, I chose a MLPs networks (thats a type of ANN) with a single hidden layer of neuronsOne of the biggest problem with ANN and machine learning in general is that is hard provide a meaningful input and at the same time avoid the overfitting problem.The overfitting problem happens when the size of the network too large for the training set.In this case, the algorithm learns extremely well to classify correctly your training data, but performs poorly on other validation data of your problem domain.Because the amount of training data was short, I had to limit both the number of neurons in the input and hidden layer.For the input layer, I used PCA, which reduces the dimension of the input feature vector.For the hidden layer I just used the rule of thumb that recommends the double of the number of neurons in the input layer.To evaluate algorithm repeatability which proved to be an issue in some cases I scanned the same strip multiple times and validate classification against them to find out how consistent the results were. 11/08/201613

Page 14Development Tools - HDevelop

Great REPL experienceDetailed feature inspectionInteractive region filtering100+ real life examples out of the boxExport to C/C++/C#

$7000 Developer license$2000 Runtime licenseNot all operators have equivalent in OpenCV

So to prototype all these algorithms I mainly used a machine vision software package developed by a German company called MVTecThis machine vision kit is called HALCON (Falcon) and includes an IDE called HDevelop with a simple yet powerful scripting languageTo the best of my knowledge there isnt anything as helpful as this tool for prototyping image processing algorithms:It has a great REPL you can set the executing cursor in whatever line you want to execute next, you can overlay images and regions to see intermediate results without having to write any code whatsoeverYou can do detailed feature inspection, create custom selection filters interactively to filter regions of interest and so much more.It also comes with more than 100 different real life examples sorted by industry, method, etc. that you can study and adapt to your specific problem with little effortThe downside is that as good as it sounds it is about $7000 per developer license and $2000 for run-time license only useful for prototyping or if developing some really expensive instrument or system (e.g. MRI, CAT scanner) Also not all operators have a direct equivalent in OpenCV but usually enough documentation is provided to understand how the algorithm works so that it could be implemented if needed (reference to research papers)11/08/201614

Page 15Algorithm prototyping with HDevelop

This screenshot for example shows the algorithm prototyped to detect blobs in lighting AOn the top left you have a graphics window where you can see the results on each execution step if run step by stepBelow you get the list of iconic variables in your executing procedure and you can clear the window and overlay the results as you likeA bit below are the control tuples that is, the control variables you use to configure your operators (list of integers, doubles and strings)On the right hand side you can see the algorithm code you are developing and testing.The green arrow is the next executing instruction, which you can change to point anywhere in the procedure or outside of it.11/08/201615

Early prototyping is critical to mitigate project riskInvest as much effort as possible on developing a large, high-quality validation set of images and ideal resultsHelps to understand the problem spaceProvide confidence in the robustness of the algorithmsInvolve the client in the annotation process of a validation set (if the project contains supervised training)Manage client expectation on what the system will be capable ofGet the client to agree on objective measurements of system performanceTransparency on the outcome of the projectNon technical stakeholders usually have high expectation of what Computer Vision can do

Page 16Lessons learnt and recommendations

Firstly, Early prototyping is critical to mitigate project risk this is no news for anyone here, but where there is image processing involved, there is always uncertainty and this is because a camera never produces the same image twice.I have worked in projects that you get to a solution that works fine 99.9% of the time, but from there to get to 100% of the requirement could take 2 or 3 times more.Secondly, spend as much effort as possible in developing a large, high-quality validation set of images and ideal resultsHelps to understand the problem space which is really important when prototypingProvide confidence in the robustness of the algorithms you know you are covering a big chunk of the problem spaceThirdly, Involve the client in the annotation process of a validation set (if the project contains supervised training)Finally, get the client to agree on objective measurements of system performance- Non technical people usually have high expectations on what computer vision can do I blame Hollywood for this

11/08/201616

Page 17Thank you!DEMO time!