Electronic board defect classification and detection with deep learning

Dan Sebban – VP of Data Analysis, Optimal+

Nissim Matatov – Machine Learning Engineer, Optimal+

What is machine learning

Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed

Machine learning and AI

Artificial intelligence

Machine learning

Deeplearning

Artificial intelligence: Any technique that enables computers to mimic human intelligence.

Machine learning: A subset of AI that includes statistical techniques that enable machines to improve at tasks with experience.

Deep learning: The subset of machine learning that permits software to train itself to perform tasks like speech and image recognition, by exposing neural networks to vast amounts of data

Artificial intelligence (AI)

enables computers to mimic human intelligence by providing key human abilities

Bunny is in picture

Computer visionkey AI capability

Deep learningkey methodology to achieve AI Bunny is here

Image classification: Who is in picture?

Seecomputervision

Hearspeech recognition

Comprehendnatural language processing

Deep learning in electronics –MotivationVisual Inspection (VI) aims to check for the presence of defects on a board

Before: Manual visual inspection

• Very laborious – impacts operational and manufacturing efficiency

• Depends on human ability to recognize defects – impacts quality

After: Computer-aided visual inspection

• Automated – improves efficiency

• More accurate in recognizing defects –improves quality

Case study introduction

• Electronic Ceramic Substrate welded through several pins to a housing

• All parts are inspected as part of the manufacturing flow using a Surface Acoustic Microscope (SAM)

• Images are generated by the microscope

• Images are labeled by technician as “Defect”/”No Defect”

Defect Image ClassificationWhether an image contains a defect

Computer vision tasks in electronics

Score : 0.08 Score : 0.32 Score : 0.92

Score for “Star” crack:

Score : 0.08 Score : 0.18 Score : 0.97

Defect ClassificationWhether an image contains a specific defect type (e.g. crack with “Star” shape)→ Allows root cause analysis

Defect DetectionWhere the defect is actually located

Business goalMaximum test time saving through VI reduction at minimum quality impact

ML goal Defect Image Classification and accurate prediction of boards without defects

ML task Supervised ML for binary classification (Defect / No Defect)

Action to save test time

Skip boards which are predicted safe (No Defect) with high degree of confidence

Evaluation Estimated number of boards with undetected defects per VI reduction level

Deep learning framing

Typical VI

inspection flow

Image Capture by

SAM

Manual InspectionPass/Fail

Proceed to next test

No Defect

Deep learning

model is

deployed to a

factory floor

Image Capture by

SAM

DL modelScore each board image for defect

Proceed to next

operation

Manual Inspection

High scored boards

Low scored boards

Defect

Engineering Disposition

Deep learning model deployment

Input data DL evaluationDL modeling

• “Defect”/”No Defect” ratio: ~1:10

• Manual labeling “Defect”/”No Defect” of 600 images

• VGG-16 Convolution Neural Network (CNN) structure

• Image augmentation pre processing procedure

• Transfer Learning

• Hyperparameter optimization for CNN

• Ensembling

• Modeling data (600 images): “Defect”/”No Defect” ratio: 1:5

• Evaluation(300 images*10 iterations): “Defect”/”No Defect” ratio: 1:10

Deep learning model deployment

No Defect Defect

No

Defect

TP(True positive)

FN(False negative)

Defect

FP(False positive)

TN(True negative)

Predicted

Classic classification performance measuresTP + TNTP + TN + FP + FNAccuracy =

Evaluation data: 12 boards (8 “No Defect” + 4 “Defect”)Random board selection → 50% skip = 2 escapes

Model based board selection (based on scoring) → 50% skip = 1 escape

Lift = 2 [DL model is 2x better than random selection]50% selection

TNTN + FPRecall(Sensitivity)

=

Lift [more intuitive metric] = 𝐑𝐚𝐧𝐝𝐨𝐦 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧𝐌𝐨𝐝𝐞𝐥 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧

AUC-ROC =

Example

Technical evaluation

Business evaluation

Skip level vs. avg. escapes tradeoff

Proposed skip level = 40%

60% of the boards will continue with human VI

40% of the boards will skip VI

Escape rate is 1%, or ~ 3 boards out of 300

Escapes ratio becomes faster for higher skip levels

Escape ratio is acceptable by customer for potential TTR

Strong DL model requires one time effort to create accurate

“Defect”/”No Defect” labeling

Existing image labeling isn’t totally accurate and negatively impacts model performance

Skip level 40%

VI escaped boards - Random (boards) 12.5

VI escaped boards - Random (% of total) 4.17%

VI escaped boards - Model (boards) 2.9

VI escaped boards - Model (% of total) 0.97%

Lift 4.3

Examples for Prediction vs. Actual

Case 1

Prediction = “No Defect” and Actual = “No Defect”The model is confident that defect is present despite poor image quality

Case 2

Prediction = “No Defect” and Actual = “Defect”Room for model improvement

Case 3

Prediction = “Defect” and Actual = “No Defect” Defect is clearly seen: labeling should be revised to improve the model

Defect detection evaluation

IoU = 0.79 IoU = 0.18

Actual bounding box

Predicted bounding boxv

Evaluation metric:

IoU (Intersection over Union)

DL model with mean IoU > 0.5 is considered strong

Case study mean IoU metric is ~0.65

Deep neural network in a nutshell

Input Layer

Pixelwise image presentation

Convolution + Pooling layers

Learn about elements of image , i.e. edges

of objects

Full Connected Layer

Learn about object presence from previous

information

Classification Layer

Express confidence about object presence

Deep learning methodologies

Data augmentation

Transform original images to create new images for learning

Transfer learning

Use previously accumulated knowledge during the learning

External feature embedding

Use other inspections along with information learned from image

Assembling

Combine partial models to provide more accurate final prediction

DL visualization

Watch that intermediate results make sense

The machine learning process

Adapt

Learn Act

Validate

Learn from data and evaluate business value

Understand changes and update model

Deploy and act upon the model

Monitor model performance and identify changes

ML/DL application challenges

Fully automated test process

High cost to interrupt the inspection process

Low volume - Low cost products

Effort on adaptivity is ineffective

New product or new technology

Not enough relevant images are available for analysis

Qualitative data

Images quality and their correct labeling

High performance DL

Computationally expensive

Model deployment and operationalizing

Model benefit is materialized and maintained

ML/DL deployment ecosystem

The algorithm is not enough – it’s the infrastructure that is challenging

Google article from 2014: Hidden Technical Debt in Machine Learning Systems

Key takeaways

• Significant value was demonstrated using Deep Learning for Defects Inspection

• Such methodology improves both operational efficiency, manufacturing throughput,

overall product quality, and reliability

• Deep learning is independent of defect types and locations, and therefore a more

suitable methodology than classical image processing on some specific tests cases

• The recent significant improvements in Deep Learning techniques allows to get faster

and more reliable models

Thank you!