Computer Aided Engineering Applications 5. Machine Visionoscar/wp-content/uploads/2015/01/5_Machine_V… · 5.3 Spatial filtering • Intensity transformations-Changing the image

Computer Aided Engineering

Applications5. Machine Vision

5.1 Introduction

5.2 The camera model

5.3 Spatial filtering

Engi 6928 - Fall 2014

5.3 Spatial filtering

5.4 Segmentation

5.5 Blob analysis

5.6 Classification

5.7 Depth perception

5.1 Introduction5.1 Introduction

• Machine vision is the technology and

methods used to provide imaging-based

automatic inspection and analysis.

Applications include : Inspection of

parts, guidance, process control.

• Digital Image processing is the process • Digital Image processing is the process

of conditioning and extracting

information from digital images.


• Applicable to many fields

RGB Images UV Images IR Images

MRI ImagesSAR ImagesUltrasound Images


• Structure of an Industrial Machine vision system

Image

Acquisition

Filtering

Transformation

10

7

NUT

x,y,z

Metric

estimation

Segmentation

Blob analysis

Object

detection

Transformation

6

1

4

2

5

9

3

8

5.2 Camera Model5.2 Camera Model

• The Pin hole camera model

{C}

CCD/CMOS image sensor

{W}

This transformation looses the depth information

Pixel Position:

Intensity:


• Intensity – The value of a pixel position

>> f=imread([pwd '\Fasteners_2_gray.pgm']);

>> close all

>> imshow(f)

>> f(1,1)

ans =

120

xpx

ypx

Finding intensity of f(0,0) in Matlab{I}

• RGB images have multiple channels. i.e. an intensity vector

for each pixel position>> f=imread([pwd '\Fasteners_2.bmp']);

>> f_info=imfinfo([pwd '\Fasteners_2.bmp']);

>> f_info.BitDepth

ans = 24

>> size(f)

ans = 1040 1392 3

>> f(1,1,:)

ans(:,:,1) = 118 ans(:,:,2) = 122 ans(:,:,3) = 114

Intensity levels : 2^24

Pixel size: 1040 x 1392

Channels: 3 (RGB)

RGB values


Image TransformationsClear; close all; clc

f=imread([pwd '\Fasteners_2_gray.pgm']);

f_info=imfinfo([pwd '\Fasteners_2_gray.pgm']);

subplot(1,2,1)

imshow(f);

m=size(f,2); %width

n=size(f,1); %height

%1. translate

T=[1 0 100;

0 1 100;

1. Translation

0 1 100;

0 0 1];

for uy=1:n

for ux=1:m

u=[ux uy 1]';

up=uint16(T*u);

if up(1)>0 && up(2)>0 && up(1)<m && up(2)<n

g(up(2),up(1))=f(uy,ux);

end

end

end

subplot(1,2,2)

imshow(g);

High level function

imtranslate(f,[tx ty]);


Image Transformations

2. Rotation

3. Scale

High level function

imscale(f,scale);

High level function

imrotate(f,angle);

5.3 Spatial filtering5.3 Spatial filtering

• Intensity transformations- Changing the image intensity level

at a location with a mathematical relation which does not

depend on the neighbouring pixels

• Spatial filtering - Changing the image intensity level at a

location with a mathematical relation which depends on location with a mathematical relation which depends on

intensity of neighbouring pixels.

1. Image Negative

%Intensity Transformation - Image Negative


g=2^8-1-f;

imshow(g);


2. Gamma correction

%% Gamma correction


g=uint8(1*double(f).^(0.8));

subplot(1,2,2)

imshow(g);

3. RGB to gray

%% RGB to gray

f=imread([pwd '\Fasteners_2.bmp']);

subplot(1,2,1)

imshow(f);

g(:,:)= 0.2989 * f(:,:,1) + 0.5870 * f(:,:,2) + 0.1140 *

f(:,:,3) ;

% g= rgb2gray(f); % high level function

subplot(1,2,2)

imshow(g);RGB Gray


4. RGB to HSV

%% RGB to hsv

f=imread([pwd '\Fruits.jpg']);

subplot(2,2,1)

imshow(f);

g = rgb2hsv(f);

h = g(:, :, 1); % Hue image.

s = g(:, :, 2); % Saturation image.

v = g(:, :, 3); % Value (intensity) image.v = g(:, :, 3); % Value (intensity) image.

subplot(2,2,2)

imshow(h)

subplot(2,2,3)

imshow(s)

subplot(2,2,4)

imshow(v)

Separates image intensity (luma) from image color (chroma).

Processing in the luma channel is robust towards lighting changes.

Intensity


• A spatial filter is viewed as a convolution (A weighted

sum of a window) using a convolution kernel.


5. Average filter (Low pass filter)

%% spatial filter - Averaging (box filter)


close all

h = ones(40,40) / 40^2;

g = imfilter(f,h);

imshow(g)

6. High Pass filter

%% spatial filter - High Pass


close all

h = -ones(5,5);

h(3,3)= 5^2;

g = imfilter(f,h);

imshow(g)

9

10x 10

4

5.4 Segmentation5.4 Segmentation

• Image histogram

A bar plot where the x axis (bins) are different

intensity levels. Y axis plots the number of pixels

having the intensity level x.

%% Gray level histogram

close all

0 50 100 150 200 2500

1

2

3

4

5

6

7

8

9

X = 74Y = 772

close all


m=size(f,1);

n=size(f,2);

g_hist=zeros(1,256);

for i=1:m

for j=1:n

g_hist(f(i,j)+1)=g_hist(f(i,j)+1)+1;

end

end

bar([1:256],g_hist)

axis([0 256 0 10^5])

% imhist(f) % high level function

772 pixels with

Gray level 74


• Gray to binary conversion

%% Gray to binary

close all


M=size(f,1);

N=size(f,2);

Thresh= 100;

The histogram exhibits 2 gray levels. A threshold

can be used to generate binary images

Thresh= 100;

for i=1:M

for j=1:N

if f(i,j)<Thresh

g(i,j)=1;

else

g(i,j)=0;

end

end

end

g=logical(g); % data type conversion to bool

%g=im2bw(f,Thresh/2^8); % high level function

imshow(g);0 50 100 150 200 250

0

1

2

3

4

5

6

7

8

9

10x 10

4

X = 74Y = 772

100


• Otsu`s Thresholding algorithm

A method to generate the threshold value automatically. The

method finds the threshold which maximizes the between

class variance.

%% Otsu's method

close all

f=imread([pwd '\Fasteners_2_gray.pgm']); 6

7

8

9

10x 10

4

72f=imread([pwd '\Fasteners_2_gray.pgm']);

threshold = graythresh(f);

g = 1-im2bw(f,threshold );

imshow(g)

0 50 100 150 200 2500

1

2

3

4

5

X = 74Y = 772

72


1. Calculate probability distribution from histogram

2. For each threshold value calculate between class

variance

3. Select the threshold which maximizes the between

• Otsu`s Thresholding algorithm

3. Select the threshold which maximizes the between

class variance

0 50 100 150 200 250 3000

5

10x 10

4

0 50 100 150 200 250 3000

0.05

0.1

0 50 100 150 200 250 3000

1000

2000


• Example


• Connected component labelling

Separates the binary image to a set of segments.

Each pixel belongs to a segment if it satisfies a pixel

connectivity condition.

Input binary image

4 connectivity labelling

8 connectivity labelling


• Connected component labelling

%% Connected components

close all


threshold = graythresh(f);

g = 1-im2bw(f,threshold );

cc = bwconncomp(g,8); %find 8 neighbor

connected componentsconnected components

L = labelmatrix(cc); %label the image pixel with

the indices of the components

RGB = label2rgb(L); %convert the label values to

unique colors

imshow(RGB)

cc = Connectivity: 8

ImageSize: [1040 1392]

NumObjects: 13

PixelIdxList: {1x13 cell}

Because of noise of the binary

image, undesired segments may

arise

㛀Ѯ


• Noise removal

Remove background connected

components which are small.

%% Noise removal - bwareaopen

close all


threshold = graythresh(f);threshold = graythresh(f);

g = im2bw(f,threshold );

g = bwareaopen(g, 5000); %remove background

connected compenents which have less than

5000px

imshow(1-g)

㛀Ѯ

5.5 Blob analysis5.5 Blob analysis

• Blob analysis quantifies different segment

descriptors to help with object identification.

• Depending on the application the segment

descriptors would require to be:

– Translation/ rotation invariant: Area, major axis length,

perimeter, generalized momentsperimeter, generalized moments

– Scale invariant: number of holes , complexity =

perimeter^2/ area, Invariant moments

㈐͵


• Matlab command regionprops calculates many

segment descriptors.

%% Blob anlysis- reigion props

clear;clc;close all


g = im2bw(f,graythresh(f));

g = 1-bwareaopen(g, 5000);

imshow(g)2

3

4

6

8

imshow(g)

cc = bwconncomp(g,8);

s = regionprops(cc,'All');

hold on

for i=1:cc.NumObjects

bbox=s(i).BoundingBox;

rectangle('Position', bbox,'EdgeColor','b')

center=s(i).Centroid;

text(center(1),center(2),[num2str(i)],'Color','r');

end

1

2

5

7

9 10

Euler number =

number of objects – number of holes


( , )m n

i jM x y f x y=∑∑

•Generalized moments can be used to represent different

descriptors of a segment.

•In the case of a digital image of size n by m pixels, the

generalized moment is given by:

1 1

( , )i j

ij

x y

M x y f x y= =

=∑∑

•For binary images the function f(x,y) takes a value of 1 for

pixels belonging to class “object” and “0” for class

“background”.


0 1 2 3 4 5 6 7 8 9

1

Xi j Mij

0 0 7 Area

∑∑= =

=n

x

m

y

ji

ij yxfyxM1 1

),(

1

2

3

4

5

6

Y

1 0

0 1

2 0

0 2

1 1

20

33

159 Moment

of Inertia64

93

䃀�


00

10

M

MX =

00

01

M

MY =

•The center of mass of a region can be defined

in terms of generalized moments as follows:

•The moments of inertia relative to the center of

mass:

00

2

010202

M

MMM −=

00

2

102020

M

MMM −=

00

01101111

M

MMMM −=

0220

1122

MM

MTAN

−=θ

•Principal axis:


䫠ѫ


•Central Moments

•Normalized central moments

•Generalized Moments

1 1

( , )m n

i j

ij

x y

M x y f x y= =

=∑∑

1 1

( ) ( ) ( , )m n

i j

ij

x y

x x y y f x yµ= =

= − −∑∑

Xc=M10/M00;

Yc=M01/M00;

%Central moments

mu=0;

for i=1:n

for j=1:m

mu=mu+ (j-Xc)^p*(i-Yc)^q*f(i,j);

end

end•Normalized central moments

•Hu`s Invariant moments

( 1)2

00

ij

ij i j

M

ηµ+ +

=

end

%Normalized Central moments

nu=mu/M00^(1+(p+q)/2);

䫠ѫ


function [Hu]=hu_moments(f)

m=size(f,2);

n=size(f,1);

[M20 mu20 nu20]=all_moments(f,2,0);






function [M,mu,nu]=all_moments(f,p,q)

m=size(f,2);

n=size(f,1);

%Moments

M=0;

for i=1:n

for j=1:m

M=M+ j^p*i^q*f(i,j);[M30 mu30 nu30]=all_moments(f,3,0);


Hu(1)=nu20+nu02;

Hu(2)=(nu20-nu02)^2+4*nu11^2;

Hu(3)=(nu30-3*nu12)^2 + (3*nu21+nu03)^2;

Hu(4)=(nu30+ nu12)^2 + (nu21+nu03)^2;

Hu(4)=(nu30-3*nu12)*(nu12+nu30)*((nu30+nu12)^2-

3*(nu21+nu03)^2) + (3*nu21-nu03)*(nu21+nu03)*(-

(nu21+nu03)^2+3*(nu12+nu30)^2) ;

M=M+ j^p*i^q*f(i,j);

end

End

%Central Moments

…

%Normalized Moments

…

䫠ѫ


Area 39303.000000 39292.000000 165640.000000 165634.000000

Major axis 416.035508 415.994256 837.636983 837.611434

Complexity 32.144061 36.683384 33.381221 37.419965

Slenderness 2.942043 2.941993 2.873593 2.873517

Hu`s moments 0.307037,

0.059264,

0.004303,

0.000003

0.307063,

0.059274,

0.003767,

0.000003

0.296804,

0.054142,

0.003507,

0.000002

0.296798,

0.054138,

0.005377,

0.000002

Tra

nsf

orm

ati

on

an

d S

cale

In

va

ria

nt

浐ѫ

5.6 Classification5.6 Classification

• Classifiers are used for object identification in

machine vision systems.

• An object has a set of features which is termed as a

feature vector.

• The classifier attempts to identify which class the

feature vector belongs to.feature vector belongs to.

• Supervised learning is the process of tuning the

classifier using a training data set.

浐ѫ


1-feature-n-class classifier

• Normalize the calculated features.

• Sort the image segments according to the selected

feature.

• Assign classes in the ascending order • Assign classes in the ascending order

– !only applicable when all classes are present

Better approach: Assign class according the closeness

to the object in the original image (Nearest

neighbour)


1. Generate feature vectors for each object in the training dataset

2. Normalize the feature vector

3. Train the classifier (assign class

Nearest Neighbour classifier

(m-feature-n-class classifier)

1

oo

oo

o

Test Data

x

1111 11

Assign to

nearest class

Fea

ture

2

3. Train the classifier (assign class mean)

1. Generate the feature vector of the test object

2. Normalize using the trained parameters

3. Find the nearest class

oo

oo

Training Data

Feature 1


Cc1 : connected components of training image

Classes1: Assigned classes of training image

Feature_vector1: feature vectors of training image

Feature_vector1_norm: Normalized feature vectors of training image

Cc2 : connected components of test image

Feature_vector2: feature vectors of test image

Feature_vector2_norm: Normalized feature vectors of test image

Weight : weight given to each feature

Nearest Neighbour classifier

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

4

7

5

1

10

8

3

2

6

9

classes2=[];

for j=1:cc2.NumObjects

for i=1:cc.NumObjects

distance_measure(i)=norm((feature_vector2_norm(j,:)-feature_vector1_norm(i,:))*weight(i));

end

[minval Id]=min(distance_measure);

classes2(j)=classes1(Id);

end

Weight : weight given to each feature

Classes2 : nearest class of each feature vector (found by the following code)

浐ѫ


GUI improvements

– Euler number checks (produce warnings for

inconsistent matches)

– Invariant features (incorporate Hus moments)

– User Visualizations and warnings (allow user to

clearly verify results and adjust critical parameters)clearly verify results and adjust critical parameters)

浐ѫ


• SURF feature matching

matched points 1

matched points 2

clear

clc

close all

I1=imread([pwd '\Fasteners_1.bmp']);

I2=imread([pwd '\Screw_6.bmp']);

I1=rgb2gray(I1);

I2=rgb2gray(I2);

points1 = detectSURFFeatures(I1);


matched points 1

matched points 2


[f1, vpts1] = extractFeatures(I1, points1);

[f2, vpts2] = extractFeatures(I2, points2);

indexPairs = matchFeatures(f1, f2) ;

matchedPoints1 = vpts1(indexPairs(:, 1));

matchedPoints2 = vpts2(indexPairs(:, 2));

figure;

showMatchedFeatures(I1,I2,matchedPoints1,matchedPoints

2);

legend('matched points 1','matched points 2');

浐ѫ

5.7 Depth perception5.7 Depth perception

1. 3D vision – Range sensors

Time of flight cameras Combined camera and laser scanner

r measured by range sensor

浐ѫ


2. 3D vision – Structured light

浐ѫ


3. 3D vision – Stereo vision/ Multi camera systems

-3-2

-10

12

3

-2

0

2

0

2

4

6

浐ѫ


4. 3D vision – Known 3D object

浐ѫ

Vision Based Inspection of Liquid

Crystal Display (LCD) modules

•Objective: to automate the

inspection of LCD modules in

order to improve quality control

•One step in the implementation

of a Six-Sigma Program (“3.4

defects per million parts”)

•The inspection must be

completed within 30 seconds for

10 predetermined LCD patterns

•System can “learn” new LCD

modules without modifying

software

浐ѫ

Vision Based Inspection of LCD modules:

System Components

•Pulnix camera with macro

lens

•High frequency fluorescent

light sources

•Coreco Bandit integrated

image acquisition and VGA

11/22/2014

Memorial University of Newfoundland

image acquisition and VGA

accelerator

•Software developed using

with WiT graphical

programming environment

in combination with

Microsoft VB

浐ѫ



Original Image Showing Error in Alignment

姀ѵ



Thresholding Operation – Image Subtraction with

respect to an image with no segments illuminated

姀ѵ



Blob Analysis – Reference Points are Identified

姀ѵ



Image Rotation and Translation

姀ѵ



Pixel by Pixel Image Subtraction from Reference

Image – Thinning Operator

姀ѵ



Blob Analysis

姀ѵ

ReferencesReferences

• Digital image processing using MATLAB - Gonzalez, R. C.,

Woods, R. E., & Eddins, S. L. (2004). Upper Saddle River, NJ

Jensen: Prentice Hall,.

• Multiple view geometry in computer vision. Hartley, Richard,

and Andrew Zisserman. Cambridge university press, 2003.and Andrew Zisserman. Cambridge university press, 2003.

Documents

Computer Aided Engineering Applications 5. Machine Visionoscar/wp-content/uploads/2015/01/5_Machine_V… · 5.3 Spatial filtering • Intensity transformations-Changing the image