47
Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of Western Ontario

An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Embed Size (px)

Citation preview

Page 1: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

An Agent Based Architecture For Designing Flexible Image Analysis Systems

Jagath Samarabandu

Dept. of Electrical and Computer EngineeringUniversity of Western Ontario

Page 2: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Multi-dimensional Imaging System

Capture Processing Visualization

Storage

Confocal Microscopy

Conventional Microscopy with Deconvolution

Tomography with Reconstruction

MR, CT, PET, SPECT, US

ROI Extraction

Digital Filtering

Segmentation

Analysis

Morphometry

Sectional Views

MIP/Sum Projections

Surface Shading

Volume Rendering

Stereo views

Animation Sequences

Interactive displays

Raw data

Region dataProcessed dataHigh level data

Single frameStereo frameMovie sequences

Page 3: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Acknowledgements Kamal Ranaweera – MESc student who

is doing most of the work Dr. Aaron Fenster for his mentorship CITO Round 3.2 research grant

Page 4: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Outline Why image analysis Image processing basics Image analysis paradigm Analysis system example A flexible framework for image analysis Knowledge organization Current status Outlook

Page 5: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Why Image Analysis Image dimensions are increasing Image sizes and resolutions are

increasing Image quality is getting better Cannot visually assess all the voxels

Page 6: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Image Processing Point Operations

Histogram Equalization Contrast Stretching

Spatial Operations Linear Filtering

Low/High pass filtering Edge Detection

Non-Linear Filtering Median filtering Morphological Filtering

Page 7: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Point OperationsHistogram EqualizationIntensity histogram of the output image has a uniform distribution

Page 8: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Spatial Operations Linear Filtering

g(x,y) = ijh(i,j) u(i+x, j+y) where u(x,y) is the input image intensity at

(x,y) and h(i,j) is the value of convolution kernel at (i,j)

1 1 11 1 11 1 1

-1 0 1-1 0 1-1 0 1

-1 -1 -1 0 0 0 1 1 1

MovingAverage

Pair of kernels for edge detection

Example kernels

Page 9: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Edge Detection Operators Gradient Operators

Finite difference approximations of orthogonal gradients (e.g.. hx and hy for 2D images)

gm (x,y) = gx(x,y)2 + gy(x,y)2

g(x,y) = tan -1 (gy(x,y) / gx(x,y))

-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1

-1 0 1 -1 -1 -1-1 0 1 0 0 0-1 0 1 1 1 1

0 1 1 0-1 0 0 -1

Roberts Prewitt Sobel

Page 10: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Edge Detection Operators Compass Operators

Directional gradients at a given number of directions (h1, h2, h3 .. hn)

gm (x,y) = MAXk {| gk (x,y)|} g(x,y) is k at which the maximum occurred

-1 0 1-1 0 1-1 0 1

0 1 1-1 0 1-1 -1 0

1 1 1 0 0 0-1 -1 -1

1 1 0 1 0 -1 0 -1 -1

1 0 -1 1 0 -1 1 0 -1

0 -1 -1 1 0 -1 1 1 0

-1 -1 -1 0 0 0 1 1 1

-1 -1 0-1 0 1 0 1 1

Kirsh templates for 8 directions

Page 11: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Edge Operators

Roberts Prewitt Sobel

Compass Kirsch Laplace

Page 12: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Spatial Operators Linear Filters

Gaussian smoothing Edge detection operators

Non Linear Filters Median filter Rank order filters morphological filters

Page 13: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Spatial Filters for Smoothing

Gaussian Noise Gaussian (7x7) Median (5x5)

Speckle noise Median (5x5)Gaussian (7x7)

Page 14: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

High Level Processing Feature Extraction

Edges and Boundaries Shape and Moments Texture

Segmentation Matching Thresholding Boundary Detection

Classification Statistical Classification Clustering

Page 15: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Canny’s Edge OperatorCombines edge operators and adaptive thresholding with

hysterisys to locate edges.

Page 16: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Image Analysis

PictureInterpretation

Image

Picture DomainCues

Scene DomainCues

Model

InstantiatedModel

Scene Domain

Picture Domain

Signal

Physical

Semantic

Kanade’s paradigm

Page 17: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Bottom Up Cycle Picture domain cues are primitive image

features Scene domain cues are the

interpretation of image features This distinction is somewhat blurred for

3D images

Page 18: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Top Down Cycle Model instantiation or hypothesis

generation Verification and refinement

Page 19: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Data Driven Approach Example

Median Filtering

Difference of Gaussian

Low Pass filter

Boundary Forming

Thresholding

Feature Aggregation

Page 20: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Feature Aggregation

This technique was able to suppress concavities

It also tended to diffuse boundaries of small structures

lyxyxH

L

yxyxl

lyx

lyx

lyx

yxeE

i

iii

iii

i

Ekwii

,| and

Otherwise

if

if

Where

,,max

min

,,

1,

1,,

00

0000

Page 21: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Processing Confocal Images

Single section of the original image

2D median filtering

Page 22: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Processing Confocal Images

Difference of Gaussian filter 3D contours of final segmentation

Page 23: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Detecting DNA Replication Sites Bottom up

Intensity maxima are detected on 2D image slices

Sites cross sections are modeled as circles centered on these maxima

Top Down Site boundaries are then refined according to the

local image features These boundaries are then combined to form 3D

structures using 3D connected component labeling

Page 24: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

DNA Replication Sites (MIP)

Page 25: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Segmented Replication Sites

Page 26: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Further Analysis

Page 27: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Further Analysis

Page 28: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Blackboard Model Provides us an elegant architecture

within which both data driven and model driven approaches can be incorporated

A problem is solved in an incremental fashion

Contains a set of knowledge sources or tools

Page 29: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Blackboard Model These tools are designed as

independent modules They interact with each other via a

global database called the blackboard Modules that generate solutions are

separated from those that determine their utility

Page 30: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Blackboard Models can handle Large solution space Noisy and unreliable data Variety of input data and a need to

integrate diverse information The need to use many independent or

semi-independent sources of knowledge to co-operate in forming a solution

Page 31: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Blackboard Models can handle The need to use multiple reasoning

methods (eg. Forward and backward) The need for multiple lines of reasoning The need for an evolutionary solution

Page 32: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Requirements for the Framework

Represent and use knowledge about task domain and processing tools

Evaluate the quality of the image processing algorithms

Transfer information between different tools and knowledge sources

Control the flow of processing Allow easy addition of new tools

Page 33: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Requirements for Processing Tools

Implements the underlying processing algorithm

Contains knowledge about its algorithm Can evaluate its suitability (matching

score) to accomplish a given task Communicates with the framework

Page 34: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Knowledge Organization Domain Expertise

Knowledge about the image Intensity characteristics of the structure of

interest Image Processing Expertise

Knowledge about image processing algorithms (e.g. filtering, detection) and their parameters

Page 35: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Domain Knowledge Initial Model (e.g. circle, line, sphere) Parameter value distribution Expected orientation Boundary definition method (e.g.

intensity +/-, texture, color etc.) Boundary definition strength (strong,

medium, weak) Boundary dispersion (short, medium,

long)

Page 36: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Image Processing Knowledge: Filter Class

Filter type (Gaussian, median, morphological etc.)

Frequency response (low pass/high pass) Linearity (linear/non-linear) Filter parameters (e.g. kernel size,

sigma, structuring-element etc.) Performance against different noise

types (Gaussian, shot noise)

Page 37: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Image Processing Knowledge: Detector Class

Detector name Primitive type

Intensity gradient edge Intensity peak Region

Effective parameter range Confidence against

Strong/medium/weak boundary definition short/medium/long boundary dispersion

Page 38: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Knowledge Representation Information is

encoded using XML Attribute names and

value types are defined for each piece of information

Sub-classing to avoid duplicating information

A 0-1 rating is used to classify behavior when applicable

<filter type=“median” linear=“no” sub-type-of “rank”>

<transfer-function type=“lowpass”/>

<parameter name=“window” min=3 max=100/>

<noise-performance type=“gaussian” rating=0.5/>

<noise-performance type=“shot” rating=0.9/>

</filter>

Page 39: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Design of Tools Each tool is an independent object (agent) Able to emit the XML description of self for

use by the framework (or others) All tools are sub-classed from a base type

that has three primary “methods” Judicious use of OOP features such as

“inheritance” and “polymorphism” helps the design of tools and the framework

Page 40: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Implementing the Tools All tools have the following functionality

Self – Describe itself using XML notation Match – Gives a compatibility score against

an input description SetInput – Defines the input data Process – Performs the processing operation GetOutput – Extract processed data

Example current.SetInput(prev);current.Process();

next.SetInput(current); …

Page 41: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Processing

SequenceGeneratio

n

Processing

Sequence Execution

Tool Library

Task Descriptio

n

Image Descriptio

n

Output

User Control

System Architecture

Page 42: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Framework First implementation was script based Simply arranged the utilities in the

prescribed order Performed the sequence of operations Had limited user interface for arranging

operations Contained two detectors (peak and edge) Mostly used as a test bed to develop the

tool interface

Page 43: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Current Implementation Resembles an expert system Inputs are

Detector type (peak, line) Edge strength and diffusion Structure size Noise level

Creates a sequence of operations Performs the sequence at user’s request

Page 44: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Limitations of the Current System

Linear progression of tools Tools with multiple inputs are not yet

supported Iterative processing methods are not

supported XML tools are not fully mature yet. Not enough tools

Page 45: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Future Work Addressing current limitations Using this system as a knowledge

acquisition tool Research on machine learning Possibilities are endless!

Page 46: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Summary Image analysis helps managing large

amounts of data and extract new information

Agent based framework attempts to use multiple processing tools to improve performance

Incorporating knowledge about analysis tools helps create a user centered system

Flexibility comes from being able to add new agents without modifying the framework

Page 47: An Agent Based Architecture For Designing Flexible Image Analysis Systems Jagath Samarabandu Dept. of Electrical and Computer Engineering University of

Thank You!

Aruni admiring a Water Lilly