Nuclei of fungus cell Paramecium Caudatum

ter Haar Romeny, FEV

Nuclei of fungus cell Paramecium Caudatum

Spatial gradient Illumination spectrum-invariant gradient

Color RGB original

Color-Scale Differential Structure

Geusebroek et al, LNCS 1852, 459-464, 1999

ter Haar Romeny, FEV

The color of an object depends on

• color of the illuminating light• illumination intensity• sensor sensitivity• direction of surface normal• surface reflectance properties

Assumptions:• Scene is uniformly illuminated• light source is colored• surface has Lambertian reflectance

ter Haar Romeny, FEV

What causes color ?

Lamp

objectcolor

spectral

color

ter Haar Romeny, FEV

400 450 500 550 600 650 700

500 1000 1500 2000

50000

100000

150000

200000

250000

300000

350000

400 450 500 550 600 650 700 400 450 500 550 600 650 700 400 450 500 550 600 650 700

Object reflectance function for the observed spectrum for a resp. 2500K, 6500K and 10,000K light source:

Spectrum reflected froman arbitrary object

8 hc 5E hc

kT 11

Emissionspectrum ofblack bodyradiator

ter Haar Romeny, FEV

Color receptive fields

x

y

x

ter Haar Romeny, FEV

Self-organization:receptive fieldsfrom Eigenpatches(12x12 pixels)

ter Haar Romeny, FEV

Blakemore C, Cooper GF (1970), Development of the brain depends on the visual environment. Nature 228, 477 - 478

ter Haar Romeny, FEV

Colour receptivefields fromEigenpatches

x

ter Haar Romeny, FEV

GD Field et al. Nature 467, 673-677 (2010)

doi:10.1038/nature09424

ter Haar Romeny, FEV

Distribution of cone cells in the fovea of an individual with normal color vision (left), and a color blind (protanopic) retina. Note that the center of the fovea holds very few blue-sensitive cones. [Wikipedia]

ter Haar Romeny, FEVGD Field et al. Nature 467, 673-677 (2010) doi:10.1038/nature09424

Full functional sampling of cone lattice by four RGC types.

ter Haar Romeny, FEV

The color opponency model

Hering E, 1964. Outlines of a Theory of the Light Sense. Harvard University Press, Cambridge, Mass.

ter Haar Romeny, FEV

Hering basis

0 10 20 30 40 50 60 70 80 90-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

Idea Koenderink: Gaussian derivatives of zero, first and second order in the wavelength domain

wavelength

RFsensitivity

How can weanalyse colordifferential structure?

ter Haar Romeny, FEV

Taylor color model

Luminance

Blue-yellowness

Purple-greenness

300 400 500 600 700 800 90000.10.20.30.40.50.60.70.80.9

LM

S

Cone sensitivity

ter Haar Romeny, FEV

Spatial color

Color scale-space starts by probing this space.

y

x

xy

s

Energy densities cannot be measured at a point, …… one probes a certain volume

ter Haar Romeny, FEV

Reflectance of light

Lamp

objectcolor

spectral

color

What are invariantproperties?

ter Haar Romeny, FEV

Reflectance model

ter Haar Romeny, FEV

Transparent materials

ter Haar Romeny, FEV

)()),,(1()()( 2 RvsneE f

The reflected spectrum is:

v = viewing directionn = surface patch normals = direction of illuminationf = Fresnel front surface reflectance coefficient in vR = body reflectance

ter Haar Romeny, FEV

),())(1)(,(),( 2 xRxxexE f

Because of projection of the energy distribution on the image plane the vectors n, s and v will depend on the position at the imaging plane. So the energy at a point x is then related to:

We assume an illumination with a locally constant color:

),())(1)(()(),( 2 xRxxiexE f

ter Haar Romeny, FEV

Aim: describe material changes independentof the illumination.

Rxxie

exRxxi

E

f

f

2

2

))(1)(()(

),())(1)((

),())(1)(()(),( 2 xRxxiexE f

Bothequationshavemanycommonterms

ter Haar Romeny, FEV

R

xR

e

e

E

EE

),(

1

)(

11ˆ

The normalized differential

determines material changes independent of the viewpoint, surface orientation, illumination direction, illumination intensity and illumination color!

ter Haar Romeny, FEV

The derivative jet to x and forms a complete family of geometric invariants:

0

ˆˆ ( ; 515 ; 55 )

n

n

EE G nm nm

These are observed properties, so we convolve with Gaussian derivatives

ˆn m

n m

E

x

ter Haar Romeny, FEV

2 2ˆ ˆx yE E

Color edges can be defined as the thresholding of the spatial gradient (color-invariant equivalent of Lw):

2 2ˆ ˆx yE E

Color invariants

ter Haar Romeny, FEV

Spatial color model and tracing color edges in microscopy

Influence of illumination color temperature on edge strength, scale is 3.0 px.

Skin tissue section illuminated by a halogen bulb at 4000 K (top) and 2600 K (bottom) color temperature.

ter Haar Romeny, FEV

1

ee

x. ex, y, eshortnotatione exzx, y, exx, y, ezx, y,

e2

x,. ex, y, eshortnotation1

e3e2 exzzx, y, 2 e exzx, y, ezx, y, exx, y, 2 ezx, y, 2 e ezzx, y,

Color-invariant multi-scale structural operators

ter Haar Romeny, FEV

Simplifyx2 y2 x, 2 y, 2;shortnotation 1

ex, y, 6 ex, y, 2ex, y, exzx, y, exx, y, ezx, y, 2

ex, y, 2ex, y, eyzx, y, eyx, y, ezx, y, 2 ex, y, 2 exzzx, y, 2 exx, y, ezx, y, 2 ex, y, 2 exzx, y, ezx, y, exx, y, ezzx, y, 2 ex, y, 2 eyzzx, y, 2 eyx, y, ezx, y, 2 ex, y, 2 eyzx, y, ezx, y, eyx, y, ezzx, y, 2

Total edge strength

ter Haar Romeny, FEV

[im,] 1e

e

color invariant 1e

e

[im,]

first wavelength derivative of

[im,] 22 second wavelength derivative of

g[im,] x2 y2 yellow-blue edges

g [im,] 2x2 2

y

2 red-green edges

g[im,] x2 y2

2x2 2

y

2 total color edge strength

Somecolor

differentialinvariants

ter Haar Romeny, FEV

Feulgen stain,red-green edges

Paramecium caudatum, Feulgen and Fast green stain

Color canny, red-green normalized edges, scale 3

ter Haar Romeny, FEV

Hematoxylin eosin stain

Pituitary gland, sheep, adenohypophysis 40x

Cell: E<0, E > 0, scale 1.0

Nuclei: E <0, E > 0, E +E < 0, scale 3.0

additional constraint added to refine selection

ter Haar Romeny, FEV

Safranin O stain

E > 0, E > 0, scale sigma 1.0

Safranin O stain for proteoglycans (mouse knee

joint)

Courtesy of Koen Gijbels and Paul Stoppie

ter Haar Romeny, FEV

Oil red O stain

Oil red O stain of fat emboli in

lung

E > 0, E > 0, scale 1.5

ter Haar Romeny, FEV

PAS stain

Lww > 0, Lvv Lww-Lvw2 > 0, E -E > 0, scale sigma 2.0

P.A.S. stain for carbohydrates (goblet cells, gut)

carbohydrates stain magenta - elliptic patches

ter Haar Romeny, FEV

Blood smear

Blood smear, Giemsa stain, 100x, JPEG

compression

RBC: E > 0, E +E > 0, scale 0.5

Leucocytes: E < 0, scale 12

Leucocyte nuclei: E < 0, E > 0, scale

3

ter Haar Romeny, FEV

Blue-yellow edges

Note the complete absence of detection of black-white edges.

ter Haar Romeny, FEV

Color edges can also be defined as the zero-crossings of the second order derivative in the spatial gradient direction (color-invariant equivalent of Lww):

0ˆˆ

ˆˆˆˆˆ2ˆˆˆ

22

22

yx

xxxxyxyyyyww

EE

EEEEEEEE

Second order color invariants

ter Haar Romeny, FEV

Color invariant edge detection

Luminance gradientedge detection

ter Haar Romeny, FEV

ter Haar Romeny, FEV

Conclusions

• Color ‘scale-space’ compatible with classical luminance

scale-space

• The model enables the design of practical image analysis

‘color reasoning’ solutions, e.g. invariance for illumination

• The color-scale invariant differential operators are building

blocks for a differential geometry on color images