Embed Size (px)
Citation preview
Colouran algorithmic approach
Thomas [email protected]
http://www.eecs.qmul.ac.uk/~tb300/pub/PhD/ColourVision2.pptx
PhD Research Topic
understanding how natural visual systems process information
Visual system: • about 30% of cortex• most studied part of
brain• best understood part
of brain
Image sensors Binary sensor array
monochromatic ‘external retina’
Luminance sensor arraydichromatic colour
Multi-Spectral sensor arraytetrachromatic colour
What do these direct links to the brain do?
Lets hypothesise … When an astronomer looks at a star, how does he code the information his sensors produce?
It was noticed that parts of spectrum were missing.
Looking our own star – the sun
• x
Each atomic element absorbs at specific frequencies …
We can Code for these elements …
We can imagine how coding spectral element lines could be used for visual perception … by a creature very different to us… a creature which hunts by ‘tasting’ the light we reflect… seeing the stuff we are made of
Colour in this case means atomic structure and chemistry…
Where do we start with humans?
Any visual system starts with the sensor.
What kind of information do these sensors produce?How do we use that information to code what is relevant to us?
Let’s first look at sensors we ourselves have designed!
Sensors we buildX
Y
The Pixel
Sensors element may be:
Binary Luminance RGB
The fundamental unit of information!
The Bitmap
2-d space
represented by integer array
0 1 2
0
1
What information is produced?
2-d array of pixels:
Black & White Pixel:– single luminance
value, usually 8 bit Colour Pixel
– 3 colour values, usually 8-bit RGB
What does RGB mean?• It is an instruction for producing
light stimuli• Light stimuli for a human
standard observer• Light stimuli produce
perception• RGB codes the re-production of
measured perceptual stimuli• It is assumed that humans are
trichromatic• It tells us nothing about what
colour means!
The Standard Observer
CIE1931 xy chromaticity diagramprimaries at: 435.8nm, 546.1nm, 700nm
The XYZ sensor response
xx
x y z
now we extract the colour information from the sensor readings
The Math:
yy
x y z
… 2-d as z is redundant
Understanding CIE chromaticity
White in center
Saturated / monochromatic colours on the periphery
Best understood as a failed colour circle
Everything in between is a mix of white and the colour
xx
x y z
yy
x y z
Does it match?The problem of
‘negative primaries’
But does it blend?
Monochromatic Colours
What the Human Visual System (HVS) does is very different!
?
Human Visual
System (HVS)
Part 1
Coding Colour
The Sensor2 systems: day-sensor & night-sensor
To simplify: we ignore night sensor system
Cone Sensors very similar to RGB sensors we design for cameras
BUT: sensor array is not ordered
arrangement is random
note:very few blue sensors, none in the centre
sensor pre-processing circuitry
First Question: What information is sent from sensor array
to visual system?
Very clear division between sensor & pre-processing (Front of Brain) andvisual system (Back of Brain) connected with very limited communication link
Receptive Fields
All sensors in the retina are organized into receptive fields
Two types of receptive field.
Why?
What does a receptive field look like?
In the central fovea it is simply a pair of sensors.
Always 2 types:• plus-centre• minus-centre
What do retinal receptive fields do?
Produce an opponent value:simply the difference between 2 sensors
This means:
it is a relative measure, not an absolute measure
and
no difference = no information to brain
Sensor Input
Luminance Levels
it is usual to code 256 levels of luminance
Linear: Y
Logarithmic: Y’
Receptive Field Function
- - -- - -- - -
+ + ++ + ++ + +
- - -- - -- - -
- - -- - -- - -
- - -- - -- - -
- - -- - -- - -
- - -- - -- - -
- - -- - -- - -
- - -- - -- - -
+ + ++ + ++ + +
- - -- - -- - -
+ + ++ + ++ + +
+ + ++ + ++ + +
+ + ++ + ++ + +
+ + ++ + ++ + +
+ + ++ + ++ + +
+ + ++ + ++ + +
+ + ++ + ++ + +
Min Zone
Max-Min Function
Output is difference between average of center and max/min of surround Max
Zone
Tip of Triangle
Dual Response to gradients
Why?
Often described assecond derivative/zero crossing
AbstractedNeurons only produce positive values.Dual +/- produces positive & negative values.
Together: called a channel means signed values.
Produces directional information
+-
+-
+-
+-
+-+- +-
+-Location, angle luminance, equiluminance and colour
Information sent to higher visual processing areas
This is a sparse representation
From this the percept is created
This is a type of data compression.Only essential information is sent!
Conversion from this format to bitmap?
starting with the sensor:Human Sensor Response
to non-chromatic light stimuli
350 400 450 500 550 600 650 7000
10
20
30
40
50
60
70
80
90
100
Wavelength (nm)
Abso
rpti
on
(%
)
RGB
HVS Luminance Sensor IdealizedSe
nsor
Val
ue
Wavelength(λ)λ
0.8
0.6
0.2
0.0
1.0
0.4
λδλ−
A linear response in relation to wavelength.Under ideal conditions can be used to measure wavelength.
Spatially Opponent
HVS:Luminance is always measured by taking the difference between two sensor values.Produces: contrast value
Sens
or V
alue
Wavelength(λ)λ
0.8
0.6
0.2
0.0
1.0
0.4
λδλ−
Sens
or V
alue
Wavelength(λ)λ
0.8
0.6
0.2
0.0
1.0
0.4
λδλ−
Which is done twice, to get a signed contrast value
Moving from Luminance to Colour
• Primitive visual systems were in b&w• Night-vision remains b&w
• Evolutionary Path– Monochromacy– Dichromacy (most mammals – eg. the
dog)– Trichromacy (birds, apes, some monkeys)
• Vital for evolution: backwards compatibility
Electro-Magnetic Spectrum
Visible Spectrum
Visual system must represent light stimuli within this zone.
Colour Vision Young-Helmholtz
Theory
Argument:Sensors are RGB thereforeBrain is RGB
3 colour model
Hering colour opponency model
Fact: we never see reddish green or yellowish blue.
Therefore:colours must be arranged in opponent pairs:
RedGreenBlueYellow
4 colour model
Colour Sensorresponse to monochromatic light
350 400 450 500 550 600 650 7000
10
20
30
40
50
60
70
80
90
100
Wavelength (nm)
Abso
rpti
on
(%
)
RGB
370 nm 445 nm 508 nm 565 nm
700 nm330 nm 400 nm 500 nm 600 nm
1.0
0.5
0.0
Human
Bird
4 sensors
Equidistant on spectrum
How to calculate spectral frequency with 2 poor quality luminance sensors.
Roughly speaking:
Sensor Value
Wavelength
0.8
0.6
0.2
0.0
1.0
0.4
λ-Δ λ λ+Δ
RG
a shift of Δfrom a known reference point
the ideal light stimulusS
en
sor
Valu
e
Wavelength
0.8
0.6
0.2
0.0
1.0
0.4
λ-δ λ λ+δ
RGMonochromatic Light
Allows frequency to be measured in relation to reference.
Problem:natural light is not ideal
Sen
sor
Valu
e
Wavelength
0.8
0.6
0.2
0.0
1.0
0.4
λ-δ λ λ+δ
RG
• Light stimulus might not activate reference sensor fully.
• Light stimulus might not be fully monochromatic.
ie. there might be white mixed in
Sens
or V
alue
Wavelength(λ, in nm)
400300 430 460 490 520 550 580 610 640 670 700
0.8
0.6
0.2
0.0
1.0
0.4
Solution:
A 3rd sensor is used to measure equiluminance.
Which is subtracted.
Then reference sensor can be normalized
Equiluminance & Normalization
Also called Saturation and Lightness.
• Must be removed first – before opponent values calculated.
• Then opponent value = spectral frequency
• Values must be preserved – otherwise information is lost.
a 4 sensor designS
en
sor
Valu
e
Wavelength(λ, in nm)
400300 430 460 490 520 550 580 610 640 670 700
0.8
0.6
0.2
0.0
1.0
0.4
2 opponent pairs• only 1 of each pair can be active• min sensor is equiluminance
,R G B y
What is Colour?Colour is calculated exactly the same as luminance contrast. The only difference is spectral range of sensors is modified.Colour channels are: RGBy
Uncorrected colour values are contrast values.But with white subtracted and normalized:Colour is Wavelength!
How many sensors?
4 primary colours require 4 sensors!
Human Retina only has 3 sensors!What to do?
We add an emulation layer.
Hardware has 3 physical sensorsbut emulates 4 sensors
Wavelength (nm)
Sens
or R
espo
nse
460 580 640520 550 610490430 670400370 700
0.8
0.6
0.2
0.0
1.0
1.2
1.4
0.4
R G B
No maths … just a diagram!
Testing Colour Opponent model
What we should see
What we do see
Unfortunately it does not matchThere is Red in our Blue
350 400 450 500 550 600 650 7000
10
20
30
40
50
60
70
80
90
100
Wavelength (nm)
Abso
rpti
on
(%
)RGB
Pigment Absorption Data of human cone sensors
350 400 450 500 550 600 650 7000
10
20
30
40
50
60
70
80
90
100
Wavelength (nm)
Abso
rpti
on
(%
)
RGB
Red > Green
Solution:HVS colour representation must be circular!
Which is not a new idea, but not currently in fashion.
540nm
620nm
480nm
Dual Opponency with Circularity
an ideal model using 2 sensor pairs
Senso
r V
alu
e
Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700
0.8
0.6
0.2
0.0
1.0
0.4
… requires 2 independent channels which give 4 primary colours
Yellow added as a primary!
Which allows a simple transform to circular representation
180°0°
90°
45°
135°
315°
270°
225°
Opponent Values HueA simple transform from 2 opponent values to a single hue value
How might HVS do this?we keep 2 colour channels but link them
180°0°
90°
45°
135°
315°
270°
225°
0
+64
+127
R = 255 G = 0 B = 128
Y - +
Ψ B
CG R
180°0°
90°
45°
135°
315°
270°
225°
Travelling the Colour Wheel (Hue)
One Chroma channel is always at max or minThe other Chroma channel is incremented or decremented
Rules:
if (CB==Max)CR--if (CR==Max)CB++if (CR==Min)CB--if (CB==Min)CR++
180°0°
90°
45°
135°
315°
270°
225°
0
-127
+128
R = 255 G = 128 B = 0
Y - +
CBΨ B
CRG R
0
-127
0
R = 255 G = 255 B = 0
Y - +
CBΨ B
CRG R
+ -
Colour Wheel
Simple rule based system that cycles through the colour wheelAllows arithmetic operations on colour
180°0°
90°
45°
135°
315°
270°
225°
0
0
+127
R = 255 G = 0 B = 0
Y - +
CBΨ B
CRG R
0
+64
+127
R = 255 G = 0 B = 128
Y - +
CBΨ B
CRG R
0
+127
+127
R = 255 G = 0 B = 255
Y - +
CBΨ B
CRG R
0
+127
+64
R = 128 G = 0 B = 255
Y - +
CBΨ B
CRG R
0
+127
0
R = 0 G = 0 B = 255
Y - +
CBΨ B
CRG R
0
+127
-64
R = 0 G = 128 B = 255
Y - +
CBΨ B
CRG R
0
+127
-127
R = 0 G = 255 B = 255
Y - +
CBΨ B
CRG R
0
+127
-127
R = 0 G = 255 B = 0
Y - +
CBΨ B
CRG R
0
+127
-127
R = 128 G = 255 B = 0
Y - +
CBΨ B
CRG R
0
-127
0
R = 255 G = 255 B = 0
Y - +
CBΨ B
CRG R
0
-127
+128
R = 255 G = 128 B = 0
Y - +
CBΨ B
CRG R
0
-64
+127
R = 255 G = 128 B = 0
Y - +
CBΨ B
CRG R
0
0
+127
R = 255 G = 0 B = 0
Y - +
CBΨ B
CRG R
Part 2Accurate Colour reproduction
First problem:
Real world is not monochromatic
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
350
367
383
400
416
433
449
466
482
499
515
532
548
565
581
598
614
631
647
664
681
697
714
730
Spectrum of a common yellow flower
Accurate Colour reproduction
second problem:
human colour vision is inaccurate
prone to ‘making stuff up’
varies from person to person
The closer the sensors the less accurate the color information.
All humans are to an extent color blind … compared to animals like birds.
Examples of real world colour?
Colours are often computed, not measured!
… an extreme example
What is the colour?
Accurate colour reproduction … for dual channel opponency
Problem # 1
very easy to solve
we simply assume monochromacy
when stimuli are not monochromatic opponent channels simply subtract to 0
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
350
367
383
400
416
433
449
466
482
499
515
532
548
565
581
598
614
631
647
664
681
697
714
730
green, yellow and red are active
r-g = 0b = 0
leaving only yellow
stimuli equivalent to monochromatic
Accurate colour reproduction … with primaries
Only primaries are true coloursall other colours are intermediary
… and can be generated by proportions of primaries!
Accurate colour reproduction … for humans
Any colour may be displayed by a combination of 2 primaries
but the location of primaries can vary between individualsand intermediary locations can be distorted
Problem # 2
Solution toAccurate colour reproduction …
for the individual human
1. primaries must be mapped for the individual
2. mid-points must be mapped
467 517 573 644503 603
545
Provides an individual colour profile … a map of the primaries and intermediary points. Can be repeated recursively for greater precision.
Does it work?
• Colour opponency requires primaries to be precisely located.
• For humans this would be virtual primaries• Is there evidence that this is so?• Do the colours match?
This will be tested empirically …
• monochromator
• Xenon light sourceequal light across visible spectrum
Apparatus
Procedure
• generate subject selectable monochromatic stimuli
• subject selects colour• virtual primaries are
calculated
Preliminary results
Blue Green Yellow Red
455 520 580 650
465 520 580 640
465 520 575 640
470 525 585 655
460 520 575 640
465 520 575 650
465 525 575 635
475 515 570 640
470 522.5 570 635
482.5 525 572.5 635
462.5 532.5 580 640
471.25 526 578.3 647
467.2 522.6 576.3 642.3
7.19 4.5 4.5 6.7Average
Std Dev
Discussion
small sample with inaccurate tools
but• primaries appear to
be are very closely grouped and spaced equidistantly – exactly as predicted
In Future
• visits to optometrist will include a colour test
• Colour displays may be set by colour ‘prescription’
http://www.eecs.qmul.ac.uk/~tb300/pub/PhD/ColourVision2.pptx
References
Poynton, C. A. (1995). “Poynton’s Color FAQ”, electronic preprint.http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html
Bangert, Thomas (2008). “TriangleVision: A Toy Visual System”, ICANN 2008.
Goldsmith, Timothy H. (July 2006). “What birds see”. Scientific American: 69–75.
Neitz, Jay; Neitz, Maureen. (August 2008). “Colour Vision: The Wonder of Hue”. Current Biology 18(16): R700-r702.
Questions?
