Computer Graphics Lecture 3staff.fmi.uvt.ro/~marc.frincu/cg/c3.pdf · West University of Timisoara Feb 13rd 2012 1 / 23. What is an Image? Image ProcessingAssignmentRecap ... Digital

What is an Image? Image Processing Assignment Recap

Computer GraphicsLecture 3

Dr. Marc Eduard Fr̂ıncuWest University of Timisoara

Feb 13rd 2012

1 / 23


Outline

1 What is an Image?Image Acquisition & DisplayHalftoning & DitheringSampling TheoryAliasing

2 Image ProcessingPixel & Convolution based operationsWarpingTransparency & Blending

3 Assignment

4 Recap

2 / 23


What is an Image

– a 2D array of pixels

More on images

The basic 2D image is a monochrome (greyscale, i.e., R = B = A for all pixels)digitized image

an image is a 2D light intensity continuous function f (x , y) : R2 → R where:(x, y) are spatial coordinatesf (x, y) is proportional to the brightness (i.e., grey value) of the (x, y) point

a digitized image:an image in which spatial and greyscale values are discreteintensity is measured across a regularly spaced (x, y) grid (grid elements = pixels =samples NOT small squares)intensities sampled on 8bits

Figure:http://www.cs.princeton.edu/courses/archive/spr03/cs426/lectures/02-sample.pdf

3 / 23

http://www.cs.princeton.edu/courses/archive/spr03/cs426/lectures/02-sample.pdf


Image Acquisition & Display

Digital Image Acquisition & Display

Acquisition

Through digital (e.g., CCD/CMOS camera) or analog (e.g., photographic film,printed paper) mediumsCCD (Charge-Coupled Device) is formed by a photoactive region (epitaxial layer ofsilicon) ⊕ shit register (the CDD)

1 image is projected through a lens on the capacitor array (1D for scanners, 2D forvideo & still cameras)

2 each capacitor accumulates electric charge proportional to the light intensity

3 after Step 2 is completed the control circuit transfers the charge of eachcapacitor to its neighbor (i.e., shift register)

4 by repeating this process the control circuit converts the entire contents of thearray to a sequence of voltages

5 the sequence is then sampled, digitized and stored in memory

In an analog device the sequence (Step 4) is processed into a continuous analogsignal (through a low pass filter) → transmitted, recorded, . . .Analog video signals (frames) are digitized through a frame grabber

4 / 23




Acquisition








4 / 23




Acquisition







In an analog device the sequence (Step 4) is processed into a continuous analogsignal (through a low pass filter) → transmitted, recorded, . . .

Analog video signals (frames) are digitized through a frame grabber

4 / 23




Acquisition








4 / 23




RAW & SNR

1 a RAW digital image file is the equivalent of a digital negative (encoded usingDCRAW for instance) ⇒ development process using Bayer matrices or Foveonchips.

2 increase Signal to Noise Ratio (SNR): stack several images

Digital color reconstruction using a Bayer matrix

Bayer matrix: pattern of RGBG (or CYMK) filters in front of each pixelRGBG filter: 1/4 pixels capture red, 1/4 blue and 2/4 greentrue color? reconstruct missing primary colors from neighboring pixelsmethods (source: http://deepskystacker.free.fr/english/index.html):

interpolation: interpolate missing primary colors from neighboring pixelssuperpixel: create super pixel from each group of RGBG pixelsBayer drizzle: compute bit by bit the true values of each primary for every pixel (goodwhen large number of images exist)

Figure:Interpolation Figure: Superpixel

Figure: Bayerdrizzle

5 / 23

http://deepskystacker.free.fr/english/index.html




RAW & SNR

1 a RAW digital image file is the equivalent of a digital negative (encoded usingDCRAW for instance) ⇒ development process using Bayer matrices or Foveonchips.

2 increase Signal to Noise Ratio (SNR): stack several images

Digital color reconstruction using a Bayer matrix

Bayer matrix: pattern of RGBG (or CYMK) filters in front of each pixelRGBG filter: 1/4 pixels capture red, 1/4 blue and 2/4 greentrue color? reconstruct missing primary colors from neighboring pixelsmethods (source: http://deepskystacker.free.fr/english/index.html):

interpolation: interpolate missing primary colors from neighboring pixelssuperpixel: create super pixel from each group of RGBG pixelsBayer drizzle: compute bit by bit the true values of each primary for every pixel (goodwhen large number of images exist)

Figure:Interpolation Figure: Superpixel

Figure: Bayerdrizzle

5 / 23

http://deepskystacker.free.fr/english/index.html




Display

– recreation of the continuous function from samples

Figure: CRT display, http://electronics.howstuffworks.com/tv3.htm

Figure: LCD display, http://www.which.co.uk/technology/tv-and-dvd/guides/which-television/lcd-tv/

6 / 23

http://electronics.howstuffworks.com/tv3.htm

http://electronics.howstuffworks.com/tv3.htm

http://www.which.co.uk/technology/tv-and-dvd/guides/which-television/lcd-tv/





Image Display Resolution

spacial resolution: size of the pixel array

possible problem: not enough spatial resolution

solution: sampling & reconstruction

intensity resolution: “depth” bits fro color/intensity

possible problem: not enough color/intensity depth (quantization)

solution: halftoning & dithering

temporal resolution: refresh rate

possible problem: refresh rate not high enough

Error computation: E 2 =∑

(x,y)(InitialPixel(x , y)− ProcessedPixel(x , y))2

7 / 23














7 / 23














7 / 23














7 / 23


Halftoning & Dithering


Quantization

– artifacts due to limited intensity resolution

uniform quantization, i.e., decreasing no. of bits per pixel:ProcessedPixel(x , y) = trunc(InitialPixel(x , y) + 0.5)

ProcessedPixel(x,y) is the value of the pixel color on the reduced bits

Reducing the quantization effects

classical halftoning (use dots to represent intensities):use clusters of pixels to represent intensity → trade spatial resolution for intensityresolutioncluster size proportional with value of InitialPixel(x, y)used in newspapers

dithering (distribute errors among pixels):randomorderederror diffusion

8 / 23




Dithering

random: ProcessedPixel(x , y) = trunc(InitialPixel(x , y) + noise(x , y) + 0.5)

1: r() = new array of random numbers ∈ [0, 255] with size x × y2: for each InitialPixel(x , y) do3: if InitialPixel(x , y) > r(i + +) then4: ProcessedPixel(x , y) = 2555: else6: ProcessedPixel(x , y) = 07: end if8: end for

average similar with the random but the random values are replaced by a singlechosen threshold

9 / 23




Dithering

random: ProcessedPixel(x , y) = trunc(InitialPixel(x , y) + noise(x , y) + 0.5)

1: r() = new array of random numbers ∈ [0, 255] with size x × y2: for each InitialPixel(x , y) do3: if InitialPixel(x , y) > r(i + +) then4: ProcessedPixel(x , y) = 2555: else6: ProcessedPixel(x , y) = 07: end if8: end for

average similar with the random but the random values are replaced by a singlechosen threshold

9 / 23




Dithering (cont.)

ordered (in use by most newspapers): uses the Bayer matrix (B) (used for RAWfiles in CCD/CMOS cameras)

size of B depends on the number of colors usedpixel values need to be scaled accordingly to the values of B

Example for B2:

B2 =

(1 34 2

)Algorithm:

1: Bn the Bayer matrix2: e = InitialPixel(x , y) scaled to the values in Bn

3: if e > Bn(x mod n, y mod n) then4: ProcessedPixel(x , y) = ceil(InitialPixel(x , y))5: else6: ProcessedPixel(x , y) = floor(InitialPixel(x , y))7: end if

10 / 23




Dithering (cont.)

Floyd-Steinberg best results of the classical methods BUT slowestpublished in ’76based on error dispersionused when converting to GIF format

Algorithm:

1: for y = 1,m do2: for x = 1, n do3: ProcessedPixel(x , y) = findClosestPalleteColor(InitialPixel(x , y))4: error = InitialPixel(x , y)− ProcessedPixel(x , y)5: InitialPixel(x + 1, y) = InitialPixel(x + 1, y) + 7/16× error6: InitialPixel(x − 1, y + 1) = InitialPixel(x − 1, y + 1) + 3/16× error7: InitialPixel(x , y + 1) = InitialPixel(x , y + 1) + 5/16× error8: InitialPixel(x + 1, y + 1) = InitialPixel(x + 1, y + 1) + 1/16× error9: end for

10: end for

11 / 23




Figure: Dithering methods – comparison,http://www.cs.princeton.edu/courses/archive/spr03/cs426/lectures/02-sample.pdf

12 / 23

http://www.cs.princeton.edu/courses/archive/spr03/cs426/lectures/02-sample.pdf


Sampling Theory

Sampling Theory

how many samples do we need in order to not loose information?

what signals can be reconstructed from a sampling?

a signal can be reconstructed from its samples if the original signal has nofrequencies above 1/2 the sampling frequency (Shannon)

the minimum sampling rate (to avoid aliasing) for a bandlimited (highestfrequency, i.e., bandwidth, is bounded) function is called Nyquist rate = twicethe bandwidth of the signal

Any signal can be written as a sum of periodic functions, e.g.,x(t) = Asin(ωt + θ), where θ : phase, ω = 2πfrequency : angular frequency, and A :amplitudeFourier transform: expresses a mathematical function of time as a function offrequency :

F (frequency) =∫∞−∞ f (t)e−2π×t×frequency dx

13 / 23


Sampling Theory

Sampling Theory







13 / 23


Sampling Theory

Sampling Theory







13 / 23


Aliasing

Aliasing

Aliasing

artifacts due to under-sampling/poor reconstruction

spatial (e.g., rough edges on oblique lines) & temporal (e.g., strobing, flickering)

Solution: antialiasing

sample at higher rate

issues: not always possible & doesnot always solve the problemworks for all primitives (less wellfor points & lines)

pre-filter to form bandwidthlimited signal (low-pass filtering):works well for points & lines

issues: trades aliasing for blurring

Figure: Aliasing vs. antialiasing effect,http://www.wikipedia.org

14 / 23

http://www.wikipedia.org


Aliasing

Aliasing

Aliasing

artifacts due to under-sampling/poor reconstruction

spatial (e.g., rough edges on oblique lines) & temporal (e.g., strobing, flickering)

Solution: antialiasing

sample at higher rateissues: not always possible & doesnot always solve the problemworks for all primitives (less wellfor points & lines)

pre-filter to form bandwidthlimited signal (low-pass filtering):works well for points & linesissues: trades aliasing for blurring

Figure: Aliasing vs. antialiasing effect,http://www.wikipedia.org

14 / 23



Aliasing

Aliasing

Supersampling

1 over-sample, e.g., 4x the pixelrate (increase resolution 4x)

2 reconstruct at the over-samplerate

3 band-limit to match pixel rate(compute average neighborvalues)

4 resample at the pixel rate ⇒ pixelvalues (decrease resolution 4x)

5 reconstruct to display

Figure: Supersampling,http://www.wikipedia.org

Sampling patterns

regular grid: divide pixel into sub-pixel array and samplesparse regular grid, e.g., NVIDIA Quincunx (center pixel + one corner)pseudo-random (Poisson disc)filter-extend (reconstruction⊕bandlimited)pattern variation: temporal (frame to frame) or spatially (pixel to pixel)shading rate vs. geometric sampling rate

Example: the accumulation buffer in OpenGL

15 / 23



Aliasing

Aliasing

Supersampling







Sampling patterns



15 / 23



Aliasing

Aliasing

Supersampling







Sampling patterns



15 / 23



Aliasing

Aliasing

Multisampling

– optimized form of supersampling that works at shading rate

eliminates jaggies due to edge aliasingefficiency and performance:

shades each fragment once (no geometric edges in a fragment)stores the color of each sample (there may be edges within a pixel)

point & area sampling

Figure: Generate fragment if unit square intersects triangle (1 gray square = 1 pixel

Example: Multisampling is trivial in OpenGL: glEnable(GL MULTISAMPLE)

16 / 23


Aliasing

Aliasing

Multisampling

– optimized form of supersampling that works at shading rate

eliminates jaggies due to edge aliasingefficiency and performance:

shades each fragment once (no geometric edges in a fragment)stores the color of each sample (there may be edges within a pixel)

point & area sampling

Figure: Generate fragment if unit square intersects triangle (1 gray square = 1 pixel

Example: Multisampling is trivial in OpenGL: glEnable(GL MULTISAMPLE) 16 / 23


Image Processing steps

1 sample the real world

2 reconstruct using discrete samples (i.e., pixels)

3 transform the reconstructed function via Fourier

4 filter (convolution) the transformed function ⇒ bandlimited function

5 sample the bandlimited function

6 reconstruct image from samples ⇒ display image

low-pass filters , i.e., Triangle & Gaussian filters – passes low frequency signals butattenuates signals with frequencies higher than a threshold

17 / 23


Image Processing steps

1 sample the real world

2 reconstruct using discrete samples (i.e., pixels)

3 transform the reconstructed function via Fourier

4 filter (convolution) the transformed function ⇒ bandlimited function

5 sample the bandlimited function

6 reconstruct image from samples ⇒ display image

low-pass filters , i.e., Triangle & Gaussian filters – passes low frequency signals butattenuates signals with frequencies higher than a threshold

17 / 23


Pixel & Convolution based operations

Basic IP

Pixel operations

brightness: ProcessedPixel(x , y) = InitialPixel(x , y)× k, where k ∈ [0, 255]

contrast: ProcessedPixel(x , y) = InitialPixel(x , y)× luminance, whereluminance = α× R + β × G + δ × B, α+ β + δ = 1

. . .

Convolution (kernel) operations

– each pixel becomes a weighted average of the sum of its neighbors

blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0

sharp: add the source image to the edge detection kernel

0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23



Basic IP

Pixel operations



. . .



blur:

1/9 1/9 1/91/9 1/9 1/91/9 1/9 1/9

edge detection:

0 −1 0−1 4 −10 −1 0


0 −1 0−1 5 −10 −1 0

18 / 23


Warping

Basic IP

Warping

– move the pixels in an image (mapping ⊕ resampling)

Notations:

(u,v) pixel in the source image

(x,y) pixel in the processed image

Operations:

scale: (x , y) = (factor × u, factor × v), where factor ∈ R+

rotate: (x , y) = (u × cos(θ)− v × sin(θ), u × sin(θ) + v × cos(θ)), whereθ ∈ [0, 360]

shear: (x , y)x = (u + factorx × v , v) or (x , y)y = (u, v + factory × u), wherefactor ∈ R+

warp: pick mapping functions fforward: ProcessedPixel(fx (u, v), fy (u, v)) = InitialPixel(u, v)

reverse: ProcessedPixel(x, y) = InitialPixel(f −1x (x, y), f −1

y (x, y))

pros & cons for each warp method?

19 / 23


Warping

Basic IP

Warping


Notations:



Operations:






y (x, y))


19 / 23


Warping

Basic IP

Warping


Notations:



Operations:






y (x, y))


19 / 23


Warping

Basic IP

Warping


Notations:



Operations:






y (x, y))


19 / 23


Warping

Basic IP

Warping


Notations:



Operations:






y (x, y))


19 / 23


Warping

Basic IP

Warping


Notations:



Operations:






y (x, y))


19 / 23


Warping

Basic IP

Warping


Notations:



Operations:




warp: pick mapping functions fforward: ProcessedPixel(fx (u, v), fy (u, v)) = InitialPixel(u, v) Many src pixels CANmap to same dst pixel & some dst pixels could not be coveredreverse: ProcessedPixel(x, y) = InitialPixel(f −1

x (x, y), f −1y (x, y)) (u, v) 6∈ N


19 / 23


Warping

Basic IP

Reverse warping

(u, v) ∈ R! Pixels must ∈ N ⇒ resample: (x , y) = resample src(u, v ,w)

point sampling: (x , y) = (trunc(u + 0.5), trunc(v + 0.5)) ⇒ aliasing

triangle filter: linear interpolation of the four closest pixels

Gaussian filter: compute weighted sum of pixel neighborhood (i.e., Gaussiankernel)

Example of reverse mapping functions – (u, v) need resampling

scale: (u, v) = (x/sx , y/sy )

rotate: (u, v) = (x × cos(θ)− y × sin(θ), x × sin(θ) + y × cos(θ))

20 / 23


Warping

Basic IP

Reverse warping






scale:

(u, v) = (x/sx , y/sy )

rotate:

(u, v) = (x × cos(θ)− y × sin(θ), x × sin(θ) + y × cos(θ))

20 / 23


Warping

Basic IP

Reverse warping






scale: (u, v) = (x/sx , y/sy )

rotate: (u, v) = (x × cos(θ)− y × sin(θ), x × sin(θ) + y × cos(θ))

20 / 23


Transparency & Blending


Transparency

pixels can also include the alpha channel α (0 – transparent; 255 – opaque)

(R, G, B, α) → pixel is colored in (R/α, G/α, B/α), i.e., components arepremultiplied by α

Blending

given images A, B & C, overlapped:1 αA blocks B by A2 1− αA shows B through A3 (1− αA)(1− αB ) shows C through A & B

given A & B the resulted colors for image C are:C = αAA + (1− αA)αB BαC = αA + (1− αA)αB

21 / 23




Transparency

pixels can also include the alpha channel α (0 – transparent; 255 – opaque)

(R, G, B, α) → pixel is colored in (R/α, G/α, B/α), i.e., components arepremultiplied by α

Blending

given images A, B & C, overlapped:1 αA blocks B by A2 1− αA shows B through A3 (1− αA)(1− αB ) shows C through A & B

given A & B the resulted colors for image C are:C = αAA + (1− αA)αB BαC = αA + (1− αA)αB

21 / 23


Assignment

ASSIGNMENT 1:

Implement the following operations on an image:

the Floyd-Steinberg dithering algorithm

the antialiasing blurring algorithm on a 45 degree oblique black line on a whitebackground

one IP from each category: pixel, convolution & warp

a basic blend between two images

Hint: use Java2D ONLY for loading and manipulating pixels.DO NOT use the Java2D’s functionality for handling IP

Deadline: Beginning of Lecture 5

22 / 23


Lecture Recap

an image is a 2D array of pixels

an image has size & depth

images are obtained by sampling the information from real scenes captured bydigital or analog devices

images suffer from halftoning & aliasing

images can be manipulated: pixel based, using convolution, by warping, mixed

23 / 23

Documents

Computer Graphics Lecture 3staff.fmi.uvt.ro/~marc.frincu/cg/c3.pdf · West University of Timisoara Feb 13rd 2012 1 / 23. What is an Image? Image ProcessingAssignmentRecap ... Digital