8/17/2019 Introduction MM Processing S08
1/70
CS 4763 Fundamentals of Multimedia Systems
- Introduction to Image Processing
Qi Tian
Computer Science Department
University of Texas at San Antonio
[email protected]://www.cs.utsa.edu/~qitian/
8/17/2019 Introduction MM Processing S08
2/70
Image Processing
Manipulation of multidimensional signals− image (photo)
− video
− CT, MRI
− Fluid flow
),( y x f
),,( t y x f
),,,( t z y x f
),,,( t z y xv
8/17/2019 Introduction MM Processing S08
3/70
A Typical Image Processing System
object observe digitize store process Refresh
/storeRecord
output
DisplayImaging
systems
Sample and
quantize
Digital
storage
(disk)
Digital
computer
On-line
buffer
X-ary, radar imaging, infrared
imaging, ultrasound imaging,
medical imaging, geophysicalimaging
A/D
8/17/2019 Introduction MM Processing S08
4/70
Fundamentals of Image Processing
Representation – acquisition, digitization, and display to mathematical
characterization of images for subsequent processing
– a prerequisite for an efficient processing techniques such as
enhancement, filtering, and restoration.
Processing Techniques
– Image compression, image restoration, and image reconstruction
– Statistical image processing techniques
Communications
8/17/2019 Introduction MM Processing S08
5/70
Multimedia Processing Techniques
– Coding/compressionStorage and communications JPEG, JPEG2000
MPEG-1 (CD, mp3), MPEG-2 (HDTV, DVD)
H.261, H.263
– Enhancement, restoration, reconstruction feature extraction for image analysis and visual information display
removal of degradation in an image, LS, ML, Max entropy, MAP
2D -> 3D image MRI, CT, Radon transform
– Analysis, detection, recognition, understandingquantitative measurements from an image to produce a description on it
– Visualization
8/17/2019 Introduction MM Processing S08
6/70
Advanced Processing Techniques
Statistical processing techniques – Hidden Markov model (HMM)
– Probabilistic graphical models
– Bayesian networks
– Markov random field
Many applications to speech recognition, pattern classification, data
compression, and channel coding, etc.
8/17/2019 Introduction MM Processing S08
7/70
History of Image/Video Coding
1950
1960
1970
1980
1990
2000+
Math PR, CV, CG
Fractal 3-D Model based
coding
Signal ProcessingBased
PCM
DPCM
Transform Coding
VQ
Subband CodingWavelets
8/17/2019 Introduction MM Processing S08
8/70
Reference:
– F. Nebeker, Signal Processing: The Emergency of a Discipline,
1948-1998
– IEEE History Center, 1998
Broadband TV (NTSC)
500 × 500 × 8 × 3
× 30 bits/sec
≈
100 Mb/sec (compression is necessary!)Modem: 56Kb/sec
Picture Element
– Pixel West coast people in USC – Pel East people in MIT
8/17/2019 Introduction MM Processing S08
9/70
Image/Video Compression
Signal-Processing Based:Encoder
),( y x f
H),( y xg
Signal
Proc.
Representation ),( y xg
Decoder 1H−),(ˆ y x f ),( y xg
8/17/2019 Introduction MM Processing S08
10/70
Image/Video Compression
3D Model-Based:Encoder
Representation P
Decoder
),( y x f
H
Analysis Model
Parameter P
Model
),(ˆ y x f P 3D
Model
8/17/2019 Introduction MM Processing S08
11/70
Image/Video Compression
Fractal-Based:Encoder
Representation S
Decoder
),( y x f System S
),( y x f
Find S for which is an Attractor.),( y x f
SAny
signal),(ˆ y x f
Iteration
8/17/2019 Introduction MM Processing S08
12/70
Image/Video CompressionStandard
Facsimile: Fax Group 1, 2, 3, 4
JBIG (Joint Bi-level Image Expert Group)
Images: JPEG (http://www.jpeg.org/)
JPEG2000
Video: H.261, H.263 P × 64 Kb/s (P =1 ~ 30)MPEG 1 1.2 Mb/s Video, CD, MP3
MPEG 2 1.2 – 20 Mb/s, sports, HDTV, DVD
MPEG 4 1 kb/s → 1Mb/s, very low speed video
coding, MultimediaMPEG 7 Multimedia description, audio/video
MPEG 21 Multimedia framework
Based on Wavelet
Transform
8/17/2019 Introduction MM Processing S08
13/70
A de facto image for the past three decade for its rich texture
Lena
8/17/2019 Introduction MM Processing S08
14/70
8/17/2019 Introduction MM Processing S08
15/70
What are Challenging Problems inMultimedia Processing?
Multimedia Processing is taken in a broad sense,
including:
Image/Video compression, enhancement, restoration,
reconstruction, analysis, recognition, understanding,
visualization, and synthesis/animation.
8/17/2019 Introduction MM Processing S08
16/70
Examples
Face modeling, detection, and recognition
Emotion recognition
Gesture recognition
Gender/age/ethnicity recognition
Audio-visual speech recognition
Image/video superresolution
Image/video browsing, indexing, and retrieval
Biometrics
8/17/2019 Introduction MM Processing S08
17/70
Face Related Research
Face modeling Face detection
Face recognition
Facial expression recognition
8/17/2019 Introduction MM Processing S08
18/70
Generic Face Model
Texture
mapping
Face model morphing
8/17/2019 Introduction MM Processing S08
19/70
Generic Face Model
The generic face model is generated from a MRI data set
8/17/2019 Introduction MM Processing S08
20/70
Customize A Genetic Face on AnIndividual
Polygon Mesh: 2240 Vertices + 3946 Triangles.Polygon Mesh: 2240 Vertices + 3946 Triangles.
Non Non--Uniform Rational BUniform Rational B--Splines (NURBS): 63 control points.Splines (NURBS): 63 control points.
8/17/2019 Introduction MM Processing S08
21/70
The iFACE system in a distributed collaborative environment. (a)
Avatar in the head mounted display, (b) avatar in the desk screen of
MIC3E, (c) avatar in the main screen of MIC3E
Avatar – talking head
University of Illinois at Urbana-Champaign
8/17/2019 Introduction MM Processing S08
22/70
Text-Driven Face Animation
“We strive to make the meter onanimation production, and are
always looking for new technology
that will enable faster, more
appealing character creation,”
said Joel Kransove, Digital Director of
Nickelodeon. (Source: Digital
Producer)
8/17/2019 Introduction MM Processing S08
23/70
Speech-Driven Face Animation
“Game characters have becomesynthetic actors and dialogue is anessential element of the effect wecreate. The quality of the lip-
synching can make or break thesense of reality,”
said Scott Cronce, vice president and CTO at
Electronic Art (Source: Gamepro)
8/17/2019 Introduction MM Processing S08
24/70
Video-Driven Face Animation
8/17/2019 Introduction MM Processing S08
25/70
Emotion Recognition
8/17/2019 Introduction MM Processing S08
26/70
Emotion Recognition
8/17/2019 Introduction MM Processing S08
27/70
Emotion Recognition
8/17/2019 Introduction MM Processing S08
28/70
Face Detection Techniques
8/17/2019 Introduction MM Processing S08
29/70
Face Detection Techniques
8/17/2019 Introduction MM Processing S08
30/70
Face Recognition: Why it is easy?
8/17/2019 Introduction MM Processing S08
31/70
Face Recognition: Why it is hard?
8/17/2019 Introduction MM Processing S08
32/70
Beauty Check
What Are the Causes and Consequences ofHuman Facial Attractiveness?
Babyfaceness
Symmetry
Social perception
Universities of Regensburg, Germany
8/17/2019 Introduction MM Processing S08
33/70
Which is more attractive?
Universities of Regensburg, Germany
8/17/2019 Introduction MM Processing S08
34/70
Babyfaceness Large head
Large curved forehead
Facial elements (eyes,
nose, mouth) located
relatively low
Large, round eyes
Small, short nose
Round cheeks
Small chinKate Moss4-year old girl
Include mature female features: high, prominent cheek bones and
concave cheeks
8/17/2019 Introduction MM Processing S08
35/70
Which one is cuter?
8/17/2019 Introduction MM Processing S08
36/70
Miss Germany (2002)
8/17/2019 Introduction MM Processing S08
37/70
A selection of the 22 contestants of the final round of
the contest
8/17/2019 Introduction MM Processing S08
38/70
8/17/2019 Introduction MM Processing S08
39/70
Image Analysis
Texture synthesis and transfer
Image Super-resolution
Image Repairs
Illumination/Lighting changes and transfer
8/17/2019 Introduction MM Processing S08
40/70
Texture Synthesis and Transfer
+
SIGGRAPH’01 Effros & Freeman, MIT, 2001
synthesis
transfer
8/17/2019 Introduction MM Processing S08
41/70
Texture Synthesis and Transfer
8/17/2019 Introduction MM Processing S08
42/70
Image Superresolution
True Sub-sampled
Intelligent guess about details of texture
8/17/2019 Introduction MM Processing S08
43/70
Image Superresolution
Gaussian filter Bicubic interpolation
8/17/2019 Introduction MM Processing S08
44/70
Image Superresolution
Median filter Wiener filter
8/17/2019 Introduction MM Processing S08
45/70
Image Superresolution
Dynamic resolutionenhancement
Amos Storkey
True
8/17/2019 Introduction MM Processing S08
46/70
Image Repairs
8/17/2019 Introduction MM Processing S08
47/70
Image Repairs
Original Image
Result
Segmentation
Image synthesis
based on Tensor
Voting
Curve connection
8/17/2019 Introduction MM Processing S08
48/70
Image Repairs
8/17/2019 Introduction MM Processing S08
49/70
Illumination Effects on Images
8/17/2019 Introduction MM Processing S08
50/70
Relighting – Basic Algorithm
Step 2: Approximate radiance environment map
Step 3: Synthesize novel appearance by adjustingthe 9 spherical harmonic coefficients
Step 1: Align image with generic 3D face model
8/17/2019 Introduction MM Processing S08
51/70
Lighting Transfer
input target results
8/17/2019 Introduction MM Processing S08
52/70
Image/Video Retrieval
Image database
CBIR b d l
8/17/2019 Introduction MM Processing S08
53/70
CBIR based on color, texture,shape/structure
MARS: Multimedia Analysis and Retrieval System
metadata
User
Interface
Similarity
ranking
memory
Featureweighting
Visual
C++
Feature
Extraction
C/C++ Color
Texture
structure
8/17/2019 Introduction MM Processing S08
54/70
State-of-the-art CBIR Systems
QBIC (IBM), PhotoBook (Media Lab), Netra (UCSB),VisualSeek (Columbia), PicHunter (NEC-NJ), Amore (NEC-
CA), EI Niňo (Praja), MARS (UIUC), Virage (Virage Inc.),
CORE, PictoSeek, Piction, InfoScope …
Research Communities
Computer Vision, Image/Video Processing, Library and
Information Science, Database and Management Systems
Leading Journals & Standard
PAMI, ACM Multimedia, IJCV, CVIU
MPEG-7
8/17/2019 Introduction MM Processing S08
55/70
MARS using global features
8/17/2019 Introduction MM Processing S08
56/70
Biometrics
Security Threats:
We now live in a global society of increasing desperate and dangerous
people whom we can no longer trust based on identification documents
which may have been compromised.
A challenging Pattern Recognition Problem
Enabling technology to make our society safer,
reduce fraud and offer user convenience.
8/17/2019 Introduction MM Processing S08
57/70
Too many passwords to remember
8/17/2019 Introduction MM Processing S08
58/70
Identification Problems
Identity Theft: Identity
thieves steal PIN (e.g., dateof birth) to open credit card
account, withdraw money
from accounts and take out
loans
3.3 million identity thefts in
U.S. in 2002; 6.7 million
victims of credit card fraud
Surrogate representations of identity such as password
and ID cards no longer suffice
8/17/2019 Introduction MM Processing S08
59/70
Biometrics
Automatic recognition of people on their
distinctive anatomical (e.g., face, fingerprint, iris,
retina, hand geometry) and behavioral (e.g.,
signature, gait) characteristics.
Person identification is now an integral part of the
infrastructure needed for diverse business sectors
such as banking, border control, law
enforcement…
8/17/2019 Introduction MM Processing S08
60/70
Biometric Applications
8/17/2019 Introduction MM Processing S08
61/70
Biometric Applications
There are ~500 million border
crossing/year (each way) in the US
8/17/2019 Introduction MM Processing S08
62/70
Want to charge it?
8/17/2019 Introduction MM Processing S08
63/70
Biometric Characteristics
8/17/2019 Introduction MM Processing S08
64/70
Biometric Market Growth
International Biometric Group
8/17/2019 Introduction MM Processing S08
65/70
“State-of-the-art” Error Rate
False accept rate
(FAR):
Proportion of
imposters
accepted
False reject rate
(FRR):
Proportions of
genuine users
rejected
8/17/2019 Introduction MM Processing S08
66/70
Multibiometrics
8/17/2019 Introduction MM Processing S08
67/70
Soft Biometrics
8/17/2019 Introduction MM Processing S08
68/70
Privacy Concerns
8/17/2019 Introduction MM Processing S08
69/70
Tracking
8/17/2019 Introduction MM Processing S08
70/70
Recommended