Automatic Modeling of 3D Human Face

Marco Andolfi

I. Ragnemalm

Institute of Technology of Linkoping

M. Schaerf

“La Sapienza” University of Rome

Supervisors:

Co-Supervisor:

M. Fratarcangeli“La Sapienza” University of Rome

The problem: Face modeling

InputModeling System Output

Our goal

InputModeling System Output

1 - Five Orthogonal Photos

2 – A “Generic” Mesh

No particular instrument

3D colored model of the subject’s head in photo

(no hair)

3 – A Landmark set points

Our system:Input - Output - Features

The Input (1) - Five orthogonal Photos

Without expression

Without Gender Features

Without Race Features

The Input:(2) – A “generic” Mesh

(almost) MPEG-4 landmark points

The Input:(3) – The landmark set of

points

Modeling System

Morphing Sub-System

ColoringSub-System

Inside the system

Morphing sub system

Morphing Sub-System

Mesh Feature

Extraction

Photo Feature

Extraction

Morphing Execution

CameraError

Correction

2D – 3D Conversio

n

Morphing sub system

Preprocessing operation

Manually picking

Mesh feature extraction

Morphing Sub-System

Mesh Feature

Extraction

Photo Feature

Extraction

Morphing Execution

CameraError

Correction

2D – 3D Conversio

n

Morphing sub system

Noisy Data

Manually picking

Photo feature extraction

Morphing Sub-System

Mesh Feature

Extraction

Photo Feature

Extraction

Morphing Execution

CameraError

Correction

2D – 3D Conversio

n

Morphing sub system

Different distance Camera-subject

Scaling

Camera Errors: the corrected ones

Different distance Camera-subject

Camera orientation

Scaling

Translation

Camera Errors: the corrected ones

Different distance Camera-subject

Camera orientation

Wrong Head Position: Z axis

Scaling

Translation

Rotation

Camera Errors: the corrected ones

Camera Errors: the negligible ones

Rotation Angle?

Wrong Head Position: X axis

Camera Errors: the negligible ones

Rotation Angle?

Wrong Head Position: X axis

Rotation No info

Wrong Head Position: Y axis

Camera Errors: the negligible ones

Rotation Angle?

Wrong Head Position: X axis

Rotation No info

Wrong Head Position: Y axis

Negligible

Prospective error

Camera Errors: Summary

Rotation Angle?

Wrong Head Position: X axis

Rotation No info

Wrong Head Position: Y axis

Negligible

Prospective error

Different distance Camera-subject

Camera orientation

Wrong Head Position: Z axis

Scaling

Translation

Rotation

Corrected Ignored

Get reference point: Why?Rotation

Scaling

Get reference point: Why?Translation

Camera Errors: Get reference point

Camera Errors: Correction

Get scaling factor: example on Y

Fy Ly

FyScaling Factor = -----------

Ly

Noisy Data

Bad Result

Get scaling factor: example on Y

Fy1 Ly1

Fy1 + Fy2 + ... + FyNScaling Factor = ---------------------------------------

Ly1 + Ly2 + ... + LyN

Fy2 Ly2

Morphing Sub-System

Mesh Feature

Extraction

Photo Feature

Extraction

Morphing Execution

CameraError

Correction

2D – 3D Conversio

n

Morphing sub system

2D to 3D convertion

Several values for each coordinate

Weighted Average

More importance to front photo

Morphing Sub-System

Mesh Feature

Extraction

Photo Feature

Extraction

Morphing Execution

CameraError

Correction

2D – 3D Conversio

n

Morphing sub system

Single step morphing

RBF Morphing functionLow stiffness parameter(as interpolation)

Noisy Data

Bad Result

Interpolation

Single step morphing: why not...

High level quality points

Low level quality points

Two different level of quality

Double step morphingLow and High level quality

points

RBF Morphing functionHigh stiffness parameter(as approximation)

RBF Morphing functionLow stiffness parameter(as interpolation)

High level quality points

First step

Second step

c

Modeling System

Morphing Sub-System

ColoringSub-System

Inside the system

Coloring sub system

Coloring Sub-System

Texture coordinate computing

Bottom texture

generation

Weight computing

Texture mapping

Remove ghost effect

SolveOverlap artifact

Coloring sub-system

Texture coordinate computing

What about the camera correction error?

Not a simple orthogonal projection

Texture coordinate computing

Coloring Sub-System

Texture coordinate computing

Bottom texture

generation

Weight computing

Texture mapping

Remove ghost effect

SolveOcclusionProblem

Coloring sub-system

β

γ

α

NP

NF

NL

NT

Weight computingWhy? Avoiding overlapping

The weight as transparency level

Weight for vertex respect front photo

Proportional to α

WF = NF ∙ NP

Set NULL negative weights

NR

WR = NR ∙ NP < 0 → WR = 0

Coloring sub-system

Coloring Sub-System

Texture coordinate computing

Bottom texture

generation

Weight computing

Texture mapping

Remove ghost effect

SolveOverlapp artifact

Bottom texture generation

Little square

only skin

No mole, No scar

Repetitionof the little

square

Final bottom texture

Bad quality? (too many squares...)

AIM: obtain a texture containig only skin

Coloring Sub-System

Texture coordinate computing

Bottom texture

generation

Weight computing

Texture mapping

Remove ghost effect

SolveOverlapp artifact

Coloring sub-system

Texture MappingThe weight as transparency

level

Coloring Sub-System

Texture mapping

Remove ghost effect

SolveOverlapp artifact

Coloring sub-system

Texture coordinate computing

Bottom texture

generation

Weight computing

Ghost effect

Coloring Sub-System

Texture mapping

Remove ghost effect

SolveOverlapp artifact

Coloring sub-system

Texture coordinate computing

Bottom texture

generation

Weight computing

Solving occlusion problem

Solving occlusion problemDetect vertex between inner eye points

Giving weight to side photos

Removing weight to side photos

Let’s see some examples