Dealing with the Complexities of Camera ISP Tuning
Clément Viard, Sr Director, R&DFrédéric Guichard, CTO, [email protected]
1© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Dealing with the Complexities of Camera ISP Tuning
2
> Basic camera image processing
> First revolution: optical and sensor defects corrections
> Second revolution: miniaturization
> ISP, A key differentiator in image and video processing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
3
Introduction
« Basic » digital image processing in a camera
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
RAW data from Sensor
4© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After « demosaicing »
5© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After exposure adaptation
6© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After « white ballance »
7© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After color rendering
8© DxO Labs 2016 | PREPARED FOR: AutoSens conference
After sharpening
9© DxO Labs 2016 | PREPARED FOR: AutoSens conference
10
A digital revolution in cameras
Optical and sensor defect corrections
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Geometric distortion
11© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Geometric distortion
12© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Volume anamorphos – perspective error with wide angle lenses
13© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Volume anamorphos – perspective error with wide angle lenses
14© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Optical vignetting
15© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Chromatic aberation
16© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Chromatic aberation
17© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping
18© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping
19© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping (with HDR sensor)
20© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Tone mapping (with HDR sensor)
21© DxO Labs 2016 | PREPARED FOR: RENAULT
Clipping – Recovering saturated area
22© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Clipping – Recovering saturated area
23© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Roll
Yaw
Pitch
X
Y
Z
Stabilisation and Electronic Rolling Shutter (ERS) effect
Translation Z
Yaw (Rotation)
Translation X
Pitch (Rotation)
Translation Y
ERS horizontal
ERS vertical
24© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Range of Digital Corrections with Advanced ISPs
25
> Optical aberrations> Distortion> Blur (spherical aberration, field
curvature, coma, astigmatism, motion)> Chromatic aberrations> Vignetting> Flare, veiling glare
> Light & Sensor> Noise> Dynamic range> Contrast> Atmospheric haze
> Sensor limitations> Field non uniformity (color, black
level,…)> Defective pixels, Dust> Clipping> Metamerism
> Others> Video stabilization> Electronic rolling shutter
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
26
Miniaturization – the second camera revolution
Will they eventually reach same performance?
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Sensor Miniaturization Challenge
27
22 times less light 4.5 stops
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Performance improvement over 10 years
28
> CMOS Sensor improvement : +1.5 stops
> Digital processing gain: +3 to +4 stops
DxOMark sensor score, APS-C cameras
+1.5 stops
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Nikon D70s, ISO 3200 – jpg
29© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v3
30© DxO Labs 2016 | PREPARED FOR: AutoSens conference
31
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v7
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
32
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v9
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
33
Nikon D70s, ISO 3200 – RAW + DxO Optics Pro v11
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
34
Nikon D70s, ISO 3200 – jpg
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Sensor color response changes with CRA
Photosite
Color response as a function of the angles350 450 550 650 750
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
36
ISP
A key differentiator in image and video processing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
« ISP » as a key differentiator
37
> Enables camera hardware and scene artefact corrections
> Only remaining differentiator since access to best sensor and lens is now ubiquitous
> Paradigm shift, cameras are designed considering possible digital corrections
> Computing power requirement consistently increasesDelivering 2,000 Ops/pixel with 240 Mpix/s (24 frames at 10Mpix per second) requires 500 GOps/s
> Very significant investment within the Mobile industry(e.g. iPhone camera engineering team ~ 800 people)
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Same HW, same ISP, same BOM same image quality?
DxO Mark Mobile score (Photos and Videos)38© DxO Labs 2016 | PREPARED FOR: AutoSens conference
« Camera Tuning »
39
For correcting each image, ISP requires 10,000+ register settings to be adapted to the situation.
> Per serie Calibration
> Per unit Calibration
> On the fly parameter estimations
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Image Quality Evaluation Challenge
> What is “Image Quality”?> Perception of how a picture “looks good”> In essence a subjective matter…that can be modeled with engineering tools
> What influences “Image Quality”> Shooting conditions: illumination, illuminant, dynamic> Content has a strong influence on Image Quality perception and image processing> Image Quality perception is different between imaging expert, professional
photographers and consumer
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Image Quality Evaluation Challenge - Subjective vs. objective
> Objective evaluation> Objective means a measurement that is neutral, operator independent: “2
cm, 15 meters, 150 kg, 2 g, -5°C, 50°C, …”> A device must provide figures (metrics) that are related to image quality> Normalizations may be necessary to have comparable metrics (cameras
with different resolutions)> Only addresses a set of metrics (some artifacts may be ignored)
> Subjective evaluation> “Subjective” means real people give an opinion like: “big, small, heavy, light,
cold, hot, …”> People can judge the quality of photographs> Methodology is key to get non-biased results
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Camera IQ assessments
42
Temporal behaviorDynamic behavior Photogrammetry and 3D
Thousands of videos/photos are required to characterize IQ along differentdimensions
Specifying AND verifying image quality targets are tough challenges
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Just Noticeable Difference (JND), a Matter of Statistics
> The smallest statistically measurable difference of perception, e.g., smallest perceivable distance between 2 parallel lines
> Typically, defined when half of the people perceive a difference and the other half are guessing (50% JND)
50% perceive
a change 50% guessing
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Anchored Scaling Concept
C D E F
Test image
??
Anchors
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Objective/subjective correlation example: Sharpness JND mapping
After extensive visual experiments, one could show that the Acutance objective metric linearly correlates with perception of sharpness (expressed with JND unit)
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
5
10
15
20
25
30
35
Acutance
JN
D
0
20
40
60
80
100
120
0.0 0.1 0.2 0.3 0.4 0.5
MTF
(%
)
Frequency (cycle/pixel)
CSF
MTF
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Back to the real world!
> Unfortunately the assumptions to build a complete multi-variate analysis based scoring system are not met > Only few metrics are linearly correlated to perception> They are not strictly orthogonal to each others> New issues comes with every technological improvement
> Since we can’t build a unified scoring system based on a set of perceptual metrics, a Multi-Modal approach is required
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Recommendation for building a relevant Image or video score
47
> Multi-modal testing approach> Objective measurements in lab environment> Perceptually-correlated metrics when available> Perceptual analysis from natural scenes
> Multi-level testing results> Overall, Photo, and Video scores> Top-level scores (open scale) non technical> Detailed reports for engineering
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Example of natural scene set used for mobile camera application
48© DxO Labs 2016 | PREPARED FOR: AutoSens conference
Conclusion
49
> Eventually, all industries would benefit from Smartphone technology breakthroughs
> ISP can fix many optical and sensor defects
> ISP technology improvements are a key enabler for miniaturization
> Mastering tuning complexity and IQ evaluation is a key differentiator
> DxO has developed robust methods to deal with this problem
© DxO Labs 2016 | PREPARED FOR: AutoSens conference
1 / DxO USA 1407 Minnesota Street San Francisco, CA [email protected]: +1 415 805 8567
2 / DxO France3, rue Nationale 92100 Boulogne-Billancourt France [email protected] Tel: +33 1 55 20 55 99
3 / DxO Korea16th Floor, Gangnam Building 1321-1 Seocho-dong, Seocho gu Seoul [email protected]: +82 2 2190 3825
4 / DxO TaiwanTai Centre 2/F-A170 Dunhua North Road Taipei 10548Taiwan Tel: +886 2 5574 0429
5 / DxO Japan3F Metlife Kabutocho Bldg 5-1 Kabutocho Chuo-ku Nihonbashi, [email protected]: +81 (0)3 6861 7565
Thank you!
50
© DxO Labs 2016 | PREPARED FOR: AutoSens conference