Raskar Npar04final

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, Yu

Image Fusion for Context Enhancementand Video Surrealism

Adrian IlieAdrian Ilie

UNC Chapel HillUNC Chapel Hill

Ramesh RaskarRamesh Raskar

Mitsubishi Electric Mitsubishi Electric Research Labs,Research Labs,

(MERL)(MERL)

Jingyi YuJingyi Yu

MITMIT

Dark Bldgs

Reflections on bldgs

Unknown shapes

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuBackground is captured from day-time scene using the same fixed camera

Night Image

Day Image

Context Enhanced Image

Mask is automatically computed from scene contrast

But, Simple Pixel Blending Creates Ugly Artifacts

Mitsubishi Electric Research Laboratories Image Fusion for Context Enhancement Raskar, Ilie, YuPixel Blending

Our Method:Integration of

blended Gradients

OutlineOutline

• Context Enhancement

• Gradient-based Fusion

• Video Enhancement

• Surrealism

Nighttime imageNighttime image

Daytime imageDaytime image

Gradient fieldGradient field

Importance Importance image Wimage W

Mixed gradient fieldMixed gradient field

GG11 GG11

GG22 GG22

GG GGxx YY

Reconstruction from Gradient Reconstruction from Gradient FieldField

• Problem: minimize errorI’ – G|• Estimate I’ so that

G = I’

• Poisson equation

I’ = div G

• Full multigridsolver

I’I’

Why Gradient-based ApproachWhy Gradient-based Approach

• Comparison of intensity values are important• Maintain gradients to capture local variations• Directly solve for desired gradients

• Maintain subtle details• Mix dissimilar images• No need for precise segmentation

ComparisonComparison

• Average– Subtle details are lost

• Pixel-wise blending– Sharp transitions

IssuesIssues

• Boundary conditions• Color shifts

Boundary ConditionsBoundary Conditions

• Assumed Neumann condition at borders, I’ · N = 0, • Enforced by haloing image with blacks

Color ShiftColor Shift• Mixing dissimilar images

• Goal: final image appearance matches input images at corresponding pixels

Ifinal(x,y) = c1Ipoisson(x,y) + c2

• SolveWi(x,y) Ioriginal(x,y) = c1

Ipoisson(x,y) + c2

• Each color channel reconstructed separately

OutlineOutline

• Surrealism

Enhanced video Note: exit ramp, lane dividers, buildings not visible in

original night video, but clearly seen here.

Overview of ProcessOverview of ProcessDay time image: By averaging

5 seconds of day videoOriginal night time traffic camera

320x240 video

Mask frame (for frame above): Encodes pixel with intensity change

Output

AlgorithmAlgorithmFrame Frame NN

Daytime imageDaytime image

TimeAveraged TimeAveraged importance maskimportance mask

Final resultFinal result

Processed binary maskProcessed binary mask

Frame Frame N-1N-1

Mixed gradient fieldMixed gradient field

OutlineOutline• Context Enhancement

• Related Work

• Surrealism

Related WorkRelated Work

• Spatio-temporal Composition– Duchamp (Nude descending a staircase)– Freeman 2002– Fels 1999, Klein 2002, Cohen 2003

• Gradient-based Techniques– Multi-spectral: Socolinsky 1999– Shadow removal: Weiss 2001– High dynamic range: Fattal 2002– Image editing: Perez 2003– Some at Siggraph’04

Rene Magritte, ‘Empire of the Light’

Surrealism

OutlineOutline

• Surrealism

Time-lapse MosaicsTime-lapse Mosaics

Maggrite StripesMaggrite Stripes

timetime

Time Lapse MosaicTime Lapse Mosaic

Sunrise at NightSunrise at Night

BiSolar SystemBiSolar System

DiscussionDiscussion• User Experience

– More effective in conveying scene context– ‘Dreamy’ appearance– Nonrealistic : False conditions

• Applications– Tools for artists– Surveillance– Amusement park rides

• Performance– ~1 sec/frame for 320x240– ~ 3 min for 4Mpixel image

Image Fusion for Context EnhancementImage Fusion for Context Enhancement

• Nonrealistic but comprehensible context• Fusion using multiple images

– Enhancing night images with day bgrnd• Gradient-based fusion• Video surrealism tools

Raskar Npar04final

Technology

Raskar Ilp Oct08 Web

Raskar 2012, Idea Hexagon

TEDxBoston Raskar: Netra and Catra as Mobile Mates

Ramesh Raskar, CompPhoto Class Northeastern, Fall 2005 Northeastern University, Fall 2005 CSG242: Computational Photography Ramesh Raskar Mitsubishi Electric

Raskar Mar09 Nesosa

What is SIGGRAPH NEXT? Intro by Ramesh Raskar

Amartya sen interview translated by_Tushar Raskar

Raskar Computational Camera Fall 2009 Lecture 01

Svr Raskar

Raskar Sig05 Display Panel July05

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar MIT Media Lab Ramesh Raskar Associate Professor

Bimber & Raskar Siggraph 2005 Spatial Augmented Reality Spatial Augmented Reality Oliver Bimber University of Weimar Ramesh Raskar Mitsubishi Electric

Raskar Pocket Proj Fid2005presented

Raskar Keynote at Stereoscopic Display Jan 2011

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar Mitsubishi Electric Research

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab

Raskar Graphics Interface May05 Web

Raskar Thesis Guidance 2011

Ramesh Raskar - MIT Media Labweb.media.mit.edu/~raskar/Talks/07ACCV...•Mask == Light Field Modulator •Intensity of ray gets multiplied by Mask ... Digital Refocusing Heterodyne

Raskar 6Sight Keynote Talk Nov09