Embed Size (px)
Citation preview
Coded Aperture and Coded Exposure PhotographyMartin Wilson
University of Cape TownCape Town, South Africa
Email: [email protected]
Fred NicollsUniversity of Cape TownCape Town, South Africa
Email: [email protected]
Abstract—This article presents an introduction to the fieldof coded photography, with specific attention given to codedaperture and coded exposure theory, methods, and applica-tions. A coded aperture is optimized for the task of defocusdeblurring, and constructed using a simple cardboard occludingmask. Furthermore, a series of coded aperture photographsare used to capture a 4D light field with a standard SLRcamera, and the captured light field is then used for depth-estimation and refocusing applications. Finally, a coded exposurepattern is optimized for motion deblurring, and coded exposurephotographs are captured by controlling the incident illuminationof the scene. The coded aperture and exposure methods areshown to be superior to traditional photographic methods interms of deblurring accuracy, and the captured light field is usedsuccessfully to produce a stereo depth estimate from a stationarycamera, and to produce post-exposure refocused photographs.
I. INTRODUCTION
Digital photography currently has applications in almostevery area of industrial and scientific research. However, thelimitations of traditional photographic techniques and equip-ment (e.g. defocus blur, motion blur, image noise, and finiteresolution) continue to restrict its usefulness and flexibility.Furthermore, traditional photography is not able to capture allof the available information within a scene’s visual appearance(e.g. scene depth, surface reflectance properties, and ray-levelstructure), and often this extra information would be veryvaluable to the application in question. Computational photog-raphy is a recently established area of research concerned withovercoming these disadvantages by utilizing computationaltechniques during image capture and post-processing.
Coded photography is a branch of computational photog-raphy that attempts to capture additional visual informationby reversibly encoding the incoming optical signal before itis captured by the camera sensor. The encoding process canoccur at multiple points in the photographic model, includingat the generation of the incident scene illumination and withinthe camera itself. Two popular in-camera methods includeusing specially engineered aperture shapes and high-frequencyexposure patterns.
When analyzing computational photography methods, it isoften useful to represent the incoming optical signal as a setof light rays rather than merely as a 2D array of intensitypixels. The set of incoming light rays can be defined as a 4Dfunction known as a light field, which specifies the intensity ofthe ray passing through each location on a 2D surface, at eachpossible 2D angular direction. Recently it has been shown that
a subset of the full light field can be practically captured bymaking relatively minor modifications to a standard camera,and that the captured light field can have a sufficient resolutionfor a variety of useful applications such as synthesizing virtualphotographs with arbitrary camera parameters.
In section II a selection of related work is presented,specifically in the fields of coded aperture, coded exposure,and light field photography. In section III the implementationdetails regarding our own experiments with coded aperture andcoded exposure photography are described. The experimentsperformed include reducing defocus and motion blur, as wellas capturing and applying a light field. The results of the ex-periments are presented in section IV, and finally conclusionsare drawn in section V.
II. RELATED WORK
Our work is inspired by related work in the fields of codedaperture photography, coded exposure photography, and lightfield acquisition. Recently a number of comprehensive surveyshave been published regarding these fields of research [1], [2],and in this section we briefly summarize a selection of relatedwork in order to provide a context for our own experiments.
A. Coded Photography
A popular reason for employing coded apertures withinan optical system is the fact that it allows the point-spread-function (PSF) to be engineered for specific purposes. Byreplacing the traditionally round aperture found in most currentcameras with a carefully designed coded aperture, the PSF canbe engineered to enhance the performance of depth estimationtechniques such as depth-from-defocus [3], [4], or it can beengineered to preserve frequency information in out-of-focusimages, thereby increasing the performance of deblurring tech-niques [3], [5]. While most coded apertures are implementedusing only occluding optics, some have been developed usingreflective elements that allow multiple apertures to be used ina single exposure [6]. The success of coded aperture methodsrelies on selecting the optimal coded aperture for a particularapplication, and therefore this non-trivial task has received asignificant amount of attention from the research community[5].
Coded apertures can also be used to capture images inhighly unconventional ways. For example, by using an apertureconsisting of multiple programmable attenuating layers it ispossible to capture an image without using a lens, and the
parameters of this lensless camera (e.g. focus, pan and tilt) canbe adjusted without any physical movement [7]. Alternatively,by taking a sequence of photographs, each with a differentcoded aperture, a 4D light field can be separated into 2D slicesand captured using an unmodified image sensor [8].
Coded exposure photography is conceptually very similar tocoded aperture photography, in that the traditional box-shapedexposure window is replaced with a coded pattern in orderto engineer the PSF of moving objects. Coded exposures canbe captured using a standard camera with an additional high-speed electronic shutter, and this has been shown to be usefulfor improving the performance of motion deblurring [9].
B. Light Field Capture and Applications
The 4D light field (also known as a lumigraph) is defined asthe radiance of every light ray passing through a 2D surface, atevery possible angle. An ideal light field cannot be physicallycaptured in its entirety, but was first proposed as a useful datarepresentation for image based rendering [10], [11]. However,due to the increasing availability of inexpensive, high qualitydigital cameras, a variety of methods for partially sampling alight field have been developed, and these partial light fieldshave since found applications in many image processing fields.
The first practical methods for capturing light fields usedeither an array of cameras [12], or a single camera on a gantry[11], to record multiple exposures of a scene from slightlydifferent locations. Despite their conceptual simplicity, cameraarrays are difficult to use in practice due to their large sizeand mechanical complexity. For this reason, building small,portable, single-camera devices for capturing light fields iscurrently a popular area of research. Capturing a light field ina single exposure requires placing additional optical elementsinto the camera, in order to prepare the 4D light field formeasurement on a 2D sensor. One method is to place a micro-lens array between the image sensor and the lens, therebymodulating the image formed on the sensor according to eachray’s angular direction[13]. Another alternative is to use ahigh-frequency attenuating mask, which creates spectral ‘tiles’of the 4D light field on the 2D image sensor, in a processsimilar to heterodyning in radio electronics [14].
Once a light field has been successfully captured, it can beused for a number of practical applications including glarereduction, depth estimation, and refocusing. Glare effects ina photograph are caused by a small subset of light rays, andtherefore if the light field can be captured, the offending rayscan be easily identified and ignored [15]. This is not possiblefor a conventional photograph due to the integrative processof traditional image capture. Light fields can also be used togenerate stereo views of a scene without requiring that thecamera be physically moved to a new position. This allowsstereo depth methods to be used in what is essentially amonocular system [8]. Lastly, virtually refocused photographscan be synthesized from a light field by placing a virtual imageplane into the model, and calculating the image formed usingray-tracing techniques [8], [13], [14].
(a) Coded Aperture Modification (b) Coded Exposure LED Array
Fig. 1. Photographs showing the two crucial elements of the prototype codedphotography camera.
III. IMPLEMENTATION
A. Coded Photography Prototype Camera
A prototype coded photography camera was constructedfrom a Canon 500D SLR camera and a Canon EF 50mm f/1.8II prime lens. The standard aperture and autofocus moduleswere removed from the lens, and replaced with a plastic bracethat allows aperture masks to be inserted from slits cut intothe lens’s external housing. Coded exposures were capturedusing coded illumination generated by a programmable LEDarray that was mounted around the lens. This method is farsimpler than replacing the camera’s shutter with a high-speedelectronic shutter, and produces good results provided thatthe ambient lighting in the scene can be minimized. Figure 1shows the modified aperture housing and programmable LEDarray.
B. Optimizing Apertures and Exposures for Deblurring
Both defocus blur and motion blur can be modelled asthe convolution between an ideal sharp image and a non-ideal PSF, or in the frequency domain, as the multiplicationof their frequency spectra [5]. Minima and zeros in thePSF’s spectrum cause information to be irreversibly lost inthe observed blurred images, thereby making the deblurringprocess ill-defined. Therefore, intuitively it can be proposedthat aperture shapes and exposure patterns that have PSFs withlarge minimum values in their frequency spectra will performwell in deblurring applications. Two specific performancemetrics were used in our experiments, Rraskar and Rzhou, whichwere proposed by Raskar et al. [9] and Zhou et al. [5] intheir respective papers. Rraskar is a formalized version of theintuitive proposal described above, and is defined as
Rraskar(fk) = α ·min(|Fk|) + β/variance(|Fk|), (1)
where fk is the aperture or exposure pattern under considera-tion, Fk is the Fourier transform of fk, and α and β are tunablescalars. The variance term is included to account for theinaccuracies that would result from errors in the estimation offk. Rzhou is a more complicated performance metric that alsotakes into account the level of noise present in the observedimage and statistics of natural images. It is defined as
Rzhou(fk) =∑ω
σ2
|Fk(ω)|2 + σ2
A(ω)
, (2)
(a) Conventional Circular Aperture
9.50mm
12m
m
(b) Optimized Coded Aperture
Fig. 2. The two aperture shapes used in the defocus deblurring experiments.
time
time
(b) Optimized Coded Exposure Pattern
(a) Traditional ‘Box’ Shaped Exposure
Fig. 3. The two exposure patterns used in the motion deblurring experiments.
where σ is the standard deviation of the assumed whiteGaussian noise and A(ω) is an approximation of the powerspectrum of natural images. For each frequency, ω, the metricindicates the degree to which noise at that frequency willbe amplified. To simplify the optimization problem, apertureshapes were restricted to an 11×11 binary grid, and exposurepatterns were restricted to a 52-bit binary string. The solutionspaces are too vast for an exhaustive search for an optimum tobe feasible, and therefore a genetic algorithm was used to findwell-performing local optima. Figure 2 shows the traditionalcircular aperture and the optimized coded aperture that wereused in the defocus deblurring experiments, while figure 3shows the traditional ‘box’ exposure and the optimized codedexposure that were used in the motion deblurring experiments.
C. Deblurring by Deconvolution
Since blur can be modelled as convolution, deblurring aphotograph requires deconvolving the observed image with theappropriate PSF. This requires that the PSF be estimated fromthe camera and scene parameters. For defocus deblurring, theshape of the PSF is equal to the shape of the aperture, and thescale is a function of depth, while for motion deblurring theshape of the PSF is determined by the velocity of the scene’srelative motion and the total exposure time. In our experimentsthe deconvolution implementation developed by Levin et al.[3] was used, which assumes a heavy-tailed, sparse derivativedistribution for natural images.
D. Light Field Capture
Figure 4 shows a popular light field representation thatdefines a ray by its intersection with two parallel planes,(u, v) and (s, t) [11]. If the (u, v) plane is defined at thephysical aperture plane, and the (s, t) plane is defined at thephysical image sensor plane, then the pixel values in a capturedphotograph represent the intensities of the rays intersectingwith the (s, t) plane, integrated over all the (u, v) locationsallowed by the particular aperture shape. Therefore, if raysare only allowed to pass through a single (u, v) location,
Aperture PlaneOriginalImage Plane
u
vs
ts’
t’
!1.0
VirtualImage Plane
Light Ray
Fig. 4. Digram showing the two-plane light field representation, as well asthe virtual image plane used to synthesize virtually refocussed photographs.
then the captured image represents a single 2D slice of the4D light field. If multiple slices are captured, each with adifferent constant (u, v) value, then the full 4D light field canbe reconstructed in software [8]. In our experiments the lightfield was captured from 81 separate exposures, each one takenwith a different block of a 9 × 9 binary coded aperture openat a time.
IV. RESULTS
A. Code Aperture Results
In this section our defocus deblurring and light field exper-iments are described, and their results are presented. Wherepossible, the performance of the coded apertures is comparedto that of a traditional circular aperture.
1) Defocus Deblurring Results: Figure 5 shows a sampleof the results obtained from defocus deblurring experiments,using both a planar resolution chart and a human face astarget scenes. The observed images were taken with the camera2.0m away from the scenes, and the lens was focused at1.0m. Since the scene objects lie far outside the focal plane,the observed images predictably show a significant level ofblurring. In the image of the resolution chart, the text iscompletely unrecognizable and only the coarsest lines aredistinguishable, while in the image of the human face, all thehard edges have been lost and only large facial features areidentifiable. It is also interesting to note that the blur causedby the conventional aperture is visually smoother than the blurcaused by the coded aperture, and this supports the claim thatthe optimized coded aperture is able to preserve more highfrequency information within the defocused areas.
The superiority of the coded aperture can clearly be seenwhen comparing the deblurred images in figure 5. The de-blurred conventional aperture images contain a significantamount of ringing, and they have failed to recover evenmoderately fine details. Only the coarsest lines in the reso-lution chart are distinguishable, and while the contrast at hardedges in the face has been improved, the edges are distortedand over-simplified. In contrast, the deblurred coded apertureimages contain far less ringing, and significant high frequencyinformation has been recovered. The medium-to-fine lines in
Obs
erve
dD
eblu
rred
(a) Circular Aperture (b) Refocus Optimized
Obs
erve
dD
eblu
rred
Fig. 5. A selection of results obtained from the defocus deblurringexperiments. The camera was focused at 1.0m and placed 2.0m away fromthe scene.
the resolution chart are now clearly distinguishable and thetext has been accurately reconstructed. In the deblurred faceimage, even the fine details such as the specular highlights inthe eyes and the texture of the facial hair have been recovered.
While results are only shown for a camera distance of 2.0m,experiments were performed for distances ranging from 1.0mto 2.0m in 10cm increments. At all of these distances theresults obtained with the coded aperture were superior to thoseobtained with the traditional aperture. However, the differencebetween the performances of the two apertures becomes lessnoticeable for distances near to the focal plane, and it isspeculated that this is due to the fact that at these distancesthe scale of the PSF becomes too small to properly define itscarefully engineered shape.
2) Light Field Results: Two visualizations of the light fieldcaptured in our experiments are shown in figure 6. Bothrepresent a subset of the full light field as a 2D array ofimages, one with uv-major indexing and the other with xy-major indexing. In the case of uv-major, each (u, v) coordinatedefines a single 2D image that represents a unique angularview of the scene. In the case of xy-major indexing, each (x, y)coordinate defines a 2D image that represents the angularintensity distribution of the light rays falling on a specific
sensor pixel.Figure 7 shows the results of an experiment in which we
attempt to calculate stereo disparity from a light field. The twoinput images were extracted from the captured light field bysetting (u, v) = (1, 5) and (u, v) = (9, 5) respectively. Theimages represent two horizontally spaced views of the testscene, and so could be directly input into a stereo disparityalgorithm without any further processing. Ignoring the noisyvalues in the background (which are due to the absence oftexture in the input images), the output disparity map hasaccurately determined the relative depths of the objects inthe test scene. While calculating stereo disparity from a pairof horizontally spaced input images is fairly commonplace,our result was obtained from single, stationary camera, and istherefore an extension of the standard method.
Results from a virtual image refocusing experiment areshown in figure 8. In (a) a photograph with the original unal-tered focal plane is shown. At the time of exposure the camerawas focussed at 1.0m and placed 1.0m away from the centreof the scene. Therefore the image of the metronome (which islocated at the centre of the scene) is in sharp focus, while theobjects in front and behind the focal plane have a significantamount of defocus blur. This is more clearly seen in sub-figure(b), which shows cropped and magnified images of the Rubik’scube, metronome, and mannequin. Refocused photographswere synthesized by placing a virtual image plane into thelight field model at various depths from the aperture plane(shown in figure 4). The original image-to-aperture distancewas normalized as α = 1.000, and refocussed photographswere produced for α values ranging from 0.980 to 1.020in 0.001 increments. Sub-figures (c) and (d) show croppedimages taken from the refocussed photographs obtained usingα = 0.995 and α = 1.005 respectively. In (c) the focalplane has been moved further away from the camera, therebybringing the mannequin into focus, while in (d) the focal planehas been brought closer to the camera, thereby bringing theRubik’s cube into focus.
B. Coded Exposure Results
This section describes a series of motion deblurringexperiments that were performed, and presents the resultsobtained. The performance of the coded exposure resultsare also compared to the results obtained with a traditionalbox-shaped exposure.
1) Motion Deblurring Results: Figure 9 shows the resultsof motion deblurring experiments involving moving sceneobjects. The observed photographs were captured using a 0.5sshutter time, and during exposure the scenes were manuallymoved vertically at an approximately constant speed (esti-mated to be ≈0.4m/s). The relative motion between the sceneand the camera has produced a significant amount of motionblur in the observed images, and almost all detail has been lostin the vertical direction. The horizontal lines and text in theresolution chart are completely unrecognizable, and only thebasic elongated shape of the face is discernible. The motion
4 5 6
4
5
6
u
v
x (1...1188)
y (1
...79
2)
(a)
500 505501 502 503 504
400
405
401
402
403
404
y
x
u (1...9)
v (1
...9)
(b)
Fig. 6. Diagram showing a subset of the captured light field as a 2D arrayof 2D images: (a) using uv-major indexing, and (b) using yx-major indexing.
0
2
4
6
8
10
12
14
16
18
20
22
(a) Input image pair extracted from light field
(b) Stereo disparity output
Fig. 7. Resulting stereo disparity map calculated from the stationary lightfield.
blur caused by the coded exposure pattern also seems tocontain more vertical structure than the smooth blur caused bythe traditional exposure, which suggests that it has preservedmore high-frequency information than the traditional exposure.
Deblurring the photographs captured using the traditionalexposure has strengthened some of the vertical contrast, butmost of the detail remains unrecovered and the backgroundnoise has been significantly amplified. The position of thetext has been recovered, but the characters themselves remainunrecognizable, and despite a slight improvement in the largefacial features (e.g. the forehead, chin, and eyebrows), theidentity of the face remains unrecognizable. In contrast, asubstantial amount of the original detail in the coded exposure
(a) Photograph with unaltered focal plane
(b) Cropped details: unaltered photograph (! = 1.000)
(c) Cropped details: refocused on mannequin (! = 0.995)
(d) Cropped details: refocused on cube (! = 1.005)
Fig. 8. Diagram showing the results of synthesizing virtually refocusedphotographs from the light field.
photographs has been salvaged by deblurring. In the resolutionchart, the text is now readable, and the medium-to-coarsehorizontal lines can even be distinguished. Also, almost allthe major facial features have been recovered, and the identityof the face has become clearly visible.
V. CONCLUSIONS AND FUTURE RESEARCH
A. Summary of Results
The results of the experiments clearly show that there aresignificant advantages to using coded apertures and exposuresfor applications such as defocus deblurring, motion deblurring,and light field capture. The coded photography techniquesthat have been covered require very simple and inexpensivehardware, and can be implemented easily by making smallmodifications to existing optical systems.
For defocus deblurring, coded apertures can be engineeredto preserve high frequency information in the blurred regions,thereby improving the results of the deconvolution operation.The deblurred photographs obtained using the optimized codedaperture contained far less ringing than when traditional circu-lar apertures are used, and the hard edges in the photographswere more accurately recovered.
Obs
erve
dD
eblu
rred
Obs
erve
dD
eblu
rred
(a) ConventionalExposure
(b) OptimizedCoded Exposure
Fig. 9. A selection of results obtained from the motion deblurring exper-iments. The scenes were moved vertically at a constant velocity during the0.5s exposure time.
Coded apertures were also used to capture a partial 4D lightfield of a 3D scene. While multiple exposures are required,this particular method can capture light fields with very finespatial resolution and flexible angular resolution. The lightfield captured in our experiments was shown to be of practicaluse for calculating depth, and synthesizing virtual photographswith adjusted focus settings.
Finally, coded exposures can be optimized to preserve highfrequency information in photographs with substantial motionblur. Our experiments with constant velocity motion showed
that coded exposure patterns produce far more accurate deblur-ring results than can be achieved with traditional exposures.
B. Recommendations for Future Research
Using an LCD-based aperture instead of the physical masksused in our experiments would allow for almost instantaneousaperture changes, which would reduce the time required tocapture a light field, and offer the ability to capture videowith a different aperture per frame. Also, using an LCDfilter to control exposure rather than controlling the incidentscene lighting would allow coded exposure photographs to becaptured outside of the laboratory environment.
Another avenue for future investigation is the use of non-binary coded apertures and exposures. Using gradient aper-tures could allow for a greater number of possible apertureshapes without increasing the diffraction effects associatedwith hard edges. Also, since most digital cameras containa Bayer-pattern colour mask, using apertures constructed outof RGB filters could allow each colour channel in a singleexposure to be captured using a different aperture shape.
ACKNOWLEDGMENT
The authors would like to thank the National ResearchFoundation, and Armscor’s PRISM program, managed by theCSIR, for their financial support.
REFERENCES
[1] G. Wetzstein, I. Ihrke, D. Lanman, and W. Heidrich, 2011. ComputationalPlenoptic Imaging. Eurographics State of the Art Report, 1-24, 2011.
[2] S.K. Nayar, 2011. Computational Camera: Approaches, Benefits andLimits. DTIC Technical Report Document, 2011.
[3] A. Levin, R. Fergus, F. Durand, and W.T. Freeman, 2007. Image anddepth from a conventional camera with a coded aperture. Proceedings ofACM SIGGRAPH 2007 26(3), July 2007.
[4] C. Zhou, S. Lin, and S. Nayar, 2011. Coded Aperture Pairs for Depthfrom Defocus and Defocus Deblurring. International Journal of ComputerVision 93(1):53-72, 2011.
[5] C. Zhou, and S. Nayar, 2009. What are Good Apertures for DefocusDeblurring? ICCP 2009 (oral).
[6] P. Green, W. Sun, W. Matusik, and F. Durand, 2007. Multi-aperturephotography. Proceedings of ACM SIGGRAPH 2007 26(3), July 2007.
[7] A. Zommet, and S. Nayar, 2007. Lensless Imaging with a ControllableAperture. IEEE Computer Society 339-346, 2006.
[8] C. Liang, T. Lin, B. Wong, C. Liu, and H. Chen, 2008. Programmableaperture photography: multiplexed light field acquisition. Proceedings ofACM SIGGRAPH 2008 27(3), August 2008.
[9] R. Raskar, A. Agrawal, and J. Tumblin, 2006. Coded exposure photog-raphy: motion deblurring using fluttered shutter. Proceedings of ACMSIGGRAPH 2006, 25(3), July 2006.
[10] S.J. Gortler, R. Grzeszczuk, R. Szeliski, and M. Cohen, 1996. TheLumigraph. Proceedings of ACM SIGGRAPH ’96, August 1996.
[11] M. Levoy, and P. Hanrahan, 1996. Light Field Rendering. Proceedingsof ACM SIGGRAPH ’96, August 1996.
[12] B. Wilburn et al, 2005. High Performance Imaging Using Large CameraArrays. Proceedings of ACM SIGGRAPH 2005, 24(3), July 2005.
[13] R. Ng, M. Levoy, M. Brdif, G. Duval, M. Horowitz, and P. Hanrahan,2005. Light Field Photography with a Hand-Held Plenoptic Camera.Stanford University Computer Science Tech Report CSTR, 2005-02.
[14] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin,2007. Dappled photography: Mask enhanced cameras for heterodynedlight fields and coded aperture refocusing. Proceedings of ACM SIG-GRAPH 2007, 26(3).
[15] R. Raskar, A. Agrawal, C. Wilson, and A. Veeraraghavan, 2008. Glareaware photography: 4D ray sampling for reducing glare effects of cameralenses. Proceedings of ACM SIGGRAPH 2008, 27(3), August 2008.
