Rendering High Dynamic Range Images on the Web · Compute the logarithm of the HDR im-age and normalize it to the interval [0 1], thereby compressing the dynamic range of HDR data

� MATLAB Digest www.mathworks.com

Products Used ■ MATLAB®

■ Image Processing Toolbox™

■ MATLAB Builder™ JA

High dynamic range (HDR) digital imag-ing systems increase visual fidelity by inte-grating contrast ratio with observer-based models of color perception across multiple exposures of a single scene. They use math-ematical transformations known as tone-mapping operators to display HDR images on low dynamic range (LDR) devices such as computer monitors.

In this article, we create HDR Explorer, a Web-based experimental test platform that can be used to quickly evaluate how different HDR algorithms render LDR images. We describe how to translate the mathematical expression of an algorithm into a MATLAB® program and then show how to turn that program into a Web application.

HDR Image Processing in MATLABThe goal of HDR image processing is to use a series of photographs of a scene to produce an image that is as close as possible to what the human eye would see. To turn a photo-graph into an HDR image, we first extract the radiance information from the photo-graphs and then tone-map it into a single low dynamic range image. Both the recon-struction and tone-mapping algorithms rely on matrix operations, making them easy to implement in MATLAB.

Reconstructing Radiance InformationThe low dynamic range of a single digital photograph contains accurate radiance in-formation only for correctly exposed pixels. Changing the exposure captures radiance

By Peter Webb

Modern digital cameras may rival the human eye’s perception of color and

spatial resolution, but they cannot yet match the eye’s ability to manage con-

trast. An image sensor or display medium’s contrast ratio defines the distance

between the darkest black and the lightest white that the device records or

displays. The eye’s contrast ratio of 1:100,000 is roughly 24 times greater

than the 1:4096 of a typical digital camera. As a result, even in correctly

exposed photographs, shadows are often too dark and highlights too

bright, creating a noticeable loss of detail.

Rendering High Dynamic Range Images on the Web

MATLAB Digest

information for different sets of pixels. To accurately analyze a sequence of photo-graphs to extract radiance information for the entire scene, we must take into account the way the camera sensor responds to dif-ferent levels of light.

Digital cameras map the radiance of a scene into image pixel values via the camera response function, a nonlinear function that combines the radiance E falling on sensor location i and the exposure time ∆t to pro-duce color value C at pixel i.

In the approach developed by Devebec and Malik, the radiance-reconstruction pro-cess samples the exposure stack to determine the inverse of the camera response function.


Translating these mathematical equa-tions into a working algorithm requires ap-proximately 200 lines of MATLAB code.

MATLAB provides a large library of built-in functions to simplify the implementation. For example, loading a JPEG image requires only a single function call, as does extract-ing the exposure information from the im-age metadata.

Tone-Mapping Radiance Data Tone mapping compresses the wide dyn-amic range of an HDR data set into the much narrower dynamic range of a display device. A good algorithm produces an image that is perceptually similar to the original scene. Because luminance largely governs our perception of contrast, most tone-map-ping algorithms operate on the luminance channel of the HDR image. The simplest form of compression is linear scaling, which involves scaling the luminance values in the HDR data set to fit into the luminance range of the output device. However, linear scal-ing removes a significant amount of detail (Figure 1B).

Because the human eye’s response to light is logarithmic, a logarithmic trans-formation produces a more recognizable image. Taking the natural log of the color values before normalizing shifts many more of the HDR values into a range that is accessible by an LDR device (Figure 1C). This approach restores much of the detail

but desaturates the colors and reduces con-trast. Many tone-mapping algorithms op-erate in the logarithmic domain, and each addresses the deficiencies of logarithmic compression differently.

Contrast-limited adaptive histogram equalization, introduced by Ward, helps maintain a perceptual equivalence between the HDR and LDR images. Recall that HDR data represents the amount of light in the scene rather than the perceived colors in an image. Ward’s tone-mapping algorithm takes the logarithm of this light, as our eyes would, increases the contrast of the image by histogram equalization to make full use of the available dynamic range, and restores the saturation and contrast lost by logarith-mic luminance compression.

The algorithm performs five steps:

1. Compute the logarithm of the HDR im-age and normalize it to the interval [0 1], thereby compressing the dynamic range of HDR data to an interval that a LDR device can display.

2. Convert the normalized image to the CIE L*a*b* color space, in which lumi-nance can be manipulated directly.

3. Apply contrast-limited adaptive histo-gram equalization to the luminance channel.

4. Adjust image contrast according to a user-specified parameter.

5. Increase or decrease color saturation as requested by the user.

Let g be the natural log of the invertible camera function, Zmin and Zmax the pixel value boundaries, N the number of samples, P the number of photographs, and w a ‘hat’ weighting function that encourages the smoothness of g. To find the range of g we minimize this objective function:

Once the matrices of coefficients have been initialized, two lines of MATLAB code solve this system of equations using Gauss-ian elimination:

X = A \ b;

g = X(1:256);

With g fully determined by a lookup table, we reconstruct the radiance map by combining the exposures, using a weighted average of the camera response function of the pixels in each exposure:

Note that N is the total number of pixels in each image and not the number of sam-pled pixels.

The MATLAB code for the numerator in equation (3) is simply

for j = 1:P

relRad = ...

relRad + weightFcn(C(j)) .* ...

(filmResponse(C(j))– ...

logExposure(j));

end

Here, a for loop replaces the outer sum-mation operator. In MATLAB code, math-ematical operators automatically iterate over their matrix inputs, making the inner summation operator implicit.

Figure 1. Single LDR exposure (A), linear scaling (B), and logarithmic compression (C).

A B C


Using MATLAB and Image Processing Toolbox™, we can implement these steps in just seven lines of code (Code sample 1).

Compare the logarithmic scaling of Figure 1C to the tone-mapped image in Figure 5. Clearly, tone mapping algorithms produce much more lifelike images. An ex-amination of the histogram of each image reveals why. The histogram of the logarith-mically scaled image is jagged (Figure 2),

indicating a loss of smooth tonal variation, and compressed along the luminance axis, which wastes available dynamic range.

The histogram of the tone-mapped im-age (Figure 3) differs from that of the loga-rithmically scaled image. Both peak in the same areas, but the LDR histogram stretches across the entire x-axis, while the HDR histogram leaves almost 10% of the x-axis unused.

Because the image contains a relatively high number of black pixels, the LDR his-togram peaks in the low mid-tone range, but the smooth shape at either end of the range indicates that neither highlights nor shadows have been clipped. The smooth-ness of the histogram in Figure 3 shows that tone mapping avoided posterization (image blotching created by compression of the color space) and restored fine variations in

% Compute normalized log of HDR data

hdr = lognormal(hdr);

% Convert from the sRGB color space to the CIE L*a*b* colorspace

Lab = sRGB2Lab(hdr);

Luminance = Lab(:,:,1);

% Increase contrast via histogram equalization

Luminance = adapthisteq(Luminance, 'NumTiles', numtiles);

% Adjust contrast (ignore shadows, clip highlights)

Luminance = imadjust(Luminance, LRemap, [0 1]);

% Restore color saturation by increasing a* and b* channels

Lab(:,:,2) = Lab(:,:,2) * saturation;

Lab(:,:,3) = Lab(:,:,3) * saturation;

Code sample 1. Implementing the tone-mapping algorithm.

Figure 2. Luminance of log-compressed HDR data, taken from image C in Figure 1.

Figure 3. Histogram of tone-mapped image.


image tone, as is clearly seen by comparing Figure 1C with the image in Figure 5.

Figure 4 shows the boundary images from a stack of 10 LDR images of a high-contrast night-time scene. Combining all 10 images to form an HDR data set and then tone-mapping this data set for a computer monitor produced the image in Figure 5. The tone-mapped image has more detail than either boundary image, and more accu-rate color fidelity. Further processing could reduce the halo effect, which is caused partly by lens flare and partly by the edge-enhanc-ing properties of histogram equalization.

Deploying HDR Explorer as a Web Application

A MATLAB based Web application requires a browser based GUI, a server-side interface to the MATLAB algorithm, and Java Script to integrate the two. The HDR Explorer application’s browser-based GUI consists of a JavaServer Pages™ (JSP) form, the server-side interface of a Java servlet and a Java API generated by MATLAB Builder™ JA. The integration code is a manually written Java Script function. The application provides two functions: LDR image stack preview and HDR data generation and tone mapping.

Designing a Browser GUI

Our HDR Explorer application needs a way to specify input images, GUI controls to adjust transformation parameters, and a name for the tone-mapped LDR image. For simplicity, the application accepts only JPEG images and requires a separate direc-tory for each LDR exposure stack. The GUI consists of several data input fields, two but-

tons, and an image display area (Figure 6). Once the user identifies a directory contain-ing an LDR image stack, the HDR Explorer application displays that stack’s median ex-posure image. The user can adjust the HDR creation and tone-mapping parameters and then create and display the tone mapped LDR image, which is automatically saved as a TIFF file.

Figure 5. HDR data, tone-mapped to LDR image.Figure 4. Minimum and maximum LDR exposure frames (from a stack of 10).

Figure 6. Web interface: a JSP form containing a MATLAB Web figure.


The GUI serves as a front end for experi-menting with HDR data creation and tone-mapping algorithms. It is simple enough to be coded by hand, using raw HTML GUI components to create radio buttons or other selectors to choose between HDR image- creation algorithms and a POST method for submitting the JSP form.

Creating a Java APIWe create the Java API automatically from the MATLAB HDR algorithm with MATLAB Builder JA. The following command creates a Java component that exports the HDR algorithm via two top-level wrapper functions, previewImageStack and jpg2hdr:

>> mcc -v -W ...

"java.(insert string here)" ...

jpg2hdr.m getWebFigure.m ...

previewImageStack.m

MATLAB Builder for JA does all the hard work: determining which functions to in-clude in the component, generating func-tions that let you call the MATLAB HDR functions from Java, and packaging the

component into a JAR file. These generated functions also manage the redistributable MATLAB runtime and automatically con-vert data between native MATLAB and Java types. When installed on a server, they be-come accessible to JSP servlets.

Integrating a Browser GUI with a MATLAB AlgorithmThe HDR Explorer GUI sends parameters to previewImageStack and jpg2hdr and hosts a MATLAB Builder JA Web fig-ure capable of displaying the full range of MATLAB graphics. The JSP form commu-nicates with the MATLAB runtime via the MATLAB Builder JA generated wrapper class HDRExplorer and a Java bean.

For example, after creating the wrapper class instance componentObject (of type HDRExplorer) in jspInit, the page embeds a scriptlet to invoke jpg2hdr when the user clicks the Create HDRI button, which is named SubmitHDR in the JSP form’s HTML (Code sample 2).

Here, guiState is the Java bean used to pass data between client and server, and

� MATLAB Digest

if (request.getParameter(“SubmitHDR”) != null)

{

MWCellArray flags = new MWCellArray(2,1);

flags.set(1,"Show");

flags.set(2,"ReturnWebFigure");

Object[] result =

componentObject.jpg2hdr(

1, // One output -- a structure array

guiState.getDirectory(), guiState.getOutputName(),

guiState.getCreateHDRGroup(), flags, 5, 250,

guiState.getLowLight(), guiState.getHighLight(),

guiState.getSaturation(), guiState.getContrast(),

figureHandle);

UpdateWebFigure(result, "webfig");

}

Code sample 2. Script to invoke jpg2hdr.

the MWCellArray type is a Java proxy for the MATLAB cell-array data type. (Cell ar-rays are N-dimensional generalizations of name-value associative lists like the C++ standard template library type map). Since jpg2hdr modifies the image displayed in the Web figure, UpdateWebFigure notifies the client-side Web figure instance to fetch the serialized image data from the server.

A User-Friendly PlatformThe matrix-based and highly mathematical nature of HDR image processing algorithms makes them easy to implement in MATLAB. Their mathematical formulas often map term-by-term into MATLAB expres-sions or statements. MATLAB Builder JA simplifies the process of turning a MATLAB program into a Web-based application by automatically generating a browser-callable Java API. With MATLAB Builder JA and your favorite HTML or servlet development tools, you have a programmer-friendly plat-form for delivering mathematically rich ap-plications to any desktop on the Internet. ■


Resources

visit www.mathworks.com

technical support www.mathworks.com/support

online user community www.mathworks.com/matlabcentral

Demos www.mathworks.com/demos

training services www.mathworks.com/training

thirD-party proDucts anD services www.mathworks.com/connections

Worldwide contactswww.mathworks.com/contact

e-mail [email protected]

91578v00 06/08

© 2008 The MathWorks, Inc. MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See www.mathworks.com/trademarks for a list of additional trade-marks. Other product or brand names may be trademarks or registered trademarks of their respective holders.

For More Information ■ Debevec, P. and Malik, J., “Recovering

High Dynamic Range Radiance Maps from Photographs,” in proceedings of the 24th annual conference on computer graphics and interactive techniques, 369-378, ACM SIGGRAPH, 1997

■ matlaB Builder Java Ja www.mathworks.com/hdri_java

■ Reinhard, E., Ward, G., Pattanaik, S., and Debevec, P., high Dynamic range imaging, Morgan Kaufmann, Boston, 2006

■ Ward, G., Rushmeier, H., and Piatko, C., “A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes,” Lawrence Berkeley National Laboratories, http://www.mathworks.com/hdri_paper

AcknowledgementsI wish to thank Jeff Mather, for introduc-ing me to the techniques used to create and process HDR images and for his imple-mentation of Ward’s tone-mapping op-erator, and Bigelow Cleaners of Newton MA, for permission to use photographs of their sign.

Documents

Rendering High Dynamic Range Images on the Web · Compute the logarithm of the HDR im-age and normalize it to the interval [0 1], thereby compressing the dynamic range of HDR data