FHDR: HDR Image Reconstruction from a SingleLDR Image using Feedback Network

Zeeshan KhanIndian Institute of Technology

Gandhinagar, Indiazeeshank606@gmail.com

Mukul KhannaIndian Institute of Technology

Gandhinagar, Indiamukul18khanna@gmail.com

Shanmuganathan RamanIndian Institute of Technology

Gandhinagar, Indiashanmuga@iitgn.ac.in

Abstract—High dynamic range (HDR) image generation froma single exposure low dynamic range (LDR) image has beenmade possible due to the recent advances in Deep Learning.Various feed-forward Convolutional Neural Networks (CNNs)have been proposed for learning LDR to HDR representations.To better utilize the power of CNNs, we exploit the idea offeedback, where the initial low level features are guided bythe high level features using a hidden state of a RecurrentNeural Network. Unlike a single forward pass in a conventionalfeed-forward network, the reconstruction from LDR to HDRin a feedback network is learned over multiple iterations. Thisenables us to create a coarse-to-fine representation, leading to animproved reconstruction at every iteration. Various advantagesover standard feed-forward networks include early reconstruc-tion ability and better reconstruction quality with fewer networkparameters. We design a dense feedback block and propose anend-to-end feedback network- FHDR for HDR image generationfrom a single exposure LDR image. Qualitative and quantitativeevaluations show the superiority of our approach over the state-of-the-art methods.

Index Terms—HDR imaging, Feedback Networks, RNN, DeepLearning.

I. INTRODUCTION

Common digital cameras can not capture the wide rangeof light intensity levels in a natural scene. This can leadto a loss of pixel information in under-exposed and over-exposed regions of an image, resulting in a low dynamicrange (LDR) image. To recover the lost information andrepresent the wide range of illuminance in an image, highdynamic range (HDR) images need to be generated. Therehas been active research going on in the area of deep learningfor HDR imaging. The advances in deep learning for imageprocessing tasks have paved way for various approaches forHDR image reconstruction using feed-forward convolutionalneural network (CNN) architectures [1] [2] [3] [4] [5]. Theabove methods specifically transform a single exposure LDRimage into an HDR image. HDRCNN [1] proposed a deepautoencoder for HDR image reconstruction which uses aweighted mask to recover only the over-exposed regions of anLDR image. The authors of DRTMO [3] designed a frameworkwith two networks for generating up-exposure and down-exposure LDR images, which are merged to form an HDRimage. The network is not end-to-end trainable and uses alarge number of parameters. Unlike the others, the proposedFHDR model provides an end-to-end trainable solution and is

able to comprehensively learn the LDR-HDR mapping whileoutperforming the existing methods.

Deeper networks are known to learn more complex non-linear relationships like the LDR to HDR mapping. Thecaveat with deeper networks is that they consume a lot ofcomputational resources and tend to over-fit the training data.To overcome this problem, we exploit the power of feedbackmechanisms, inspired by [6], for the task of HDR imagereconstruction. A feedback block is an RNN whose output isfed back to guide its input via a hidden state. A feedbacknetwork can run for many iterations for a single trainingexample. Considering the number of iterative operations onthe shared network parameters, a feedback network is virtuallydeeper than the corresponding feed-forward network with thesame physical depth. Here, virtual depth = physical depth ×number of iterations.

For improving the reconstruction at every iteration, the lossis calculated for the output of each iteration. By doing so,the network is forced to create a coarse-to-fine representation,being able to reconstruct HDR content right from the firstiteration and improve with every subsequent iteration. Wepropose a global feedback block which consists of smallerlocal feedback blocks of densely connected layers, inspiredby the Dense-Net architecture [7]. Dense connections allowthe network to reuse features. This helps in learning robustimage representations, even with lesser network parameters.

The performance of our framework is evaluated on the stan-dard City Scene dataset [8] and another dataset prepared fromthe list of HDR image datasets suggested in [1]. Qualitativeand quantitative assessments of the network suggest that evenwith fewer network parameters, the proposed FHDR modeloutperforms the state-of-the-art methods.

II. HDR RECONSTRUCTION FRAMEWORK

A. Feedback system

Feedback systems are adopted to influence the input basedon the generated output, unlike the conventional feed-forwardnetworks, where information flow is unidirectional and is notdirectly influenced by the generated output. The authors in[6] exploited the feedback mechanism using an RNN, wherethe output of the feedback block travels back to guide itsinput via a hidden state. Their architecture uses ConvLSTMcells as the basic RNN units and is designed for the task of

arX

iv:1

912.

1146

3v1

[cs

.CV

] 2

4 D

ec 2

019

image classification. Even with lesser network parameters ascompared to other feed-forward networks, such networks areable to learn better representations. Recently, authors of [9]designed a feedback network specifically for the task of imagesuper-resolution which achieved state-of-the-art performance.

Inspired by the success of feedback networks, we designeda feedback network for learning the LDR to HDR mappingthat has been explained in detail in the following sections.

B. Model architecture

Fig. 1: FHDR Architecture

Our architecture consists of three blocks similar to [9], asshown in Fig. 1. The first block is the Feature Extraction block(FEB), followed by the Feedback block (FBB) and an HDRreconstruction block (HRB). Inspired by [10], we use a globalresidual skip connection for bypassing low level LDR featuresat every iteration to guide the HDR reconstruction block in thefinal layers. For every training example, the network runs for niterations. Here each iteration from t = 1 to t = n is a forwardpass in time in an unfolded RNN. The FEB is responsible forextracting the low-level feature information Fin from the inputLDR image ILDR.

Fin = fFEB(ILDR). (1)

Here, fFEB represents the operations of the FEB. Toachieve the feedback mechanism, Fin is fed to the FBB,combined with the output of the FBB from the previousiteration, using a global hidden state as below.

F tFBB = fFBB(Fin, F

t−1FBB). (2)

Here, F tFBB represents the output of the feedback block at

iteration t. At t = 1, when there is no feedback, the hiddenstate is initialised with the values of the extracted features Fin.

At every iteration, the low level LDR features from the firstconvolutional layer in FEB are added to the output of the FBBusing a global residual skip connection as below.

F tres = Fin(1) + F t

FBB . (3)

Here, F tres represents the global residual feature map learned

at iteration t. Fin(1) stands for the low level LDR features fromthe first convolutional layer in FEB. F t

res is passed to the HRBto generate an HDR image at every iteration as below.

Htgen = fHRB(F

tres) (4)

Here, fHRB represents the operations of the HRB and Htgen

represents the HDR image generated at tth iteration. For everyLDR image, a forward pass can run for n iterations, thereforegenerating n HDR images. Each generated image is coupledwith a loss, hence resulting in improved reconstruction at eachiteration through back-propagation in time.

C. Feedback block

Fig. 2: Feedback block

We have designed a novel feedback block for the task oflearning LDR-to-HDR representations, as shown in Fig. 2. Thebasic unit of the feedback block is a Dilated Dense Block(DDB) shown in Fig. 3. It is a modification of the Dense blockproposed in [7]. Dilated convolutions help in increasing thereceptive field of the network [11]. A DDB helps in utilisingall the hierarchical features from the input. Other than thetwo 1 × 1 convolutional layers for feature compression, eachDDB houses four dilated 3 × 3 convolutional layers, each ofwhich uses the information from all the previous layers usingdense skip connections. This reuse of features due to the denseforward connections allows for reduced network parametersand improves the learning ability. Three of such DDBs cometogether to form the feedback block of the network.

We implement global and local feedback mechanisms de-scribed as follows.

1) Global Feedback: The global feedback block, FBB isconsidered as an RNN with a global hidden state. Highlevel features are transferred from the output of the feedbackblock at the t − 1th iteration to its input at the tth iteration.The hidden state is concatenated with FTin and a 1 × 1compression convolution layer is applied for high-level andlow-level feature fusion as shown in Fig. 2. The fused featuresare passed to the dilated dense blocks, followed by a 3 × 3convolution layer for further processing.

2) Local Feedback: We argue that a feedback connectionis always beneficial as it helps to guide the low-level featureswhich are in some way blind to the higher level features.Hence, we have implemented local feedback connections overeach DDB which aim to improve the features generatedlocally. These connections run parallel to the global feedbackconnections and increase the overall effectiveness of the net-work. Each DDB can be considered as an RNN similar to theglobal feedback block, transferring features from its output toits input via a local hidden state.

Fig. 3: Dilated Dense BlockD. Loss function

Loss calculated directly on HDR images is misrepresenteddue to the dominance of high intensity values of images with awide dynamic range. Therefore, we tonemap the generated andthe ground truth HDR images to compress the wide intensityrange before calculating the loss function value. We use theµ-law for tonemapping, as suggested by [12]. The µ-law isrepresented as below.

T (Htgen) =

log(1 + µHtgen)

log(1 + µ)(5)

Here, T represents the tonemapping operation and µ definesthe amount of compression, which is set to 5000 for theexperiments. In addition to the L1 loss suggested by previousfeedback networks, we use a perceptual loss [13] for improv-ing the visual quality of the generated image. We calculate theL1 loss and the perceptual loss at every iteration t and takean average over all the iterations. The average L1 loss LL1 isgiven below.

LL1 =1

n

n∑t=1

∣∣∣∣T (Htgen)− T (Hgt)

∣∣∣∣ (6)

Here, Hgt represents the ground truth image. The averageperceptual loss Lp can be represented as below.

Lp =1

n

n∑t=1

fV GG19

(T (Ht

gen), T (Hgt))

(7)

Here, fV GG19represents the perceptual loss calculated be-

tween the tonemapped ground truth and generated images. Thefinal loss function L is given below.

L = Lp + λLL1 (8)

Here, λ is set to 0.1 for all the experiments. We have observedthat applying only the L1 distance produces dark artifacts.However, a combination of L1 loss and perceptual loss hasresulted in improved visual quality of the generated image.

E. Implementation details

All the convolutional layers (conv2d) in the network have3×3 kernels, followed by a ReLU activation, unless mentionedotherwise. There are 2 conv2d layers in the FEB, 20 (6×3+2)conv2d layers in FBB, and 2 conv2d layers in the HRB. Thesize of the feature maps remains same throughout the network.The depth of the feature maps also remain same i.e. 64, exceptfor in the DDBs. A growth rate of 32 has been implementedfor the DDBs, meaning that 3×3 conv2d layers of each denseblock output a 32 channel feature map that gets concatenatedwith features from all the preceding layers. This accumulated

feature map depth is brought down using a 1×1 conv2d layerat the end of the dense block.

We trained our network on Geforce RTX 2070 GPU witha batch-size of 16 for the City Scene dataset and 6 for thecurated HDR dataset. Adam optimizer [14] was adopted withmomentum parameters β1 = 0.5 and β2 = 0.999. All thevariants of the proposed model were trained for 200 epochswith an initial learning rate of 2× 10−4 for first 100 epochs,decayed linearly over the next 100 epochs.

III. EXPERIMENTS

A. Dataset

We have trained and evaluated the performance of ournetwork on the standard City Scene dataset [8] and anotherHDR dataset curated from various sources shared in [1].The City Scene dataset is a low resolution HDR dataset thatcontains pairs of LDR and ground truth HDR images of size128 × 64. We trained the network on 39460 image pairsprovided in the training set and evaluated it on 1672 randomlyselected images from the testing set. The curated HDR datasetconsists of 1010 HDR images of high resolution. Training andtesting pairs were created by producing LDR counterparts ofthe HDR images by exposure alteration and application ofdifferent camera curves from [18]. Data augmentation wasdone using random cropping and resizing. The final trainingset consists of 11262 images of size 256×256. The testingset consists of 500 images of size 512×512 to evaluate theperformance of the network on high resolution images.

B. Evaluation metrics

For the quantitative evaluation of the proposed method, weuse the HDRVDP-2 Q-score metric, specifically designed forevaluating reconstruction of HDR images based on humanperception [19]. We also use the commonly adopted PSNR andSSIM image-quality metrics. The PSNR score is calculated indB between the µ-law tonemapped ground truth and generatedimages.

C. Feedback mechanism analysis

To study the influence of feedback mechanisms, wecompare the results of four variants of the network based onthe number of feedback iterations performed - (i) FHDR/W(feed-forward / without feedback), (ii) FHDR-2 iterations, (iii)FHDR-3 iterations, and (iv) FHDR-4 iterations. To visualisethe overall performance, we plot the PSNR score valuesagainst the number of epochs for the four variants, shown inFig. 5. The significant impact of feedback connections canbe easily observed. The PSNR score increases as the numberof iterations increase. Also, early reconstruction can be seenin the network with feedback connections. Based on this, wedecided to implement an iteration count of 4 for the proposedFHDR network.

IV. RESULTS

Qualitative and quantitative evaluations are performedagainst two non-learning based inverse tonemapping methods-(AKY) [15] and (KOV) [16], two deep learning methods-

LDR AKY [15] KOV [16] DRTMO [3] HDRCNN [1] FHDR/W FHDR GROUND TRUTH

Fig. 4: Qualitative evaluation against five described methods on the curated HDR dataset. (HDR images have been tonemappedusing Reinhard tonemapping algorithm [17] for displaying.)

Fig. 5: Convergence analysis for four variants of FHDR,evaluated on the City Scene dataset.

HDRCNN [1] and DRTMO [3] and the feed-forward counter-part of the proposed FHDR method. We use the HDR toolbox[20] for evaluating the non-learning based methods. The deeplearning methods were trained and evaluated on the describeddatasets, as mentioned in earlier sections.

DRHT [4] is the state-of-the-art deep neural network forthe image correction task. It reconstructs HDR content as anintermediate output and transforms it back into an LDR image.Due to unavailability of DRHT network implementation, re-sults of their network on the curated dataset are not presented.Performance metrics of their LDR-to-HDR network, trainedon the City Scene dataset has been sourced from their paperbecause of the same experiment setup in terms of the trainingand testing datasets.

A. Quantitative evaluation

A quantitative comparison of the above mentioned HDRreconstruction methods has been presented in Table 1. Theproposed network outperforms the state-of-the-art methodsin evaluations performed on both the datasets. Results ofDRTMO [3] could not be calculated on the City Scenedataset because the network does not accept low resolutionimages. Even though the feed-forward counterpart of FHDRoutperforms other methods, the proposed FHDR network (4-iterations) performs far better.

B. Qualitative evaluationAs can be seen in Fig. 4, non-learning based methods

are unable to recover highly over-exposed regions. DRTMObrightens the whole image and is able to recover only theunder-exposed, regions. HDRCNN, on the other hand, isdesigned to recover only the saturated regions and thus underperforms in recovering information in under-exposed darkregions of the image. Unlike others, our FHDR/W pays equalattention to both under-exposed and over-exposed regions ofthe image. The proposed FHDR network with the feedbackconnections enhances the overall sharpness of the image andperforms an even better reconstruction of the input.

Methods City Scene Dataset Curated HDR DatasetPSNR SSIM Q-score PSNR SSIM Q-score

AKY [15] 15.35 0.44 35.40 9.58 0.20 33.47KOV [16] 16.77 0.59 35.31 12.99 0.41 29.87HDRCNN[1]

13.21 0.38 54.34 12.13 0.34 55.32

DRTMO [3] - - - 11.4 0.28 58.85DRHT [4] - 0.93 61.51 - - -FHDR/W 25.39 0.89 63.21 16.94 0.74 65.27FHDR 32.54 0.95 67.18 20.3 0.79 70.97

TABLE I: Quantitative comparison against state-of-the-artmethods.

V. CONCLUSION

We propose a novel feedback network FHDR, to reconstructan HDR image from a single exposure LDR image. Thedense connections in the forward-pass enable feature-reuse,thus learning robust representations with minimum parameters.Local and global feedback connections enhance the learningability, guiding the initial low level features from the highlevel features. Iterative learning forces the network to createa coarse-to-fine representation which results in early recon-structions. Extensive experiments demonstrate that the FHDRnetwork is successfully able to recover the underexposed andoverexposed regions outperforming state-of-the-art methods.

VI. ACKNOWLEDGEMENT

This research was supported by the SERB Core ResearchGrant.

REFERENCES

[1] G. Eilertsen, J. Kronander, G. Denes, R. K. Mantiuk, and J. Unger,“Hdr image reconstruction from a single exposure using deep cnns,”ACM Transactions on Graphics (TOG), vol. 36, no. 6, p. 178, 2017.

[2] D. Marnerides, T. Bashford-Rogers, J. Hatchett, and K. Debattista, “Ex-pandnet: A deep convolutional neural network for high dynamic rangeexpansion from low dynamic range content,” in Computer GraphicsForum, vol. 37, pp. 37–49, Wiley Online Library, 2018.

[3] Y. Endo, Y. Kanamori, and J. Mitani, “Deep reverse tone mapping.,”ACM Trans. Graph., vol. 36, no. 6, pp. 177–1, 2017.

[4] X. Yang, K. Xu, Y. Song, Q. Zhang, X. Wei, and R. W. Lau, “Imagecorrection via deep reciprocating hdr transformation,” in Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition,pp. 1798–1807, 2018.

[5] S. Lee, G. H. An, and S.-J. Kang, “Deep chain hdri: Reconstructing ahigh dynamic range image from a single low dynamic range image,”IEEE Access, vol. 6, pp. 49913–49924, 2018.

[6] A. R. Zamir, T.-L. Wu, L. Sun, W. B. Shen, B. E. Shi, J. Malik,and S. Savarese, “Feedback networks,” in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pp. 1308–1317, 2017.

[7] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Denselyconnected convolutional networks,” in Proceedings of the IEEE confer-ence on computer vision and pattern recognition, pp. 4700–4708, 2017.

[8] J. Zhang and J.-F. Lalonde, “Learning high dynamic range from outdoorpanoramas,” in Proceedings of the IEEE International Conference onComputer Vision, pp. 4519–4528, 2017.

[9] Z. Li, J. Yang, Z. Liu, X. Yang, G. Jeon, and W. Wu, “Feedback networkfor image super-resolution,” in The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2019.

[10] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual densenetwork for image super-resolution,” in Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition, pp. 2472–2481,2018.

[11] F. Yu and V. Koltun, “Multi-scale context aggregation by dilatedconvolutions,” arXiv preprint arXiv:1511.07122, 2015.

[12] N. K. Kalantari and R. Ramamoorthi, “Deep high dynamic rangeimaging of dynamic scenes.,” ACM Trans. Graph., vol. 36, no. 4,pp. 144–1, 2017.

[13] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-timestyle transfer and super-resolution,” in European conference on computervision, pp. 694–711, Springer, 2016.

[14] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.

[15] A. O. Akyuz, R. Fleming, B. E. Riecke, E. Reinhard, and H. H. Bulthoff,“Do hdr displays support ldr content?: a psychophysical evaluation,” inACM Transactions on Graphics (TOG), vol. 26, p. 38, ACM, 2007.

[16] R. P. Kovaleski and M. M. Oliveira, “High-quality reverse tone mappingfor a wide range of exposures,” in 2014 27th SIBGRAPI Conference onGraphics, Patterns and Images, pp. 49–56, IEEE, 2014.

[17] E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda, “Photographic tonereproduction for digital images,” ACM Trans. Graph., vol. 21, pp. 267–276, July 2002.

[18] M. D. Grossberg and S. K. Nayar, “What is the space of camera responsefunctions?,” in 2003 IEEE Computer Society Conference on ComputerVision and Pattern Recognition, 2003. Proceedings., vol. 2, pp. II–602,June 2003.

[19] M. Narwaria, R. Mantiuk, M. P. Da Silva, and P. Le Callet, “Hdr-vdp-2.2: a calibrated method for objective quality prediction of high-dynamicrange and standard images,” Journal of Electronic Imaging, vol. 24,no. 1, p. 010501, 2015.

[20] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, Advanced highdynamic range imaging. AK Peters/CRC Press, 2017.