End-to-End Differentiable Learning to HDR Image Synthesis for Multi-exposure Images

Recovering the scene luminance with given LDR images and reconstructing HDR images requires estimating the intensity-to-luminance mapping function of the individual camera. The estimating process of the mapping function is called the radiometric calibration. The commonly used radiometric calibration estimates mapping function, which is the camera response function (CRF), from a given multi-exposure stack and corresponding exposure values. Based on the assumption about the shape of the CRF, most approaches can be categorized into two classes: parametric and non-parametric methods.

The parametric methods assume the CRF to have a specific and analytic functional form, such as a gamma function(Mann and Picard 1994), or a polynomial function (Mitsunaga and Nayar 1999). Furthermore, Grossberg and Nayar (Grossberg and Nayar 2003) modeled CRF using the principal component analysis (PCA) to collect vectors from a large number of real CRFs. Besides, PCA based modeling methods were incorporated into recent deep learning methods (Li and Peers 2017; Liu et al. 2020). However, parametric approaches suffer from making explicit assumptions on the analytical form of the CRFs, which is not adequate for monotonic modern camera configurations (Chen, McCloskey, and Yu 2019).

Non-parametric methods focus on estimating the CRF in a discrete function with the lookup table structure. Debevec and Malik (Debevec and Malik 2008) proposed a least-square formulation with the smoothness constraints to recover CRF in discrete function form. Lee et al.(Lee et al. 2012) utilized the observation that images in the multi-exposure stack are linearly dependent on reconstructing HDR images. Badki et al.(Badki, Kalantari, and Sen 2015) proposed a radiometric calibration method to compensate significant motions in images using a random sample consensus (RANSAC)-based method. Furthermore, a recent deep learning-based approach (Endo, Kanamori, and Mitani 2017; Lee, An, and Kang 2018b) applied the most commonly used non-parametric method: Debevec and Malik’s approach recovering the CRF. However, since the non-parametric radiometric calibration recovers a CRF as a discrete function and non-differentiable form, the whole end-to-end network implementation considering the multi-exposure stack has been limited.

Debevec and Malik (Debevec and Malik 2008) proposed the HDRI pipeline that estimates the CRF using the non-parametric radiometric calibration, which is commonly used. Given LDR images with different exposures, estimating the CRF or inverse CRF is modeled as the least-square problem as follows:

$$\begin{split}&O=\sum_{i}^{N}\sum_{j}^{P}[g(Z_{ij})-\ln E_{i}+EV_{j}]^{2}\\ &+\lambda\sum_{z=Z_{min}+1}^{Z_{max}-1}g^{\prime\prime}(z)^{2},\end{split}$$

(1)

where $O$ denotes an objective function, $g$ denotes an inverse CRF, and $Z_{ij}$ as a pixel intensity value of $i$-th pixel with $j$-th exposure value. $Z_{min}$ and $Z_{max}$ indicate minimum and maximum intensity values of given LDR images. $N$ and $P$ are the number of images and exposure values of the stack, and $i$, and $j$ are their corresponding indices, respectively. $E_{i}$ denotes the luminance value of $i$-th pixel and $EV_{j}$ denotes the $j$-th exposure value. The exposure value can substitute exposure time with a fixed aperture and ISO value. The second term of the objective function regularizes the CRF to be smoothened with the hyperparameter $\lambda$. By minimizing the objective function, we can obtain the discrete CRF of $g$, which maps 8-bit pixel intensity values to 32-bit luminance values. With the recovered inverse CRF $g$, the pixel intensity value can be remapped to the luminance value as follows:

$$\ln{E_{i}}=\ g(Z_{ij})-EV_{j}.$$

(2)

The scene luminance is remapped with Eq. 2; however, as inverse CRF has the form of the non-differentiable function, we transform the inverse CRF with a linear approximation technique.

Let an inverse CRF be $g=[p_{0},p_{1},\cdots,p_{N}]$ with $N$ denoting the maximum intensity value of multi-exposure images. We define the derivative of the linearized function $\hat{g}$ as follows:

$$\frac{\partial\hat{g}}{\partial Z_{ij}}=\left\{\begin{array}[]{lr}g(0),&\mbox{if }Z_{ij}=0\\ g(Z_{ij})-g(Z_{ij}-1),&\mbox{otherwise.}\end{array}\right.$$

(3)

Fig. 2 illustrates our approach to piecewise-linearize the inverse CRF. With the sampled pixels using the Grossberg and Nayar’s method (Grossberg and Nayar 2003), we linearize the function with the prior assumptions of the CRF having the characteristic of monotonically increasing with the shape of the non-linear curve. We reformulate the function with a piece-wise linear form to back-propagate the difference between the function value and the one before, as shown in Eq. 3. The simple linearization method enables the propagation of gradients to each pixel of the multi-exposure stack with the chain rule (Goodfellow, Bengio, and Courville 2016). The gradients from the loss of luminance values flow to pixel intensity values of each image, which imposes constraints on the generated multi-exposure stack to have correlated values with Eq. 3. Hence, our novel framework enables the networks to accomplish both the multi-exposure stack generation task and the HDR synthesis task, with the optimal objective of reconstructing high-quality HDR images.

Furthermore, we implemented the polynomial curve fitting approach proposed by Mitsunaga and Nayar(Mitsunaga and Nayar 1999) in the differentiable HDR synthesis layer. Polynomial curves can be differentiated; however, as modern cameras’ CRF have monotonic and resembling shape (Chen, McCloskey, and Yu 2019), the higher-order models are not necessary. Therefore, we focused on the piece-wise linearization approach with further experiments (See the comparison between the piece-wise linearization and the polynomial curve fitting approach in the supplementary material). We verified our differentiable HDR layer outputs to reproduce the identical results with the MATLAB HDR Toolbox (Banterle et al. 2017) and to generate a pixel-wise gradient.

We incorporate the recursive generation of the multi-exposure image stack with the prior knowledge of the exposure manifold space (Lee, An, and Kang 2018b). We propose the recurrent-up and recurrent-down networks to be distinct from conventional methods (Lee, An, and Kang 2018a, b). Since the process is defined as a recursive process, we implement the convolutional gated recurrent unit (Conv-GRU) (Siam et al. 2017) to construct the recurrent network. In addition, as multi-exposure images have different over-exposed and under-exposed regions regarding their exposure values, we decompose the exposure transfer task into two path-ways. From a given single image, our model learns the global tone and local details with the global network and local network, respectively. With decomposed images, the refinement network integrates global and local components to generate fine-tuned images.

Fig. 3 shows the structures of sub-networks in our model. Our recurrent-up and recurrent-down networks contain three sub-networks of U-Net structures (Ronneberger, Fischer, and Brox 2015) to transfer exposures to the images with the relative up and down EVs: the global, local, and refinement networks. The global and local networks are constructed with $5$-level and $4$-level structures, respectively, with $2$ convolutional layers for each level. We implemented the Swish activation (Ramachandran, Zoph, and Le 2017) on each convolution layer to alleviate a gradient vanishing problem in recurrent models. Note that the refinement network shares the same structure with the global network except for the Conv-GRUs on bottleneck layers. We impose the global and local networks to focus on adaptively responding to the number of recursions, and the refinement network to focus on integrating the global and local components, which are global tones and gradient-based edge structures of a target LDR image, respectively. The image decomposition approach resolves the complexity of a single network (Lee, An, and Kang 2018b), as the adaptive response and integration are learned separately for the network to explicitly learn each corresponding task (See supplementary material for details).

The recurrent-up (or recurrent-down) network exploits the same weights for transferring exposures, even with the recurrent state that differs from the exposure value of an input. However, both the recurrent-up and recurrent-down networks should adaptively produce the over-exposed and under-exposed images corresponding to the exposure value of an input. Therefore, we use the conditional instance normalization to standardize feature maps of different exposure values. The normalization transforms a feature map, $X$, of which the shape is $C\times H\times W$, into a normalized map $Y$ by using two learnable parameters of $\gamma_{e}$ and $\beta_{e}$ with the target exposure value of $e$, which are in $\mathbb{R}^{C}$. The normalized map is formulated as $Y=\frac{\gamma_{e}(X-\mu)+\beta_{e}}{\sigma}$, where $\mu$ and $\sigma$ are the mean and the standard deviation of $X$ taken across spatial axes, respectively. In other words, our networks select the scale and shift factors according to the exposure value of an input LDR image. By using the conditional instance normalization, we can assist the network to focus on detecting subtle differences between the estimated and target images. Thus, we implemented a conditional instance normalization layer on the decoding layers of each level.

Our model is designed to facilitate the recurrent structure, which shifts the exposure level of the image gradually, as presented in Fig. 1. Specifically, the recurrent-up and recurrent-down networks are trained separately with a given single LDR image to generate the multi-exposure stack recursively. For the sub-networks, the global and local networks are trained in advance for 10k iterations, then we jointly trained the entire network, including the refinement network. The loss functions are defined independently with each sub-network. Specifically, the global network is trained with the pixel-wise $L_{1}$ loss ($L_{1}$) and histogram loss ($L_{hist}$) to constraint the network to generate the image with a similar global tone to the target image (See the supplementary material for details). The local network is trained with pixel-wise $L_{1}$ loss ($L_{edge}$) on edge maps computed with Canny edge detector (Canny 1986) of $\sigma=2$. The refinement network is trained with $L_{1}$ loss ($L_{1}$), the contextual bilateral loss ($L_{CoBi}$) (Zhang et al. 2019), and the HDR loss ($L_{HDR}$). For the HDR loss, we used a tone-mapped HDR loss with $\mu$-law to stabilize the training process (Yan et al. 2019). Note that $L_{CoBi}$ alleviates the ghosting artifacts due to the misaligned images by minimizing the distances between the matching features extracted from the $3$-rd and $4$-th layer of the pre-trained VGG-19 network (Simonyan and Zisserman 2014) with the bilateral filtering. Overall loss functions are formulated as follows:

$$\begin{split}&L_{global}=\lambda_{1}L_{1}+\lambda_{2}L_{hist}\\ &=\frac{\lambda_{1}}{N\cdot E}\sum^{E}_{e}\sum^{N}_{i}|\hat{I}^{e}_{i}-I^{e}_{i}|\hskip 51.21495pt\\ &+\frac{\lambda_{2}}{L\cdot E}\sum^{E}_{e}\sum^{L}_{l}|cnt_{l}(\hat{I}^{e})-cnt_{l}(I^{e})|,\end{split}$$

(4)

$$L_{local}=\lambda_{3}L_{edge}=\frac{\lambda_{3}}{N\cdot E}\sum^{E}_{e}\sum^{N}_{i}|\hat{E}^{e}_{i}-edge(I^{e}_{i})|,$$

(5)

$$\begin{split}&L_{refine}=\lambda_{4}L_{1}+\lambda_{5}L_{HDR}+\lambda_{6}L_{CoBi}\\ &=\frac{\lambda_{4}}{N\cdot E}\sum^{E}_{e}\sum_{i}^{N}|\hat{I}^{e}_{i}-I^{e}_{i}|+\frac{\lambda_{5}}{N}\sum_{i}^{N}|log\frac{1+\mu\hat{H_{i}}}{1+\mu H_{i}}|\\ &+\frac{\lambda_{6}}{M}\sum_{j}^{M}\min_{k}(\mathbb{D}_{p_{j},q_{k}}+w_{s}\mathbb{D^{\prime}}_{p_{j},q_{k}}),\end{split}$$

(6)

where $N$, $E$, $L$, and $M$ denote the number of pixels, exposure values, intensity levels, and features respectively, and for all the equations, $\hat{\cdot}$ represents the prediction of the network. $I_{i}^{e}$ denotes the $i$-th pixel value in image $I$ of exposure value $e$, and $cnt_{l}(\cdot)$ indicates the number of pixels which has a rounded down intensity $l$ in the input image $I$. $edge(\cdot)$ extracts gradient-based edge maps from the image $I$, and $E_{i}$ denotes the $i$-th pixel value in predicted edge map. $H_{i}$ is a pixel luminance in the HDR image, and $\mu$ is the compression parameter of the HDR image, where we set the value with $5000$. $\mathbb{D}_{p,q}$ indicates the sum of cosine distances between all the matched features of $p$ and $q$, and ${\mathbb{D^{\prime}}}_{p,q}$ indicates spatial coordinate distance. Note that $j$ and $k$ indicate indices of the matched feature of $p$ and $q$ respectively. We set the hyperparameters $\lambda_{1}=\lambda_{3}=\lambda_{4}=\lambda_{5}=1$ and $\lambda_{2}=\lambda_{6}=0.1$ in our experiments to stably train the networks.

The comparison evaluations were performed with 6 recent deep learning-based methods, both direct methods(HDRCNN (Eilertsen et al. 2017), ExpandNet (Marnerides et al. 2018), FHDR (Khan, Khanna, and Raman 2019), SingleHDR (Liu et al. 2020)) and multi-exposure stack-based methods (DrTMO (Endo, Kanamori, and Mitani 2017), Deep recursive HDRI (Lee, An, and Kang 2018b)) as benchmarks. The interchangeability of training datasets between methods is limited as the direct methods need a large amount of LDR-HDR image pair datasets, and the multi-exposure stack-based method requires an adequate amount of images of different exposures. Therefore, we used pre-trained models for ExpandNet, DrTMO, FHDR (2-iteration), and SingleHDR.

We evaluated the effectiveness of the individual components in our model on the VDS dataset, as shown in Table 3. We added modules incrementally on the U-Net structure (Ronneberger, Fischer, and Brox 2015), which is a baseline of our model with $5$-level and $2$ convolutional layers for each level, and evaluated with the HDR-VDP-2 score. The overall results show that our method using all modules improved $9.305$ and $4.483$ with HDR-VDP-2 score and PSNR, respectively.