Automatic Exposure Selection and Fusion
for High Dynamic Range Photography via Smartphones

Here gradient information is used to guide the luminance fusion. The rationale behind using gradient information is that a well-exposed image provides a better representation of edges, that is to say it provides higher gradient values compared to a poorly-exposed image Ref18 . The gradients are extracted from the well-exposed regions of $\textbf{Y}_{UE}$, $\textbf{Y}_{NE}$, and $\textbf{Y}_{OE}$ and merged into gradient maps along the horizontal ($\mathbf{\Lambda}_{h}$) and vertical ($\mathbf{\Lambda}_{v}$) directions. An initial estimate (X) of the fused luminance (Y) is also obtained by averaging $\textbf{Y}_{UE}$, $\textbf{Y}_{NE}$, and $\textbf{Y}_{OE}$. Using the extracted gradients and the initial estimate, the fused luminance (Y) is devised by solving the optimization problem that is stated below.

where $\|.\|_{F}^{2}$ denotes the Frobenius matrix norm, $\nabla^{h}$ and $\nabla^{v}$ indicate the horizontal and vertical gradient operators, and $\lambda$ is a weighting parameter. The solution to this optimization problem is given by (see Appendix A):

where u and v are the pixel coordinates, $\mathcal{F}$ and $\mathcal{F}^{-1}$ denote the forward and inverse Fourier transforms, respectively, and $\dagger$ denotes complex conjugate.
For luminance fusion, the images $\textbf{Y}_{UE}$, $\textbf{Y}_{NE}$, and $\textbf{Y}_{OE}$, are available and the output image Y is estimated. As stated earlier, X, $\mathbf{\Lambda}_{h}$ and $\mathbf{\Lambda}_{v}$ are obtained from the available image data in the solution provided in (2) to estimate Y. In this work, the initial estimate X is chosen to be the average of the images captured at different exposure settings because the average luminance conveys the brightness of a scene as illustrated in Fig. 2. In other words, X is set to be

The horizontal ($\mathbf{\Lambda}_{h}$) and the vertical ($\mathbf{\Lambda}_{v}$) gradients are extracted from the luminance components using the clustering outcome obtained in the exposure selection step. In other words, the gradient in the dark, normal, and bright regions is obtained from the over-exposed, normal-exposed, and under-exposed images, respectively. For this purpose, three binary cluster maps (R) are generated which correspond to the three clusters obtained from the exposure selection step . For a pixel k of $\textbf{I}_{in}$, if its gray-level belongs to the cluster 1, 2, or 3 (specified by the dark, normal, and bright areas, respectively), then the corresponding pixel in $\textbf{R}_{UE}$, $\textbf{R}_{UE}$, or $\textbf{R}_{UE}$ is set to 1, where $\textbf{R}_{UE}$, $\textbf{R}_{NE}$, and $\textbf{R}_{OE}$ represent the binary cluster maps for the under-exposed, normal-exposed, and over-exposed clusters, respectively. Hence, for a pixel k

where $C_{\textbf{I}_{in}^{k}}$ denotes the cluster label of the $k^{th}$ pixel of $\textbf{I}_{in}$. Then, $\mathbf{\Lambda}_{h}$ and $\mathbf{\Lambda}_{v}$ for the pixel k are computed as follows:

The method described above causes sharp transitions on cluster borders since image gradients get multiplied by binary cluster maps. In order to avoid such sharp transitions, a low-pass filter is applied to every binary map. Here, the so-called guided filter Ref13 is applied to the binary masks to make transitions of cluster borders smooth. In other words, the smooth cluster maps ($\mathbf{\Omega}$) for the three clusters are obtained as follows:

where X acts as a guide and r and $\varepsilon$ indicate the filter parameters. The smoothed cluster maps are then normalized. The normalized weights for a pixel k are computed this way

Once the normalized weights are computed, $\mathbf{\Lambda}_{h}$ and $\mathbf{\Lambda}_{v}$ are obtained as follows:

X, $\mathbf{\Lambda}_{h}$ and $\mathbf{\Lambda}_{v}$ computed via (3) and (8) are then inserted into (2) to obtain Y.

Unlike the luminance component, chrominance fusion is achieved through weighted averaging. The saturation value of a pixel is used as a weight to fuse its chrominance components. The saturation $s$ for a pixel k with red, green, and blue components of $r_{k}$, $g_{k}$, $b_{k}$ is computed this way

Let $s_{UE}^{k}$, $s_{NE}^{k}$, and $s_{OE}^{k}$ represent the saturation of the pixel k in $\textbf{I}_{UE}$, $\textbf{I}_{NE}$, and $\textbf{I}_{OE}$, respectively. The normalized weights are obtained as follows:

And the chrominance components of the pixel k are then fused according to these equations:

The capturing part involves taking $\textbf{I}_{in}$ followed by taking $\textbf{I}_{OE}$, $\textbf{I}_{NE}$, and $\textbf{I}_{UE}$. A button named Capture is provided in the app to launch capturing images. Once the Capture button is pressed by the user, the capturing process starts as described in Algorithm 1. The output of Algorithm 1 consists of three images in the YCbCr format, corresponding to the under-exposed, normal-exposed, and over-exposed conditions as well as the clustering outcome in the form of three binary maps.

When the three images are made available from the capturing part, the processing outlined in section 3, which is coded in C, is performed and an HDR image as described in Algorithm 2 is generated. This image is then sent back to the Java environment for getting saved in memory.

The parameters of the exposure fusion part included the regularization parameter $\lambda$ of the optimization in (2), as well as the guided filter parameters, i.e. the filter radius r and the regularization parameter $\varepsilon$ in (6). For the guided filter, since maximal smoothing within short distances was desired, the minimal value of 1 for r together with a large value of 1 for $\varepsilon$ were considered. The kernels $[-1,1]$ and $[-1,1]^{T}$ were used as the horizontal and vertical gradient operators, respectively.
The experiments revealed that the optimal $\lambda$ was highly dependent on the image size. To address this issue, resized copies of the images were generated by scaling the dimension from 0.1 to 1 in steps of 0.1. The fusion algorithm was then applied to all the images, the original images and resized ones. $\lambda$ was varied according to (12) and the widely used image quality measure of Tone-Mapped Quality Index TMQI Ref24 was computed for every $\lambda$ ,

where $SIZE$ denotes the diameter of the input image and $s_{\lambda}$ is a scaling factor. The average TMQI over all the images for different scale values is shown in Fig. 3. This figure indicates that the scale value of 2 generated the best TMQI. Hence, $\lambda$ was set to $\lambda=SIZE/4$ for the subsequent experimentations.

Furthermore, in order to gain computational efficiency, another experiment was conducted by applying the guided filtering operation to the down-sampled binary maps (instead of the original size) and TMQI was recomputed. The binary cluster maps were scaled to a smaller size, after applying the guided filter, the filtering outcome was rescaled to the original size. All the other steps of the exposure fusion were kept the same. The dimensions of the binary cluster maps were lowered as follows:

where $SCALE$ denotes the down-sampling ratio and $s_{g}$ represents a scaling factor. The average TMQI over all the images for different $s_{g}$ values is shown in Fig. 4. As can be seen from this figure, TMQI dropped monotonically as $s_{g}$ was increased. However, since the decrease in TMQI was more as $s_{g}$ varied from 3 to 4 compared to the previous decreases, $s_{g}=3$ was chosen in the smartphone app. In other words, for computational efficiency reasons on a smartphone platform, the binary clustered maps were down-sampled with a factor of 8 before applying the guided filter.

This section provides objective and subjective comparisons with several representative existing methods. The method introduced in this paper, named OB, together with iCam Ref10 , Weighted Least Squares (WLS) Ref11 , Guided Filtering (GF) Ref13 , and Exposure Fusion (EF) Ref15 methods were applied to the image sequences. Sample image sequences as well as the corresponding fusion results are displayed in Fig. 5. As can be seen from Figs. 5 (c) and (d), the fusion results using the EF and OB methods were found to be better compared to the other methods. However, the light region in Fig. 5 (c) appeared over-exposed and the wall region above the light in Fig. 5 (d) lost some color information when using the EF method whereas these effects did not occur when using the OB method.

The TMQI measure averaged over all the 10 scenes are provided in Fig. 6. As can be seen from this figure, the performance of the EF and OB methods were found comparable and higher than those of the other methods.

The processing times of the aforementioned methods were measured by applying them to the images in the dataset resized to 1 megapixels. These times were found to be 2.67, 22.61, 1.07, 2.37, and 0.99 seconds for the iCam, WLS, GF, EF, and OB methods, respectively, when using a 2.66 GHz machine with all the methods coded in Matlab. It is also worth mentioning that the processing times for the iCam, WLS, and GF methods, which all fall under the tonal mapping category of HDR, only reflect the time associated with the tonal mapping and do not include the time associated with the scene irradiance map generation. Since the TMQI measure for the iCam, WLS, and GF methods was found to be lower compared to those for the EF and OB methods, the processing time comparison was limited to the EF and OB methods which is stated below.
The computational complexities of the EF and OB methods are $C_{1}O\big{(}N^{2}log(N)\big{)}$ and $C_{2}O\big{(}N^{2}log(N)\big{)}$, respectively, where $C_{1}$ and $C_{2}$ denote constants and $N$ denotes the image size (height or width). Although the two methods have the same big $O$, the constants $C_{1}$ and $C_{2}$ have a noticeable impact on the processing time as $N$ increases. Fig. 7 shows the average processing times of the EF and OB methods over the resized versions of the captured scenes in terms of the image size expressed in pixels. The processing times were measured for $N$=64, 128, 256, 512, 1024, 2048. In addition, $C_{1}N^{2}log(N)$ and $C_{2}N^{2}log(N)$ curves were fitted to the processing times thus generating the values of 8.47e-8 and 2.25e-7 for the constants $C_{1}$ and $C_{2}$, respectively. As can be seen from this figure, the two dashed lines fit the processing times closely. As a result, the processing time of the OB method is on average $C_{2}/C_{1}=0.37$ that of the EF method, that is the OB method is about 60% faster than the EF method. More computational gain can be achieved noting that the Fourier transforms of $\nabla^{h}$ and $\nabla^{v}$ in (2) are input image-independent and can be computed offline and stored in memory.

Although the introduced solution is intended for use in scenes that do not change during the time that different exposure images are captured, the exposure selection and capturing parts are designed and implemented in such a way that there is minimal delay in capturing the three exposure images. Thus, in practice, one experiences little or no misalignments between the three exposure images because of possible changes that may occur in a scene. Furthermore, the exposure fusion part is tolerant to slight misalignments since it extracts the fusion information locally from the auto-exposed image rather than the other exposure images. An example of an HDR image in the presence of misalignments that are generated by the EF and OB methods is shown in Fig. 8. As can be seen from this figure, the ghost effect is noticeable around the cables and the edges of the monitor and the mouse in the image generated by the EF method while no ghost artifacts are present in the image generated by the OB method. It is worth noting that it is possible to use computationally efficient image registration techniques, e.g. Ref4 , Ref25 , to align different exposure images before performing fusion for situations where severe misalignments may occur.

In this subsection, the actual smartphone implementation outcome of the OB method is reported. This HDR app can be downloaded from http://www.utdallas.edu/
$\sim$kehtar/HDRApp.apk and run on an Android smartphone. A sample exposure selection and fusion outcome obtained by running the developed app on a smartphone is shown in Fig. 9 together with the corresponding under-exposed, normal-exposed, and over-exposed images.

The average processing time for images of size $768\times 1024$ pixels on a modern smartphone was found to be about 1.8s. Note that the intention here has been to show the practicality of running the introduced solution on smartphones. It should be realized that it is possible to achieve lower processing times as the app code running on the smartphone does not utilize vector processing on the NEON coprocessor present on nearly all modern smartphones.