Neural Augmentation Based Panoramic High Dynamic Range Stitching

The accuracy of the physics-driven approach is crucial for the neural augmentation. Therefore, a novel histogram bins-based algorithm is proposed to estimate the IMFs [13]. The physics-driven generation of panoramic LDR images with different exposures builds upon the new IMF estimation algorithm.

The IMFs of the images $Z_{i}$ and $Z_{j}$ are computed by using their OFOV. The proposed method is different from the algorithms in [11] in the sense that the CRFs are required to be known in advance and the IMFs are computed from the CRFs in [11]. Therefore, the generation of panoramic LDR images with different exposures from the differently exposed LDR images with OFOVs is naturally a good case for the application of neural augmentation [11]. With some abuse of notation, their OFOVs are also denoted as $Z_{i}$ and $Z_{j}$.

Let $p$ be a pixel position. $\Omega_{i}(z)$ is defined as $\{p|Z_{i}(p)=z\}$, and $|\Omega_{i}(z)|$$(\doteq H_{i}(z))$ is the number of pixels in the set $\Omega_{i}(z)$. The cumulative histogram $C_{i}(z)$ of the image $Z_{i}$ is then computed as $\sum_{k=0}^{z}H_{i}(k)$. A function $\psi^{i\to j}(z)$ can be defined from the non-decreasing property of the IMFs as [13]

where $\psi^{i\to j}(-1)$ is 0. With the function $\psi^{i\to j}(z)$, the correspondence between the histogram bins of the images $Z_{i}$ and $Z_{j}$ is first built up as follows: the first and last bins in the image $Z_{j}$ corresponding to the bin $\Omega_{i}(z)$ are $\Omega_{j}(\psi^{i\to j}(z-1))$ and $\Omega_{j}(\psi^{i\to j}(z))$, respectively. The cardinalities of sub-bins (or bins) in the image $Z_{j}$ corresponding to the bin $\Omega_{i}(z)$ is denoted by $\hat{H}^{i\to j}(k,z)$. $\hat{H}^{i\to j}(k,z)$ is defined as $H_{i}(z)$ if $\psi^{i\to j}(z-1)$ is equal to $\psi^{i\to j}(z)$. Otherwise, it is defined as

The new IMF $\Lambda^{i\to j}(z)$ is then computed by using the correspondance at the histogram-bin level as

Obviously, our new IMF $\Lambda^{i\to j}(z)$ is a weighted average of all the sub-bins (or bins) in the image $Z_{j}$ corresponding to the bin $H_{i}(z)$. Therefore, our new IMF estimation algorithm is named as the WHA. Unlike the pixel-level correspondence in [19], the correspondence at the histogram-bin-level is robust to moving objects and camera movements. On the other hand, the high-frequency information will be lost when the IMFs are used to synthesize images.

The IMFs $\Lambda^{1\to 2}(z)$ and $\Lambda^{2\to 1}(z)$ are computed from the sub-images $Z_{1,2}^{o}$ and $Z_{2,1}^{o}$. Similarly, the IMFs $\Lambda^{2\to 3}(z)$ and $\Lambda^{3\to 2}(z)$ are obtained from the sub-images $Z_{2,3}^{o}$ and $Z_{3,2}^{o}$. Subsequently, the IMFs $\Lambda^{1\to 3}(z)$ and $\Lambda^{3\to 1}(z)$ can be computed. As shown in the section V-C, the accuracy of the IMFs plays an important role in the proposed neural augmentation framework. Thus, the resolutions of the sub-images $Z_{1,2}^{o}$, $Z_{2,1}^{o}$, $Z_{2,3}^{o}$ and $Z_{3,2}^{o}$ are required to be sufficiently large such that the IMFs $\Lambda^{1\to 2}(z)$, $\Lambda^{2\to 1}(z)$, $\Lambda^{2\to 3}(z)$ and $\Lambda^{3\to 2}(z)$ are highly accurate. Differently exposed images of each view are generated as

Even though the proposed physics-driven approach is independent of training data, its representation capability is usually a little limited. As a result, the quality of images $Z_{m}^{i\to j}$ $(1\leq i\neq j\leq 3)$ is poor and all the images need to be refined via a data-driven approach as in the next subsection.

The IMFs are physical models on the correlation among different exposed images. Since their representation capabilities are usually limited, there are unmodelled information. The unmodelled information $(Z_{T}^{i\to j}-Z_{m}^{i\to j})(1\leq i\neq j\leq 3)$ can be further represented by the data-driven approach $f_{d}^{i\to j}(\cdot)$.

The unmodelled information can be easily compensated by the $f_{d}^{i\to j}(\cdot)$ because it is sparser. Since the DNNs cannot learn the high-frequency information well [21], it is crucial for the data-driven approach to preserve the high-frequency information. A new MEAN is proposed in Figs. 4 and 5 to achieve these objectives. The key components of the proposed MEAN are EAGB, SRRGs, and multi-scale feature refinement. The SRRG is designed by adding short-skip connections to the RRG in [30] in order to pass shallow layers’ information into deep layers. As such, the high-frequency information has a potential to be well preserved. The proposed SRRG can propagate more informative features among different layers. The multi-scale feature refinement structure can further help preserve the high-frequency information.

The proposed EAGB consists of several compensations blocks (CBs) with each of them including two $3\times 3$ convolutional layers and one activation layer. The EAGB guides the SRRG to restore the saturated areas more effectively by properly binary masks $M_{l}^{i\to j}$$(1\leq i\neq j\leq 3)$. The masks $M_{l}^{1\to i}$$(i=2,3)$ are defined for the image $Z_{1,l}$ as

It should be pointed out that when bright images are mapped to dark images, their binary masks are different. For example, the masks $M_{l}^{i\to 1}$$(i=2,3)$ are

Obviously, the proposed masks $M_{l}^{i\to j}$$(1\leq i\neq j\leq 3)$ are different from the soft mask. Besides the structure of the proposed MEAN, loss functions are also important for the data-driven approach. It is very important to guarantee the fidelity of the reconstructed image $Z^{i\to j}$ with to the ground truth image $Z_{T}^{i\to j}$. A reconstruction loss function $L_{r}$ is

Color distortion is a possible issue for pixels in the underexposed and overexposed regions of $Z^{i\to j}$ [31]. To remove the color distortion, another loss function $L_{c}$ is given as

where $\angle(Z^{i\to j}(p),Z_{T}^{i\to j}(p))$ is the angle between $Z^{i\to j}(p)$ and $Z_{T}^{i\to j}(p)$ [32].

One more feature-wise loss function $L_{f}$ is computed as

where $W_{k,l}$ and $H_{k,l}$ are the dimensions of the respective feature maps, $\phi_{k,l}(\cdot)$ denotes the feature map which is provided by the $l$-th convolution before the $k$-th pooling layer in the VGG network.

Our overall loss function $L$ is calculated as

where $w_{c}$ and $w_{f}$ are two hyper parameters. Since the reconstruction loss function $L_{r}$ is much more important than the other two items, $w_{c}$ and $w_{f}$ are empirically selected as $0.01$’s, respectively [11].

The numbers of SRRG and EAGB in the proposed MEAN are set as $5$ and $4$, respectively. The MEAN is trained using an Adam optimizer. The batch size is 8. The learning rate is initially set to $10^{-5}$, then it is decreased through a cosine annealing method. Our MEAN is trained for 1000 epochs.

Our neural augmentation framework is convergent faster than the corresponding data-driven approach. Note than the data-driven approach only generates a refined term for the physics-driven approach. As such, the amount of training data is reduced by our new framework to achieve an accuracy [29]. Moreover, the physics-driven approach is independent of the training data, our new framework can be applied to to obtain a more general representation of the data than the corresponding data-driven approach.

Once all the images $Z^{i\to j}(1\leq i\neq j\leq 3)$ are generated by the proposed neural augmentation framework, the three panoramic LDR images with different exposures are first produced as in the equations (10)-(22), and will then be combined together to generate an information enriched panoramic LDR image in the next section.

The accuracy and computational cost of the physics-driven approach are important for our neural augmentation. In this subsection, the proposed WHA is compared with existing IMF estimation algorithms including HHM [7], PCRF [12], TG [14], AM [15], 3MS [16], GPS [17], MV [18], GC [19], and CHM [20] by using the VETHDR-Nikon dataset [11]. The PSNR, SSIM, FSIM, iCID and running time are chosen as the evaluation metrics. All the algorithms are tested by using the Matlab R2019a on a laptop with Intel Core i7-9750H CPU 2. 59GHz and 32. 0 GB memory.

In the VETHDR-Nikon dataset [11], each pair of differently exposed LDR images is well-aligned. In order to simulate cases with camera movements, those images in the data-set are cropped according to the following two equations:

where $W$ and $H$ are the weight and height of the images in the dataset. As demonstrated in Table I, the proposed WHA achieves the best performance from all the PSNR, SSIM, FSIM, and iCID points of view, and its speed is slightly slower than the CHM [20].

The proposed WHA can be applied to investigate ghost removal of differently exposed LDR images. There are many ghost removal algorithms for differently exposed images [31]. The proposed WHA is adopted to improve the ghost removal algorithm in [31]. One of the input images is selected as the reference image. Pixels in all other images are divided into inconsistent and consistent pixels. All the inconsistent pixels are corrected by using the correlation between the reference image and the corresponding image. The IMFs which are estimated by the CHM in [20] are used in [31] to detect all the inconsistent pixels and correct them. The CHM in [20] is replaced by the proposed WHA. As shown in Fig. 6(b), there is serious color distortion in the wall by using the CHM in [20]. Fortunately, the color distortion is significantly reduced by our WHA. It should be pointed out that the WHA was recently applied in [34] to design an optical flow estimation algorithm for differently exposed LDR images in LDR domain. It was shown in [34] that the WHA outperforms the gamma correction which was widely utilized by deep-learning based ghost removal algorithms.

Four important components of the proposed neural augmentation frameworks are the physics-driven approach, the EAGB, the short-skip connection, and the multi-scale refinement. Their performances are evaluated in this subsection.

With vs without the physics-driven approach: Our neural augmentation framework and the data-driven approach are compared with each other. The data-driven approach is almost identical to the data-driven approach of our neural augmentation framework, except for disabling the EAGB. This is because that the initial synthetic images is required by the EAGB. Table II demonstrates that our framework can enhance the SSIM, PSNR, and FSIM. Additionally, Fig. 7 illustrates that our new framework converges faster than the data-driven approach. It is noteworthy that case 1 in Table II is the physics-driven approach. Clearly, the data-driven approach can significantly enhance the physics-driven approach.

With vs without the EAGB: As demonstrated in Fig. 7, the EAGB can be applied to improve the stability of the proposed framework, and it can also achieve higher PSNR and SSIM values at different epochs.

With vs without the short-skip-connection: As shown in Fig. 5, our proposed SRRG can transmit information from shallow layers to deep layers. The PSNR and SSIM at different epochs are depicted in Fig. 7, indicating that the mode with the short-skip connection can obtain superior results.

Multi-scale vs single-scale: The proposed multi-scale framework and another mode with a single-scale structure are compared with each other. The multi-scale structure achieves higher SSIM, PSNR, and FSIM values.

It was shown in [29] that the physics-driven approach can be improved by the data-driven approach in the neural augmentation. However, one important issue was ignore in [29], i.e., is the accuracy of the physics-driven approach important for the neural augmentation? Experimental results are given to address the issue in this subsection. More specifically, highly accurate IMFs are compared with lowly accurate IMFs.

Two different overlapping area, denoted as R1 ($480\times 200$) and R2 ($480\times 100$), are compared to answer the question. The IMFs of the former are more accurate than those of the latter. As depicted in Fig. 7, the former converges faster than the latter. The PSNR and SSIM of R1 are also increased. Therefore, the accuracy of the physics-driven approach is important for the proposed neural augmentation framework. To reduce the inference complexity of the neural augmentation, the physics-driven approach is also required to be simple.

It is also demonstrated in Fig. 7 that data-driven approach is typically locally stable. Thus, a neural augmentation on top of an accurate physics-driven approach usually outperforms the corresponding data-driven approach.

The proposed panoramic HDR stitching algorithm is compared with four state-of-the-art (SOTA) panoramic stitching algorithms in [6, 35, 7, 8].

As illustrated in Fig. 8, serious brightness inconsistencies are evident in the stitched images produced by the algorithms in [6, 35, 8]. These inconsistencies are overcome by the algorithms in [7] and our algorithm. Information in the darkest regions of the first set and information in the brightest regions of the second set are preserved better using our algorithm compared to the algorithms in [6, 35, 8, 7]. Additionally, as highlighted by the red boxes, the algorithm in [7] produces color distortions. The MEF-SSIM metric in [36] is also employed to compare the all these. The MEF-SSIM is computed by using the ground-truth images of three panoramic LDR images with different exposures and a panoramic LDR image stitched by one of these algorithms. As shown in Table III, our algorithm noticeably outperforms those in [6, 35, 7, 8] from the MEF-SSIM perspective. Our algorithm (45) extract real high-frequency information from each set of images and can further enlarge the average MEF-SSIM. Clearly, the HDR stitching is highly demanded for real-world HDR scenes.

The proposed local stitching algorithm (45) is evaluated in this subsection. As shown by enlarged parts in Fig. 9, high-frequency components such as sharp edges, textures, fine details, etc are indeed preserved better by the local one. Therefore, the local algorithm can indeed be applied to achieve instance adaptation. It should be mentioned that different users have different preferences on the sharpness of the enhanced image $\hat{Z}$. This can be achieved via a weighted combination $(\nu\tilde{Z}_{s}+(1-\nu)\hat{Z})$ with $\nu$ in the interval $[0,1]$. An interactive mode can be provided by allowing tuning the $\mu$ on-line.

A new panoramic stitching system is proposed in this subsection. Instead of capturing a series of images by existing panoramic stitching systems, multiple differently exposed images are captured with varying orientations. For simplicity, the case of two differently exposed images is studied in this subsection. The new system can be easily extended to the case of three different exposed images. The differences between the orientations of these two images are about 15 degrees as shown in Fig. 14. Since the exposures of the two images are different, the algorithm in [5] cannot be applied to align these images directly. The right images is first mapped to the brightness of the left one by using the IMFs between them. They are then aligned by simultaneously using homography transformations and thin-plate spline transformations. The homography transformation provides a global linear transformation, while the thin-plate spline transformation allows local nonlinear deformation. These two images are finally stitched by using the proposed neural augmentation based algorithm. Two panoramic LDR images with different exposures are produced by using the same set of homography transformation and thin-plate spline transformations. They are fused to generate the final image as shown in Fig. 10 (d).

Such a new panoramic HDR stitching system is compared with two conventional panoramic LDR stitching systems. One is from a smart phone and the other is the algorithm in [5]. The exposure times of those images are supposed to be almost the same for the conventional panoramic LDR stitching systems. As demonstrated in Fig. 10, the images produced by the new panoramic HDR stitching system is much better than those generated by the conventional panoramic LDR stitching systems. The panoramic LDR image captured with a mobile phone exhibits noticeable distortion in the bridge deck, as highlighted by these two red boxes. The panoramic LDR image generated using [5] includes visible seams, and has three different brightnesses in different regions. These artifacts are removed by using the proposed panoramic HDR stitching algorithm.