Devignet: High-Resolution Vignetting Removal via a Dual Aggregated Fusion Transformer With Adaptive Channel Expansion

A limited number of studies in the field of traditional vignetting removal has proposed methods that are based on mathematical principles, statistical analysis, and prior knowledge. SIVC (Zheng et al. 2013) utilizes the symmetry properties of semicircular tangential gradients and RG distributions to estimate the optical center and correct vignetting. Goldman et al. assume that the vignetting in the image exhibits radial symmetry around its center (Goldman 2010). RIVC (Lopez-Fuentes, Oliver, and Massanet 2015), addresses vignetting removal through the minimization of the log-intensity entropy.

Traditional methods (Wang et al. 2013; Guo, Li, and Ling 2016; Jobson, Rahman, and Woodell 1997; Cai et al. 2017; Fu et al. 2016, 2015) for Low-Light Image Enhancement often refers to the Retinex theory or histogram equalization.

Recently, the utilization of these learning-based approaches gain traction as a prevalent solution for enhancing low-light images. Several widely recognized datasets are utilized by these learning-based methods. For instance, Wei et al. propose a Low-Light dataset (LOL) (Wei et al. 2018) containing pairs of low/normal-light images. The MIT-Adobe FiveK dataset (Bychkovsky et al. 2011) comprises 5,000 photos that are manually annotated. To ensure the highest quality, the dataset underwent retouching by a team of 5 trained photographers, rendering it well-suited for supervised learning in the context of LLIE. In terms of effective methods, certain technologies have also made contributions. Wang et al. propose Uformer (Wang et al. 2022) employing both local and global dependencies to restore images. The KinD  (Zhang, Zhang, and Guo 2019) method proposed by Zhang et al. can be trained and achieves impressive results without the need for explicitly defining a ground truth dataset. Liu et al. (Liu et al. 2021) present a retinex-based network that is both lightweight and efficient. Guo et al. enables end-to-end training without the need for reference images (Guo et al. 2020). Li et al. (Li, Guo, and Loy 2021) introduce an adaptive LLIE network that can operate under diverse lighting conditions without dependence on paired or unpaired training data. Lim et al. (Lim and Kim 2020) propose a method that can independently recover global illumination and local details from the original input, and gradually merge them in the image space. Jiang  et al. (Jiang et al. 2021) put forward unsupervised learning into the realm of LLIE using GAN.

VigSet employs a variety of three distinct capture devices: Fujifilm X-T4, ONE PLUS 10PRO, and iPhone SE.

In well-lit situations, vignetting is generally not noticeable. Therefore, we use an ND filter to reduce illumination and capture photos with vignetting. The center light, which passes through an ND filter, travels a shorter distance compared to the light at the edges. This difference in distance contributes to the occurrence of vignetting.

Additionally, to ensure that there is no displacement between each pair of photos and to enhance data diversity, it is crucial to avoid dynamic objects such as swaying tree leaves, moving vehicles, and glass reflections on specific objects that are susceptible to motion. Therefore, we select multiple distinct indoor environments for the data collection. To reduce device vibration caused by manual camera shutter presses, we use remote shutter control to capture steps 2 and 4, as illustrated in Figure 2 (c).

Figure 2 (a) provides a preview of our experimental configuration designed for capturing VigSet. The process of collecting data is facilitated by adjusting the white ND value controller located on the camera ND filter, as depicted in Figure 2 (c). During the data collection process, we capture images from real-world scenes utilizing experimental equipment. We use a tripod to stabilize the camera and control unrelated variables. The steps of the data collection process are as follows.

1. 1.

   The camera settings, such as focal length, aperture, exposure, and ISO, are set to fixed values.

2. 2.

   A vignetting-free image is captured with an ND value of 0, as shown in Figure 3 (a)

3. 3.

   Continuously modify the value of the ND filter until noticeable vignetting becomes visible in the image.

4. 4.

   An image with vignetting is captured, as shown in Figure 3 (b)

We exclude images exhibiting obvious motion distortion or lack of focus. Moreover, we conduct group reviews of these photographs to identify and remove any outliers and duplicates.

The Dual Aggregated Fusion Transformer (DAFT) is a neural network designed specifically for handling low-frequency information in images. As the foundational architecture, the Fusion Transformer employs multiple attention mechanisms to capture various points of focus, enabling the model to effectively incorporate both local and global information within its representation. Fusion Transformer significantly boosts the expressive capacity of the Transformer network. Figure 4 illustrates the location of these four fusion transformers, wherein each transformer comprises two modules. Each module contains 1, 2, 3, or 4 Transformer Blocks (Dosovitskiy et al. 2021) from left to right. The output of the first module is passed on to the second module, and the final output is obtained by adding the output of the second module to the output of the first module.

Inspired by (Cun, Pun, and Shi 2020), the Aggregation Node is designed as integration operations on the input features. By aggregating the semantic information extracted by the fusion transformer, a richer and more comprehensive global feature representation can be obtained. This Structure facilitates an improved understanding of the image’s structure, texture, and global properties by the model in low frequency, leading to enhancing vignetting removal.

While the primary cause of vignetting is low-frequency color information, some edge information of the images remains. Therefore, motivated by U-Net (Ronneberger, Fischer, and Brox 2015), we propose a structure that integrates Adaptive Contrast Enhancement Module (ACEM) and Laplacian pyramid reconstruction for optimal vignetting removal results in high-frequency.

ACEM is a lightweight module that does not employ any activation functions, and it has been proven that there will be no decrease in performance. Inspired by (Ba, Kiros, and Hinton 2016), the beginning section of ACEM is LayerNorm, which improves stability and reduces computational overhead. The following convolutional layers capture feature information at varying scales. Inspired by (Chen et al. 2022), the Simplified Channel Attention (SCA) and SimpleGate techniques are utilized to enhance network performance. SimpleGate divides the feature maps into two channel dimensions and multiplies them together, leading to a reduction in computational load and complexity. The formula is as follows:

$$SimpleGate(\mathbf{X},\mathbf{Y})=\mathbf{X\odot Y},$$

(1)

where $\mathbf{X}$ and $\mathbf{Y}$ are feature maps of the same size. Simplified Channel Attention (SCA) can be considered as a streamlined adaptation of Channel Attention (CA). It captures the essence of CA by retaining only two vital components: aggregating global information and enabling channel-wise interaction. SCA can be described as :

$$SCA(\mathbf{X})=\mathbf{X}*Wpool(\mathbf{X}).$$

(2)

$W$ denotes a fully connected layer. $pool$ represents the global average pooling procedure which combines spatial data into channels. $*$ indicates a channel-wise multiplication.

Inspired by (Wang et al. 2023), The Hierarchical Channel Attention Module (HCAM) module is dedicated to the hierarchical fusion of features and the acquisition of learnable correlations across diverse layers. The primary function of HCAM is to compute and apply attention weights to the input feature map, resulting in the refinement and enhancement of vignetting features. Initially, HCAM performs a transformation on the input $I\in R^{H\times W\times 3}$ to yield $R^{N\times H\times W\times C},N=3$, from successive layers. Subsequently, the $Q$ (query), $K$ (key), and $V$ (value) features are extracted using convolutional layers. Subsequently, the $Q$, $K$, and $V$ features are extracted using convolutional layers.

In this module, Inner-Product Attention plays a crucial role in calculating attention scores by computing the dot product between the query and the key. HCAM adaptively fuses features from different hierarchical levels by conducting weighted operations between values and attention scores. The corresponding output feature ${\mathbf{R}}_{\text{out }}$ can be described as:

$${\mathbf{R}}_{\text{out }}=W_{1\times 1}LA({\mathbf{Q}},{\mathbf{K}},{\mathbf{V}})+{\mathbf{R}}_{\text{in }},$$

(3)

where $LA$ represents Layer Attention, $W_{1\times 1}$ indicates Convolution $1\times 1$, $LA$ can be written as:

$$LA(\hat{\mathbf{Q}},\hat{\mathbf{K}},\hat{\mathbf{V}})=\hat{\mathbf{V}}\operatorname{softmax}(\hat{\mathbf{Q}}\hat{\mathbf{K}}/\boldsymbol{\alpha}),$$

(4)

During the training phase, we employ two loss functions, namely Mean Squared Error ($L_{MSE}$) and Structural Similarity Index Measure ($L_{SSIM}$) (Wang et al. 2004), which offer significant advantages in preserving the intricate details of the image. The mathematical expressions for these losses are as follows:

$$L_{total}=L_{MSE}+\lambda*L_{SSIM},$$

(5)

The weight of $L_{SSIM}$ is empirically set to 0.4.

In order to ensure the highest level of fidelity and reliability in our experiments, we have made the deliberate decision to exclusively use the VigSet dataset for vignetting removal. Moreover, we also conduct experiments in the field of LLIE to demonstrate the versatility of DeVigNet in global illuminance adjustment.

In our experiments, we leverage the following datasets:

VigSet: This dataset contains 983 image pairs. We allocate 803 pairs for training purposes and reserve the remaining 180 pairs for testing.

MIT-Adobe FiveK (Bychkovsky et al. 2011) & LOL-v1 (Wei et al. 2018): For these datasets, we adopt the experimental settings delineated in (Wang et al. 2023), ensuring consistency with prior research.

We have performed a comprehensive evaluation of state-of-the-art vignetting removal methods on the VigSet dataset. The compared vignetting methods include SIVC (Zheng et al. 2008) and RIVC (Lopez-Fuentes, Oliver, and Massanet 2015). These methods serve as benchmarks for evaluating the performance of our proposed vignetting removal technique.

Furthermore, we also conduct additional experiments with state-of-the-art LLIE methods. We compare DeVigNet with traditional LLIE methods such as LIME (Guo, Li, and Ling 2016), MSRCR (Jobson, Rahman, and Woodell 1997), and NPE  (Wang et al. 2013). Additionally, we evaluate the performance of learning-based LLIE methods, including WV-SRIE (Fu et al. 2016), PM-SRIE (Fu et al. 2015), Uformer (Wang et al. 2022), KID (Zhang, Zhang, and Guo 2019), DSLR (Lim and Kim 2020), ELGAN (Jiang et al. 2021), JieP (Cai et al. 2017), RUAS (Liu et al. 2021), Zero-DCE (Guo et al. 2020), and Zero-DCE++ (Li, Guo, and Loy 2021).

Qualitative results, as shown in Figure 5 demonstrate that our method exhibits significant superiority over others in the vignetting dataset. Our results display enhanced image clarity and more prominent vignetting removal. Moreover, our method achieves competitiveness on the LLIE dataset as shown in Figure 6.

In terms of quantitative results, by comparing our proposed method with these state-of-the-art vignetting and LLIE methods as displayed in Table 1 and Table 2, we can assess its effectiveness and performance in relation to existing methods. This comprehensive evaluation provides insights into the strengths and weaknesses of different approaches and helps to advance the field of vignetting removal.

Conventional methods heavily depend on assumptions based on physics, which are not consistently accurate. Additionally, current LLIE methods are not well-suited for addressing the issue of vignetting. Therefore, based on the data from Table 2, it is evident that traditional methods and LLIE methods exhibit substantial disparities compared to other methods in multiple metrics, particularly in terms of MAE. Additionally, we conducted supplementary quantitative experiments on a LLIE dataset, as presented in Table 1. The results illustrate the strong performance of our method in LLIE. Besides, with the advancement of current technology, capturing high-resolution images has become an essential routine. DeVigNet stands out as the most efficient approach in achieving superior results in vignetting removal across various resolutions when compared to other existing methods.

We maintain a similar parameter count across all combinations to ensure a fair comparison. DAFT and ACEM are removed respectively using the comment-out method in the experiments. As shown in Table 3, we conduct ablation studies evaluating various components of our model at different image resolutions. These components include varying the depth of the Laplacian pyramid, as well as ablating the ACEM and DAFT modules.

As the depth of the Laplacian pyramid increases, the magnitude of the low-frequency component decreases. This reduction directly impacts the quality of vignetting removal, especially with respect to preserving low-frequency color information.

DAFT plays a pivotal role in capturing global color information within the low-frequency component of an image. Therefore, models lacking DAFT suffer performance degradation due to the loss of global color context.

Meanwhile, ACEM is primarily utilized to retain edge details during image reconstruction, focusing on the high-frequency portion. Removing ACEM causes models to lose textural information in the reconstruction phase, thereby deteriorating performance due to the absence of high-frequency context.

Additionally, we replace DAFT with DHAN (Cun, Pun, and Shi 2020), the results are shown in Table 3. Accordingly, DAFT and ACEM are responsible for the low-frequency and high-frequency components, respectively. While each component contributes individually to improved vignetting removal performance, their combined effect yields optimal results by leveraging complementary global and local context.