LTCF-Net: A Transformer-Enhanced Dual-Channel Fourier Framework for Low-Light Image Restoration

The LAB and YUV color spaces offer significant advantages over the traditional RGB color space, particularly in terms of separating light information. This separation facilitates independent manipulation of the light and color channels, enhancing the process’s flexibility and effectiveness. The LAB color space, grounded in human visual perception, allows for precise adjustments as demonstrated by the standard LAB conversion formula presented in Eq.4.

$\displaystyle\begin{array}[]{r}L^{*}=116\times f\left(\frac{Y}{Y_{n}}\right)-16\\ a^{*}=500\left[f\left(\frac{X}{X_{n}}\right)-f\left(\frac{Y}{Y_{n}}\right)\right]\\ b^{*}=200\left[f\left(\frac{Y}{Y_{n}}\right)-f\left(\frac{Z}{Z_{n}}\right)\right]\end{array}$

(4)

Meanwhile, the YUV color space is particularly beneficial for low-light image enhancement due to its ability to independently adjust luminance (Y) — critical for enhancing visibility — without affecting the chrominance channels (U and V). The formulation of the YUV color space is specified in Eq. 5.

$$\begin{array}[]{r}Y=0.299R+0.587G+0.114B\\ U=-0.14713R-0.28886G+0.436B\\ V=0.615R-0.51499G-0.10001B\end{array}$$

(5)

The MHSA Block, inspired by transformer architecture, begins by transforming the input feature $\mathbf{F}_{in}\in\mathbb{R}^{H\times W\times C}$ into a reshaped format $\mathbf{X}\in\mathbb{R}^{HW\times C}$. This reshaped feature is then partitioned into multiple ‘heads’ as $\mathbf{X}=\left[\mathbf{X}_{1},\mathbf{X}_{2},\cdots,\mathbf{X}_{k}\right]$.

Each segment, $\mathbf{X}_{i}\in\mathbb{R}^{HW\times d_{k}}$, where $d_{k}=\frac{c}{k}$ and $i=1,2,…,k$, undergoes processing by three bias-free fully connected layers ($f_{c}$), which project $\mathbf{X}_{i}$ into the query $\mathbf{Q}_{i}$, key $\mathbf{K}_{i}$, and value $\mathbf{V}_{i}$ components, which can be defined by $\mathbf{Q}_{i}=\mathbf{X}_{i}\mathbf{W}_{\mathbf{Q}_{i}},\mathbf{K}_{i}=\mathbf{X}_{i}\mathbf{W}_{\mathbf{K}_{i}},\mathbf{V}_{i}=\mathbf{X}_{i}\mathbf{W}_{\mathbf{V}_{i}}$.

The learnable parameters of the fully connected layers are represented by $\mathbf{W}_{\mathbf{Q}i}$, $\mathbf{W}_{\mathbf{K}i}$, and $\mathbf{W}_{\mathbf{V}_{i}}\in\mathbb{R}^{d_{k}\times d_{k}}$. This structure allows the model to adaptively respond to varying lighting conditions across different image regions, enhancing areas that are typically more challenging due to darkness. Each attention head functions independently, employing the self-attention mechanism as defined below:

$$\text{Attention}(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i})=\text{Softmax}\left(\frac{\mathbf{Q}_{i}\mathbf{K}_{i}^{\mathrm{T}}}{\sqrt{d_{k}}}\right)\times\mathbf{V}_{i}$$

(6)

The outputs from all attention heads are concatenated and integrated using a positional encoder, then reshaped back to the original input dimensions, resulting in the final output feature $\mathbf{F}_{\mathrm{out}}\in\mathbb{R}^{H\times W\times C}$.

The CD Block leverages a four-scale U-shaped network [32], incorporating the MHSA mechanism introduced above at the network bottleneck. This integration of convolutional and attention-based methodologies allows for robust feature extraction and effective denoising. The module consists of multiple convolutional layers characterized by two types of strides and includes skip connections to enhance detail retrieval and noise reduction.

This block processes color information from two distinct color spaces, represented by $R_{i}$, where $i=4$ corresponds to the color space components $A$, $B$, $U$, and $V$. The process initiates with these color components being processed through a $3\times 3$ convolutional layer with a stride of one to extract preliminary features, denoted by $F^{(0)}=\mathrm{Conv}_{3\times 3}^{s=1}(R)$. The signal then sequentially traverses three $3\times 3$ convolutional layers each with a stride of two, progressively capturing multi-scale features. For each convolution layer $k=1,2,3$, the operation results in $F^{(k)}=\mathrm{Conv}_{3\times 3}^{s=2}(F^{(k-1)})$, reducing the spatial dimensions of the feature map by half with each convolution, transitioning from $F^{(1)}\in\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times C}$ to $F^{(3)}\in\mathbb{R}^{\frac{H}{8}\times\frac{W}{8}\times C}$.

At the network bottleneck, minimum-scale feature mapping captures global dependencies effectively, utilizing the MHSA Block. Subsequent stages involve up-sampling, matched in scale to the prior down-sampling phases. The initial up-sampling step, employing a deconvolution with a stride of two, produces $G^{(1)}=\mathrm{Deconv}_{3\times 3}^{s=2}(F^{(3)})$. This up-sampling process is repeated twice to restore the feature map to its original dimensions, $H\times W$. The final output is then refined through two additional convolutional layers followed by a Tanh activation function to maintain color fidelity. The ultimate output, $\mathbf{F}_{\mathrm{out}}$, restores the feature dimensions to $\mathbf{F}_{\mathrm{out}}\in\mathbb{R}_{i}^{H\times W\times C}$.

The MSEF Block is engineered to integrate multiple feature enhancement mechanisms [20], specifically designed to enhance the model’s capability to accurately reconstruct and refine details in images by capturing both global and local features associated with illumination and color information. Initially, the input feature is subjected to Layer Normalization to stabilize its mean and variance. Subsequently, the normalized features are processed through the Squeeze-and-Excitation Block (SEBlock), which employs a two-stage recalibration of feature importance: squeezing to reduce redundancy and excitation to emphasize informative features.

Within the SEBlock, features $\mathbf{F}_{in}$ initially pass through Global Average Pooling (GAP) to form a global descriptor for each channel, capturing essential contextual information that is broadly representative of the entire image. This descriptor $\mathbf{D}_{\mathrm{re}}$ is then refined through a fully connected layer equipped with a ReLU activation function, as outlined in Eq. 7, enhancing significant features while filtering out less relevant data. The process continues with another fully connected layer featuring a Tanh activation function, which expands the features back to their original dimensions to achieve optimal recalibration:

$$\begin{array}[]{r}\mathbf{D}_{\mathrm{re}}=\operatorname{ReLU}\left(\mathbf{W}_{1}\cdot\operatorname{GP}\left(\operatorname{LN}\left(\mathbf{F}_{\text{in }}\right)\right)\right)\\ \mathbf{D}_{\mathrm{ex}}=\operatorname{Tanh}\left(\mathbf{W}_{2}\cdot\mathbf{D}_{\mathrm{re}}\right)\cdot\operatorname{LN}\left(\mathbf{F}_{\text{in }}\right)\end{array}$$

(7)

This process not only emphasizes and restores key features but also enhances the model’s ability to detect variations in color and illumination, producing images that appear more natural. This is particularly vital for low-light image enhancement, where maintaining accurate color and detail is crucial. Finally, residual connections also help preserve the original input features, ensuring stability during training and preventing the loss of important details.

The Fourier Branch Processing Block is crucial for enhancing and restoring image brightness by manipulating the luminance channel in the frequency domain through the application of Fourier Transform techniques [3]. This approach allows for precise differentiation and manipulation of high-frequency and low-frequency components, thereby significantly improving the details in brightness across the image.

Initially, the luminance channel undergoes transformation into its frequency-domain representation, with the real and imaginary parts processed separately to refine frequency-domain features. A convolutional layer reduces the number of channels to 16, streamlining features while preserving essential details. This is followed by a Leaky ReLU activation function that captures finer details and mitigates the issue of dying ReLUs in areas of low brightness. Subsequently, features are further refined through an additional convolutional layer that maintains the channel count, ensuring that critical nuances are preserved. A final convolutional layer re-expands the features to match the original input dimensions. This restoration is crucial for aligning the enhanced features with the original image structure, facilitating seamless integration back into the overall image processing workflow. Detailed experimental validations of this process are presented in Section 3.4.

In this section, we introduce our loss function framework, dividing six loss variables into two main categories: pixel-level and perceptual-level losses, each targeting different aspects of image quality enhancement.

Specifically, $\mathcal{L}_{\mathrm{S1}}$ is a smooth L1 loss function, which is defined for two parameters: the predicted image ($y_{\mathrm{pred}}$) and the ground truth image ($y_{\mathrm{true}}$). This loss function has been proven effective in Fast R-CNN because it is robust to outliers and can provide stable gradients during the training process. Its formula is shown in Eq. 8

$\displaystyle\mathcal{L}_{\text{S1}}(y_{\mathrm{true}},y_{\mathrm{pred}})$

$\displaystyle=\sum_{i\in\{x,y,w,h\}}\text{smooth}_{L_{1}}(y_{\mathrm{true}}^{i}-y_{\mathrm{pred}}^{i})$

(8)

Where the function $smooth_{L_{1}(x)}$is defined as:

$$\begin{array}[]{r}smooth_{L_{1}(x)}=\begin{cases}0.5x^{2},&\text{if }|x|<1\\ |x|-0.5,&\text{otherwise}\end{cases}\end{array}$$

(9)

$\mathcal{L}_{\mathrm{PSNR}}$ measures and optimizes image quality by assessing the mean squared error between ($y_{\text{pred}}$) and ($y_{\text{true}}$). It is designed to enhance pixel accuracy, as detailed in Eq. 10.

$\displaystyle\mathcal{L}_{\text{PSNR}}=40.0-\sum_{i=1}^{n}20\cdot\log_{10}\left(\frac{1}{\sqrt{\text{MSE}}}\right)$

(10)

The $\mathcal{L}_{\mathrm{Color}}$ ensures color fidelity by minimizing color discrepancies between the predicted and reference images, calculated as shown in Eq. 11.

$\displaystyle\mathcal{L}_{\text{Color}}=\sum_{i=1}^{n}\left|\text{mean}(y_{\text{true}}^{i})-\text{mean}(y_{\text{pred}}^{i})\right|$

(11)

The histogram loss function aligns the global statistical features such as brightness and contrast by comparing histograms, enhancing overall image consistency as illustrated in Eq. 12.

$\displaystyle\mathcal{L}_{\text{Hist}}=\sum_{i=1}^{n}\left|\frac{1}{N}\sum_{n=1}^{N}H_{\text{true}}^{i,n}-\frac{1}{N}\sum_{n=1}^{N}H_{\text{pred}}^{i,n}\right|$

(12)

Where $H_{\text{true}}^{i,n}$ and $H_{\text{pred}}^{i,n}$ are the histogram values for the ground truth and predicted images, respectively, at bin $n$. This loss encourages the model to generate predictions whose color distributions match those of the ground truth more closely.

$\mathcal{L}_{\mathrm{per}}$ utilizes features extracted from a VGG19 network, this perceptual loss measures differences that affect the perceptual quality between the predicted and ground truth images. Its formula is shown in Eq. 13.

$$\begin{array}[]{r}\mathcal{L}_{\text{per}}=\frac{1}{C_{j}H_{j}W_{j}}\left\|\phi_{j}(y_{true})-\phi_{j}(y_{pred})\right\|_{2}^{2}\end{array}$$

(13)

Additionally, $\mathcal{L}_{\mathrm{SSIM}}$ is based on multi-scale structural similarity, this loss ensures structural consistency across different scales, thereby enhancing the perceptual similarity of the generated image. Its formula is shown in Eq. 14. Where $y$ is the original image and $\hat{y}$ is the target image.

$$\mathcal{L}_{\text{SSIM}}(y,\hat{y})=1-\frac{(2\mu_{y}\mu_{\hat{y}}+C_{1})(2\sigma_{y\hat{y}}+C_{2})}{(\mu_{y}^{2}+\mu_{\hat{y}}^{2}+C_{1})(\sigma_{y}^{2}+\sigma_{\hat{y}}^{2}+C_{2})}$$

(14)

As shown in Eq 15, the final loss function is a combined loss between pixel level losses and perceptual level losses, where $\alpha_{1}$ to $\alpha_{5}$ are hyperparameters for each loss component. This final loss function includes provides a comprehensive evaluation metric that optimizes various aspects of image quality, ultimately leading to a more realistic and high-quality output.

	$\displaystyle\mathcal{L}_{Pixel}$	$\displaystyle=\mathcal{L}_{\mathrm{S1}}+\alpha_{1}\mathcal{L}_{\mathrm{PSNR}}+\alpha_{2}\mathcal{L}_{\mathrm{Color}}+\alpha_{3}\mathcal{L}_{\mathrm{Hist}}$		(15)
	$\displaystyle\mathcal{L}_{Perc}$	$\displaystyle=\alpha_{4}\mathcal{L}_{\mathrm{Per}}+\alpha_{5}\mathcal{L}_{\mathrm{SSIM}}$
	$\displaystyle\mathcal{L}_{Total}$	$\displaystyle=\mathcal{L}_{Pixel}+\mathcal{L}_{Perc}$

We eveluate our method on LOL(v1 [42] and v2-real [49]), SID [8], SMID [7], SDSD [39], and FiveK [5] datasets.

In addition to the above eight benchmarks, we also tested our method on five datasets: LIME [18], NPE [40], MEF [31], DICM [26], and VV [36], which do not have ground truth annotations.

Additionally, as depicted in Fig. 4, models like URetinex [45], KinD++ [51], and DiffLL [22] show noticeable color inaccuracies. Moreover, Retinexformer [6] is unable to recover text on a sticker, underscoring its limitations. These observations confirm that our method outperforms others in restoring contrast and detail, particularly in the SID and SMID datasets. Including Fig. 5, in SDSD-indoor and outdoor datasets, our method can restore many details.

Additional, we evaluate the impact of various enhancement algorithms on object detection using the FiveK dataset [5]. We used YOLO-v4-tiny [2] as the detector, and then compare the object detection performance on raw images and the images enhanced by different low-light enhancement methods.

Table 2 shows the average precision (AP) scores. Our model achieves the highest mean AP of 37.7, surpassing the leading self-monitoring method, URetinex, by 1.5 AP. Notably, our approach outperforms in seven object categories: Bicycle, Boat, Bus, Dog, Motor, People, and Table.

Fig. 7 illustrates the qualitative detection results in a low-light scene (left) and after LTCF-Net enhancement (right). The enhanced image enables the detector to identify more objects with higher accuracy, demonstrating the crucial role of our low-light enhancement method in improving object detection performance.

To validate the effectiveness of each component within the LTCF-Net and its contribution to training optimization, we conducted detailed ablation experiments on the SID dataset, specifically examines the dual color branches, the MSEF module, and the FBP module.