L²UWE: A Framework for the Efficient Enhancement of Low-Light Underwater Images Using Local Contrast and Multi-Scale Fusion

Underwater image enhancement. Some early approaches that attempted the enhancement of underwater images include: the color correction scheme of Chambah et al. [14] that aimed for a better detection of fish, the work of Iqbal et al. [27], which focused in adjusting the contrast, saturation and intensity to boost images’ quality, and the method of Hitam et al. [26], where the equalization of the histogram from underwater images is proposed as a means to achieve enhancement. More recently, Ancuti et al. [6] introduced a popular framework that derived two inputs (a color-corrected and a contrast-enhanced version of the degraded image), as well as four weight maps that guaranteed that a fusion of such inputs would have good sharpness, contrast and color distribution.

Multiple underwater image processing methods [15, 47, 5, 8, 38, 19, 10] make use of aerial dehazing techniques, given that the issues that plague hazy images (absorption and scattering) create outputs that are similar to those captured underwater. Fu et al. [20] proposed a framework based on the Retinex model that enhances underwater images by calculating their detail and brightness, as well as performing color correction. Berman et al. [10] used the color attenuation and different models of water types to create a single underwater image enhancer. Cho and Kim [16], inspired by simultaneous location and mapping (SLAM) challenges, derived an underwater-specific degradation model. Drews et al. [19] considered only two color channels when using the dehazing approach of He et al. [24], adapting this method to underwater scenes. Marques et al. [39] introduced a contrast-based approach inspired in the Dark Channel Prior [24] that significantly improved the quality of low-light underwater images.

Aerial images dehazing. Dehazing methods aim for the recovery of the original radiance intensity of a scene and the correction of color shifts caused by scattering and absorption of light by fluctuating particles and water droplets. Initial approaches proposed to address this challenge [35, 42, 36, 30, 44] were limited by the need of multiple images (captured under different weather conditions) as input. He et al. [24] proposed a popular method for single-image dehazing that introduced the Dark Channel and the Dark Channel Prior (DCP), which allow for the estimation of the transmission map and atmospheric lighting of a scene, both important parameters of the dehazing process (as detailed in sub-section 2.2). Comparative results [3, 40, 2, 39] attest to the performance of this method in scenarios where only a single hazy image is available. Numerous underwater-specific enhancement frameworks [39, 19, 13, 15, 21, 32] are based on variations of the DCP. Recently a number of data-driven methods [12, 40, 50, 31, 41, 48] focused on the use of Convolution Neural Networks (CNNs) to train systems capable of efficiently performing the dehazing/image recovering task. However, these systems typically require time-consuming processes of data gathering and curation, hyper-parameter tuning, training, and evaluation.

Low-light image enhancement. Dong et al. [18] observed that dark regions in low-light images are visually similar to haze in their inverted versions. The authors proposed to use the DCP-based dehazing method to remove such haze. Ancuti et al. [4] proposed to use a non-uniform lighting distribution model and the DCP-based dehazing method to generate two inputs (an additional input is the Laplacian of the original image), followed by a multi-scale fusion process. Zhang et al. [49] introduced the maximum reflectance prior, which states that the maximum local intensity in low-light images depends solely on the scene illumination source. This prior is used to estimate the atmospheric lighting model and transmission map of a dehazing process (refer to sub-section 2.2 for details). Guo et al. [22] proposed LIME, where the atmospheric lighting model is also initially constructed by finding the maximum intensity throughout the color channels. The LIME framework refines this initial model by introducing a structure prior that guarantees structural fidelity to the output, as well as smooth textural details. Data-driven approaches were also proposed for the enhancement of low-light images [28, 46, 43, 29]. These methods employ CNNs to extract features from datasets composed of low-light/normal-light pairs of images ([29] requires only the degraded images), and then train single-image low-light enhancement frameworks.

Dark Channel Prior-based dehazing of single images. Equation 1 describes the formation of a hazy image $I$ as the sum of two additive components, direct attenuation and airlight.

where $J$ represents the haze-less version of the image, $x$ is an spatial location, transmission map $t$ indicates the amount of light that reaches the optical system and $A_{\infty}$ is an estimation of the global atmospheric lighting. The first term, direct attenuation $D(x)=J(x)t(x)$, indicates the attenuation suffered by the scene radiance due to the properties of the medium. The second term, airlight $V(x)=A_{\infty}(1-t(x))$, is due to previously scattered light and may result in color shifts on the hazy image. The goal of dehazing efforts is to find the haze-free version of the image ($J$) by determining $A_{\infty}$ and $t$.

He et al. [24] introduced the Dark Channel and the Dark Channel Prior, which can be used to infer atmospheric lighting $A_{\infty}$ and to derive a simplified formula for the calculation of $t$. The dark channel is described in Equation 2.

where $x$ and $y$ represent two pairs of spatial coordinates in the dark channel $J^{dark}$ and in the hazy image $I$ (or any other arbitrary image), respectively. The intensity of each pixel in the single-channel image $J^{dark}$ is the lowest value between the pixels inside patch $\Omega$ in $I^{c}$, where $c\in\{R,G,B\}$. The DCP is an empirical observation stating that pixels from non-sky regions have at least one significantly low intensity across the three color channels. Thus the dark channel is expected to be mostly dark (i.e., intensities closer to 0).

A single three-dimensional global atmospheric lighting vector $A^{c}_{\infty}$ ($c\in\{R,G,B\}$) can be inferred by looking at the 0.1% [24] or 0.2% [39] brightest pixels in the dark channel (which represent the most haze-opaque regions from input), then considering the brightest intensity pixels in these same spatial coordinates from the input image $I$. Ancuti et al. [4] observed that a single global value might not properly represent the illumination of low-light scenes, thus proposing the estimation of local atmospheric light intensities $A^{c}_{L\infty}$ inside patches $\Psi$ following Equation 3.

where $x$, $y$ and $z$ represent, respectively, spatial coordinates in the local atmospheric image, “minimum” image $I_{\min}(z)=\min_{z\in\Omega(y)}(I(z))$, and hazy image $I$. For each color channel $c\in\{R,G,B\}$, the local atmospheric lighting $A^{c}_{L\infty}$ is calculated by first finding $I^{c}_{\min}$, which represents the minimum intensities inside patch $\Omega$ on $I^{c}$, and then calculating the maximum intensities inside a patch $\Psi$ on $I^{c}_{\min}$. By arguing that the influence of lighting sources goes beyond patch $\Omega$, Ancuti et al. [4] used patch $\Psi$ with twice the size of $\Omega$. The DCP is used in [24] to derive Equation 4 for the calculation of the transmission map $t$.

where $A^{c}_{\infty}$ indicates the atmospheric lighting in the range $[0,255]$, resulting in a normalized image ($\frac{I^{c}(y)}{A^{c}_{\infty}}$) ranging from $[0,1]$. The constant $\omega$ $(0\leq\omega\leq 1)$ preserves a portion of the haze, generating more realistic output. Note that for local estimation of atmospheric lighting, $A^{c}_{\infty}$ would be substituted by $A^{c}_{L\infty}$ in Equation 4. The haze-less version of the image, $J(x)$, is obtained using Equation 5 [24].

Since the direct attenuation $D(x)=J(x)t(x)$ can be close to zero when $t(x)\approx 0$, [24] adds the $t_{0}$ term as a lower bound to $t(x)$, effectively preserving small amounts of haze in haze-dense regions of $I$.

Contrast-guided approach for the enhancement of low-light images. Marques et al. [39] observed and addressed three problems that arrive from the use of single-sized patches $\Omega$ throughout the image dehazing process: 1) small patch sizes would result in oversaturation of the radiance from the recovered scene (non-natural colors); 2) large patch sizes better estimate and eliminate haze, but since they consider that the transmission profile (i.e., amount of light that reaches a camera) is constant inside patch $\Omega$, undesired halos might appear around intensity discontinuities and 3) a single patch size is typically not optimal for images of varying scales.

Marques et al. [39] argues that homogeneous regions of an image possess lower contrast, and that the assumption that their transmission profile is approximately the same holds for patches $\Omega$ of varying sizes (in particular, from $3\times 3$ up to $15\times 15$). In these scenarios, the use of larger patch sizes generates darker dark channels (strengthening the DCP) and are, therefore, preferred. For regions with complex content (i.e., intensity changes), [39] uses smaller patch sizes to capture the local transmission profile nuances. To determine the correct patch size for each pixel in image $I$, [39] introduced the contrast code image ($CCI$), whose calculation is summarized in Equation 6.

where $\Omega_{i}(x)\in I$ represents a square patch of size $(2i~{}+~{}1)\times(2i~{}+~{}1)$ centered at the pair of spatial coordinates $x$, $i=\{1,2,...,7\}$, and $\sigma$ represents the standard deviation between intensities inside of patch $\Omega_{i}$ (considering the three color channels). Therefore, the $CCI$ is populated by $i$ (referred to as code $c$ in [39]), which indicates the size of the patch $\Omega$ that generated the smallest $\sigma$ in each pixel location $x$. A variable named $tolerance$ is also introduced to incentivize the use of larger patch sizes by changing the measured values of $\sigma$ for different $i$.

The $CCI$ is then used to calculate the transmission map and dark channel. For a pixel location $x$, one would use patches of size $(2c~{}+~{}1)\times(2c~{}+~{}1)$ (where $c=CCI(x)$) instead of fixed-size patches. This constrast-guided approach significantly mitigates the three aforementioned problems.

Multi-scale fusion for image enhancement. The work of Ancuti et al. [7] proposes to perform dehazing by first calculating two inputs $\mathcal{I}^{k}$ ($k=\{1,2\}$) from the original image: a white-balanced and a contrast-enhanced version of it. Then the authors introduced the now-popular multiscale fusion process, where three weight maps containing important features of each input $\mathcal{I}^{k}$ are calculated: 1) Luminance $\mathcal{W}_{L}^{k}$: responsible for assigning high values to pixels with good visibility, this weight map is computed by observing the deviation between the R, G and B color channels and luminance $L$ (average of pixel intensities at a given location) of the input; 2) Chromaticity $\mathcal{W}_{C}^{k}$: controls the saturation gain on the output image, and can be calculated by measuring the standard deviation across each color channel from the input; 3) Saliency $\mathcal{W}_{S}^{k}$: in order to highlight regions with greater saliency, this weight map is obtained by subtracting a Gaussian-smoothed version of the input by its mean value (method initially proposed by Achanta et al. [1]). Aiming to minimize visual artifacts introduced by the simple combination of the weight maps, [7] uses a multiscale fusion process where a Gaussian pyramid is calculated for the normalized weight map $\bar{\mathcal{W}}^{k}$ (i.e., per-pixel division between the multiplication of all three weights and their sum) of each input, while the inputs $\mathcal{I}^{k}$ themselves are decomposed into a Laplacian pyramid. Given that the number of levels from both pyramids is the same, they can be independently fused using Equation 7.

where $l$ indicates the number of levels (typically 5), $L\{\mathcal{I}\}$ represents the Lapacian of $\mathcal{I}$, and $G\{\bar{\mathcal{W}}\}$ denotes the Gaussian-smoothed version of $\bar{\mathcal{W}}$. The fused result is a sum of the contributions from different layers, after the application of an appropriate upsampling operator.

The authors applied the multi-scale fusion with different strategies in other works, for example by changing the inputs to gamma-corrected and sharpened versions of a white-balanced underwater image [8] or by calculating new weight maps (e.g., local contrast weight map [4]).

Using parts of the dark channel to derive a single global estimate $A_{\infty}$ for underwater images creates overly-bright and washed-out results. The underlying assumption in [39] that the lighting in inverted input images is mostly white is not reasonable for underwater scenes: see the “inversion” step of Figure 1, where the lighting presents non-white colors. Given that the images from the OceanDark dataset [39] are captured using artificial (and often multiple) lighting sources, a single global estimate $A_{\infty}$ can not properly model their non-uniform nature. L²UWE addresses these problems by calculating the atmospheric lighting for each color channel, and by considering the $CCI$ code at each spatial position $x$ when determining local estimates of lighting. This approach is similar to that described in Equation 3 [4], but instead of using a fixed-size patch $\Psi$, we introduce the contrast-guided patch $\Upsilon$ for lighting calculation.

We observed that regions with high contrast (i.e., lower code $c$ in the $CCI$) should have their local lighting component modeled by considering a larger $\Upsilon$, based on the heterogeneous influence that illumination sources place on them. On the other hand, since regions with lower contrast (i.e., higher codes $c$ in the $CCI$) are illuminated in a roughly homogeneous manner, only smaller patches $\Upsilon$ need to be studied to properly model local illumination. We offer a formalization of this reasoning in Equation 8, which specifies the relationship between $CCI$ code $c$ and the size of the lighting square patch $\Upsilon$ used in our local atmospheric lighting model calculation.

where $m$ represents an arbitrary multiplication factor and $c=\{1,2,...,7\}$ represents code $c$ in the $CCI$ at a certain position. Parameter $m$ has a multiplicative effect on the contrast-guided patch, but an offset is also added to progressively constrain it based on the patch size: smaller patches will be more influenced than larger patches. For example, for $m=15$ and a position $x$ where $CCI(x)=1$, patch $\Upsilon(x,m)$ would be of dimensions $S_{\Upsilon}(15,1)\times S_{\Upsilon}(15,1)$, or $45\times 45$. Similarly, for $m=15$ and $CCI(x)=5$ (lower contrast), patch $\Upsilon(x,m)$ would be of dimensions $25\times 25$.

With the use of contrast-aware patch $\Upsilon$ and multiplication factor $m$, we define the local, contrast-guided atmospheric intensity $A^{c}_{LCG\infty}$ for a color channel $c$ as:

The main distinction between $A^{c}_{LCG\infty}(x,m)$ (Equation 9) and $A^{c}_{L\infty}(x)$ (Equation 3) is that the former uses contrast-aware patches $\Upsilon(x,m)$, while the latter uses fixed-size patches $\Psi(x)$. Thus we refer the reader to Equation 3 and its discussion on sub-section 2.2. Since $A^{c}_{LCG\infty}$ is calculated for each color channel $c$, by maximizing contrast-aware patch $\Upsilon(x,m)$ one is actually determining a local position $y$ where the radiance for a certain color channel $c$ is maximum in the $I^{c}_{min}$ (i.e., a dark channel specific for color $c$). Figure 2 compares the different atmospheric lighting models obtained using $A_{LCG\infty}$ and $A_{\infty}$. The $A_{LCG\infty}$ is filtered using a Gaussian kernel with $\sigma=10$ to prevent the creation of abrupt, square-like intensity discontinuities (“halos”) when normalizing the input image by the atmospheric lighting (see Equation 4). Differently from $A_{\infty}$, $A_{LCG\infty}$ captures the color characteristics of the illumination, as well as its local distribution throughout the image.

The choice of parameter $m$ in Equations 8 and 9 determines the size of local window $\Upsilon$ and therefore the radius of influence from each illumination source. In other words, higher $m$ creates brighter lighting models. While this generates output with less darkness, it also might apply an excess of radiance correction (i.e., bringing intensities close to saturation) in regions of the image and hide important intensity changes (as previously discussed about [39]). In order to harvest the advantages of both choices, we derive two $A_{LCG\infty}$: one with $m=5$ and the other choosing $m=30$. These lighting models are used with Equation 4 to determine two transmission maps. These maps are filtered using a Fast Guided filter [25], and finally used to recover two haze-less versions of the original image (Equation 5). The image enhancement results obtained using each $A_{LCG\infty}$ serve as inputs $\mathcal{I}^{k}$ ($k=\{1,2\}$) to a multi-scale fusion process. Figure 3 shows two inputs generated with $m=\{5,30\}$ for an OceanDark sample.

By fusing the inputs generated from different $A_{LCG\infty}$, we preserve two important aspects of the enhanced images: the efficient darkness removal of the input obtained with $m=30$ (Figure 3 right) and the edges, textures and overall intensity changes of the input generated with $m=5$ (Figure 3 center). Experiments evaluating different pairs of $m$ values yielded the best performance when using $m=5$ and $m=30$. We also performed tests using a higher number of inputs, but similar results were obtained as for fusing the inputs corresponding to $m=5$ and $m=30$, at the expense of a higher computational complexity.

In order to properly combine the two inputs, we calculate three weight maps from each of them: saliency, luminance and local contrast. These weight maps guarantee that regions in the inputs that have high saliency and contrast, or that possess edges and texture variations will be emphasized in the fused output [7, 4].

As discussed in sub-section 2.2, the saliency weight map is calculated by subtracting a Gaussian-smoothed version of input $k$, $\mathcal{I}^{G_{s}}_{k}$, by the mean intensity value of this same input, $\mathcal{I}_{k}^{\mu}$ (constant for each input), as detailed in Equation 10.

where $x$ represents a spatial location of input $k$ and $\mathcal{I}^{G_{s}}_{k}$ is obtained with a $5\times 5~{}(\frac{1}{16}[1,4,6,4,1])$ Gaussian kernel.

Considering that saturated colors present higher values in one or two of the $R,G,B$ color channels [7], we use Equation 11 to calculate the luminance weight map $\mathcal{W}_{L}^{k}$.

where $L^{k}$ represents, at each spatial position, the mean of R, G, B intensities for input $k$. $R^{k},G^{k}$ and $B^{k}$ are the three color channels of input $\mathcal{I}^{k}$.

The local contrast weight map $\mathcal{W}_{LCon}^{k}$, also used in [4], is responsible for highlighting regions of input $\mathcal{I}^{k}$ where there is more local intensity variation. We calculate it by applying a $\frac{1}{8}\big{[}\begin{smallmatrix}-1&-1&-1\\ -1&8&-1\\ -1&-1&-1\\ \end{smallmatrix}\big{]}$ Laplacian kernel on $L^{k}$.

The three weight maps ($\mathcal{W}_{LCon}^{k}$, $\mathcal{W}_{L}^{k}$, $\mathcal{W}_{S}^{k}$) are combined into a normalized weight map $\bar{\mathcal{W}}^{k}$, from which a $5$-level Gaussian pyramid $G\{\bar{\mathcal{W}}^{k}\}$ is derived. Our choice of using Gaussian pyramids is based on their efficacy in representing weight maps, as demonstrated by Ancutiet al. [7]. Figure 4 illustrates the three weight maps calculated for one input image, as well as their equivalent normalized weight map.

Following the procedure of [4, 7, 8, 11], each input $\mathcal{I}^{k}$ is decomposed into a $5$-level Laplacian pyramid $L\{\mathcal{I}^{k}\}$. The multi-scale fusion process is then carried out using Equation 7. The output of L²UWE (Figure 5) reduces the darkness from the original image, retains colors and enhances important visual features.