Semantic-embedded Unsupervised Spectral Reconstruction
from Single RGB Images in the Wild

The physical degradation model of RGB images from HS images could be generally formulated as

$\displaystyle\mathbf{X}=\mathbf{C}\mathbf{Y}+\mathbf{N}_{y},\vspace{-0.2cm}$

(1)

where $\mathbf{X}\in\mathbb{R}^{3\times hw}$ denotes the vectorial representation of an RGB image of spatial dimensions $h\times w$, $\mathbf{Y}\in\mathbb{R}^{s\times hw}$ is the corresponding HS image with $s$ ($s\gg 3$) spectral bands to be reconstructed, $\mathbf{C}\in\mathbb{R}^{3\times s}$ is the camera spectral response function (SRF), and $\mathbf{N}_{y}\in\mathbb{R}^{3\times hw}$ is the noise.

Apparently, one can design and train a DNN with paired RGB and HS images to learn the non-linear mapping from $\mathbf{X}$ to $\mathbf{Y}$. However, it is costly, laborious and inconvenient to collect massive such image pairs for training in practice. Besides, due to different spatial resolution, field of views, and focal lengths, it is non-trivial to well register RGB and HS images captured by dual cameras. Alternatively, one may consider synthesizing RGB images from real HS images to form paired training data. However, the inevitable gap between real and synthetic RGB images may result in the DNN trained with synthetic data cannot be well generalized to real data.

To this end, as shown in Fig. 1, we propose an unsupervised, lightweight, and end-to-end learning-based HS image reconstruction framework from single RGB images captured by commercial cameras, where the corresponding ground-truth HS images of input RGB images are not required during training. Our framework is mainly composed of four modules:

(1) Degradation model-aware HS image generation: Let $\widehat{\mathbf{Y}}$ be the recovered HS image from $\mathbf{X}$, and $\widehat{\mathbf{X}}$ be the re-projected RGB image by applying the camera SRF of $\mathbf{X}$ to $\widehat{\mathbf{Y}}$. According to Eq. (1), if $\widehat{\mathbf{Y}}$ approximates the ground-truth one well, the difference between $\mathbf{X}$ and $\widehat{\mathbf{X}}$ should be very small. Based on this intrinsic degradation relationship, we propose an HS image generation network, where a coarse HS image is first generated and then progressively refined by spreading useful information embedded in the difference between $\mathbf{X}$ and $\widehat{\mathbf{X}}$. The design of our generation network elegantly adapts to such an unsupervised scenario. See Section 3.2.

(2) Prior-driven camera SRF estimation: To realize the re-projection of the generation network, the camera SRFs of input RGB images have to be known. However, for RGB images in the wild, their SRFs are camera model-dependent and usually unavailable. Based on the fact that the SRFs of commercial cameras actually lie in a low-dimensional convex space (e.g., the first two principle components contain over 97 % of total variance of SRFs of 28 different camera models) [19], we propose a prior-driven SRF estimation network, in which based on extracted deep features from input RGB images, SRFs are estimated as the convex combination of the atoms of a pre-defined SRF dictionary. See Section 3.3.

(3) $\mathcal{L}_{1}$ gradient clipping boosted adversarial learning: To enable the learning of the generation network under the lack of paired ground-truth HS images, we adopt the adversarial learning manner, i.e., a discriminator is used to promote the distribution of generated HS images to approach that of real HS images. Besides, due to the inherent possibility of gradient explosion [37], it is non-trivial to train GANs [37, 48]. Inspired by traditional gradient clipping [7], we propose a simple yet effective strategy, namely $\mathcal{L}_{1}$ gradient clipping, to stabilize the training process and likewise boost performance, in which the magnitude of the gradient matrix from the discriminator is adaptively clipped with reference to the gradient of $\mathcal{L}_{1}$ loss function. See Section 3.4

(4) Semantic-embedded regularization: Owing to the rich spectrum information, the pixels of an HS image are discriminative [8], i.e., pixels corresponding to different (resp. identical) objects/materials always have distinct (resp. similar) spectral signatures [30, 10]. By leveraging this unique property, we propose a semantic-embedded regularization network to further regularize the solution space of the unsupervised generation network for meaningful reconstruction, i.e., during training, the generated HS images by the generation network are fed into a scene parsing/semantic segmentation network trained with the ground-truth semantic maps of the input RGB images to predict semantic maps. In contrast to the discriminator that globally regularizes the distribution of HS images, this module is able to locally regularize the pixels of generated HS images. See Section 3.5. Note that RGB image-based semantic segmentation is a popular image analysis task and there are many publicly available datasets, and thus our method will not introduce additional costs. Moreover, the emerging weakly-supervised scene parsing methods [42, 9] will lessen the requirements of the ground-truth semantic annotations.

Note that our framework is end-to-end trained with the objective functions in Section 3.6, and the two modules adversarial learning and semantic-embedded regularization are not needed during testing. In what follows, we will show the technical details of each module.

As shown in Fig. 2, this module generates an HS image from an input RGB image in a coarse-to-fine fashion via the following three steps:

Coarse generation. We first learn a coarse HS image denoted as $\mathbf{Y}^{(1)}$ using an efficient sub-CNN $\mathsf{G}^{(0)}(\cdot)$:

$\displaystyle\widetilde{\mathbf{Y}}^{(1)}=\mathsf{G}^{(0)}(\mathbf{X}).$

(2)

Progressive refinement. We then progressively refine $\mathbf{Y}^{(1)}$ via a multi-stage structure, and at the $t$-th ($1\leq t\leq T-1$) stage, the refinement is carried out as

$\displaystyle\widetilde{\mathbf{Y}}^{(t+1)}=\widetilde{\mathbf{Y}}^{(t)}+\mathsf{G}^{(t)}(\mathbf{C}_{o}\widetilde{\mathbf{Y}}^{(t)}-\mathbf{X}),$

(3)

where $\mathsf{G}^{(t)}(\cdot)$ stands for the sub-CNN at the $t$-th stage, and $\mathbf{C}_{o}\in\mathbb{R}^{3\times s}$ is the predicted camera SRF explained in Section 3.3. Note that $\mathsf{G}^{(1)}(\cdot)$, $\cdots$, $\mathsf{G}^{(T-1)}(\cdot)$ have the same network architecture equipped with different parameters, and the network architecture of $\mathsf{G}^{(0)}(\cdot)$ is different from that of $\mathsf{G}^{(t)}(\cdot)$ because $\mathsf{G}^{(t)}(\cdot)$ perceives error maps, while $\mathsf{G}^{(0)}(\cdot)$ takes RGB images as input.

Non-negative thresholding. Being aware that the pixel intensity of a physical meaningful HS image should be non-negative, we finally employ the LeakyReLU, which does not obstruct gradient propagation, to guarantee the non-negativity of generated HS images:

$\displaystyle\widetilde{\mathbf{Y}}=\mathsf{LeakyReLU}\left(\widetilde{\mathbf{Y}}^{(T)}\right).$

(4)

where $\widetilde{\mathbf{Y}}\in\mathbb{R}^{s\times hw}$ is the generated HS image.

Denote by $\mathcal{D}_{c}=\{\mathbf{C}_{1},\mathbf{C}_{2},\cdots,\mathbf{C}_{N}\}$ a pre-defined dictionary consisting of $N=28$ commonly-used camera SRFs [19]. We first estimate an intermediate SRF $\widehat{\mathbf{C}}\in\mathbb{R}^{3\times s}$ as

	$\displaystyle\widehat{\mathbf{C}}(i)=\sum_{j=1}^{N}\widehat{\mathbf{w}}_{i}(j)\times\mathbf{C}_{j}(i),~{}~{}(i=1,~{}2,~{}3),$
	$\displaystyle{\rm with}~{}~{}\widehat{\mathbf{w}}_{i}=\mathcal{S}_{max}(\textbf{W}(i)),~{}~{}\mathbf{W}=\mathsf{M}(\mathbf{X}),$		(5)

where $\mathsf{M}(\cdot)$ is a sub-CNN extracting deep features from $\mathbf{X}$ and outputs the weight matrix $\mathbf{W}\in\mathbb{R}^{3\times N}$; $\widehat{\mathbf{C}}(i)$, $\mathbf{C}_{j}(i)$, and $\mathbf{W}(i)$ are the $i$-th row of $\widehat{\mathbf{C}}$, $\mathbf{C}_{j}$, and $\mathbf{W}$, respectively; $\widehat{\mathbf{w}}_{i}(j)$ is the $j$-th entry of $\widehat{\mathbf{w}}_{i}\in\mathbb{R}^{1\times N}$; $\mathcal{S}_{max}(\cdot)$ denotes the softmax operator.

Besides, real RGB images by commercial cameras are usually adjusted to be visually-pleasing via white balance. That is, the red, green, and blue three channels are separately re-scaled to have the same average value via the widely-used Gray World white balance algorithm [24]. Considering its pixel position-independent property, we mimic this operation by re-scaling the SRF:

	$\displaystyle\mathbf{C}_{o}(i)=$	$\displaystyle\frac{\sum_{j=1}^{N}\widetilde{\mathbf{W}}(i,j)}{\sum_{j=1}^{N}\sum_{k=1}^{3}\widetilde{\mathbf{W}}(k,j)/3}\times\widehat{\mathbf{C}}(i),~{}~{}(i=1,~{}2,~{}3),$
		$\displaystyle{\rm with}~{}~{}\widetilde{\mathbf{W}}=\mathsf{Sigmoid}(\mathbf{W}),$		(6)

where $\mathbf{C}_{o}(i)$ is the $i$-th row of the finally estimated camera SRF $\mathbf{C}_{o}\in\mathbb{R}^{3\times s}$, $\mathsf{Sigmoid}(\cdot)$ is the sigmoid function, and $\widetilde{\mathbf{W}}(k,j)$ denotes the $(k,j)$-th entry of matrix $\widetilde{\mathbf{W}}\in\mathbb{R}^{3\times N}$. The use of the sigmoid function is able to avoid the cancellation when computing the the summation of positive and negative values, and also maintains the original order.

It is commonly-known that training a good GAN is challenging, and the poor convergence may severely compromise the final performance. Instead of using existing regularization methods [28, 5, 16], which may compromise the discriminant ability of the discriminator, we propose $\mathcal{L}_{1}$ gradient clipping to stabilize the training process and likewise boost performance. Specifically, the existing gradient clipping [7], which was originally designed for solving the gradient issue of training RNNs, re-scales the intermediate gradient to a certain level, i.e., ,

$\displaystyle\mathcal{C}(\mathbf{G},\tau)=\mathsf{Min}\left(1,\frac{\tau}{\|\mathbf{G}\|_{F}}\right)\cdot\mathbf{G},$

(7)

where $\mathcal{C}(\cdot,\cdot)$ is the gradient clipping function for training, $\tau$ is the gradient threshold, $\mathbf{G}\in\mathbb{R}^{c\times hw}$ is the gradient matrix to be regularized, and $\|\cdot\|_{F}$ is the Frobenius norm of a matrix. Although gradient clipping can regularize the gradient magnitude without changing discriminator parameters and weakening its discriminant ability, the threshold $\tau$ has to be carefully set. Based on the observation that the classic $\mathcal{L}_{1}$ loss function can produce a gradient with a fixed $\mathcal{F}$-norm, which only varying with the input size, we modify Eq. (7) and propose $\mathcal{L}_{1}$ gradient clipping, which re-scales the gradient matrix as $\mathcal{C}(\mathbf{G},\frac{1}{\sqrt{c\times hw}})$. We apply this $\mathcal{L}_{1}$ gradient clipping to the gradient matrix back-propagated from the discriminator to the HS image generation network. We experimentally validated the impressive performance of such a simple strategy in Table 5.

We utilize existing scene parsing datasets as the source of input RGB images during training, which contain semantically-labeled RGB images in pixel-wise. As it is usually complex and time-consuming to train a scene parsing network, we propose the following simple manner: (1) first utilizing a pre-trained scene parsing network [40] as the backbone, denoted as $\mathsf{E}_{1}(\cdot)$, to extract features denoted as $\mathbf{\Phi}_{X}$ from $\mathbf{X}$ and a plain sub-CNN, denoted as $\mathsf{E}_{2}(\cdot)$, to extract features denoted as $\mathbf{\Phi}_{Y}$ from $\widetilde{\mathbf{Y}}$; and (2) then fusing $\mathbf{\Phi}_{X}$ and $\mathbf{\Phi}_{Y}$ via another sub-CNN, denoted as $\mathsf{SE}(\cdot,\cdot)$, to predict the semantic map $\textbf{P}\in\mathbb{R}^{M\times hw}$ with $M$ being the total number of semantic classes. It is expected that via back-propagation, the generation network will be promoted to produce $\widetilde{\mathbf{Y}}$ which could be distinguished by the semantic-embedded network trained with ground-truth semantic maps of $\mathbf{X}$. We also apply $\mathcal{L}_{1}$ gradient clipping to stabilize the fluctuated gradient from the semantic-embedded network.

Analogy to most GAN-based frameworks, we alternately train the generator (including HS image generation, semantic-embedded regularization, and camera SRF estimation) and the discriminator. The objective function of the generator is

	$\displaystyle\mathcal{L}_{G}=\mathcal{B}(\mathsf{D}(\widetilde{\mathbf{Y}}/\overline{y}),~{}1)+\lambda_{b}\mathcal{S}_{bp}(\mathbf{C}_{o},\widetilde{\mathbf{Y}},\mathbf{X})+\lambda_{s}\mathcal{S}_{1}(\widetilde{\mathbf{Y}})$			(8)
	$\displaystyle+\lambda_{p}\mathcal{S}_{pos}(\widetilde{\mathbf{Y}})+\lambda_{sem}\mathcal{S}_{sem}(\mathbf{P},\mathbf{L}),$

where $\lambda_{b}$, $\lambda_{s}$, $\lambda_{p}$, and $\lambda_{sem}$ are non-negative parameters to balance different terms, which are empirically set to $1e^{2}$, $1e^{1}$, $1e^{-2}$, and $1e^{0}$, respectively; $\overline{y}$ is the mean value of $\widetilde{\mathbf{Y}}$; $\mathcal{B}(\cdot)$ is the binary cross-entropy loss; $\mathsf{D}(\cdot)$ stands for the discriminator; the $\mathcal{S}_{bp}(\cdot,\cdot,\cdot)$ is the back-projection constraint, $\mathcal{S}_{1}(\cdot)$ is the first-order smoothness constraint, $\mathcal{S}_{pos}(\cdot)$ is the positive-definite constraint, and $\mathcal{S}_{sem}(\cdot,\cdot)$ is the loss for the semantics embedding regularization, where are respectively defined as

$\displaystyle\mathcal{S}_{bp}(\mathbf{C}_{o},\widetilde{\mathbf{Y}},\mathbf{X})=\frac{1}{3\times hw}\left\|\mathbf{C}_{o}\widetilde{\mathbf{Y}}-\mathbf{X}\right\|_{F}^{2},$

(9)

$\displaystyle\mathcal{S}_{1}(\widetilde{\mathbf{Y}})=\frac{1}{(s-1)\times hw}\sum_{i=1}^{s-1}\left\|\widetilde{\mathbf{Y}}(i)-\widetilde{\mathbf{Y}}(i+1)\right\|_{1},$

(10)

$\displaystyle\mathcal{S}_{pos}(\widetilde{\mathbf{Y}})=\left\|\widetilde{\mathbf{Y}}-\left\lvert\widetilde{\mathbf{Y}}\right\rvert\right\|_{F}^{2},$

(11)

$\displaystyle\mathcal{S}_{sem}(\mathbf{P},\mathbf{L})=\frac{1}{hw}\sum_{n=1}^{hw}-\mathbf{L}(n)\log(\mathbf{P}(n)),\vspace{-0.3cm}$

(12)

where $\lvert\cdot\rvert$ computes the absolute value of the input in element-wise, $\|\cdot\|_{1}$ is the $\ell_{1}$-norm of a vector, $\mathbf{P}(n)\in\mathbb{R}^{M\times 1}$ is the semantic prediction of the $n$-th pixel, and $\mathbf{L}(n)\in\mathbb{R}^{M\times 1}$ is the ground-truth of $\mathbf{P}(n)$. The objective function of the discriminator is

$\displaystyle\mathcal{L}_{D}=\mathcal{B}(\mathsf{D}(\widetilde{\mathbf{Y}}/\overline{y}),~{}0)+\mathcal{B}(\mathsf{D}(\mathbf{Y}_{r}/\overline{y}_{r}),~{}1),$

(13)

where $\mathbf{Y}_{r}\in\mathbb{R}^{s\times hw}$ is a real HS image, $\overline{y}_{r}$ is the mean value of $\mathbf{Y}_{r}$. We normalize the $\widetilde{\mathbf{Y}}$ and $\mathbf{Y}_{r}$ respectively with their mean values to focus the reconstruction process much on the spectral angular of HS images. Besides, the proposed $\mathcal{L}_{1}$ gradient clipping operation in Section 3.4 will be used to regularize the backward gradients.

Experiment settings. Although there is a gap between synthetic and real data, we still conducted experiments on synthetic data as done in existing works [49], [36] to have a quantitative and intuitive understanding of our framework.

We selected the last 20 HS images of ICVL to form the test dataset, and the remaining ones as the training dataset. All the HS images of NTIRE’20 were used as the test dataset. For fair comparisons, we applied the same dictionary of camera SRFs in Section 3.3 to the training HS images to synthesize RGB images, generating paired training data, which were used to train HSCNN-D [36], FM-Net [49], and CHS-GAN [2] with the paired synthetic RGB and HS images. Following the original implementation [14], we first trained DBF to learn a basis function and then trained it on the synthetic RGB images with the learnt basis function. The proposed method was trained with PASCAL-Context and the aforementioned training HS images from ICVL as RGB and HS image sources, respectively, with the batch size of 24. The number of stages $T$ was set to $4$. We adopted the ADAM [22] optimizer for optimization of the generation network, the discriminator, and the camera SRF estimation network. The exponential decay rates were set as $\beta_{1}=0.9$ and $\beta_{2}=0.999$ for the first and second moment estimates, respectively. We also adopted the TTUR scheme [18] to set the learning rate of the SRF estimation network, HS image generation network, and the HS image discriminator to $1e^{-4}$, $1e^{-4}$, and $5e^{-4}$, respectively. Meanwhile, for the semantic embedding module, we fixed $\mathsf{E}_{1}(\cdot)$ and trained $\mathsf{E}_{2}(\cdot)$ and $\mathsf{SE}(\cdot)$ using the SGD optimizer with the base learning rate, the momentum, and the weight decay equal to $1e^{-3}$, 0.9, and $1e^{-4}$, respectively. The poly learning rate policy with the power of 0.9 is used for dropping the learning rate [40]. The proposed method is trained 50 epochs with 208 iterations each epoch. We tested all the methods with the same protocols on the synthetic RGB images generated from the widely-used camera SRF— Nikon D700. To quantitatively compare different methods, we adopted 3 commonly-used metrics, i.e., Peak Signal-to-Noise Ratio (PSNR), Average Structural Similarity Index (ASSIM) [43], and Spectral Angle Mapper (SAM) [47]. We reported all the methods with the training results of the last epoch.

Quantitative and visual results. Table 1 lists the results of different methods on ICVL, where it can be seen that our method significantly outperforms the unsupervised DBF and achieves comparable performance to the supervised HSCNN-D and CHS-GAN, which are credited to the carefully designed framework. But the gap between our method and FM-Net — the most recent supervised method also indicates the potential of this topic. However, as listed in Table 2, our method even exceeds FM-Net on NTIRE’20, which somewhat demonstrates better generalization ability. As visualized in Fig. 3, we can see that the error between the reconstructed spectral band and the corresponding ground-truth one is much smaller than those of compared methods. Such an advantage may be benefit from the semantic information of the PASCAL-Context dataset containing the class of sky. Besides, the spectral signatures of selected pixels are closer to ground-truth ones. Finally, the proposed method consumes much less time than HSCNN-D and FM-Net.

For real RGB images, the well registered ground-truth HS images are usually unavailable, making it impossible to quantitatively evaluate different methods, as done in Section 4.1. Besides, due to the high spectral resolution, it is also difficult to visually compare the results of different methods. Considering that the rich spectrum information makes pixels of HS images discriminative, which will be benefit high-level vision tasks, we introduced the HS video-based visual tracking experiment to evaluate different HS image reconstruction methods. It is expected a better method will generate HS images that are closer to unknown ground-truth ones, which then produce high tracking accuracy.

Experiment settings. We compared with HSCNN-D and FM-Net, which always achieve the top two performance among all the compared methods in the previous scenario. For all methods, we directly adopted the trained models in Section 4.1 without additional training to reconstruct the 10 randomly selected RGB videos from HOTAC into HS videos, which were then fed to a pre-trained HS video-based tracker named MHT [44] to realize tracking. Besides, we also performed the MHT tracker on the 10 HS videos provided by HOTAC, corresponding to the 10 selected RGB videos. The tracking performance of such a setting, denoted as “Raw”, could be thought of as the upper-bound of the reconstruction methods. We quantitatively measured the tracking performance by using Average Precision (AP) among all the distance threshold, AUC, and location error, three commonly-used metrics in visual tracking.

Quantitative results. Table 4 and Fig. 4 show the results, where it can be seen that our method even produces better tracking performance than the two supervised methods FM-Net and HSCNN-D, and as expected, the performance of all the three computation-based reconstruction methods is lower than that of “Raw”. Please refer to our Video Demo for more visual results.

Regularization terms. We experimentally validated the effectiveness of the smooth constraint $\mathcal{S}_{1}(\cdot)$, the positive constraint $\mathcal{S}_{pos}(\cdot)$, the back-projection constraint $\mathcal{S}_{bp}(\cdot,~{}\cdot,~{}\cdot)$, $\mathcal{L}_{1}$ gradient clipping, and the semantic-embedded module. As shown in Table 3, we can see that our framework benefits from each of these components. Among them, the back-projection regularization, semantic embedding module and $\mathcal{L}_{1}$ gradient clipping have the pivotal influence. Besides, only using traditional regularization terms such as $\mathcal{S}_{bp}(\cdot,~{}\cdot,~{}\cdot)$, $\mathcal{S}_{pos}(\cdot)$, and $\mathcal{S}_{1}(\cdot)$, the method produces very limited performance comparable to that of Bicubic interpolation, indicating that the unsupervised HS image reconstruction actually is a very challenging task.

Camera SRF estimation. Even with supervision, camera SRF estimation of RGB images is a quite challenging task [21]. To evaluate the effectiveness of our camera SRF estimation module, we randomly chose ten SRFs to generate synthetic RGB images from real HS images, which were then fed into our estimation module to estimate the SRFs. We calculated the precision of estimated SRFs using the AUC metric. As shown in Fig. 5, the average precision of our estimation module achieves $77.79\%$. Note that our SRF estimation module is unsupervised, i.e., the ground-truth camera SRFs were not employed as supervision during training.

$\mathcal{L}_{1}$ gradient clipping. To show the advantage of the proposed $\mathcal{L}_{1}$ gradient clipping, we trained our framework with different gradient regularization methods [37, 16], where we only changed the gradient regularization terms and remained the other modules unchanged. As listed in Table 5, the proposed $\mathcal{L}_{1}$ gradient clipping exceeds the other gradient clipping methods to a large extent, demonstrating its superiority.

Various training datasets. To further explore the performance of our method on different datasets, we conducted experiments by training our method with different image datasets as training data. We tested all the trained models with the 20 samples of ICVL dataset. As listed in Table 6, we can see that for the RGB image source, the performance of using PASCAL-Context is better than that of using Cityscapes, which may be credited to the more diverse samples contained in the PASCAL-Context dataset.