DTR: A Unified Deep Tensor Representation Framework for Multimedia Data Recovery

Matrices and tensors are denoted by capitalized letters, e.g., $\mathbf{X}$ and capitalized calligraphic letters, e.g., $\mathcal{X}$, respectively. For a matrix $\mathbf{X}\in\mathbb{R}^{n_{1}\times n_{2}}$, we use $\mathbf{X}(i,j)$ to denote the $(i,j)$-th element of $\mathbf{X}$. Meanwhile, for a third-order tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times n_{2}\times n_{3}}$, we use $\mathcal{X}(i,j,k)$ to denote the $(i,j,k)$-th element of $\mathcal{X}$. $\mathcal{X}^{(k)}\in\mathbb{R}^{n_{1}\times n_{2}}$ denotes the $k$-th frontal slice of $\mathcal{X}$. $\odot$ denotes the point-wise multiplication (i.e., Hadamard product). The Frobenius norm of tensor $\mathcal{X}$ is defined as $\|\mathcal{X}\|_{F}=\sqrt{\sum_{i,j,k}\left|\mathcal{X}(i,j,k)\right|^{2}}$. $\Delta$ denotes the face-wise product between two tensors [32]. Given $\mathcal{X}\in\mathbb{R}^{n_{1}\times n_{2}\times n_{3}}$, $\mathcal{Y}\in\mathbb{R}^{n_{2}\times n_{4}\times n_{3}}$, the face-wise product between $\mathcal{X}$ and $\mathcal{Y}$ is $\mathcal{X}\Delta\mathcal{Y}=\mathcal{Z}\in\mathbb{R}^{n_{1}\times n_{4}\times n_{3}}$, where $\mathcal{Z}^{(k)}=\mathcal{X}^{(k)}\mathcal{Y}^{(k)}(k=1,\ldots,n_{3})$.

According to above mentioned two definitions, we have

An indispensable component in the transform-based tensor representation is the transform that captures the frontal slice relationships of multi-dimensional data. Over the past decade, researchers have explored various transforms [22, 28, 20, 30, 17] to facilitate the most classic transform-based tensor representation, i.e., tensor singular value decomposition (t-SVD). For instance, Kilmer et al. [23] first introduced the DFT along the third dimension under the t-SVD framework. Madathil et al. [22] adopted the real-valued DCT to avoid the high computational cost caused by complex operations in DFT-based original t-SVD. Lu et al. [21] deduced the new tensor tubal rank based on the invertible linear transform and provided the theoretical bound for the exact recovery under certain tensor incoherence conditions for tensor completion. Unlike previous invertible transforms, Jiang et al. [20] employed the non-invertible framelet transform to construct a framelet latent tensor. In addition to the previously pre-defined transforms, some data-driven transforms have emerged promising results in multi-dimensional data recovery problems. For example, Song et al. [28] constructed a data-dependent unitary transform, which can be obtained by taking the SVD of the unfolding matrix. Kong et al. [30] defined the new tensor Q-rank based on novel data-dependent transform, which can be learned by principal component analysis. Luo et al. [34] proposed a nonlinear transform-based tensor nuclear norm (TNN), where the nonlinear transform is obtained by learning a multi-layer neural network solely using the observed tensor in a self-supervised manner.

Another indispensable component in the transform-based tensor representation is the characterization of the frontal slices of latent tensor. The original t-SVD [23] adopted the standard matrix SVD to characterize the frontal slices of latent tensor. Zhou et al. [24] introduced NMF as an alternative approach to efficiently characterize the low-rankness of the frontal slices of latent tensor. In addition, to further enhance the accuracy and efficiency of the characterization, later work [31, 25] adopted an approximate SVD method based on QR decomposition to characterize the frontal slices of latent tensor, which not only reduces the computational complexity but also yields superior results compared to SVD-based method [16]. Despite the benefits of shallow matrix factorization (such as SVD, NMF, and QR decomposition), they may not be representative enough to characterize the latent tensor. Moreover, shallow matrix factorization characterizes the intra-slice relationships of latent tensor separately, which neglects the inter-slice relationships of latent tensor.

Inspired by the transform-based tensor representation, we propose a unified deep tensor representation framework, i.e., DTR, by synergistically combining the deep latent generative module and the deep transform module. The diagram of our DTR framework is shown in Fig. 1, and the mathematical formulation can be expressed as follows:

where $g_{\theta}(\cdot)$ is the deep latent generative module, $\theta$ denotes the network parameter of $g(\cdot)$, and $\mathcal{Z}$ is a random noise. $f_{\xi}(\cdot)$ is the deep transform module, $\xi$ denotes the network parameter of $f(\cdot)$, $\circ$ denotes the composition of functions, and $\mathcal{X}\in\mathbb{R}^{n_{1}\times n_{2}\times n_{3}}$ is the multi-dimensional data represented by the proposed DTR. Two indispensable components in the proposed DTR framework, i.e., the deep latent generative module $g_{\theta}(\cdot)$ and the deep transform module $f_{\xi}(\cdot)$, complement to each other and work together to make up a unified framework. Now, we detailedly introduce the two indispensable components in our DTR framework.

To examine the representation ability of the proposed DTR, we consider the representative multi-dimensional data recovery task and suggest an unsupervised DTR-based multi-dimensional data recovery model, as follow:

Here, $\mathcal{O}$ is the observed multi-dimensional data, $\mathcal{X}$ is the multi-dimensional data represented by the proposed DTR, $\mathcal{M}$ is a binary mask tensor that sets the observed location to be ones and others be zeros, $g(\cdot)$ is the deep latent generative module, and $f_{\xi}(\cdot)$ is the deep transform module.

Although the model (8) is highly non-convex and nonlinear, there are still numerous available methods that can solve it. Specifically, we consider using the popular gradient descent optimization algorithm in deep learning, i.e., the adaptive moment estimation (Adam) [37] algorithm, to effectively optimize $\xi$ and $\theta$. Once the optimal $\xi$ and $\theta$ are obtained, the multi-dimensional data $\mathcal{X}$ can be achieved by Eq. (4).

$\mathbf{Compared\ Methods.}$ To comprehensively evaluate the effectiveness of DTR, we compare it with five classic multi-dimensional data recovery methods, i.e., TNN [16], TRLRF [38], FTNN [20], FCTN [39], and HLRTF [17]. These methods are all tensor decomposition-based methods. Note that TNN and HLRTF adopt shallow SVD and NMF to characterize the frontal slices of latent tensor, respectively. To ensure optimal performance, all parameters associated with the compared methods are carefully adjusted according to the suggestions provided by the respective authors in their articles.

$\mathbf{Implementation.}$ We employ two HSIs (i.e., WDC²²2Available at https://engineering.purdue.edu/biehl/MultiSpec/
hyperspectral.html and Pavia³³3Available at https://www.ehu.eus/ccwintco/index.php/
Hyperspectral_Remote_Sensing_Scenes), eleven MSIs⁴⁴4Available at http://www.cs.columbia.edu/CAVE/databases/multispectral/ (i.e., Beads, Cloth, Flowers, Cd, Beers, Feathers, Statue, Toys, Jelly, Painting, and Watercolors), and two videos⁵⁵5Available at http://trace.eas.asu.edu/yuv/. (i.e., Bird and Sunflower) to test the performance of the proposed DTR. The sampling rates for both random and tube missing cases are set as $\text{SR}=\{0.1,0.2,0.3\}$. Before the experiment, the gray values of all datasets are normalized band-by-band into the interval $[0,1]$. All experiments are performed on a PC equipped with two Intel(R) Xeon(R) Silver 4210R CPUs, two NVIDIA RTX 3090 GPUs, and 128GB RAM. The proposed DTR and HLRTF are implemented by using the Python and the PyTorch library with GPU calculation, and other comparison methods are implemented by MATLAB (R2018a) with CPU calculation. In our experiments, $g_{\theta}(\cdot)$ is set as the U-Net, $f_{\xi}(\cdot)$ is set as the FCN, $\hat{n}_{3}$ value is set as the $n_{3}$, the layer number for both modules is set to 2, the Adam algorithm is adopted to train the model with learning rate 0.001, and the maximum iteration step for both HLRTF and the proposed DTR is set to 7000.

$\mathbf{Evaluation\ Metrics.}$ Two quantitative evaluation metrics peak-signal-to-noise-ratio (PSNR) and structural similarity (SSIM) are selected to evaluate the overall quality of the recovered results. In general, higher PSNR and SSIM values indicate better results.

Tables I-II list the evaluation metrics of the recovered results by different methods under the random missing and tube missing cases, respectively. Especially, we highlight best and second-best results by bold and underline, respectively. From Tables I-II, we can observe that FTNN and FCTN perform well for random missing cases, while the TNN performs well for tube missing cases. The quantitative evaluation results of HLRTF are superior compared to TNN and FTNN for nearly all cases. The reason behind this is that the nonlinear transform in HLRTF obtained by learning a multi-layer neural network can better capture the frontal slice relationships of multi-dimensional data. Additionally, it is worth noting that DTR significantly outperforms HLRTF⁶⁶6Due to the additional TV regularization term existing in HLRTF, some recovered results of HLRTF are better than DTR. for most cases, which can be attributed to the powerful characterization ability of deep latent generator in contrast to shallow matrix factorization.

To visually compare the recovered results, we present visual illustrations of the recovered results obtained by different methods in Figs. 3-6. Aiming at a better visual comparison, we mark one local area and enlarge it under each image. From Figs. 3-6, we can observe that all the compared methods can somewhat recover the multi-dimensional data. The results of TNN and TRLRF contain some spatial blurring effects on MSIs (e.g., Cd and Beads). Although the FTNN and FCTN have better recovery effects than TNN and TRLRF visually, there are still some unsatisfactory performance in local details. The results of HLRTF have more spatial edges or textures compared to FTNN and FCTN, but small blurring effects still can be found. On the contrary, the proposed DTR performs better on the preservation of global structure and local details compared with the comparative methods.

Overall, the proposed DTR achieves superior performance on multi-dimensional data, including HSIs, MSIs, and videos. The superior performace of DTR can be attributed to the powerful representation ability of deep latent generative module and deep transform module. These two indispensable components allow DTR to more faithfully capture the rich textures and details in multi-dimensional data, which is particularly beneficial for multi-dimensional data recovery tasks.

The deep transform module $f_{\xi}(\cdot)$ is another indispensable component in the proposed DTR. Table IV presents the quantitative comparison of the proposed DTR with different the layer number $K$ of $f_{\xi}(\cdot)$ for HSI Pavia. It can be observed that using the deep transform module can achieve better recovered results, which reflects the effectiveness of deep transform module. Particularly, when $K$ = 1, $f_{\xi}(\cdot)$ degenerates into a linear transform. We can observe that nonlinear transform (i.e., $K\geq 2$) can achieve better recovered results, which indicates that nonlinear transform can better capture the frontal slice relationships of multi-dimensional data. Similar to the deep latent generative module, we can observe that increasing $K$ enhances the performance when $K$ is small. However, when $K$ becomes larger, the performance tends to drop. Thus, in our experiments, we set the layer number $K$ of $f_{\xi}(\cdot)$ to 2.