Deep-RLS: A Model-Inspired Deep Learning Approach to Nonlinear PCA

Principal component analysis (PCA) is a linear orthogonal transformation that determines the eigenvectors or the basis vectors associated with the covariance matrix of the observed data. The output of the standard PCA is the projection of the data onto the obtained basis vectors, resulting in principal components which are mutually uncorrelated.

As a related problem, blind source separation (BSS)  [1] aims to recover statistically independent source signals from linear mixtures whose composition gains are unknown. In fact, the only knowledge about the source signals to be recovered in BSS is their independence. This is a stronger assumption than what one requires for PCA, i.e. uncorrelated signal sources. This suggest that performing PCA on such a mixture of signals may not successfully separate them, and that higher order statistics is required to verify and ensure the independence of the source signals.

In light of the above, the notion of nonlinear correlation was introduced to verify the independence of signals more effectively [2]. It was also shown that an extension of PCA to the nonlinear domain (known as nonlinear PCA) can separate independent components of the mixture [2]. In [1], an objective function targeting the fourth-order moments of the distribution of the recovered signals is suggested. It is proved that minimizing this objective function can obtain the independent source signals under the assumption that all sources have negative kurtosis, i.e. they are sub-Gaussian. Minimizing such an objective functions introduces nonlinearities in the signal recovery and the learning process.

In this paper, we propose a deep learning approach to nonlinear PCA that relies on the unfolding of an iterative algorithm into the layers of a deep neural network in order to perform the nonlinear PCA task; i.e. to estimate the source signals for BSS. In particular, we use the classical recursive least squares [3] to construct the network structure and to iteratively estimate the source signals. We experimentally verify the performance of our algorithm and compare it with the traditional RLS algorithm.

Deep neural networks (DNN) are one of the most studied approaches in machine learning owing to their significant potential. They are usually used as a black-box without incorporating any knowledge of the system model. Moreover, they are not always practical to use as they require a large amount of training data and considerable computational resources. Hershey, et al., in [8], introduced a technique referred to as deep unfolding or unrolling in order to enable the community to address the mentioned issues with DNNs. The deep unfolding technique lays the ground for bridging the gap between well-established iterative signal processing algorithms that are model-based in nature and deep neural networks that are purely data-driven. Specifically, in deep unfolding, each layer of the DNN is designed to resemble one iteration of the original algorithm of interest. Passing the signals through such a deep network is in essence similar to executing the iterative algorithm a finite number of times, determined by the number of layers. In addition, the algorithm parameters (such as the model parameters and forgetting parameter in the RLS algorithm) will be reflected in the parameters and weights of the constructed DNN. The data-driven nature of the emerging deep network thus enables improvements over the original algorithm. Note that the constructed network may be trained using back-propagation, resulting in model parameters that are learned from the real-world training datasets. In this way, the trained network can be naturally interpreted as a parameter optimized algorithm, effectively overcoming the lack of interpretability in most conventional neural networks [9]. In comparison with a generic DNN, the unfolded network has much fewer parameters and therefore requires a more modest size of training data and computational resources. The deep unfolding technique has been deployed for many signal processing problems and has dramatically improved the convergence rate of the state-of-the-art model-based iterative algorithms; see, e.g., [10, 11, 12] and the references therein.

Due to the promises of deep unfolding in addressing the shortcomings of both generic deep learning methods and model-based signal processing algorithms, we have been motivated to deploy this technique to improve the recursive least squares solution for nonlinear PCA. As shown in [6], when applied to a linear mixture of source signals (i.e., the BSS problem), the RLS algorithm usually approximates the true source signals well and successfully separates them. However, the number of iterations needed to converge may vary greatly depending on the initial values and the forgetting parameter $\beta$. Inspired by the deep unfolding technique, we introduce Deep-RLS, our deep learning-based framework which is designed based on the modified iterations of the algorithm 1. More precisely, the dynamics of the $k$-th layer of Deep-RLS are given as:

where $\mathbf{x}(k)$ is the data vector at the time instance $k$, $\mathbf{g}(.)$ is a nonlinear activation function which can be chosen by considering the distribution of the source signals, $\omega_{k}\in\mathds{R}$ represents the trainable forgetting parameter, and $\mathbf{H}_{k}\in\mathds{R}^{m\times m}$ and $\mathbf{b}_{k}\in\mathds{R}^{m}$ denotes the set of trainable weights and biases of the $k$-th layer, respectively. In [5], it was shown that for source signals ${\mathbf{s}(t)}$ with sub-Gaussian distribution, $\mathbf{g}(\mathbf{x})=tanh(\mathbf{x})$ results in convergence of the nonlinear PCA to the true source signals.

Given $T$ samples of the data vector ${\mathbf{x}(t)}$, our goal is to optimize the parameters $\{\Gamma_{k}\}_{k=1}^{T}$ of the DNN, where

The output of the $k$-th layer is an approximation of the source signals at the time instance $k$. For training of the proposed Deep-RLS network, we consider cumulative MSE loss of the layers. In designing the training procedure, one needs to consider the constraint that the forgetting parameter must satisfy $0<\beta\leq 1$. Hence, in order to impose such a constraint, one can regularize the loss function ensuring that the network chooses proper weights $\{\omega_{k}\}_{k=1}^{T}$ corresponding to a feasible forgetting parameter at each layer [13]. Accordingly, we define the loss function used for the training of the proposed architecture as follows:

where $\mathrm{ReLU}(\cdot)$ is the well-known Rectifier Linear Unit function extensively used in the deep learning literature. We have presented the employed training process in Algorithm 2.

In this section, we demonstrate the performance of the proposed Deep-RLS for nonlinear PCA in the case of blind source separation. The proposed framework was implemented using the PytTorch library [14] and the Adam stochastic optimizer [15] with a learning rate of $10^{-3}$ for training purposes.

The training was performed based on the data generated via the following model. For the time interval $t=0,1,...,T$, each element of the vector $s(t)$ is to be generated from a sub-Gaussian distribution. For data generation purposes, we have assumed the source signals to be i.i.d. and uniformly distributed, i.e., $s(t)\sim\mathcal{U}(0,1)$. The mixing matrix $\mathbf{A}$ is assumed to be fixed and generated once according to a Normal distribution, i.e., $\mathbf{A}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. We performed the training of the proposed Deep-RLS using the batch learning process with a batch size of 40 and trained the network for $N=50$ epochs. A training set of size $10^{3}$ and test set of size $10^{2}$ was chosen. We used the average of the mean-square-error (MSE), $\mathrm{MSE}=(1/T)\sum_{k=1}^{T}||\mathbf{s}(k)-\mathbf{y}(k)||^{2}_{2}$, as the performance metric. In Fig. 1, the performance of RLS implemented with $\beta=0.99$ at different number of iterations is compared with the Deep-RLS network trained with the same number of layers. It can be observed that the average MSE obtained using the proposed architecture is much lower than RLS, even with fewer number of layers/iterations. Fig. 2 demonstrates the performance of the proposed Deep-RLS network and the original RLS algorithm in terms of the average MSE for a growing number of source signals $m$. It can be observed from both Figs. 1 and 2 that the proposed method achieves a far superior performance than that of the original RLS algorithm.

We considered the application of model-based machine learning, and specifically the deep unfolding technique, for nonlinear PCA. A deep network based on the well-known RLS algorithm, which we call Deep-RLS, was proposed that outperforms its traditional model-based counterparts.