A Neural Network-Prepended GLRT Framework for Signal Detection Under Nonlinear Distortions

We consider two time-series signals, $a_{1}$ and $a_{2}$, transmitted from two different directions, impinging an $M$ element antenna array. Throughout this work, we will denote the interference signal as $a_{1}$ and the SOI, which we are interested in detecting, as $a_{2}$. A snapshot of the output at the antenna array at time $t$ is given by

where $\mathbf{z}(t)\in\mathbb{C}^{M\times 1}$ is a column vector that denotes the time sample of the $M$ sensor outputs at time $t$, $\mathbf{x}(u_{1})$ and $\mathbf{x}(u_{2})$ are the array response vectors of signals $a_{1}$ and $a_{2}$ arriving from directions $u_{1}$ and $u_{2}$ (assuming that the direction parameters $u_{1}$ and $u_{2}$ are scalars), respectively, and $a_{1}(t)$ and $a_{2}(t)$ are the amplitudes of $a_{1}$ and $a_{2}$, respectively, at time $t$. Lastly, $\mathbf{n}(t)\sim\mathcal{C}\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$ is complex additive white Gaussian noise (AWGN) with variance $\sigma^{2}$, and $L$ is total number of time samples.

Let $\mathbf{Z}=[\mathbf{z}(1),\cdots,\mathbf{z}(L)]\in\mathbb{C}^{M\times L}$ denote the complete observed signal at the antenna array. Given $\mathbf{Z}$, we are interested in determining when the SOI turns on (i.e., when $a_{2}$ becomes active). Here, we define the null hypothesis, $\mathcal{H}_{0}$, as the case where $a_{2}$ is absent in $\mathbf{Z}$ (i.e., $a_{2}=\mathbf{0}$), and the alternate hypothesis, $\mathcal{H}_{1}$, as the case where $a_{2}$ is present in $\mathbf{Z}$ (i.e., $a_{2}\neq\mathbf{0}$). Formally, we characterize the distribution of $\mathbf{Z}$ under each hypothesis as

where $\mathbf{R}$ is the true, unknown covariance of $\mathbf{Z}$. Now, under $\mathcal{H}_{1}$, we are exclusively considering a system with one SOI in $\mathbf{Z}$. Here, $\mathbf{X}=[\mathbf{x}(u_{2})]\in\mathbb{C}^{M\times 1}$ is the antenna array response of the SOI, $\mathbf{A}\in\mathbb{C}^{1\times S}$ is the row vector of signal amplitudes, and $\mathbf{T}\in\mathbb{C}^{S\times L}$ is the matrix $[\mathbf{0}\quad\mathbf{I}_{S}]$ ($S=L-T_{0}$) corresponding to the time samples during which the SOI is active (i.e., when $a_{2}$ is present in $\mathbf{Z}$), thus yielding $\mathbf{AT}=[0,0,\ldots,0,a_{2}(T_{0}),a_{2}(T_{0}+1),\ldots,a_{2}(L)]$, where $T_{0}$ denotes the time sample at which $a_{2}$ becomes active in $\mathbf{Z}$. Our objective is to arrive at a metric that will allow us to determine the active hypothesis (i.e., where the SOI became active in $\mathbf{Z}$).

In our case, where $\mathbf{X}$ is completely unknown, we can base detection on the equivalent monotonically related suboptimal statistic of the GLRT [6] given by

where we will reject $\mathcal{H}_{0}$ if $\gamma\geq\gamma_{0}$ for some threshold $\gamma_{0}$, which is empirically determined. Here, $\hat{\mathbf{R}}_{\text{old}}=\mathbf{Z}_{\text{old}}\mathbf{Z}_{\text{old}}^{\text{H}}$ is the empirical covariance of $\mathbf{Z}_{\text{old}}$, and $\hat{\mathbf{R}}_{\text{new}}=\mathbf{Z}_{\text{new}}\mathbf{Z}_{\text{new}}^{\text{H}}$ is the empirical covariance of $\mathbf{Z}_{\text{new}}$. Intuitively, the GLRT solution scans the spatial (covariance) domain on blocks of time samples and results in a contrast peak (further shown and discussed in Sec. III) when a signal (i.e., the SOI) impinging the antenna array from a different direction than the interference turns on in $\hat{\mathbf{R}}_{\text{new}}$ that was not present in $\hat{\mathbf{R}}_{\text{old}}$.

Now, let us consider the effect of the GLRT test statistic given in (5) when a nonlinearity is induced on the received signal. In this case, we will assume that we receive $f(\mathbf{Z}):\mathbb{C}^{M\times L}\rightarrow\mathbb{C}^{M\times L}$ instead of $\mathbf{Z}$, where $f(\cdot)$ is some unknown nonlinear function that is individually applied to each element in $\mathbf{Z}$. Specifically, the nonlinearity effect is given by

The element-wise effect of $f(\cdot)$ is critical because it will affect each sample differently. For example, certain received time samples in $\mathbf{Z}$ will remain in the linear operating region and thus will not be significantly distorted by $f(\cdot)$, whereas other samples will experience high degrees of clipping and distortion due to the nonlinearity. The clipping distortion of each antenna element can be observed in the eigenvalue spread of the covariance matrix, $\hat{\mathbf{R}}_{\text{old}}$.

The induced nonlinearity directly changes the distribution of $\mathbf{Z}$ and, as a result, the initial assumptions about the distributions of the interference and SOI in (2) no longer hold. Furthermore, since the nonlinear function, $f(\cdot)$, is unknown, we are unable to characterize the distribution of the signal after it has been transformed via $f(\cdot)$, thus preventing us from deriving a new GLRT-based test statistic to operate on $f(\mathbf{Z})$. The inability to analytically characterize nonlinearly-processed received signals motivates a data driven approach to signal detection (as described in Sec. II-D), where data samples can be augmented such that the GLRT is able to operate accurately.

To combat the breakdown of the GLRT solution on nonlinear signals, we implement a dense (fully connected) neural network (DNN) classification model, which employs the time samples comprising $f(\mathbf{Z})$ for training. Specifically, our objective is to train the DNN to classify excessively nonlinear time samples, i.e., $f(\mathbf{z}(t))$, using both a linear received signal, $\mathbf{Z}$, and its corresponding nonlinear transformation, $f(\mathbf{Z})$. At test time, when we only receive $f(\mathbf{Z})$ and are unaware of $\mathbf{Z}$, we will use the DNN to identify and eliminate excessively nonlinear samples from $f(\mathbf{Z})$ before applying the GLRT test statistic for detection. Our overall DNN-prepended GLRT proposed framework is shown in Fig. 1.

For compatibility with real-valued DNNs, we will represent each time sample (column vector) as a real-valued vector consisting of the sample’s corresponding real and imaginary components. Specifically, we denote $f(\mathbf{z}(t))$ as $f(\mathbf{z}(t))\in\mathbb{R}^{2M\times 1}$ with the form

where $\Re\{\cdot\}$ and $\Im\{\cdot\}$ denote the real and imaginary components of $\{\cdot\}$ and the subscript denotes the antenna element from which the sample was collected at time $t$.

Using a set of training data for which we have both the linear and nonlinear representation of the received signal, $\mathbf{z}(t)$ and $f(\mathbf{z}(t))\hskip 2.84526pt\forall\hskip 2.84526ptt$, we measure the Euclidean distance between each time sample, i.e., between $\mathbf{z}(t)$ and $f(\mathbf{z}(t))$, and populate a one-hot label vector $\mathbf{y}(t)\in\mathbb{R}^{2\times 1}$ for each sample. Specifically, for each time sample, $t$, we compute

with the corresponding label determined as

where $d_{\text{T}}$ is an empirically determined threshold from the training data, and $\mathbf{y}(t)$ indicates whether the corresponding time sample is excessively nonlinear.

Next, we train a DNN, $\zeta:\mathbb{R}^{2M\times 1}\rightarrow\mathbb{R}^{2\times 1}$, using $f(\mathbf{z}(t))$ and $\mathbf{y}(t)\hskip 2.84526pt\forall\hskip 2.84526ptt$ to classify the time samples, $f(\mathbf{z}(t))$, that are highly distorted from their linear counterparts. Note that the input to the DNN is $f(\mathbf{z}(t))$ since, during test time, we will only receive the nonlinear signal, and thus need to classify its time samples that have contributed to the nonlinearity to the greatest extent. We consider a $p$-layered DNN, consisting of $A$ units each, with a softmax output layer yielding the DNN prediction $[\hat{\mathbf{y}}_{1}(t),\hat{\mathbf{y}}_{2}(t)]$, where the input is predicted to be nonlinear if $\hat{\mathbf{y}}_{1}(t)<\hat{\mathbf{y}}_{2}(t)$. In place of a two-dimensional softmax output, the DNN could output a one-dimensional sigmoidal value. However, we empirically found that the former setup yields a more accurate nonlinear sample classifier.

Using $f(\mathbf{z}(t))$ and $\mathbf{y}(t)\hskip 2.84526pt\forall\hskip 2.84526ptt$ as training samples, the DNN minimizes the mean squared error loss function. At test time, each column (i.e., time sample) is forward propagated through the DNN, and time samples classified as highly distorted due to the nonlinearity are deleted. The edited matrices of $f(\mathbf{Z}_{\text{old}})$ and $f(\mathbf{Z}_{\text{new}})$, denoted as $\mathbf{Z}_{\text{old}}^{\prime}$ and $\mathbf{Z}_{\text{new}}^{\prime}$, respectively, correspond to $f(\mathbf{Z}_{\text{old}})$ and $f(\mathbf{Z}_{\text{new}})$ after their time samples that were classified as nonlinear by $\zeta$ were removed. $\mathbf{Z}_{\text{old}}^{\prime}$ and $\mathbf{Z}_{\text{new}}^{\prime}$ are then used to compute the GLRT test statistic in (5).

Here, we analyze the computational complexity of our proposed framework during inference. The GLRT statistic in (5) consists of $2$ $M\times k$ matrix Hermitian operations and $2$ $M\times k$ matrix multiplications followed by an $M\times M$ matrix inverse and an $M\times M$ matrix multiplication. Thus, the computational complexity of the GLRT is $\mathcal{O}(k^{2}+M^{2})$. Since the DNN pre-pended to the GLRT in our proposed framework consists of $p$ dense layers with $A$ units each, it adds an additional complexity of $\mathcal{O}(2(p+1)A^{3})$. This yields a total time complexity, for our method, of $\mathcal{O}(pA^{3}+k^{2}+M^{2})$. During inference, our method requires roughly $60$ ms to compute the test statistic in (5) for the simulation environment considered in Sec. III. Thus, despite the overhead incurred from the DNN, our method remains computationally feasible for practical use.

For our empirical evaluation, we generate an interference signal, $a_{1}$, and a SOI, $a_{2}$. Each signal contains a sequence of random bits, is modulated according to the BPSK constellation, and transmitted with a carrier frequency of $f_{c}=10$ MHz and a bandwidth of $1$ MHz. At the receiver, we use $M=4$ antenna elements and consider a received signal with an observation window of $L=2000$ time samples. In addition, we use $k=48$ time samples to construct each initial instance of $\mathbf{Z}_{\text{old}}$ and $\mathbf{Z}_{\text{new}}$. Here, we consider two setups by scaling the AWGN: (a) an interference to noise ratio (INR) of 40 dB and a signal to noise ratio (SNR) of 15 dB (corresponding to perhaps an RF application); and (b) an INR of 57 dB and an SNR of 37 dB (corresponding to perhaps an audio application). The large power difference between the SOI and interference introduces an additional challenge for detection, which is achievable in the linear case but much more challenging in nonlinearity.

In our analysis, we use the hyperbolic tangent function as the induced nonlinearity, i.e., we define $f(\mathbf{Z})=\text{tanh}(\mathbf{Z})$. This function is a representative model for hardware nonlinearities, since it consists of a linear and a nonlinear region, thus reflecting the effect of amplifiers when only certain received samples experience nonlinear saturation [14].

The DNN is trained on a generated received signal $\mathbf{Z}_{\text{Train}}$ and its nonlinear counterpart $f(\mathbf{Z}_{\text{Train}})$ to identify excessively nonlinear signals based on the method in Sec. II-D. The employed DNN contains $p=3$ layers with $A=10$ hyperbolic tangent units each. During training, we use a batch size of 64, a learning rate of $0.001$, and 500 epochs with a patience on the training loss of 25 epochs as the stopping criterion.

We evaluate our methodology on $f(\mathbf{Z}_{\text{Test}})$, and in addition, we generate the linear counterpart, $\mathbf{Z}_{\text{Test}}$, to compare the nonlinear and linear detection performance. During implementation, we scale the nonlinearity based on a factor $\alpha$ as $f(\mathbf{\alpha Z})/\alpha$ to achieve a desirable nonlinearity in each considered setup. Fig. 2 shows the GLRT test statistic on (i) a theoretical (linear) signal, (ii) a nonlinear signal, and (iii) the same nonlinear signal after DNN pre-processing for setups (a) and (b). The SOI turns on at $T_{0}=1000$. As shown by the metric on the linear signal, a distinctive contrast peak is captured at the time in which the SOI turns on in the received transmission. When the signal undergoes a nonlinearity, and $f(\mathbf{Z}_{\text{Test}})$ is received without DNN pre-processing, several false alarm peaks are produced at multiple incorrect time samples, and no distinctive contrast peak is captured at $T_{0}$ when the SOI turns on. When our proposed methodology is applied to the nonlinear signal, however, the GLRT test statistic produces a distinctive contrast peak correctly at $T_{0}=L/2$, albeit to somewhat of a lesser degree than in the theoretical case. This behavior stems from the remaining samples in $\mathbf{Z}_{\text{old}}^{\prime}$ and $\mathbf{Z}_{\text{new}}^{\prime}$ still containing nonlinear samples. Using the curves in Fig. 2, we vary the detection threshold and measure the resulting false positive versus true positive detection rates to obtain the performance curves shown next in Fig. 3.

Fig. 3 shows the receiver operating characteristic (ROC) detection performance curves of the GLRT for each setup. In addition, we compare our proposed method to two baselines: (i) end-to-end deep learning (EtE DL) [15], where a deep learning classifier is trained on both linear and nonlinear signals to perform detection directly in place of the GLRT; and (ii) data driven (DD) pre-processing [16], where the received nonlinear signal is first pre-processed in an attempt to recover the linear signal prior to applying the GLRT. In our adoption of [16], we train an autoencoder (AE) to map nonlinear signals (taken as input to the AE) to their linear counterparts (given as output by the AE). The GLRT test statistic in (5) is then applied on the AE output. Finally, for completion, we also employ our proposed DNN pre-processing methodology on a linear signal to show its effect when it is applied to a signal that does not suffer from nonlinear distortions.

From Fig. 3, we see that our method is able to improve the area under the curve (AUC) of the ROC curve in both cases over using the nonlinear signal without DNN pre-processing and the other considered baselines. The lower AUC, in comparison to the theoretical linear case, is attributed to the increased magnitudes of the false positive contrast peaks produced by the GLRT in Fig. 2. In particular, we note that our method significantly improves the true positive detection rate at low false alarm levels, which is often desirable in the context of wireless signal detection. The nearly random performance of the EtE DL classifier is attributed to the large power difference between the interference and SOI, resulting in a lack of salient features for the classifier to perform detection on. Similarly, DD pre-processing faces challenges in the training step since the same nonlinear clipped time sample corresponds to several linear time samples (i.e., a one-to-many mapping), thus resulting in poorly trained AE models. Our proposed method avoids the sensitivity of mapping non-linear time samples to an approximation of their linear counterparts by instead classifying and removing nonlinear samples that degrade detection performance and then applying the GLRT. Finally, we see that applying our proposed framework to a linear signal only slightly degrades performance in comparison to the theoretical linear performance. However, since we can deduce when a received signal has undergone non-linear distortions from hardware constraints, we can employ our proposed framework on non-linear signals while resorting to the classical GLRT on linear observations.