Ensemble Wrapper Subsampling
for Deep Modulation Classification

Our strategy employs a Convolutional Neural Network (CNN), a Convolutional Long Short-term Deep Neural Network (CLDNN), and a Residual Network (ResNet), whose details we provide below. For all architectures, we use the Adam optimizer and the categorical cross entropy loss function. We also use ReLu activation functions for all layers, except the last dense layer, where we use Softmax activation functions. Robustness and diversity were the key design factors that guided our choice of architectures. The former indicates that each architecture is well fit for the task, even at low sampling rates, which we verified through experimental results, and the latter indicates that the three architectures are independently trained and rely on different mechanisms for capturing task-relevant features. While a CNN relies on a fixed hierarchical representation that first extracts lower-level features through a large number of convolutional kernels, and then captures higher-level semantics through less kernels whose outputs have a wide input receptive field, the ResNet relies on shortcut connections between convolutional layers that are far apart, which provides stable training for deeper layers and allows for dynamically choosing the effective architecture while training (see [28, Chapters $8$-$9$] for more illustration). Also, the ResNet is significantly deeper than the CNN, which makes it likely to reach very different solutions. Further, unlike these two architectures, the CLDNN includes a gated LSTM layer that captures long-term temporal correlations in the convolutional output feature maps.

The first building block in our ensemble method that employs the architectures provided above, is a supervised wrapper feature selection algorithm that we call the Subsampler Net, which uses a deep neural network to rank the importance of each sample in accordance to the relative drop in classification accuracy that occurs when that sample is removed (set to 0). Similar to other wrapper feature selection methods, a Subsampler Net relies on a classifier to rank sample importance. We will refer to this classifier as the Ranker Model. We use pre-trained models for each of the above three architectures as Ranker Models to construct three separate Subsampler Nets.

Suppose we want to obtain $k$ samples from a pool of $d$ samples. As shown in Figure 3, the Subsampler Net first starts by setting a sample to $0$ (setting both the real and imaginary parts of that sample to $0$) in a batch of training validation examples and evaluating it using the Ranker Model, which will then provide us with a classification accuracy for that batch. Setting a sample to $0$ means that the input neurons to the model for the two features corresponding to that complex sample are dead, which allows us to simulate the removal of that sample from the signal as all the weights from these neurons in the input layer will not contribute to the outcome of the model. This is done $d$ times by setting each of the $d$ samples to $0$. After evaluating each of the $d$ samples, we are left with $d$ classification accuracies that correspond to the ability of the model to classify the signal if each of the samples were to be removed. The most important sample, which is the sample whose removal results in the lowest classification accuracy, is then permanently set to $0$ for this batch of training examples and added to the final set of samples. Now, we are left with $d-1$ samples and this process is repeated until we are finally left with $k$ samples. The Subsampler Net construction is detailed in Algorithm 1.

While attempting this method, we found that normalizing the data by setting the mean of each sample to $0$ and the variance to $1$ improves performance because when we set an input sample to $0$, we are effectively setting it to the mean, and lower variances now manifest as lower weights in the input layer [29]. We also observed, from experiments that rely on a discrete set of SNR values, that the sample indices chosen for batches of validation examples belonging to the same SNR value were the same in most cases, while they were likely to be distinct from those chosen for batches belonging to different SNR values. Therefore, we divide the available data according to SNR value and obtain a set of $k$ sample indices for each SNR.

We next introduce the notion of the Holistic Subsampler, which combines the ability of all three Ranker Models in order to capitalize on their diversity of performance. After the set of samples, that match the required sampling rate, are collected for each of the three Ranker Models, we divide these samples into three tiers.

The Venn diagram in Figure 4 illustrates the division of the tiers. Tier 1 consists of the intersection of all three sets of samples, Tier 2 consists of the samples that belong to two of the three sets of samples, and Tier 3 consists of samples that are selected by only one Subsampler Net. The samples within each of the tiers are sorted according to the sum of the classification accuracy values that occur when that sample is removed using the corresponding Ranker Models. For example, for Tier 2 samples, we sort the samples using the sum of the two obtained classification accuracy values when the sample is removed (set to 0). Akin to the Subsampler Net, the Holistic Subsampler is a recursive algorithm that first selects the best sample, then sets the value of the corresponding sample index to 0 for the whole training set, and calls itself again to find the next best sample. This is done until the desired number of samples $k$ is reached. To find the best sample, the top sample - corresponding to the lowest sum of classification accuracy values - is selected from Tier 1. If Tier 1 is empty, then the top sample from Tier 2 is selected. If Tier 1 and Tier 2 are both empty, then the top sample from Tier 3 is selected.

We note that the Subsampler Net merely selects the best sample at each iteration, without regard to how the selection of a subsequent sample will affect the importance of the currently selected sample to the classification task. We chose to do this in the interest of saving training time of Subsampler Nets in order to render the implementation of the proposed method feasible using low-power communication devices. To tackle this problem, we next propose a variant of the $\epsilon$-greedy algorithm [30] in order to explore candidate combinations for subsequent best samples while taking into account dependence relationships between the selected samples.

According to Algorithm 2, we introduce $\epsilon$, the exploration factor that determines the number of candidate samples considered for selection at each step. If $\epsilon=0.1$, then in every step, we explore the $10\%$ best samples according to the ranking provided by the Holistic Subsampler. This is unlike the conventional $\epsilon$-greedy algorithm, where $\epsilon$ represents the probability that the decision taken deviates from the top greedy choice. Our variant of the algorithm explores all of the top routes and $\epsilon$ is the parameter that determines the number of top routes explored. The Multi-Armed Bandit problem [31], which is one of the most popular applications of the traditional $\epsilon$-Greedy Algorithm, is based on a scenario where there is a single top choice and all the other choices are of the same importance before exploration. However, in our case, the choices apart from the top choice have a ranking that reflects their relative importance. Therefore, unlike the Multi-Armed Bandit problem, we do not need to waste time exploring unknown paths, and this is the rationale behind our deviation from the conventional $\epsilon$-Greedy Algorithm.

As observed in Figure 5, the $\epsilon$-Greedy Search can be represented as a traversal algorithm over an $\epsilon d$-ary tree whose depth is equal to the desired number of samples $k$, where the subsampling rate is $\frac{k}{d}$. Each node in the tree corresponds to the selection of a combination of sample indices. The nodes at the same depth are arranged by increasing order of accuracy when removed, which implies that the combination of samples corresponding to the left child of a node has higher priority than that corresponding to the right child of the same node, for the case when $\epsilon d=2$. The root node does not represent any sample and is added just for the sake of illustration of how the tree is formed. The leaf nodes are searched from left to right until a classification accuracy that is satisfactory is reached.

We note that setting $\epsilon=\frac{1}{d}$ is equivalent to having an unaltered Holistic Subsampler that greedily chooses the best sample at each iteration, which leads to the leftmost leaf of the tree in Figure 5. This corresponds to node $0000$ because only a tree with a single branch is formed with the nodes 0, 00, 000, and 0000. Increasing the value of $\epsilon$ expands the scope of exploration.

We here finalize the specification of the proposed approach. Given input data with $d$ samples per example, we first initialize $\epsilon$ to $\frac{1}{d}$ and invoke the $\epsilon$-Greedy Search. As mentioned earlier, this is the same as invoking the Holistic Subsampler on its own. Next, we proceed to the next SNR value available in the training set. The $\epsilon$-Greedy Search is invoked with an $\epsilon$ value of $\frac{1}{d}$ for this next training set belonging to the next SNR value. If the accuracy is lower than the accuracy for the previous SNR value, then $\epsilon$-Greedy Search is invoked again after doubling the $\epsilon$ value to $\frac{2}{d}$. The $\epsilon$-Greedy Search stops exploring once the accuracy is greater than the accuracy obtained for the previous SNR value. We repeatedly double $\epsilon$ and invoke $\epsilon$-Greedy Search until this stopping criterion is met. The pseudocode for this strategy is given in Algorithm 3.

The function described in Algorithm 3 returns the selected set of $k$ sample indices for each SNR value. Note that douling the value of $\epsilon$ for the $\epsilon$-Greedy Search corresponds to searching for combinations of sample indices in a tree that has twice the arity.

We use the RadioML2016.10b synthetic dataset generated in [23] as the input data. Details about the generation of this dataset can be found in [32]. Figure 6 shows a high-level framework for the data generation process. For digital modulations, the entire Gutenberg works of Shakespeare in ASCII is used, with whitening randomizers applied to ensure equiprobable symbols and bits. For analog modulations, a continuous voice signal consisting of acoustic voice speech with some interludes and off times is used as input. The modulation rate is 8 samples per symbol.

The dataset is split equally among all ten considered modulation types. For the channel model, physical environmental noises like thermal noise and multipath fading were simulated. The models for generating random channel and device imperfections, that determine the parameters in (2), are detailed in [32]¹¹1Dataset generation parameters are also available at https://github.com/radioML/dataset. When packaging data, the output stream of each simulation is randomly segmented into vectors as the original dataset with a sample rate of 1M samples per second. Similar to the way that an acoustic signal is windowed in voice recognition tasks, a sliding window extracts 128 samples with a shift of 64 samples, which forms a sample vector in the dataset. 160,000 sample vectors generated using the GNU-radio library developed in [32] are segmented into training and testing datasets. Each example consists of 128 samples, that are represented as a 2$\times$128 vector with real and imaginary parts separated. The SNR of the samples is uniformly distributed from -20 dB to 18 dB, with a step size of 2 dB, i.e., the dataset is equally split among all SNR dB values in $\{-20,-18,-16,\cdots,18\}$.

We note from the frequency domain representation of the input waveform depicted in Fig. 7 that the sampling rate of the input waveform is around 6 times the Nyquist rate.

We used Keras with TensorFlow as a backend, and a GPU server equipped with three Tesla P100 GPUs with 16 GB memory. For all architectures, we used a batch size of 1024, and a learning rate of 0.001. Only the training set is used by the subsampling algorithm described in Section III with a validation split of 0.25. After selecting the set of sample indices for each of the 20 considered SNR values, we train the ResNet classifier with the corresponding samples, as we found it to deliver the best performance among the three considered architectures²²2Code is available at: https://github.com/dl4amc/dds.

We present the obtained results at different sampling rates in Figure 8. From our results, we note the following.

• •

  Subsampling can lead to higher accuracy: Applying the proposed ensemble wrapper subsampling strategy can result in dramatic improvements in classification accuracy, particularly at low SNR values. The direct cause for this phenomenon at high SNR is resolving confusions between the QAM16/QAM64 and AM-DSB/WBFM pairs, as we illustrate in Section V. To the best of our knowledge, state-of-the-art methods fail at resolving these confusions. We believe that this is because our subsampling strategy reduces overfitting, as we elaborate in Section VI-B.

• •

  Sub-Nyquist Sampling: As noted above, the considered data is originally sampled at around 6 times the Nyquist rate. A subsampling rate of $\frac{1}{16}$ hence corresponds to around $37.5\%$ of the Nyquist rate, and leads to slightly higher classification accuracy at very high SNR and significantly higher accuracy at very low SNR, than the case with no subsampling. This observation could carry important implications in practice, as the sampling hardware requirements can be dramatically simplified (see [33] for more illustration).

• •

  Minimal Sample Set: Based on the loss in classification accuracy, we can choose the smallest set of samples (smallest value of $k$) that gives us a classification accuracy higher than a given classification accuracy requirement. For instance, $20$ is the smallest number of samples (around Nyquist rate) that can be selected such that the classification accuracy has to be higher than $99\%$ at $18$ dB SNR. Similarly, $8$ is the smallest number of samples $\left(\text{around }37.5\%\text{ of Nyquist rate}\right)$ that can be selected given that the classification accuracy has to be higher than $90\%$ at $18$ dB SNR.

As a result of subsampling, the training time of the classifier is reduced due to the reduced input dimensions. We show the reduction in training times and high SNR classification accuracy of the ResNet classifier for different subsampling rates in Table I. Note that a subsampling rate as low as $\frac{1}{16}$, which corresponds to the sub-Nyquist regime with around $37.5\%$ of the Nyquist rate, and results in approximately $\frac{1}{3}$ of the original training time, still results in a classification accuracy higher than that without subsampling.

We first discuss the performance of a Gaussian naive Bayes classifier, to assess the difficulty of the considered modulation classification task. It will then become clear how deep learning significantly outperforms the naive Bayes classifier. The Gaussian naive Bayes classifier can be described through the conditional probabilities:

where $P(x_{i}|y)$ is the likelihood of an observed instance $x_{i}$ belonging to a certain class $y$, $\sigma_{y}^{2}$ is the observed variance of class $y$, and $\mu_{y}$ is the observed mean of class $y$. The predicted output of $x_{i}$ is the class that maximizes the likelihood function. Instead of trying to classify all ten modulation types, we only used certain pairs to further demonstrate the performance of the Gaussian naive Bayes classifier for simpler tasks.

From the results in Table II, we note that even when the Bayes classifier is trained to distinguish pairs that are not challenging at the maximum SNR value, its maximum accuracy is $82\%$. Further, for challenging pairs, the performance is similar to random guessing even at high SNR.

We present in Figure 9 the classification accuracy of the considered architectures using the considered dataset with no subsampling. We note that - similar to previous work on deep learning for modulation classification - most of the misclassifications at high SNR are due to confusions between the AM-DSB/WBFM and the QAM16/QAM64 pairs, which is evident from the ResNet 18 dB confusion matrix depicted in Figure 9a. We observe by comparing to Figure 8 how the proposed data-driven subsampling method leads deep neural network classifiers to clear this confusion. We believe that this is due to overfitting reduction, as further illustrated in Section VI. We further note that we also considered a pure LSTM architecture by fine tuning that of [34] for the task. Even though this architecture delivered good performance with no subsamlping as shown in Figure 9b, we chose not to use it in our proposed method as it suffered drastic performance degradation with subsampling. We believe that this is due to extreme sensitivity of the captured temporal correlations to absence of few samples.

We provide in Figure 10a a comparison between the proposed approach and four different subsampling techniques; namely: 1) Uniform Subsampling: where a sample is taken every fixed amount of time, 2) Random Subsampling: where the indices of selected samples is determined randomly with equal probabilities, 3) Magnitude Rank Subsampling: where the indices corresponding to samples with top magnitude values in each example are selected, and 4) Principal Component Subsampling (PCS), where first Principal Component Analysis (PCA) is done over the training set, and the indices corresponding to samples with the top PCA coefficient total magnitude values are selected. A more thorough explanation of these methods is available in [35]. We note that the proposed method leads to - uniformly across all SNR values - superior performance than all these methods. The figure demonstrates results slightly below the Nyquist rate, but this observation extends to all considered subsampling rates. In particular, random subsampling delivers better performance that the other three methods at lower rates, which agrees with the intuition in [33], but this performance remains lower than that of the proposed approach.

Feature selection algorithms aim at identifying important input vector elements. In the considered setup, a direct application of a feature selection algorithm for $2\times 128$ input vectors, would treat each of the $256$ elements separately. We handle this with a slight modification to tie the real and imaginary parts of each sample, as illustrated below. In Figure 10b, we show a comparison between the proposed method and four popular feature selection algorithms; namely: 1) Laplacian Score [36]: which is an unsupervised filter feature selection algorithm that selects features with the objective of preserving the data manifold structure through a graph representation [37], 2) Fisher Score [38]: which is a supervised filter feature selection algorithm that selects features such that the features of samples within the same modulation type are similar while the features of samples belonging to other modulation types are as distinct as possible [37], 3) Efficient and Robust Feature Selection (RFS) [39]: which is a computationally efficient embedded feature selection method that exploits the noise robustness - through rotational invariance - property of the joint $\ell_{2,1}$-norm loss function [40, 41], by applying the $\ell_{2,1}$-norm minimization on both the loss function and its associated regularization function, and 4) Feature Quality Index (FQI) [42]: which is a wrapper feature selection algorithm that utilizes the output sensitivity of the considered model to changes in the input to rank features. FQI can be considered as a simplified version of our Subsampler Net that relies on the Mean Squared Error (MSE) loss instead of the model’s loss and uses only the initial sample ranking that we use to select the first sample. For each of the techniques examined, except FQI where we have sample scores, we add the two scores obtained for the two features belonging to a sample to obtain a sample score before proceeding to rank the samples. We note how the proposed method delivers significantly better performance than all considered methods, and that this holds for all other considered subsampling rates not shown in the figure. Particularly, our method delivers better performance than FQI, which demonstrates the need for re-ranking samples after each iteration in the Subsampler Net described in Section III-B. Also, we note that this re-ranking does not require model retraining, unlike most other wrapper feature selection algorithms, which makes our method computationally feasible in a wide range of settings.

We observe from Figure 11 how the relative performance of the three considered Subsampler Nets changes with different sampling rates and SNR values. This is the main motivation behind the Holistic Subsampler that benefits from the performance diversity among the three architectures. However, even the Holistic Subsampler, suffers from significant drops in classification accuracy for a wide range of SNR values at sampling rates well below the Nyquist rate. This motivated our $\epsilon$-Greedy step of the proposed approach. In particular, the slope of the classification accuracy curve for the Holistic Subsampler becomes negative towards -$12$, $4$, and $8$ dB with a $\frac{1}{16}$ subsampling rate and towards -$12$, $2$, and $10$ dB with a $\frac{1}{32}$ subsampling rate. To obtain the results shown in Figure 8, our ensemble wrapper data-driven subsampling algorithm, illustrated in Algorithm 3, applied $\epsilon$-Greedy Search with $\epsilon=\frac{1}{64}=\frac{2}{d}$ for the $4$ and $8$ dB SNR values with $\frac{1}{16}$ subsampling rate, as well as at $10$ dB SNR with $\frac{1}{32}$ subsampling rate. For the -$12$ dB SNR value with both $\frac{1}{16}$ and $\frac{1}{32}$ subsampling rates, an $\epsilon=\frac{1}{32}$ had to be used because having $\epsilon=\frac{1}{64}$ was insufficient, and the same held for $2$ dB SNR with $\frac{1}{32}$ rate. It is important to note that our $\epsilon$-Greedy step has a time complexity of the Order $O((\epsilon d)^{k})$, if the number of explored combinations has the same order as the number of the constructed tree leaves. Fortunately, this step is typically needed only at very low subsampling rates, where the value of $k$ is small, and with small values of $\epsilon$.

We note how the Holistic Subsampler achieves better results than any individual Subsampler Net, even though the final classifier relies on a single ResNet architecture. Furthermore, even though each of these architectures is trained to classify the data when all the samples are present at the input, when used as Subsampler Nets, one or more of these samples are set to 0. Hence, we use the trained deep neural network classifiers in two ways other than their intended application that they are trained on: 1- They are used to select samples for another classifier, 2- They are used with only a subset of samples present. This is only possible because of the transferability property of these deep neural network architectures. In general, we believe that exploiting transferability has great potential for various wireless communication tasks that rely on processing received signal samples.

In Section V-B, we saw that all the deep learning architectures suffered from the same drawback of the AM-DSB/WBFM and QAM16/QAM64 misclassification when no subsampling is used. To further analyze why the proposed subsampling method leads to higher classification accuracies with fewer samples, we will be using Principal Component Anaysis (PCA) [43] and t-Distributed Stochastic Neighbor Embedding (t-SNE) [44] to visualize how subsampling allows us to reduce overfitting, particularly for the aforementioned class pairs. We first subsample the training dataset at $18$ dB SNR with a rate of $1/2$. After subsampling, we have $64$ samples, corresponding to $128$ features. Finally, we implement PCA and t-SNE to obtain a $3$-Dimensional projection of the training dataset for better visualization. We chose to implement both PCA and t-SNE because PCA clarifies the distinction between AM-DSB and WBFM while t-SNE clarifies the distinction between QAM16 and QAM64 after subsampling, as shown in Figure 12³³3The t-SNE plots were generated with a perplexity value of $20$, a learning rate of $10$, and were run for $250$ iterations.. Observing the figure, we believe that the higher accuracy values stem from the subsampling strategy enabling simpler decision boundaries to distinguish, with high fidelity, between the considered class pairs, which improves generalization performance and reduces overfitting.

The performance of a Subsampler Net heavily depends on the performance of the model used to rank the features. In some cases, however, even the state-of-the-art models do not have high classification accuracy values. In such cases, we believe that better feature selection results can be obtained with a Subsampler Net by training the ranker model for more epochs beyond what is suggested by the Early Stopping algorithm (see e.g., [28, Chapter $8$]). This is as we found this strategy to be useful in multiple settings and further observed that it can significantly increase the discrepancy in weight magnitudes across the different features. For example, we considered a toy example constructed in TensorFlow Playground with a single-hidden-layer network that distinguishes between two classes based on two features, and the second is more salient as it enables forming a decision boundary that allows for better classification. Each of these features have three weights in the input layer and as expected, the weights connected to Feature $2$ quickly manifest, after several epochs, into weights of higher average magnitude than those belonging to Feature $1$ as shown in Fig. 13. As the number of training epochs increases - even from $100$ to $1000$ which is way more than needed for this small network - the difference in the average magnitudes of the weights increases. This implies that the ranker will be able to better rank the saliency of features because the accuracy difference will increase when suppressing each of these features.

Even though both the final classifier and the ranker models used in the proposed subsampling method are trained using the whole training set across all considered SNR values, the selected set of sample indices is different for each SNR value, and hence, we expect a real time system employing this method to have an accurate estimate of the SNR value, in order to know the right set of sample indices. We made this choice, as we found it to deliver a significantly superior performance to the extreme alternative, where the same set of sample indices is selected for all SNR values. In future work, we plan to investigate the impact of small errors in such an estimate, by comparing the different sets of selected sample indices for adjacent SNR values. We plan to also benefit from analyzing these sets of sample indices, to better understand the roles of different ranker models at different SNR values.