SR2CNN: Zero-Shot Learning for Signal Recognition

Signal is a special data type, which is very different from image. While it is easy to give a description of semantic attributes of images in terms of visual information, extracting semantic features of signals without relying on any computation is almost impossible. Hence, a natural way to automatically extract the semantic information of signal data is using feature extractor networks $F$. Considering about the unique features of signals, the input shape of $F$ should be a rectangle matrix with 2 rows rather than square matrix. In our scheme, $F$ consists of four convolutional layers and two fully connected layers.

Generally, $F$ can be represented by a mapping from the input space $X$ to the latent semantic space $Z$. In order to minimize the intra-class variations in space $Z$ while keeping the inter-classes’ semantic features well separated, center loss [46] is used. Let $x_{i}\in X$ and $y_{i}$ be the label of $x_{i}$, then $z_{i}=F(x_{i})\in Z$. Assuming that batch size is $N$, the center loss is expressed as follows:

where $c_{y_{i}}$ denotes the semantic center vector of class $y_{i}$ in $Z$ and the $c_{y_{i}}$ needs to be updated as the semantic features of class $y_{i}$ changed. Ideally, entire training dataset should be taken into account and the features of each class need to be averaged in every iterations. In practice, $c_{y_{i}}$ can be updated for each batch according to $c_{y_{i}}\leftarrow c_{y_{i}}-\alpha\Delta_{c_{y_{i}}}$, where $\alpha$ is the learning rate and $\Delta_{c_{y_{i}}}$ is computed via

where $\delta(\cdot)=1$ if the condition inside $()$ holds true, and $\delta(\cdot)=0$ otherwise.

The classifier $C$ will discriminate the label of samples based on semantic features. It consists of several fully connected layers. Furthermore, cross entropy loss $L_{ce}$ is utilized to control the error of classifier $C$, which is defined as

where $C(F(x_{i}))$ is the prediction of $x_{i}$.

Further, auto-encoder [47, 48, 49] is used in order to retain the effective semantic information in $Z$. As shown in the right part of Fig 2, decoder $D$ is used to reconstruct $X$ from $Z$. It is made up of deconvolution, unpooling and fully connected layers. Among them, unpooling is the reverse of pooling and deconvolution is the reverse of convolution. Specifically, max unpooling keeps the maximum position information during max pooling, and then it restores the maximum values to the corresponding positions and set zeros to the rest positions as shown in Fig. 3(a). Analogously, average unpooling expands the feature map in the way of copying it as shown in Fig. 3(b).

The deconvolution is also called transpose convolution to recover the shape of input from output, as shown in Fig. 3(c). See appendix A for the detailed convolution and deconvolution Operation, as well as toy examples.

In addition, reconstruction loss is utilized to evaluate the difference between original signal data and reconstructed signal data.

where $D(F(x_{i}))$ is the reconstruction of signal $x_{i}$. Intuitively, the more complete signal is reconstructed, the more valid information is carried within $Z$. Thus, the auto-encoder greatly helps the model to generate appropriate semantic features.

As a result, the total loss function combines cross entropy loss, center loss and reconstruction loss as

where the weights $\lambda_{ct}$ and $\lambda_{r}$ are used to balance the three loss functions. We have carefully designed the total loss function. The cross entropy loss is used to learn information from labels. And center loss minimizes the intra-class variations in the semantic space while keeping the inter-classes’ semantic features well separated, which also helps unknown classes to separate. Reconstruction loss makes model learn more information about signal data, because of well-reconstructed data. Ablation study in Section V also validates the above points. The whole learning process with loss $L_{t}$ is summarized in Algorithm 1, where $\theta_{F}$, $\theta_{C}$, $\theta_{D}$ denote the model parameters of the feature extractor $F$, the classifier $C$ and the decoder $D$, respectively.

The discriminator $P$ is the tail but the core of the proposed framework. It discriminates among known and unknown classes based on the latent semantic space $Z$. For each known class $k$, the feature extractor $F$ extracts and computes the corresponding semantic center vector $S_{k}$ as:

where $m$ is the number of all training samples. When a test signal $\mathcal{I}$ appears and $F(\mathcal{I})$ is obtained, the difference between the vector $F(\mathcal{I})$ and $S_{k}$ can be measured for each $k$. Specifically, the generalized distance between $F(\mathcal{I})$ and $S_{k}$ is used, which is defined as follows:

where $A_{k}$ is the transformation matrix associated with class $k$ and $A_{k}^{-1}$ denotes the inverse of matrix $A_{k}$. When $A_{k}$ is the covariance matrix $\Sigma$ of semantic features of signals of class $k$, $d(\cdot,\cdot)$ is called Mahalanobis distance. When $A_{k}$ is the identity matrix²²2This is also the only possible choice in the case when the covariance matrix $\Sigma$ is not available, which happens for example when the signal set of some class is singleton. $I$, $d(\cdot,\cdot)$ is reduced to Euclidean distance. $A_{k}$ also can be $\Lambda$ and $\sigma^{2}I$ where $\Lambda$ is a diagonal matrix formed by taking diagonal elements of $\Sigma$ and $\sigma^{2}\triangleq\frac{trace(\Sigma)}{t}$ with $t$ being the dimension of $S_{k}$. The corresponding distance based on $A_{k}=\Lambda$ and $A_{k}=\sigma^{2}I$ are called the second distance and third distance. Note that when the Mahalanobis distance, second distance and third distance are applied, the covariance matrix of each known class needs to be computed in advance.

With the above distance metric, we can establish our discriminant model which is divided into two steps. Firstly, distinguish between known and unknown classes. Secondly, discriminate which known classes or unknown classes the test signal belongs to. The first step is done by comparing the threshold $\Theta_{1}$ with the minimal distance $d_{1}$ given by

where $S$ is the set of known semantic center vectors. Let us denote by $y_{\mathcal{I}}$ the prediction of $\mathcal{I}$. If $d_{1}<\Theta_{1}$, $y_{\mathcal{I}}\in K$, otherwise $y_{\mathcal{I}}\in U$. Owing to utilizing the center loss in training, the semantic features of signals of class $k$ are assumed to obey multivariate Gaussian distribution. Inspired by the three-sigma rule [50], we set $\Theta_{1}$ as follows

where $\lambda_{1}$ is a control parameter referred to as the discrimination coefficient.

Two remarks are made as follows to explain the Gaussian distribution assumption and the choice of $\Theta_{1}$, respectively.

The second step is more complicated. If $\mathcal{I}$ belongs to the known classes, its label $y_{\mathcal{I}}$ can be easily obtained via

Obviously the main difficulty lies in dealing with the case when $\mathcal{I}$ is classified as unknown in the first step. To illustrate, let us denote by $R$ the recorded unknown classes and define $S_{R}$ to be the set of the semantic center vectors of $R$. In this difficult case with $R\subseteq\varnothing$, a new signal label $R_{1}$ is added to $R$ and $F(\mathcal{I})$ is set to be the semantic center vector $S_{R_{1}}$. The unknown signal $\mathcal{I}$ is saved in set $G_{R_{1}}$ and let $y_{\mathcal{I}}=R_{1}$. While in the difficult case with $R\not\subseteq\varnothing$, the threshold $\Theta_{2}$ is compared to the minimal distance $d_{2}$ which is defined by

Intuitively, a good choice of $\Theta_{2}$ may be made based on the distance between $F(x)$ and $S_{k}$’s. $d_{1}$ is the minimum distance which is firstly used in our test of choice of $\Theta_{2}$. Actually, we test a set of choices of $\Theta_{2}$ and numerically find that unknown classes can be often correctly identified when $\Theta_{2}$ is set between $d_{1}$ and $d_{med}$, where $d_{med}$ is the median distance between $F(x)$ and each $S_{k}$. Therefore, the threshold $\Theta_{2}$ is finally set as

where $\lambda_{2}$ is used to balance the two distances $d_{1}$ and $d_{med}$.

To proceed, let $n_{R}$ denote the number of recorded signal labels in $R$. Then, if $d_{2}>\Theta_{2}$, a new signal label $R_{n_{R}+1}$ is added to $R$ and set $y_{\mathcal{I}}=n_{R}+1$. Note that we don’t impose any prior restrictions on the value of $n_{R}$ (the size of set $R$), i.e., our model can never know the number of the unknown classes pending to be discriminated. Then if $d_{2}\leq\Theta_{2}$, we set

and save the signal $\mathcal{I}$ in $G_{y_{\mathcal{I}}}$. Accordingly, $S_{y_{\mathcal{I}}}$ is updated via

where $\#(G_{y_{\mathcal{I}}})$ denotes the number of signals in set $G_{y_{\mathcal{I}}}$. As a result, with the increase of the number of predictions for unknown signals, the model will gradually improve itself by way of refining $S_{R_{u}}$’s.

To summarize, we present the whole procedure of the discriminator in Algorithm 2. We emphasize that our SR2CNN is different from the common open-set recognition methods. Assuming that there are $n$ known classes and an uncertain number of unknown classes, the traditional open-set recognition method will only distinguish the test samples into $n+1$ classes, while SR2CNN will distinguish the test samples into $n+n_{R}$ classes via Algorithm 2, where $n_{R}$ is the number of unknown classes recognized by the discriminator. Specifically, for the case when a test sample belongs to an unknown class, we determine whether it belongs to an existing unknown class or a new unknown class by comparing $d_{2}$ with threshold $\Theta_{2}$. Hence, the notable advantage of SR2CNN over the common open-set recognition method lies in that SR2CNN can roughly distinguish how many unknown classes there are in the test set, not just label the test sample as unknown.

Basically, the average softmax accuracy of the known test set will converge roughly to $80\%$ on both 2016.10A and 2016.10B, while to $94\%$ on 2016.04C, as indicated in Fig. 4. Note that there’s almost no perceptible loss on the accuracy when using the clustering approach (i.e., the distance measure-based classification method described in Section IV) to predict instead of softmax, meaning that the semantic feature space established by our SR2CNN functions very well. For ease of exposition, we will refer to the known cluster accuracy as upbound (UB).

During the training course, the cross entropy loss undergoes sharp and violent oscillations. This phenomenon makes sense, since the extra center loss and reconstruction loss will intermittently shift the learning focus of the SR2CNN.

The most critical results are presented in Table II. To better illustrate it, we will firstly make a few definitions in analogy to the binary classification problem. By superseding the binary condition positive and negative with known and unknown respectively, we can similarly elicit true known (TK), true unknown (TU), false known (FK) and false unknown (FU). Subsequently, we get two important indicators as follows:

Furthermore, we define precision likewise as follows:

where $S_{correct}$ denotes the total number of known samples that are classified to their exact known classes correctly. $U_{dominantly\_correct}$ denotes the total number of unknown samples that are classified to their exact newly-identified unknown classes correctly. For evaluation, the real label of a certain newly-recorded unknown class is determined as the label of the most signal samples in that class. Note that sometimes unexpectedly, our SR2CNN may classify a small portion of signals into different unknown classes but their real labels are actually identical and correspond to one certain unknown class (we name these unknown classes as isotopic classes). In this rare case, we only count the identified unknown class with the highest accuracy in calculating $U_{dominantly\_correct}$.

For ZSL, we test our SR2CNN with several different combinations of aforementioned parameters $\lambda_{1}$ and $\lambda_{2}$, hopefully to snatch a certain satisfying result out of multiple trials. Fixing $\lambda_{2}$ to 1 simply leads to fair performance, though still, we adjust $\lambda_{1}$ in a range between 0.05 and 1.0. Here, the pre-defined indicators above play an indispensable part to help us sift the results. Generally, a well-chosen result is supposed to meet the following requirements: 1. the weighted true rate (WTR):  $0.4\times$TKR+$0.6\times$TUR  is as great as possible; 2. KP$>0.95\times$UB, where UB is the upbound defined as the known cluster accuracy; 3. $\#^{j}_{isotopic}<=$2 for all possible $j$, where $\#^{j}_{isotopic}$ denotes the number of isotopic classes corresponding to a certain unknown class $j$.

In order to better make a transverse comparision, we compute two extra indicators, average total accuracy in ZSL scenario and also average known accuracy in completely supervised learning, shown as italics in Table II. On the whole, the results are promising and excellent. However, we have to admit that ZSL learning somewhat incurs a little bit performance loss as compared with the fully supervised model. Looking vertically, among all modulations, the performance loss especially occurs in the class AM-DSB. While looking horizontally among all datasets, the performance loss especially occurs in dataset 2016.10B. After all, when losing sight of the two unknown classes, SR2CNN can only acquire a segment of the intact knowledge that shall be totally learned in a supervised case. It is this imperfection that presumably leads to an apparent variation on each class’s accuracy when compared with supervised learning. Among these classes, the poorest victim is always AM-DSB, with considerable portion of its samples rejected as unknown ones. Besides, the features, especially those of the unknown classes, among these three datasets are not exactly in the same difficulty levels of learning. Some unknown features may even be similar to those known ones, which can consequently cause confusions in the discrimination tasks. It is no doubt that these uncertainties and differences in the feature domain matter a lot. Take 2016.10B, compared with its two counterparts, it emanates the greatest loss (more than 10%) on average accuracy (both total and known), and also a pair of inferior true rates. Moreover, it is indeed the single case, where both two unknown classes are separately identified into two isotopic classes.

It is obvious that average accuracy strongly depends on the weighted true rate (WTR). Since the clearer for the discrimination between known and unknown, the more accurate for the further classification and identification. Therefore, to better study this discrimination ability, we depict Fig. 5 to elucidate its variation trends regarding discrimination coefficient ($\lambda_{1}$). At the same time, we introduce a new concept discrimination interval as an interval where the weighted true rate is always greater than 80%. The width of the above interval is used to help quantify this discrimination ability.

Apparently, the curves for the primary two kinds of true rate are monotonic, increasing for the known while decreasing for the unknown. The maximum points of these weighted true rate curves for each dataset, are about 0.4, 0.2, and 0.4 respectively. These points exactly correspond to the results shown in Table II. Besides, the width of the discrimination interval of 2016.10B is only approximately one third of those of 2016.10A and 2016.04C. This implies that the features of 2016.10B are more difficult to learn, and just accounts for its relatively poor performance.

In this subsection, we explain the necessity of each of the three loss functions. Relevant experiments are mainly based on 2016.10A.

Fig. 6 presents the known accuracy in absence of cross entropy loss, center loss and reconstruction loss respectively during training. In general, we found that the best performance in training will be degraded when any one of these three loss functions is excluded. It can be observed that both cross entropy loss and reconstruction loss make a positive impact on the known accuracy, boosting about 3% to 5%, while center loss seems slightly weaker.

Analyzing Table III, we can easily discern the effect of these three loss functions in the test course, especially the center loss. Results show that the F1 score in absence of cross entropy loss, center loss and reconstruction loss decreases by 1.8%, 1.7% and 2.0% respectively for the known classes. For the unknown classes, the minimum degradation in F1 score is 0.8% after removing cross entropy loss, while the maximum degradation in F1 score is 16.7% after removing center loss. Actually, Fig. 7 indicates that the usage of center loss on 2016.10A indeed helps our model to discriminate more distinctly, resulting in a notably broader discrimination interval. Besides, we have also made an attempt at applying L1 loss [52] to calculate center loss (Eq. (2) in Section IV) and reconstruction loss (Eq. (5) in Section IV). Those related results are presented in the last column of Table III. It is seen that L1 loss can indeed slightly increase the F1 score of known classes by 0.8%, however, at the cost of a decrease in the F1 score of unknown classes by 6.4%.

In sum, the three loss functions, though not exactly promoting our SR2CNN in the same way and in the same fields, are indeed useful.

We tentatively change several unknown classes on 2016.10A, seeking to excavate more in the feature domain of data. As shown in Table IV, both known precision (KP) and unknown precision (UP) are insensitive to the change of unknown classes, proving that the classification ability of SR2CNN are consistent and well-preserved for the considered dataset. Nevertheless, obviously, the unknown class CPFSK is always the hardest obstacle in the course of discrimination. Since accuracy of CPFSK is always the lowest as well as some isotopic classes are observed in this case. Especially, when class CPFSK and GFSK simultaneously show up in the unknown roles, the performance loss (on both TKR and TUR) is quite striking. We speculate that the unknown CPFSK and GFSK may share a considerable number of similarities with some known classes, which will unluckily mislead SR2CNN about the further discrimination task.

To justify SR2CNN’s superiority, we compare it with a couple of traditional methods prevailing in the field of outlier detection, as well as two open-set recognition methods, i.e., OpenMax [8] and MDL4OW [43]. For outlier detection methods, the detected outlier will be regarded as an unknown sample. For OpenMax, an extra dimension is appended to the output vector to indicate the probability of the current sample being unknown. While for MDL4OW, the extreme value theory is adopted to detect the unknown classes by modeling the distribution of loss. The results are presented in Table V. It is found that our SR2CNN significantly outperforms both outlier detection methods and open-set recognition methods in terms of the true unknown rate. Furthermore, we find that most of the aforementioned methods cannot correctly identify GFSK as unknown. For example, in our experiment, OpenMax wrongly classifies all GFSK samples as known. As for MDL4OW, it identifies a small percentage of GFSK samples at the cost of true known rate. However, it can be found from the experiment results that our SR2CNN can still work very well for this open-set recognition task.

Note that there are no unknown classes identification tasks launched, only discrimination tasks are considered. Hence, here, for a certain unknown class $j$, we compute its unknown rate, instead of accuracy, as $\frac{\#^{j}_{unknown}}{N_{j}}$, where $N_{j}$ denotes the number of samples from unknown class $j$, while $\#^{j}_{unknown}$ denotes the number of samples from unknown class $j$, which are discriminated as unknown ones.

In addition, relevant ROC curves for the above comparison experiments are depicted in Fig. 8. It is observed that SR2CNN has the largest AUC, indicating its superiority over other methods. Besides, notably, there seems as if a steep ‘cliff erecting’ where False Known Rate approximately equals to 0.5, particularly for EllipticEnvelope, LocalOutlierFactor, and OpenMax. This means that almost half samples of unknown classes are not easy to be correctly discriminated. Correspondingly, according to Table V, we can speculate that these ‘hard’ samples all come from unknown class GFSK.