RTFN: Robust Temporal Feature Network

The attention mechanism aims at mapping a given query and a set of key-value pairs to an output, where the output is a weighted sum of the values. For each value, its associated weight is calculated based on a compatibility function of the query with the corresponding key [10]. For any input data, attention can mine different positions to derive the representation of input data.

Just like our eyes, the attention mechanism focuses on the sensitive information we look for, capture the areas we are interested in, and discover rich features and relationships among them. Its use is to discover the hidden information and relationships that classical networks are unaware of, which helps to further complement the time-series and semantic features. Attention has been applied successfully in various areas, such as object detection [11], image caption [12], activity recognition [13], speech recognition [14], question answering [15] and software-defined networking [16].

LSTM is a widely used recurrent neural network. For a sequential model, LSTM can explore the features of input sequence by using specific internal units, and make use of these features to mine more underlying information and relationships from the internal structure [17]. Besides, the internal memory units in LSTM store the new features transmitted to the next processing units. In this way, when the features being transmitted are forgotten, they can be recovered by those stored in the internal memory units. Hence, LSTM can obtain abundant features from input data. Recently, LSTM has been incorporated into various network structures for addressing time-series tasks, for instance, ERA-LSTM [5], FCN-LSTM [6] and multivariate FCN-LSTM [18].

Motivated by the research related to the attention mechanism and LSTM, we design a robust temporal feature network that integrates LSTM into an attentional learning framework, which is named as attentional LSTM network. By taking advantages of the two techniques, the proposed network is able to not only explore long and short term temporal features, but also focus on different positions of a sequence to capture the shapelets hiding in the input data.

The structure of RTFN is shown in Fig. 1. It primarily consists of temporal feature networks and attentional LSTM networks. The former is based on residual networks and multi-head convolution neural networks, responsible for capturing basic temporal features from the input time-series data. Especially, in the temporal feature networks, we place self-attention networks [10] between the two multi-head convolutional neural networks to relate various positions of the shapelets that the first multi-head convolutional neural networks obtain, which enriches temporal features. The latter pays attention to digging out the complex temporal relationships from the input, which helps to compensate for the loss of features that the temporal feature networks may not capture. Therefore, combining the two types of networks together promotes the abundance of both the basic and in-depth temporal features. In this paper, RTFN is embedded into supervised structure first and unsupervised clustering later, in the context of time series classification.

Motivated by the original attention mechanism, we propose an innovative Attentional LSTM to concentrate on mutual temporal relationships. Its details are shown in Fig. 2.

An attentional LSTM network is a hybridized network that incorporates LSTM networks into an attentional structure. In this network, a temporal query and a set of key-value pairs are mapped to an output, where the query, key, and value are vectors obtained from the feature extraction by the LSTM networks. The output is defined as a weighted sum of the values with sufficient shapelets and relationships. In this paper, each value is obtained by a compatibility function that helps to mine the hidden relationships between a query and its corresponding key that already carry basic features, thus strengthening the robustness of the attentional LSTM network. The calculation of the attentional LSTM network is defined in Eq. (1).

where $f_{Q},f_{K}$ and $f_{V}$ represent the matrixes of the queries, keys and values, respectively.

After the sequential data go through the LSTM networks, the queries, keys and values carry sufficient long- and short-term features as input, which are then used to capture the in-depth shapelets and relationships hiding behind as output.

The RTFN-based supervised structure is shown in Fig. 3, where a dropout layer and a fully-connected layer are cascaded to the output of RTFN. To be specific, we introduce the dropout layer to avoid overfitting during the training process. The fully-connected layer performs as the classifier of the RTFN-based supervised structure. The reason we simply use the dropout and fully-connected layers is that the features extracted by RTFN are sufficiently good and thus a complicated classifier network is not necessary. Like other commonly used supervised algorithms, we use the cross entropy function to compute the difference between the prediction result and ground truth label, shown as Eq. (2).

where $\widehat{Y}_{i}^{train}$ represents the ground truth label of the $i$-th time-series class, and $p_{i}$ $\{i=1,2,...,n\}$ is the prediction output of the RTFN-based supervised structure.

The RTFN-based unsupervised clustering is based on a widely adopted auto-encoder unsupervised architecture, as shown in Fig. 4. To be specific, it is primarily composed of RTFN as the encoder, a decoder and a K-means algorithm [19]. RTFN is responsible for obtaining basic and in-depth shapelets from the input data as many as possible. The decoder is made up of four fully-connected layers, helping to reconstruct the features captured by RTFN. Besides, the K-means algorithm acts as unsupervised classifier.

Different from that the total loss takes k-means loss into account in [20][21][22], the loss of the RTFN-based unsupervised clustering simply depends on a reconstruction loss defined in Eq. (3).

where $\breve{X}_{i}^{train}$ is the input data and $X_{i}^{rec}$ represents the decoder output.

The UCR 2018 archive is one of the authoritative temporal data archives, which contains 128 datasets with different lengths in a variety of application areas. We select 85 and 11 standard datasets to evaluate performance of the RTFN-based supervised structure and unsupervised clustering, respectively. All experiments are run on a computer with Ubuntu 18.04 OS, Nvidia dual-GTX 1070Ti GPU with 8GB $\times$ 2 GRAM and AMD R5 1400 CPU with 16G RAM. For encouraging scientific comparison, our code is available at http://X.X.X.X.

To evaluate performance of the RTFN-based supervised structure, we compare it with a number of state-of-the-art supervised algorithms against the top-1 accuracy. We use ‘win’, ‘tie’ and ‘lose’ to rank the algorithms for comparison, i.e. on how many datasets that an algorithm performs better than, equivalent to, or worse than the others. For each algorithm, the number of the ‘best’ cases is the summation of the numbers of the corresponding ‘win’ and ‘tie’ cases.

Table 1 shows the statistical results obtained by different algorithms on the 85 selected datasets in the UCR 2018 archive. For each dataset, the existing SOTA represents the best algorithm on that dataset [4], including BOSS [23], COTE [24], DTW [25] and so on; similarly, for each dataset, the Best:lstm-fcn is the best-performance approach on that dataset, e.g. it involves LSTM-FCN, ALSTM-FCN, etc, in [6]. Note that the existing SOTA did not consider the last 20 of the 85 datasets. The original results are shown in the appendix.

It is seen that the proposed RTFN-based supervised structure performs the best in ‘tie’ and the second best in ‘win’, guaranteeing its first position in ‘best’. To be specific, ours wins in 11 cases and performs no worse than any other algorithm in 29 cases, which leads to 40 ‘best’ cases in the competition. Besides, the Best: lstm-fcn and the Vanilla: ResNet-Transormer achieve the second and third ‘best’ performance, with respect to the ‘best’ score. Especially, the former is a winner on 16 datasets, indicating its outstanding performance. On the other hand, the ResNet-Transformer 2 [4] is the worst algorithm, with 2 ‘win’ scores and 20 ‘tie’ scores only.

In addition, to further reflect the difference between the proposed structure and the others in the long time-series datasets, we select 12 out of the 85 datasets (see Table 2 for details) and show the top-1 accuracy results in Table 3. Besides, we add the RTFN-based supervised structure without the attentional LSTM (ours w/o a-LSTM) to the comparison, which helps to verify the contribution of the attentional LSTM.

On the one hand, if focusing on ours and ours w/o a-LSTM, one can easily see the former outperforms the latter on each dataset, as the former always obtains higher accuracy. This clearly unveils the attentional LSTM plays a non-trivial role in the performance improvement, helping our structure to gain 8 ‘best’ scores and the highest mean accuracy. This is because the embedded attentional LSTM is capable of extracting the underlying temporal shapelets and relationships that are simply ignored by the temporal feature networks.

On the other hand, the Best: lstm-fcn also achieves decent performance with respect to the ‘best’ score and mean accuracy, because its LSTM helps to extract additional features from input data to enrich the features obtained by the fcn networks. The Vanilla: ResNet-Transformer is no doubt the one with the best performance among all the transformer-based networks for comparison. The reason behind this is the embedded attention mechanism can further relate different positions of time series data and thus enhance the accuracy.

To evaluate performance of the RTFN-based unsupervised clustering, we compare it with a number of state-of-the-art unsupervised algorithms against three metrics, including the ‘best’ score as used in Section 4.2, the average ranking, and the rand index [26]. The average (AVG) rank scores are computed according to the average Geo-ranking approach, which measures the average difference between the accuracies of a model and the best accuracies among all models [22]. As a widely adopted performance indicator, the rand index (RI), $RI$, is defined in Eq. (4).

where $PTP$ and $NTP$ are the numbers of the positive and negative time-series pairs in the clustering, respectively, and $s$ is the dataset size.

We select 11 representative datasets from the UCR 2018 archive for performance evaluation and show the results in Table 4. Clearly, our RTFN-based unsupervised clustering achieves the best performance on 4 datasets. Meanwhile, it obtains the best AVG rank and AVG RI values, i.e. 2.13 and 0.7189. On the other hand, USSL [36] and DTCR [22] both achieve 4 ‘best’ scores, which are, to some extent, statistically equivalent to ours. This is because the pseudo-labels help to guide the training process in USSL while the mistake correction helps to facilitate the temporal reconstruction in DTCR. However, for USSL, there is no efficient mechanism to alleviate the negative impact on the training process when mistakes exist in pseudo-labels. Besides, the temporal reconstruction in DTCR is not good at mining cluster-specific representations from input data. This is why DTCR needs an additional K-means objective to its seq2seq structure.

On the contrary, our unsupervised clustering simply adopts an auto-encoding structure to update the parameters of our model and utilizes a K-means algorithm to classify the features obtained by RTFN. Although it is quite simple in structure, ours achieves decent performance on the 11 datasets, which depends on the strong feature extraction ability of RTFN.

In order to further testify that the proposed networks contribute to the accuracy improvement of the K-means algorithm, we compare the RTFN-based unsupervised clustering with a separate K-means algorithm on the 11 datasets above and show the RI results in Fig. 5. One can see that ours outperforms the separate K-means algorithm on each dataset. Sometimes, RI can be significantly improved, such as, on Wafer, Prox.Phal.out.ageGr and OSULeaf datasets. This is because our unsupervised clustering is provided with sufficient features obtained by the proposed RTFN, especially those hiding deeply in the input data which are beyond the exploration abilities of ordinary feature extraction networks. The results of the AVG ranks of all algorithms for comparison are shown in Fig. 6.