ECRTime: Ensemble Integration of Classification and Retrieval for Time Series Classification

The time series classification (TSC) problem represents a foundational challenge within the domain, with the academic community having proposed a multitude of effective algorithms to contend with this intricacy. Backoff-2023[4] summarizes the current eight state-of-the-art (SOTA) Time Series Classification (TSC) classifiers, which are HIVE-COTE 2.0[5], Hydra-MR[6], InceptionT[3], RDST[7], WEASEL-D[8], RSTSF[9], FreshPRINCE[10] and PF[11]. We categorize them into ensemble-based and feature-based methods. Among these, the most accurate ensemble method is HIVE-COTE 2.0, which is also currently the best time series classifier. It employs more advanced ensemble techniques compared to its predecessor, Hive-COTE 1.0[12]. It includes the following: STC, Temporal Dictionary Ensemble (TDE)[13], Diverse Representation Canonical Interval Forest (DrCIF), and Arsenal. Among these methods, DrCIF, developed by the authors of this paper, extends the Canonical Interval Forest (CIF) [14]. Meanwhile, Arsenal, an ensemble of compact ROCKET classifiers, generates valuable probability values for each class during predictions using CAWPE. Presently, these ensemble methods represent the state-of-the-art in time series classification. Nevertheless, they typically exhibit high time complexity, posing substantial challenges for practical application. In our research, we employed an ensemble approach that balances high accuracy with comparatively lower time complexity.

Hydra-MR stands as the most accurate feature-based method to date, achieving an excellent balance between accuracy and time consumption. It amalgamates the Hydra algorithm[6] with the MR (MultiRocket) model[15]. Building on the foundational work of Rocket[16] and MiniRocket[17], MultiRocket introduces a variety of pooling operations and transformations, enhancing the diversity of feature distributions. This advancement not only boosts classification accuracy but also maintains computational efficiency. HYDRA, a dictionary-based method, transforms input time series data using a collection of randomly selected convolutional kernels grouped together. It quantifies the frequency of kernels that most closely match the input time series at each point in time. These quantifications are then utilized to train a linear classifier.

The current academic emphasis on resolving TSC challenges predominantly resides in the domain beyond deep learning, where non-deep learning methods prevail in terms of both prevalence and performance. DL-review[18], as an influential work in the field of Time Series Classification (TSC) within the deep learning domain, summarizes nine advanced deep learning-based time series classifiers, including: Resnet[2], FCN[2], Encoder[19], MLP[2], Time-CNN[20], TWISEN[21], MCDCNN[22], MCNN[23], and t-LeNeT[24]. Among these, Resnet and FCN exhibit relatively optimal performance. Resnet consists of three residual blocks, each followed by a Global Average Pooling (GAP) layer and a softmax classifier at the end. The number of neurons in the classifier corresponds to the number of classes in the dataset. Within each residual block, three convolutions are initially performed, the output of which is added to the block’s input before being passed to the subsequent layer. FCN comprises three convolutional blocks, each containing three sequential operations: a convolution, batch normalization, and a ReLU activation function. The output of the third convolutional block undergoes averaging across the entire time dimension, forming the Global Average Pooling (GAP) layer. Subsequently, a conventional softmax classifier is connected to the output of the GAP layer. Subsequently, it was found in the [25] that directly ensembling these deep learning models could further enhance the algorithm’s performance. Based on this discovery, InceptionT[3] was initially inspired by the Inception-v4[26] network from computer vision tasks and designed the “AlexNet” of TSC - the Inception net. It then ensembled these five models, ultimately achieving an accuracy on UCR85 comparable to HIVE-COTE 1.0. Moreover, Backoff-2023[4] comprehensively reviews and summarizes numerous deep learning models in current TSC tasks, ultimately concluding that InceptionT is currently the best deep learning-based time series classifier.

Figure 5 illustrates the overall structure of the ECR, which includes two stages: training and inference, as shown in Figure 5(a) and Figure 5(b) respectively.

We use ${x}\in{R^{1\times L}}$ to represent a single time series, $1$ to indicate that it is univariate, and $L$ to represent the length of the series. During the training phase, each mini-batch input is denoted as $X$, the input label set as $Y$, and the batch size as $B$. Each mini-batch includes $C$ different categories, and each category contains $m$ samples, thus resulting in $B=C\times m$, $X\in{R^{B\times 1\times{\rm{L}}}}$, $Y\in{\{1,2,...,C\}^{B}}$.

As shown in Figure 5(a), the training framework consists of two independent forward branches: one trained based on classification method and the other based on retrieval method, both sharing the same input $X$. During training, $X$ passes through the classification backbone and retrieval backbone to extract feature vectors ${F_{Cls}}\in{R^{B\times d\times L}}$ and ${F_{Ret}}\in{R^{B\times d\times L}}$, respectively. Note that the two backbones have the same structure but do not share weights. $d$ represents the number of channels in the feature vector. Subsequently, ${F_{Cls}}$ and ${F_{Ret}}$ are both dimensionally reduced to ${{F}_{Cls}^{{}^{\prime}}}\in{R^{B\times d}}$ and $F_{Ret}^{{}^{\prime}}\in{R^{B\times d}}$ through Global Average Pooling(GAP) on the $L$ dimension. In the classification branch, ${F}_{Cls}^{{}^{\prime}}$ is followed by a Fully Connected Layer, and the output is then fed into the classification loss for learning. In the retrieval branch, $F_{Ret}^{{}^{\prime}}$ , after undergoing L2 norm operation, is input into the retrieval loss.

In the inference phase shown in Figure 5(b), for the test sequence, features are extracted based on the backbone networks in the two branches, and then both are reduced in dimension and normalized through Global Average Pooling and L2 Normalization. Additionally, we will pre-extract features of all sequences in the training set in this manner and construct both a classification feature library and a retrieval feature library. Subsequently, the classification and retrieval features of the test sequence are compared with corresponding library features in the Distance module. If there are $N$ features in the library, $1\times N$ classification distance vectors and retrieval distance vectors are outputted respectively. Finally, the predicted category is outputted by ensembling these two types of distances. In the following sections, we will specifically introduce the key modules in the training and inference, including: Backbone, Loss, Distance and Ensemble.

During these years of rapid development in deep learning, many backbone networks for feature extraction have emerged, yet ResNet[27], based on residual connections, remains the preferred choice in numerous application scenarios. The residual structure, without introducing additional parameters and computational load, can effectively mitigate gradient vanishing on one hand, ensuring the continuity of parameter learning; on the other hand, deeper networks can be constructed based on residual connections, and generally speaking, the deeper the network, the stronger its feature extraction capability. Existing works such as [2, 28, 29] have already applied ResNet in the TSC field. Based on the structures validated in these methods, we designed the backbone module of this paper, as shown in Figure 6.

Similar to the structure in [2], the input in Figure 6 first goes through a BN (Batch Normalization) layer for normalization, then through three residual blocks for feature extraction. We also follow the approach in [28] to convert each layer to 1D form, including the convolutional layers and BN layers. The 1D structure is more suitable for one-dimensional time series input, requires less padding, and reduces computational complexity. Furthermore, we adopt a (9, 7, 5, 3) kernel size combination, in contrast to the (8, 5, 3) configuration cited in [2, 28]. This choice is due to the general preference for odd-sized kernels in convolutional networks, which minimize layer distortion and ensure symmetry around the output pixel in the preceding layer. In Table 1, we use a sequence with an input size of $1\times 1460$ as an example to show the specific parameters of each layer in the first block. The other two blocks have similar parameter formats, differing only in the channel dimension.

As shown in Figure 5(a), the training phase employs two types of loss functions: the classification branch utilizes cross-entropy loss, denoted as ${L_{Cls}}$, and the retrieval branch employs the hard version of triplet loss, denoted as ${L_{Ret}}$. Based on the symbol definitions in Section 3.1, ${L_{Cls}}$ is defined in the following form:

In Eq. (1), $z$ represents the vector obtained after ${{F}_{Cls}^{{}^{\prime}}}$ passes through the FC layer, and the indicator function $\mathbbm{1}({y^{i}}=j)$ is defined as follow:

Based on Eq. (2), Eq. (1) can be further simplified to the following form:

For retrieval tasks, triplet loss[30] is generally used. It originally works on an anchor series $A$, a positive sample $P$ from the same class and a negative sample $N$ from a different class. The objective is to minimize the distance between $A-P$, while push away the $N$. The formula of triplet loss is as follow:

Eq. (4) iterates over and calculates each sample in the mini-batch input, taking the average as the final loss. In this context, $g$ denotes the vector after L2 norm, $H$ indicates the count of positive or negative samples. The term $\alpha$ represents the margin between positive and negative samples. Additionally, the subscripts $P$ and $N$ correspond to positive and negative samples, respectively. The original triplet loss introduces many easily satisfied triplets, which lack contribution to the training, leading to slower and less efficient convergence. Therefore, this paper follows the approach in [30] and uses hard triplet loss for training. This loss narrows the distance between the anchor sample and the farthest positive, while increasing the distance between the anchor and the closest negative, defined as follow:

In the training phase, two pivotal modules, the Backbone and the Loss, have been delineated earlier. Subsequently, in the testing phase, we elucidate the Distance module, conceived on the 1-NN classifier paradigm. As depicted in the testing procedure (Figure 5(b)), each time series input ${{x}^{i}}$ undergoes feature extraction via the classification and retrieval backbone, followed by processing through GAP and L2Norm, yielding output vectors $f_{Cls}^{i}$ and $f_{Ret}^{i}$. Concurrently, features are extracted from every sequence in the training set in a similar fashion, leading to the formation of two libraries: the classification library ${F_{Cls\_lib}}=\{j\in[0,N)|f_{Cls}^{j}\}$ and the retrieval library ${F_{Ret\_lib}}=\{j\in[0,N)|f_{Ret}^{j}\}$, where $N$ signifies the total number of sequences in the training set.

In the Distance module, by iterating and calculating the Euclidean distance between $f_{Cls}^{i}$ and all features in the classification library ${F_{Cls\_lib}}$, we obtain $D(f_{Cls}^{i},{F_{Cls\_lib}})={\{D_{Cls}^{1},D_{Cls}^{2},...,D_{Cls}^{N}\}^{N}}$, representing the distance between $f_{Cls}^{i}$ and ${F_{Cls\_lib}}$. The distance between vector $f_{Ret}^{i}$ and the retrieval library ${F_{Ret\_lib}}$ is denoted as $D(f_{Ret}^{i},{F_{Ret\_lib}})={\{D_{Ret}^{1},D_{Ret}^{2},...,D_{Ret}^{N}\}^{N}}$, calculated in a similar manner.

Following the Distance module is the Ensemble module, and this paper involves two stages of ensemble operations, as illustrated in Figure 7. The first stage ensembles classification and retrieval models to obtain ECR, and the second stage ensembles multiple ECRs to derive the final ECRTime model. In the first ensemble, as shown in Figure 5(b), we average all corresponding elements in sets $D(f_{Cls}^{i},{F_{Cls\_lib}})$ and $D(f_{Ret}^{i},{F_{Ret\_lib}})$ to obtain the final distance set $D({f^{i}},{F_{lib}})={\{(D_{Cls}^{1}+D_{Ret}^{1})/2,(D_{Cls}^{2}+D_{Ret}^{2})/2,...,(D_{Cls}^{N}+D_{Ret}^{N})/2\}^{N}}$. Finally, the category of the corresponding sequence in the library, predicted for the test sequence ${x^{i}}$, is identified by taking $\arg\min(*)$ of $D({f^{i}},{F_{lib}})$.

It is important to note that L2 normalization is applied to the classification features during testing to ensure uniformity in the value ranges of $D(f_{Cls}^{i},{F_{Cls\_lib}})$ and $D(f_{Ret}^{i},{F_{Ret\_lib}})$. This uniformity is crucial as it prevents the averaging of prediction results from being skewed by differing value ranges. Additionally, for two vectors with an L2 norm of 1, their Euclidean distance can be reformulated as $\sqrt{2(1-\cos\theta)}$ where $\theta$ is the angle between the vectors. This ensures that both parties being averaged have the same value range, which is $[0,2]$.

Furthermore, in the second ensemble, based on Eq. (6), multiple ECRs are integrated to obtain the final ECRTime model presented in this paper. In the formula, $\widehat{y}_{i,c}$ represents the ensemble’s output probability that the input time series, $x_{i}$ belongs to class $c$, This is equivalent to the average logistic output $\sigma_{c}$ across $n$ randomly initialized ECRs.

In this section, we evaluate the ECR and ECRTime on 112 datasets in the UCR univariate time series archive. ECRTime refers to an ensemble of three ECR modules, while the “ECRTime(n)” notation is used to denote an ensemble of n ECR modules. “UCR112” is used to denote the 112-version of the UCR archive in the following text. The experimental results are available on the website¹¹1https://anonymous.4open.science/r/ECRTime-3834. Initially, the experimental setup is introduced.

Datasets and SOTAs: To compare with numerous advanced algorithms while avoiding excessively time-consuming experiments, we followed the approach used in [5, 4], conducting experimentation with the 112 equal length problems in the 2019 version of the UCR archive. The comparison includes classifiers based on deep learning summarized in DL-review[18] and state-of-the-art classifiers in the field of Time Series Classification(TSC) compiled in Backoff-2023[4]. The corresponding comparison results are respectively sourced from ²²2https://github.com/hfawaz/dl-4-tsc/blob/master/results/results-ucr-128.csv and ³³3https://github.com/time-series-machine-learning/tsml-eval/tree/main/tsml_eval/publications/y2023/tsc_bakeoff/results.

Configuring ECRTime: Since the datasets in the UCR only consist of training and test sets, lacking a validation set, it is not possible to tune hyperparameters such as epochs. Therefore, we refer to the ResNet classifier in DL-review[18] for hyperparameter settings. During the training phase, we set each mini-batch input to contain 4 categories, with 4 samples per category, resulting in a batch size of 16. The margin in the hard triplet loss is set to 0.1, and the optimizer used is Adam. For the classification branch, the learning rate is set at 1e-3, and for the retrieval branch, the learning rate is set at 1e-4. The scheduler used is ReduceLROnPlateau from PyTorch, the number of training epochs is set to 1500. Experimental environment configuration: PyTorch 2.0, Python 3.9.

To facilitate a comparison with a range of deep learning classifiers, we evaluated ECR against the eight methods detailed in DL-review[18]. For result validation, we adhered to DL-review’s methodology, training ECR for five iterations on UCR112. During these iterations, only the random seed varied, while the model’s structure and training hyperparameters remained unchanged. The final reported accuracy represents the mean of these iterations. Figure 8 features a critical difference diagram that illustrates the accuracy comparisons between ECR and various deep learning models. The horizontal thick line across different models indicates no significant difference between them (p-value>0.05). It is noted that ECR significantly outperforms all methods depicted in the figure (p-value<0.05), including ResNet and FCN, which were identified as the most precise deep learning classifiers in DL-review at that time. In the subsequent Section 4.5.5, the discussion will focus on how ECRTime, an ensemble of ECR, markedly surpasses ECR, indicating that ECRTime also surpasses the aforementioned deep learning-based methods. Consequently, to reduce the redundancy of the experiments, we refrained from conducting a parallel comparative analysis for ECRTime as in Figure 8.

In the preceding section, we discussed ECR’s significant outperformance of most deep learning classifiers on UCR112. This section broadens the comparative analysis to include all Time Series Classification (TSC) methods. We evaluated ECR against the eight leading state-of-the-art (SOTA) classifiers listed in Backoff-2023[4], namely: (1) HIVE-COTE 2.0[5], (2) Hydra-MR[6], (3) InceptionT[3], (4) RDST[7], (5) WEASEL-D[8], (6) RSTSF[9], (7) FreshPRINCE[10], and (8) PF[11]. Of these, HIVE-COTE 2.0 is recognized as the most accurate algorithm for TSC issues, albeit with considerable computational demands. InceptionT, on the other hand, is currently the most accurate deep learning-based TSC classifier. Detailed analysis of these SOTA classifiers can be found on website ⁴⁴4http://timeseriesclassification.com/results.php. Adhering to the methodology outlined in Backoff-2023, we conducted training and testing on the original UCR112 train/test set, as depicted in Figure 9. The results indicate that ECR outperforms PF (the leading distance-based method), FreshPRINCE (the top feature-based method), and RSTSF (the foremost interval-based method). However, it is surpassed by the other five methodologies.

To extend our comparative analysis, we utilized the more efficacious ECRTime, an ensemble of three ECR models, against other SOTA methods. We provided scatter charts for detailed pairwise comparisons: between ECRTime and the most accurate deep learning-based method, InceptionT, as illustrated in Figure 10(a), and with the leading algorithm, HIVE COTE 2.0, in Figure 10(b). As depicted in Figure 1, ECRTime marginally outperforms InceptionT, thereby becoming the most precise deep learning classification method currently used in TSC, although its margin over the second-ranked Hydra-MR is not statistically significant (p-value>0.05). Nonetheless, it considerably trails behind HIVE-COTE 2.0 (HC2), the most accurate classifier. In Figure 10(a), ECRTime and InceptionT demonstrate comparable performance on half of the UCR112, each exhibiting superiority in the remaining datasets, with ECRTime having a narrow advantage (Wins: 29 versus 27). Figure 10(b) shows a notable gap between ECRTime and HC2 (Wins: 23 versus 44), with ECRTime displaying over a 20% shortfall in datasets such as Rocket, SemgHandMovementCh2, and SemgHandSubjectCh2. Future efforts will be directed towards enhancing ECRTime’s performance in these specific datasets.

Upon synthesizing the comparison results, it becomes clear that the ECRTime model introduced in this study matches the current top-performing deep learning classifier, InceptionT, in terms of overall effectiveness. To conduct a more detailed analysis of their strengths and weaknesses, we compared them based on the dataset types present in UCR112, namely the sequence source types. As depicted in Figure 12(a), UCR112 encompasses 13 dataset types[4]. The DEVICE, IMAGE, MOTION, SENSOR, SIMULATED, and SPECTRO categories, which constitute 85% of the datasets, are the most significant, with DEVICE and SIMULATED being the smallest (9 datasets each) and IMAGE being the largest (32 datasets). The ”others” category comprises 7 dataset types, each containing only 1-3 datasets. Due to the dominance of the first six categories in UCR112, we utilized boxplots to illustrate the accuracy variances between ECRTime and InceptionT across these categories. As shown in Figure 12(b), ECRTime exceeds InceptionT in the DEVICE and SENSOR categories, equals InceptionT in the IMAGE and SPECTRO categories, and is marginally less effective than InceptionT in the MOTION and SIMULATED categories. These insights provide valuable guidance for researchers in choosing the most suitable approaches for diverse practical applications.

To enable a comparison of time efficiency with ECRTime, we extracted the average training time of various state-of-the-art methods on UCR142 from Backoff-2023 as an approximate for the time spent on UCR112. These comparative findings are presented in Table 2. ECRTime exhibits a reduced training duration compared to InceptionT. While it is more time-intensive than other CPU-based methods like RDST and Hydra-MR, leveraging parallel training on multiple GPUs can decrease its training time, as ECRTime is GPU-based. It is important to note that for this study, ECRTime was trained using an RTX3060 GPU, a less powerful consumer-grade graphics card. Utilizing more advanced graphics cards could lead to a further decrease in training duration.

For a comprehensive comparison of accuracy and training duration across various methods, we utilized the average training time from Table 2 as the X-axis and the mean rank on the UCR112 dataset as the Y-axis to construct Figure 13. This figure indicates that Hydra-MR, a member of the ROCKET family, presents the most favorable balance between accuracy and training time. HC2 achieves the highest accuracy but requires the longest training period, whereas ECRTime is ranked third in accuracy but benefits from a relatively short training duration. To examine the scalability of ECRTime and InceptionT in depth, we assessed the relationship between training time and sequence length, as well as training time and the number of samples in the training set, using the EthanolLevel dataset from UCR112. As illustrated in Figure 11(a) and Figure 11(b), ECRTime demonstrates a slightly more gradual increase in training duration compared to InceptionT, in response to longer sequence lengths or larger training sample sizes. Consequently, when considering factors such as training time, accuracy, and scalability, ECRTime emerges as a more advantageous option compared to InceptionT.

We explore the effect of key parameter choices on accuracy over UCR112 for ECR and ECRTime:

• –

  1-NN classifier versus SoftMax classifier.

• –

  hard triplet loss versus triplet loss.

• –

  ensemble in ECR.

• –

  ensemble in ECRTime.

• –

  batch size.