DeepVATS: Deep Visual Analytics for Time Series ¹¹footnotemark: 1

Datasets have to be “logged” as an artifact in the storage module before they can be used in any other part of the system. In order to define a time series dataset to be used in DeepVATS, some considerations have to be taken:

• •

  Only one time series at a time can be analysed. Therefore, the tool is suitable for long time series that present cyclical patterns and anomalies, and not in cases when the time series is not cyclical or when one wants to detect similarities differences between different samples (e.g., clinical records of different patients). In the future, we are planning to extend the tool so that it is applicable to datasets made of multiple time series too.

• •

  The tool works in the same way regardless of the number of variables present in the time series, i.e., whether the time series is univariate or multivariate. Additionally, natural timestamps (i.e., datetimes) are not needed, since the time information is not used as input in the neural network encoder.

• •

  The series is assumed to be regular, i.e., time steps must be evenly spaced.

• •

  Missing values in the series, if any, are imputed through linear interpolation.

• •

  In case the time series is too long, the user can choose an option to resample it into a lower frequency. The data will then be aggregated using the mean.

By default, the time series will be logged in the storage module as a single artifact. This is appropriate for enabling offline data analysis, where the range of data that is analysed is the same that has been used to train the backbone encoder. However, DeepVATS also provides the option of splitting the time series manually and logging multiple artifacts out of it, namely the training and test artifacts. This is especially relevant to examine the capabilities of DeepVATS for online analysis, i.e., for situations where an expert needs to analyse new data from a process often, and therefore, retraining the backbone encoder every time new data is available is infeasible. In these cases, one has to test the trained model on future time windows (See Figure 3), to check if it produces satisfactory results out of the training interval. Note that this opens the possibility of doing zero-shot DVA, i.e., of having an instant DL-based analysis of a dataset without having to retrain or fine-tune the backbone model. Additionally, the online mode is also suitable for very long time series, where the cost of training the backbone model on the whole series is unnecessarily high, and just using a slice of data is enough to create a model that will then be used to process (in inference time) the whole series in a fast way.

The encoder is the central piece of the DL module, and arguably of the whole DeepVATS framework. We also refer to it in this work as the backbone model, or simply the model. An encoder, in deep learning, is an artificial neural network used to turn raw input variables from a high-dimensional space into efficient and compressed vector representations of a lower dimensionality, often known as embeddings, hidden states, or just the model’s latent space. These representations are typically used as an intermediate step in machine learning tasks.

Training the encoder properly needs a dataset with a big number of time series samples. However, as seen in the previous section, the input artifact logged in the storage module is a single long time series. Therefore, before passing it to the encoder, the input series is divided into separate, contiguous and fixed-size windows of data, or patches, in a process often known as sliding window. The window size in this process, to which we will refer as $w$, is then a crucial hyperparameter that can affect the dataset that is passed to the model, and thus, it can affect the representations obtained by the encoder and the visualisations that we get out of it. By default, the stride of the sliding window, i.e., the number of data points that the window is moved ahead along the series is set to 1. Therefore, given a time series of length $T$, a window size $w$ and a stride of $s$, the dataset after the sliding window will have a total of $N=\lfloor\frac{T-w}{s}\rfloor+1$ time windows. Despite using such a low stride causes contiguous time windows to almost completely overlap with each other, this prevents the training process from having a bias from the choice of this value.

In order to watch for overfitting, the dataset created with the sliding window process is split into training and validation subsets. We do this differently depending on whether there are one or two time series artifacts logged for the dataset in the storage module. If there are two, namely the training and test artifacts, we create the validation set by holding out the last 20% of the training set, to be temporally coherent with respect to the fact that the model will be tested on future data. Otherwise, the validation set is created by randomly holding out 20% of the time windows. With regard to normalisation, it can be applied either batch-wise, sample-wise, or dataset-wise, depending on the needs of the use case.

The use of encoders in deep learning is commonly associated with a so-called encoder-decoder architecture, where there is a decoder network fed with the outputs of the encoder, which turns them into an actual prediction. One special case of the encoder-decoder architecture, which will be employed in this work, is the AutoEncoder (AE), in which the input and output spaces are the same, i.e., the task of the model is to reconstruct the input data, by learning first a compressed representation of it. Note that this is a self-supervised task, since the target needed to train the model is inherent in the data itself. Note also that, despite DeepVATS uses by default an autoencoder architecture, its modular design allows it to plug in any other encoder.

We refer to the autoencoder used in DeepVATS as the Masked Time Series AutoEncoder (MTSAE). The term “mask” refers to the idea of hiding (or zeroing) sections of the time series, and training the model to reconstruct them [17]. One of the advantages of having a masked learning framework is the flexibility in the design of the masking strategy, as it can be seen in Figure 4(b). Masks are applied stochastically over the time series, controlled by the probability of masking, $r$. An uniform distribution is followed by default, but a “stateful” mode can be selected too, in which a geometric distribution is applied so that average mask length meets the value of another parameter called $lm$. This is required in order to force the model beyond trivial prediction methods such as padding, replication or linear interpolation, which might offer a sufficiently good approximation for very short masked sequences. Additionally, one can elect to use the same mask synchronised over all variables for a given object (synchronised mode), or instead, generate a new mask for each in order to encourage the model to learn both relationships between different values along individual sequences, as well as inter-dependencies between variables in order to improve the modelling. If masks are applied at the end of the time series, the MTSAE acts effectively as a forecasting model, which makes forecasting a special case of a masked autoencoder. In any of these cases, a binary sequence is generated based on each variable in the series, and the model is tasked with predicting the values of the time series over which the mask is set to 0.

With regard to the architecture used to build the MTSAE, we clearly differentiate the encoder and decoder sections. For the encoder, we employ the InceptionTime architecture [41]. This is, to the best of our knowledge, the best deep neural architecture for time series tasks among the family of 1-dimensional Convolutional Neural Networks (CNNs). In this work, we use a network made up of six sequential inception modules which maintain residual connections, a common strategy to build deeper CNNs. On the other hand, the decoder is made up of just one output layer, in which the shape of the original input data is reconstructed by using a convolutional layer with a number of filters equal to the number of channels in the input data, and a filter size of 1 (See Figure 5). Note that this decoder layer can be created in the same way regardless of the encoder architecture. Therefore, new encoder architectures can be plugged into the framework without making any change in the training process.

One advantage of using a CNN such as InceptionTime in the encoder is that the neural architecture does not change depending on the sequence length of the input, unlike for vanilla RNNs or Transformers. This fact can be used to implement variable window sizes during training (See Figure 6), which allows the model to be trained with different window sizes in each training iteration, obtaining a model less sensitive to the choice of this hyperparameter. To do this, we set an interval $[w_{\textnormal{min}},w_{\textnormal{max}}]$, and for each training iteration, the batch will be randomly truncated at a given value within that range. This has a regularisation effect too, which prevents overfitting in cases where the dataset is small. Note that, under this setup, $w_{\textnormal{max}}$ has to be lower or equals to the value of the window size $w$ used in the sliding window process. However, setting $w_{\textnormal{max}}<w$ causes that the data points between $w_{\textnormal{max}}$ and $w$ in each batch are lost, since they will never be passed to the model. Therefore, by default DeepVATS sets $w=w_{\textnormal{max}}$.

We train the model using an adaptation of the Mean Squared Error (MSE) metric, such that it only considers the predictions over masked values. For a given object with $i$ channels and $t$ time steps, this loss is defined as,

Once the model has been trained and validated properly, it is logged as an ”encoder artifact” in the storage module so that it can be used for inference in the VA module. This artifact will contain not only the weights of the neural network, but also metadata associated to the training process such as the window size employed to slice the dataset, the masking type and probability, etc.

In this part of the GUI, the user can configure a variety of controls that will affect how the rest of the plots of the GUI look like. These controls are:

• •

  Dataset selector: It allows the user to select the dataset that wants to be analysed, among the list of time series artifacts that have been logged before in the storage module by the DL module (See Section 3.1.1).

• •

  Encoder selector: It allows the user to select the encoder that wants to be used to encode the selected dataset, among the list of encoders that have been trained in the DL module for the selected dataset and logged in the storage module (See Section 3.1.2).

• •

  Window size selector: The value selected here will be used to slice the selected time series dataset following a sliding window process. If the selected encoder has been trained with variable window sizes, this selector will allow to choose values in the interval $[w_{\textnormal{min}},w_{\textnormal{max}}]$, i.e., the same one used for training the encoder. Otherwise, in case a fixed window size approach has been employed, the range of selectable values is $[w-w/2,w+w/2]$, where $w$ is the value of the window size used in training. In any case, by default the selector is set to the value $w$.

• •

  Stride selector: Unlike in training, where the stride of the sliding window process is always set to 1 to maximise the amount of data windows created, increasing this stride in inference can help create clearer projections of the embeddings which reveal better information. Additionally, it reduces the amount of data windows, reducing the inference time too, which is crucial for a fluid exploration of the data. The values to select for the stride range between 1 and the value of the window size chosen in the above selector.

• •

  Projection method selector: Three possible dimensionality reduction algorithms can be applied to the encoded data windows, namely Uniform Manifold Approximation and Projection (UMAP) [19], t-Distributed Stochastic Neighbor Embedding (t-SNE) [18], and Principal Component Analysis (PCA) [42]. All of them are run directly on GPU using the RAPIDS library [43], in order to maximise their performance.

With the configuration set in the control panel, the selected dataset is sliced into data windows through a sliding window process using the selected window size and the selected stride. Then, the whole set of windows produced is passed to the selected encoder. During this inference, we are not interested in the output of the model, but in its latent space, i.e., in its activations within one of its internal layers, which contain the encoded version (the embeddings) of each of the data windows. By default, if the proposed MTSAE is used as architecture for the encoder (See Section 3.1.2), the embeddings will be taken from the last layer of the encoder block, i.e., from the output of the last inception module from the InceptionTime architecture (See Figure 8(a)). Otherwise, in case a custom encoder is used, the layer from which to get the embeddings must be set. Normally, it will be the last layer before the head of the network, due to latter layers in a neural network having higher abstractions than earlier ones [44].

Given a time series with $v$ variables, sliced into $N$ data windows of size $w$, the input tensor that will be passed to the model will have a shape of $N\times v\times w$. In the proposed InceptionTime-based MTSAE, the dimensionality of the embedding space at the last layer of the last inception module is 128 (that is, the number of filters of that layer is 128). There are no pooling layers in those modules that shrink the sequence length of the internal feature maps across the network. Therefore, at the end of the encoder section of the MTSAE the shape of the embeddings is $N\times 128\times w$ (See Figure 8(a)). We average that tensor on the sequence dimension reducing it to $N\times 128$. At that moment, each data window is represented by a single embedding of 128 features. Finally, the selected dimensionality reduction algorithm is applied, projecting the embeddings into a $N\times 2$ table, ready to be visualised on a 2D canvas.

The central plot to visualise projected embeddings is a connected scatter plot, as it was in the TimeCluster framework [3]. In this plot, data points represent the projection of the embedding of each data window, and straight lines are used to connect subsequent data windows to each other, in order to keep an idea of time within the plot. The size and colour of all the points is the same. The user can interact with this plot by zooming, cropping, select points and select areas of the 2D space.

Under the embedding projections canvas, we plot the time series dataset, and provide two-way interactions in both plots to explore and understand where the different parts of the embedding space are found on the original input space. When a point or area of the embeddings plot is highlighted, the time windows represented by those points are highlighted in the time series plot too, which allows the user to explain the patterns and anomalies of the embedding space based on the data (See the green area highlighted in Figure 7).

Other possible interactions available in the time series plot include zooming, cropping, and, in case of having multivariate time series, selecting a subset of the variables to be visualised, in order to reduce information overload in the plot.

We constructed the synthetic time series as a linear combination of sinusoidal functions with different amplitudes and seasonalities. To this end, we assumed a discrete generation, taking minutes as the regular time step. We considered sinusoidal components for the hourly, daily, weekly seasonalities and different components for groups of $n$ hours, with $n$ taking the values of $2$, $3$, $4$, $6$, $8$ and $12$ hours. Based on these considerations, the generic function used to generate the synthetic time series is:

With this generation technique, it is possible to synthesise both univariate and multivariate time series by generating the different variables independently. Additionally, we can concatenate time series generated with different parameters, thus giving rise to series with different stages. Following these procedures, we generated four synthetic time series datasets, whose equations and plots can be found in A:

• •

  S1: a univariate time series spanning 24 days of data, made up of four segments of respective size $7.5$, $6.5$, $10$, and $4$ days, with different patterns constructed by varying the amplitudes of the different seasonal components. More specifically, two of them, the first and the third, maintain the same components with seasonality less than a day, and only the components with a daily and weekly seasonality vary, maintaining a very similar daily pattern in both segments.

• •

  S2: a univariate time series with 20 days of data, built from a synthetic series used as a base, perturbing two segments of one hour each, at the beginning of the fifth day and at the end of the fourteenth, thus causing two anomalies in these segments of the time series.

• •

  S3: a univariate time series spanning 20 days of data, built with a single configuration of generation parameters, but with a new term in its definition to include a time-dependent increasing trend.

• •

  S4: a univariate time series spanning 20 days of data, constructed, like the series S2, from a base synthetic series, but this time perturbing a single segment of one hour in the part end of the series, on the nineteenth day.

In order to test the capabilities of the tool to detect multidimensional time series motifs, we use a synthetic dataset called M-Toy, gathered from the stumpy library for time series data mining [45]. It has three parallel variables, but only the first two are correlated and conform a multidimensional pattern. The third one is actually a “random walk” that was purposely included as a red herring, not related to the other ones. This is meant to test whether the algorithm is robust to irrelevant dimensions and spurious data.

We validated the capabilities of DeepVATS on real datasets for which we have access to expert information of what should be found on them. It should be noted that we selected them according to their purpose in the experimentation and not based on other factors such as their size, even though DeepVATS is especially useful for long time series. Below are brief descriptions of the three datasets employed, whose plots can be found in B:

• •

  Arterial Blood Pressure (ABP): Firstly presented in [46], it contains data from a healthy volunteer resting on a medical tilt table. At time 2400, the table was tilted upright, invoking a response from the homeostatic reflex mechanism, and producing two clear segments of data.

• •

  Parking: Presented in [47], it contains the hourly parking occupancy rate in 6 streets of Caserta and Naples, Italy. From them, we take the data of one of the streets, namely Piazza Vanvitelli. The parking occupancy rate is defined as the ratio between the number of occupied parking slots and the total number of parking slots present in a specific area. Hourly and daily seasonalities are present in the data.

• •

  Kohl’s: The data contains a decade-long Google Trend query volume (collected weekly from 2004-2014) for the keyword Kohl’s, an American retail chain. The time series has a growing trend, and features a significant but unsurprising “end-of-year holiday bump”. We can see that the bump is generally increasing over time. This dataset was presented in [48], and used in the time series data mining library stumpy [45] for the analysis of time series chains, i.e., motifs that evolve or drift in some direction over time.

Time series semantic segmentation, or just time series segmentation, is the task of dividing the series into homogeneous regions [46]. The timestamps when the time series changes from one segment to another are called change points, and there are specific methods to detect them [49]. Here, we will show how DeepVATS allows extracting both the compact segments of the series and the change points.

Based on the first synthetic dataset (S1), we trained a model in the deep learning module following a masking strategy with state (stateful) and masking probability $r=0.4$, with variable window in the range $[36,72]$. Once trained, we loaded the model in the visual analytics module to explore its content. By exploring with the tool, we found that a window size $w$ of $54$ timestamps, a stride $s$ of $2$ positions and UMAP as projection algorithm produced valuable insights. As it can be seen in Figure 10, the projection graph shows three clearly differentiated areas, with circular shapes, connected by gray segments. The circular shape of these clusters of points is no accident, and the reason for this is detailed in the next section. Each of these areas corresponds to a segment of the time series that have certain common characteristics. In the first three representations of the figure it can be seen that DeepVATS detected the three homogeneous segments that had been defined in the construction of the synthetic series, having considered the first and third fragments of the series as a single homogeneous segment, since both have common characteristics related to the seasonality of their components.

Since each point of the projection plot corresponds to a time window, and the points of contiguous windows are joined by a grey line, it is possible to detect windows with change points, by looking at the points at the beginning and end of the line that join the different segments. The smaller the window size used in inference, the narrower the area of change between segments will be. In C, we complement this experiment with one analogous made on a real use case, namely the ABP dataset.

In this section we aim at spotting repetitive patterns and anomalies with DeepVATS. On the one hand, anomaly (or outlier) detection refers to the task of finding points or subsequences in the time series that differ from the common behaviour of the rest of the data. On the other hand, the recognition of repetitive or cyclical patterns involves finding subsequences that appear recurrently throughout the time series, due to either the seasonal nature of the series or to the presence of a motif, .i.e, a previously unknown pattern that carry precise information about the underlying source of the series [50].

To account for this analysis, we trained MTSAEs on two datasets, namely S2, built specifically for this purpose, and Parking (See Section 4.1.2). Here we show the results on Parking, leaving the results on the synthetic data in D. We trained the encoder following a stateless masking strategy with $r=0.5$ and a variable window size of $[8,24]$, given that the series have a time step of 1 hour and presents daily seasonalities. Then, we explored the model in the VA module and found the results shown in Figure 11. The first aspect to highlight in the projections plot is that points are distributed following an elliptical shape, which indicates that the model has been able to detect the cycles present in the data (this is in accordance with the results found in [3]). However, we can see several grey segments that do not follow the elliptical path and cross the middle of the figure. Examining the windows corresponding to the extreme points of these segments, we see that they correspond to time windows that contain the days December 25 and January 1 respectively. On those days, the parking occupancy followed a different pattern, and in fact, if we observe the series those days in an extended way (see the right graphic representation with the highlighted windows in green on the time series) we can confirm that its behaviour differs from the surrounding days. Outliers, therefore, appear in DeepVATS as points that do not follow the path common to the rest of the time windows.

Considering the groups of points formed along the cyclic trajectories, the second screenshot of Figure 11 shows how all the points located in the lower right part of the trajectory correspond to the pattern of occupation of the daytime hours, leaving outside the night hours. The following screenshot of this same figure highlights certain points in the projection space that represent occupancy patterns on weekdays, while the last one shows the points that correspond to the repetitive pattern of weekends and some holidays. Therefore, we can conclude that clusters within the elliptical path correspond to windows that follow the same recurrent pattern, beyond the seasonality periods of the series.

The masking strategy that has provided the best results for detecting anomalies and repetitive patterns is the default stateless masking strategy. In these cases, the probability of masking follows a uniform distribution, increasing the probability that isolated timestamps are masked. This helps the encoder to identify point-based anomalies in and not ignore them when it comes to reconstructing the masked parts in the decoder, as it happens with classic autoencoders.

Trend detection in a time series tries to find significant and prolonged changes over time. Some classical approaches accomplish this task using statistical tests [51]. In DeepVATS, we will try to detect these changes from the shapes that follow the trajectories of points in the projection space. Among the different masking options available to train models in DeepVATS, we found that the future masking strategy (See Figure 4(b)) is the one that best suits this task. A possible explanation for this is that having to reconstruct the latest values of the time windows based on previous knowledge makes the model learn to interpret the trends in the data in order to make the reconstruction of the future values take those trends into account.

As a first approximation analysing the ability of DeepVATS to detect trends in the data, we trained a model with the synthetic dataset S3. We employed a future masking strategy and a variable window size in the range $[32,96]$. The results after exploring the model in the VA module are shown in Figure 12. Although the series is cyclical, the trajectories in the projection space do not seem to have an elliptical or circular shape (as happened with the models of the previous section), being the points distributed in a sort of parabolic trajectory. However, if we carefully examine the groups of points within this trajectory and the segments that join them, we appreciate how they retain a certain circular shape, with the centre of the circle of each cycle shifting as the series progresses (see carefully screenshots 3, 4 and 6 in Figure 12). The projections contain the cycles, but display them in displacement as a result of the trend of this series. Therefore, although a certain idea of trend is captured in the two-dimensional space, it is not reflected as clearly as it has occurred with anomalies, patterns and segments.

We corroborated this hypothesis after applying the tool to the Kohl dataset (See Section 4.1.2). We trained the MTSAE on this dataset employing a future masking strategy, analogous to that applied in the synthetic use case above, and a variable window size of $[6,12]$ timestamps. The results after exploring the model embeddings in the VA module are shown in Figure 13. Unlike S3, Kohl is a time series whose amplitude on each cycle is not constant. In addition to having a growing global trend, it grows in amplitude each year. This fact is translated into a projection space in DeepVATS that provides little valuable knowledge about the series. As it can be seen in Figure 13, the projected points seem to follow a linear trajectory, but if we look at the time windows corresponding to some of the groups of points that are close in the trajectory, we don’t see a clear mapping between them and continuous segments of the original series. Furthermore, the points associated to time windows containing peaks are not clustered together in a region of the projection space. In conclusion, we cannot extract valuable insights about trends using DeepVATS, as it is developed so far.

Finally, we test the capabilities of DeepVATS to analyse multivariate time series. Note that we do not have to change anything from the pipeline used for the univariate analysis above, since both the neural architecture and the masking strategy used to create the backbone MTSAE work for multivariate time series out of the box. By default, DeepVATS creates unsynchronised masks to train the MTSAE, which means that the same time step can be part of the mask or not depending on the variable we are focusing on. This naturally makes the training robust to spurious dimensions and allows the network to find patterns between a subset of dimensions. However, this can be changed as part of the configuration, making the mask synchronised across all variables, which is only recommended in case one knows in advance that all dimensions of the series are involved in the patterns of interest.

We trained a MTSAE on the M-toy dataset (see Section 4.1.1, using a fixed window size of $w=30$, as it is done in the stumpy library from which the dataset was gathered. The results after exploring the model embeddings in the VA module are shown in Figure D, along with a screenshot from the stumpy documentation ³³3https://stumpy.readthedocs.io/en/latest/Tutorial_Multidimensional_Motif_Discovery.html with the results of running a 1-dimensional matrix profile based motif discovery algorithm on the dataset. This algorithm resulted in a real multidimensional motif found between the first two channels of the series (T1 and T2), and a spurious motif in channel T3, which we would like to avoid because, as it was said in Section 20, that channel was artificially introduced as noise.

As it can be seen in the projections of Figure 14, there are two long segments in the connected scattered plot with no point in between, which represent “irregularities” in the embedding space. In fact, by analysing the points at the ends of each segment, we can see that they correspond to the two appearances of the motif that we were looking for (the one between T1 and T2). Since there are no other irregularities in the embedding space, we can conclude that DeepVATS has helped to easily spot this motif, and it has avoided the effects of noisy variables such as T3.