CSCLog: A Component Subsequence Correlation-Aware Log Anomaly Detection Method

Traditional methods for log anomaly detection can be generally divided into two classes, i.e., supervised learning-based traditional methods and unsupervised or self-supervised learning-based traditional methods.

Supervised learning-based traditional methods aim to classify log sequences by extracting statistical features, e.g., the number of templates and keyword distribution, which require the data with labels. Chen et al. (Chen et al., 2004) introduced decision trees, which have strong interpretability, to model the number of templates in log sequences. Liang et al. (Liang et al., 2007) extracted six different features from log sequences, e.g., time interval and the number of templates, as the inputs of their classifier. Recently, Bodik et al. (Bodik et al., 2010) employed the Support Vector Machine (SVM) based on the Gaussian kernel to model features in log messages, e.g., frequency of occurrence, periodicity, and correlation of anomalies. Since the frequency of anomalies in a system is low and it is difficult to get a large amount of labeled log data, unsupervised or self-supervised learning-based traditional methods are proposed. Lou et al. (Lou et al., 2010) introduced Invariant Mining (IM) to derive linear relations of the number of log templates, which can capture the co-occurrence patterns of log templates. Xu et al. (Xu et al., 2009) applied Principal Component Analysis (PCA) to log anomaly detection, which uses the count vector of log templates and the parameter value vector of log messages as the inputs. Lin et al. (Lin et al., 2016) used the Agglomerative Hierarchical Clustering algorithm (AHC) to classify log sequences into normal and anomalous clusters, which detects whether a newly arriving log sequence is anomalous by calculating its distance to the two clusters.

However, due to the differences in the distribution of statistical features of different datasets, these methods fail to achieve good performance on different datasets. In addition, when an anomaly occurs, a series of program executions are logged in the system, while traditional methods fail to capture the sequential dependencies in log sequences.

With the development of deep learning, neural network-based methods began to emerge. These methods usually capture the sequential dependencies in log sequences by infusing RNNs and their variants, e.g., LSTMs and GRUs. To portray the differences in patterns between normal and anomalous sequences, these methods model the patterns of log sequences into a unified feature representation. Du et al. (Du et al., 2017) predicted the log template of the next log message by employing LSTMs to learn the patterns of log sequences when the system runs normally. To capture the possible anomalies in the log parameters, the same LSTMs were applied to check the parameter values of log messages. Zhang et al. (Zhang et al., 2019) exploited bidirectional LSTMs to detect sequential anomalies with semantic features extracted by TextBoxes models (Liao et al., 2017). Meng et al. (Meng et al., 2019) captured quantitative anomalies by introducing the number of templates and detecting linear relations between log messages. Li et al. (Li et al., 2020) introduced time intervals in log sequences, and the bidirectional LSTMs were used to capture both the sequential dependencies and the patterns of time intervals.

To capture the complex log patterns, some methods enhance the perception of the features by introducing component sequences, auxiliary datasets, and multi-scale designs. Yin et al. (Yin et al., 2020) thought that the calling relations of components also contain log patterns and extracted the component sequences from log data as the inputs of the LSTMs. Nedelkoski et al. (Nedelkoski et al., 2020) introduced auxiliary datasets from other systems as the negative sample set, and assumed that the effectiveness of log anomaly detection methods mainly depended on the ability of the model to distinguish between normal and anomalous sequences. Wang et al. (Wang et al., 2021) introduced a multi-scale design to slice the log sequence into fixed-length subsequences containing local log patterns, and LSTMs were used to extract the sequential dependencies at different scales.

Besides RNNs, other deep neural networks, e.g., GANs, CNNs, and GNNs, are also employed to address different challenges. Han et al. (Han and Yuan, 2021) and Xia et al. (Xia et al., 2021) introduced GANs to generate new log data, which can enhance the ability of the model to capture anomalous information. To model the temporal correlation of log templates, Zhang et al. (Zhang et al., 2021) and Wang et al. (Wang et al., 2022) arranged log templates as a matrix and captured the temporal dependencies in log sequences with CNNs and their variants. Wan et al. (Wan et al., 2021) transferred the log sequence into a graph, where nodes represent log templates and edges represent the order of log templates in the sequence, and GNNs were employed to model the information interactions in the graph and capture the log patterns. However, these methods ignore the interactions of subsequences, which can demonstrate the complex log patterns of components.

In this work, we propose CSCLog, a Component Subsequence Correlation-Aware Log anomaly detection method, which extracts subsequences from log sequences based on components and introduces an implicit correlation encoder to model their implicit correlations adaptively.

The framework of CSCLog is shown in Fig. 2, which consists of three modules: (1) subsequence modeling; (2) implicit correlation modeling; (3) subsequence feature fusion. For the subsequence modeling module, subsequences are extracted from the log sequence based on components, and LSTMs are introduced to capture the sequential dependencies in subsequences. For the implicit correlation modeling module, an implicit correlation encoder is introduced to model the implicit correlations of subsequences adaptively, and GCNs are employed to accomplish the interactions of subsequences, which can model the influences between subsequences. For the subsequence feature fusion module, attention mechanisms are exploited to fuse the embeddings of all subsequences. The details of these modules are introduced in Sections 4.2 to 4.4.

We extract subsequences from the log sequence, where the log messages in the same subsequence contain the same component, and create empty subsequences for components that are not contained in the log sequence. We denote the set of subsequences as $C=\left\{P_{1},P_{2},\ldots,P_{N_{\rm c}}\right\}$, where $N_{\rm c}$ is the number of components.

To capture anomalous information with different features in log data, we extract semantic and temporal features in log sequences and subsequences. The semantic features are the descriptions of the unstructured text in the log templates. We first create a set of keywords extracted from the text of the log templates by removing the non-character tokens and prepositions. Then, to obtain semantic information of the words in the set, we map each word to a 768-dimensional word vector by a pre-trained BERT model (Sun et al., 2019). Finally, we calculate the weighted average of the vectors of all keywords in the log template to obtain its semantic vector $\boldsymbol{v}^{\prime}$, which is formalized as:

where $\boldsymbol{v}_{i}$, $w_{i}$, and $N_{\rm e}$ denote the word vector of the $i$th keyword, the Term Frequency-Inverse Document Frequency (TF-IDF) weight of the $i$th keyword, and the number of keywords in the log template, respectively. By extracting semantic features for all log messages in the log sequence, we obtain its semantic features $\boldsymbol{V}\in\mathbb{R}^{N\times 768}$ of the log sequence, where $N$ denotes the length of the sequence.

Temporal features focus on extracting the difference in time of log messages. Specifically, the timestamp of the first log message in the log sequence is regarded as the start time, and we take the interval of the timestamp of a log message and the start time as its temporal feature $t^{\prime}$, which is formalized as:

where $t_{i}$ denotes the timestamp of the $i$th log message, $t_{i}\in\mathbb{N}$. By extracting the temporal features for all log messages in the log sequence, we obtain its temporal features $\boldsymbol{t}\in\mathbb{R}^{N\times 1}$.

We introduce a feature extractor to map features with different dimensions into a low-dimensional feature space. Semantic and temporal features are first sent to a corresponding Multilayer Perceptron (MLP) to obtain semantic and temporal embeddings, whose dimensions are $N\times d_{\rm sem}$ and $N\times d_{\rm tim}$, respectively. Then, we concatenate these two embeddings to obtain the feature embedding $\boldsymbol{X}\in\mathbb{R}^{N\times d}$, which is formalized as:

where $\varphi_{1}$ and $\varphi_{2}$ denote embedding operations, which are accomplished by MLP, and ${\rm Concat}$ denotes the concatenation operation. In addition, to balance the effects of semantic and temporal features on model performance, a threshold $\alpha_{\rm emb}\in(0,1)$ is applied to control the dimensions of the two embeddings, i.e., $d_{\rm sem}=\alpha_{\rm emb}d$ and $d_{\rm tim}=(1-\alpha_{\rm emb})d$.

Finally, we employ the LSTMs to capture the sequential dependencies in the log sequence and obtain its sequential embedding $\boldsymbol{x}^{\prime}\in\mathbb{R}^{1\times d^{\prime}}$, which is formalized as:

Similarly, we obtain the sequential embeddings of all the subsequences $\boldsymbol{X}_{\rm c}\in\mathbb{R}^{N_{\rm c}\times d^{\prime}}$, where $N_{\rm c}$ denotes the number of components. The parameters of the LSTMs are shared between subsequences.

To model the interactions of subsequences, we introduce the implicit correlation encoder to learn the implicit correlation of different subsequences end-to-end. Specifically, to avoid the correlation of any subsequence being lost, the embeddings of subsequences are concatenated in pairs based on the assumption that interactions exist between any subsequence pair (Kipf et al., 2018). We obtain the correlation embedding $\boldsymbol{X}_{\rm edge}\in\mathbb{R}^{N_{\rm edge}\times 2d^{\prime}}$, where $N_{\rm edge}$ denotes the number of subsequence pairs. The correlation embedding of a subsequence pair is formalized as:

where $\boldsymbol{X}_{\rm c}^{i}$ and $\boldsymbol{X}_{\rm c}^{j}$ denote the embedding of the $i$th and $j$th subsequences, respectively, $\sigma_{1}$ denotes the ReLU activation function, and $\varphi_{3}$ denotes the MLP. Then, we obtain the correlation weight $\boldsymbol{x}_{\rm rel}\in\mathbb{R}^{N_{\rm edge}\times 1}$ of each subsequence pair based on their correlation embeddings, which is formalized as:

where $\boldsymbol{X}_{\rm edge}^{i,j}$ denotes the correlation embedding of the $i$th and $j$th subsequences, ${\rm Conv}$ denotes the convolution operation, and $\sigma_{2}$ denotes the Sigmoid activation function, which remains the correlation weight between 0 and 1.

Finally, GCNs are employed to accomplish the interactions of subsequences. Specifically, we construct the implicit correlation graph, in which the embeddings of subsequences are regarded as the embeddings of nodes in the graph, and the correlation weights between different subsequences are regarded as the weights of edges, which indicate the importance of the interactions of subsequences. The updated subsequence embedding $\boldsymbol{X}_{\rm m}\in\mathbb{R}^{N_{\rm c}\times d^{\prime}}$ is obtained by stacking the GCN layers, which is formalized as:

where $\boldsymbol{X}_{\rm c}$ denotes embeddings of subsequences, $\boldsymbol{x}_{\rm rel}$ denotes the correlation weight, ${\rm GCN}$ denotes the message passing operation, and $\sigma_{3}$ denotes the ReLU activation function.

Attention mechanisms are employed to fuse the embeddings of different subsequences without considering the effect of positional relations. Specifically, we introduce a global vector $\boldsymbol{u}_{\rm att}\in\mathbb{R}^{1\times d^{\prime}}$, and the attention weights $\beta_{i}$ of the embeddings of different subsequences are calculated by the attention mechanisms (Chen et al., 2021), which is formalized as:

where ${\rm AttScore}$ denotes the attention scoring function, which is accomplished by the dot product of vectors, and $\boldsymbol{X}_{\rm m}^{i}$ denotes the embedding of the $i$th subsequence. To avoid the weight $\beta_{i}$ jumping between two neighboring iterations, resulting in unstable training, we introduce a constraint on it, which is formalized as:

where $\gamma$ denotes the dynamic threshold and $\beta_{i}^{j}$ denotes the attention weight of the $i$th subsequence at the $j$th iteration. Then, we fuse the embeddings of all subsequences to get the fused embedding $\boldsymbol{x}_{\rm att}\in\mathbb{R}^{1\times d^{\prime}}$, which is formalized as:

where $\boldsymbol{X}_{\rm m}^{i}$ denotes the embedding of the $i$th subsequence and $\beta_{i}$ denotes the attention weight of the $i$th subsequence.

Finally, to obtain the probabilities of the log template of the next log message $\hat{\boldsymbol{x}}\in\mathbb{R}^{N_{\rm event}\times 1}$, where $N_{\rm event}$ denotes the number of log templates, we concatenate the fused embedding with the sequential embedding of the log sequence and send it to the classifier, which is formalized as:

where $\boldsymbol{x}^{\prime}$ denotes the sequential embedding, $\boldsymbol{x}_{\rm att}$ denotes the fused embedding of subsequences, $\varphi_{\rm out}$ denotes the transformation function, which is implemented by the MLP, and $\sigma_{\rm out}$ denotes the Softmax activation function.

CSCLog is trained on the template prediction task. The training objective is to minimize the cross-entropy loss, which is formalized as:

where ${\hat{\boldsymbol{x}}}_{i}$ denotes the predicted probability of the $i$th log template, and $y_{i}$ denotes whether the predicted log template is the same as the true log template, which is formalized as:

To improve the efficiency of the calculation of the model, the input data are calculated in parallel using the batch processing method.

To determine whether a log sequence is normal or anomalous, we use a sliding window to get a set of log sequence samples. Each sample $S_{\rm t}=\left\{m_{1},m_{2},\ldots,m_{N_{\rm w}}\right\}$, where $N_{\rm w}$ denotes the length of the sliding window, is sent to the trained model to get the set of top-k ranked templates. $S_{\rm t}$ is regarded as an anomaly sample when the true log template of the next log message is not in the set, which is formalized as:

where $e_{\rm t}$ denotes the true log template of the next log message. We consider that an anomaly occurs in the log sequence when the number of anomalous samples reaches the set threshold $\alpha_{\rm anom}$.

To evaluate the performance of CSCLog, we conduct experiments on four public log datasets, i.e., HDFS (Xu et al., 2009), BGL (Oliner and Stearley, 2007), ThunderBird (Oliner and Stearley, 2007), and OpenStack (Du et al., 2017), which contain anomaly labels. The details of the experimental datasets are shown in Table 1.

HDFS: The HDFS dataset is generated by Hadoop-based MapReduce jobs deployed on more than 2,000 Amazon’s EC2 nodes, which contains 11,175,629 log messages for 39 hours. The log sequences are extracted directly based on the block_id in a log message, which are manually labeled as anomaly or normal by the Hadoop domain experts.

BGL: The BGL dataset contains 4,747,963 log messages generated by the BlueGene/L supercomputer deployed at Lawrence Livermore National Laboratory, with a time span of 7 months. Each log message in the dataset is manually labeled as anomaly or normal by the domain experts. A sliding window with the length of 10 seconds is used to extract the log sequences. Log sequences are labeled as anomaly when containing anomalous log messages.

ThunderBird: ThunderBird is a large dataset of over 200 million log messages, generated on the ThunderBird supercomputer system at Sandia National Laboratories (SNL). We extracted 2 million continuous log messages from 9:18:40 on November 11, 2005 to 20:28:59 on November 13, 2005, with a time span of about 59 hours. Similar to BGL, each log message is manually labeled as anomaly or normal by the domain experts. A sliding window with the length of 10 seconds is used to extract the log sequences. Log sequences are labeled as anomaly when containing anomalous log messages.

OpenStack: OpenStack is a small dataset generated by a cloud operating system with 10 compute nodes, which contains 207,636 log messages for 30 hours. The dataset is divided into two normal log files and an anomaly file. A sliding window with the length of 10 seconds is used to extract the log sequences, and the log sequence is labeled with the same label as the file.

CSCLog is implemented in Python with PyTorch 1.10.0 (Paszke et al., 2019) and PyTorch Geometric 2.1.0 (Rozemberczki et al., 2021), and the source code is released on GitHub¹¹1https://github.com/Hang-Z/CSCLog. We conduct all the experiments with an Intel Xeon Gold 5118 CPU and an NVIDIA GeForce RTX 2080Ti GPU. We parse the log messages by a parsing tool Drain (He et al., 2017) with default parameters. All the datasets are split into training, validation, and test sets by the ratio of 7:1:2 according to the temporal order of log sequences.

Adam (Kingma and Ba, 2015) is adopted as the optimizer, and the weight decay rate is set to 0.0001. The number of iterations is set to 20. To avoid overfitting, we apply the early stop (Yao et al., 2007) in the training stage, where the stop patience is set to 50% of the number of iterations by default. The number of layers of the GNNs in implicit correlation encoder is set to 2. The number of layers of the LSTMs in subsequence modeling module is set to 2. The step length of the sliding window for sampling is set to 1. The global vector $\boldsymbol{u}_{\rm att}$ in subsequence feature fusion module is initialized with the Xavier initialization. For other hyper-parameters in the model, we exploit the Neural Network Intelligence toolkit (NNI)²²2https://nni.readthedocs.io/en/latest/ to search for the best value automatically. Due to the differences in the number of log templates for different datasets, the hyper-parameter number of top-k ranked templates is chosen differently for them. The search spaces of these hyper-parameters and the configures of NNI are given in Table 2. The best hyper-parameters on different datasets are given in Table 3.

We use Accuracy, Precision, Recall, and Macro F1-Measure as evaluation metrics. Macro F1-Measure combines Precision and Recall and allocates same weight to every class, which reduces the impact of class imbalance on the evaluation results, and higher Macro F1-Measure demonstrates better performance, which is formalized as:

where $k$ denotes the index of classes and $N_{\rm class}$ denotes the number of classes. The final reported result is the average macro F1-Measure across five independent runs by varying seeds.

To justify the superiority of CSCLog, we compare it with other log anomaly detection methods, all of which are unsupervised or self-supervised learning-based methods. Among these methods, Invariant Mining (IM) (Lou et al., 2010), Principal Components Analysis (PCA) (Xu et al., 2009), and LogCluster (Lin et al., 2016) are traditional methods, and DeepLog (Du et al., 2017), LogAnomaly (Meng et al., 2019), LogC (Yin et al., 2020), and OC4Seq (Wang et al., 2021) are neural network-based methods. Due to the additional introduction of anomalous information, supervised learning-based methods usually perform better than unsupervised or self-supervised learning-based methods (Le and Zhang, 2022), so that they are not considered in our comparison experiment. We conduct experiments on the anomaly detection task and template prediction task, respectively. All compared methods are implemented by the code provided by the original papers. For the sake of fairness, all compared methods follow the same experimental settings and search spaces of hyper-parameters as CSCLog, and their parameters are also optimized. In addition, the kernel size of the one-class classifier of OC4Seq is set to 5 according to the original paper. The detailed descriptions of baselines are as follows:

IM (Lou et al., 2010), PCA (Xu et al., 2009), and LogCluster (Lin et al., 2016): IM, PCA, and LogCluster are traditional methods that detect anomalies by extracting the statistical features in log sequences. To use them on the log anomaly detection task, we extract the number of log templates for IM and PCA, and construct the set of frequent words in log messages for LogCluster. To a newly arriving log sequence, we detect whether it is anomalous according to the patterns captured from the statistical features.

DeepLog (Du et al., 2017): DeepLog predicts the log template of the next log message by modeling the log sequences as natural language sequences and employing LSTMs to learn the patterns of log sequences when the system runs normally. To a newly arriving log sequence, DeepLog detects whether it is anomalous according to whether the true log template is in the predicted log templates.

LogAnomaly (Meng et al., 2019): LogAnomaly captures quantitative anomalies by introducing the count vector of log templates in log sequences, and captures sequential anomalies by learning the patterns of log sequences when the system runs normally. Similar to DeepLog, LogAnomaly detects whether a newly arriving log sequence is anomalous according to whether the true log template is in the predicted log templates.

LogC (Yin et al., 2020): LogC extracts the component sequences from log data, and captures the calling relations of components and the patterns of log sequences when the system runs normally. To a newly arriving log sequence, LogC detects whether it is anomalous according to whether the true log template is in the predicted log templates and the true component is in the predicted components.

OC4Seq (Wang et al., 2021): OC4Seq introduces a multi-scale design to slice the log sequence into fixed-length subsequences containing local log patterns. LSTMs are used to capture the sequential dependencies at different scales. A one-class classifier is introduced to detect whether a newly arriving log sequence is anomalous by capturing the features of sequential embeddings.

Table 4 shows the Accuracy, Precision, Recall, and Macro F1-Measure of CSCLog and baselines on the anomaly detection task on HDFS, BGL, ThunderBird, and OpenStack datasets, from which we can observe the following phenomena:

1) Neural network-based methods (DeepLog, LogAnomaly, LogC, and OC4Seq) outperform traditional methods (IM, PCA, and LogCluster), which indicates the effectiveness of capturing the sequential dependencies in log sequences. In addition, the high differences in the performances of traditional methods on different datasets demonstrate the limitations of extracted statistical features.

2) The performance of LogAnomaly is slightly better than that of DeepLog, which indicates that the introduced number of templates can enhance the ability to capture anomalous information. However, LogAnomaly fails to perform well on the BGL dataset. The reason is that the large number of log templates (over 1000) contributes to the sparse count vector of templates, so that the model fails to capture the linear relations between log templates.

3) LogC fails to perform well on the ThunderBird dataset, which contains a large number of components. The reason is that LogC focuses on modeling the calling relations of components, and the large number of components make the relations more complex, which presents a challenge to the model to capture the complex log patterns of components.

4) As the SOTA method on the anomaly detection task, OC4Seq performs better than DeepLog, LogAnomaly, and LogC. OC4Seq can capture both global and local sequential dependencies in log sequences by slicing the subsequences of different lengths. In addition, by introducing the one-class classifier to classify log sequences directly, OC4Seq reduces the impact of the length of sequences on performance.

5) CSCLog outperforms the best baseline with relative improvements of 12.86%, 5.64%, 1.19%, and 9.96% in Macro F1-Measure on HDFS, BGL, ThunderBird, and OpenStack datasets, respectively. The results justify the advantage of capturing the sequential dependencies in subsequences and modeling the implicit correlations of subsequences.

Table 5 shows the Accuracy, Precision, Recall, and Macro F1-Measure of CSCLog and neural network-based methods on the template prediction task on the HDFS dataset. To demonstrate the comprehensive results, we set the number of top-k ranked templates to 1, 3, and 5. We replace the one-class classifier of OC4Seq with the same classifier as CSCLog to accomplish the template prediction task. We can observe that CSCLog outperforms all the baselines and performs better than the best baseline with relative improvements of 3.37%, 14.97%, and 5.85% in Macro F1-Measure. This further demonstrates the advancement of CSCLog.

To justify the advantage of capturing the sequential dependencies in subsequences and modeling the implicit correlations of subsequences, we compare CSCLog with two simplified models. For the sake of fairness, all variants follow the same experimental settings as CSCLog, and their parameters are also optimized. The detailed descriptions of simplified models are as follows:

CSCLog w/o IC: CSCLog w/o IC removes the implicit correlation encoder and sets the correlation weight between subsequences to 1, which means that the correlations of subsequences are fixed and identical. The rest is the same as CSCLog.

CSCLog w/o LSTM: CSCLog w/o LSTM removes the LSTMs of the subsequence modeling module from CSCLog and only utilizes the average of the feature embeddings of a log sequence as its sequential embedding, which means that the model cannot capture the sequential dependencies. The rest is the same as CSCLog.

Table 6 shows the Accuracy, Precision, Recall, and Macro F1-Measure of CSCLog and simplified models on the anomaly detection task on the HDFS dataset, from which we can observe that CSCLog outperforms CSCLog w/o IC, which indicates the effectiveness of using the implicit correlation encoder to model the implicit correlation of subsequences. CSCLog outperforms CSCLog w/o LSTM, which indicates the effectiveness of employing LSTMs to capture the sequential dependencies in log sequences. The performance degradation of CSCLog w/o IC is more than that of CSCLog w/o LSTM, which indicates that modeling the implicit correlation of subsequences is more effective to improve the ability to capture anomalous information.

To justify the effectiveness of introduced features, we remove semantic and temporal features to study the impact on the model. CSCLog w/o SEM denotes the model without semantic features and CSCLog w/o TIME denotes the model without temporal features.

Table 7 shows the Accuracy, Precision, Recall, and Macro F1-Measure of CSCLog and simplified models on the anomaly detection task on the HDFS dataset, from which we can observe that CSCLog outperforms CSCLog w/o SEM and CSCLog w/o TIME, which indicates the effectiveness of the introduced semantic and temporal features. The performance degradation of CSCLog w/o SEM is more than that of CSCLog w/o TIME, which indicates the importance of semantic features. The reason is that the semantic features contain rich text information of log templates.

To study the impact of several important parameters, including the length of sliding window, the number of top-k ranked templates, and the threshold of anomaly detection, we conduct the parameter sensitivity analysis on the anomaly detection task.

We evaluate the impact of the length of sliding window, increasing it from 9 to 23 with a step size of 2. Fig. 5 shows the evaluation results on the HDFS dataset, from which we can observe that with the increase of the length of sliding window, the performance of CSCLog rises gradually. The best performance is achieved when the length of sliding window is 19. The reason is that when the length of sliding window increases from 9 to 19, the patterns of log sequences increase and the ability to capture anomalous information is improved. As the length of sliding window increases from 19 to larger, the performance gradually drops. The reason may be that the increase of sliding window causes the increase of the number of extracted subsequences, which brings a challenge for the model to capture the correlation of subsequences.

We evaluate the impact of the number of top-k ranked templates, increasing it from 1 to 11 with a step size of 2. Fig. 5 shows the evaluation results on the HDFS dataset, from which we can observe that when the number of top-k ranked templates is less than 7, the performance rises as the number increases, which indicates that increasing the number of top-k ranked templates within a certain range can improve the fault tolerance of the model. When the number of top-k ranked templates is larger than 7, the performance drops slightly as the number increases. The reason may be that when the number is too large, it is easy for the top-k ranked templates to hit the true log template, which makes the model misclassifies the anomalous sequences as normal.

We evaluate the impact of the threshold of anomaly detection, increasing it from 1 to 5 with a step size of 1. Fig. 5 shows the evaluation results on the HDFS dataset, from which we can observe that with the increase of the threshold of anomaly detection, the performance first increases and then drops. The best performance is achieved when the threshold of anomaly detection is 4. The reason may be that when the threshold is set too small, the log sequences are easy to be detected as anomalies, which contributes to the higher false positive rate and the lower accuracy. When the threshold is set too large, the log sequences are difficult to be detected as anomalies, which contributes to the higher false negative rate and the lower recall.

We evaluate the computation costs of CSCLog and baselines, including the parameter number, training time, and inference time. Table 8 shows the evaluation results on the HDFS dataset, from which we can observe that DeepLog has the least parameter number and runs fastest in these methods, but it gets the worst anomaly detection performance. Compared with LogAnomaly, LogC, and OC4Seq, CSCLog gets the best anomaly detection performance while needing the least time for inference, which is more in line with the requirements of modern software systems for real-time anomaly detection. Overall, comprehensively considering the significant anomaly detection performance improvement and the computation costs, CSCLog exhibits its superiority over existing methods.

To intuitively reveal the superiority of CSCLog, we perform several case studies.

In Fig. 6, we use the t-SNE method (Van der Maaten and Hinton, 2008) to visualize the embeddings of the output of the final MLP (before the Softmax activation function) on the OpenStack dataset. The log sequence samples whose log templates of the next log message are “T1”, “T2”, “T3”, “T4”, and “T5” are selected. We can find that the sample embeddings of CSCLog are more clustered than those of DeepLog, and the sample embeddings of different log templates are more distantly distributed. It means that CSCLog can better capture the complex log patterns by modeling the implicit correlations of subsequences.

Table 9 shows the predicted top-5 ranked templates of two log sequence samples on the template prediction task on the OpenStack dataset, where the two samples are labeled as normal and anomaly and their log templates order are both “T1, T2, T1, T2, T3, T4, T3, T4, T5, T3”. We can find that to the normal sample, both DeepLog and CSCLog predict the log template of the next log message correctly, and CSCLog has a higher ranking to the true log template. To the anomalous sample, the true log template is in the predicted top-5 ranked templates of DeepLog but not in that of CSCLog, which means CSCLog can detect the anomaly while DeepLog cannot. It indicates that CSCLog can improve the ability to capture anomalous information by introducing the implicit correlation encoder and modeling the correlation of subsequences.

Fig. 7 shows the distribution of the correlation weights of the subsequences of the above two log sequence samples. These two samples contain components “nova.scheduler.host.manage”, “nova.meta-data.wsgi.server”, and “nova.osapi_compute.wsgi.server”, whose subsequences are labeled as 0, 1, and 2, respectively. To the normal sample, subsequences 0, 1, and 2 are “T1, T2, T1, T2”, “T3, T4, T3”, and “T3, T4, T5”, respectively. To the anomalous sample, subsequences 0, 1, and 2 are “T1, T2, T1, T2”, “T3, T4, T3”, and “T4, T5, T3”, respectively. We can find that in the normal sample, subsequences 1 and 2 are strongly correlated, and subsequences 0 and 1 are weakly correlated. In contrast, in the anomalous sample, the correlation between subsequences 0 and 1 is extremely strong, and the correlations between the remaining two subsequence pairs are weak. It indicates that CSCLog can capture the complex log patterns of components.