ASGNet: Adaptive Semantic Gate Networks for Log-Based Anomaly Diagnosis

In this research, the log-based anomaly diagnosis task can be described as a tuple of three elements $(S,I,y)$, where $S=[s^{1},s^{2},\dots,s^{g}]$ represents the log sequence whose length is $g$. $I$ denotes the current log message task ID and $y\in Y$ conveys the anomaly diagnosis specific labels for logging exceptions, currently diagnosable exceptions are Stream exception, Connection broken, Redundant request, Unexpected error, etc.

In this section, we first give the formal definition of the log statistics vector, which is defined as follows. Given a word $w$ in a log message and a set of log exception labels $C=\{c_{1},c_{2},\cdots,c_{n},\}$, the statistical information vector of $w$ is:

The statistical information vector captures the global distribution of log anomaly labels as a feature of the words in the log.

These statistical features are primitive but can be used for feature selection by determining word-relatedness. Intuitively, if a word $w$ has very high or very low frequency across all labels, then we can assume that $w$ has a limited contribution to the anomaly diagnosis task. In contrast, if a word appears more frequently in specific label class, we assume this word is discriminative. Here, the term statistical label vector dictionary $E$ is only obtained from the training set.

Inspired by the text classification [13],Log Statistics information Representation (V-Net) aims to convert statistical features into an efficient representation. Log statistics vectors contain integer counts of words, thus initially incompatible with semantic features in both dimension and scale. Therefore, V-Net employs an autoencoder to map discrete vectors of log statistics into a latent continuous space to obtain a global representation of the statistics. In this work, we employ a variational autoencoder(VAE)[15] to encode log statistics vectors.

We generate log statistics vectors for all log messages in the data set to obtain $S=\{E_{(i)}^{L}\}_{i=1}^{N}$, which consists of $N$ discrete log statistics Vector composition. We assume that all log statistics vectors are generated by a stochastic process $p\theta(E|S)$ involving a latent variable S sampled from a prior distribution $p\theta(S)$. Since the posterior $p\theta(S|E)$ is intractable, we cannot directly learn the generative model parameters $\theta$. Next, we employ a variational approximation $q\phi(S|E)$ to jointly learn the variational parameters $\phi$ and $\theta$. Therefore, we can optimize the model by maximizing the marginal likelihood that is composed of a sum over the marginal likelihoods of individual $E$:

Because the $KL$ divergence term is non-negative, we can derive the likelihood term ${\L}(\theta,\phi;E)$ to obtain a variational lower bound on the marginal likelihood, i.e.:

By training an unsupervised VAE model, we can obtain latent variables $E^{s}$ through a probabilistic encoder, which would be a global representation of statistical features. The training of V-Net is independent of the main classifier, and the representation $E^{s}$ is generated in the preprocessing stage and is fed to the classifier through the adaptive semantic threshold mechanism.

Log Deep Semantic Representation (S-Net) extracts semantic features from log message input and projects the semantic features into the information space for confidence evaluation. The input of S-Net is a log message $W=[w_{i},w_{2},\cdots,w_{m},]$ with fixed length $m$.

In this paper, we use the pre-training model to obtain the semantic representation of the log. Specifically, we use the pre-trained RoBERTa[16] to extract the feature map of the input log text:

Then, we map the semantic feature map $C$ into the information space through dense layers. We use the sigmoid to activate values in the representation.

As described in Section 2.3, the semantic representation of log statistics features $E^{s}$ is obtained offline. To flexibly utilize statistical features, we apply dense layers to project $E^{s}$ into the information space shared with semantic features:

The valve component is fused with $H^{C}$ and $H^{E}$, and the semantic feature map $H^{O}$ enhanced by statistical information is output through the AdaSemGate function,

We employ global attention to combine the consolidated semantic representation $H^{O}$ with the original feature map $C$:

Note that if we reject all statistical information (i.e., $\epsilon=0$),Eqn.(11) will become self-attention[22] as $H^{O}=C$.

After passing through fully-connected layers and a softmax layer, feature vectors are mapped to the label space for label prediction and loss calculation. To maximize the probability of the correct label $Y_{True}$, we deploy an optimizer tominimize cross-entropy loss $L$.

We evaluate our model ASGNet on seven public datasets. The statistics of these seven datasets are listed in Table 2.

We fix all the hyper-parameters applied to our model. Specifically, We use the basic version of RoBERTa as the pre-trained embeddings in our experiments. The $\epsilon$ set to 0.2, which empirically shows the best performance. The algorithm we choose for optimization is Adam Optimizer with the first momentum coefficient $\beta_{1}$= 0.9 and the second momentum coefficient $\beta_{2}$=0.999. We use the best parameters on development sets and evaluate the performance of our model on test sets.

To analyze the effectiveness of our model, we take some of the most advanced methods as baselines on the above-mentioned seven datasets, to validate the performance of our ASGNet model. The results are demonstrated as follows.

As shown in Figure 3, NeuralLog [23] , LogRobust [9] and LogAnomaly [24] are strongest baseline models of the Log-Based Anomaly Detection task. The experiment results show that our model ASGNet achieves best performance on seven datasets. For instance, on the BGL dataset, our model outperforms baseline model NeuralLog [23] by 2.17% in F1-Score (p $<$ 0.05 on student t-test). LogRobust [9] by 1.3% in F1-Score (p $<$ 0.05 on student t-test). LogAnomaly [24] by 4.37% in F1-Score (p $<$ 0.05 on student t-test). It also showed good performance compared to the comparison method on the other six datasets.

The reason is our proposed model takes into account not only the semantic features behind the log execution logic but also the statistical features of the log text in the log exception diagnosis task. In order to better fit the two features, we propose well-designed threshold mechanism which effectively selects the statistical features to consolidate the semantic features, instead of using all statistical features. The threshold mechanism introduces the information flow into the classifier based on the confidence of the semantic feature in the decision, which is conducive to training a robust classifier and can solve the overfitting problem caused by the use of statistical features.Therefore the final experimental results demonstrate that our method achieves remarkable performance in the log-based anomaly diagnosis task.

To thoroughly figure out the effect of each key model component, we carry out a series of ablation study to decompose the whole model into three derived models.

Model (1): only use log statistics information representation – The entire model only uses the statistics information Representation of the log sequence.

Model (2): only use log deep semantic representation – The entire model only uses the log deep semantic representation of the log sequence.

Model (3): without adaptive semantic gate networks – The entire model excludes the adaptive semantic gate networks.

As shown in Figure 4, we can see that in our model, the log deep semantic representation module contributes more to task log-based anomaly diagnosis than the log statistics information representation module, mainly because the semantic information of the text of the logs can better express the deep logical semantic information of the logs, while the statistical features can only express the high and low frequency distribution of words, so log deep semantic representation module shows more advantages. In addition, it can be seen that the adaptive semantic gate module is indispensable in the whole component of our model, and removing the adaptive semantic gate module will affect the overall performance of the text proposed model.

As shown in Figure 5, firstly, we can see that on two datasets the f1-score of our model shows an upward trend when the dimension size is less than 300, especially achieving highest when the dimension size is exactly 300, which indicates that a large dimension size could contribute to model performance. However, when the dimension size is larger than 300, the F1-Score of the model drops on both HDFS and BGL Dataset, possibly due to insufficient training data.

Secondly, we compare the performance of our model using the gate function $\epsilon$ on two datasets. As illustrated in Figure 5, our model achieves best f1-score with the $\epsilon=0.2$ thus demonstrating that not all statistical features are useful for the model.