Incorporating Gradients to Rules: Towards Lightweight, Adaptive Provenance-based Intrusion Detection

The first step of solving the optimization problem is to define the objective function, i.e., the loss function in the context of learning tasks. The learning module aims to find the parameters that can reduce false alarms while maintaining the sensitivity to the true malicious events. Therefore, the loss function comprises two terms: the term of false alarms and the regularizer term.

The false alarm term mainly focuses on penalizing erroneously triggered alarms. Specifically, an event $e$ triggers a false alarm means $f(e)$ should be greater than 0 but it does not. Since $f(e)\in[-1,1]$, we can use the Mean Squared Error $(y_{e}-f(e))^{2}$ as the loss function for all events that trigger a false alarm. $y_{e}$ is set as $1$ for those benign events, and $-1$ for the malicious events. Because we only use benign data for training, all $y_{e}$ should be $1$. Please also note that we do not calculate the loss for the correctly classified events, i.e., we do not make $f(e)$ close to $1$ if $f(e)$ is already greater than $0$. This is because the events that do not cause alarms are significantly more than the alarm-triggering events, considering them, therefore, is inefficient and would make the detection system insensible to the malicious events. Thus, the false alarm term in the loss function can be formalized as

$$\mathcal{L}(e)=max(0,(1-f(e))^{2}-1)$$

(3)

Next, we introduce the second component: the regularizer term. As we mentioned, Captain is trained on benign data to learn the normal behaviors in the detection environment. This is due to the fact that benign behaviors on a system are more consistent than attack activities. Another reason is the malicious training data is much harder to acquire. However, a big challenge is how to guarantee that the detection capability would not be affected if there is no malicious sample in the dataset, i.e. we would not be too lenient to capture the true alarm when reducing false alarms.

To capture all potential malicious events, the adaptive parameters are configured conservatively before training. During training, we only want to make small adjustments to a small portion of the parameters according to the false alarms on training data. For the rest parameters, we would like to keep them as conservative as possible so that the sensitivity to the maliciousness is not compromised. In the area of machine learning, One-class classification (OCC) algorithms are proposed to deal with the situation where only one class of samples is available in the training set [45]. Inspired by [46, 47], we add a regularizer term in the loss function to avoid the parameters becoming too lenient when the malicious training sample is absent. The essential thought is to make the adaptive parameters as close to the default values ($A_{0},G_{0},T_{0}$) as possible. By minimizing the $l_{2}$ distance between the adaptive parameters and the default values, we protect the detection capability during training. Therefore, the detection loss can be formalized as

$$Loss=\sum_{e\in E}\mathcal{L}(e)+\alpha||A-A_{0}||_{2}+\gamma||G-G_{0}||_{2}+\tau||T-T_{0}||_{2}$$

(4)

where, $\alpha$, $\gamma$, and $\tau$ are the regularizer coefficients. In the evaluation section, we discuss the effect of the regularizer coefficients (§VI-D2) and methods used to fine-tune them (§VI-A2).

One of the fundamental steps of using gradient descent algorithms is calculating the gradient of each variable. To adjust the adaptive parameters based on the loss function, we need to design a differentiable detection framework, which allows us to keep the gradients of the adaptive parameters with respect to $Loss$.

According to the chain rule, for the parameter $a_{n}$, we have

$$\frac{\partial Loss}{\partial a_{n}}=\sum_{e\in E}\frac{\partial\mathcal{L}(e)}{\partial f}\cdot\frac{\partial f}{\partial a_{n}}+\alpha\cdot(a_{n}-a_{0})$$

(5)

The equations for $g_{e}$ and $thr_{e}$ (presented in Appendix §C) hold similarly to Eq. 5. The most challenging part of building an adaptative rule-based PIDS is to compute the gradients, i.e., $\partial f/\partial a_{n}$, $\partial f/\partial g_{e}$, and $\partial f/\partial thr_{e}$. This is because the rule-based detection process is usually modeled as the branch selection based on rules, rather than a differentiable function. To the best of our knowledge, no previous work has formalized or recorded the gradients of parameters in a rule-based PIDS. In Captain, we model the detection process as a differentiable function and calculate the gradients of each adaptive parameter. When the tags are propagated, the corresponding gradients are updated and recorded. In so doing, we associate the adaptive parameters with the loss, making it possible to perform the gradient descent algorithm to find the optimal parameters that minimize the loss.

As per Eq. 2, $\partial f/\partial thr_{e}=-1$. This means every time when we want to increase the value of $f$ (the direction of benign), we have to decrease the value of $thr_{e}$, and vice versa. This aligns with our intuition because a lower tag value means more maliciousness in our system. Therefore, if we want to be more lenient in the detection, we should set a lower threshold.

The formalization and calculation of $\partial f/\partial a_{n}$ and $\partial f/\partial g_{e}$ are more complicated. According to Eq. 2, calculating the gradients of the detection function $f$ is essentially computing the gradients with respect to the variable $tag$ for each node.

We start from the gradients of $a_{n}$. According to the propagation policies shown in Table VIII, the updated tag value $tag_{rule}$ is either the lower tag values of involved nodes $min(tag_{src},tag_{dest})$ or a constant value $c$. Note the subscripts src and dest denote the propagation direction. If $tag_{rule}=c$, then

$$\frac{\partial tag_{dest}^{new}}{\partial a_{n}}=(1-g_{e})\frac{\partial tag_{dest}}{\partial a_{n}}$$

(6)

If $tag_{rule}=tag_{src}$, then

$\displaystyle\frac{\partial tag_{dest}^{new}}{\partial a_{n}}$

$\displaystyle=g_{e}\frac{\partial tag_{src}}{\partial a_{n}}+(1-g_{e})\frac{\partial tag_{dest}}{\partial a_{n}}$

(7)

Since $a_{n}$ is defined as the initial value of the tag on node $n$, the initial gradient of $a_{n}$ is

$$\frac{\partial tag_{n^{\prime}}}{\partial a_{n}}=\begin{cases}1,&\text{if }n^{\prime}=n\\ 0,&\text{if }n^{\prime}\neq n\end{cases}$$

(8)

Then we focus on calculating gradients of $g_{e}$, which is more complicated. From Eq. 1, we know that $tag_{dest}^{new}$ is determined by $tag_{dest}$, $tag_{rule}$, and $g_{e}$. And $tag_{dest}$, $tag_{rule}$ are related to the propagation rates of previous events. Therefore, $tag_{dest}^{new}$ is influenced by the propagation rates of the current event (denoted by $g_{e}$) and all previous events (denoted by $g_{e^{\prime}}$) happened on $src$ and $dest$ nodes. Consequently, for each propagation, we must update $\partial tag_{dest}^{new}/\partial g_{e}$ and $\partial tag_{dest}^{new}/\partial g_{e^{\prime}}$ simultaneously. The case of $g_{e^{\prime}}$ is similar to that of $a_{n}$, we have

$\displaystyle\frac{\partial tag_{dest}^{new}}{\partial g_{e^{\prime}}}$

$\displaystyle=g_{e}\frac{\partial tag_{rule}}{\partial g_{e^{\prime}}}+(1-g_{e})\frac{\partial tag_{dest}}{\partial g_{e^{\prime}}}$

(9)

Calculating gradients of $g_{e}$ involves the Product Rule in calculus, which is

$\displaystyle\frac{\partial tag_{dest}^{new}}{\partial g_{e}}$

$\displaystyle=g_{e}\frac{\partial tag_{rule}}{\partial g_{e}}+(1-g_{e})\frac{\partial tag_{dest}}{\partial g_{e}}+tag_{rule}-tag_{dest}$

(10)

The full proof of Eq. 9 and Eq. 10 can be found in the Appendix §C.

As previously stated, $g_{e}$ are set according to the event features (src_node_feature,event_type,dest_node_feature). Therefore, before the propagation happens,

$$\frac{\partial tag_{n}}{\partial g_{e}}=0,\forall\ e\in E,\forall\ n\in N$$

(11)

We use Eq. 11 to initialize the gradient of $g_{e}$ and update them according to Eq. 9 and  10.

In summary, to employ the gradient descent algorithm, we calculate the gradients of the adaptive parameters with respect to the loss. We first calculate the initial gradient for each node. Afterward, the gradients are updated according to our equations when the tags are propagated. Please note that unlike GNN models such as GCN [48] and GraphSage [49], which only consider the n-hop neighbor of a node, Captain fine-tunes every edge and node in the graph that can influence the detection results by propagating their gradients.

After calculating the gradients of the adaptive parameters with respect to the loss using the differentiable detection framework, we can now utilize the gradient descent algorithm to optimize the parameters. In machine learning, this process is referred to as “training”. The learned parameters are stored as a customized configuration, which is then used to set up Captain prior to testing.

As shown in Fig. 3, an epoch in training comprises the forward and backward propagation. Before training starts, all adaptive parameters, $A$, $G$, and $T$, are configured as the default settings $A_{0}$, $G_{0}$, and $T_{0}$. Instead of random initialization[50], we use the most conservative setting as the starting point to keep the sensitivity to the maliciousness during training.

When Captain processes the events, tags are propagated among the graph, updating the gradients according to Eq. 7,  9 and  10, and generate the detection results. Next, we calculate the loss and back-propagate the gradients of loss according to Eq. 5. Finally, parameters are updated in the opposite direction of the gradient as follows:

$$p_{new}=p_{old}-l\cdot\frac{\partial Loss}{\partial p_{old}}$$

(12)

where $p$ is the adaptive parameters (which could be $a_{n}$, $g_{e}$, or $thr_{e}$), $p_{new}$ is the new parameters, $p_{old}$ is the old one, $\frac{\partial Loss}{\partial p_{old}}$ is the gradient, and $l$ is the learning rate. We repeat the training process when the maximum epoch is reached or the changes in the result become sufficiently small. Before testing, we configure the parameters according to what we learned in the training stage. Then, Captain can process audit data and conduct detection as introduced in § IV-A.

We evaluate Captain using public forensic datasets from the DARPA Transparent Computing program and datasets generated within simulated environments in collaboration with an SOC.

We deployed Captain and performed all experiments on an Ubuntu 22.04.3 Linux Server with an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz and 1.0 TB memory. We partition each dataset into training and testing sets, adhering to the assumption stated in §II-C that the training set should not contain any malicious activities. Specifically, we find the starting time of the first attack. Then, all data produced before that date becomes the training set, whereas the remaining data becomes the testing set.

To avoid biased results due to overfitting, we performed cross-validation when evaluating the performance of Captain. However, standard k-fold cross-validation is unsuitable for streaming logs because the time series cannot be freely split into k groups, as the subsequent tags depend on the previous ones. Therefore, we utilized time series cross-validation, which preserves the chronological order of data. We set a time window and trained the parameters using the data within this window. Afterward, we moved the time window forward until creating k different training sets. In our experiment, we set k=3 and the length of the time windows to be around 2/3 of the total length of the training set. We also discuss overfitting in § VII-A.

Like many other machine learning systems, Captain also relies on appropriate hyperparameters, especially three regularizer coefficients $\alpha,\gamma,\tau$, to train the accurate and robust model (the effect of these hyperparameters are shown in §VI-D2). We provide two methods to fine-tune the hyperparameters, depending on whether a validation set is available. If a validation set exists, we perform grid searching, a generic hyperparameter tuning approach. We train multiple models with different hyperparameter combinations and evaluate their performance on the validation set. The default conservative detector is run on the validation set as the baseline. We select the hyperparameters from the model with the fewest false alarms and equivalent true alarms to the baseline. For instance, the optimal hyperparameters of E3-Cadets is $\alpha=0.1,\gamma=0.1,\tau=0.1$ and the learning rate is 0.01. However, a validation set is usually not available for a PIDS in practice. Even if it exists, it may not encompass all possible attacks. In this case, we propose a heuristic-based method to fine-tune the regularizer coefficients. Recall that the training starts from the conservative adaptive parameters, false alarms in the training set will loosen the conservative settings. The regularizer coefficients are added to the loss function to avoid being too lenient and missing the true alarms. Therefore, the values of the regularizer coefficients $\alpha$, $\gamma$, and $\tau$ are determined by the following question: how many false alarms caused by a node/edge can we tolerate in the training set to avoid being overly lenient? Before training starts, we set a number for the allowed false alarms ($N$), and we can estimate an approximate value of the regularizer coefficients based on $N$: $\alpha\approx 3N,\gamma\approx 3N,\tau\approx 12N$ (see Appendix §D for the mathematical proof). Please note that these are approximate estimations depending on the extent to which we trust the “benign” training data. A practical way might be: set regularizer coefficients using heuristics; if missing alarms occur compared to the conservative baseline, multiply the coefficients by a factor (e.g., 10) and retrain. Conversely, if no alarms are missed for a long time $T$, reduce the coefficients by a factor and retrain.

We compare Captain with five SOTA PIDS: Flash [16], Kairos [15], ShadeWatcher [13], NodLink [18], and Morse [22] to evaluate their performance from different perspectives. We chose Flash, Kairos, and NodLink because they are the SOTA embedding-based PIDS. In addition, their implementations are open-sourced, allowing us to test them on different datasets. Since some other embedding-based PIDS [20, 19, 23] have already been evaluated in these works and the result shows that Flash, Kairos, and NodLink outperform them, we didn’t include them in our evaluation. We chose Morse since it is the SOTA rule-based PIDS, and we want to evaluate the improvements from our differentiable adaptation framework in Captain. To get a fair and unbiased comparison, we use the settings from Morse in the evaluation. According to [15, 16] and our communication with the authors, the detection systems of ShadeWatcher and ProGrapher are not fully open-source due to proprietary license restrictions. Although the data preprocessing module of ShadeWatcher is open-sourced, we tried our best but could not locate the preprocessing output files used for the following training and testing. We then realized that it was not feasible for us to perfectly replicate their systems for an unbiased comparison. Therefore, we used the code to evaluate the efficiency and latency of data preprocessing and the detection results reported in their paper to evaluate detection accuracy. For Flash, Kairos, and NodLink, we used their open-sourced code as the basis for the evaluation. We reimplemented Morse according to their paper.

We first focus on the reduction of false alarm events, which requires the detection granularity at the event level. We compare Captain with the SOTA event-level PIDS ShadeWatcher and the classical rule-based PIDS Morse. Due to the close-source nature of ShadeWatcher, we used the reported detection results in their paper. The results show that both Morse and Captain successfully detected all attacks in the testing dataset. But Captain reduces the false alarm rate by over 93% (15.66x) compared to ShadeWatcher and over 95% (20.45x) compared to Morse. Additionally, ShadeWatcher holds 80% of events for training and only 10% for testing, while the testing set of Captain is over seven times larger than the testing set of ShadeWatcher. This shows that Captain does not require as much data as ShadeWatcher for training.

One advantage of the rule-based PIDS like Captain and Morse is that it can offer semantic-rich alarms, while ShadeWatcher only flags deviations from patterns observed during training without explaining the alarms. We then compared the number of false alarms generated by Captain and Morse into different categories, shown in Table IV.

Table IV shows that Captain can reduce false alarms by over 90% (11.49x) on average for all datasets compared with the non-adaptive Morse. It outperforms Morse in every alarm category. Those remaining false alarms, especially in C-5, S-2, and S-3, cannot be removed for the following reasons. First, some “false” alarms are not purely benign. They are related to some (potential) attack nodes (e.g., configuration files in /tmp/atScript/atomic-red-team-Gray_dev1.0/* in S-2) but cannot be mapped to specific attack steps directly, thus not labeled as “malicious.” Second, some nodes and edges do not exist or trigger any alarm in the training set. Third, since our approach aims to be as conservative as possible, some parameters are tuned just enough to eliminate false alarms in the training set. However, these conservative settings may still cause false alarms when applied to the testing set.

While Captain operates as an event-level detector, we still compare Captain with the SOTA entity-level PIDS Flash, Kairos, and NodLink(Kairos and NodLink give the result at graph-level, but they also support entity-level detection). We transformed the alarm events triggered by Morse and Captain into alarm entities using the following method: an entity is considered an alarm entity if it is involved in any alarm events.

It is noteworthy that only a subset of each DARPA dataset is utilized for training and testing in [20, 16, 15]. To simulate the real scenarios in SOC, we utilize the entire dataset for evaluation, dividing the training and testing sets as described in §VI-A2. We retrained the model and performed detection using their code. Additionally, we noticed Flash and ThreaTrace did not count the FPs within the two-hop distance from the labeled attack entities in the ground truth while counting the TPs within the two-hop distance from the detected entities [15]. To ensure a fair comparison without providing excessive leniency, we report the TP and FP results, as well as the one-hop FP result, in which we exclude the FP entities within a one-hop distance from the ground truth. We also did not count the attack entities in the 2-hop neighborhood of the detected entities as TP like [16, 15, 20] did. These strict experiment settings and data labeling methods can explain the difference between our results and the results in their papers, but we believe it is fair and necessary to assess the performance of PIDS in real-world scenarios.

We evaluated the detection accuracy on Cadets, Trace, and Theia from Engagement 3 since those datasets were commonly used by the baselines. Table V illustrates the comparison among Captain and the baselines. On all datasets, Captain demonstrates superior performance in identifying more TPs while maintaining fewer FPs. Flash can report a fair amount of TPs but produce an excessive amount of FPs. It leverages GNN to learn $k$-hop neighborhood structures. While this technique shows promising performance when there is a significant anomaly within $k$ hops, it could degrade when the training set is limited [16]. Kairos can filter out FPs with the Anomalous Time Window Queue [15]. Although it guaranteed decent detection accuracy at the time-window level; the entity-level accuracy, however, is not satisfactory. NodLink detects most attack processes. However, missing other relevant entities, such as files and network sockets, requires additional expert efforts to investigate the reported alarms. Morse and Captain achieve a lower false positive rate in node-level detection. This is attributed to the semantic-rich alarms that provide additional information for alarm filtering. For instance, a file corruption alarm involves two entities: the process and the corrupted (benign) files. As the corrupted files should not be reported as malicious entities, they can be easily filtered out. This explains why Morse triggers thousands of false alarms at the event level, but only hundreds of nodes are incorrectly alarmed. For embedding-based PIDS, there is no such semantics for alarm filtering. We include the filtered results in Table V as we believe it highlights an advantage of rule-based PIDS.

Captain successfully detects all attacks in the dataset. As an event-level detector, Captain does not report entities that are not directly related to the attack events. For example, if a malware is downloaded and executed, Captain reports the MalFileCreation alarm and the FileExec alarm, which reveals the malware process, file, and parent process. We do not immediately report the entry network node used by attackers to avoid alarm fatigue, which explains the FNs of Captain. The investigations on the relevant nodes can be conducted after the first response.

The detection latency analysis encompasses three dimensions: buffer time, preprocessing time, and detection time. We have already introduced buffer time in II-A. Preprocessing time refers to the duration taken to convert the raw audit logs into a data structure that the detection systems can process. It involves data parsing, data cleaning, noise reduction, preprocessing, feature extraction, and so on. Detection time refers to the interval between the completion of data processing and the moment the result is produced. It is noteworthy that since the detection granularity is different in different PIDS, the detection time might not reflect the actual latency of each system. For instance, a detector based on the whole graph may take longer to deliver detection results of each graph compared with a detector on the entity/event level. However, this does not necessarily mean the former is slower because a large graph could contain many entities and events. Therefore, we calculate the total time spent on detection across the entire dataset.

We evaluated the latency on Trace and Cadets from DARPA Engagement 3 since they are covered in the experiments of all baselines. Our evaluation focuses on the testing stage since training can be conducted offline. We modified their code to obtain unbiased results. For example, Flash divides the data processing into two stages in their code and uses files to store intermediate results. And Kairos employs the PostgreSQL database and stores the intermediate results in files. These are the steps not required in a real-time streaming pipeline. Consequently, we exclude the I/O time to focus solely on measuring the “pure” preprocessing and detection time.

The result is shown in Table VI. Unlike other PIDS that embeds the graphs and thus require a buffer time ranging from several minutes to hours, Captain processes audit logs in a streaming fashion, eliminating the need for any buffer time. Captain is also faster in preprocessing because the logs are not preprocessed for the machine-learning model. Moreover, the tags of Captain are much simpler than the state vectors used by Flash and Kairos. By avoiding the use of text embedding or GNNs to aggregate semantic and contextual graph information, Captain achieves detection speeds over 10 times faster than the baseline methods. Lastly, compared to the rule-based PIDS Morse, Captain only takes a few seconds longer to detect logs spanning 4 to 7 days, once again demonstrating the superiority of rule-based PIDS in terms of latency.

Another important metric reflecting the efficiency of a PIDS is the runtime overhead to the system. In this section, we evaluate the runtime overhead of Captain during detection and conduct a comparative study with other SOTA PIDS. We also analyze the memory consumption of Captain during the training stage.

Runtime Overhead in Detection: We use the Python resource module to evaluate the resource consumption of detectors on CPU mode. As admitted in [13], GPUs may not be available in most real-life threat detection scenarios. We calculate the total CPU time in user mode. The result is shown in Fig. 4(a). Please note that the total CPU time used in Fig. 4(a) could be more than the detection time in Table VI because of the multi-core CPU usage by deep learning models. Since Captain only relies on straightforward arithmetic operations on real-number tags, it requires significantly less CPU time than the embedding-based PIDS (around 2% of Flash and around 0.02% of Kairos), which need complex matrix and vector operations due to their use of neural networks. Moreover, Captain’s fine-grained detection rules only increase CPU time by a modest 5.6% compared to Morse. Given the significant improvement on Morse in detection accuracy (over 90% reduction in false alarms) and the substantial decrease in CPU time compared to other embedding-based PIDS, we believe the slightly higher CPU usage than Morse is acceptable in real-world detection scenarios.

We use the psutil library in Python to monitor the live memory usage during detection. The comparison of memory usage over time is shown in Fig 4(b). Captain finishes detection much faster than the embedding-based PIDS (as already shown in Table VI) and achieves the lowest memory usage throughout the entire detection process. The memory usage and detection time of Captain and Morse are similar, shown by the overlapping curves in Fig. 4(b). The slight difference in memory usage between Captain and Morse is due to the storage of fine-grained rules. In general, Captain retains the lightweight and fast characteristics of the rule-based PIDS.

Runtime Overhead in Training: The training of the detectors can be performed offline, where users can allocate ample time and computing resources, including GPUs. Therefore, we did not evaluate the runtime overhead and the time used for training. However, as Captain records and propagates the gradients for each tag among the big provenance graph, a reasonable concern is that saving those gradients would consume much memory when there is a dependency explosion problem. We evaluated the number of non-zero gradients saved by Captain during training. The results, shown in Fig. 8, illustrate that most nodes only need to store a small number of non-zero gradients during training. Due to the space limit, the detailed analysis can be found in §E-A.

Mimicry attacks on PIDS involve altering the provenance data and incorporating more benign features to “mimic benign behaviors”, thereby evading detection. In this section, we evaluated the robustness of Captain against mimicry attacks using the attack methodology in [35, 16], i.e. inserting benign structures into the attack graphs. Our mimicry attack contains two steps. First, we extract some events of benign system entities from the normal training data. Next, we create “fake” events by substituting the benign entities with the attack entity, simulating the attack entity performing activities similar to those of the benign entities. To verify the effectiveness of the mimicry attack, we use Flash as the baseline. For evaluation purposes, we use the Cadets from Engagement 3 because of its relatively small scale. The details about our mimicry experiment can be found in the Appendix E-B. The result is shown in Fig. 5.

In general, both Captain and Flash showed robustness against our mimicry attack attempt to some extent. The attack entity /tmp/test is still detected by both systems even after we add some normal activities. However, we see a significant drop in the anomalous score of Flash when we introduce a relatively small number of events (fewer than 10), which is also confirmed in their paper [16]. On the other hand, Captain remains unaffected against the mimicry attack, demonstrating superior robustness compared to the baseline.

The robustness of Captain results from three main reasons. First of all, the propagation of Captain is guided by heuristic-based rules, while the propagation and aggregation of Flash and other embedding-based PIDS are based on the neural network. Second, Captain is an event-level detector with a finer granularity than entity-level detectors. As mentioned before, Captain gives the detection result for each streaming event without any buffer time, which means any mimicry insertion after the real attack activities is useless. For those PIDS with the buffer time, any events within the same time window and related to the same entity can affect the final detection result. Third, Captain is not an anomaly-based detector. In other words, the detection is performed based on the conservative, heuristic rules, while the normal data is only used to reduce the false alarms. Therefore, adding normal features can not compromise the detection capability as it is guaranteed by the conservative detection rules.

Training set poisoning refers to the malicious manipulation of the training set. In the context of cyber security and intrusion detection, it usually involves polluting the benign training set with adversarial events/entities. When the detector is trained on a “benign” dataset that has been polluted, it learns patterns or features introduced by attackers. These patterns and features, once learned, could assist attackers in evading detection in future instances.

We add the regularizer term to the loss function to control the sensitivity of Captain to the training set. This enhances Captain’s robustness against dataset poisoning attacks. In this section, we present a real case of Pine Backdoor & Phishing Email attack from Engagement 3 Trace. In this attack, a vulnerable pine connected to the email server 128.55.12.73, downloaded and executed an email attachment. However, IP 128.55.12.73 had multiple activities and caused some “false” alarms in the training set. Please note that this violates our assumption stated in § II-C that the training set must not be compromised by attack entities or events. Nevertheless, this makes it a perfect example to test the robustness against data poisoning attacks.

We analyze the critical parameters for detecting this attack: the initial integrity tag ($a$) of IP node 128.55.12.73, to illustrate the effect of the regularization term. Without the regularization term in the loss function, 128.55.12.73 will be assigned a high integrity score (near 1.0) due to its activities in the training set, which lets us miss the phishing email attack starting from it. However, with the help of the regularizer terms ($\alpha=10$), Captain is more cautious during training. Like the “grey node” motivating example in §III, Captain assigns the integrity score as the mediocre 0.488, which removes many false alarms caused by this IP while successfully capturing the phishing email attack during testing (more details can be found in §VI-F). The propagation rate of event (pine, read, 128.55.12.73) is also more conservative against the polluted training set due to the regularizer term.

We conduct an ablation study on each adaptive parameter. As each parameter can be learned independently, we have seven separate experiments, each optimizing a subset of the parameters, i.e. $\{A\}$, $\{G\}$, $\{T\}$, $\{A,G\}$, $\{A,T\}$, $\{G,T\}$, and $\{A,T,G\}$.

According to Fig.6, adjusting the threshold ($T$) alone may not have much effect on the detection results because without changing the initial tags and the propagation rates, the updated tags still tend to be 0 or 1. Fig.6 also shows that tuning all these three parameters can accelerate the training process. Although $\{A,T\}$ and $\{A,T,G\}$ achieve similar good detection results on the testing set, $\{A,T,G\}$ converges using fewer epochs during training. It should be noted that tuning $A$ plays the most major role in mitigating false alarms among these three adaptive parameters. In certain datasets (S-3 and C-3), adjusting $A$ alone sometimes results in fewer false alarms than tuning parameters $A$, $G$, and $T$ together because, without the benefit of learning propagation rates and thresholds, we have to assign higher integrity scores to more nodes to counteract false alarms during training. However, reducing false alarms more crudely carries the risk of being excessively lenient.

We conducted an experiment on different learning rates on the DARPA Engagement 3 Cadets. The detailed results can be found in § E-C. As illustrated in Fig. 9, a higher learning rate usually leads to a sharper loss decline during the initial training epochs. However, for ongoing training, a lower learning rate may yield a finer-tuned search process and lower loss, necessitating a balance between training speed and training quality.

The most significant improvement of Captain is its fine-grained detection capability, enabled by the adaptive parameters. As mentioned in § III, unlike the discrete, human-assigned trustworthiness levels and universal rules used in [22, 21, 23], the fine-tuned numerical parameters of each entity and event in the graph can better help distinguish nuances between benign and malicious patterns.

Fig. 7(a) demonstrates how Captain outperforms the baselines in distinguishing between two similar behaviors using fine-tuned parameters. In both cases, process pine reads from 128.55.12.73 (possibly the email server) and creates, writes, and changes the permissions of a new file. It is difficult for previous rule-based PIDS [21, 22] to assign a tag for 128.55.12.73. Take the SOTA Morse as an example - if Morse initialize the node 128.55.12.73 as a benign node with the data integrity score as 1.0, it would miss at least the first stage of the attack. On the other hand, if Morse assigns the initial tag as “Untrusted” (this is what they did in order to capture all attacks), it would trigger multiple false “Malicious File Creation” and “Change Permissions” alarms on the benign graph. Instead, Captain assigns a moderate initial tag $a_{n}$, 0.488, to this IP and slightly tunes down the threshold $thr$ for the event (pine, create, /home/admin/pinerc174500) from 0.5 to 0.478. As $0.478<0.488$, the creation of the file will not be alarmed, but in real attacks, the $thr$ for the event (pine, create, /tmp/tcexec) is still 0.5, meaning creating a malicious file would still trigger alarms ($0.5>0.488$).

Captain’s fine-grained rules also help address the dependency explosion issue caused by libresolv.so.2. Unlike Morse, which sets universal decay/attenuation factors for all processes, Captain learns distinct propagation rates for different events. As shown in Fig. 7(b), Captain assigns lower propagation rates to events that frequently cause dependency explosions (e.g., being read by sshd or firefox) while maintaining high propagation rates for other events. Even for the same process, firefox, Captain learns to suppress the dependency propagation when reading this file ($g_{e}=0.63$) and remains sensitive to maliciousness propagation when loading this file ($g_{e}=0.99$). Consequently, Captain avoids being overly lenient for potential attack events when addressing the dependency explosion issue.

The fine-grained parameters of Captain also outperform many embedding-based PIDS in detecting mimicry attacks. For instance, in Fig. 7(a), the attacker can modify the file name of /tmp/tcexec with the knowledge that pine might interact with a file like /home/admin/pinerc+numbers. Since generalization techniques like Word2Vec and Sentence2Vec are widely used in embedding-based PIDS, they can manipulate the file name by appending random digits to pinerc, making it similar to pinerc174500 after embedding. However, to evade Captain, the attacker must know the exact file name of pinerc174500. In addition, deep learning usually relies on the amount of training data. But there are only four events related to file /home/admin/pinerc174500 during the 7-day training set, making it challenging for the neural networks to learn such patterns.

Some previous rule-based PIDS [7] can customize their detection system using benign training data. However, the adjustments are based on the training set rather than the detection results. For example, NoDoze assigns a distinct anomaly score to each event based on its frequency in the training set. However, due to training set poisoning and living-off-the-land attack techniques, the frequent events in the training set cannot be fully trusted. For instance, the events related to dbus-daemon are very common in the training set of Trace in Engagement 3. Typical events such as dbus-daemon writes or reads /var/run/dbus/system_bus_socket occur over 100,000 times during seven days. Such events and the dbus-daemon entity would be assigned with a low anomaly score in frequency-based systems like NoDoze. However, the low anomaly scores allow the attackers to deceive the detectors using living-off-the-land techniques related to the D-Bus process [56]. In contrast, Captain does not change the parameters related to dbus-daemon because, despite the large number of events associated with dbus-daemon in the training set, no false alarms are triggered by it. Captain adjusts the parameters related to an event based on the false alarms caused by it rather than the frequency of it.