Enhancing Robustness Against Adversarial Examples in Network Intrusion Detection Systems

Anomaly-based NIDS can be built in many different ways. Figure 1 illustrates a typical anomaly-based NIDS. These NIDS have two major components: a feature extractor and an anomaly detector. In a nutshell, the feature extractor receives a stream of packets and extract features from them to feed them to the anomaly detector. Then, the anomaly detector outputs a score for each input it receives that gets compared against a threshold. If the score is less than the threshold the corresponding input will be predicted as benign otherwise it’ll be predicted as malicious. The feature extractor can be built in two different ways. It can output a feature vector for each packet it receives. This feature vector is not based solely on the current packet; it can also consider the history of packets it has seen earlier. In this case, the features can be as simple as raw values extracted from packet headers or complex features created manually. We call the NIDS leverage such feature extractors packet-based NIDS. On the other hand, there are NIDS where their feature extractor builds a single feature vector with high-level features for a whole flow. We call these NIDS flow-based NIDS. The anomaly detector component can also be created in several different ways. Kitsune [11] which is a packet-based NIDS proposed to leverage an ensemble of autoencoders which are neural-networks trained to reconstruct a given input based on the full observation of that input. Then during the execution time, it calculates the final score for each input based on the reconstruction errors of the autoencoders. Zong et al. proposed DAGMM [12] which trains two neural-network in an end-to-end fashion to calculates the energy of each flow in the Gaussian Mixture Model (GMM) framework and uses that energy to detect anomalies. Zenati et al. introduced a BiGAN-based approach [13] that uses the reconstruction error of the generator and the output of the discriminator in the GAN (Generative Adversarial Network) framework to detect anomalies. We refer the readers to those papers for more details about each approach. We first show how one can build a packet-based NIDS with the RePO technique and compare it with Kitsune. Then we demonstrate that an anomaly detector built with RePO can also be utilized in the flow-based context and compare it with DAGMM and the BiGAN-based NIDS which were evaluated in such a setting. We chose these methods as our baselines as they were among the most highly cited NIDS which were published recently.

It has been shown that deep neural networks used for tasks such as image classification [18, 19], speech-to-text systems [20], face recognition [21], autonomous driving [22], malware detection [23], etc. are vulnerable to adversarial example attacks (i.e. evasion attacks). Adversarial example attacks are carried out during the inference phase. For this attack, the attacker doesn’t change any of the model’s parameters but modifies its own inputs in a way to make the model predict the inputs as the desired class.

Recently, it has been shown that it is possible to craft adversarial examples against NIDS, as well [15, 16]. Aiken et al. [16] showed one can craft adversarial examples against an NIDS deployed on an SDN for a DDoS attack by increasing packets payload size, decreasing packet rate and forging traffic with the reverse source and destination to that of the attack packets. Hashemi et al. [15] have designed a more general method to craft adversarial examples for a wide range of attacks. They’ve shown that by applying a combination of legitimate transformations such as splitting a packet into multiple packets, changing the delay between packets, etc. one can fool the NIDS while not breaking the underlying network protocols. In this paper, in order to evaluate our NIDS in an adversarial setting, we follow their threat model and use the algorithms and transformations introduced by them to craft adversarial examples against our NIDS. More specifically, we consider that the attacker has a copy of the NIDS deployed on the victim network and knows all of its parameters. Also, we assume the NIDS deployed on the victim network receives a copy of all the packets that travel through the network entrances. Note, This can easily be achieved in an SDN with the help of the controller to make the switches forward all the network traffic to the NIDS node.

As we saw in Kitsune, the reconstruction error which is calculated based on the difference between the output and input of an autoencoder can be used as a score function to detect anomalies in an unsupervised manner. The assumption is that the model learns to reconstruct the inputs from the distribution it is trained on (e.g. benign traffic) better than inputs that come from another distribution (e.g. malicious traffic). We argue that this assumption is not necessarily true. The problem with using autoencoders and their reconstruction error in this way is that a neural network with a large enough capacity would have the capability to over-generalize and learn to rebuild the anomalous inputs, as good as the benign inputs. In other words, the score of malicious inputs will be very similar to the benign inputs making it hard to distinguish between them resulting in poor detection rate.

Therefore, in order to solve the issue of over-generalization mentioned above, we use denoising autoencoders to force the model to solve a harder problem. We train the model in a way to reconstruct a given input based on observing some part of it. This way, the model has to not only reconstruct the visible parts of the input but also to generate the hidden parts of it. As a result, the reconstruction errors of the malicious inputs become larger than the benign inputs as the model can’t reconstruct the hidden parts of the malicious inputs well enough which leads to better distinguishment between malicious and benign traffic and a higher detection rate. Also, note that as malicious inputs get further away from the decision boundary of the model (i.e. the threshold) it becomes harder to craft adversarial examples for them. Therefore, when we train the model in a way to make the gap between the malicious and benign scores larger it also becomes more robust against the adversarial example attack.

In our approach, since $r$ is a random matrix, by having different masks the model outputs different scores. In such a setting even if an input sample is normal, the random mask might block the most important parts of the input. Therefore, the model may not be able to reconstruct the input from what it observes. In this case, the reconstruction error would also be high. In order to solve this issue, during inference time, we replicate each sample 100 times and feed them in parallel to the model such that each of them is masked with a different mask $r_{i}$. We then calculate the score of all of them and group them equally into 5 groups, and from each group, we keep the minimum score. Finally, we calculate a new score by adding these 5 minimum scores together and we use this new score for deciding whether or not an input is anomalous. When we use 100 different masks, it becomes more likely to have ”better” masks. By choosing the ones which have smaller reconstruction errors, we essentially ignore the cases that have high error because of a ”bad” mask (i.e. a mask which blocks the essential features of a sample which are needed to correctly classify it). Note that, here, we don’t train an ensemble of 100 different models that can take a very long time to be trained. We only train one single model that receives multiple parallel copies of each input masked with different masks during inference time. We call this approach RePO+, and we empirically found that it has a higher detection rate.

Also, note that crafting adversarial examples against RePO+ becomes harder as for a given input the adversary should plan to modify a larger set of features to fool the model compared to when there is only one mask. This is because if only a few features get changed, they can get masked by one of the masks with a high chance which makes those changes ineffective. Moreover, since the adversary can’t directly query the NIDS deployed on the victim’s network (otherwise would be detected) and has to craft the adversarial example by the help of their local copy of the NIDS, Even if the adversary can fool the local copy when the sample goes through the original NIDS because of a different random mask, it would generate a different score which might be higher than the threshold and therefore will be detected. This stochastic nature of RePO+ also makes it more robust against adversarial examples. In Section IV-D, we empirically show the extent to which the model is more robust due to this property.

Here, we discuss how to build a packet-based NIDS using the RePO technique. ¹¹1Our code is available at: https://github.com/s-mohammad-hashemi/repo To build a packet-based NIDS with RePO, we first group the received packets by their sender and receiver IPs (i.e. the packets from A to B would be in the same group as packets from B to A). Then for each packet, we build a feature vector with the following features:

• •

  Inter-arrival time: The time between this packet and the previous packet in the group.

• •

  Features extracted from Ethernet header: the length of the frame.

• •

  Features extracted from IP header: IP header length, IP length, IP flags (df, mf, rb), TTL.

• •

  Features extracted from TCP header: source port, destination port, sequence number, acknowledgment number, TCP flags (res, ack, cwr, ecn, fin, ns, push, reset, syn, urg), TCP window size, urgent pointer. These features would be zero if the current packet is not a TCP packet.

• •

  Features extracted from UDP header: UDP length, source port, destination port. These features would be zero if the current packet is not a UDP packet.

• •

  Features extracted from ICMP packet: ICMP type. This feature would be zero if the packet is not ICMP packet.

• •

  Direction: This feature is a binary feature that shows whether the packet is from the IP which started the communication or from the other end.

In total, the feature vector corresponding to each packet contains 29 features. In order to make a decision for a given packet, in addition to the features of that packet, we also send the features extracted from the previous 19 packets from that group to our model. Therefore the decision is made based on observing 20 consecutive packets, and we feed 580 features to the model in each case. If there are not enough packets before a given packet, we pad it with feature vectors which all of their elements are zero. The model architecture we used to train RePO as a packet-based NIDS is a light-weight neural-network with only one hidden layer with 2048 neurons. During the training phase, we first normalized each feature separately by a min-max scaling approach. We set the batch size to be 512 and we trained the model for 30000 different batches which were selected randomly with learning rate 0.001, 0.0001 and 0.00001 each for 10000 iterations.

Note, it is not required to utilize RePO only in the packet-based context. RePO can also be used to build a flow-based NIDS when combining it with a feature extractor that extracts features at the flow level. The model architecture we used for detecting anomalies at the flow level is a fully-connected network with 6 hidden layers each with size 256 and ReLU non-linearity in addition to 2 dropout layers after the third and fifth hidden layers of the network. During the training phase, we first normalized each feature by a min-max scaling approach. We used a batch size of 256 and trained the model for 5 epochs with a learning rate of 0.001.

To evaluate our approach, we used a highly cited dataset containing network traces of twelve network attacks from the Canadian Institute of Cybersecurity (CIC) ²²2The dataset can be downloaded at: https://www.unb.ca/cic/datasets/ids-2017.html  [24]. These network attacks were carried out over a 5-day work week in a controlled environment and are as follows: FTP-Patator, SSH-Patator, DoS slowloris, DoS slowhttptest, DoS Hulk, DoS GoldenEye, Heartbleed, Web Attacks, Infiltration, Botnet, PortScan and DDoS. These network attacks make 10.33% of our test set at the packet level and 24.22% at the flow level. They are also distributed unevenly (e.g. Dos Hulk alone makes up 48% of malicious packets). Therefore, we demonstrate the detection rate of NIDS for each attack separately. The whole dataset contains more than 56 million packets (2.8 million flows). We extracted features directly from the packets as described in the previous section to evaluate RePO in the packet-based context and used the flow-level features provided in this dataset to evaluate our NIDS in the flow-based context. Each flow was labeled as either benign or with the specific attack name and we labeled each packet based on the flow labels ourselves with the same procedure as Hashemi et. al in [15]. We trained the packet and flow-based NIDS on the Monday traffic, which solely contains over 11.6 million benign packets (529,481 flows). The NIDS were then tested on the network traffic generated during Tuesday-Friday, which contains both benign and network attack traffic. We excluded web attacks from our evaluations because in both packet-based and flow-based cases the features were collected from packet headers and in order to detect web attacks packet payloads should also be inspected.

In order to evaluate our approach in an adversarial setting we use the crafting procedures introduced by Hashemi et al. in [15]. Also, we follow the experiment setting introduced in [15] and set the threshold of our NIDS in a way to keep FPR at 0.1. This is because our baselines didn’t perform well at the low FPR (0.01) even in a normal setting and there wasn’t too much malicious traffic that is detected by those NIDS at the low FPR to craft adversarial example for them. Therefore, for evaluation in an adversarial setting, we used a higher FPR (0.1) in which the robustness of different NIDS can be compared better against each other.

Our NIDS is also time-efficient. In our experiments, by using an Nvidia Titan V GPU it only took 25 seconds to train RePO+ as a flow-based NIDS. Moreover, During inference time, RePO+ could process 17,550 flows per second. That is to say, the whole test set including 2.3 million flows collected over a course of 4 days could be processed in only 131 seconds. Also, as a packet-based NIDS, we could train RePO+ in 7.5 minutes and it could process 3,285 packets per second during test time. This means that the whole test set which contains more than 44 million packets could be processed in 226 minutes. Note that, this processing rate is for when we classify every single packet which is not necessary but in section IV-C1 we did it this way to have a fair comparison against Kitsune which outputs a score for every single packet. That is to say in our evaluation we considered a window with size 20, grouped 20 consecutive packets, fed them into the model and moved the window one step forward. We can simply move this window 20 steps forward to reduce the redundancy while still considering all of the packets. In this case, RePO+ can process 65,700 packets per second making it capable of processing the whole test set in only 12 minutes. It’s also worth saying that in this setting we don’t sacrifice detection rate at all. The average detection rate when FPR is 0.01 is still 34.83% which is almost the same as what reported in IV-C1. In addition, since RePO+ can become fully parallelized we can use more GPUs to process a higher number of flows/packets per second if needed. In this case, the number of flows/packets processed per second will be multiplied by the number of GPUs used.