TabSec: A Collaborative Framework for Novel Insider Threat Detection

Internet threats pose the foremost risk to the security of enterprise assets [1, 2, 3], IoT applications [4, 5, 6, 7], security or privacy-sensitive machine learning systems [8, 9, 10] or some edge-cloud cooperation applications [11, 12, 13, 14, 15]. These threats are characterized by diverse and complex attack methods, which, once occurred, can lead to severe customer privacy breaches and significant asset losses [16, 17]. Therefore, detecting internet security threats is of utmost importance. To ensure asset security, enterprises traditionally employ IDS and UEBA technologies to detect both external and insider threats. Due to their legitimate access, insiders can be challenging to detect when they launch attacks on the system. Traditional UEBA techniques attempt to classify normal and abnormal users by establishing user group profiles [1, 7]. However, these methods struggle to distinguish users who have been long-term masqueraders within the system or those who gain access through User to Root (U2R) and Remote to Local (R2L) attacks [18], resulting in low prediction accuracy.

With the vast attack surface of the internet, enterprises face the challenge of dual threats from both internal and external sources[19]. Attackers may exploit U2R and R2L attacks to gain system access, allowing them to escalate their privileges to that of internal users. The growing sophistication of these attacks has blurred the boundaries between external and insider threats. This shift in attack trends reveals the limitations of traditional standalone detection techniques, which often have low detection rates against unknown attacks. Furthermore, the scarcity of robust and representative attack data hinders the ability of existing machine learning models to effectively learn and generalize attack behaviors. This limitation exacerbates the shortcomings of detection techniques, resulting in suboptimal performance and reduced efficacy in identifying and mitigating complex threats. This inability to effectively differentiate between normal and malicious actions poses a serious security risk, as it allows sophisticated attackers to bypass detection, escalate privileges, and operate within the system as trusted users. The failure of these traditional methods to detect such nuanced and covert threats underscores a significant vulnerability in current cybersecurity defenses, highlighting the urgent need for more advanced and adaptive detection strategies.

In summary, current threat detection technologies face the following critical issues:

1. 1.

   Traditional threat detection techniques often struggle to effectively distinguish between legitimate activities and sophisticated evasive behaviors, particularly when dealing with external attackers who leave backdoors, such as those executing U2R and R2L attacks.

2. 2.

   Traditional threat detection technologies struggle with the critical issue of effectively learning from rare class data[20, 21, 22], leading to significantly low prediction accuracy for these uncommon yet highly consequential attacks. This inadequacy poses a severe risk, as rare class attacks — often representing the most sophisticated and damaging threats — go undetected or are misclassified.

3. 3.

   Traditional threat detection technologies typically operate independently and fail to account for the transition from external to insider threats, which has become increasingly common in sophisticated attack scenarios. This limitation undermines their effectiveness in modern security environments, as attackers often exploit initial external breaches to gain insider access. As a result, traditional systems are frequently unable to provide comprehensive coverage, leaving critical blind spots in threat detection and response.

The main contribution of this paper can be summarized as:

1. 1.

   We integrated IDS and UEBA technologies to detect masquerader attacks evolving from U2L and R2L, and validated this approach using the NSL-UEBA and KDD-UEBA datasets. The results demonstrate that this combined strategy significantly improves detection accuracy and effectively identifies advanced masquerader attacks, while addressing the detection blind spots present in traditional insider threat solutions (Section VI).

2. 2.

   We utilize the TabNet classifier for threat detection. TabNet’s superior Attentive Transformer can generate feature selection masks to choose the most relevant features at each decision step, thereby enhancing the detection stability (Section IV).

3. 3.

   The integrated system effectively identifies complex, multi-stage attacks, particularly those involving the progression from external to insider threats (such as U2R and R2L attacks). By leveraging collaborative analysis and cross-domain threat correlation, the integrated approach addresses the blind spots present in traditional detection methods, significantly enhancing the system’s responsiveness to emerging and unknown threats. This integration strategy optimizes the monitoring and analysis of the entire attack chain (Section IV).

Previous research has extensively studied ontologies and detection methods for insider threats. Homoliak et al. [1] conducted a survey on the classification, modeling, and mitigation of insider threats, categorizing them into two groups as malicious and unintentional. Existing work employs data logging and behavioral analysis to detect insider threats [2, 3]. Rashid et al. [23] utilized hidden Markov models to learn the behavioral characteristics of normal users within the system, thereby enhancing the ability to identify anomalous behavior. This approach is effective in analyzing long-term user behavior and detecting threats. Zhang et al. [24] proposed a behavior log detection method based on ensemble learning and self-supervised learning. This method employs a TF-IDF-based entity embedding technique and performs a self-supervised classification task on detecting inputs, distinguishing between valid and malicious behaviors of specific users. Other studies have also applied various machine learning methods, such as one-class SVM [25] and clustering techniques [26].

However, the exceptional performance of deep learning in tasks such as representation learning, sequence modeling, and heterogeneous data integration offers significant advantages for insider threat modeling and detection [27]. Various neural networks, including RNN and CNN, have been employed to extract behavioral features from user activity sequences or malicious sessions [28, 29, 30]. To address the issue of critical information loss due to the lack of emphasis on the connectivity between entities, Wei et al. [31] first proposed a Graph Neural Network model to construct the relationships between user behaviors and entities. They employed a weighting function to quantify the structural information between users, thereby matching behavior logs with known attack patterns. He et al. [32] utilized LSTM to extract user behavior sequences and differentiate between various user behaviors to identify anomalies. Their work leveraged an attention mechanism based on users’ historical behaviors to learn the distinctions in user behavior.

Despite the aforementioned works providing feasible detection methods, most of these models primarily detect unintentional and malicious attacks based on behavioral characteristics, but cannot analyze the process by which external attackers gain legitimate access to the system through U2R or R2L attacks. Our model integrates IDS and UEBA, in the meanwhile having the ability to track and identify masqueraders and lurkers inside the system.

TabITD consists of the following three main components, as illustrated in Fig. 1. The network architecture related to TabNet is shown in Fig. 2.

In TabNet, feature selection is facilitated through a learnable mask matrix $\mathbf{M}[i]\in\mathbb{R}^{B\times D}$, enabling the adaptive selection of the most salient features. This approach ensures that the model’s learning capacity is focused on the most relevant features, thereby enhancing parameter efficiency. The mask operation is multiplicative, represented as $\mathbf{M}[i]\cdot\mathbf{f}$, and an attentive transformer is employed to generate the masks from the processed features of the previous step, $\mathbf{a}[i-1]$. As shown in (1), The mask $\mathbf{M}[i]$ is determined using the sparsemax function.

Sparsemax normalization encourages sparsity by mapping the Euclidean projection onto the probabilistic simplex, thus ensuring the model concentrates on the most critical features. The mask values are normalized such that, as illustrated in (2), the sum over the elements equals one.

To regulate the sparsity of the selected features, a sparsity regularization term $L_{\text{sparse}}$ is incorporated into the overall loss function, as defined in (3). This term, derived from entropy, is formulated as:

where $\epsilon$ is a small value for numerical stability. This regularization term encourages the selection of fewer features, providing a favorable inductive bias, particularly beneficial for datasets with many redundant features.

The filtered features are processed using a feature transformer, which is split to handle decision step output and subsequent step information. Formally, for each decision step $i$, we have (4).

where $\mathbf{d}[i]\in\mathbb{R}^{B\times N_{d}}$ and $\mathbf{a}[i]\in\mathbb{R}^{B\times N_{a}}$.

To ensure parameter efficiency and robust learning with high capacity, the feature transformer should consist of layers that are shared across all decision steps and layers that are decision step-specific. This is implemented as a concatenation of two shared layers and two step-dependent layers.

Each fully connected (FC) layer is followed by batch normalization (BN) and a gated linear unit (GLU) nonlinearity, as shown in (5).

This structure is eventually connected to a normalized residual connection. Normalization with $\sqrt{0.5}$ stabilizes learning by maintaining consistent variance throughout the network.

To expedite training, large batch sizes with BN are employed. For input features, we use ghost BN, where the virtual batch size $B_{V}$ and momentum $m_{B}$ form, as expressed in (6), BN normalizes the input features by subtracting the mean and dividing by the standard deviation.

For non-input features, we employ standard BN. Finally, inspired by decision-tree-like aggregation, the decision output $d_{\text{out}}$ is computed by (7).

A linear mapping $\mathbf{W}_{\text{final}}\cdot d_{\text{out}}$ is used for the output construction.

TabNet’s feature selection masks illuminate the importance of features at each decision step. When $\mathbf{M}_{b,j}[i]=0$, the $j$-th feature of the $b$-th sample contributes nothing to the decision. If $f_{i}$ were a linear function, $\mathbf{M}_{b,j}[i]$ would directly reflect the importance of feature $\mathbf{f}_{b,j}$. Despite each decision step employing non-linear processing, their outputs are aggregated linearly. To measure the combined importance of features across steps, we introduce a coefficient $\eta_{b}[i]$ that weighs the significance of each decision step for the $b$-th sample. Specifically, we have (8).

This coefficient denotes the aggregate decision contribution at the $i$-th decision step. Intuitively, if $\mathbf{d}_{b,c}[i]<0$, all features at step $i$ have zero contribution to the final decision. As $\eta_{b}[i]$ increases, it indicates a greater role in the overall linear combination. Scaling the decision mask at each step by $\eta_{b}[i]$, we propose the aggregate feature importance mask $\mathbf{M}_{\text{agg},b,j}$, which is defined in (9).

This aggregate mask $\mathbf{M}_{\text{agg},b,j}$ quantifies the overall importance of each feature by integrating its significance across all decision steps.

There is a decoder architecture aimed at reconstructing tabular features from TabNet encoded representations. The decoder comprises feature transformers, followed by fully connected (FC) layers at each decision step. These outputs are aggregated to generate the reconstructed features. The task involves predicting missing feature columns from others, using a binary mask $\mathbf{S}\in\{0,1\}^{B\times D}$.

The TabNet encoder inputs $(1-\mathbf{S})\cdot\hat{\mathbf{f}}$ and the decoder outputs the reconstructed features, $\mathbf{S}\cdot\hat{\mathbf{f}}$. We initialize $\mathbf{P}[0]=(1-\mathbf{S})$ in the encoder to ensure the model focuses solely on known features, while the decoder’s final FC layer is multiplied by $\mathbf{S}$ to output the unknown features.

The reconstruction loss during the self-supervised phase, as defined in (10), is:

This normalization uses the population standard deviation of the ground truth, beneficial due to potential range differences in features. The mask $\mathbf{S}_{b,j}$ is sampled independently from a Bernoulli distribution with parameter $p_{s}$ at each iteration.

In this section, we comprehensively describe the experimental environment utilized in our study. We also identify and discuss the current leading detection models chosen for comparison, establishing the basis for the subsequent evaluation of our proposed model’s performance.

With the increasing frequency of insider threat incidents, their detection has become a critical issue. This paper proposes a collaborative detection-based insider threat detection scheme, aiming to address the shortcomings of traditional insider threat detection techniques. This scheme utilizes TabNet for threat behavior detection, and performance test results demonstrate that our approach can identify various malicious activities. Currently, the main limitation of this study lies in the adaptive hyperparameter tuning, which is crucial for selecting optimal model parameters and thus affects the model’s detection performance. The hyperparameters of TabNet are often interdependent, with complex relationships between certain hyperparameters. This interdependence makes simple grid search or random search methods insufficient for effective hyperparameter tuning, necessitating the use of more advanced optimization algorithms. Therefore, improving hyperparameter tuning remains a challenge, and further research is needed to optimize and refine this scheme. In the future, we plan to employ Hyperband optimization and adopt a resource allocation strategy based on bandit algorithms to dynamically allocate computational resources for evaluating different hyperparameter combinations, testing this approach in real-world wireless testbed [33] scenarios.