Detecting Masquerade Attacks in Controller Area Networks Using Graph Machine Learning

CAN is a robust communication protocol that has become a standard in automotive and industrial applications for facilitating the exchange of messages between electronic control units (ECUs). Initially introduced by Robert Bosch GmbH, the protocol’s latest version (2.0) was released in 1991, marking a significant advancement in automotive communication technologies [25]. CAN was further standardized internationally in 2011 by ISO 11898-1:2011, which specified the requirements for high-speed communication and data exchange among various vehicle subsystems. CAN operates on a multi-master serial bus standard for ECUs, which allows devices to communicate with one another within a vehicle without a central computer. This protocol is designed to operate at high speeds, up to 1 Mbps, depending on the distance and configuration, with the ability to support distributed real-time communication efficiently [16].

CAN operates as a broadcast protocol, enabling all ECUs connected to the bus to receive signals or messages transmitted over the bus. These signals, called frames, contain messages instructing the vehicle or system on executing operations. CAN encompasses the first two layers of the open systems interconnection (OSI) model, namely the physical and data-link layers, as shown in Fig. 1. Within CAN, ECUs such as the transmission control module, engine control unit, and the antilock braking system broadcast data frames that provide information about the vehicle’s current state.

A CAN frame comprises several fields, each defined below along with their possible values:

• •

  Start of frame – This bit indicates the beginning of a CAN frame transmission — 1 bit, dominant (0). A dominant value ensures that all nodes on the network recognize the start of a new frame.

• •

  Identifier (ID) – This unique identifier represents the message priority, where lower values indicate higher priority — 11 bits for standard ID, 29 bits for extended ID. IDS are crucial for arbitration when multiple ECUs transmit simultaneously.

• •

  Remote transmission request (RTR) – Indicates whether the frame is for data transmission (dominant) or for requesting data (recessive) — 1 bit. Dominant (0) for data frames, recessive (1) for remote request frames. Dominant indicates an actual data frame, while recessive indicates a request.

• •

  Identifier extension bit (IDE) – Specifies whether the identifier is standard or extended — 1 bit. Dominant (0) for standard ID, recessive (1) for extended ID. This bit differentiates between the two ID formats.

• •

  Reserved – Reserved for future use and must be dominant to ensure compatibility — 1 bit. Must be dominant (0) but accepted as either dominant or recessive. Ensures consistency and compatibility across different CAN standards.

• •

  Data length code (DLC) – Indicates the length of the payload, specifying how many bytes of data are in the frame — 4 bits, representing 0–8 bytes. This is essential for the receiving ECU to know how much data to expect.

• •

  Payload – The actual data carried by the frame, encapsulated in bits — 0–8 bytes. Contains the information being transmitted, such as sensor readings or control commands.

• •

  Cyclic redundancy check (CRC) – Used to ensure the integrity of the transmitted data by detecting errors — 15 bits. Helps in error detection to ensure reliable communication.

• •

  Acknowledge (ACK) – Indicates whether the frame was acknowledged by any node on the network — 1 bit, recessive (1) when the frame is sent. A dominant value (0) signifies that at least one node successfully received the frame.

• •

  End of frame (EOF) – Denotes the end of a CAN frame — 7 bits, recessive (1). Marks the completion of the frame transmission.

• •

  Inter-frame spacing (IFS) – The space between the end of one frame and the start of the next — 3 bits, must be recessive (1). Ensures proper spacing between frames to avoid collisions.

In our research, we use only the ID and data fields. The ID serves as the message header that identifies the frame and is used for arbitration, the process of prioritizing frames when multiple ECUs transmit simultaneously, i.e., the lower the ID, the higher the priority. The data field holds the actual message contents of up to 8 bytes, where each unique information carried in the message is termed a signal. CAN frames with the same ID encode the same set of signals in the same format and are typically transmitted at a fixed frequency to relay updated signal values.

CAN protocol’s design ensures reliable communication in environments that are prone to electrical noise and disruptions, making it ideal for automotive applications where safety and efficiency are paramount [26]. Furthermore, the signal-level representation of CAN data is facilitated using a database for CAN (DBC) file, which translates binary payloads into real-valued signals, providing a time series representation of the network’s activity. This feature is crucial for the development of ML-based intrusion detection systems (IDS) for masquerade attacks as it captures a structured representation of CAN data, allowing to characterize the regular relationships between physical signals in a system [25, 27].

The security of CAN can be analyzed using the confidentiality, integrity, and availability (CIA) triad, which are fundamental principles of cybersecurity.

A graph $G$ is defined as an ordered pair $G=(V,E)$, where $V$ represents a set of vertices (or nodes), and $E\subseteq(V\times V)$ represents a set of edges (or links) [32, 33]. Vertices act as entities and edges capture the relationships between nodes. For example, let us consider a toy graph $G$ with a vertex set denoted by $V(G)=\{v_{1},v_{2},v_{3},v_{4}\}$ and edge set by $E(G)=\{e_{1},e_{2},e_{3},e_{4},e_{5},e_{6}\}$. Fig. 2 depicts this graph for detailed visualization and to support the subsequent descriptions and definitions.

The MSG is a crucial tool for analyzing the behavior of the CAN network [21]. It captures the sequence pattern of messages, reflecting their normal sequence of messages in CAN. Nodes in this graph correspond to unique CAN identifiers (IDs), denoting different ECUs or functions within the vehicle network. Edges map the sequence of messages between these IDS illustrating how information flows within the system. The MSG models regular patterns of communication between ECUs within specified sliding windows, allowing for detailed observation of both normal and potentially malicious activities. Typical sequence patterns found during various vehicle operations, such as parking or acceleration, can be captured by the MSG. These patterns form the basis for detecting deviations that may indicate cybersecurity threats like fabrication and suspension attacks.

Constructing an MSG involves parsing the CAN data to determine the sequence of CAN IDS and quantifying the frequency of these sequences. This process transforms the raw, time-stamped messages into a structured graph that highlights the dynamics of the network’s communication. For instance, typical sequences observed in vehicle operations—such as those related to increasing speed—involve a predictable order of CAN messages related to fuel delivery, RPM, and speed. Fig. 4 in Section IV provides a visualization of these sequences within the MSG. By focusing on the sequence and frequency of message exchanges, the MSG provides a powerful method for identifying disruptions in normal communication patterns indicative of potential attacks. This graph-based approach enhances the ability to scrutinize the integrated behavior of ECUs and ensure the integrity of vehicular communications.

Graph embeddings map nodes, edges, or entire graphs into a lower-dimensional vector space, preserving their structural and relational information [36]. Here, we focus on embedding the whole graph as our purpose is to detect intrusions in MSGs [37]. Consider $G=\{G_{1},\ldots,G_{m}\}$ as a collection of $m$ graphs, with each graph $G_{i}=\{V_{i},E_{i},l_{i}\}_{i=1}^{m}$ consisting of sets of vertices $V$ and edges $E$, and $l_{i}$ representing the class label of $G_{i}$. Whole-graph embedding aims to map an entire graph into a low-dimensional space $\mathbb{R}^{d}$ where $d\ll|V|$. This embedding vector should ideally retain as much of the graph’s structural information as possible [38].

In our analysis, we select node2vec[39], renowned for its effective network feature learning capabilities including node classification, link prediction, and community detection [39],[40]. Whole-graph embeddings are constructed by aggregating the node embeddings generated by node2vec, which captures the overall structure of the graph by summarizing the features of its individual nodes. This aggregation process typically involves averaging or pooling to combine node features into a single representation, providing a comprehensive view of the graph’s topology [41]. node2vec is particularly suitable for CAN bus data due to its proficiency in preserving the network topology and learning meaningful patterns of interactions between ECUs [42]. This is critical for identifying and classifying masquerade attacks within vehicular networks. The node2vec algorithm excels by extracting detailed feature representations for ECUs based on their communication patterns and roles within the network. node2vec parameters include walk length (l), which is crucial in determining the scope of each random walk, influencing whether the embeddings show immediate connectivity or broader network contexts. This adjustment allows feature learning to cover both local interactions and the entire network structure. The number of walks (r) initiated from each node deepens network exploration, ensuring a thorough representation by capturing diverse neighborhood configurations. Control over the random walks is further adjusted through the return parameter (p) and the in-out parameter (q). These parameters help direct the walk’s focus, switching between close neighborhoods and more distant connections. The return parameter (p) affects how likely the walk is to return to a node soon after departing, influencing the focus on local versus broad network connections. Conversely, the in-out parameter (q) shapes whether the walk tends to explore regions close to or far from the starting node, helping to balance the emphasis on tight community structures and a node’s role in the larger network topology. Additionally, the dimensions (d) parameter determines the size of the embedding vectors, balancing computational efficiency with the ability to capture rich graph-related properties.

Islam et al. [43] proposed a four-stage IDS utilizing the chi-squared method to identify both strong and weak cyberattacks on the CAN bus. Their approach, which is among the first graph-based defense systems for CAN, showed misclassification rates of 5.26% for DoS attacks, 10% for fuzzy attacks, 4.76% for replay attacks, and zero misclassification for spoofing attacks. The dataset used for evaluation was obtained from the Hacking and Countermeasure Research Lab [44]. This study emphasized the need for robust security mechanisms in modern vehicles and demonstrated superior accuracy compared to existing ID sequence-based methods.

Jedh et al. [21] proposed a message injection attack detection solution. Their approach leveraged MSGs and used graph-based analytics and anomaly detection to detect malicious message injections with high accuracy and low detection time. They validated their approach using a dataset collected from a moving vehicle under attack conditions. This study highlighted the importance of addressing the security of in-vehicle communication networks without relying on proprietary ECU information.

Islam et al. [45] introduced a novel graph-based Gaussian Naive Bayes (GGNB) algorithm for CAN IDS. They leveraged graph-based features from the nodes and edges. The GGNB method applied to the real raw CAN dataset, achieved high detection accuracy across various attack types, including DoS, fuzzy, spoofing, and replay attacks. The study also utilized the OpelAstra dataset [46] and reported significant improvements in training and testing times compared to SVM classifiers. This research showed the efficiency and effectiveness of the GGNB-based methodology in a resource-constrained environment.

Sreelekshmi et al. [47] proposed a graph-based IDS for CANs achieving notable accuracy in detecting various types of attacks. Their method leverages bidirected graphs and parameters like degree variance to identify anomalies. The system was tested on the Car Hacking: Attack & Defense Challenge 2020 dataset [48] demonstrating superior performance with an accuracy of 98.38%.

Refat et al. [49] presented a lightweight IDS that translates CAN traffic into temporal graphs and applies neighborhood-based graph similarity techniques to detect message injection attacks. The system was evaluated using real vehicle data [50], achieving a detection accuracy of 96.01% for spoofing, fuzzy, and DoS attacks. This work contributed to vehicle security by providing a computationally efficient method without requiring changes to the CAN protocol.

Zhang et al. [23] proposed a CAN bus anomaly detection system using graph neural networks (GNNs) to detect message injection, suspension, and falsification attacks. Their approach involved creating directed attributed graphs from CAN message streams and training a two-stage classifier cascade. The evaluation on a Ford Transit 500 dataset [51] demonstrated the system’s efficiency in real-time detection and its ability to handle new anomalies through federated learning training.

Park et al. [52] introduced the G-IDCS, a graph-based IDS integrating a threshold-based IDS and a ML-based attack classifier. This system reduced the number of required CAN messages for detection and improved detection accuracy by over 9% compared to existing methods. They used the Car Hacking: Attack & Defense Challenge 2020 dataset [48] for evaluation. Their study emphasized the system’s robustness to changes in attack types and its potential use in digital forensic investigations.

Xiao et al. [14] proposed the CAN-GAT model based on graph attention networks for in-vehicle networks. By transforming CAN bus messages into graph structures, the model captured the correlation between traffic bytes, improving detection accuracy and efficiency. The model was evaluated using datasets from the Hacking and Countermeasure Research Lab [44], demonstrating superior performance among compared GNNs.

Meng et al. [53] developed GB-IDS, an IDS leveraging a novel graph structure and a variational autoencoder for training classifiers without negative samples. Their system, evaluated on the OTIDS dataset [54], achieved high detection success rates for DoS, fuzzing, and impersonation attacks. This study addressed the limitations of traditional IDS, i.e., [49] by avoiding the need for protocol parsing and large training datasets.

In this paper, we propose a unique framework for detecting masquerade attacks in CANs, leveraging graph ML and time series analysis. By representing CAN frames as MSGs and annotating their nodes with time series features, we fill a gap in the current research. Our approach builds upon previous graph-based methods, such as those introduced by Islam et al.[43], Jedh et al.[21], and demonstrates competitive performance against recent studies using the ROAD dataset [31] [55]. Unlike methods focusing solely on time series data, our framework captures both structural and temporal aspects of CAN frames, enhancing detection capabilities. The flexibility of our framework, adjustable via the window length and offset in a sliding window environment, allows for fine-tuning performance in constrained vehicular systems filling the gaps in [23]. We test our framework on the most realistic masquerade attacks from the ROAD Dataset [15], demonstrating its potential for enhancing automotive cybersecurity.

Zhou et al. [56] developed BTMonitor, a bit-time-based IDS for CANs, achieving a sender identification accuracy of 99.76% on a real vehicle. Their approach leverages the small discrepancies in bit times to fingerprint sender ECUs and can detect new types of masquerade attacks that other IDS fail to identify. They evaluated the system using a three-node CAN bus prototype and a 2012 Buick Regal production vehicle dataset.

Ying et al. [57] proposed a formal analysis of clock skew-based IDS and introduced the cloaking attack, a new type of masquerade attack that manipulates message inter-transmission times to avoid detection. Their study, validated using hardware testbeds and the UW EcoCAR dataset [58], predicted the attack success probability with an average prediction error of within 3.0%.

Hanselmann et al. [59] proposed CANet, a neural network architecture for detecting intrusions on the CAN bus. CANet, the first deep learning-based IDS capable of handling the high dimensionality of CAN bus data, was evaluated on real and synthetic CAN data. The method outperformed previous ML-based approaches with a high true negative rate, typically over 0.99, demonstrating robustness in detecting a large number of synthetic masquerade attacks [59].

Moriano et al. [31] focused on detecting masquerade attacks on the CAN bus by analyzing time series extracted from raw CAN frames. Using hierarchical clustering, this study demonstrated that changes in cluster assignments could indicate anomalous behavior. The proposed forensic tool was tested on the ROAD dataset [15], showing significant differences in time series clustering similarity on benign and attack conditions.

Shahriar et al. [55] developed CANShield, a deep learning-based signal-level intrusion detection framework for the CAN bus. CANShield handles high-dimensional CAN data streams, analyzing time-series data from different temporal scales using multiple deep autoencoder networks. Evaluation on the SynCAN [59] and ROAD [15] datasets demonstrated the framework’s robustness against various advanced attacks, improving the overall AUC-ROC by 6.40% compared to conventional methods.

Sharmin et al. [8] contributed to CAN IDS benchmarking efforts by comparing six CAN IDS algorithms using the ROAD dataset [15]. The study included algorithms based on ID sequences, entropy, Hamming distance, frequency and isolation forest. Their results showed that entropy- and frequency-based algorithms performed better emphasizing the need for fine-tuning existing methods to detect sophisticated attacks like targeted ID and masquerade attacks.

Shahriar et al. [3] proposed CANtropy, a feature engineering-based lightweight CAN IDS. CANtropy explores a comprehensive set of features from both temporal and statistical domains utilizing PCA for anomaly detection. Evaluation on the SynCAN dataset [59] showed CANtropy’s effectiveness, with an average AUC-ROC score of 0.992, outperforming existing DL-based baselines.

Moriano et al. [60] conducted a benchmark study on four non-deep learning unsupervised online IDS for masquerade attacks in CANs. They controlled streaming data conditions in a sliding window setting and used realistic attacks from the ROAD dataset [15]. They noticed that there is no one-size-fits-all solution for detecting masquerade attacks in CAN in an online setting, i.e., there is no single algorithm that is optimal for detecting each attack type. Notably, a method that detects changes in the hierarchical structure of clusters of time series outperformed others in detecting various attack types at the expense of computational overhead.

We use sliding windows of fixed size to partition and process the stream of CAN data [61]. The raw CAN data is first transformed into a structured dataframe format where each row represents a message with its timestamp, process ID, and data payload (see Fig. 4). This data is then partitioned into discrete time slices defined by specific window sizes and offsets, enabling a continuous analysis of CAN data.
We now define the key parameters controlling the sliding windows:

• •

  Window size ($\omega$): The length of each window determining the extent of data included in each graph representation.

• •

  Offset ($\delta$): The sliding step between consecutive windows, where $\delta$ represents the number of samples separating each window, ensuring no loss of data and the capture of communication patterns that may span multiple windows.

A MSG is constructed within each defined sliding window to model CAN activity following the procedure introduced by [21] and [43]. Graphs are empirically recognized as effective tools to model relationships among relational data that are too complex for tabular representation or other simpler data structures [62]. The process of building MSGs is crucial as it sets the stage for subsequent analyses, including node annotation and embedding generation. By modeling CAN data using MSG on sliding windows, we enhance our ability to detect and respond to potential masquerade attacks by finetuning the portion of analyzed data. We now detail each of the components of the MSG:

• •

  Nodes: Each node in the MSG corresponds to a unique ID in CAN representing the header of the CAN frame. This alignment allows for the detailed representation of each frame’s interactions on streams of CAN data.

• •

  Edges: Directed edges are established based on the sequential relationships of messages. That is, an edge is formed when one ID follows another, indicating the flow of communication [43]. This method effectively maps sequences of CAN messages enabling the detection of irregular patterns that may signify an attack. Edge weights are assigned to edges of MSGs patterns to quantify the frequency of observed communication sequences within a sliding window. Specifically, the weight of an edge from node $v_{i}$ to node $v_{j}$ is computed as the number of times the CAN message with ID $v_{j}$ directly follows the CAN message with ID $v_{i}$ within the same sliding window. Weights help to identify anomalous recurrent patterns that could indicate security threats.

Fig. 4 shows how MSGs are built and their corresponding components. Our modeling framework ensures that both structural and temporal aspects of CAN communications are captured, facilitating a comprehensive analysis of the CAN dynamics. Algorithm 1 describes the creation of MSGs.

To ensure the correctness of Algorithm 1, we establish a loop invariant for the main while loop. A loop invariant is a property that holds true before and after each iteration of the loop, helping to trace the algorithm’s functionality and correctness. We demonstrate that the loop invariant holds through three key steps:

1. 1.

   Loop Invariant: At the start of each iteration of the while loop, current_time is a valid start time for a new window, and all CAN messages up to current_time have been processed into the main graph $G$.

   • •

     Initialization: The invariant holds when current_time is set to the start time of the CAN data. No messages have been processed yet, and $G$ is empty. This ensures that the loop invariant is true before the loop starts.

   • •

     Maintenance: During each iteration, a window is defined from current_time to current_time + $\omega$. All messages in this window are extracted and processed into a subgraph $G_{\text{sub}}$, which is then merged into the main graph $G$. current_time is incremented by $\delta$, updating it to a new valid start time for the next window, ensuring all messages up to the new current_time have been processed. This demonstrates that the property holds after each loop iteration.

   • •

     Termination: The loop terminates when current_time + $\omega$ exceeds the end_time of the CAN data. At this point, there are no more complete windows of size $\omega$ that can start at or after current_time. The invariant holds true because current_time has moved past the end of the data, confirming that all messages in the valid windows have been processed. The loop invariant confirms that the algorithm functions as intended and supports its correctness.

   By maintaining the loop invariant throughout the execution, we demonstrate the correctness and robustness of the MSG creation algorithm.

2. 2.

   Time Complexity: The time complexity of our MSG creation algorithm depends on $\omega$ and $\delta$. In general, it can be expressed as $O\left(\frac{n}{\delta}\cdot\omega\right)$, where $n$ is the total number of messages in the CAN data. This is calculated as follows: The outer loop in Algorithm 1 iterates approximately $\frac{n}{\delta}$ times, processing the CAN data in steps of $\delta$. Within each iteration, a window of size $\omega$ is processed, involving operations such as extracting messages and constructing subgraphs, which take $O(\omega)$ time. Therefore, the overall time complexity is the product of the number of iterations and the operations per iteration, leading to $O\left(\frac{n}{\delta}\cdot\omega\right)$.

   In our experiments, we test various combinations of $\omega$ and $\delta$, ranging from (2, 1) to (15, 14) for time-based windows and from (50, 50) to (400, 400) for sample-based windows. For each attack, we run the algorithm with these combinations to analyze the impact of $\omega$ and $\delta$ on performance and detection accuracy. For example, with a combination of $\omega=4$ and $\delta=4$, the time complexity simplifies to $O(n)$, as each message is processed exactly once. This reflects the trade-off between $\omega$ and $\delta$, with larger $\omega$ or smaller $\delta$ increasing the number of operations. The flexibility of our implementation allows for easy adjustment of these parameters, enabling a balance between granularity of analysis and computational efficiency.

Concurrently with the creation of the MSGs, raw CAN messages are decoded using a DBC as described in [63]. In doing so, CAN data is transformed into a time series format to capture detailed patterns of the signals contained in every CAN ID per sliding window. We used CAN-D [64] to extract timeseries from CAN logs. Up to the time of this writing, CAN-D is still the state-of-the-art method for CAN reverse engineering [65]. Fig. 5 shows a decoded representation of the signals representing the four wheels’ speed of a vehicle [15]. The plot is generated by translating the CAN frames into a continuous time series format, which captures the network activity over time. This representation facilitates a detailed characterization of masquerade attacks within CANs highlighting variations and trends that may not be evident relying on raw CAN data.

We enrich MSGs further by annotating each node with statistical attributes derived from the time series data. In particular, we include the mean and standard deviation of each signal associated with each CAN ID. Note that other time series features can be computed but we opted for simpler statistical features that have constant complexity [3]. Our hypothesis is that enriching MSGs with contextual data from the time series significantly enhances our ability to conduct graph ML and detect subsequent masquerade attacks. Note that the graph annotation process is performed as MSGs become available. This approach ensures that CAN communications’ temporal and semantic aspects are captured comprehensively. Fig. 6 shows how each node is annotated with statistical attributes, specifically the mean ($\mu$) and standard deviation ($\sigma$) of the associated signals. Algorithm 2 describes the pseudocode of this process.

Graph embeddings provide a powerful way to represent the complex structure of graphs into a lower dimensional space such that ML algorithms can effectively take advantage [66, 67, 68]. In our study, we focus on shallow graph embeddings and in particular the node2vec algorithm [39], which is well-suited for capturing the nuances of weighted and directed graphs characteristic of MSG. Moreover, node2vec facilitates the transformation of MSG nodes into vector representations, preserving nodes’network topology. We configure node2vec with specific parameters—64 dimensions (d), a walk length (l) of 15, 100 walks (r) per node, a return parameter (p) of 1.5, and an in-out parameter (q) of 0.5. This configuration allows capturing the unique characteristics of MSGs while excelling at inference tasks included in the graph annotation phase [39]. Our configuration ensures that the embedding process remains efficient, thereby supporting near real-time detection capabilities, which is critical for timely identification of masquerade attacks in CANs. Note that this relatively low embedding dimension has shown to be enough for capturing rich graph-related properties [69]. In our work, instead of focusing on node embeddings, we construct whole-graph embeddings [32]. We compute them by averaging node embeddings within sliding windows leading to a unified representation of the network’s behavior. Specifically, we stack them per node in the MSGs. Each of these graphs is labeled indicating the normal or anomalous activity in CAN, i.e., we frame our IDS problem as a graph classification problem [70]. Algorithm 3 describes how we generate our node2vec embeddings.

The result of this process is a data frame that not only encapsulates node embeddings but also includes the augmented node attribute features with labels indicative of normal or anomalous activity states in the CAN. Before classification, we create a comprehensive feature vector for each graph by concatenating the averaged graph embeddings with the time series statistical features (mean and standard deviation of signals). This representation combines both structural (graph) and temporal (time series) information, providing a robust input for ML classifiers. These comprehensive feature vectors serve as the foundation for our supervised learning approach to masquerade attack detection.

To evaluate the effectiveness of our feature representation and overall anomaly detection framework, we employ supervised learning techniques. We experiment with two ensemble learning methods: Random Forest (RF) [71] and XGBoost [72]. These decision tree-based ensemble learning methods are renowned for their high accuracy, scalability, and robustness, making them exceptionally suitable for dealing with the high-dimensional space characteristic of CAN data. The RF model is instantiated with 100 trees, and depth is capped at 20 to prevent overfitting. A combination of a minimum samples split of 6 and minimum samples leaf of 2 is used to control the growth of the trees further. As the training dataset exhibits a 1:10 ratio between the minority (attack) and majority (normal) classes, we balance the dataset using the synthetic minority over-sampling technique (SMOTE) [73]. A 5-fold stratified k-fold cross-validation is performed to ensure the model’s generalizability during training. The XGBoost model yielded similar results to the RF classifier in our experiments. Despite the comparable performance, we opted to report only results for the RF classifier due to its simpler interpretability and less intensive hyperparameter tuning process, which aligns well with the goals of this study.

Evaluation is performed at the sliding window level. In doing so, we verify the proportion of sliding windows coinciding with an attack region in the testing captures, which refers to the recorded CAN bus data used for testing. We found that for the combinations of $\omega$ and $\delta$, the proportion of windows containing attacks tended to be balanced, i.e., greater than 40%. To quantitatively assess the performance of our anomaly detection framework, we use the area under the receiver operating characteristic curve (AUC-ROC) [74]. The AUC-ROC is a comprehensive evaluation metric for classification problems across various threshold settings. It reflects the model’s ability to distinguish between classes, with higher values indicating better performance. The ROC curve is a plot of the true positive rate (TPR) against the false positive rate (FPR) at different threshold levels. The TPR, also known as sensitivity or recall, measures the proportion of actual positives correctly identified, while the FPR measures the proportion of actual negatives incorrectly identified as positive. Mathematically, the AUC-ROC is defined as:

The value of the AUC-ROC ranges between 0 and 1. An AUC-ROC of 1 indicates perfect classification, whereas an AUC-ROC of 0.5 suggests no discriminative power, equivalent to random guessing. An AUC-ROC below 0.5 indicates that the model performs worse than random guessing. In practical terms, the AUC-ROC can be interpreted as the probability that the classifier will rank a randomly chosen positive instance higher than a randomly chosen negative instance. This interpretation is particularly useful in our context, where we aim to detect subtle masquerade attacks embedded within normal CAN traffic.

Our unifying framework is evaluated on the Real ORNL Automotive Dynamometer (ROAD) dataset, a comprehensive set of CAN data collected from a real vehicle at Oak Ridge National Laboratory (ORNL) [15]. This dataset is particularly valuable due to the inclusion of physically verified fabrication and simulated masquerade attacks that provide a realistic environment for testing CAN security methods. The ROAD dataset stands out for its unparalleled quality of CAN information covering the most credible CAN attacks reflecting various driving scenarios. The dataset comprises 3.5 hours of recorded data, with 3 hours allocated for training and the remaining 30 minutes set aside for testing. The test data incorporates five masquerade attacks including ‘correlated signal’, ‘max engine’, ‘max speedometer’, ‘reverse light off’, and ‘reverse light on’ attacks. These attacks are designed to manipulate specific vehicle states. This dataset enables us to conduct a comprehensive evaluation of our framework under a variety of attack conditions. Table I provides a comprehensive overview of the masquerade attacks used in our study and their impacts on the vehicle’s state.

Our evaluation begins with the establishment of a baseline model that relies solely on graph embeddings, i.e., focusing only on the topology of the MSGs. This baseline model provides a fundamental comparison point and is essential for assessing the effectiveness of incorporating additional features. Fig. 8 shows the performance metrics for this baseline, serving as the benchmark for our subsequent enhancements. Specifically, the AUC-ROC values for ‘correlated signal attack’ reach an optimal performance of 0.98 with a $\omega$ of 8 seconds and an $\delta$ of 1 second, showcasing strong model effectiveness in this configuration. In contrast, ‘reverse light on attack’ achieves its highest AUC-ROC of 0.97 with $\omega=7$ seconds and $\delta=4$ seconds, respectively, demonstrating its dependence on extended time frames. In general for the remaining attacks, we notice that high classification results can be obtained for $\omega$ and $\delta$ of several seconds. These insights confirm that adjusting the configurations to suit different types of attacks is crucial for maximizing detection capabilities. The variance in performance across different settings shows the importance of fine-tuning the sliding window parameters to enhance the efficacy of anomaly detection systems in vehicular networks.

Building upon our baseline, we incorporate time series features of CAN signals alongside the graph embeddings. This setting harnesses both the structural patterns and the temporal dynamics of CAN frames aiming to achieve a more comprehensive anomaly detection. Fig. 8 depicts the performance metrics of this enhanced model. The inclusion of time series data significantly improves detection capabilities, as evidenced by the AUC-ROC scores. For ‘correlated signal attack’, the model achieves an AUC-ROC peak of 0.99 at a $\omega$ of 3 seconds and an $\delta$ of 3 seconds. This suggests a strong model response to quick changes in CAN signal patterns. Similarly, ‘reverse light on attack’ records its highest AUC-ROC at 0.99 for a $\omega$ of 7 seconds and an $\delta$ of 4 seconds, indicating effective monitoring of critical engine parameters. In general, we notice that when integrating time series features, the required time to get optimal performance is lower than when using only graph embeddings. Specifically, across the five masquerade attacks, the average optimal $\omega$ decreases from 9.8 seconds to 6 seconds, while the average optimal $\delta$ increases from 3.2 seconds to 4.8 seconds when combining graph embeddings with time series features. This 38.8% reduction in $\omega$ coupled with a 50% increase in $\delta$ allows for more granular analysis of CAN messages while spacing the analyses further apart, enabling our method to capture detailed patterns more effectively.

We compare the effectiveness of two detection settings: using embeddings only and embeddings combined with time series features. The evaluation focuses on the AUC-ROC metric (see Section IV F) to compare the performance of these settings against various attack scenarios.

Table II summarizes results across heatmaps. We place attack categories in the columns and detection settings in the rows. This means that each cell in this table shows the summary statistics from the heatmaps for a specific detection setting and attack. We focus on summary statistics including mean ($\mu$), standard deviation ($\sigma$), median ($\eta$), minimum ($\min$), and maximum ($\max$). In addition, we also compute average ($\overline{\max}$) and standard deviation ($\sigma_{\max}$) of the maximum values for each of the attack detection settings and attack types in the ROAD dataset. They are displayed as last rows and columns in the table.

Graph embeddings combined with time series features generally outperform the graph embeddings-only setting across all attack types. Specifically, for ‘correlated signal attack’, the graph embeddings with time series features setting achieves a mean AUC-ROC ($\mu$) of 0.96 with a standard deviation ($\sigma$) of 0.02, compared to the embeddings-only setting with a mean AUC-ROC of 0.94 and a standard deviation of 0.02. For ‘max engine coolant temperature attack’, the embeddings with time series setting has a mean AUC-ROC of 0.94 and a standard deviation of 0.04, which is higher than the graph embeddings-only setting with a mean AUC-ROC of 0.91 and a standard deviation of 0.04. In the case of ‘max speedometer attack’, the mean AUC-ROC for the graph embeddings with time series setting is 0.94 with a standard deviation of 0.03. In contrast, the embeddings-only setting achieves a mean AUC-ROC of 0.91 with a standard deviation of 0.03. For ‘reverse light off attack’, the graph embeddings with time series setting achieves a mean AUC-ROC of 0.96 with a standard deviation of 0.02, while the graph embeddings-only setting has a mean AUC-ROC of 0.93 and a standard deviation of 0.02. Finally, for ‘reverse light on attack’, the mean AUC-ROC for the graph embeddings with time series setting is 0.94 with a standard deviation of 0.03, compared to the graph embeddings-only setting with a mean AUC-ROC of 0.92 and a standard deviation of 0.03.

In ‘correlated signal attack’, the graph embeddings with time series setting achieves a median ($\eta$) of 0.96, a minimum ($\min$) of 0.92, and a maximum ($\max$) of 0.98. These metrics provide further insights into the detection performance by highlighting the central tendency and range of the results. The average ($\overline{\max}$) and standard deviation ($\sigma_{\max}$) of the maximum values obtained by each attack and experiment type are also reported. These values illustrate the peak performance and consistency of the settings, respectively. For instance, for ‘max engine coolant temperature attack’, the average maximum AUC-ROC ($\overline{\max}$) is 0.98 for the embeddings only setting and 0.99 for the embeddings with time series setting. The standard deviation ($\sigma_{\max}$) for both settings is 0.00, demonstrating consistent peak performance across different experiments.

To further validate the differences in detection effectiveness of both settings, we employed the Mann-Whitney $\mathrm{U}$ test and the Kolmogorov-Smirnov $(\mathrm{KS}$) test. The Mann-Whitney $\mathrm{U}$ test [75] is a non-parametric statistical test that evaluates whether there is a difference between two independent samples. The $\mathrm{KS}$ test [76] is another non-parametric test that assesses whether two samples come from the same distribution. In our analysis, we compared the AUC-ROC value distributions of the two settings: one using only embeddings and the other combining these with time series features. Here, the null hypothesis is that the AUC-ROC values for the setting combining embeddings with time series features are less than or equal to those for the setting using only embeddings, while the alternative hypothesis is that the AUC-ROC values for the setting combining embeddings with time series features are greater than those for the setting using only embeddings. A low $p$-value indicates a significant difference between these two settings. We focused on a significance level of 0.05. Table III shows significant differences between the observed distributions for all masquerade attacks. In each of the attacks, the Mann-Whitney U and $\mathrm{KS}$ tests produce low $p$-values, indicating a significant deviation from the expected distribution. This supports our hypothesis of relying on annotated MSGs to effectively detect masquerade attacks at various windows and offset combinations. Consequently, we reject the null hypothesis for all tested masquerade attacks, confirming that the combined setting performs significantly better.

We investigate the effect of sample-based windows in our detection framework’s performance, as opposed to time-based windows. Our aim is to understand the impact of varying the number of CAN message samples in each sliding window on masquerade attack detection. In this context, time-based windows refer to windows defined by a fixed duration, whereas sample-based windows are defined by a fixed number of CAN messages. We experiment with different window sizes and offsets to find the optimal configurations for detecting anomalies.

Figs. 10 and 10 are heatmaps that provide visual comparisons of the AUC-ROC values for different $\omega$ and $\delta$ combinations for both settings. These heatmaps show that the combined setting generally outperforms the embeddings-only setting, especially at certain window sizes and offsets. However, the optimal configuration varies significantly depending on the attack type, emphasizing the importance of careful parameter selection in practical applications. For instance, in the ‘correlated signal attack’, the combined setting achieves a higher AUC-ROC value of 0.84 with a $\omega$ of 100 and an $\delta$ of 100, compared to the embeddings-only setting, which peaks at 0.79 with the same configuration. Similarly, in the ‘max engine coolant temperature attack’, the combined setting reaches an AUC-ROC value of 0.97 with a $\omega$ of 350 and an $\delta$ of 300, outperforming the embeddings-only setting, which attains a maximum of 0.89 with a $\omega$ of 200 and an $\delta$ of 200. In the ‘reverse light off attack’, the combined setting shows superior performance with an AUC-ROC value of 0.81 with a $\omega$ of 400 and an $\delta$ of 400, while the embeddings-only setting reaches 0.76 with the same configuration. Note that in the remaining attacks (i.e., ‘max speedometer attack’ and ‘reverse light on attack’), combining graph embeddings with time series features allows the classifier to obtain higher AUC-ROC values but requires a lower number of samples.

Table V presents the summary statistics for all masquerade attacks in the ROAD dataset. For both settings, the highest and lowest performances occur in the ‘max engine coolant temperature attack’ scenario, with varying window sizes and offsets. In the graph embeddings-only setting, we achieve a mean AUC-ROC ($\mu$) of 0.61 with a standard deviation ($\sigma$) of 0.06. When we combine graph embeddings with time series features, the mean AUC-ROC ($\mu$) improves to 0.68, with a standard deviation ($\sigma$) of 0.05. Our findings back previous results that the integration of time series features with graph embeddings generally enhances the model’s performance.

The granular window approach proves effective with performance varying significantly across different attack types and window configurations. Individual attack scenarios reveal interesting patterns. For instance, ‘correlated signal attack’ shows consistent improvement when time series features are added, with the mean AUC-ROC increasing from 0.68 to 0.74. The ‘max engine coolant temperature attack’ exhibits the highest variability in performance, indicating high sensitivity to $\omega$ and $\delta$ selection. The ‘max speedometer attack’ and ‘reverse light on attack’ prove more challenging to detect, with lower mean AUC-ROC values in both settings. The ‘reverse light off attack’ shows moderate improvement with the addition of time series features. We observed that the average $\omega$ increased slightly from 240 to 260 samples when incorporating time series features, while the average $\delta$ remained constant at 220 samples. Despite this minor increase in $\omega$, the combined approach consistently demonstrated superior detection performance across various attack scenarios, as evidenced by the improved AUC-ROC values. For instance, in ‘correlated signal attack’, the AUC-ROC improved from 0.79 to 0.84, while in ‘max engine coolant temperature attack’, it increased from 0.89 to 0.97. These results suggest that the integration of time series features with graph embeddings enhances the model’s ability to detect anomalies, even when requiring slightly larger sample windows. This improvement in detection capability outweighs the small increase in computational requirements, further validating the effectiveness of our combined approach in various window configurations.

We also found the distributions of AUC-ROC values to be significantly different between the two settings. The Mann-Whitney $\mathrm{U}$ and $\mathrm{KS}$ tests confirmed this difference, indicating the superior detection performance of the combined setting. Table IV summarizes the results, showing significant differences across all tested masquerade attacks. For instance, ‘correlated signal attack’ yielded a Mann-Whitney U statistic of 1069.5 with a $p$-value of 9.84e-07, and a KS statistic of 0.58 with a $p$-value of 2.61e-06. Similar trends were observed for other attacks, such as ‘max engine coolant’ and ‘max speedometer’ attacks, confirming the effectiveness of incorporating time series features with graph embeddings. The results for each attack type consistently indicate that the combined setting outperforms the embeddings-only setting, validating the improvements seen in the AUC-ROC metrics.