A systematic mapping study on security countermeasures of in-vehicle communication systems

Several review articles have been published in this topic area. Hu and Luo [5] discussed the state-of-the-art techniques of secure communication for in-vehicle networks including message authentication, data encryption, and intrusion protection, and concluded technical requirements of cryptographic mechanisms and intrusion detection policy. Zoppelt and Kolagari [6] described seven typical automotive security and dependability mechanisms, examined them with a set of known attacks, and led to the conclusion that most countermeasures against attacks are hardly effective.

Other than the reviews at a general level, some researchers focus on specific techniques. Loukas et. al. [7] conducted a survey on the intrusion detection systems (IDS) of vehicles, proposed a classification scheme, and discussed learned lessons and open issues in relevant industries. Lokman, Othman, and Abu-Bakar [8] investigated the IDS implementation of the CAN network systems in four aspects (i.e. detection approaches, deployment strategies, attacking techniques, and technical challenges) and categorized them based on seven detection strategies. Grimm, Pistorius, and Sax [9] presented a classification of network monitoring techniques for security purposes and compared typical security measures in enterprise information technology systematically with the ones in the vehicle security field.

Additionally, some reviews are on a certain type of communication media. Gmiden et. al. [10] and Bozdal et. al. [11] discussed the vulnerabilities of the CAN bus protocol, presented and compared various countermeasures against attacks. Kishikawa et. al.[12] highlighted the vulnerabilities of the FlexRay and discussed countermeasures against spoofing attack on FlexRay networks. Boatright and Tardo [13] examined the security aspects of the Ethernet AVB backbone and discussed various Ethernet security solutions including protocols, algorithms, and encryption mechanisms appropriate for vehicular requirements.

All mentioned reviews are not systematic literature reviews or systematic mapping studies and mainly focus on the research findings with no attention to research activities.

Reviewing is an essential research method, which searches, collects, summarizes, and synthesizes existing studies to present current knowledge and research states of a research area. Literature review (LR), systematic literature review (SLR), and systematic mapping study (SMS) (or called scoping study, mapping review) are three common methods for review studies.

LR summarizes evidence on a topic using informal or subjective methods to collect and interpret studies qualitatively to provide a summary or overview of a topic [14]. This kind of review can be found in the introduction or related work sections of research articles or in some review articles.

Due to the informality and subjectivity of LR, SLR is a better choice to conduct a review with low biases. SLR is a high-level overview of primary research that identifies, selects, synthesizes, and appraises all relevant high-quality research evidence and eliminates bias [14].

Although SLR is good at answering focused questions and providing evidence with low biases, it does not present an overview of a topic area. By contrast, an SMS is a defined method to build classification schemes and structure a research field [15] to discover research trends and gaps [16]. SMS is not only based on the findings presented in publications but also the activities related to the findings [17] like when and where a paper was published. The outcome of an SMS is an inventory of papers on the topic area which is mapped to a classification scheme [16].

SMS was originally used in medical fields and then started to get more attention in software engineering around 2007 [15]. Petersen et al. [15] provided a guideline to conduct an SMS in software engineering in 2008, which is the most used SMS guidelines in software engineering [16]. In 2015, Petersen et al. [16] conducted an SMS of SMS to identify improvement potentials in conducting the mapping studies and proposed an updated guideline for conduction SMS in software engineering.

Although the mentioned SMS guidelines and studies are proposed in software engineering fields, the SMS processes are independent of software engineering and are also applicable to other research areas. For example, Zanella Gomes et al. [18] conducted a survey on the provisioning of ubiquitous intelligent services for vehicular users and identified clusters of research interest on this topic. Lun et al. [19] focused on how security is addressed in the cyber physical systems from an automatic control perspective and provided a systematic map of this subject. Both examples are conducted based on the systematic process proposed in [16]. In this article, we report a systematic map on the security countermeasures of the in-vehicle communication systems to present current research states and identify trends and gaps in this topic area by using the updated guideline proposed by Petersen et al. [16].

The goal of this study is to summarize the current research states and identify the trends and gaps in the target research area, which is the security countermeasures of in-vehicle communication systems. This study also serves as a preliminary investigation for further systematic literature reviews on a particular security issue or solution.

Four research questions (RQ) are proposed as follows.

• 1.

  RQ1: What countermeasures were proposed or evaluated for in-vehicle communication systems?

• 2.

  RQ2: How were the proposed or existing countermeasures evaluated?

• 3.

  RQ3: When and where were related studies conducted and published?

• 4.

  RQ4: What are the research trends and gaps in this topic area?

The detailed workflow of the study identification is shown in Figure 2.

All reviewers should first discuss and agree on the study identification strategy and criteria. After the search, publications that are obviously duplicated or violating the inclusion criteria were removed using reference management tools (i.e. Zotero and Mendeley). Then, papers’ meta-data was input into a screening tool called Rayyan [20] for further decision. Two reviewers did the inclusion and exclusion work separately according to the defined criteria. The most inclusive decision rule was defined to specify the behavior of an individual reviewer, which is: if the reviewer is not completely sure that a paper should be excluded, this paper should be considered as included. Similarly, the most inclusive rule for the result summary was defined, which is: if a paper is not excluded by both reviewers, it should be considered as included. When a decision conflict occurred, the two reviewers discussed with each other to solve it. If it failed to reach an agreement, an additional reviewer was involved and made the final decision.

Note that the decision in this step is based on the metadata of papers (i.e. title, keywords, and abstract). The output list of the included paper is preliminary. The final decision would be made based on the full text in the paper extraction process.

Figure 3 shows the paper numbers in the study identification process. In practice, we did the search in two rounds due to the time schedule. The main round started in July 2020 to identify and analyze papers from 2010 to Jun. 2020, and the supplementary search was performed in Jan. 2021 to include the latest papers published in the second half of 2020.

According to the data extraction strategy and criteria defined in section 3.4, we extracted data of each included paper and recorded values in a table. Then, we abstracted the unenumerated values and developed classification schemes for them.

For the values of the main countermeasure, nine abstracted countermeasures are identified, which are listed as follows.

• 1.

  Anomaly detection: Anomaly or intrusion detection methods of a system, e.g. a specification-based or machine-learning-based algorithm to detect the anomaly.

• 2.

  Anomaly reaction: Strategies or methods of system reaction when anomaly or intrusion has been detected, e.g. a method to disable the attacker when being detected.

• 3.

  Hardware support for security: Research on hardware to support better security countermeasures, e.g. a hardware accelerator for cryptographic calculation, realize a secure protocol in hardware.

• 4.

  Key management: Approach for key management in security mechanisms, e.g. key exchange protocols, methods of generating and storing keys, ect.

• 5.

  Secure communication scheme: Protocols or mechanisms to ensure the security of the data transmission in various networks, e.g. secure communication protocol, ID shuffling mechanisms, etc.

• 6.

  Secure component design: Security design methods for a single component in a system, e.g. a secure gateway, secure ECU design, etc.

• 7.

  Security mechanism at application layers: Mechanisms that are specific for a type of applications, e.g. a secure protocol for Unified Diagnostic Service in vehicles, a secure strategy for braking dynamics of cooperative driving, etc.

• 8.

  System security design: System-level design approaches like overall design architecture, framework, etc., e.g. distributed firewall deployment, a comprehensive framework including a set of countermeasures to protect the system.

• 9.

  Others: Other topics that are relevant but can not be classified into the rest classes, e.g. a study of bit level permutation which is used in the cryptographic environment.

For the properties concerned, we recorded many metrics like error rate and average delay originally from the papers. Then, we divided them into five classes, which are introduced below.

• 1.

  effectiveness: Whether the solution is effective to solve the problem. Concrete metrics in this class include detection accuracy metrics (e.g. precision, recall, true positive, true negative, false positive, false negative, f-score), output randomness, etc.

• 2.

  efficiency: Whether the solution is efficient. Typical metrics in this class are latency, detection speed, round trip time, standby time, process throughput, etc.

• 3.

  performance: The performance metrics of the system include network throughput, busload, bandwidth, and metrics related to the system quality (e.g QoS, robustness, reliability, etc.). This class investigates the impacts of the countermeasures on the original system.

• 4.

  resource: This class represents the resource required by the proposed solution, like financial or computational cost, memory consumption, etc.

• 5.

  compatibility: Whether the solution is compatible with other contexts, e.g. be applicable for other media types, be compatible for other protocols.

For the validation or evaluation approach, we adapted the original classification to better fit our study. All research in PS and VR papers used validation methods to validate solutions, which means that they were all conducted in laboratory environments (including stimulations on PCs and experiments in the real-world). Two kinds of abstracted methods for the validation are “prototyping and then doing experiments (PE)” and “Theoretical analysis/proofing (TA)”.

For the security goals, we applied nine security properties classes proposed in the EVITA project [22], which are data origin authenticity, integrity, authorization, freshness, non-repudiation, privacy, anonymity, confidentiality, and availability. Since many papers do not mention the security goals explicitly, we developed a mapping (Table 4) to map the possible attacks or common mechanisms mentioned in publications to the nine security goals. If there is no specific security goals or related attacks explicitly mentioned in the paper, the security goal of this paper is marked as “None”.

For future work, we abstracted the next-step work mentioned by authors and divided them into categories, which are listed below.

• 1.

  Improving performance: To improve the performance of the proposed solutions, including improving the practicability or effectiveness, implementing HW modules, overcoming current limitations, reducing cost, etc.

• 2.

  Extend application scope: To extend application scope for more scenarios, including fighting against or detecting more attacks, integrating the proposed solution with other protocols, supporting in other contexts, trying other HWs, etc.

• 3.

  Further evaluation: To evaluate the proposed solution under other conditions, including with a more extensive set of attacks, with other parameters, with real-world data and scenarios, with large scale, etc., or to investigate and compare the solution with other relevant ones.

• 4.

  Add features: To add new functional features to the proposed solution or to integrate the proposed solution with other methods, like adding a reaction process after the anomaly detection, etc.

• 5.

  Exploit other methods: To find other possible solutions for the mentioned issues or other problems, which is completely different from the proposed solution.

• 6.

  None: No concrete future work is mentioned in the paper.

After we defined all the abstraction and classification, we updated the extraction table based on the classification schemes and output a classified data table for further analysis.

To visualize and analyze the extracted data, we chose the bubble chart and the (stacked) bar chart for data visualization. These kinds of charts can display the relations between two or three variants and are the most commonly used means for the visulization [15]. We developed R scripts for efficient data statistics and drawing the plots. The detail of the data visualization and analysis is presented in section 4.

We evaluate the study from three aspects. First, the validity threats are discussed, including the potential biases and countermeasures we conducted to migrate the biases. Then, the identified study set is compared with a known paper set to evaluate the study identification. Finally, the whole mapping study process is evaluated based on the activities involved according to the evaluation rubric proposed by the guideline [16]. The detail of the evaluation is presented in section 5

Writing a report and publishing it is the final step of an SMS. Mapping strategies, conducting details, and relevant discussion should be reported clearly. We used the workflow (Figure 1) as a checklist to ensure that we do not miss any important information in the report. As for the venue, we would prefer a journal for the publication due to the space limits of conference papers. The study in [16] also found a higher number of mapping studies published in reputable journals rather than conferences.

To conclude meaningful information from the extracted data, we used five keywords to guide the data statistic and visualization process, which are “year”, “country”, “countermeasure”, “validation approach”, and “PS”. The “year” charts display the paper distribution related to the publication time and present the publication profile throughout the past 10 years. The “country” charts show the distributions related to authors’ countries and fields. The “countermeasure” charts display the distribution related to the abstracted countermeasures of papers and represent the research interests in this topic area. The “Validation approach” charts show the validation approach distributions used in the PS and VR papers, including abstracted approaches, tools, and properties concerned in the validation. The “PS” charts show the main security goals and future work of the PS type papers.

Table 5 summarizes the detailed information about the planned charts. We set a rule for the data statistic in this step. If a paper includes two target elements, it should be considered as two records and counted twice. For example, if a paper has two authors from two different countries, this paper was counted twice and belongs to two countries separately when counting countries in the statistic.

With the help of the charts generated based on the data visualization plan in Table 5, four research questions are answered in this section.

The discussion of the validity threat is an essential step to analyze the quality of the mapping study via inspecting potential biases in the process. We follow the validity classification scheme used in the guideline, which includes descriptive validity, theoretical validity, generalizability, interpretive study, and repeatability [16].

To evaluate the quality of the study identification results, we compared the final included papers to a known set, which has been prepared before the study and includes relevant papers we already knew in previous research. Table 6 shows the list of known papers and the comparison results. 8 out of 14 known papers are covered by the final included paper set but 6 are not hit by the identification process. We analyzed the result and tried to find reasons for missing papers. We found that all missing papers mainly focus on particular techniques (No. 4 is on the firewall; No. 10 is on the secure gateway, No. 6 to 8 are on intrusion detection) or concrete application scenarios (No. 11 is about diagnostics in repair shops) and do not use the common words we picked for the search string. This searching bias has been mentioned in the theoretical validity discussion. However, on the other hand, if we do not use more general words like “security countermeasure” in the initial search, we may have no idea what are the specific countermeasures that could be used in this field.

To check how many papers we can newly find if we use more specific words in the search, we designed a supplementary search string, which is “((anomaly detection) OR (intrusion detection) OR (secur* gateway) OR (secur* diagnostic) OR firewall) AND ((in-vehicle OR automotive OR automobile) AND (network OR communication))”. By using more specific words, we found an additional huge number of records (2096). To balance the workload and the size of the collection, we do not add the records found by specific words into the initial collection. Therefore, the papers we identified are those that discuss the security issues using more general words. If the researchers want to achieve an overview of a specific technique (e.g. anomaly detection approaches), they can narrow the scope down to conduct an SMS or SLR on the particular technique with an acceptable workload and a relatively complete collection.

Applying snowballing is also a good choice to migrate bias caused by the search words since it can found content-related papers regardless of the keywords used. However, snowballing requires a huge amount of workload especially starting from a big paper set. Reviewers should make a trade-off decision between the size of sampling and the work efficiency. Since the topic in this SMS is a general one and our purpose of this study is to provide an overview of this topic area instead of finding all evidence, we do not use snowballing in this mapping. In further possible SLRs, the forward and backward snowballing is a good supplementary search strategy to find all evidence on a more specific topic with an acceptable workload.

Note that we don’t add the missing known papers into the included paper set because they were found by manual search randomly and not consistent with the paper identification process in this study.

Finally, we evaluate the mapping process by an evaluation rubric devised in the guideline [16]. A total of 26 relevant activities that can be applied in mapping studies have been identified. The guideline suggests calculating the ratio of the number of actions applied comparing to the total number. Table 7 shows the total activities and the applied ones in this study. 14 activities were conducted and the ratio is 54$\%$. According to the study [16] on the SMS conducted before 2015, the maximum and median ratio is about 48$\%$ and 33$\%$. Therefore, the quality of this study can be considered as high.