Cyber-Physical Taint Analysis in Multi-stage Manufacturing Systems (MMS): A Case Study

Multistage Manufacturing System (MMS) refers to a manufacturing system consisting of multiple components, stations, or stages required to finish the final product. Almost all modern manufacturing processes (e.g., assembly, machining, semiconductor fabrication, pharmaceutical manufacturing) fit this category. In this report, a multi-stage hot rolling mill automation system testbed will be used for case studies and experiments. It consists of digital motor drives, Programmable Logic Controllers (PLC), control desks equipped with touchscreen LCD operator interfaces, remote I/O units, digital communication infrastructure and SCADA computer(s). All components of the system are connected to a digital communications network, to provide fast and safe data exchanges. Motor drives residing on this network are under the control of PLC for a safe start and rolling operation. A multistage hot rolling process often has 10 to 30 rolling stations, and each rolling station has multiple rolls driven by motors and gear boxes. All the rolls (motors) need to be controlled in a synchronized and cascaded speed, which is designed to match the cross-section reduction ratio of the rolled bars. If a security attack occurs to alter one (or several) roll (mo- tor) speed, it may cause bar surface quality defects, stretch/compress of rolling bar before/after the speed altered station, and even severer incidents such as broken bars or cobbles in the rolling process, inducing significant production system damage or failure.

In MMS, each manufacturing machine is controlled by a Programmable Logic Controller (PLC), and those machines are interconnected in both cyber and power networks for coordinated/planned manufacturing processes. The cyber-threats may compromise the integrity of machine controllers and man- ufacturing machines through the communication networks or other means such as Trojan or insider attack. The integrity attacks are stealthy and difficult to detect in the cyber-space, but may be observed in the physical world or signals.

In our threat model, integrity attacks may be launched from Stuxnet-like malware (e.g. [3]) or malicious insiders. Malware can compromise PLC programs. In addition, we assume that insiders can compromise PLC source code to intentionally result in integrity violations (e.g., PLC logic bombs [5]). It is found in survey studies [6, 7] that insider attacks are top security challenges for air-gapped ICS (Industry Control Systems). As a result, (a) PLC source code and configurations may not be trustworthy; (b) the SCADA computer may not be trustworthy either. In our threat model, integrity attacks may also be launched from a manufacturing communication network. For example, the widely-known Maroochy Water Services Attack [8] injects bogus radio commands (i.e. messages) to the sewage equipment, causing 800,000 liters of raw sewage to spill out into local parks, rivers and even the grounds of a hotel. This example indicates that network intrusions should be a main part of our threat model. Regarding what are trusted, we assume that the rest of the manufacturing system, including the digital motor drives, the remote I/O units, the DSP components, the RTOS (real-time operating systems), as well as our data collection mechanisms (e.g., the sensors) are trusted.

The integrity attacks, in general, may bypass the traditional cyber-security measures but could have been detected by monitoring physical-system signals, such as electric waveform and product quality signals in MMS. To further trace down the root causes, the system needs to audit cyber signals and perform taint analysis. The root causes could include intrusion attack, malwares, trojans, insider or hardware faults (which is not a cyber-threat).

The existing cyber-security research [9, 10, 11] most relies on cyber signals, integrating physical signals for cyber-threat detection is still in its infancy. Methods were developed to detect cyber threats by monitoring and analyzing structural health of the finished parts [12], multiple physical signals of manufacturing process (such as acoustic signals, vibration, magnetic intensity, coal feed) [13, 14], vision and acoustic signals [15]. Traditionally, quality controls are used in MMS to reduce product variability and detect the equipment aging or failure with the goal to ensure the quality of manufacturing processes. In recent years, efforts are made to use quality control methods to address cyber security issues in manufacturing system [16, 17, 11, 18]. While these works provide crucial insights in cyber-physical security in manufacturing, they are limited in analyzing the effect of cyber threat on a single existing quality control tool, such as control chart. As MMSs are increasingly vulnerable to cyber threats, those existing quality controls are not effective to detect the malicious cyberattacks that cause machine failure and quality degradation [11, 19]. There lacks a generalized methodology to detect and diagnose cyber and physical threats in manufacturing systems.

The cybersecurity of Industrial Control Systems (ICS), including PLC security, has been drawing increasing attention in the research community (e.g., [20, 21, 22]). The existing works can be classified into 6 bodies.

• •

  Novel cyber-attacks: In [20], a Rootkit attack is proposed to compromise PLCs. In [23], ransomware for ICS are proposed. In [24], stealthy attacks (e.g., gradually decreasing the system integrity) on ICS are proposed. In [25], three classes of new cyber-attacks are proposed to disrupt the power grid with an IoT botnet.

• •

  Intrusion Detection: In [26], temporal execution behavior is analyzed to detect intrusions in PLCs. In [27], event-driven finite state machines are leveraged to detect data-oriented attacks on cyber-physical systems. In [21], a framework is proposed to generate invariants for anomaly detection in ICS. In [28], a mechanism is proposed to detect stealthy attacks on ICS.

• •

  Statically verifying PLC logic: In [29, 30, 31, 32, 33], PLC logic is statically verified in a formal manner.

• •

  Detection of PLC safety violations: In [34, 35], dynamic simulations of runtime behaviors are leveraged to detect PLC safety violations. In [36, 37], symbolic execution on PLC code is leveraged to detect PLC safety violations.

• •

  Safety vetting of PLC code: In [22], a new program-analysis-based approach is proposed to generate event sequences that can be used to automatically discover hidden safety violations.

• •

  Reverse engineering: In [38], an ICS reverse engineering framework is proposed to reverse engineer PLC binaries.

As manufacturing machines and equipment are connected through power networks, cyber-threats will cause electrical signals/waveforms changes (that might include energy consumption, voltage, current, harmonics) in power networks. A method was developed to detect data integrity attacks modifying the G-code movement commands of 3D printing systems by monitoring current supplied to each electric machine [39]; however, it is infeasible to monitor the current of each machine in a large-scale manufacturing systems. In smart grid, cyber security studies used waveform data from phasor measurement unit (PMU), microPMU, and smart meters [40, 41, 42, 43, 44, 45]. While smart grid security studies provide necessary technical foundation, they mainly address the cyber threats affecting the functionality, stability, and cost of large-scale power networks rather than the function and precision of the devices and equipment, which are generally concerned in manufacturing systems. There are no existing works on analyzing electrical waveforms to detect threats in MMS. Researchers have obtained promising results from their preliminary works [46, 47] on cyber-security of electrical machines in power networks.

As shown in Figure 4, from the viewpoint of the MMS sensing components, the proposed information flow model is anchored by five new DTA notions: events, attributes, defect patterns, (physical) signals, and analysis results. In particular, (a) when a harmful event (e.g., motion of a motor) is tainted, all of the event’s attributes should be tainted. An event is recorded at every millisecond in our current data set. (b) Since one attribute of the event is usually a particular defect pattern associated with the product, the defect pattern should be tainted. (c) When the defect pattern is being sensed, the physical signals (e.g. quality and electrical signals) should be tainted. (d) All the analysis results of any tainted signals should be tainted. For example, a previous event at an upstream motor is a main cause of the defect. (e) As soon as the previous event is tainted, we will reuse the above-mentioned rules to further propagate the taint.

To better understand the information flow embedded in the testbed, we have this mapping relationship as the bridge to connect the testbed and the information flow together. As shown in Figure 2, there are three main components in the current testbed setup: a PC running FreeMASTER, a S32K144 ARM board, and a motor. The PC communicates with the ARM board through LPUART (Figure 2), which corresponds to the PC provides speed reference to Speed Controller on the ARM board(Figure 3). The ARM board, as the cyber-space, sends the PWM control signals to the motor through the FTM(PWM) module (Figure 2), which corresponds to the current controller sends PWM control signals to winding voltage in the motor drive (Figure 3). The motor, as the physical world, sends the current signals and the speed signal back to the ARM board through ADC module and FTM(QUAD) module respectively, which can be mapped to winding current sends current signals back to the current controller and rotor speed sends the speed signal back to the speed controller correspondingly. In this way, we could explicitly know how the information flows through the testbed.

Then, we will demonstrate how the control information flow is depicted with our proposed information flow model. Since our current focus is on how to use the collected data to generate the taint graph, we use a PC running FreeMASTER to represent the infected PLC program to send a malicious command to the ARM board as the attack scenario. Under such an assumption, we explain what the five new DTA notions in our information flow model stand for in the control information flow. Specifically,

• •

  (Figure 4) When a suspicious harmful event occurred (e.g., unexpected speed change of a motor), which corresponds to the change of rotor speed in Figure 3, and such an event will be tainted firstly.

• •

  Usually an event has its intrinsic attributes like some intermediate parameters in the control testbed (e.g., Us_d, Us_q in Figure 2) as well as a particular defect pattern associated with the product, and the defect pattern should also be tainted.

• •

  When the defect pattern is being sensed, the associated physical signals should be tainted (e.g., quality and electrical signals), which corresponds to the current signals and speed signal in Figure 3.

• •

  All the analysis results of the tainted signals should be tainted. For example, the statistical analysis results could reveal that a previous event at an upstream motor is a main cause of the defect.

• •

  (Figure 4) As long as a sensor is tainted (e.g., current sensor and encoder in Figure 3), the taint will be propagated to the next event as part of the inputs.

At this point, we have acknowledged how the Cyber-Physical information flow loops express the information flow in the mini-testbed. Next, we will dive into the details about how to generate the taint graph.

As mentioned in Section IV, a set of the physical measurements has been collected from the pre-deployed sensors on the testbed. Such a dataset is used to construct our taint graph aligning with the Cyber-Physical information flow loops. The dataset is recorded in 99 time cycles and every cycle has 64 milliseconds. In every millisecond, 8 variables are collected, which include is_a, is_b, is_c, theta, omega_actual_mech, us_d, us_q, and m_PWM. The detailed description of the first 5 variables can be found in Table II, the last one m_PWM represents the modulate index for PWM control. Basically, the modulate index is used to generate the switching signal (s1-s6 in the diagram), but the index is not shown in the diagram 2.

At this point, it is known that how the information flow model expresses the information flow of the testbed. As shown in Figure 5, basically, the taint graph is to explicitly show the information flow embedded in the testbed. The whole taint graph is generated following the time sequence of the collected data. At every specific millisecond (marked as one event), there is one sub-graph for the information flow of the current running testbed. For every sub-graph, we have taint source node, taint propagation, and taint sink node. The followings are how specifically the whole taint graph is constructed based on the proposed Cyber-Physical information flow model and the acquired data set of the testbed.

To construct the taint graph, we developed a software tool written in Python. There are three stages to generate the taint graph: 1) reorganize the dataset and assign the variable IDs to the collected 8 variables; 2) prepare a file for the event at 0 millisecond including the connection information between these variables, both taint information and node type of these variables, and use this file as input into our reading procedure to generate the same kind of file, which contain the above information for all recorded events; 3) the generated file will be fed into our tainting procedure to construct the taint graph.

When a malicious command (e.g., modifying the speed reference to a bad one) is launched at the ARM board, through forward taint tracking on the sample taint graph, a series of damage and compromised components could be identified.

To answer the question “if the speed reference is maliciously modified, what is the damage?”, we have the following answers.

• •

  In Figure 6, once the speed reference node (1-0-0) is maliciously modified, following the forward taint propagation, the arm boar node (d 1-0-1), the connected attribute node (1-0-2) & (1-0-3), m_PWM node, the connected sensor data node (1-0-5), (1-0-6), (1-0-7), (1-0-8), (1-0-9) will all be part of the damage.

• •

  Furthermore, because the mini-testbed utilizes the control information flow in Figure 3, so both the maliciously modified speed reference and the generated sensor data will serve as the inputs for the next event, which means all the following events will all belong to the damage.

However, in the real world setting, some attacks can not be detected in real-time, which means some damage is observed firstly, and then intrusion diagnosis steps in to identify how the intrusion happened and what the root cause is. For example, if defect Y was found in one product produced by motor M at time T, how could we identify the causing factors and even the root cause?

To answer the question “if damage Y is observed, would X be a causing factor?”, we have the following answers.

• •

  In our collected dataset, variable omega_actual_mech is the feedback motor rotating speed, which directly impacts the product quality. In Figure 6, once we know at what time the damage Y is observed, we could locate the abnormal sensor data node omega_actual_mech (aa-bb-09), which will be used as the taint source node for backward taint tracking.

• •

  Through backward taint tracking on the taint graph, as shown in Figure 6, the taint starting from the abnormal sensor data node omega_actual_mech could be firstly propagated from (1-1-9) to (1-1-4). Through (1-1-2) & (1-1-3), the taint is propagated from (1-1-4) to (1-1-1). At this point, the taint will be propagated within the current sub-graph to (1-1-0) and to the sensor data nodes in the previous event respectively. In the former case, the speed reference could be one the candidates who account for the damage Y.

• •

  For the latter case, the taint will be propagated through the events backwardly. During taint propagation, the values of those tainted nodes, e.g., (1-0-4), (1-0-5), (1-0-7), are compared with their normal values recorded under the same testbed configuration. Some obvious mismatching will be marked as suspicious candidates causing the damage Y.