Gotcha! I Know What You are Doing on the FPGA Cloud: Fingerprinting Co-Located Cloud FPGA Accelerators via Measuring Communication Links

Long wires, one type of FPGA routing resources that are used to connect configurable logic blocks (CLBs), have been proved to be a source of side-channel information leakage. In (Giechaskiel et al., 2018), the authors find that when a long wire on FPGA is transmitting a logical $1$, the delay of the nearby long wires is shorter than when it is transmitting a logical $0$. Based on this phenomenon, the authors propose to measure the delay of long wires by connecting ring oscillators (ROs) to them. When the target long wire is transmitting a logical $1$, the delay of the nearby long wires will decrease, which causes the frequency of the ROs to increase. By monitoring the frequency change of the ROs in a fixed time interval, the authors successfully recovered $99\%$ of the bits that are being transmitted in the target long wire. Similarly, in (Ramesh et al., 2018) the authors recovered the secret key of an AES implementation using the long-wire side-channel attack.

In certain FPGA circuits (e.g. cryptographic circuits), power consumption may be influenced by data being processed in the circuits hence this information may be monitored and used to recover secrets, e.g. cryptographic keys. Normally, deploying power side-channel attacks requires physical access to the FPGA boards in order to assess the system’s power usage. Although direct access to cloud FPGAs is not achievable, Zhao et.al (Zhao and Suh, 2018) propose a power side-channel attack using FPGA as a power monitor. The authors created an RO-based on-chip power monitor and prove that the RO-based FPGA power monitor may be utilized for a power analysis attack on an RSA crypto module on the same FPGA. Furthermore, in (Gravellier et al., 2019), the authors propose a new design for the RO-based power sensor, which can measure the internal voltage in nanosecond scale. They are able to successfully retrieve the secret key of an AES encryption circuit using the power side-channel.

Power side-channel can also be used for accelerator fingerprinting, as shown in (Gobulukoglu et al., 2021). However, in this paper we will show that communication side-channel can be a better option, which has less stringent requirements for attackers and can achieve better performance.

PCIe contention side-channel has been utilized before to retrieve secret information from CPU-GPU systems (Tan et al., 2021). In (Tian et al., 2021), the authors used PCIe contention to perform an attack on the AWS server. The authors observe that the difference in locations of PCIe slots in the PCIe topology can result in disparate latency and bandwidth. Based on this, they are able to detect the bandwidth change when different FPGAs in the same sever attempt simultaneous memory accesses to generate PCIe contention and successfully reverse-engineer the locality of different FPGAs in the same AWS server. However, unlike our work, (Tian et al., 2021) focuses on revealing infrastructure information instead of revealing information about applications on the same FPGA.

The existing attacks targeting the FPGA cloud presume the knowledge of co-located victim circuit is provided. While in fact this information is nearly impossible to be obtained directly. In response, our work focuses on inferring co-located FPGA-accelerated workloads using PCIe contention side-channel information. Previously proposed attacks will benefit from our attack method.

In our PoC benchmark, the host program is responsible for:

1. (1)

   Assign appropriate resources for the operation of accelerator kernels;

2. (2)

   Invoke and orchestrate accelerator kernels;

3. (3)

   Measure kernel performance using low-level function calls.

There are $3$ design parameters in our host program: BUFFER_NUM, BUFFER_SIZE and REPEAT_NUM. The workflow of our host program is defined as follows. First, the host program will allocate BUFFER_NUM memory trunks of size BUFFER_SIZE. Then, these BUFFER_NUM memory trunks will be accessed by the FPGA accelerator in a pre-defined order. Each of the BUFFER_NUM memory trunks will be read and written by the accelerator, with traffic passing through the communication link. During the operation to a memory trunk, the time it takes to execute the kernel will be recorded using profiling APIs provided by OpenCL. This information will be further used for calculating the bandwidth of the communication link when all operations to a memory trunk are finished. The operations to a single memory trunk may be repeated for REPEAT_NUM times and averaged to cancel the effect of noise. Finally, the BUFFER_NUM measurement of bandwidth will be combined together to form a trace with length BUFFER_NUM. The pseudo-code for the host program is shown in Figure 2.

The measurement accelerator we use in this paper focuses on measuring the I/O bandwidth performance. Similar to previous works that target measuring PCIe performance (Neugebauer et al., 2018) or stressing the PCIe connection (Tian et al., 2021), our measurement accelerator implementation follows a similar method and stresses the PCIe communication link via massive read and write communication. There is one design parameter called ACCESS_NUM that controls how much data is written to the host. The code of our benchmark accelerator implemented as an OpenCL kernel is shown in Figure 3.

First, our benchmark accelerator takes in an address pointer (dst). dst is defined as a pointer pointing to pre-allocated host memory. By doing so, we guarantee that our FPGA accelerator will be able to access legally allocated host memory and the generated traffic will pass the FPGA-host communication link. Our kernel then obtains an arbitrarily assigned index to access the host memory space. The exact index is not important in our implementation, and we only use the OpenCL API get_global_id() for convenience.

Second, our accelerator enters an execution loop where host memory is accessed multiple times via an array update operation. The same location (dst[id]) in host memory will be updated ACCESS_NUM times, where ACCESS_NUM is a design parameter of our benchmark accelerator. The operation listed in Figure 3 ensures that a certain amount of data is transferred and the compiler will not optimize out the operation since every time there will be a new value written to the host memory.

All our FPGA-related experiments are conducted on Intel DevCloud (dev, 2021). DevCloud allows free SSH access to their servers and FPGA resources from academic users. In our experiments, we select to use nodes with Xeon CPUs and Intel Arria 10 series FPGAs. The environment version is development stack release 1.2.1. To compile our OpenCL kernels, we utilize the tool-chain provided by Intel, which is available on these nodes. Results are all obtained from node named s005-n007.

The experiment process is controlled by our host program. After the initial setup of OpenCL environments (getting platform information, setting up context, command queues, etc.), we launch a victim kernel which will run continuously during the experiment. Meanwhile we also launch the proposed benchmark to perform multiple measurements on the PCIe communication link and gather data. In each measurement, we initialize a new memory buffer item in host memory and execute benchmark kernel for BUFFER_NUM times, aggregate acquired data and calculate the average bandwidth as the result of this measurement. For each accelerator, we collect $50$ traces, with BUFFER_NUM points in each trace (default value $100$). Since victim FPGA circuits and our benchmark circuits (both are synthesized from OpenCL kernel implementation) run concurrently and there is no synchronization step between the two kernels, our experimental setting resembles a multi-tenant FPGA cloud setting.

In our experiments, we select $8$ different FPGA-accelerated workloads provided by Xilinx Vitis Accelerator Example repository (Xilinx, 2020) and FPGA-synthesized GPU workloads from Rodinia benchmark (Che et al., 2009). We make necessary modifications to deploy them on DevCloud. Detailed description and abbreviation codes are listed in Table 1. These accelerators cover different critical areas for FPGA accelerators, including image processing, signal processing, numerical simulation and neural network acceleration thus can serve as representative workloads.

The collected data will be further analyzed by our learning models. In our experiment, we build several different models based on Python machine learning libraries like Pytorch (Paszke et al., 2019) and Scikit-learn (Pedregosa et al., 2011). The configurations of these models are listed as follows:

• •

  Random-forest: RandomForestClassifier() from Scikit-learn library (Pedregosa et al., 2011) is used.

• •

  SVM: SVC() classifier from Scikit-learn library (Pedregosa et al., 2011) is used.

• •

  MLP: built in Pytorch (Paszke et al., 2019), using learning rate $0.001$, cross-entropy loss and stochastic gradient descent (SGD) optimizer, being trained for $1500$ epochs.

• •

  1D-Convolution: built in Pytorch (Paszke et al., 2019), using learning rate $0.001$, cross entropy loss and Adam optimizer (Kingma and Ba, 2014), being trained for $1500$ epochs.

In this paper, we consider two attacking scenarios, i.e., closed-world and open world scenario. For closed-world testing, we assume all accelerators are known to the attacker, and we consider the fingerprinting problem as an $n$-way classification problem, with $n$ being the number of accelerators in this closed-world. For the more realistic open-world scenario, we consider it as a binary classification problem (since most attackers will have only one specific target for side-channel attacks) and each classifier will be trained to recognize a single accelerator.

Under both attacking scenarios, we split all collected traces into $7:3$ for training and testing respectively to obtain accuracy data. Additionally, for open-world scenario, we manually remove traces from certain classes in the training set, making these accelerators agnostic to the classifier. The test set will still include traces from these classes to simulate the real-world scenario, where traces from unknown accelerators are collected.

In our experiments, we aim to answer $2$ research questions (RQs):

1. (1)

   RQ1: Does our measurement circuit capture the communication patterns, and what is the accuracy of fingerprinting?

2. (2)

   RQ2: How do the parameter settings of benchmark impact the attacking results?

To answer RQ $1$, we first present the visualization of measured PCIe side-channel traces and fingerprinting accuracy in Figure 5 and Table 2, respectively. All data are obtained from the $8$ FPGA-accelerated workloads mentioned above.

We first present the collected traces with using t-Distributed stochastic neighbour embedding (t-SNE) (Van der Maaten and Hinton, 2008), which is a widely used data visualization method. It projects high-dimensional data to a $2$-D plane and can show how the data points can be clustered. In Figure 5, we collect and visualize the bandwidth traces of our benchmark accelerator when it is running concurrently with one of the $8$ victim circuits. In Figure 5 (a), bandwidth data are normalized with minimum and maximum bandwidth values in the data set and range between $0$ and $1$. We can see that traces belonging to different accelerators are separable, which indicates that our benchmark circuit is able to capture the unique communication patterns existing in the execution of the victim accelerators and generate fingerprints for each of them. The bandwidth difference exposes a vulnerability of inferring the co-located victim circuit. T-SNE results in Figure 5 (b) also prove that the data traces are separable.

Based on findings in Figure 5, which indicates that these traces contain information that can help differentiate different accelerators, we further consider two fingerprinting attacking scenarios, i.e. closed-world and open-world scenarios. Closed-world fingerprinting aims to classify the types of accelerators within a known accelerator set, while open-world fingerprinting only interests in one sensitive accelerator and classify others as ”unrelated”.

Closed-world.

Table 2 shows the closed-world classification accuracy performance of our selected models. Among our models, Random Forest achieves the highest classification accuracy, reaching $88\%$ accuracy in this task. The $10$-fold cross-validation results of our Random Forest model is provided in Figure 6. From the distribution of model metrics like accuracy, precision, recall and F1-score we can see that the model, we can see that the model performance is relatively stable. There are similar but different FPGA workload fingerprinting works (Gobulukoglu et al., 2021; Drewes et al., 2023), where the authors utilize power side-channel to perform classification of different cryptographic cores. Compared to their works, we focus on fingerprinting general computing circuits and utilize a different side-channel. Also, our implementation stays at HLS level.

We select the two models with the best accuracy performance, i.e. SVM and random forest, and provide further details to show how well our model performs in this specific task. Confusion matrices are provided in Figure 7. It shows how accurate the selected models are able to classify each of the victim accelerators. Values in each cell of the confusion matrix represent the number of samples of each (Predicted Label, True Label) pair. We can see from the figure that both models have acceptable accuracy performance ($69.2\%$ and $88.3\%$) and are able to differentiate the $8$ accelerator classes, although random forest works better with fewer misclassified samples and outperforms with a great margin. This could be due to the intrinsic features of the data traces, which are potentially more suitable for the algorithm of random forest and decision trees (Goodfellow et al., 2016).

In the experiments above, we only use standard min-max scaling pre-processing and standard models. With further customization (filtering data, modifying predictive models), the accuracy can be potentially higher, resulting in a higher success rate and lower costs for continuing side-channel attacks.

Summary. Our benchmark accelerator is able to capture communication patterns of co-located accelerators on FPGA and use these generated fingerprints to classify at a higher accuracy. This accuracy performance is sufficient for use in a real-world scenario. From the classifier side, we find that Random forest model achieves the highest classification accuracy and can reach $88\%$ classification accuracy. Surprisingly, the most complicated model, 1D-Convolution, achieves the worst classification accuracy performance. In our experiments, simpler models like random forest and SVM achieve significantly better accuracy performance. This could be potentially attributed to the limited number of traces in our data set.

As mentioned in Section 4, our benchmark has $4$ different design parameter:

• •

  ACCESS_NUM, which corresponds to how much traffic is generated by benchmark accelerator.

• •

  REPEAT_NUM, which is the number of times the kernel is executed and it relates to our measuring granularity and data stability.

• •

  BUFFER_SIZE, which determines how large each buffer is.

• •

  BUFFER_NUM, which relates to how many data points are collected within one performance trace.

The following parameter settings:

• •

  ACCESS_NUM$=1000$,

• •

  REPEAT_NUM$=10$,

• •

  BUFFER_SIZE$=4$ Bytes,

• •

  BUFFER_NUM$=100$.

will be later referred to as our default setting.

In this experiment, we screen all parameters and provide t-SNE visualization and compare their classification accuracy with the one under the default setting. To make visualization results clearer, we drop the simplest accelerator (noisegen) and the most complex accelerator (hotspot) and only perform attacks on the remaining $6$ accelerators. To explore the effects of each of the $4$ parameters, we fix the other $3$ parameters to the default settings and vary the value of the target parameter, then collect data on the $6$ victim accelerators. The t-SNE visualization of the collected data traces as well as classification accuracy traces of our $4$ models regarding the $4$ parameters are shown in Figure 9 - 12. Corresponding accuracy performance of the $4$ models are provided in Figure 13. Figures corresponding to configurations that are identical with default settings are omitted to avoid repetition, the results are the same as in Figure 5 (b). In these figures, we obtain t-SNE results from normalized communication bandwidth data. Overall, the use of different parameter values results in varying trace patterns and can hence affect the accuracy of different models. The analysis of the results we obtain in this experiment is shown as follows.

ACCESS_NUM. In Figure 9, the influence of the parameter ACCESS _NUM is shown. From Figure 9 (a) - (d), we can observe a change in the visualization results, i.e. the traces collected from different victim accelerators show different separability. This indicates that the ability of our accelerator benchmark to capture the differentiable patterns in I/O operations existing in our victim accelerators can vary according to ACCESS_NUM. We can see that after ACCESS_NUM $\geq 2000$, data points from several accelerator classes tend to be mixed together. By looking at Figure 13 (a), we can see that the random forest model achieves the highest accuracy result, reaching an accuracy over $90\%$ at ACCESS_NUM$=1000$. SVM achieves over $85\%$ accuracy and MLP achieves over $70\%$ classification accuracy, both at the same point. However, the 1D-Convolution model is only able to achieve $60\%$ accuracy at its highest. We can also see that as ACCESS_NUM increases, except 1D-Convolution model, the accuracy generally increases first (though the accuracy of our random forest model stays over $90\%$ relatively stably). After reaching the highest accuracy at ACCESS_NUM$=1000$, the accuracy starts to drop.

We speculate that the reason behind the fingerprinting accuracy difference is due to the measurement granularity differences when ACCESS_NUM ranges from $250$ to $4000$. Initially, the growth of ACCESS_NUM introduces more data to be read and written hence the effects of noise can be better cancelled and communication patterns can be better captured until ACCESS_NUM reaches $1000$. However, since the execution time of our benchmark accelerator also increases as ACCESS_NUM grows, the measurement will become more coarse-grained since the change in I/O performance variance of victim accelerators within this execution time period will be amortized. After a certain point (in our experiment, between $1000$ and $2000$), the extended execution time of benchmark accelerator causes the benchmark circuit to lose the ability to accurately capture victim communication patterns, thereby inducing a drop in classification accuracy.

REPEAT_NUM. For parameter REPEAT_NUM, the t-SNE visualization results are shown in Figure 10. Same as in Figure 9, data points belonging to different accelerator classes in Figure 10 (a) - (d) are separable, where the clearest clustering results appear at REPEAT_NUM$=5$ and REPEAT_NUM$=10$ (see Figure 5). The classification results for the machine learning models in Figure 13 (b) also match this observation, with the highest classification results achieved at REPEAT_NUM$=5$ and REPEAT_NUM$=10$, where the accuracy of random forest is again over $90\%$ and the highest accuracy results of SVM and MLP are around $85\%$ and $70\%$, respectively. In Figure 13 (b) we can observe a similar trend as in Figure 13 (a), where the accuracy first increases to an optimal point and starts to drop as REPEAT_NUM grows.

The explanation for the accuracy trend is similar. As REPEAT_NUM determines how many times our benchmark is executed when operating on a memory buffer, increasing REPEAT_NUM will: ($1$) cancel the effects of noise and obtain a more precise measurement of the performance; ($2$) extend the time it takes to operate on a single buffer, i.e. the time it takes to generate a data point in the performance trace. As the execution time of the accelerator kernel task is relatively short, when REPEAT_NUM is low, the measurement will be finished within a short period of time and the dynamic communication patterns cannot be captured by our benchmark accelerator. This, combining the influence of noise, is the reason why all models achieve poor classification accuracy results at REPEAT_NUM$=1$. When REPEAT_NUM increases, the communication patterns start to be captured. However, if REPEAT_NUM is too large, same as the situation in ACCESS_NUM the whole measurement process becomes too coarse-grained. Changes in the I/O communication traffic may be amortized, thus classification models cannot extract detailed information from the collected traces.

BUFFER_SIZE. The experimental results of trace visualization and classification results under different BUFFER_SIZE settings are shown in Figure 11. We can see that all the BUFFER_SIZE settings we use are able to preserve the layering information in the victim accelerators. From Figure 13 (c), we also observe that the influence of parameter BUFFER_SIZE is not as much as ACCESS_NUM and REPEAT_NUM. However, there is an optimal point for the random forest and SVM to work on (BUFFER_SIZE$=4$ Bytes). We will keep using this empirical value since it is the best work point for our most accurate model. However, as BUFFER_SIZE increases beyond the optimal point, there is a slight drop in classification accuracy. This can be due to certain details of the implementation of low-level runtime drivers.

BUFFER_NUM. Results of varying the parameter BUFFER_NUM is shown in Figure 12. By increasing BUFFER_NUM, a longer period of execution of the victim accelerators will be probed and the trace can include more information. However, surprisingly, from Figure 13 (d), the classification accuracy does not change much. With BUFFER_NUM$=50$, our random forest classifier is able to reach over $90\%$. Other models have a similar trend of accuracy.

Summary. In our experimental results, we show that for a relatively wide range of parameter choices, random forest model is able to achieve satisfying classification accuracy. This helps loose the constraints on attackers’ benchmark accelerator implementation. Under ACCESS_NUM$=1000$, REPEAT_NUM$=5$, BUFFER_NUM$=100$, and BUFFER_SIZE$=4$Bytes our model is able to achieve the highest accuracy. However, in real world, under some other hardware or software settings (different FPGA models, communication link hardware, or a different heterogeneous computing software stack) these values may vary. To maximize attack performance, attackers are recommended to conduct some offline screening prior to launching the attack to obtain near-optimal parameters. This parameter search does not need to be accurate, since our most powerful model can achieve over $92\%$ accuracy performance under a relatively wide range of attack accelerator parameter choices in the selected accelerator set, which is sufficient for fingerprinting tasks. From the benchmark accelerator side, we conclude that:

1. (1)

   Our benchmark accelerator is able to capture the I/O patterns of each of the victim accelerators.

2. (2)

   Both ACCESS_NUM and REPEAT_NUM affect the granularity of measurement and can significantly influence the performance of classification models. There are optimal values for these two values, as shown in Figure 13 (a) and Figure 13 (b).

3. (3)

   Buffer related parameters BUFFER_SIZE and BUFFER_NUM have less influence on classification accuracy. However, the optimal parameter values still exist.

In our proposed attack, one prerequisite for attackers is to obtain servers and FPGAs that are identical to the servers and FPGAs used in FPGA clouds. In reality, instead of purchasing hardware and building up the system locally, it’s better for attackers to use the cloud itself for data collection. By running data collection steps in the cloud multiple times and recording the underlying hardware and software platform, the attackers can eventually have a set of models that are able to cover heterogeneous hardware and software platforms in the cloud. Doing this step on the victim cloud is more realistic and economic, considering the high cost to set up required hardware and software environments locally.

The fingerprinting attack we propose relies on the intrinsic features of victim accelerators, and we make an assumption that attackers are aware of the target accelerator and can limit the range of accelerators running on the cloud. Therefore, to defend against the proposed fingerprinting attack, FPGA cloud users should be careful about using existing public intellectual property cores (IPs) since these IPs are possibly already in the attackers’ database. To achieve this, these users can modify their accelerators and insert noisy I/O or computation operations (additional writes to an unimportant memory location, inserting additional computation between two I/O operations, etc.) to distort the performance traces the attacker may obtain.

In the meantime, it is worth noting that exploiting this security vulnerability also relies on physically residing on the same FPGA where victim accelerator is running. The simplest way for users to avoid being attacked is to obtain ownership of the whole FPGA board as well as the hosting server. This may result in higher costs in deploying FPGA accelerators (since it requires users to pay more to cloud service providers), but it completely eliminates the threat of side-channel-related attacks induced by sharing FPGA resources with unknown users.

We suggest cloud service providers enhance their infrastructure interface as mentioned in Section 6.1. Though this may add additional performance overhead, it can efficiently prevent the proposed attack.

Also, FPGA cloud service providers can consider improving their scheduling policy to scatter users’ FPGA accelerators on the cloud. It can dramatically reduce the chance of victims’ malicious accelerators co-locating with victims’ accelerators and hence mitigating side-channel attacks or fingerprinting attacks that require attackers and victims to be placed together.