Privacy-Preserving Feature Coding for Machines

To fit into the edge-cloud collaborative setting, the YOLOv5 model is split into two parts: the front-end, which is deployed on an edge device, and the back-end, which resides in the cloud. Choosing a split point is a design issue [25, 26], which depends on energy considerations, computational resources at the edge, the type of connection between the edge and the cloud, and so on. Here, our selection of the split point is mainly based on information-theoretic considerations.

Let $\mathbf{X}$ be the input image and $\mathbfcal{Y}_{i}$ be the feature tensor at the $i$-th layer. Since $\mathbf{X}\to\mathbfcal{Y}_{i}\to\mathbfcal{Y}_{i+1}$ forms a Markov chain, by the data processing inequality [22] we have that

where $I(\cdot;\cdot)$ is the mutual information. Hence, deeper layers carry less information about the input and are therefore more resilient to model inversion attacks. Also, as shown in [27], deeper layers are more compressible. These arguments would suggest choosing a split point as deep as possible.

On the other hand, limited computation and energy resources on the edge device favor selecting a shallower split point. Moreover, the YOLOv5m model branches out at layer 5, meaning that if we select the split point after layer 5, we would have to encode and transmit multiple feature tensors, which would increase both the complexity and the total bitrate. Hence, we choose to split the YOLOv5m model at layer 5.

At the split point, we insert an autoencoder, whose purpose is to both reduce the dimensionality and modify the features, such as to improve resistance to model inversion. This is a plug-and-play strategy and can be used for other models and tasks as well. The autoencoder is shown in Fig. 1: its encoder portion is referred to as AE and decoder as AD. They consist of $Conv(n,k,s)$ layers, whose structure is specified in the caption of Fig. 1, and ResBlocks, whose structure is shown in Fig. 2. The AE outputs the bottleneck feature tensor whose channel dimension ($64$) is lower than the input tensor’s channel dimension ($192$), but the spatial dimensions are unchanged. This is done to preserve the spatial precision of subsequent object detection. The resulting bottleneck feature tensor is tiled, pre-quantized to 8-bits per element, and encoded using Versatile Video Coding (VVC)-Intra [28]. At the cloud side, the encoded bitstream is decoded by a VVC decoder, then AD, then fed to the YOLOv5m back-end.

Since the goal is to create bottleneck features with improved resilience towards model inversion, we construct an auxiliary DNN model, called Reconstruction Network (RecNet), whose goal is to reconstruct the input image from bottleneck features. As depicted in Fig. 1, the architecture of RecNet is roughly a mirror of the YOLOv5m front-end and AE. We train the autoencoder and RecNet together in an adversarial manner [8]. Over the training process, RecNet tries to recover the input image from the bottleneck features as best it can, while the AE simultaneously tries to disrupt RecNet’s performance by manipulating the generated bottleneck features. At the same time, both AE and AD attempt to keep the object detection accuracy high. Note that the pre-trained YOLOv5m model is kept intact, and its weights are frozen during the entire training process. The training process for each batch of data is summarized in the following steps:

1. 1.

   The input $\mathbf{X}$ goes through the front-end, AE, and RecNet and the reconstruction loss is computed as follows:

   $$\begin{split}L_{rec}=\frac{1}{n}\|\mathbf{X}-\widehat{\mathbf{X}}\|_{1}&+\frac{\beta}{n}\|S_{x}*\mathbf{X}-S_{x}*\widehat{\mathbf{X}}\|_{1}\\ &+\frac{\beta}{n}\|S_{y}*\mathbf{X}-S_{y}*\widehat{\mathbf{X}}\|_{1},\end{split}$$

   (2)

   where $\widehat{\mathbf{X}}$ is the reconstructed input, $\|\cdot\|_{1}$ is the $\ell_{1}$-norm, $S_{x}$ and $S_{y}$ are the horizontal and vertical Sobel filters, respectively, $*$ is the convolution operator, and $n$ is the total number of tensor’s elements in the batch. The value of $\beta$ is empirically set to $5$. We adopted the Sobel filter in the reconstruction loss in order to emphasize the edges since private information is usually associated with fine detail. As a result, RecNet becomes more powerful in reconstructing the edges, which will force AE to remove edge information from the bottleneck features.

2. 2.

   Gradient of $L_{rec}$ backpropagates only through the RecNet and updates its weights. Note that the autoencoder’s weights are frozen at this step.

3. 3.

   The same batch of images goes through the whole network, and the total loss is computed as follows:

   $$L_{tot}=L_{obj}-w\cdot L_{rec},$$

   (3)

   where $L_{obj}$ is the YOLOv5 object detection loss [24], and $w$ is the balancing weight between the reconstruction and object detection. We empirically set it to $0.1$.

4. 4.

   Gradient of $L_{tot}$ backpropagates only through the autoencoder and updates its weights. RecNet’s weights are frozen in this step. Note that the negative sign of the $L_{rec}$ in (3) leads AE to make reconstruction more difficult for RecNet. Meanwhile, the positive sign of $L_{obj}$ causes AE and AD to improve object detection accuracy.

As mentioned before, RecNet is an auxiliary DNN model exploited in the adversarial training stage. So, RecNet is not part of the final pipeline. In a real situation, however, if an adversary can get hold of the edge device, they can try to train their own input reconstruction model using input-bottleneck pairs. In order to test our model against this attack, we train a new, randomly initialized RecNet on the bottleneck features generated by the autoencoder obtained in the adversarial training stage. This RecNet is trained with a $\ell_{1}$-norm loss (the first term in (2)) and is called RecNet-bottleneck. For comparison, we have also trained another RecNet, whose first three layers are removed, on the original YOLOv5m latent space (without the AE). We call this model RecNet-latent.

We test the resistance against model inversion attack by applying RecNet-latent to YOLOv5m features at layer 5 (without VVC compression) and applying RecNet-bottleneck to our bottleneck features (also without VVC compression). Input reconstruction performance is measured using conventional Peak Signal-to-Noise Ratio (PSNR), as well as a new quality metric called edge-PSNR that emphasizes PSNR near the edges. To compute the edge-PSNR, the horizontal and vertical Sobel filters are applied to the original and reconstructed images to capture the image gradients in horizontal and vertical directions. Then, the magnitude of the gradient is considered as a new image based on which the edge-PSNR is calculated. The values of PSNR and edge-PSNR are given in Table II. These results show that our AE is able to remove some information required for input reconstruction, especially near the edges. In particular, reconstruction from our bottleneck features is 1.4 dB worse than reconstruction from YOLOv5m features, and this loss is mostly concentrated near the edges since edge-PSNR is 2.5 dB lower. Some visual examples are also provided in Fig. 3. As can be seen, the edges are more distorted in the output of RecNet-bottleneck, the text is not readable, and the faces and facial expressions are not easily recognizable.

To measure the impact of feature compression, we encode the bottleneck features of the COCO validation images using VVC-Intra, specifically, its VVenC [31] implementation with lowdelay-faster preset. Prior to that, the $64$ channels in the bottleneck are clipped, quantized to $8$ bits, and tiled into an $8\times 8$ matrix to create a gray-scale image. On the decoder side, the bitstream is decoded, and the resulting tensors are passed to AD and the YOLOv5m back-end for inference.

We found that most of the feature values in the bottleneck lie in the range of $[-6,6]$. As noted in [32], feature compression performance can be improved via clipping. To this end, we tested three clipping ranges: $[-6,6]$, $[-3,3]$, and $[-1.5,1.5]$. We computed Bjøntegaard-Delta values [33] between their associated Rate-Accuracy curves (not shown due to space constraints), where object detection accuracy is measured via mean Average Precision (mAP) at the IoU threshold of 0.5. QP values $\{34,36,38,40,41,42\}$ were used to obtain these results. Using the performance for the $[-1.5,1.5]$ clipping range as the anchor, BD-Rate calculated based on the MPEG-VCM reporting template [34] shows that clipping ranges $[-3,3]$ and $[-6,6]$ reduce the bitrate by $8.1\%$ and $7.1\%$ on average, respectively. Hence, we selected $[-3,3]$ as the clipping range for further experiments.

Finally, we compare our proposed method against direct coding of YOLOv5m layer 5 features, which we call the “Anchor.” Similar to encoding the bottleneck, the $192$ channels in the YOLOv5m latent space are clipped, quantized to 8 bits, tiled into a $12\times 16$ matrix, and coded using VVC-Intra.

Fig. 4 shows the Rate-Accuracy and Quality-Accuracy curves obtained by QP $\in\{34,36,38,40,41,42\}$ for the “Proposed” and QP $\in\{39,41,42,43,44,45\}$ for the “Anchor.” BD-Rate between the curves presented in Fig. 4(a) is $-31.3\%$. Hence, our proposed method reduces the bitrate by over $30\%$ on average for the equivalent accuracy. This reduction is not surprising given the reduced dimensionality of the bottleneck features. However, the novel benefit of the proposed method is the reduction of input reconstruction ability at the same accuracy, which is depicted in Fig. 4(b). Here, BD-PSNR between the curves is $-0.76$ dB, meaning that our bottleneck features lower the input reconstruction ability by about $0.8$ dB on average. As seen earlier, by design, most of the degradation is near the edges, which is important for privacy.