An Encryption Method of ConvMixer Models without Performance Degradation

Many model encryption methods have been studied to be applied to adversarial defense, model protection [5, 6, 7, 8, 9] and privacy-preserving image classification [3, 10, 11, 12, 13, 14, 15]. Almost all model encryption methods are carried out by training models with images encrypted with a secret key, but the methods can degrade the performance of the models compared with that when using non-encrypted models due to the influence of encryption.

Model encryption methods have to satisfy two requirements. The first requirement is that authorized users with a secret key can obtain almost the same performance as that of non-encrypted models from encrypted models. The second is that performance of the encrypted models is not high for unauthorized users without a correct key. The proposed method aims not only to avoid the influence of encryption but to also provide an extremely degraded accuracy to unauthorized users.

ConvMixer [4] is well-known to have a high performance in image classification tasks, even though it has a small number of model parameters. ConvMixer is a type of isotropic network. It is inspired by the vision transformer (ViT) [16], so the architecture has a unique feature, called patch embedding.

Figure 1 shows the architecture of the network, which consists of two main structures: patch embedding and ConvMixer layer. First, an input image $x\in\mathbb{R}^{C\times H\times W}$ is replaced with $z_{0}$ by patch embedding with patch size $P$ and embedding dimension $d$ as

$\displaystyle z_{0}$

$\displaystyle=\mathrm{BN}(\mathrm{\sigma}\{\mathrm{Conv}_{C\rightarrow d}(x,\mathrm{kernel\_size}=P,\mathrm{stride}=P)\}).$

(1)

where $H$, $W$, and $C$ are the height, width, and the number of channels of $x$. Also, $\mathrm{Conv}_{C\rightarrow d}$ is a convolution operation with $C$ input channels and $d$ output channels, $\mathrm{BN}$ is a batch normalization operation, and $\sigma$ is an activation function. In addition, to simplify the discussion, we assume that $H$ and $W$ are divisible by $P$. Next, $z_{0}$ is transformed into $z_{l},l\in\{1,2,...,L\}$ by using $L$ ConvMixer layers. Each layer consists of depthwise convolution $(\mathrm{ConvDepthwise})$ and pointwise convolution $(\mathrm{ConvPointwise})$ as follows.

	$\displaystyle z^{\prime}_{l}$	$\displaystyle=\mathrm{BN}(\mathrm{\sigma}\{\mathrm{ConvDepthwise}(z_{l-1})\})+z_{l-1}$		(2)
	$\displaystyle z_{l}$	$\displaystyle=\mathrm{BN}(\mathrm{\sigma}\{\mathrm{ConvPointwise}(z^{\prime}_{l})\})$

Finally, the output of the $L$ th ConvMixer layer is transformed by Global Average Pooling and a softmax function to obtain a result.

In this paper, we utilize patch embedding in Eq.(1) to encrypt a model. Patch embedding can be done in two steps.

1. 1.

   Reshape an input image $x$ into a sequence of flattened $2D$ patches ${x}_{\mathrm{p}}\in\mathbb{R}^{N\times(P^{2}C)}$, where $N=HW/P^{2}$ is the number of patches.

2. 2.

   Map each patch $\boldsymbol{x}^{i}_{\mathrm{p}}\in\mathbb{R}^{P^{2}C}$ to $\boldsymbol{z}^{i}_{0}$ with dimensions of $d$ as

   	$\displaystyle\boldsymbol{z}^{i}_{0}$	$\displaystyle=\boldsymbol{x}^{i}_{\mathrm{p}}\mathbf{E}$		(3)
   	$\displaystyle\mathbf{E}$	$\displaystyle\in\mathbb{R}^{(P^{2}C)\times d}.$

A kernel in $\mathrm{Conv}_{C\rightarrow d}$ in Eq.(1) corresponds to $\mathbf{E}$ in Eq.(3). In this paper, we show that model encryption can be carried out by transforming $\mathbf{E}$ with a secret key. Also, this encryption does not degrade the performance of ConvMixer.

Figure 2 shows an overview of the proposed method, where it is assumed that the third party is trusted, and the provider is untrusted. The third party trains a model by using plain images and transforms the trained model with a secret key. The transformed model is given to the provider, and the key is sent to a client. The client prepares a transformed test image with the key and sends it to the provider. The provider applies it to the transformed model to obtain a classification result, and the result is sent back to the client. Note that the provider has neither a key nor plain images. The proposed method enables us to achieve this without any performance degradation compared with the use of plain images.

First, we address a block-wise image transformation method with a secret key to encrypt test images. As shown in Fig. 3, the procedure of the transformation consists of three steps: block segmentation, block transformation, and block integration. To transform an image $x\in[0,1]^{C\times W\times H}$, we first divide $x$ into $W_{\mathrm{b}}\times H_{\mathrm{b}}$ blocks, as in $\left\{B_{11},B_{12},...,B_{W_{\mathrm{b}}H_{\mathrm{b}}}\right\}$, where $W_{\mathrm{b}}=\frac{W}{M}$ is the number of blocks across width $W$, $H_{\mathrm{b}}=\frac{H}{M}$ is the number of blocks across height $H$, and $M$ is the block size. In this paper, we assume that the block size of the segmentation is the same as the patch size of ConvMixer. Next, each block is flattened, and it is concatenated again to obtain a block image $x_{\mathrm{b}}\in[0,1]^{W_{\mathrm{b}}\times H_{\mathrm{b}}\times p_{\mathrm{b}}}$, where $p_{\mathrm{b}}=M^{2}C$ is the number of pixels in each block. Then, $x_{\mathrm{b}}$ is transformed to ${x^{\prime}}_{\mathrm{b}}\in[0,1]^{W_{\mathrm{b}}\times H_{\mathrm{b}}\times p_{\mathrm{b}}}$ in accordance with block transformation with key $K$. Finally, ${x^{\prime}}_{\mathrm{b}}$ is transformed so that it has the same $C\times H\times W$ dimensions as those of the original image $x$, and encrypted image $x^{\prime}\in[0,1]^{C\times W\times H}$ is obtained.

In addition, the block transformation is carried out by using the three operations shown in Fig. 3. Details on each operation are given below.

In model transformation, some parameters in models trained with plain images are transformed by using a secret key. In this paper, a model transformation method is proposed to achieve the combined use of models and images transformed with the same key.

ConvMixer utilizes patch embedding (see Fig. 1), so it has the ability to adapt to the pixel order by patch embedding; patch embedding can be adapted to pixel shuffling and bit flipping because they can be expressed as an invertible linear transformation.

In the proposed method, it is assumed that the patch size $P$ used for patching is the same as the block size used for image encryption, and the number of patches is equal to that of blocks in an image. The transformation of parameters in trained models is described below.

To confirm the effectiveness of the proposed method, we evaluated the accuracy of an image classification task on the CIFAR-10 dataset (with 10 classes). CIFAR-10 consists of 60,000 color images (dimension of $3\times 32\times 32$), where 50,000 images are for training, 10,000 for testing, and each class contains 6,000 images. Images in the dataset were transformed by the proposed encryption algorithm, where the block size was $4\times 4$.

We used the PyTorch [17] implementation of ConvMixer, where the patch size was $4$, the number of channels after patch embedding was $d=256$, the kernel size of depthwise convolution was $9$, and the number of ConvMixer layers was $8$. The ConvMixer model was trained for $200$ epochs with Adam, where the learning rate was $0.001$.

First, we evaluated the proposed method in terms of the accuracy of image classification under the use of ConvMixer. Table 1 shows the classification result of ConvMixer. “Proposed” means that ConvMixer models and test images were transformed by the proposed method. As shown in Table 1, the proposed method did not degrade the performance at all for the transformed images. In contrast, the performance for plain images was degraded. Therefore, the proposed method was effective for model protection.

Next, we confirmed the performance of images encrypted with a different key from that used in model encryption. We prepared 100 random keys, and test images encrypted with the keys were input to the encrypted model. From the box plot in Fig. 5, the accuracy of the models was not high under the use of wrong keys. Accordingly, the proposed method was confirmed to be robust against a random key attack.

The use of a large key spaces enhances robustness against various attacks in general. In this experiment, the key space of pixel shuffling and bit flipping ($O_{\mathsf{p}}$ and $O_{\mathsf{b}}$) are given by $O_{\mathrm{p}}=p_{\mathrm{b}}!$ and $O_{\mathrm{b}}=\frac{p_{\mathrm{b}}!}{(p_{\mathrm{b}}/2)!\cdot(p_{\mathrm{b}}/2)!}$. Therefore, the key space of the proposed method is $O=O_{\mathrm{p}}\times O_{\mathrm{b}}\simeq 2^{543.8}$. The key space $O$ is sufficiently large, so it is difficult to find the correct key by random key estimation.