A Compressive Multi-Kernel Method for Privacy-Preserving Machine Learning

Compressive Privacy [16, 17] is a privacy paradigm of which the main principle is based on lossy encoding. In Compressive Privacy, the utility and privacy are known to the system designer. Consequently, the lossy encoding scheme can be designed such that it is information-preserving with respect to the utility, but information-lossy with respect to the privacy.

Another important principle of Compressive Privacy is that the data privacy protection should be managed by the data owners. Therefore, Compressive Privacy defines two different spheres in the world of data privacy – the private sphere and the public sphere. The private sphere is where the data owners generate and manage the lossy encoding of the data, whereas the public sphere is where the data are available to the authorized users and, at the same time, to the possible adversary. Therefore, in Compressive Privacy, data need to be transformed in a privacy-lossy, utility-preserving way in the private sphere before being transferred to the public sphere.

The study of the multi-kernel method arises from the traditional need to hand-design a kernel for an application, and the idea that combining multiple kernels should be more powerful than a single kernel. This has been a problem of interest in the research community both in theoretical, and application viewpoints. Several works have investigated different ways of formulating kernel combination including using semidefinite programming [7], semi-infinite programming [23], convex combination [24, 25, 26], non-stationary combination [27], and sparse combination [28]. At the same time, several applications of the multi-kernel method have been proposed such as in image analysis [29, 30], neuroimaging [31], signal processing [32], and stock market prediction [33].

Arguably, the most common kernel combination method is the non-negative linear combination of kernels, which can be viewed as a convex combination of a finite set of kernels. This multi-kernel is, thus, in the form,

where $\mathbf{K}_{l}$ is the base kernel, $P$ is the number of kernels used, and $\mu_{l}$ is the weight corresponding to each kernel in the finite set. The design of $\mu_{l}$ has, hence, been the focus of the research in this area, where traditionally, the uniform weight has been shown to perform reasonably well [10], and more recently, the alignment-based method has been proposed to be the improvement [5]. For its wealth in past study in the literature and its simplicity, this work, therefore, follows this form of the multi-kernel method.

The tasks considered are classification problems of the utility and privacy classes. Given $M$-dimensional feature data of $N$ samples, $\mathbf{X}\in\mathbb{R}^{M\times N}$, and the corresponding utility label, $\mathbf{y}_{u}$ and privacy label $\mathbf{y}_{p}$, the goal is to design a system to predict $\mathbf{y}_{u}$ well, but not $\mathbf{y}_{p}$. The scenario considered is the mobile sensing application. The utility task is defined to be activity recognition, while the privacy concern is defined to be the identification of the data owner.

The proposed compressive multi-kernel learning system consists of two steps – the compression step and the multi-kernel step. In the compression step, the training feature data and utility label are provided and $P$ rank-reduced kernels, $\hat{\mathbf{K}}_{1},\hat{\mathbf{K}}_{2},\ldots,\hat{\mathbf{K}}_{P}$, are produced. These rank-reduced kernels are then combined in the multi-kernel step to form the final compressive multi-kernel in the form $\hat{\mathbf{K}}_{\mu}=\sum_{l=1}^{P}\mu_{l}\hat{\mathbf{K}}_{l}$. This compressive multi-kernel is provided to the kernel-based classifier.

As common in the framework of Compressive Privacy [16, 17], the private and public spheres are separated. Both compression and multi-kernel steps are assumed to occur in the private sphere, possibly by a trusted centralized server, while the classification step is considered to be in the public sphere. Therefore, the final compressive multi-kernel $\hat{\mathbf{K}}_{\mu}$ is considered public, whereas the original feature data, $\mathbf{X}$, are private. Moreover, to evaluate the system against a strong adversary, the training data, including the feature data, the utility label and the privacy label, are assumed to be public, and the adversary is assumed to have the knowledge of the procedure of the system. The testing data are, on the other hand, considered private. Figure 1(a) summarizes the system and marks the private and public spheres.

In addition, the proposed compressive multi-kernel learning system can be visualized as a neural network illustrated in Figure 1(b). The network has four hidden layers. The first hidden layer provides the kernel feature vector mapping (as described in more detail in [1]). The second and third hidden layers act as the compression step, while the the fourth hidden layer serves as the multi-kernel step. The illustration in Figure 1(b) shows the system with two kernels, as separated by the dashed line. The last layer in the network is simply the kernel-based classifier in the system that produces a prediction as the output.

The proposed system is evaluated on two datasets – Mobile Health (MHEALTH) [20, 21], and Human Activity Recognition Using Smartphones (HAR) [22]. Both datasets are related to activity recognition using mobile-sensing features, and the feature data are collected from multiple individuals, so two different labels are available – activity recognition label, and person identification label. In all experiments, activity recognition is defined as the utility classification, whereas person identification is defined as the privacy classification. Therefore, the goal of a privacy-preserving classifier is to be able to recognize the activity label with high accuracy, but identify the person with low accuracy.

The MHEALTH dataset has 23 features, and 4,800 samples are used for training, while 900 samples are left out for testing. The utility label consists of six activity classes, whereas the privacy label consists of 10 individuals. The HAR dataset has 561 features, and 5,451 samples are kept for training, while 1,080 are used for testing. The utility label has six activities, and the privacy label has 20 individuals.

In all experiments, support vector machine (SVM) is used as the classifier. The parameters of SVM are selected using cross-validation. To judge the performance of differential compressive multi-kernel methods fairly, SVM is used for all methods, and the results from single compressive kernels are also reported for comparison. Five types of kernels are used in the experiments and are defined as follows:

• •

  Linear kernel: $\mathbf{K}_{l}(\mathbf{x}_{i},\mathbf{x}_{j})=\mathbf{x}_{i}^{T}\mathbf{x}_{j}$

• •

  Polynomial kernel: $\mathbf{K}_{l}(\mathbf{x}_{i},\mathbf{x}_{j})=(\gamma\mathbf{x}_{i}^{T}\mathbf{x}_{j}+c_{0})^{p}$

• •

  Radial basis function (RBF) kernel: $\mathbf{K}_{l}(\mathbf{x}_{i},\mathbf{x}_{j})=\exp(-\gamma\left\|\mathbf{x}_{i}-\mathbf{x}_{j}\right\|^{2})$

• •

  Laplacian kernel: $\mathbf{K}_{l}(\mathbf{x}_{i},\mathbf{x}_{j})=\exp(-\gamma\left\|\mathbf{x}_{i}-\mathbf{x}_{j}\right\|_{1})$

• •

  Sigmoid kernel: $\mathbf{K}_{l}(\mathbf{x}_{i},\mathbf{x}_{j})=\tanh(\gamma\mathbf{x}_{i}^{T}\mathbf{x}_{j}+c_{0})$

For each dataset, two sets of experiments are performed. The first experiment is on multiple RBF kernels with various gamma values. The gamma values are chosen from the set of $a\times 10^{b}$, where $a=\{1,3,5\}$ and $b=\{0,1,2,3,4,5\}$, such that, in cross-validation, the single compressive kernels are sufficiently different in individual performances, and the multi-kernel performs reasonably well . The second experiment, on the other hand, uses more than one kind of kernel. Similarly, the choice of kernels is based on sufficient difference in individual performances and reasonable multi-kernel performance in the cross-validation of all above kernels.

For comparison, four methods are investigated in each experiment.

• •

  The compressive single kernel, where only one compressive kernel is used.

• •

  The compressive uniform multi-kernel, which uses the uniform weight for the value of $\mu_{l}$. Hence, $\mu_{l}=1/P$ for all kernels.

• •

  The compressive alignment-based multi-kernel, which uses the utility-target alignment maximization algorithm according to [5] on the compressive kernels to determine the value of $\mu_{l}$.

• •

  The proposed compressive SNR-based multi-kernel, which uses the SNR score described in Section (III-D) to determine the value of $\mu_{l}$. The value of the ridge term, $\rho_{snr}$, is chosen to be zero and 0.1 across all experiments.

Finally, the details of the choice of kernels in each dataset are described as follows:

The results from the evaluation confirm that the compression step indeed provides privacy protection as the privacy classification accuracy is around random guess in all experiments. This is true even when multiple compressive kernels are used. This success in privacy-preservation is achievable from the fact that large amount of information is loss during the compression step.

The compression step is actually general as other dimensionality reduction techniques can also be used. Future work can be carried out on studying the effect on the choice of the compression method. Some possible ideas for future work are the following:

• •

  Principal Component Analysis (PCA) may be used to maximize data reconstruction quality when the utility and privacy labels are not known in advance.

• •

  Desensitized RDCA method [17] may be used when the privacy label is known at the time of compression but the utility label is not.

• •

  Utility-to-privacy ratio [35, 16] maybe used when both utility and privacy labels are known.

In the current system, the multi-kernel step is primarily for utility improvement. The results from the evaluation conform with the goal. Especially, the SNR-based kernel weight score proposed is shown to be fairly effective and improving upon the state-of-the-art. Nevertheless, the current work primarily focuses on using the multi-kernel weight optimization for enhancing utility only, but as the application of privacy-preserving machine learning involves maximizing utility, while minimizing privacy, it is interesting to seek an alternative optimization criterion that fits this objective.

Fortunately, the SNR score can be extended to such setting by considering the utility-to-privacy ratio similar to that in [35, 16]. By utilizing both the utility label $\mathbf{y}_{u}$ and privacy label $\mathbf{y}_{p}$, the utility-to-privacy ratio can be defined as,

where $\mathbf{S}_{B_{U}}$ is the between-class scatter matrix derived from $\mathbf{y}_{u}$, $\mathbf{S}_{B_{P}}$ is the between-class scatter matrix derived from $\mathbf{y}_{p}$, and $\rho_{upr}$ is, again, the ridge term. Naturally, the utility-to-privacy ratio can also be derived in the empirical space:

where $\mathbf{K}_{B_{U}}$ is the counterpart of $\mathbf{S}_{B_{U}}$ and $\mathbf{K}_{B_{P}}$ is the counterpart of $\mathbf{S}_{B_{P}}$ in the empirical space.

Alternatively, the alignment approach in [6, 7] can also be extended to consider both utility and privacy together by defining the alignments between the multi-kernel matrix and the targeted utility and privacy labels, respectively, as:

Then, following the line of thought in the Differential Utility/Cost Analysis (DUCA) formulation [16], the utility-to-privacy ratio can be maximized as the following:

This optimization criterion can be formulated into a simple quadratic programming problem. First, defining the following two vectors:

Then, it can be shown that the solution $\boldsymbol{\mu}^{*}$ is given by $\boldsymbol{\mu}^{*}=\mathbf{v}^{*}/\left\|\mathbf{v}^{*}\right\|$, where $\mathbf{v}^{*}$ is the solution of the following quadratic program:

The proof of this proposition directly follows the proof of the alignment maximization algorithm in Proposition 3 of [6].

These two optimization criteria will be an interesting formulation for future study, especially for the case when compression alone is not sufficient for privacy protection.

One advantage of the multi-kernel in the form of Equation (15) is the independence among the kernels, which, as a result, allows each kernel to be designed for different purposes. This lends itself nicely to the application when there are more than one utility label. For example, different kinds of kernel can be chosen for each utility label, and the multi-kernel should then be useful for more than one utility simultaneously. Alternatively, under the compressive kernel scheme, the DCA-derived lossy projection matrix used for kernel compression can be derived from DCA with respect to different utility labels for different kernels. This would allow the multi-kernel to be applicable to more than one utility label and possible reduce conflict in the compression design.