PRICURE: Privacy-Preserving Collaborative Inference in a Multi-Party Setting

A feed-forward neural network (FFNN) is a network of information processing units called neurons organized into layers. In a FFNN, information travels only forward without looping, starting from input nodes, then through hidden nodes, and finally to output nodes. The goal of the FFNN is to approximate some function $F_{\theta}$, and then map an input $x$ to a label $y$ as $y=F_{\theta}(x)$, where $\theta$ is the learned parameter vector.

The network architecture consists of the following: an input layer that captures raw inputs, hidden layers that apply a series of transformations to the input, and the output layer that produces the mapping of $x$ to $y$. For a multi-class classifier, the output layer has as many neurons as the class labels. The coefficients of connections between two neurons are referred to as weight, and are typically initialized as small random values to bootstrap the training process.

Suppose that the FFNN has $l$ number of hidden layers, and we denote the number of neurons at layer $l$ as $k_{l}$. The output layer has $o$ number of neurons. The input vector ($x_{1},x_{2},...,x_{d}$) is the $d$-dimensional input to the network. Without loss of generality, the label $y$ is computed as: $y=F(x,\theta)=\sum_{j=0}^{k}\xi_{j}\theta_{j}$, where $\theta$ are model parameters, $\xi$ is a vector of non-linear parametric basis functions where $\xi_{0}(x)=1$. Now, given $k_{1}$ number of neurons in the $1^{st}$ hidden layer, the output of $j^{th}$ neuron of the $1^{st}$ hidden layer is computed as: $r^{1}_{j}(x,W)=h(\sum_{i=1}^{d}W_{ij}(1)x_{i}+b_{j})$, where $W_{ij}(1)$ is the weight from connection between the input to connection of the $1^{st}$ hidden layer of $j^{th}$ neuron, $b_{j}$ is the bias and $h$ is the non-linear differentiable activation function. As a result, for the $1^{st}$ hidden layer, we have the output neurons $[r^{1}_{1},r^{1}_{2},....,r^{1}_{k_{1}}]$. These output neurons will then be the input vectors to the $2^{nd}$ hidden layer. In the same vein, the output of the $j^{th}$ neuron from the $l^{th}$ hidden layer is computed as: $r^{l}_{j}(x,W)=h(\sum_{i=1}^{k_{l-1}}W_{ij}(l)r^{l-1}_{i})$. Finally, the result of the $j^{th}$ neuron in the output layer is computed as: $y_{j}(x,W)=h(\sum_{i=1}^{k_{l}}W_{ij}(l+1)r^{l}_{i})$, where the coefficient $W_{ij}(l+1)$ is the weight vector from the final hidden layer $l$ to the $j^{th}$ node of the output layer. The resultant vector for all neurons of the output layer will be $[y_{1},y_{2},...,y_{o}]$. Note that, the weight vector $W(1)\in R^{(d\dots k_{1})}$, $W(2)\in R^{(K_{1}\dots k_{2})}$, …, $W(l+1)\in R^{(k_{l}\dots o)}$, are concatenated into the parameter vector $\theta=[W(1),...,W(l+1)]$.

Secure multi-party computation (SMPC) (Bayatbabolghani and Blanton, 2020) enables a set of parties $P=\{P_{1},...,P_{m}\}$, where party $P_{i}$ holds sensitive input $x_{i}$, to jointly compute a function $y=f(x_{1},...,x_{m})$ while protecting each $x_{i}$. The computation needs to result in the correct value of $y$ (called the correctness property) and at the end of the computation each $P_{i}$ learns nothing beyond $y$ (called the privacy property). SMPC has several applications, for example in: privacy-preserving decision making on distributed data (Attema et al., 2018), privacy-preserving machine learning (Gilad-Bachrach et al., 2016), secure auctions (Joan Feigenbaum and Saint-Jean, 2004), and secure voting (Nair et al., 2015).

The two popular implementations of SMPC are garbled circuit (Yao, 1986) and secret sharing (Ben-Or et al., 1988). In garbled circuit, the $P_{i}$’s construct a (large, encrypted) circuit and evaluate it at once, while in secret sharing they need to interact for each circuit gate. Garbled circuit allows for constant number of rounds but requires larger bandwidth (i.e., fewer messages, but bigger messages are sent). Secret sharing has typically low bandwidth (i.e., small messages per gate) and high throughput, where the number of rounds is determined by the depth of the circuit. In this work, we use secret-sharing-based MPC. Intuitively, a $(t,n)$-secret sharing scheme splits a secret $s$ into $n$ shares $s_{i}$ and at least $t$ shares are required to reconstruct the secret. We use $\langle s\rangle=(s_{1},...,s_{n})$ to indicate the sharing of $s$ among $n$ parties.

Additive secret sharing (Shamir, 1979) allows a secret $s$ to be split into random parts and shares them with secure workers (real/virtual instances that perform computations such as addition and multiplication securely). For example, suppose there are two workers $A$ and $B$ and a secret $s$. The above notation then becomes $\langle s\rangle=(s_{1},s_{2})$ because $n=2$. Accordingly, $A$ and $B$ receive share values $s_{1}^{a}$ and $s_{2}^{b}$, respectively. The workers $A$ and $B$ perform computation (e.g., compute function $F$) directly on the share values. After finishing the computation, the workers produce intermediate results as follows:

These intermediate results $y_{a}$ and $y_{b}$ are then combined using private additive scheme and then the true output result is revealed. This secret sharing operation does not use floating point numbers, rather performed in a mathematical space called Integer Quotient Ring (Jacobson, 1957), which contains the integer between 0 to Q-1. Here Q is a prime number that has to be big enough so that the space is able to contain all the numbers that would be used in our experiments. A conceptual proof that illustrates additive secret sharing is described in the Appendix (Section 8.2).

Our implementation of pricure builds on private additive sharing introduced in SecureNN (Wagh et al., 2019) and incorporated in PySyft (OpenMined, 2020). SecureNN builds on MPC to implement exact non-linear functions while avoiding the use of inter-conversion protocols as well as general-purpose number-theoretic libraries. The mathematical operations like matrix multiplication, summation, private comparison, division, and max-pooling operations are built based the MPC property. PySyft implements SPDZ (Damgård et al., 2012), a secret sharing scheme that enables more complex operations. In the Appendix (Section 8.3), we provide an illustrative proof on how SPDZ functions.

For two neighboring datasets $D_{i}$ and $D_{j}$, which differ in just one data point $x$, suppose $F_{i}$ is trained on $D_{i}$ and $F_{j}$ is trained on $D_{j}$. Let the output space of $F_{i}$ and $F_{j}$ be $S$ such that for an input $x$, $F_{i}(x)\in S$ and $F_{j}(x)\in S$. Differential privacy (DP) (Dwork et al., 2006) guarantees that a randomized mechanism $M(F_{i}(x))$ and $M(F_{j}(x))$ does not enable an observer (adversary) to distinguish whether $M$’s output was based on $D_{i}$ or $D_{j}$, i.e., whether or not $x$ is used as a training example for $F_{i}(x)$ or $F_{j}(x)$. The indistinguishability of $x$’s membership in $D_{i}$ or $D_{j}$ protects the identifiability of $x$ (e.g., a person’s medical record). In $\epsilon$-DP, the indistinguishability of the outputs of $F_{i}(x)$ and $F_{j}(x)$ is parametrized by $\epsilon$ (also called the privacy budget). Equation  3 formalizes the notion of ($\epsilon$)-DP as follows:

Intuitively, lower $\epsilon$ values indicate stronger privacy protection. Equation 3 has the $(\epsilon,\delta)$ variant: $P[M(F_{i}(x))\in S]\leq e^{\epsilon}\times P[M(F_{j}(x))\in S]+\delta$, where $\delta$ refers to the failure probability of the mechanism M, and when $\delta=0$, we say $M$ is $\epsilon$-DP. The $\epsilon$-DP formalism in Equation 3 guarantees individual data item privacy in the most extreme case where $D_{i}$ and $D_{j}$ are so similar that only one data point sets them apart. In pricure, each $D_{i}$ is unique since the datasets come from independent data owners. Hence, using the DP notion, pricure guarantees that individuals who contribute privacy-sensitive data have bounded privacy guarantee.

In DP, randomization is at the core of ensuring the indistinguishability of an individual’s record in a dataset. A typical way to achieve output randomization is to add noise to an output (e.g., $F_{i}(x)$) using, for instance, the Laplace mechanism. For a privacy budget $\epsilon$ and $F_{i}$’s sensitivity value of $s$, using the Laplace mechanism $Lap(b)$ centered at $0$ and scale $b$, $noise$ is computed as $Lap(\frac{s}{\epsilon})$.

Problem: We consider a setting where multiple model owners $P=\{P_{1},…,P_{m}\}$ hold private models $F_{1},…,F_{m}$ such that $F_{i}$ is trained on $P_{i}$’s private data, and client $C$ submits their private input $x=[x_{1},…,x_{d}]$ to obtain an inference result: $y=\mathbf{\Phi}_{i:1...m}(F_{i}(x))$, where $\mathbf{\Phi}$ is an aggregation function that leverages the collective inference capability of $F_{i}$’s. Taking a feed-forward neural network $F_{i}$ with number of layers $l$, the collaborative inference result can be expanded in terms of weights and biases as:
$y=\mathbf{\Phi}_{i:1...m}(W_{l}\cdot F^{i}_{l-1}(...F^{i}_{1}(W_{1}\cdot x+b_{1})...)+b_{l})$.

Goals: Our first goal is to obtain $y$ while keeping $F_{i}$’s and $x$ private. In particular, after each prediction, $C$ learns nothing about $F_{i}$’s (i.e., $W_{1},W_{2},...,W_{l}$ and $b_{1},b_{2},...,b_{l}$) and $P_{i}$’s learn nothing about $x$ and other model owners. Our second goal is to deter a semi-honest client $C$ from mounting membership inference attacks.

Challenges: To achieve the aforementioned goals, one needs to address specific research questions with regards to a) accuracy/privacy trade-off, b) scalability with growing number of model owners, c) the prospect of attacks such as membership inference even in the presence of privacy-preserving schemes, and d) performance implications of privacy-enhancing schemes in a multi-party setting. We address these questions in Sections 5.2 - 5.5.

Here, we describe our threat model with respect to model owners, workers, aggregator, and client.

Model Owners: We consider the semi-honest setting for model owners, whereby they do not trust each other to share their training data and/or models, because of data protection and privacy regulations (e.g., HIPAA) or competitive advantage (e.g., they compete in the same business).

Client: The client is assumed to be semi-honest for it may use pricure as an oracle to initiate membership inference (Shokri et al., 2017), model extraction (Tramèr et al., 2016), or model inversion attacks (Fredrikson et al., 2015). Moreover, it does not have access to model parameters of any of the model owners.

Workers: In the honest-but-curious sense, they may analyze the secret-shares they receive from model owners or intermediate computation results, but are trusted enough not to collude with each other to exchange their respective secret-shares or intermediate computation results. Given the partial result they compute, workers are assumed to have no access to the final inference result.

Aggregator: We assume the aggregator is trusted by model owners and clients not to reveal true inference results to any third party, and will not mount model extraction/membership inference style attacks. Moreover, it doesn’t have access to model parameters of the model owners and the data of the client. When sending the prediction results to the client, the aggregator encrypts the result with the client’s public key (shared with the aggregator a priori).

Figure 1 highlights our proposed system, pricure. We consider a setting of $m$ model owners (e.g., hospitals) who may not share training data and models due to regulatory or trust reasons, yet they want to participate in a collaborative inference task $T$ (e.g., medical image classification). As a result, model owners train $m$ private models $F_{1}$, …, $F_{m}$. pricure combines secure multi-party computation (SMPC) and differential privacy (DP) to enable collaborative inference among the $m$ model owners on task $T$, while $(a)$ ensuring secrecy of each $F_{i}$ and samples submitted for inference from client $C$ and $(b)$ limiting adversarial advances of a semi-honest client to mount membership inference attack.

Why Combine SMPC and DP? It is because these two schemes are complementary to each other, and they provide orthogonal protections to address $(a)$ and $(b)$. Given an input $x$ and a function $f(x)$, SMPC is aimed at answering “what information about $x$ is leaked in the course of computing $f(x)$?”, which addresses $(a)$. DP, on the other hand, is concerned with “what can be inferred by analyzing $f(x)$?”, which addresses $(b)$. In the sense of the collaborative inference we consider in pricure, SMPC is relevant for pre-inference protection of data while DP is aimed at post-inference protection to avoid attacks such as membership inference.

After training a model on a private dataset, each model owner secret-shares its model parameters with non-colluding secure servers, which we call workers. A client $C$ (e.g., another hospital, a medical researcher) who wants to obtain an inference result (e.g., classification) on sample $x$ also secret-shares the sample with the workers. A single iteration of collaborative inference proceeds as follows: when the workers receive a secret-share of an input $x$, they use each model’s partial inference function to compute and return a partial result to a trusted aggregator, which reconstructs each model’s partial inference results into full inference output for sample $x$. The aggregator then performs confidence-weighted aggregation of the inference results, and finally leverages differential privacy to add random noise to the final inference result and sends it to the client.

The secrecy of client’s inputs and model parameters of each owner is protected using additive secret sharing. Noisy inference via $\epsilon$-DP is used to deter a semi-honest client who might attempt to mount membership inference attacks. The communication link between the client and the aggregator is a secure channel. The client shares its public key with the aggregator, that it uses to encrypt an inference result so the client decrypts it with its private key to obtain the final output of the collaborative inference for sample $x$.

Each model owner maintains its private training data from which it trains a private model, and does not share both training data and model details with other model owners, the aggregator, or a third party. To enable collaborative inference that benefits from the collective intelligence of what is learned by each $F_{i}$, model owners are, in principle, free to use any model architecture, but need to agree a priori on a common feature representation (e.g., pixel intensities for images) and the inference output format (e.g., label only, probability score). A model owner $P_{i}$ may train its respective model $F_{i}$, with or without privacy. We note that when a model is trained in a privacy-preserving manner (e.g., via gradient perturbation as in differentially private stochastic gradient descent (DP-SGD) (Abadi et al., 2016) or using output perturbation (Jayaraman et al., 2018)), the privacy budget will add up with the inference output perturbation pricure introduces at the aggregator, which may incur more penalty on accuracy.

Number of Model Owners: The minimum value for $m$ is 2 while there is no limit on the maximum. Intuitively, the higher the value of $m$, the better the collective inference accuracy of the participants. This, however, depends on the accuracy of each model and the privacy budget (amount of noise added on the final inference output). Our model owner setup shares some similarity with PATE (Papernot et al., 2017). However, our setting differs both in the goals and the details. For instance, in PATE, the larger $m$ implies potentially better privacy guarantee. In our case, the interpretation of how large $m$ may not inherently entail better privacy guarantee, since each model owner is acting on its own training data (unlike PATE, where training data is partitioned into $m$ disjoint sets). The disjoint $m$ partitions in PATE is equivalent to the typically disjoint $m$ model owners in pricure. Model owners in pricure may use varying size of training data and the choice of $m$ may have implications on the accuracy/privacy trade-off (in PATE, depending on the dataset size, the choice of $m$ may affect accuracy/privacy trade-off). In Section 5.3, we will analyze the relationship between number of model owners, inference accuracy, and privacy guarantee. Next, we describe how model owners secret-share model parameters with workers.

Secret-Sharing of Model Parameters: Model owners secret-share their model parameters with the virtual workers via a secure channel. Once they secret-share their model parameters, model owners are not required to stay connected with the workers. Instead, they can terminate the secure connection and initiate it later as needed (for instance when a model is retrained and updating the model parameters is deemed necessary).

Once a model owner trains a model, the learned model parameters are captured via weights ($W$) and bias ($b$). Suppose for model owner $P_{i}$ ($P\in[1,m]$), the weight vectors are $\theta=[W(1),....,W(l+1)]$, where, the weight vector $W(1)$ represents the weight parameters from input to the $1^{st}$ hidden layer, similarly $W(2)$ represents the weight parameters from $1^{st}$ hidden layer to the $2^{nd}$ hidden layer and $W(l+1)$ represents the weight parameters from $l^{th}$ hidden layer to the output layer. Using Equation 4 and 5, model owner $P_{i}$ secret-shares its $j^{th}$ model parameters with worker $A$ and $B$, respectively as follows:

Finally, all secret-shared model parameters are computed as follows:

The secret-share is performed over a secure channel, and once the share is saved by the two workers, the model owner can go offline.

When the client wants to obtain inference result on input $x$, it connects to workers $A$ and $B$ and secret-shares its private input. Moreover, it generates a private-public key pair and sends the public key to the aggregator so that when the inference result is ready, before sending it the aggregator encrypts it with the client’s public key. When it receives an encrypted label from the aggregator, the client performs a decryption operation using its private key and produces the clear-text label for input $x$. In Section 4.5, we will describe how the aggregator produces the final inference result. Next, we describe how the client secret-shares private input $x$.

Secret-Sharing of Client Input: For simplicity, consider $x_{1}$ is the first element of the client’s input feature vector $x=[x_{1},...,x_{d}]$. $x_{1}$ is divided into two parts to be secret-shared with two workers $A$ and $B$, using Equations 4 and 5:

x${}_{1}^{a}$ and x${}_{1}^{b}$ are large numbers that fall in the range $[0,Q-1]$, and do not individually reveal the real data. $x_{1}$ is obtained by combining $x_{1}^{a}$ and $x_{1}^{b}$ using Equation 6 below:

Extending the secret-share mechanism from $x_{1}$ to the whole feature vector $x$ of dimension $d$, the secret-share of input $x$ will be done by extending Equations 4 and 5 as follows:

Note that for $i\in[1,d]$ each $x_{i}^{a}$ and $x_{i}^{b}$ is respectively computed using Equation 4 and 5.

Next, let us assume we have our network of neurons with total $l$ number of hidden layers, $d$ be the dimension of input $x$ to be secret-shared with worker $A$ and worker $B$. The output results for the $j^{th}$ neuron of the $1^{st}$ hidden layer for worker $A$ and worker $B$ are computed as follows:

Here, $W^{a}_{ij}(1)$ and $W^{b}_{ij}(1)$ are the weight vectors from input to the $j^{th}$ neuron of the $1^{th}$ hidden layer. For the $1^{st}$ hidden layer, we have the output neurons $[r^{a,1}_{1},r^{a,1}_{2},....,r^{a,1}_{k_{1}}]$ for worker $A$ and $[r^{b,1}_{1},r^{b,1}_{2},....,r^{b,1}_{k_{1}}]$ for worker $B$. Note that $k_{1}$ is the total number of neurons in the $1^{st}$ hidden layer. These output neurons will make the input vector for the $2^{nd}$ hidden layer. Accordingly, the output of the $j^{th}$ neuron of the $2^{nd}$ hidden layer is obtained as:

Similarly, the output of the $j^{th}$ neuron of the $l^{th}$ hidden layer is computed as follows:

Here, $W^{a}_{ij}(l)$ and $W^{b}_{ij}(l)$ are the weight vectors from the $(l-1)^{th}$ hidden layer to the $j^{th}$ neuron of the $l^{th}$ hidden layer.

Each worker uses the secret-shared private input of the client and private model parameters of each model owner to produce intermediate results, which, when combined, produce the true inference output. Finally, the intermediate results for worker $A$ and worker $B$ for the $j^{th}$ output neuron are computed as:

The coefficients $W^{a}_{ij}(l+1)$ and $W^{b}_{ij}(l+1)$ are the weight vectors from the final hidden layer $l$ to the $j^{th}$ neuron of the output layer for worker $A$ and worker $B$, respectively.

Finally, the resultant intermediate vector for the output layer of model owner $P_{i}$ for $o$ number of neurons is $y_{P_{i}}^{a}=[y^{a}_{P_{i},1},y^{a}_{P_{i},2},....,y^{a}_{P_{i},o}]$ for worker $A$ and $y_{P_{i}}^{b}=[y^{b}_{P_{i},1},y^{b}_{P_{i},2},....,y^{b}_{P_{i},o}]$ for worker $B$, respectively. These final vectors $y_{P_{i}}^{a}$ and $y_{P_{i}}^{b}$ are sent to the aggregator via a secure communication channel.

Reconstruction of Inference Results: When it receives intermediate results $y_{a}$ and $y_{b}$ from the workers, the aggregator first combines $y_{a}$ and $y_{b}$ to obtain the true inference result $y$. Across each $y$ that corresponds to a model owner, the aggregator then performs majority vote-based aggregation of inference results. The aggregator then adds random noise sampled from the Laplace distribution to perturb the inference output (and hence deter membership inference style attacks). Lastly, the aggregator encrypts the inference output with client’s public key before sending it to the client.

In general, for each model owner $P_{i}$, the aggregator receives $y_{P_{i}}^{a}=[y^{a}_{P_{i},1},y^{a}_{P_{i},2},..,y^{a}_{P_{i},o}]$ and $y_{P_{i}}^{b}=[y^{b}_{P_{i},1},y^{b}_{P_{i},2},..,y^{b}_{P_{i},o}]$ from worker A and B respectively. To get the true inference value from the intermediate results, the aggregator combines them as follows:

As a result, the constructed inference result for model owner $P_{i}$ is $y_{P_{i}}=[y_{P_{i},1},y_{P_{i},2},....,y_{P_{i},o}]$ for $o$ number of output neurons. Hence, the output for $m$ model owners is represented as a vector of final inference results: $[y_{P_{1}},y_{P_{2}},....,y_{P_{m}}]$.

Noisy Aggregation of Inference Results: The aggregator aggregates all the inference results for $m$ model owners as follows:

In Equation 12 above, $y_{noise}$ is the noisy aggregated output for $m$ number of model owners, and $\Phi$ is the aggregation function (e.g., majority vote, confidence-weighted sum of inference probabilities). Here, $y_{noise}=[y_{1},y_{2},...,y_{o}]$ is an $o$-dimensional vector with a total of $o$ number of noisy aggregated output inferences. The final label is produced using the $argmax$ operation as $y=argmax([y_{1},y_{2},...,y_{o}])$, where $y$ is the output label for input vector $x$. The aggregator encrypts this output with the client’s public key and sends it to the client over a secure channel.

We use four datasets that focus on: handwritten digit recognition (MNIST (LeCun et al., 2020)), clothing image classification (Fashion-MNIST (Xiao et al., 2017)), breast cancer detection (IDC (Kaggle, 2020)), and in-ICU length-of-stay prediction for patients (MIMIC (Johnson et al., 2016)). We chose these datasets because they were used as benchmarks in related work (Papernot et al., 2017; Mohassel and Zhang, 2017; Riazi et al., 2018; Gilad-Bachrach et al., 2016) and also are relevant for privacy-sensitive domains (e.g., IDC and MIMIC are medical datasets). Next, we first briefly describe each dataset and then discuss model architectures and training setup.

MNIST is a collection of 70K gray-scale images of handwritten digits with training set of 60K images and a test set of 10K images. Each sample has a width-height dimension of 28x28 pixels of handwritten digits 0 to 9, which make up the 10 classes. We divide the 60K training samples into multiple chunks to train multiple models. For instance, for 50 model owners, we divide the training set into 50 disjoint datasets so that each model owner has 1.2K samples to train their model on. We use 500 samples from the test set for the evaluation of pricure.

Fashion-MNIST is a dataset from Zalando’s article images with 60K training set and 10K test set. Each example is a 28x28 gray-scale image, associated with a label from 10 classes. The labels are for T-shirt/top, trouser, dresses, etc. For the highest number of model owners of 40, we divide the training set into 40 disjoint datasets so each model owner has 1.5K samples to train their model. We use 500 samples from the test set to evaluate pricure.

IDC is the Invasive Ductal Carcinoma (IDC) dataset with 277,524 patches of size 50×50 extracted from 162 whole mount slide images of breast cancer specimens scanned at 40x. Out of the 277,524 samples, 198,738 test is negative (benign) and 78,786 test positive with IDC. Using under-sampling to balance positive and negative samples, we use 277,024 samples for training a binary classifier and 500 samples as testing samples for pricure.

MIMIC is the Medical Information Mart for Intensive Care (MIMIC) dataset with 60K samples with details such as demographics, vital signs, laboratory tests, medications and personal information about patients. The classifier predicts the length of stay (LOS) of a patient in the hospital. We use a multi-class classifier where LOS is divided into four classes: class 0 (0-4 days), class 1 (4-8 days), class 2 (8-12 days), and class 3 (12 or more days). Our training dataset contains 58, 386 samples and test set has 590 samples.

Model Architectures and Setup: For all the four datasets, we train a FFNN model. For MNIST and Fashion-MNIST, the input layer is $28\times 28$ and the output layer has $10$ neurons, with two hidden layers with ReLU activation. The output layer is linear and we use Mean Square Error loss instead of the negative log-likelihood function. Since log-likelihood is performed using softmax function, it requires computing logarithm and exponential functions which are not practical for pricure because we convert all real number parameters to integers. Since our computations are over a finite integer field, we convert all the floating point tensors into fixed precision tensors with a rounding at the second decimal digit (e.g., .456 to 45). For MIMIC, we use one hidden layer with $30$ input neurons and $4$ output neurons. For IDC, we use the same FFNN model where the number of input neurons is $7500$ and the number of output neurons is 2. Model architecture details are described in the Appendix (Section 8.1). The learning rate is 0.001 across the four datasets. The number of epochs for MNIST and Fashion-MNIST is $100$, while we used $80$ epochs for MIMIC and $200$ epochs for IDC. Finally, for each dataset, each model owner secret-shares their respective model parameters with workers A and B.

We first analyze the inference accuracy of pricure with respect to privacy budget $\epsilon$. Smaller $\epsilon$ values imply stronger privacy guarantees (e.g., against membership inference) and vice-versa. The ideal/optimal case is when we achieve high inference accuracy while providing strong privacy (smallest $\epsilon$ value), but striking the sweet spot that fulfills both is generally non-trivial for the trade-off depends on training data, number of model owners, and aggregation scheme used to decide the final inference result. In Figures 5–5, $x$-axis shows privacy budget $\epsilon$ and $y$-axis is inference accuracy of pricure. Across the four datasets, the smallest number of model owners is $m=10$, while the maximum is in the range $40-100$ (e.g., $40$ for Fashion-MNIST, $50$ for MNIST, and $100$ for IDC), depending on the size of the training set and inference accuracy.

On MNIST and Fashion-MNIST Benchmarks: Figure 5 shows inference accuracy with respect to privacy budget $\epsilon$ (in the range [10^-2, 1]) for pricure. Note that we have divided the $60$K MNIST training set into range of 10, 25, and 50 disjoint datasets with $6K$, $2.4K$, and $1.2K$ per-model owner samples, respectively. The client has $500$ input samples as the inference set. For MNIST, when number of model owners $m=50$, the inference accuracy is $55.53\%$ for $\epsilon=0.03$. Compared to the non-private $92.4\%$ inference accuracy for MNIST, we observe that pricure incurs significant accuracy loss for the privacy gain it provides in exchange. Interestingly, for $\epsilon=0.05$, the inference accuracy for pricure is $92.6\%$ (i.e, no accuracy loss), and the inference accuracy remains fairly stable for $\epsilon\in[0.5,1]$ (see Figure 5). This observation is consistent with prior work on ensemble of differentially private teacher models by Papernot et. al. (Papernot et al., 2017, 2018b). The best trade-off is with $\epsilon=0.05$ and inference accuracy=92.6% when $m=50$.

Figure 5 shows accuracy/privacy trade-off analysis for Fashion-MNIST for 10, 20, and 40 model owners. For $m=40$, the inference accuracy is 82.8% with $\epsilon=0.1$. For $\epsilon<0.1$, the inference accuracy drops dramatically. For example, when $\epsilon=0.05$, inference accuracy is $74.6\%$ which is smaller than our best case accuracy on Fashion-MNIST of $82.8\%$ when $m=40$. Note that non-private inference accuracy is $82.8\%$, which, similar to MNIST, suggests that there is no accuracy loss. The best trade-off is obtained with $\epsilon=0.1$ and inference accuracy = 82.8% for $m=40$.

On IDC and MIMIC: Figure 5 shows accuracy/privacy trade-off analysis for IDC. When $m=100$, inference accuracy is 71.6% for $\epsilon$=0.05, which is the optimal case in terms of accuracy/privacy trade-off. Hence, pricure offers an acceptable inference accuracy for IDC with smaller privacy budget, which offers a strong privacy guarantee. For $\epsilon<0.05$, the inference accuracy is lower, for example, for $\epsilon=0.03$, inference accuracy is 71.6% with $m=100$, which suggests that the acceptable noise range for IDC is $\epsilon\in[0.05,1]$. Figure 5 shows accuracy/privacy trade-off for MIMIC. The best case inference accuracy is 76.6% with $\epsilon=0.05$ for $m=40$. For $\epsilon<0.05$, the inference accuracy is much lower (e.g., $34\%$ for $\epsilon=0.01$).

Comparison with Closely Related Work: Papernot et. al.(Papernot et al., 2017) show with a CNN model on MNIST, their non-private model is 99.18% accurate while the private ensemble of 250 teacher models is 93.18% accurate. In our case, the non-private FFNN model on MNIST is 96.6% accurate, while its private counterpart of 50 model owners is 92.6% with $\epsilon=0.05$. We note that initial differences in the non-private accuracy of the models could be attributed to the different in model architectures, i.e., CNN (theirs) vs. FFNN (ours).

$\bigstar$ Take-away: With respect to RQ1, our analysis of the accuracy/privacy trade-off suggests that pricure provides acceptable privacy guarantees with very minimal trade-off on inference accuracy, showing its practical viability in a multi-party setting.

As we indicated in Section 4.2, intuitively, a higher value for number of model owners $m$ would potentially entail better collective inference accuracy. However, this depends on the accuracy of each model and the privacy budget. With that background, here, we analyze the impact of $m$ on inference accuracy and privacy budget.

On MNIST: From Figure 5, as $m$ increases from 10 to 25 and then 50, we notice a roughly consistent trend, especially for $\epsilon>0.03$. With progressive increase in $\epsilon$ from $\epsilon=0.05$ to $\epsilon=1$, inference accuracy seems to be inversely proportional to increase in $m$. Keeping the same trend for $\epsilon\in[0.01,1]$, the highest inference accuracy values are achieved for the lowest number of model owners ($m=10$), and the lowest inference accuracy values correspond to the highest number of model owners ($m=50$). In particular, for $m=50$, the inference accuracy is 92.4% (with $\epsilon=0.05$). On the other hand, inference accuracy jumps to 94.52% (with $\epsilon=0.3$) and then 96.56% (with $\epsilon=0.5$) for $m=25$ and $m=10$, respectively.

Overall, from these observations we synthesize that higher number of model owners (e.g., $m=50$) result in lower inference accuracy as opposed to lower number of model owners (e.g., $m=25$, $m=10$). It is, however, noteworthy that higher inference accuracy for lower number of model owners offer relatively lower privacy guarantees. For instance, with $m=25$ and $m=10$, inference accuracy of 94.52% and 96.56% are achieved with $\epsilon=0.3$ and $\epsilon=0.5$, respectively.

Fashion-MNIST, IDC, and MIMIC: We observe similar results for Fashion-MNIST and IDC from Figure 5 and Figure 5. For Fashion-MNIST, the best case of inference accuracy is 82.8% for $\epsilon=0.03$, 84.2% for $\epsilon=0.3$, and 84.39% for $\epsilon=0.5$, respectively, for 40, 20, and 10 model owners. On the other hand, for IDC, the best case of inference accuracy is 71.6% for $\epsilon=0.05$, 72.2% for $\epsilon=0.05$, and 73.4% for $\epsilon=0.3$, respectively, for 100, 50, and 10 model owners. As a result, for Fashion-MNIST and IDC dataset, we observe that pricure achieves better inference accuracy for smaller number of model owners, but with a caveat of relatively lower privacy guarantee. For MIMIC, from Figure 5 we observe that for 40 model owners, the best case for inference accuracy is 76.6% with $\epsilon=0.05$, while for 10 model owners, the best case for inference accuracy is 80.53% with $\epsilon=0.3$ which also supports our claims.

$\bigstar$ Take-away: With respect to RQ2, we note that if the number of model owners $m$ is higher, it allows a larger noise level (i.e., offers stronger privacy guarantee), while for smaller $m$ values the inference accuracy is more sensitive to noise. Our findings here are inline with PATE (Papernot et al., 2017) with respect to larger number of models resulting in lower privacy budget on MNIST.

We recall that one of the goals of pricure is to deter a semi-honest client who may mount membership inference attack (MIA). To that end, we perform DP-aggregation of inference results before we release each inference result to the client. The goal of MIA is to identify (with some confidence) whether a given sample belongs to a training set of a target model (Shokri et al., 2017). To examine the utility of DP in limiting MIA, we measure the accuracy, precision and recall of MIA for both noiseless and noisy aggregation of inference results. Accuracy measures the percentage of examples that are correctly predicted to be members of the target model’s training dataset. Precision measures the proportion of true membership inference with respect to all reported attacks, while recall measures the coverage of the attack, as the fraction of the training records that the attacker can correctly deduce as members of the training set. For the purpose of this evaluation, we reproduce the MIA by Shokri et. al. (Shokri et al., 2017), and we use the same size of member and non-member data to maximize the inference uncertainty, with baseline accuracy of $50\%$. We use Fashion-MNIST and MIMIC to examine the effect of differential privacy on MIA.

MIA Against Fashion-MNIST Model: We use 5,000 samples to train the target model. There are $20$ shadow models and their training size is set to $5,000$. Training dataset of shadow models are disjoint with the training dataset of the target model. The shadow models are expected to mimic the behavior of the target model as the target model and the shadow models are all trained on data coming from the same population. As a test dataset, we use the samples with equal number of members and non-members.

Figure 7 shows per-class accuracy of MIA for both private and non-private model on Fashion-MNIST. Without differentially private prediction, the maximum, minimum, and average MIA accuracy, respectively, is $69.3\%$ (for class-4), $46\%$ (for class-9), and $57.13\%$ (for all 10 classes). On the other hand, when we add Laplacian noise with $\epsilon=0.1$ post-aggregation on each inference result, average MIA accuracy is $51.17\%$ with maximum of $54.1\%$ for class-4 and minimum accuracy of $45.3\%$ for class-9. The attack accuracy degradation (by 5.96% to be exact) implies that due to the presence of Laplacian noise, the model owner’s training data are less vulnerable to MIA. We also observe that, adding more noise gradually decreases the MIA accuracy, thus mitigating training data exposure.

The average precision value for all class without DP is $54.22\%$, suggesting that for the non-private model, $54.22\%$ of the images that were inferred as members by the attacker are true members. On the other hand, average recall score for the non-private model for all classes is $79.02\%$, implying that the attacker can correctly infer an averages of $79.02\%$ of the training images as members. On the other hand, precision and recall with privacy budget = $0.1$ are $51.2\%$ and $46.24\%$, respectively. The lower precision (by 3.02%) and lower recall (by 32.78%) for the private model suggest that $\epsilon$-DP mitigates the membership inference attack.

MIA Against MIMIC Model: We use $5,000$ samples for both the target model and shadow model, with $20$ shadow models. We use the same number of members and non-members for evaluation. Figure 7 shows per-class accuracy of MIA for both private and non-private model on MIMIC. In a non-private setting, the maximum MIA accuracy is $69\%$ for class-2 and minimum MIA accuracy is $53\%$ for class-0. The average MIA accuracy is $61.18\%$ for all four classes, suggesting $61.18\%$ examples of the target model’s training data are correctly predicted to be members. With the privacy budget $\epsilon=0.1$, the maximum MIA accuracy is $52.3\%$ for class-1 and minimum MIA accuracy is $49\%$ for class-3. For these settings, the average accuracy of MIA is $52.1\%$ across all four classes. The average accuracy degradation is $9.08\%$, which suggests that $\epsilon$-DP mitigates MIA by $9.08\%$. Looking at precision and recall for the non-private model, we obtain $56.47\%$ and $96.2\%$, respectively. On the other hand, precision and recall with privacy budget = $0.1$ are $50.95\%$ and $51.48\%$, respectively. The lower precision (by 5.52%) and lower recall (by 44.72%) for the private model suggest that $\epsilon$-DP mitigates the attacker’s pursuit of members’ records.

Comparison with Closely Related Work: While PATE (Papernot et al., 2017) discusses intuitive guarantee against MIA, they did not provide quantitative measurement of MIA based on noise scale. In (Rahman et al., 2018), the effect of DP-based training against MIA is examined. On MNIST, with $\epsilon=2$ and $\epsilon=1$, they show that DP can eventually mitigate MIA accuracy and bring it close to baseline. Unlike (Rahman et al., 2018), in pricure we use $\epsilon$-DP after aggregating the inference results of model owners, with sizable MIA mitigation within our noise boundary.

$\bigstar$ Take-away: With respect to RQ3, differentially private aggregation limits membership inference attack in a way that both the inference about members and the coverage of attack are minimized.

We evaluate the overall, per-sample, and per-model performance overhead incurred by pricure. In particular, we measure time taken by a) a model owner to secret-share their model parameters with the workers and b) per-sample overhead for inference.

Hardware Specification: To run our experiments, we used Google Colab virtual machine which provides 2 Intel(R) Xeon(R) model 79, family 6 CPUs with 2.2GHz each, with 12.72GB RAM and 107.77GB disk space. To speed-up computations, we used Python’s multi-threading features to efficiently utilize the two CPUs at a time, by dividing the task between them and get output inferences from two model owners in one cycle.

Per-model Overhead for Secret-Sharing: Figure 9 shows the duration of secret sharing of model parameter across the four datasets, and for different number of model owners. This performance is influenced by the number of model parameters ($W$ and $b$). The $x$-axis shows the different datasets and the $y$-axis shows the time needed, in milliseconds, to share a single model owner’s model parameters with the two workers. Overall, the per-model secret-sharing overhead is insignificant. Comparatively, IDC seems to incur more delay (up to $392.4$ms) of model parameter secret sharing because it produces vast model parameters compared to others. The other data sets need much less time in the range 8ms-48ms to share their model parameters, which is insignificant.

Per-sample Overhead for Inference: In Figure 9, the $y$-axis shows overall duration (in $log_{10}$ scale) to produce inference result for an input with SMPC-DP (pricure) and without SMPC-DP (non-private). This performance is affected by number of model owners $m$, number of classes, and feature vector dimension of the input. From Figure 9, across the board pricure incurs a significant overhead to produce the inference result. For example, on MNIST, pricure requires 772.56s to produce an inference result for one input (with $m=50$), while producing non-private prediction in just 0.025s. While the overhead seems significant on the surface, given the commodity hardware we used, which can be accelerated with parallelization and GPUs, we do not consider the overhead to be a deal-breaker for practical deployability of pricure.

Comparison with Closely Related Work: SecureNN (Wagh et al., 2018) proposes a DNN-centric protocol for secure training and prediction of 3- and 4-party setup. It was ran on Amazon EC2 c4.8x large instances in a LAN setting and WAN setting with average bandwidth of 625MB/s and 40MB/s, respectively. For per-sample inference on MNIST, 3-party and 4-party SecureNN require 0.34sec and 0.23sec, respectively. On the other hand, with 50-party setup ($m=50$), pricure requires 772.56sec to compute inference result for an image. Note that for $m=1$, pricure takes $1.5$sec to compute a per-sample prediction, which is comparable to SecureNN given the difference in hardware specifications and latency setup.

Chameleon (Riazi et al., 2018) is evaluated based on a 5-layer CNN on MNIST with inference label as output. Experiments were ran on Intel Xeon ES-1620 CPU, 3.5 GHz with 16 GB of RAM. Per-sample classification took 2.7sec with testing accuracy of 99%. In pricure, $m$ is orders of magnitude larger (e.g., $m=50$, $m=100$, depending on the dataset) and the inference is collaborative.

CryptoNets (Gilad-Bachrach et al., 2016) is a cloud-based platform where the server can provide encrypted prediction to client’s computing on their encrypted data so both parties’ data can be secure. CryptoNets was ran on a similar hardware as (Mohassel and Zhang, 2017) and per-sample classification took 297.5s with 99% test accuracy.

$\bigstar$ Take-away: With regards to RQ4, secret-sharing model parameters and aggregating inference vectors incur negligible overhead. We also note that inference overhead in pricure is fairly significant compared to its non-private counterpart. We, therefore, see room for optimizations through hardware acceleration, parallelization, and efficiency advances in SMPC.

Scalability: pricure’s effectiveness depends on: number of model owners, and complexity of models and inputs. Other factors kept constant, increasing number of model owners slows down the inference process, because more models means more load on the workers. Model complexity/architecture plays an important role in system performance. Model details such as number of hidden of layers, activation functions, and pooling operations determine the effectiveness of pricure. The complexity of feature vectors of inputs also determines the speed of computing predictions and the complexity of the learned parameters. For instance, images of 28x28 pixels for MNIST result in a model far less lightweight than images of 50x50 pixels for IDC. While our evaluations are encouraging as to real-life deployability of pricure, its scalability with more complex model architectures is worth exploring as future work. In addition, as the number of models increase, at some point, the performance load on the workers will be too much to bear, which demands performance enhancement alternatives such as hardware acceleration, multi-threading, and introducing more workers (albeit more complexity).

Performance Overhead Measurements: Due to our limitations of hardware resources, we measured performance overhead of pricure on a single machine. As a result, some measurements (e.g., secret-sharing delay, sending intermediate results from workers to the aggregator) may not fully capture an actual distributed setup. As in prior work (e.g., (Wagh et al., 2018)), production performance overhead measures need to be done in distributed setup to measure network latency for secret-sharing model parameters, computing aggregated inference results, and sending them to the client.

Model Architecture of MNIST and Fashion-MNIST Number of input neurons $=28\times 28$ Number of output neurons $=10$ Number of hidden layers $l=2$ Number of neurons in $1^{st}$ layer =$128$ Number of neurons in $2^{nd}$ layer =$64$ Activation Function: ReLu Error estimation method: MSE

Model Architecture of IDC Regular Number of input neurons $=7500$ Number of output neurons $=2$ Number of hidden layers $l=1$ Number of neurons in hidden layer =$500$ Activation Function: ReLu
Error estimation method: MSE

Model Architecture of MIMIC Critical Care Number of input neurons $=30$ Number of output neurons $=4$ Number of hidden layers $l=1$ Number of neurons in hidden layer =$500$ Activation Function: ReLu
Error estimation method: MSE

For additive secret sharing, first we have a finite field ${0,1,2,......Q-1}$ where all the secret-shares belong where $Q$ is a very large prime number. Now, suppose we split up $x$ into $i$ shares such that $x$ = $x_{1},x_{2},x_{3},...,x_{i-1}$ and the $i^{th}$ share will be $x_{i}=x-(x_{1}+x_{2}+,...,+x_{i})$, and the reconstruction will be $x=\sum_{k=1}^{i}x_{k}$. (Haagh et al., 2018). For two workers A and B, input value $x_{1}$ is split into two secret-shares and sent to A and B as follows:

The reconstruction is done using the following procedure:

The reconstruction can be proved with as follows:

For summation, suppose we want to compute $x+y$ via secret sharing. We divide them using Equation 4 and 5 as follows:

Now, the system sends $x^{a}$ and $y^{a}$ to worker $A$ and $x^{b}$ and $y^{b}$ to worker $B$. The summation operation on each worker side will be:

The actual sum is reconstructed using Equation 6 as follows:

This reconstruction can be proved as follows:

Since Q is a large prime compared to $x$ and $Y$, the above illustration proves that additive secret sharing reconstructs $x+y$.

For two numbers $x$ and $y$, our goal is to compute $x*y$ securely. We divide them using Equation 4 and 5 as follows:

The multiplication operation will be:

We can observe from above that to calculate $x^{a}y^{b}$ and $x^{b}y^{a}$ Worker A and Worker B need each other’s share, which conflicts with the privacy assumptions of our approach. To resolve the conflict, the SPDZ protocol (Beaver, 2012) is used for performing secret sharing multiplication. For this technique, independent of $x$ and $y$, values called ‘multiplication triplets’ are generated in offline phase. In this phase, three numbers $q1$, $q2$ and $q3$ are generated and shared with two parties ahead of running the protocol. The numbers, $q1$, $q2$ and $q3$ are generated as follows: