FT-PrivacyScore: Personalized Privacy Scoring Service for Machine Learning Participation

Machine learning (ML) models have shown promising performance in many applications. Consequently, industrial practitioners are collecting customer data, building or fine-tuning ML models with the collected data, and deploying models in their products to enhance functionalities and revenues. However, the widespread deployment of ML models raises concerns about data leakage and privacy breaches (Gu and Chen, 2023). Differentially private machine learning, e.g., DP-SGD (Abadi and et al., 2016), is a well-known approach to addressing the data privacy issue, and a few companies have started experimenting with it in production environments. Methods like DP-SGD allow data contributors to quantitatively gauge the amount of privacy loss, e.g., via a global privacy setting $\epsilon$ (Abadi and et al., 2016) or a personalized local privacy setting (Jorgensen and et al., 2015). While the practical meaning of such a setting is still arguable, a major drawback is the significantly reduced model quality due to noise addition and gradient clipping (Abadi and et al., 2016), which might not be acceptable to many model builders. Focusing on data utility, an alternative approach, controlled data access, is still actively adopted by major agencies, e.g., NIH. For example, the NIH All of Us project (of Health, 2015) allows authorized researchers to work within an online workbench web service to access individual records confidentially. In the controlled access environment, no data perturbation is applied to guarantee full data utility, and the data curator and authorized data users are trusted to protect data privacy well. However, individual participants still have the right to understand their privacy risks and decide whether to participate in a study cohort. It’s also important for the model builder to weigh the risks and gains of incorporating a specific contributor or record into their modeling. However, there is no formal quantitative privacy risk evaluation tool like differential privacy for such a controlled data access scenario. Inspired by the recent development in the hypothesis-based membership inference (Carlini and et al., 2022a), i.e., the likelihood-ratio test (LiRA) method, we design an efficient privacy scoring tool for evaluating the potential risk of each individual data contributor participating in a cohort-based fine-tuning modeling task. We consider a scenario where the model builder continuously fine-tunes the model with fresh instances from data contributors. For each fresh instance, the scoring tool will tell the data contributor and the model builder the privacy risk score of including this specific instance. As such, the data contributor can decide whether she/he wants to attend, and the data contributor may also estimate the risk and the gain (e.g., via another utility tool) to include such an instance. This tool can also be used to determine instance-specific incentives for participants – potentially more usages to be explored. However, directly deploying the LiRA method has a major performance challenge. The original method depends on training many models with sample sets from the whole dataset, which are too expensive to be practical. An initial evaluation on 25,000 sample models trained with or without the target record on one GPU server takes 6.5 hours to test just one sample. Although the whole batch of model training can be parallelized, the total cost of GPU hours is still substantial. Therefore, we consider two improvements to significantly lower the cost and make the demo practical. First, we focus on model fine-tuning tasks, which are more practical for large models. Second, we employ a batch evaluation method for calculating the privacy scores for a batch of submitted records together, which significantly reduces the per-instance cost. We find the proposed method can significantly reduce the cost to 3 minutes per instance. The core of this demo is the FT-PrivacyScore privacy scoring service that takes information from both the data contributors and the model builder. Data contributors submit their records to be evaluated, and the model builder provides the training data distribution and the base model to be fine-tuned. The scoring service will generate hundreds of fine-tuned models conducted on random sample sets (details in Section 2) and perform the LiRA test on the fine-tuned models for each record to be evaluated to generate the privacy score. The main contributions of this demonstration include: (1) It’s the first novel application of the offline LiRA method to scoring privacy risks efficiently for users participating in a model fine-tuning task; (2) The demo system provides sufficient details and an interactive interface for the audience to assess the practicality of the proposed approach.

Figure 1 illustrates the overall architecture for evaluating the privacy level of new instances collected from data contributors. The service will first request and register the modeling task information, e.g., a base model and the fine-tuning strategies provided by the model builder. After the service receives a sufficient number of requests for privacy scoring (e.g., $>100$ records) from one or multiple contributors participating in the task, the core scoring procedure is applied to calculate and deliver a privacy score for each record to the corresponding contributor. The privacy score is evaluated based on the susceptibility of a sample to membership inference attacks across multiple models, assessing how easily the sample can be identified successfully (Carlini and et al., 2022b). We adopted the idea of Likelihood-ratio Attack (LiRA) (Carlini and et al., 2022b) and made it work efficiently on batch data and fine-tuned models. The LiRA method was designed for testing individual records and has not been applied to fine-tuned models. Background: LiRA. Carlini et al. (Carlini and et al., 2022a) introduce both online and offline LiRA tests. The online LiRA test involves training thousands of shadow models on randomly sampled datasets that may contain or not contain the target record. Hypothesis testing is applied to determine the risk of the target record being identified as a member of the training data. The whole process is very expensive as it requires thousands of models trained for the tested record. In contrast, the offline LiRA test does not consider the tested record when preparing the shadow models, which are used to estimate the approximate distribution of the log-likelihood-ratio $\log((1-p)/p)$, for instance, where $p$ is the highest probability among all possible classes in classification modeling, for out-domain samples’ output probabilities. For a new sample $x$ to be tested, we apply the specific model $f(x)$ and test whether its output’s log-likelihood-ratio $\log(1-p_{x})/p_{x})$ is significantly higher than the typical out-domain’s. The offline LiRA can significantly reduce the computational burden with slightly reduced accuracy.

Efficient Privacy Scoring. The service consists of two stages: the preparation stage and the production stage. In the preparation stage, the privacy scoring service prepares domain-agnostic models, denoted $\{O_{1},...,O_{k}\}$, and fit a Gaussian distribution $\mathcal{N}(\mu,\sigma^{2})$ (Carlini and et al., 2022a) for out-domain samples’ log-likelihood-ratio distribution in offline LiRA test, which will be shared for scoring later. Then, in the production stage (Algorithm 1), the service starts receiving data samples to be tested from contributors, and also the corresponding original model $M_{O}$, and the fine-tuning strategy, $S$, from the model builder. Once we have received enough $m$ test samples, denoted as $t_{j},j=1..m$, we build $n$ fine-tuned models, $M_{i},i=1..n$ with the fine-tuning strategy, each of which randomly takes m/2 samples from the submitted test samples. As a result, for each tested sample in $t_{j}$, roughly $n/2$ models’ training data contains this sample for fine-tuning, and the other half does not. For each sample, $t_{j}$, the offline LiRA is used to attack this sample for each model $M_{i}$, which will report either 0 or 1. The membership prediction is compared with the ground truth, i.e., we know already whether $M_{i}$’s fine-tuning has used $t_{j}$, denoted $G(t_{j},M_{i})\in\{0,1\}$. The privacy score for $t_{j}$ is then calculated as $|2\sum_{i=1}^{n}\mathbb{I}(\text{LiRA}(t_{j},M_{i})==G(t_{j},M_{i}))/n-1|$. In the worst-case scenario, the LiRA test gives a random guess, resulting in a privacy score of 0. This batch-based method has two unique features: (1) specifically designed for fine-tuning, which is more practical for large models, and (2) the cost of training $n$ fine-tuned models is spread to the $m$ samples. To validate whether our privacy score makes sense, we tested our service on 100 random samples from the CIFAR-10 dataset and compared the results with the expensive per-sample-based non-fine-tuning approach (Carlini and et al., 2022b). As shown in Figure 2, the privacy scores obtained from our service closely align with the more expensive version that takes 6 hours to evaluate one score, while our method completes the evaluation in just 3 minutes per score.

The audience will experience the demonstration through the following components: 1. Introduction: The first part of the demonstration uses a poster to outline the problem, the architecture, the privacy score evaluation pipeline, and the demonstration system. 2. Live System: The audiences will be able to interact with the system to evaluate the privacy scores of pre-selected samples using pre-fine-tuned models or by running the entire pipeline with their setup.

The demonstration showcases a personalized privacy scoring service for machine learning participation based on the recently developed offline LiRA attack. The purpose is to show that with the proposed batch-based offline-online combined processing strategy, we are able to make fast privacy score evaluations for data contributors to determine the risk of participating in a model fine-tuning task. The audience can interactively explore the demo, which we believe will help researchers and practitioners better understand the basic idea and practicality of our approach.

This research was partially supported by the National Science Foundation (Award# 2232824).