More Than Privacy: Applying Differential Privacy in Key Areas of Artificial Intelligence

A stable learning algorithm is one in which the prediction does not change much when the training data is modified slightly. Bousquet et al. [30] have proved that stability is linked to the generalization error bound of the learning algorithm, indicating that a highly stable algorithm leads to a less over-fit result. However, increasing the stability of the algorithm is challenging when the size of the testing data is limited. This is because the validate data sometimes are reused and lead to an incorrect learning model. To preserve statistical learning validity, analysts should collect new data for a fresh testing set.

Differential privacy can be naturally linked to learning stability. The concept of differential privacy ensures that the probability of observing any outcome from an analysis is essentially unchanged by modifying any single record. Dwork et al. [31, 4] showed that differential privacy mechanisms can be used to develop adaptive data analysis algorithms with provable bounds for over-fitting, noting that certain stability notions are necessary and sufficient for generalization. Therefore, differential privacy is stronger than previous notions of stability and, in particular, possesses strong adaptive composition guarantees [32].

Dwork et al. [4] show that, by adding noise to generate fresh testing data, differential privacy mechanisms can achieve highly stable learning. For a dataset $D$, an analyst learns about the data by running a series of analyses $f_{i}$ on the dataset. The choice of which analysis to run depends on the results from the earlier analyses. Specifically, the analyst first selects a statistic $f_{0}$ to query on $D$ and observes a query result $y_{1}=f_{0}(D)$. From the $k^{th}$ analysis, the analyst selects a function $f_{k}$ based on the query result ${y_{1},...,...y_{k-1}}$. To improve the generalization capability of the adaptive scenario, noise is added in each analysis iteration. For example, $y_{k}=Lap(\frac{\Delta f}{\epsilon})+f_{k-1}(D)$ [31].

This type of adaptive analysis can be linked to machine learning. The dataset $D$ can be randomly partitioned into a training set $D_{t}$ and a testing (holdout) set $D_{h}$. The analyst can access the training set $D_{t}$ without restrictions but may only access $D_{h}$ through a differentially private interface. The interface takes the testing and training sets as inputs and, for all functions given by the analyst, provides statistically valid estimates of each function’s results.

For a sufficiently large testing set, the differential privacy interface guarantees that for function $f:D\rightarrow[0,1]$, the mechanism will return a randomized value $v_{f}$. When $v_{f}$ is compared to the query result $y$, we have $|v_{f}-y|\leq\tau$ with a probability of at least $1-\beta$, where $\tau$ is the analyst’s choice of error and $\beta$ is the confidence parameter. The probability space is over the data elements in $D_{h}$ and $D_{t}$ and the randomness introduced by the interface. A multiplicative weight updating mechanism [33] could also be included in the interface to conserve the privacy budget.

The idea of adding randomization during data analysis to increase stability has been widely accepted. MacCoun et al. [34] believed: when deciding which results to report, the analyst interacts with a dataset that has been obfuscated through adding noise to observations, removing some data points, or switching data labels. The raw, uncorrupted, dataset is only used in computing the final reported values. Differential privacy mechanisms can follow the above rules to significantly improve learning stability.

Fairness issues are prevalent in every facet of our lives including education, job application, the parole of prisoners and so on [35, 36, 37]. Instead of resolving fairness issues, modern AI techniques, however, can amplify social inequities and unfairness. For example, an automated hiring system may be more likely to recommend candidates from specific racial, gender or age groups [38, 39]. A search engine may amplify negative stereotypes by showing arrest-record ads in response to queries for names predominantly given to African-American babies but not for other names [40, 41]. Moreover, some software systems that are used to measure the risk of a person recommitting crime demonstrate a bias against African-Americans over Caucasians with the same profile [42, 43]. To address these fairness issues in machine learning, great effort has been placed on developing definitions of fairness [44, 45, 46] and algorithmic methods for assessing and mitigating undesirable bias in relation to these definitions [47, 48]. A typical idea is to make algorithms insensitive to one or multiple attributes of datasets, such as gender and race.

Dwork et al. [49] classified individuals with the goal of preventing discrimination against a certain group while maintaining utility for the classifier. The key idea is to treat similar individuals similarly. To implement this idea, these researchers adopted the Lipschitz property, which requires that any two individuals, $x$ and $y$, with a distance of $d(x,y)\in[0,1]$ must map to the distributions $M(x)$ and $M(y)$, respectively, such that the statistical distance between $M(x)$ and $M(y)$ is at most $d(x,y)$ [50]. In other words, if the difference between $x$ and $y$ is $d(x,y)$, the difference of the classification outcomes of $x$ and $y$ is at most $d(x,y)$. A connection between differential privacy and the Lipschitz property has been theoretically established, in that a mapping satisfies differential privacy if, and only if, this mapping satisfies the Lipschitz property [49].

Zemel et al. [51] extended Dwork et al.’s [49] preliminary work by defining the metrics between individuals. They learned a restricted form of a distance function and formulated fairness as an optimization problem of finding the intermediate representation that best encodes the data. During the process, they preserved as much information about the individual’s attributes as possible, while removing any information about membership with other protected subgroup. The goal was two-fold: first, the intermediate representation should preserve the data’s original features as much as possible. Second, the encoded representation is randomized to hide whether or not the individual is from the protected group.

Both ideas take advantage of randomization in differential privacy. Considering an exponential mechanism with a carefully designed score function, the framework can sample fresh data from the universe to represent original data with the same statistical properties. However, the most challenging part of this framework is designing the score function. This is because differential privacy in fairness assumes that the similarity between individuals is given; however, estimating similarity between individuals in an entire feature universe is a tough problem. In other words, the evaluation of similarity between individuals is the key obstacle of model fairness, making score function design an obstinate problem. Therefore, differential privacy in model fairness needs further exploration.

Recently, researchers have attempted to adopt differential privacy to simultaneously achieve both fairness and privacy preservation [52, 53]. This research is motivated by settings where models are required to be non-discriminatory in terms of certain attributes, but these attributes may be sensitive and so must be protected while training the model [54]. Addressing fairness and privacy preservation simultaneously is challenging because they have different aims [55, 53]. Fairness focuses on the group level and seeks to guarantee that the model’s predictions for a protected group (such as female) are the same as the predictions made for an unprotected group. In comparison, privacy preservation focuses on the individual level. Privacy preservation guarantees that the output of a classification model is independent of whether any individual record is in or out of the training dataset. A typical solution to achieve fairness and privacy preservation simultaneously was proposed by Ding et al. [53]. Their solution is to add a different amount of differentially private noise based on different polynomial coefficients of the constrained objective function, where the coefficients relate to attributes in the training dataset. Therefore, privacy is preserved by adding noise to the objective function, and fairness is achieved by adjusting the amount of noise added to the different coefficients.

The best methods of similarity measurement and composition are open problems in fairness models. Further, differential privacy in fairness models has been directed toward classification problems. There are also some works on fairness in online settings such as online learning, bandit learning and reinforcement learning. However, how to use differential privacy mechanisms to benefit those online settings of fairness needs further investigation.

Composition fairness is also a big challenge. Here, fairness means that if each component in the algorithm satisfies the notion of fairness, the entire algorithm will satisfy the same [6]. This composition property is essential for machine learning, especially for online learning. Dwork et al. [56] explored this direction, finding that current methods seldom achieve this goal because classification decisions cannot be made independently, even by a fair classifier. Also, classifiers that satisfy group fairness properties may not compose well with other fair classifiers. Their results show that the signal provided by group fairness definitions under composition is not always reliable. Hence, further study is needed to figure out how to take advantage of differential privacy to ensure composition.

One of the most common privacy attacks is an inference attack where the adversary maliciously infers sensitive features and background information about a target individual from a trained model [60]. Typically, there are two types of inference attacks in deep learning.

The first type is a membership inference attack. The aim here is to infer whether or not a given record is in the present training dataset [61]. Membership inference attacks can be either black-box or white-box. Black-box means that an attacker can query a target model but does not know any other details about that model, such as its structure [62]. In comparison, white-box means that an attacker has full access to a target model along with some auxiliary information [63]. The attack model is based on the observation that machine learning models often behave differently on training data versus the data they “see” for the first time.

The second type of attack is an attribute inference attack. The aim of an attribute inference attack is to learn hidden sensitive attributes of a test input given access to the model and information about the non-sensitive attributes [64]. Fredrikson [64] describes a typical attribute inference attack method, which attempts to maximize the posterior probability estimate of the sensitive attributes based on the values of non-sensitive attributes.

Some of the properties of differential privacy are naturally resistant to membership and attribute inference attacks. An intuitive way to resist inference attacks is to properly add differentially private noise to the values of the sensitive attributes before using the dataset to train a model. In a typical deep learning algorithm, there are four places to add noise, as shown in Figure 5. The first place is the training dataset, where the noise is derived from an input perturbation mechanism. This operation occurs before the training starts and is usually done to resist attribute inference attacks.

The second place is the loss function, which yields an objective perturbation mechanism. This operation occurs during training and is usually done to resist membership inference attacks.

The third place is the gradients at each iteration, i.e., a gradient perturbation mechanism. Gradients are computed using the loss function to do partial-derivative against the weights of the deep neural network. Likewise, this operation occurs during training and is usually done to resist membership inference attacks.

The fourth place is the weights of the deep neural network, constituting the learned model, called an output perturbation mechanism. This operation happens once training is complete. The operation is easy to implement and can resist both membership and attribute inference attacks. However, directly adding noise to the model may significantly harm its utility, even if the parameter values in differential privacy have been carefully adjusted.

Of these four places, adding noise to the gradients is the most common method. However, because the gradient norm may be unbounded in deep learning, a bound must be imposed before applying gradient perturbation. A typical way is to manually clip the gradient at each iteration [65]. Such a clipping can also provide a sensitivity bound with differential privacy. Table III summarizes the properties of the mentioned attacks and their defence strategies.

From Table III, we can see that adding noise to a dataset can defend against attribute inference attacks. Since the aim of attribute inference attacks is to infer the values of sensitive attributes, directly adding noise to these values is the most straightforward and efficient method of protecting them. However, this method may significantly affect the utility of the learned model, because it is heavily dependent on the values of the attributes in the training dataset, and using a dataset with modified attribute values to learn a model is similar to using a “wrong” dataset.

By comparison, adding noise to the loss function or gradient only slightly affects the utility of the learned model. This is because the noise is added during the training process and the model can be corrected by taking the noise into account. Adding noise to the loss function or gradient can resist membership inference attacks, which can be guaranteed by the properties of differential privacy. However, adding noise to the loss function or gradient does not offer much resistance to attribute inference attacks. As mentioned before, a typical attribute inference attack needs two pieces of information: 1) the underlying distribution of the training dataset; and 2) the values of non-sensitive attributes. These two pieces of information are not modified when adding noise to the loss function or gradient.

Finally, adding noise to the weights or classes of a neural network can resist both membership and attribute inference attacks. This is because adding noise to the weights will modify the learned model and both of these types of attacks need to access the learned model to launch an attack. The downside is that adding noise to a learned model after training may drastically affect its utility, and retraining the model will not correct the problem as noise simply needs to be added again. Noise could be added to the weights in each iteration of training. However, this method might affect convergence, since the output of the algorithm is computed based on the weights. Hence, if noise is added to each weight, the total amount of noise might become large enough to make the loss never convergent. Adding noise to the classes has the similar disadvantage for the same reason.

Conventional deep learning is limited to a single-machine system, where the system has all the data and carries out the learning independently. Distributed deep learning techniques, however, accelerate the learning process. Two main approaches are applied in distributed deep learning: data parallelism and model parallelism [58]. In data parallelism, a central model is replicated by a server and distributed to all the clients. Each client then trains the model based on her own data. After a certain period of time, each client summarizes an update on top of the model and shares the update to the server. The server then synchronizes the updates from all the clients and improves the central model. In model parallelism, all the data are processed with one model. The training of the model is split between multiple computing nodes, with each computes only a subset of the model. As data parallelism can intrinsically protect the data privacy of clients, most research on distributed deep learning focuses on data parallelism.

As mentioned in the previous subsection, differentially private noise can be added to five places in a deep neural network. The following review is divided into methods based on adding noise.

Adding noise to input datasets. Heikkila et al. [66] proposed a general approach for privacy-preserving learning in distributed settings. Their approach combines secure multi-party communication with differentially private Bayesian learning methods so as to achieve distributed differentially private Bayesian learning. In their approach, each client $i$ adds a Gaussian noise to her data and divides them and the noise into shares. Each share is then sent to a server. In this way, the sum of the shares discloses the real value, but separately they are just random noise.

Adding noise to loss functions. Zhao et al. [67] proposed a privacy-preserving collaborative deep learning system that allows users to collaboratively build a collective learning model while only sharing the model parameters, not the data. To preserve the private information embodied in the parameters, a functional mechanism, which is an extended version of the Laplace mechanism, was developed to perturb the objective function of the neural network.

Adding noise to gradients. Shokri and Shmatikov [68] designed a system that allows participants to independently train on their own datasets and share small subsets of their models’ key parameters during training. Thus, participants can jointly learn an accurate neural network model without sharing their datasets, and can also benefit from the models of others to improve their learning accuracy while still maintaining privacy.

Abadi et al. [65] developed a differentially private stochastic gradient descent algorithm for distributed deep learning. At each iteration during the learning, Gaussian noise is added to the clipped gradient to preserve privacy in the model. In addition, their algorithm also involves a privacy accountant and a moment accountant. The privacy accountant computes the overall privacy cost during the training, while the moment accountant keeps track of a bound on the moments of the privacy loss random variable.

Cheng et al. [69] developed a privacy-preserving algorithm for distributed learning based on a leader-follower framework, where the leaders guide the followers in the right direction to improve their learning speed. For efficiency, communication is limited to leader-follower pairs. To preserve the privacy of leaders, Gaussian noise is added to the gradients of the leaders’ learning models.

Adding noise to weights. Jayaraman et al. [70] applied differential privacy with both output perturbation and gradient perturbation in a distributed learning setting. With the output perturbation, each data owner combines their local model with a secure computation and adds Laplace noise to the aggregated model estimator before revealing the model. With the gradient perturbation, the data owners collaboratively train a global model using an iterative learning algorithm, where, at each iteration, each data owner aggregates their local gradient within a secure computation and adds Gaussian noise to the aggregated gradient before revealing the gradient update.

Phan et al. [71] proposed a heterogeneous Gaussian mechanism to preserve privacy in deep neural networks. Unlike a regular Gaussian mechanism, this heterogeneous Gaussian mechanism can arbitrarily redistribute noise from the first hidden layer and the gradient of the model to achieve an ideal trade-off between model utility and privacy loss. To obtain the property of arbitrary redistribution, a noise redistribution vector is introduced to change the variance of the Gaussian distribution. Further, it can be guaranteed that, by adapting the values of the elements in the noise redistribution vector, more noise can be added to the more vulnerable components of the model to improve robustness and flexibility.

Adding noise to output classes. Papernot et al. [72] developed a model called Private Aggregation of Teacher Ensembles (PATE) which has been successfully applied to generative adversarial nets (GAN) for privacy guarantees [75]. PATE consists 1) an ensemble of $n$ teacher models; 2) an aggregation mechanism; and 3) a student model. Each teacher model is trained independently on a subset of private data. To protect the privacy of data labels, Laplace noise is added to the output classes, i.e., the teacher votes. Last, the student model is trained through knowledge transfer from the teacher ensemble with the public data and privacy-preserving labels. Later, Papernot et al. [73] improved the PATE model to make it applicable to large-scale tasks and real-world datasets. Zhao [74] also improved the PATE model by extending it to the distributed deep learning paradigm. Each distributed entity uses deep learning to train a teacher network on private and labeled data. The teachers then transfer the knowledge to the student network at the aggregator level in a differentially-private manner by adding Gaussian noise to the predicted output classes of the teacher networks. This transfer uses non-sensitive and unlabeled data for training.

Although a number of privacy-preserving methods have been proposed for distributed deep learning, there are still some challenging issues that have not yet been properly addressed. The first issue is synchronization. If data parallelism has too many training modules, it has to decrease the learning rate to ensure a smooth training procedure. Similarly, if model parallelism has too many segmentations, the output from the nodes will reduce training efficiency [58]. Differential privacy offers potential for solving this issue. Technically, the challenge is a coordination problem, where modules or nodes collaboratively perform a task, but each has a privacy constraint. This coordination problem can be modeled as a multi-player cooperative game, and differential privacy has been proven as effective for achieving equilibria in this game [76].

The second issue is collusion. Most of the existing methods assume non-collusion between multiple computation parties. This assumption, however, may fail in some situations. For example, multiple service providers may collude to obtain a user’s private information. Joint differential privacy may be able to address this issue, as it has been proven to successfully protect any individual user’s privacy even if all the other users collude against that user [77].

The third issue is privacy policies. Most existing methods rely on privacy policies that specify which data can be used by which users according to what rules. However, there is no guarantee that all the users will strictly follow the privacy policies. Differential privacy may be available to deal with this issue, as differential privacy can guarantee users will truthfully report their types and faithfully follow the recommendations given by the privacy policies.

Federated learning enables individual users to collaboratively learn a shared prediction model while keeping all the training data on the users’ local devices. Federated learning was first proposed by Google in 2017 as an additional approach to the standard centralised machine learning approaches [78], and has been applied to several real-world applications [79].

Fig. 6 shows the structure of a simple federated learning framework. First, the training centre distributes a general learning model, trained on general data, to all the smart devices in the network. This model is used for general purposes, such as image processing. In a learning iteration, each user downloads a shared model from the training centre to their local device. Then, they improve the downloaded model by learning from their own local data. Once complete, the changes to each user’s local model are summarized as a small update, which is sent to the training centre through a secure communication channel. Once the cloud server receives all the updates, the shared model is improved wholesale. During the above learning process, each user’s data remain on their own device and is never shared with either the training centre or another user.

However, although private user data cannot be directly obtained by others, it may be indirectly leaked through the updates. For example, the updates of the parameters in an optimization algorithm, such as stochastic gradient descent [80], may leak important data information when exposed together with data structures [81]. Differential privacy can, however, resolve this problem, as explained next.

Although no training data is transferred from mobile devices to the cloud centre in federated learning, simply keeping data locally does not provide a strong enough privacy guarantee when conventional learning methods are used. For example, adversaries can use differential attacks to discover what data was used during training through the parameters of the learning model [2]. To protect against these types of attacks, several algorithms that incorporate differential privacy have been proposed that ensure a learned model does not reveal whether the data from a mobile device was used during training [82].

Adversaries can also interfere with the messages exchanged between communicating parties, or they can collude among communicating parties during training to attack the accuracy of the learning outcomes. To ensure the resulting federated learning model maintains acceptable prediction accuracy, approaches using both differential privacy mechanisms and secure multiparty computation frameworks have been created, providing formal data privacy guarantees [83].

Geyer et al. [2] incorporated differential privacy mechanisms into federated learning to ensure that whether an individual client participates in the training cannot be identified. This approach protects the entire data of an individual client. To achieve this aim, in each communication round, a subset of the total clients is randomly selected. Then, the difference between the central model and each of the selected client’s local model is calculated, and Gaussian noise is added to the difference.

Shi et al. [84] investigated a distributed private data analysis setting, where multiple mutually distrustful users co-exist and each of them has private data. There is also an untrusted data aggregator in the setting who wishes to compute aggregate statistics over these users. The authors adopted computational differential privacy to develop a protocol, which can output meaningful statistics with a small total error even when some of the users fail to respond. Also, the protocol can guarantee the privacy of honest users, even when a portion of the users are compromised and colluding.

Agarwal et al. [85] combined two aspects of distributed optimization in federated learning: 1) quantization to reduce the total communication cost; and 2) adding Gaussian noise to the gradient before sending the result to the central aggregator to preserve the privacy of each client.

The main advantage of the federated leaning model is that none of the training data needs be transferred to the cloud centre which satisfies the basic privacy concerns of mobile device users can be satisfied. However, federated learning has some unique challenges, mainly in the following three respects:

• •

  Issues related to attacks on various vulnerabilities of the federated learning model and the countermeasures to defend against these attacks. For example, adversaries can use differential attacks to determine which mobile users have been included in the learning process [86]; messages can be tampered with; and adversaries can use model poisoning attacks to cause the model to misclassify a set of chosen inputs with high confidence [87, 88].

• •

  Issues related to the learning algorithms, such as the requirements of accuracy, scalability, efficiency, fault-tolerance, etc. [78, 89].

• •

  Issues related to the structure of the federated learning system, including its communication efficiency, the computational and power limitation of the mobile devices, the reliability of the mobile devices and communication system, etc. [86]. This issue can potentially be tackled through the composition property of differential privacy by fixing the privacy budget and forcing all communications to consume that budget.

To effectively use federated learning in various applications, we first need to overcome the challenges related to attacks, the system structures, and the learning algorithms. Therefore, intensive research addressing these challenges will be required in the near future. The second future development will be to explore the power and benefits of federated learning for both new and existing applications, especially now that mobile devices are ubiquitous. The third future development will be the automation of tools that use federated learning and the emergence of companies providing such services to meet the various needs of business and individual customers.

Differential privacy mechanisms can provide a security guarantee that a malicious agent being in or out of a multi-agent system has little impact on the utility of other agents. As the probability of selecting neighbors to ask for advice is based on the reward history provided by neighbors, exponential or Laplace mechanisms can be applied to this step to diminish the impact of malicious agents on security purpose. Moreover, the composition of the privacy budget can naturally control the communication overhead, namely by limiting the amount of advice allowed throughout the whole system.

Ye et al. [8] proposed a differentially private framework to deal with malicious agents and communication overhead problems. Using Fig. 7 as an example, suppose each agent in the grid environment wants to move to the corner and has the moving knowledge that can be shared with others.

Fig. 8 illustrates the process of agent interaction in this example. Agent $2$ send out an advice request to its neighbors. Agent $1$ and the malicious agent would give advice to agent $2$. Agent $1$’s advice will include the best action in agent $2$’s state according to agent $1$’s knowledge. However, the malicious agent will always give false advice. After receiving advice from both neighbors, agent $2$ performs a differentially private adviser selection algorithm, which applies an exponential mechanism to adjust the chosen probabilities. Because the malicious agent always provides false advice, the exponential mechanism may filter its advice with a high probability. So that the impact of the malicious agent will be diminished. The system will stop communicating when the privacy budget is used up, so that the privacy budget can be used to control the communication overhead.

Differential privacy technology has been proven to improve the performance of agent learning in addition to its use as a privacy-preservation tool. Compared to the benchmark broadcast-based approach, the differential privacy approach achieves a better performance (normally evaluated by the total reward of the system and the convergence rate) with less communication overhead when malicious agents are present. However, further exploration of differential privacy technique in new environments, such as a dynamic or an uncertain environment, would be worthwhile.

Although differential privacy in auctions has been widely accepted, these approaches typically assume a seller or an auctioneer can directly interact with all potential buyers. This assumption, however, may not be applicable to some real situations, where sellers and buyers are organized in a network, e.g., social networks [114, 115]. Auctions in social networks introduce new challenging privacy issues.

The first issue is the bidding propagation. In a social network, a bid from a buyer cannot be sent directly to the seller but has to be propagated by other agents to the seller. These intermediate agents may be potential buyers and are thus competitive to that buyer. Therefore, the bid value of that buyer is private and cannot be disclosed to others. An intuitive way to protect the privacy of that buyer is to add Laplace noise to the bid value. However, this does not stop the problem of a seller receiving a fake bid and making a wrong decision.

The second issue is social relationships. During bid propagation, a trajectory forms which indicates who is engaged in the propagation. By investigating this trajectory, the seller may learn things about the buyer’s social relationships.

Kearns et al. [76] developed a differentially private recommender mechanism for incomplete information games. Thanks to differential privacy techniques, their mechanism achieves equilibria of complete information games even with a large number of players and any player’s action only slightly affects the payoffs for other players. Their mechanism offers a proxy that can recommend actions to players. Players are free to decide whether to opt in to the proxy, but if players do opt in, they must truthfully report their types. In addition to satisfy the game-theoretic properties, the mechanism also guarantees player privacy, namely that no group of players can learn much about the type of any player outside the group.

Rogers and Roth [122] improved the performance and defense of malicious users by expanding on Kearns et al.’s work [76] to allow players to falsely report their types even if the players opt in to the proxy. They theoretically show that by using differential privacy to form an approximate Bayes-Nash equilibrium, players have to truthfully report their types and faithfully follow the recommendations.

Pai et al. [102] improved game performance as well. They studied infinitely repeating games of imperfect monitoring with a large number of players, where players observe noisy results, generated by differential privacy mechanisms, about the play in the last round. The authors find that, theoretically, folk theorem equilibria may not exist in such settings, which concern all the Nash equilibria of an infinitely repeated game. Based on this finding, they yield antifolk theorems [123], where restrictions are imposed on the information pattern of repeated games such that individual deviators cannot be identified [124].

Lykouris et al. [125] increased game stability by analyzing the efficiency of repeated games in dynamically changing environments and population sizes. They draw a strong connection between differential privacy and the high efficiency of learning outcomes in repeated games with frequent change. Here, differential privacy is used as a tool to find solutions that are close to optimal and robust to environmental changes.

Han et al. [126] developed an approximately truthful mechanism to defend against malicious users in an application to manage the charging schedules of electric vehicles. To ensure users to truthfully report their specifications, their mechanism takes advantage of joint differential privacy which can limit the sensitivity of the scheduling process to changes in user specifications. Therefore, any individual user cannot benefit from misreporting his specification, which results in truthful reports.

Hsu et al. [127] modelled the private query-release problem in differential privacy as a two-player zero-sum game between a data player and a query player. Each element of the data universe for the data player is interpreted as an action. The data player’s mixed strategy is a distribution over her databases. The query player has two actions for each query. The two actions are used to penalize the data player, when the approximate answer to a query is too high or too low. An offline mechanism, based on the Laplace mechanism, is then developed to achieve the private equilibrium of the game.

Zhang et al. [77] developed a general mobile traffic offloading system. They used the Gale-Shapley algorithm [128] to optimize the offloading station allocation plan for mobile phone users. In this algorithm, to protect users’ location privacy, they proposed two differentially private mechanisms based on a binary mechanism [129]. The first mechanism is able to protect the location of privacy of any individual user when all the other users are colluding against this user, but the administrator is trusted. The second mechanism is stronger than the first mechanism because it assumes that even the administrator is untrusted.

Zhou et al. [130] adopted an aggregation game to model spectrum sharing in large-scale and dynamic networks, where a set of users compete for a set of channels. They then applied differential privacy techniques to guarantee the truthfulness and privacy of the users. Specifically, they use a Laplace mechanism to add noise to the cost threshold and users’ costs to protect this information. Moreover, they use an exponential mechanism to decide the mixed strategy aggregative contention probability distribution for each user so as to preserve the privacy of users’ utility functions.

The current research on differential privacy in game theory has mainly focused on static environments. These same issues in dynamic environments are generally still open. Of the little research that does consider dynamic environments [125], only changes in population are considered while overlooking changes in other areas, such as the strategies available to each agent or the utility of each of those strategies. Studying game theory with changing available strategies is a challenging issue, as these types of changes may result in no equilibria between agents. This is because no matter which strategy is taken by an agent, other agents may always have strategies to defeat that agent. In other words, other agents are incentivized to unilaterally change their strategies. However, since differential privacy can be used to force agents to report truthfully, it may also be used to force agents to reach equilibria.

When an agent is in an unfamiliar state during a multi-agent learning process, it may ask for advice from another agent [106]. These two agents then form a teacher-student relationship. The teacher agent offers advice to the student agent about which action should be taken. Existing research is based on a common assumption that the teacher agent can offer advice only if it has visited the same state as the student agent’s current state. But this assumption might be relaxed by using differential privacy technique.

The property of differential privacy can be borrowed to address the advice problem. Two similar states are interpreted as two neighbouring datasets. The advice generated from the states is interpreted as the query result yielded from datasets. Since two results from neighbouring datasets can be considered approximately identical, two pieces of advice generated from two similar states can also be considered approximately identical. This property can thus guarantee that advice created in a state can still be used in another similar state. Hence, this may be an interesting way to improve agent learning performance.

When agents transfer knowledge between each other to improve learning performance, a key problem discussed is that privacy needs to be preserved [107]. Existing methods are typically based on homomorphic cryptosystems [153, 154, 155]. However, homomorphic cryptosystems have a high computation overhead and, therefore, may not be very efficient in resource-constrained systems, e.g., wireless sensor networks. Differential privacy, with its light computation overhead, therefore, could be a good alternative in these situations.

Reasoning is an ability that enables an agent to use known facts to deduce new knowledge. It has been widely employed to address various real-world problems. For example, knowledge graph-based reasoning can be used in speech recognition to parse speech contents into logical propositions [156], and case-based reasoning can be adopted to address the data-sparsity issue in recommendation systems by filling in the vacant ratings of the user-item matrix [157]. A typical reasoning method is based on the Belief, Desire and Intention (BDI) model [158]. An agent’s beliefs correspond to information the agent has about the world.

Reasoning is a powerful tool in AI especially when it is combined with deep neural networks. For example, Mao et al. [159] has recently proposed a neuro-symbolic concept learner which combines symbolic reasoning with deep learning. Their model can learn visual concepts, words and semantic parsing of sentences without explicit supervision on any of them. As reasoning requires querying known facts which may contain private information, privacy preservation becomes an issue in reasoning process. Tao et al. [160] propose a privacy-preserving reasoning framework. Their idea is to hide the truthful answer from a querying agent by providing the answer “Unknown” to a query. Then, the querying agent cannot distinguish between the case that the query is being protected and the case that the query cannot be inferred from the known facts. However, simply hiding truthful answers may seriously hinder the utility of querying results. Differential privacy, with its theoretical guarantee of utility of querying results, may be a promising technique for privacy-preserving reasoning.

Recently, there have been great efforts to combine differential privacy with reasoning [161, 162, 163]. These works, however, aim to take advantage of reasoning to prove differential privacy guarantees of programs, instead of using differential privacy to guarantee privacy-preserving reasoning. Therefore, a potential direction of future research may be introducing differential privacy into reasoning process to guarantee the privacy of known facts.