Federated Unlearning via Active Forgetting

Yuyuan Li [email protected] Zhejiang UniversityChina , Chaochao Chen [email protected] Zhejiang UniversityChina , Xiaolin Zheng [email protected] Zhejiang UniversityChina and Jiaming Zhang [email protected] Zhejiang UniversityChina

(2023)

Abstract.

The increasing concerns regarding the privacy of machine learning models have catalyzed the exploration of machine unlearning, i.e., a process that removes the influence of training data on machine learning models. This concern also arises in the realm of federated learning, prompting researchers to address the federated unlearning problem. However, federated unlearning remains challenging. Existing unlearning methods can be broadly categorized into two approaches, i.e., exact unlearning and approximate unlearning. Firstly, implementing exact unlearning, which typically relies on the partition-aggregation framework, in a distributed manner does not improve time efficiency theoretically. Secondly, existing federated (approximate) unlearning methods suffer from imprecise data influence estimation, significant computational burden, or both. To this end, we propose a novel federated unlearning framework based on incremental learning, which is independent of specific models and federated settings. Our framework differs from existing federated unlearning methods that rely on approximate retraining or data influence estimation. Instead, we leverage new memories to overwrite old ones, imitating the process of active forgetting in neurology. Specifically, the model, intended to unlearn, serves as a student model that continuously learns from randomly initiated teacher models. To preserve catastrophic forgetting of non-target data, we utilize elastic weight consolidation to elastically constrain weight change. Extensive experiments on three benchmark datasets demonstrate the efficiency and effectiveness of our proposed method. The result of backdoor attacks demonstrates that our proposed method achieves satisfying completeness.

Federated learning, Machine Unlearning, Poisoning Attack

^†^†copyright: acmcopyright^†^†journalyear: 2023^†^†doi: XXXXXXX.XXXXXXX^†^†conference: ACM xxx 2023; 2023; Ottawa, Canada^†^†ccs: Security and privacy Privacy protections^†^†ccs: Computing methodologies Cooperation and coordination

1. Introduction

With the prevalence of Machine Learning (ML) in various areas (LeCun et al., 2015; Jumper et al., 2021; OpenAI, 2023), there is a growing concern regarding the potential negative impacts of ML models. In response, several regulatory requirements have emerged to promote privacy in ML systems. For example, the General Data Protection Regulation (GDPR) (Voigt and Von dem Bussche, 2017) in the European Union allows individuals to request the removal of their data, including any influence it may have had on training models, i.e., the Right To Be Forgotten (RTBF). Similarly, the California Consumer Privacy Act (CCPA) (Pardau, 2018), proposed as a state law in the U.S., requires businesses to disclose what personal information they collect and gives consumers the right to request deletion.

Refer to caption — Figure 1. An overview of different unlearning approaches. $\theta$ and $\theta_{\lnot}$ denote original and unlearned models respectively. $\mathcal{D}$ , $\mathcal{R}$ , and $\mathcal{D}^{\prime}$ denote the original training dataset, the target data (to be unlearned), and the updated dataset without the target data ( $\mathcal{D}\backslash\mathcal{R}$ ) respectively.

Machine unlearning is an effective approach that can help preserve privacy in ML systems, which focuses on removing previously used data and learned information from ML models. On the one hand, individuals can remove their sensitive information learned by ML models, in addition to removing their training data. On the other hand, companies can proactively unlearn the dirty data that is no longer accurate (Li et al., 2016). The most straightforward method of unlearning is retraining the model from scratch using the updated dataset, without including the target data, i.e., data to be unlearned. However, retraining from scratch is often impractical due to the significant computational overhead involved when training ML models. Based on the degree of unlearning completeness, unlearning methods can be categorized into two approaches, i.e., exact unlearning (full completeness) and approximate unlearning (partial completeness).

In recent years, federated learning has emerged as a promising approach for training ML models in a distributed manner, without compromising user privacy. Despite its potential benefits, federated learning still falls short in addressing the aforementioned concerns, i.e., RTBF and fairness. As a result, researchers have turned their attention to exploring unlearning in a distributed manner, known as federated unlearning. Existing federated unlearning methods predominantly focus on approximate unlearning. This is due to the fact that implementing exact unlearning, which typically relies on the partition-aggregation framework, in a distributed manner does not yield substantial improvements in time efficiency. Existing federated (approximate) unlearning methods are inadequate as they are plagued by imprecise data influence estimation (Basu et al., 2021), significant computational burden (Guo et al., 2021), or both, leaving much room for improvement. To be specific, imprecise estimation will significantly affect the completeness of unlearning and hinder model utility, while computational burden reduces the efficiency of unlearning.

In this paper, we propose a novel federated unlearning method, named FedAF, which is independent of specific models and federated settings. As shown in Figure 1, our proposed FedAF presents a distinct design in comparison to the prevailing unlearning approaches. Taking inspiration from active forgetting in neurology (Davis and Zhong, 2017), FedAF enables the target model, i.e., the model in a client with the intention to unlearn, to use new memories to overwrite the old ones, mimicking the function of dopamine-producing forgetting cells that accelerate the elimination of memorization. This allows for a seamless unlearning process that is integrated into the original federated learning procedure, which fundamentally overcomes the limitations of existing federated unlearning methods, without the need for storing historical updates or estimating data influence.

Specifically, we implement the design of active unlearning based on incremental learning, which focuses on the task of learning multiple tasks in sequence. In the context of unlearning, the original learning process and the subsequent unlearning process can be regarded as two sequential tasks. Here we treat all unlearning requests as one process for conciseness. The key distinction between incremental learning and federated unlearning lies in the presence of task conflict. Incremental learning acquires sequentially new knowledge without any conflict between two sequential tasks. In federated unlearning, there is a conflict as the subsequent task requires unlearning previously acquired knowledge in the previous task.

When using incremental learning to unlearn, there are two critical points to consider: (i) generating effective new memories, and (ii) overcoming catastrophic forgetting. FedAF consists of two modules, i.e., memory generator and knowledge preserver, to address these issues, respectively. Firstly, we need to generate knowledge-free and easy-to-learn new memories to overwrite the old ones. Knowledge-free means containing no effective information about the data, e.g., a random label vector. Meanwhile, the new memories have to be easy-to-learn so that it will not cost considerable computational overhead to accomplish unlearning. Memory generator follows a teacher-student learning pattern to generate fake labels and pairs them with original features. Secondly, previous studies found that conventional deep learning methods fail to tackle incremental learning due to the phenomenon of catastrophic forgetting (Chen et al., 2021). In other words, besides removing the influence of target data, the influence of non-target data is also removed. Knowledge preserver alleviates the issue of unintentional forgetting by elastically constraining the model’s parameters with our derived loss, which is to address the problem of task conflict. The main contributions of this paper are summarized as follows:

•

We propose a novel federated unlearning method, i.e., FedAF based on the concept of active forgetting, which fundamentally overcomes the limitations posed by existing federated unlearning methods.
•

To effectively generate new memories to overwrite the old ones, we adopt a teacher-student learning pattern. The model in a target client, i.e., student, distills knowledge from manipulated data which is generated by teacher models.
•

To alleviate the catastrophic forgetting phenomenon of non-target data, we first derive a new loss to address the problem of task conflict, and then dynamically constrain the change of model’s parameters with elastic weight consolidation.
•

We conduct extensive experiments on three benchmark datasets to evaluate the performance of FedAF. The results show that our proposed FedAF outperforms compared methods in terms of efficiency, utility, and completeness.

2. Preliminaries

In this section, we first introduce the notations of federated learning and unlearning, followed by the principles of federated unlearning. Afterwards, we clarify the unlearning target in this paper.

2.1. Notation

Federated Learning. Federated learning allows multiple clients to collaboratively train a global model without sharing their private data. A typical architecture of federated learning, i.e., FedAvg (McMahan et al., 2017) can be formulated as follows: There are $K$ clients denoted by $\mathcal{K}=\{1,2,\dots,K\}$ , and a server. Each client $k\in\mathcal{K}$ has a local dataset $\mathcal{D}_{k}$ with $n_{k}$ denting the number of samples in it. The goal is to obtain a global model parameterized by $\theta^{*}$ that minimizes the empirical risk over all clients:

(1)

\theta^{*}=\arg\min_{\theta}\frac{1}{K}\sum_{k=1}^{K}w_{k}L_{k}(\theta),

where $w_{k}=n_{k}/\sum_{k}n_{k}$ is the weight assigned to client $k$ , and $L_{k}(\theta)$ is the local loss function of client $k$ . At $t$ -th federated iteration, each client $k$ performs $E$ epochs of stochastic gradient descent on its local dataset $\mathcal{D}_{k}$ using the current global model parameters $\theta_{t}$ , and then uploads the updated model parameters $\theta_{k,t+1}$ to the server for aggregation. The server aggregates the model updates from all clients using a weighted average:

(2)

\theta_{t+1}=\sum_{k=1}^{K}=w_{k}\theta_{k,t+1}.

This process continues until the global model $\theta$ converges or reaches the maximum federated iterations.

Federated Unlearning. As formulated above, each client $k$ possesses a local dataset $\mathcal{D}_{k}$ . Thus, the client $k$ has the right to remove any of its data and corresponding influence in the global model. This process is termed as unlearning. Formally, the client $k$ submits a request asking to unlearn a specific target $\mathcal{R}\in\mathcal{D}_{k}$ . Following (Liu et al., 2022; Gao et al., 2022), we assume that the requests are submitted after the end of federated training, i.e., the globe model has reached convergence. In practice, clients may submit unlearning requests during federated training, but comparing the performance of the global model before and after unlearning during federated training is inconvenient. After receiving the requests, the server can instruct all clients to take any necessary steps to unlearn the target data while ensuring that their local data remains private. We denote the parameters of the ground-truth unlearned model by $\theta_{\lnot}^{*}$ which is collaboratively retrained across all clients on $\mathcal{D}\backslash\mathcal{R}$ from scratch.

2.2. Unlearning Principles

We identify four principles that we consider as the pillars of achieving successful federated unlearning. Similar objectives of the first three principles can also be found in (Bourtoule et al., 2021; Chen et al., 2022).

P1: Unlearning Completeness. Completely unlearning the influence of target data refers to entirely revoking the target data information learned by ML models, making it irretrievable. In some real-world scenarios, completeness may be partially sacrificed for efficiency.

P2: Unlearning Efficiency. Efficiency is another important principle of unlearning. Practical ML models often involve large-scale datasets and parameters, which results in significant computational overheads in terms of both time and space. As a consequence, retraining from scratch is prohibitive.

P3: Model Utility. It is evident that both clients and servers desire to maintain model performance after unlearning. However, it is important to acknowledge that unlearning a significant amount of data lineage would inevitably diminish the model utility, because unlearning is equivalent to reducing the amount of training data. Thus, an adequate unlearning method needs to generate an unlearned model that achieves comparable performance to a model retrained from scratch.

P4: Data Privacy. In the context of federated learning, the raw data is possessed by a set of clients who are unwilling and not authorized to share it with each other. Due to this inherent limitation, it becomes imperative to ensure that user privacy, i.e., sharing raw data, is not compromised during the federated unlearning process.

2.3. Unlearning Targets

Unlearning targets can be mainly classified into four categories based on their scope, i.e., client-wise (all data of a client), class-wise (all data of a class), sample-wise (a data sample) and feature-wise (one feature dimension of a data sample). In federated learning, the data is possessed by various clients without permission for sharing. Thus, in this paper, the class-wise target refers to a specific class of data that is possessed by the target client. Except for feature-wise targets, our proposed method (FedAF) is capable of handling the other three types of targets, and is particularly well-suited for class-wise targets. To fully exploit the capability of FedAF, in this paper, we conduct experiments of class-wise unlearning, without compromising the generality.

3. Related Work

3.1. Machine Unlearning

Machine unlearning methods can be categorized into two approaches: exact unlearning (full completeness) and approximate unlearning (partial completeness). The choice of approach depends on the situation and desired outcome.

Exact Unlearning (EU). This approach aims to completely remove the influence of target data, guaranteeing that there is no residual influence in the unlearned model. Retraining from scratch is a naive but algorithmic, i.e., naturally enabling full completeness, approach. As retraining takes considerable computational overhead in practice, EU approach mainly focuses on enhancing retraining efficiency. The main idea behind the existing EU methods is partition-aggregation, which involves partitioning the dataset or model into sub-components, training them individually, and then aggregating them at the end (Cao and Yang, 2015; Ginart et al., 2019; Schelter et al., 2021; Bourtoule et al., 2021; Yan et al., 2022). With this idea, EU methods can limit the overhead of retraining to sub-components, and avoid retraining from scratch. However, EU methods suffer from a trade-off between unlearning efficiency (P2) and model utility (P3). On the one hand, increasing the number of sub-components can enhance unlearning efficiency, but this may lead to the issue of weak learners, i.e., models with poor utility. On the other hand, limiting the number of sub-components can preserve model utility, but this simultaneously restricts the unlearning efficiency.

Approximated Unlearning (AU). This approach aims to expedite unlearning by approximating the parameters of the ground truth model, i.e., retraining from scratch. Existing AU methods estimate the influence of target data and directly remove it through reverse gradient operations (Sekhari et al., 2021; Wu et al., 2022b; Mehta et al., 2022). Estimating the data influence is mainly based on influence function (Koh and Liang, 2017; Koh et al., 2019). Despite the theoretical ability of AU methods to improve unlearning efficiency, the associated computational overhead required for influence estimation remains a significant and limiting factor, particularly for large-scale models. The latest AU methods manage to accelerate influence estimation by approximation, i.e., approximate the approximation (Wu et al., 2022b; Mehta et al., 2022), which inevitably results in decreased accuracy of influence estimation.

3.2. Federated Unlearning

Due to the collaboration among clients in federated learning, achieving exact federated unlearning costs extra-prohibitive computational overhead. Therefore, existing federated unlearning methods focus on approximate unlearning, which can be further divided into the following three categories.

Retrain Unlearning. This approach mimics EU approach, but does not adopt the partition-aggregation framework, which can not improve time efficiency in the context of federated learning. Instead, retrain unlearning methods accelerate retraining by approximating the gradients using previously stored historical updates (Liu et al., 2021; Yuan et al., 2023; Liu et al., 2022). This approach is hindered by inaccurate approximation and the burden of data storage.

Reverse Unlearning. This approach follows the idea of AU, removing the estimated influence through reverse gradient operations, e.g., loss maximization (Halimi et al., 2022) and stochastic gradient ascent (Wu et al., 2022a). Analogous challenges to the AU approach are encountered by this approach.

Others. There are other federated unlearning methods using knowledge distillation (Wu et al., 2022c), scaled gradients (Gao et al., 2022), and channel pruning (CNN-specific) (Wang et al., 2022). The theoretical underpinnings of these methods are notably weaker than that of the above two approaches, placing them at a greater risk of encountering privacy concerns.

As for comparison, our proposed FedAF is based on a novel approach that continues the original federated learning process to achieve unlearning.

3.3. Incremental Learning

Incremental learning, which is also known as continual learning (Li et al., 2019) and lifelong learning in literature, aims to sequentially learn multiple tasks. Different from transfer learning and multi-task learning, incremental learning focuses on achieving high performance across all tasks while only having access to the data of new task(s). This aligns with the setting of federated unlearning, where it can be arduous to reach for historical updates. There are mainly three approaches to address catastrophic forgetting in incremental learning:

•

Selective Synaptic Plasticity elastically constrains parameter change based on synaptic importance to preserve learned knowledge (Aljundi et al., 2018; Kolouri et al., 2020). It is known for Elastic Weight Consolidation (EWC) framework.
•

Additional Neural Resource Allocation allocates new parameters for new tasks (Schwarz et al., 2018; Li et al., 2019). This approach changes the model structure, which is obviously unsuitable for the unlearning problem.
•

Memory Reply stores previous data or generates pseudo-data, and replays this data with the new data (Wu et al., 2018; Hu et al., 2019). This approach either increases extra computational overhead or changes the original learning process. Thus it is out of our consideration.

4. Active Forgetting Framework

Motivation. The challenge of federated unlearning arises due to the extensive collaboration between clients and the server in federated learning. Existing federated unlearning methods suffer from imprecise data influence estimation, significant computational burden, or both. Inspired by active forgetting (Davis and Zhong, 2017) in Neurology, we propose a novel federated unlearning framework named FedAF. Different from natural forgetting, i.e., passive forgetting, active forgetting can induce the forgetting of specific memories (Anderson and Hulbert, 2021).

Advantages. To actively forget the target data influence, our proposed Federated Active Forgetting Framework (FedAF) continually trains the model with new memories, i.e., manipulated data, which are generated from the old memories that need to be forgotten, i.e., target data. Compared with existing framework, our proposed FedAF has the following advantages:

•

FedAF’s unlearning process can be considered as a part of the extended learning process, reducing the additional impact on the model utility to a minimal level.
•

FedAF has wide applicability. As it seamlessly integrated the unlearning process into the learning process, it is independent of specific models and federated settings.
•

Compared with retrain unlearning, FedAF neither requires additional storage for historical updates nor spends extra computational overhead for non-target data, which reduces the deployment cost of unlearning in practice.
•

Compared with reverse unlearning, FedAF avoids estimating the influence of data, which is found analytically intractable (Graves et al., 2021), and continues the original training to achieve unlearning. This enables FedAF to unlearn with higher precision.

4.1. Framework Overview

FedAF learns from the new memories to achieve unlearning. The new memories contain no effective knowledge of the target data, which can overwrite the old ones. Figure 2 shows the learning (black arrows) and unlearning (blue arrows) workflow of FedAF. In the learning workflow, FedAF neither interferes with the original federated learning process nor stores any historical update. As described in Section 2.1, we can condense the original federated learning process into two loops, i.e., local training loop and federated training loop. In the unlearning workflow, FedAF incorporates an unlearning loop within the local training loop, and then broadcast the unlearned update through the federated training loop.

The unlearning loop in FedAF consists of two modules, i.e., memory generator and knowledge preserver. These two modules tackle the aforementioned issues respectively, i.e., generating effective new memories and overcoming catastrophic forgetting. In general, the memory generator initializes a set of teacher models to generate fake labels, and then pairs them with the original feature to produce the manipulated data. The knowledge preserver continually trains the model on the manipulated data. With the help of the new loss function derived from EWC framework, the knowledge preserver manages to alleviate the catastrophic forgetting phenomenon. Algorithm 1 summarizes the details in the unlearning workflow, where lines 1 to 6 represent memory generator and line 7 represents knowledge preserver.

Algorithm 1 Federated Active Forgetting Framework

1:The client

k

submits an unlearning request

\mathcal{R}\in\mathcal{D}_{k}

2:The client

k

receives the globe model

\theta

3:Let new memories

\mathcal{M}=\emptyset

;

4:Initialize

Q

teacher models without training;

5:for

(x,y

) in

\mathcal{R}

6: Generate fake label

\tilde{y}

by Eq (3) from teacher models;

7: Add (

x,\tilde{y}

) to

\mathcal{M}

;

8:end for

\triangleright

Teacher models can be released after use.

9:Train

\theta

\mathcal{M}

and

\mathcal{R}

by Eq (7), produce the local unlearned model

\theta_{\lnot k}

;

10:Aggregate local unlearned model

\theta_{\lnot k}

to the server by Eq (2);

11:return The global unlearned model

\theta_{\lnot}

;

4.2. Memory Generator

The goal of memory generator is to produce knowledge-free and easy-to-learn new memories for overwriting. In this paper, we focus on supervised learning, where the knowledge usually lies in labels. Therefore, we manipulate labels to generate new memories.

4.2.1. Knowledge-Free

The primary characteristic of new memories is knowledge-free. Removing the influence of the target data means making the model behave as if it has never seen the target data before. In other words, the model is supposed to have no knowledge about the target data. Following the idea of active forgetting, we generate knowledge-free labels, and overwrite the previously learned knowledge by training the model on these manipulated data (original features with new labels).

Assuming there is a binary classification task, and the label of a target data point can be $[0,1]^{\top}$ , which means it belongs to the second class. There are two types of straightforward knowledge-free labels: i) the uniform label, e.g., $[0.5,0.5]^{\top}$ , and ii) the random label, e.g., $[r\sim\mathcal{U}(0,1),1-r]^{\top}$ where $r$ is a random value sampled from a uniform distribution $\mathcal{U}(0,1)$ . We compared with both types of the above labels in our ablation study (Section 5.5.1).

4.2.2. Easy-to-Learn

The above two introduced knowledge-free labels do not take prior knowledge of models into consideration, which makes them hard to learn by the model. Our empirical study (Figure 7) shows that both the uniform and random labels have residual old memories and unstable performance. These hard-to-learn labels cause two potential problems during unlearning: i) the incapacity of completely removing the influence of target data, and ii) increasing the computational overhead for the unlearning process.

To make it easier to learn, we distill the prior knowledge from the model into labels. Specifically, we adopt a teacher-student learning pattern (Hinton et al., 2015). We first initialize a set of untrained models as teacher models, since they have no learned knowledge about training data. Then we feed teacher models with features of target data, producing the teacher predictions. Finally, the teacher label is obtained by averaging the teacher predictions. Formally, the teacher label $\hat{y}$ is computed as $\hat{y}=\frac{1}{Q}\sum_{i=1}^{Q}\theta_{i}(x)$ , where $x$ is data features and $Q$ is the number of teachers. Note that the teacher models do not require training and can be released after generating the fake labels. Thus, the memory generator has both space and time advantages over the space-for-time strategy in existing retrain unlearning methods.

However, the prior knowledge can be so strong that it results in bias on particular data. The bias is added purely by the model or algorithm, which is referred to as algorithmic bias (Mehrabi et al., 2021). Our empirical study (Figure 7) shows that the untrained model naturally has prediction bias on particular classes, which makes the testing accuracy of these classes relatively high. This potentially leaks knowledge about the data, which is conflicted with the primary characteristic, i.e., knowledge-free. Consequently, we import a debias vector $\nu$ to underweight the target class (Espín-Noboa et al., 2021). Each element of $\nu$ represents the propensity score $\nu_{i}$ of the corresponding class. We set the propensity score of target class $\nu_{\text{target}}=\sigma\in[0,1]$ and the other classes $\nu_{\text{non-target}}=1$ . Formally, debias teacher label is computed as

(3)

\tilde{y}=\text{debias}(\hat{y})=\frac{\nu\hat{y}}{|\nu\hat{y}|_{1}},\enspace\nu=\boldsymbol{1}+(\sigma-1)y,

where $\boldsymbol{1}$ is an all-one vector, $y$ is the original label, and $\sigma$ is debias weight. $\nu\hat{y}$ is scaled by its L1 norm to satisfy the summation constraint, i.e., $\sum y=1$ . The debias weight can be dynamically determined by the quotient of the target prediction weight and average prediction weight. In this way, we enjoy a balanced trade-off between the characteristics of knowledge-free and easy-to-learn.

4.3. Knowledge Preserver

While unlearning target data, we also need to preserve the remaining knowledge learned from non-target data. Conventional deep learning methods suffer from the phenomenon of catastrophic forgetting when incrementally learning a sequence of tasks. To imitate active forgetting in the human brain, our proposed FedAF is based on the idea of incremental learning. The target client $k$ sequentially trains the model on two tasks, i.e., $\mathcal{D}_{k}$ and $\mathcal{M}$ . To avoid confusion, we directly name the task by its dataset. Tasks $\mathcal{D}_{k}$ and $\mathcal{M}$ stand for the learning and unlearning process, respectively. Catastrophic forgetting happens when training on the latter task, i.e., $\mathcal{M}$ . The new memories not only overwrite the old memories of target data, but also make the model forget the memories of non-target data. Thus, we introduce EWC training to alleviate this phenomenon.

As introduced in Section 4.1, we do not change the learning process, which means that task $\mathcal{D}_{k}$ adopts conventional training. In this paper, we take empirical risk minimization as an example. The client $k$ optimizes model $\theta$ by minimizing the loss of task $\mathcal{D}_{k}$ as follows:

(4)

L_{\mathcal{D}_{k}}(\theta)=\frac{1}{|\mathcal{D}_{k}|}\sum_{x\in\mathcal{D}_{k}}\ell(\theta,x).

At the end of training, $\theta$ is supposed to converge to a solution space $\theta^{*}_{\mathcal{D}_{k}}$ where $\forall\theta\in\theta^{*}_{\mathcal{D}_{k}}$ is an acceptable solution for task $\mathcal{D}_{k}$ .

The next step is the unlearning process, where we train on task $\mathcal{R}$ . We explain the difference between conventional training and EWC training as follows.

Conventional Training. The model $\theta^{*}_{\mathcal{D}_{k}}$ continues training by minimizing the loss of task $\mathcal{M}$ , i.e., $L_{\mathcal{M}}(\theta)$ . As shown in Figure 3, conventional training methods suffer from catastrophic forgetting, which is detrimental to model performance.

EWC Training. A straightforward way to preserve knowledge is to train the model on both tasks simultaneously. As shown in Figure 3, this method is based on the assumption that there is an overlapping solution space of both tasks. We empirically validate this assumption in Section 5.5.1. Ignoring the scaling term $1/|\mathcal{D}^{\prime},\mathcal{M}|$ , the overall loss on both tasks is defined as

(5)

L_{\mathcal{D}^{\prime},\mathcal{M}}(\theta)=L_{\mathcal{M}}(\theta)+L_{\mathcal{D}^{\prime}}(\theta),

where $\mathcal{D}^{\prime}=\mathcal{D}_{k}\backslash\mathcal{R}$ denotes the task of non-target data, since we are required to unlearn the target data $\mathcal{R}$ . However, computing the loss of $\mathcal{D}^{\prime}$ requires non-target data, which is against our intention of avoiding extra computational overhead. EWC provides us an approximation of $L_{\mathcal{D}_{k}}(\theta)$ when having $\theta^{*}_{\mathcal{D}_{k}}$ , but not having access to $\mathcal{D}_{k}$ . The overall loss $L_{\mathcal{D}_{k},\mathcal{M}}(\theta)$ is approximated by computing the second order Taylor expansion of $L_{\mathcal{D}_{k}}(\theta)$ at $\theta^{*}_{\mathcal{D}_{k}}$ as follows:

(6)		$\displaystyle L_{\mathcal{D}_{k},\mathcal{M}}(\theta)$	$\displaystyle=L_{\mathcal{M}}(\theta)+L_{\mathcal{D}_{k}}(\theta)$
		$\displaystyle\approx L_{\mathcal{M}}(\theta)+\frac{\lambda}{2}(\theta-\theta^{}_{\mathcal{D}_{k}})^{\top}H_{\mathcal{D}_{k}}(\theta-\theta^{}_{\mathcal{D}_{k}})+\epsilon,$

where $\lambda$ is a hyper-parameter introduced to have a trade-off between learning $\mathcal{M}$ and not forgetting $\mathcal{D}_{k}$ , $H_{\mathcal{D}_{k}}=\partial^{2}L(\theta^{*}_{\mathcal{D}_{k}})/\partial^{2}\theta^{*}_{\mathcal{D}_{k}}$ denotes Hessian matrix at $\theta^{*}_{\mathcal{D}_{k}}$ , and $\epsilon$ accounts for all constants. Please refer to (Kolouri et al., 2020) for more details of derivation. As shown in Eq (6), Hessian matrix can be interpreted as regularizer weight matrix of $(\theta-\theta^{*}_{\mathcal{D}_{k}})$ , where each element represents synaptic importance of a parameter. The more important the parameter is, the more regularizer weight it should be added to, making the parameter change less when learning $\mathcal{M}$ . The Hessian can be efficiently computed by Fisher information matrix from first order derivatives (Martens, 2020).

Based on the above approximation of $L_{\mathcal{D}_{k}}(\theta)$ , we compute the overall loss on tasks $\mathcal{D}^{\prime}$ and $\mathcal{M}$ , i.e., the unlearning loss, as follows:

(7)		$\displaystyle L_{\mathcal{D}^{\prime},\mathcal{M}}(\theta)$	$\displaystyle=L_{\mathcal{M}}(\theta)+L_{\mathcal{D}^{\prime}}(\theta)=L_{\mathcal{M}}(\theta)+L_{\mathcal{D}_{k}}(\theta)-L_{\mathcal{R}}(\theta)$
		$\displaystyle\approx L_{\mathcal{M}}(\theta)-L_{\mathcal{R}}(\theta)+\frac{\lambda}{2}(\theta-\theta^{}_{\mathcal{D}_{k}})^{\top}H_{\mathcal{D}_{k}}(\theta-\theta^{}_{\mathcal{D}_{k}})+\epsilon,$

where $\mathcal{R}$ and $\mathcal{M}$ denote the original target data (old memories) and the manipulated target data (new memories) respectively. Consequently, the model $\theta^{*}_{\mathcal{D}_{k}}$ continues training on our proposed new loss to alleviate the phenomenon of catastrophic forgetting. In this way, the knowledge preserver can unlearn the target data without breaking the original training procedure.

5. Experiments

We evaluate the effectiveness of our proposed method on three widely used datasets based on three principles, i.e., unlearning completeness, unlearning efficiency, and model utility. To further investigate our proposed method, we also conduct an ablation study.

5.1. Dataset

Our experiments are conducted on three benchmark datasets, i.e., MNIST (LeCun et al., 1998), CIFAR10 (Krizhevsky et al., 2009), and CelebA (Liu et al., 2015). We provide the information about these datasets in Table 1. For MNIST and CIFAR10, following their original papers, we leave 10,000 samples for testing and use the others for training. For CelebA, we select two representative attributes, i.e., Male and Mouth_Slightly_Open. We label the samples based on whether they possess these attributes or not, which results in four distinct classes, i.e., having both attributes, having none of the attributes, only having the Male attribute, and only having the Mouth_Slightly_Open attribute. We use 80%, 10%, and 10% of the original dataset for training, validation, and testing, respectively.

Table 1. Summary of datasets.

Dataset	Dimensionality	Size	# Classes
MNIST	28 $\times$ 28	70,000	10
CIFAR10	32 $\times$ 32 $\times$ 3	60,000	10
CelebA	178 $\times$ 218 $\times$ 3	202,599	4

5.2. Experimental Settings

Model. We use a network that consists of 2 convolutional layers followed by 1 fully-connected layer for MNIST, ResNet-10 (He et al., 2016) for CIFAR10 and CelebA. For simplicity, we did not sufficiently fine-tune the models to get their optimal performance, which is not the focus of this paper.

Training Details. We initialize all model parameters, including teacher models, with Xavier Initialization (Glorot and Bengio, 2010). As initialization involves randomness, we run all models for 10 trials and report the average results. We adopt the cross entropy loss function and stochastic gradient descent to train the models. The learning rate is set as 0.001 for MNIST, 0.01 for CIFAR10, and 0.01 for CelebA. In this paper, we conduct class-wise unlearning, and unlearn one class of a specific client at one time for all classes. All models are implemented with PyTorch. We run all experiments on an Ubuntu 20.04 LTS system server with 256GB RAM, and NVIDIA GeForce RTX 3090 GPU.

Federated Settings. For federated learning, we adopt the widely acknowledged FedAvg (McMahan et al., 2017). Specifically, we set the number of clients as 4, and the local epoch as 1. Ensuring convergence of the global model, the maximum round of federated iteration is set as 5, 20, and 10 for MNIST, CIFAR10, and CelebA, respectively.

Hyper-parameters. In our proposed FedAF, there are three hyper-parameters, i.e., EWC training epoch, trade-off coefficient $\lambda$ , and the number of teacher models $Q$ . For the EWC training epoch, we investigate it in {1, 2, 3, 4, 5}. For $\lambda$ , we investigate it in {0.1, 0.5, 1, 10, 50}. Based on empirical results, we set the EWC training epoch as 1 and $\lambda$ as 10 for the following experiments in this paper. We report empirical results in Appendix A. For $Q$ , we investigate it in {5, 10, 15, 20} and finally set $Q=10$ , since we observe that a larger number over 10 cannot significantly improve the overall performance of teacher models.

Table 2. Accuracy (Acc) of unlearning different classes. The accuracy of testing data (Test Acc) before unlearning is i) MNIST: 98.30

\pm

0.21, ii) CIFAR10: 82.04

\pm

0.36, and iii) CelebA: 90.31

\pm

0.22. The BD Acc and Test Acc are used to evaluate unlearning completeness and model utility respectively. We bold the top results of each class, other than Retrain.

MNIST
Class ID	BD Acc (%)	BD Acc (%) after unlearning				Test Acc (%) after unlearning
Class ID	before unlearning	Retrain	FRR	FedAF	FedAF-C	Retrain	FRR	FedAF	FedAF-C
0	81.14 $\pm$ 2.86	0.12 $\pm$ 0.01	0.73 $\pm$ 0.13	0.58 $\pm$ 0.18	0.60 $\pm$ 0.35	96.83 $\pm$ 0.13	91.14 $\pm$ 0.12	92.22 $\pm$ 0.26	40.03 $\pm$ 0.51
1	81.57 $\pm$ 3.32	0.11 $\pm$ 0.14	0.55 $\pm$ 0.08	0.82 $\pm$ 0.43	0.68 $\pm$ 0.52	96.84 $\pm$ 0.07	91.06 $\pm$ 0.12	92.29 $\pm$ 0.61	39.99 $\pm$ 0.41
2	81.58 $\pm$ 3.29	0.03 $\pm$ 0.00	0.94 $\pm$ 0.15	0.65 $\pm$ 0.41	0.40 $\pm$ 0.35	96.39 $\pm$ 0.18	90.75 $\pm$ 0.20	92.06 $\pm$ 0.39	39.75 $\pm$ 0.32
3	80.73 $\pm$ 3.19	0.08 $\pm$ 0.01	0.62 $\pm$ 0.04	0.79 $\pm$ 0.45	0.71 $\pm$ 0.58	96.68 $\pm$ 0.07	90.42 $\pm$ 0.28	92.19 $\pm$ 0.38	39.94 $\pm$ 0.33
4	81.16 $\pm$ 2.87	0.01 $\pm$ 0.00	0.60 $\pm$ 0.02	0.39 $\pm$ 0.05	0.93 $\pm$ 0.18	96.42 $\pm$ 0.10	90.52 $\pm$ 0.15	92.12 $\pm$ 0.64	39.92 $\pm$ 0.62
5	80.82 $\pm$ 3.13	0.37 $\pm$ 0.15	0.97 $\pm$ 0.20	0.57 $\pm$ 0.32	0.62 $\pm$ 0.41	96.78 $\pm$ 0.13	90.42 $\pm$ 0.22	92.12 $\pm$ 0.62	40.10 $\pm$ 0.24
6	80.25 $\pm$ 4.06	0.04 $\pm$ 0.00	1.04 $\pm$ 0.17	0.73 $\pm$ 0.20	0.55 $\pm$ 0.07	96.74 $\pm$ 0.05	91.03 $\pm$ 0.21	91.85 $\pm$ 0.60	40.17 $\pm$ 0.47
7	80.41 $\pm$ 3.50	0.27 $\pm$ 0.15	1.71 $\pm$ 0.54	1.27 $\pm$ 0.62	1.29 $\pm$ 0.45	96.54 $\pm$ 0.15	91.01 $\pm$ 0.14	92.04 $\pm$ 0.24	39.94 $\pm$ 0.45
8	81.86 $\pm$ 3.57	0.13 $\pm$ 0.04	0.77 $\pm$ 0.21	0.82 $\pm$ 0.18	0.43 $\pm$ 0.10	96.68 $\pm$ 0.24	90.21 $\pm$ 0.22	91.90 $\pm$ 0.42	40.11 $\pm$ 0.71
9	79.60 $\pm$ 5.30	0.22 $\pm$ 0.06	1.41 $\pm$ 0.47	1.05 $\pm$ 0.50	1.33 $\pm$ 0.22	96.38 $\pm$ 0.21	89.76 $\pm$ 0.34	92.38 $\pm$ 0.65	40.07 $\pm$ 0.37
CIFAR10
0	71.00 $\pm$ 1.73	0.01 $\pm$ 0.00	1.64 $\pm$ 1.05	1.45 $\pm$ 1.02	1.45 $\pm$ 1.87	81.03 $\pm$ 0.63	80.04 $\pm$ 0.61	79.55 $\pm$ 1.92	29.91 $\pm$ 6.74
1	70.59 $\pm$ 4.62	1.63 $\pm$ 1.28	2.38 $\pm$ 1.91	2.81 $\pm$ 1.80	2.17 $\pm$ 1.44	80.84 $\pm$ 0.27	79.04 $\pm$ 0.31	79.76 $\pm$ 0.96	28.27 $\pm$ 4.11
2	74.17 $\pm$ 4.02	0.86 $\pm$ 0.08	1.88 $\pm$ 0.91	1.20 $\pm$ 0.91	1.50 $\pm$ 0.48	81.78 $\pm$ 0.57	81.50 $\pm$ 0.56	80.82 $\pm$ 0.75	27.87 $\pm$ 3.60
3	69.85 $\pm$ 1.29	1.59 $\pm$ 1.25	2.15 $\pm$ 1.71	1.76 $\pm$ 0.83	1.79 $\pm$ 1.10	81.61 $\pm$ 1.06	80.60 $\pm$ 1.09	80.63 $\pm$ 1.76	28.18 $\pm$ 8.25
4	70.55 $\pm$ 4.44	0.06 $\pm$ 0.00	1.34 $\pm$ 1.04	1.30 $\pm$ 1.07	1.55 $\pm$ 0.95	81.25 $\pm$ 0.40	80.15 $\pm$ 0.35	80.27 $\pm$ 0.81	27.07 $\pm$ 4.61
5	74.66 $\pm$ 1.07	0.03 $\pm$ 0.01	1.34 $\pm$ 1.03	1.70 $\pm$ 1.12	1.96 $\pm$ 1.24	80.57 $\pm$ 0.25	79.59 $\pm$ 0.25	80.50 $\pm$ 1.74	27.85 $\pm$ 5.35
6	70.38 $\pm$ 2.42	0.09 $\pm$ 0.01	1.30 $\pm$ 0.94	1.28 $\pm$ 1.06	1.76 $\pm$ 1.19	81.17 $\pm$ 0.81	80.18 $\pm$ 0.84	81.05 $\pm$ 1.02	30.19 $\pm$ 4.42
7	73.66 $\pm$ 6.68	0.05 $\pm$ 0.00	1.34 $\pm$ 1.05	1.53 $\pm$ 1.22	1.56 $\pm$ 1.12	80.90 $\pm$ 0.42	80.44 $\pm$ 0.41	80.78 $\pm$ 0.50	25.45 $\pm$ 2.42
8	79.05 $\pm$ 4.81	0.02 $\pm$ 0.00	1.54 $\pm$ 0.92	1.41 $\pm$ 0.66	1.84 $\pm$ 0.85	80.71 $\pm$ 0.29	79.68 $\pm$ 0.33	80.37 $\pm$ 0.25	29.14 $\pm$ 3.48
9	81.70 $\pm$ 6.54	1.08 $\pm$ 0.07	1.81 $\pm$ 1.07	1.55 $\pm$ 1.15	1.75 $\pm$ 1.46	80.54 $\pm$ 0.57	79.45 $\pm$ 0.64	80.28 $\pm$ 1.12	29.14 $\pm$ 2.67
CelebA
0	74.16 $\pm$ 0.37	0.68 $\pm$ 0.30	2.18 $\pm$ 1.30	1.91 $\pm$ 1.32	1.76 $\pm$ 1.31	88.71 $\pm$ 0.36	88.69 $\pm$ 0.39	88.44 $\pm$ 0.47	31.29 $\pm$ 4.91
1	74.58 $\pm$ 1.65	0.39 $\pm$ 0.35	1.43 $\pm$ 0.86	1.34 $\pm$ 1.15	1.29 $\pm$ 1.14	89.29 $\pm$ 0.24	89.09 $\pm$ 0.26	89.10 $\pm$ 1.18	32.35 $\pm$ 2.73
2	84.55 $\pm$ 3.40	1.33 $\pm$ 0.61	2.58 $\pm$ 1.44	2.43 $\pm$ 1.65	2.52 $\pm$ 2.26	86.15 $\pm$ 0.25	86.10 $\pm$ 0.29	85.14 $\pm$ 0.31	29.99 $\pm$ 4.11
3	62.13 $\pm$ 6.33	0.63 $\pm$ 0.80	1.94 $\pm$ 1.31	1.98 $\pm$ 1.73	1.82 $\pm$ 1.24	89.34 $\pm$ 0.37	88.11 $\pm$ 0.20	89.74 $\pm$ 0.56	29.83 $\pm$ 3.94

5.3. Compared Methods

We compare our proposed FedAF with two representative unlearning methods which are applicable for class-wise unlearning.

•

Retrain: Retraining from scratch is the ground-truth unlearning method with heavy computational overhead.
•

FRR (Liu et al., 2022): Federated Rapid Retraining (FRR) is the State-Of-The-Art (SOTA) sample-wise federated unlearning method, which can be applied to class-wise unlearning. We implement FRR using the published code.

5.4. Results and Discussions

5.4.1. Unlearning Completeness

Following (Sommer et al., 2022; Hu et al., 2022), we utilize backdoor attack (Gu et al., 2019) to evaluate the unlearning completeness. Specifically, we implant backdoor triggers into the target data that we aim to unlearn, and flip their labels to a random class. Through this process, the backdoor attack enforces the model to build a mapping between the trigger pattern and the flipped label. As a consequence, we can evaluate unlearning completeness by comparing the performance of target data before and after unlearning. We report BackDoor Accuracy (BD Acc) of the unlearned data in Table 2. From it, we have the following observations: i) All compared methods significantly decrease the BD Acc after unlearning (from 76.21% to 1.00% on average), indicating that the influence of target data is significantly reduced; ii) FRR and FedAF cannot achieve as low BD Acc (after unlearning) as Retrain. Compared with the BD Acc of Retrain, FRR and FedAF have an average increase of 1.02% and 0.90%, respectively; iii) There is only a marginal difference between FRR and FedAF, regarding performance on completeness. Overall, FedAF slightly outperforms the SOTA method in terms of unlearning completeness.

Table 3. Running time of unlearning one class in a client.

Time (s)	Retrain	FRR	FedAF
MNIST	24.55	28.60	1.62
CIFAR10	891.46	3,554.32	9.29
CelebA	1,774.75	16,696.48	57.20

5.4.2. Unlearning Efficiency

We use running time to measure the unlearning efficiency of the compared methods. Specifically, we report the average running time of unlearning the first class in a specific client in Table 3. As shown in Table 3, our proposed FedAF on average improves the efficiency of Retrain by 39.51 times. On the contrary, FRR spends a significantly greater amount of time than Retrain due to the precise computation required by the Newton optimizer. This is totally unacceptable because the design ethos behind unlearning methods is to achieve more efficient unlearning than Retrain.

5.4.3. Model Utility

As we mentioned in Section 2.2, preserving the model utility is also an important principle of federated unlearning. An adequate unlearning method is supposed to avoid over-remove the influence of target data. Therefore, we compare the after-unlearning model utility by evaluating the model performance on the testing set, and report the results in Table 2. From it, we observe that all compared methods, i.e., Retrain, FRR, and FedAF, achieve Test Acc that is close to what was achieved before unlearning. Specifically, regarding Retrain as a baseline, FRR and FedAF can limit the difference between their Test Acc and that of Retrain to below 3.12% and 2.31% respectively, indicating that there is no significant forgetting of the non-target data.

5.5. Ablation Study

5.5.1. Memory Generator

We conduct an overlapping validation experiment for two purposes: i) to empirically validate that there is an overlapping solution space for tasks $\mathcal{D}^{\prime}$ and $\mathcal{M}$ . As we mentioned in Section 4.3, the existence of overlapping solution space is one of the most fundamental requirements for EWC training, and ii) to compare the performance of the aforementioned labels, including two benchmark labels, i.e., uniform label and random label, and two our proposed labels, i.e., teacher label and debias teacher label. Thus, we train the model on the mixed task $\mathcal{D}^{\prime}+\mathcal{M}$ , and observe the performance, which indicates the upper bound of EWC training. For task $\mathcal{D}^{\prime}$ , we remove the target class $\mathcal{R}$ from the original local dataset $\mathcal{D}_{k}$ . For task $\mathcal{M}$ , we replace the label of the target class with a fake label to construct manipulated data. We mix the data from the above two tasks for training and test the model on the testing set. Manipulating one class at each time, we report the accuracy of the target class in Figure 7 and the accuracy of non-target data in Appendix B.

If the overlapping solution space exists, the model is supposed to achieve desirable performance on both tasks, i.e., i) on task $\mathcal{D}^{\prime}$ : having high accuracy on non-target data, and ii) on task $\mathcal{M}$ : having low accuracy on target class, which means it has no knowledge about target class. From the accuracy of non-target data, we observe that all types of labels achieve high accuracy, indicating that using whichever label can learn the knowledge from task $\mathcal{D}^{\prime}$ . From Figure 7, we observe that uniform and random labels fail to have the overlapping solution space. They suffer from the phenomenon of intransigence. They achieve considerably high accuracy and only have low accuracy in few classes, which means they have residual knowledge of the target class. Their performance is also unstable, which proves that they are hard to learn. Teacher label has low accuracy in most classes. However, it fails in a few classes. This is because the untrained model potentially has prediction bias on particular classes, making the accuracy on these classes relatively high (Espín-Noboa et al., 2021). Debias teacher label has low testing accuracy for all classes, proving that it exists an overlapping solution space robustly.

5.5.2. Knowledge Preserver

To investigate the effectiveness of EWC, we compare FedAF with FedAF-C which continually trains the model with conventional loss. As shown in Table 2 (after unlearning), FedAF-C achieves comparable DB Acc as Retrain. But it also shows a decrease in Test Acc by 62.14% when compared with Retrain. This indicates that FedAF-C unlearns the target data, but also forgets some of the non-target data (catastrophic forgetting). As for comparison, our proposed FedAF utilizes EWC training to alleviate catastrophic forgetting.

6. Conclusion and Future Work

In this paper, we propose a novel federated unlearning method, named FedAF. Inspired by active forgetting in neurology, FedAF unlearns by leveraging new memories to overwrite the old ones, which fundamentally overcomes the limitations of existing federated unlearning methods, without requiring the storage of historical updates or the estimation of data influence. In general, we build FedAF based on the idea of incremental learning to achieve active forgetting. Specifically, FedAF consists of two modules in each client, i.e., memory generator and knowledge preserver. Memory generator produces knowledge-free and easy-to-learn fake labels based on teacher-student learning, and pairs them with the target data features to generate new memories. Knowledge preserver continually trains the model with new memories, and adopts EWC training with our derived loss to alleviate catastrophic forgetting. The target client triggers these two modules by submitting unlearning requests, while other clients follow the standard training process. Experiments conducted on three benchmark datasets demonstrate that our proposed FedAF can not only efficiently unlearn the target data of a specific client from the global model, but also preserve the knowledge of non-target data.

Appendix A Hyper-parameters

We empirically investigate the effect of two hyper-parameters, i.e., EWC training epoch and trade-off coefficient $\lambda$ . To properly achieve unlearning, our proposed framework is supposed to have i) low accuracy on target data (ideally 0% accuracy), and ii) relatively high accuracy on non-target data (comparable results with learning process). Therefore, we choose adequate hyper-parameters based on the above two metrics.

A.1. EWC Training Epoch

For the EWC training epoch, we investigate it in {1, 2, 3, 4, 5} for all datasets.

Target data. Figures 5(a), 5(c), and 5(e) report the accuracy of target data. From them, we observe that EWC training can already achieve almost 0% accuracy with the minimal epoch options on both datasets.

Non-target data. Figures 5(b), 5(d), and 5(f) report the accuracy of non-target data. From them, we observe that EWC training achieves the best non-target accuracy with the minimal epoch options on both datasets, and then gradually declines.

Summary. Therefore, we choose the minimal epoch options, setting EWC training epoch as 1.

A.2. Trade-off Coefficient $\lambda$

For $\lambda$ , we investigate it in {0.1, 0.5, 1, 10, 50} for all datasets and report the results in Figure 6. $\lambda$ balances the trade-off between unlearning target data (low accuracy on target data) and not forgetting non-target data (high accuracy on non-target data).

Target data. As it is shown in Figures 6(a), 6(c), and 6(e), when $\lambda\leq 10$ , our proposed framework can achieve almost 0% accuracy of target data on both datasets. The target data accuracy gradually grows when further increasing $\lambda$ , which means the model does not completely unlearn the target data.

Non-target data. As it is shown in Figures 6(b), 6(d), and 6(f), the accuracy of non-target data increases with the value of $\lambda$ , but the degree of increase decreases gradually.

Summary. Based on the empirical results in Figure 6, we set $\lambda$ as 10, since it strikes a good balance.

Appendix B Overlapping Validation

Manipulating one class at each time, we report the accuracy of non-target data in Table 4 and the results of target data (CIFAR10 and CelebA) in Figure 7. As shown in Table 4 all types of labels achieve high accuracy, indicating that using whichever label can learn the knowledge from task $\mathcal{D}^{\prime}$ . The results presented in Figure 7 demonstrate a consistent observation with those found in experiments on MNIST. Specifically, we observe that uniform and random labels do not yield an overlapping solution space. While the teacher label exhibits low accuracy across most classes, it does demonstrate a bias towards certain ones (relatively high accuracy).

Table 4. Non-target data accuracy (%) of different types of fake labels when manipulating each class.

Class ID	MNIST				CIFAR10				CelebA
Class ID	Uniform	Random	Teacher	Debias	Uniform	Random	Teacher	Debias	Uniform	Random	Teacher	Debias
0	98.9 $\pm$ 0.1	98.9 $\pm$ 0.1	98.9 $\pm$ 0.1	98.9 $\pm$ 0.1	91.1 $\pm$ 0.6	90.8 $\pm$ 0.6	90.6 $\pm$ 0.6	90.8 $\pm$ 0.7	95.0 $\pm$ 0.6	94.9 $\pm$ 0.5	95.1 $\pm$ 0.6	95.2 $\pm$ 0.7
1	98.7 $\pm$ 0.1	98.7 $\pm$ 0.2	98.7 $\pm$ 0.2	98.8 $\pm$ 0.1	89.6 $\pm$ 0.9	89.5 $\pm$ 0.7	89.7 $\pm$ 0.5	90.0 $\pm$ 0.6	95.1 $\pm$ 0.5	95.0 $\pm$ 0.6	95.2 $\pm$ 0.4	95.2 $\pm$ 0.5
2	99.0 $\pm$ 0.1	99.0 $\pm$ 0.2	98.9 $\pm$ 0.2	99.0 $\pm$ 0.1	92.1 $\pm$ 0.6	91.9 $\pm$ 1.0	91.7 $\pm$ 0.6	91.6 $\pm$ 0.9	95.0 $\pm$ 0.4	95.2 $\pm$ 0.7	94.9 $\pm$ 0.5	95.1 $\pm$ 0.7
3	98.9 $\pm$ 0.1	99.0 $\pm$ 0.1	98.9 $\pm$ 0.2	98.9 $\pm$ 0.1	93.7 $\pm$ 0.9	93.7 $\pm$ 0.8	91.0 $\pm$ 0.6	91.8 $\pm$ 0.6	95.2 $\pm$ 0.4	94.8 $\pm$ 0.4	95.1 $\pm$ 0.6	95.2 $\pm$ 0.6
4	98.9 $\pm$ 0.3	99.0 $\pm$ 0.1	98.9 $\pm$ 0.1	99.0 $\pm$ 0.1	90.4 $\pm$ 1.9	91.2 $\pm$ 0.8	91.1 $\pm$ 0.6	90.4 $\pm$ 1.2	-	-	-	-
5	99.0 $\pm$ 0.1	99.0 $\pm$ 0.1	99.0 $\pm$ 0.1	99.0 $\pm$ 0.1	92.5 $\pm$ 0.3	92.9 $\pm$ 0.7	91.8 $\pm$ 1.8	91.9 $\pm$ 0.5	-	-	-	-
6	99.0 $\pm$ 0.2	99.0 $\pm$ 0.1	99.0 $\pm$ 0.1	98.8 $\pm$ 0.2	89.8 $\pm$ 1.7	90.2 $\pm$ 1.0	90.4 $\pm$ 0.5	90.4 $\pm$ 0.3	-	-	-	-
7	99.0 $\pm$ 0.2	99.1 $\pm$ 0.1	99.0 $\pm$ 0.1	99.0 $\pm$ 0.2	89.9 $\pm$ 1.0	90.2 $\pm$ 0.3	90.1 $\pm$ 0.5	90.2 $\pm$ 1.0	-	-	-	-
8	99.1 $\pm$ 0.1	99.1 $\pm$ 0.1	99.1 $\pm$ 0.1	99.0 $\pm$ 0.1	89.7 $\pm$ 0.5	90.5 $\pm$ 0.5	89.7 $\pm$ 0.6	89.8 $\pm$ 1.0	-	-	-	-
9	99.1 $\pm$ 0.1	99.1 $\pm$ 0.1	99.2 $\pm$ 0.1	99.1 $\pm$ 0.1	90.7 $\pm$ 0.4	90.6 $\pm$ 0.4	90.4 $\pm$ 0.5	90.3 $\pm$ 0.4	-	-	-	-

References

(1)
Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In ECCV. 139–154.
Anderson and Hulbert (2021) Michael C Anderson and Justin C Hulbert. 2021. Active forgetting: Adaptation of memory by prefrontal control. Annual Review of Psychology (2021).
Basu et al. (2021) S Basu, P Pope, and S Feizi. 2021. Influence Functions in Deep Learning Are Fragile. In ICLR.
Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In Proceedings in the 42nd IEEE Symposium on Security and Privacy (SP).
Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In Proceedings in the 36th IEEE Symposium on Security and Privacy (SP). 463–480.
Chen et al. (2022) Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. 2022. Recommendation unlearning. In Proceedings of the ACM Web Conference 2022. 2768–2777.
Chen et al. (2021) Pei-Hung Chen, Wei Wei, Cho-Jui Hsieh, and Bo Dai. 2021. Overcoming catastrophic forgetting by bayesian generative regularization. In ICML. PMLR, 1760–1770.
Davis and Zhong (2017) Ronald L Davis and Yi Zhong. 2017. The biology of forgetting—a perspective. Neuron 95, 3 (2017), 490–503.
Espín-Noboa et al. (2021) Lisette Espín-Noboa, Fariba Karimi, Bruno Ribeiro, Kristina Lerman, and Claudia Wagner. 2021. Explaining classification performance and bias via network structure and sampling technique. Applied Network Science 6, 1 (2021), 1–25.
Gao et al. (2022) Xiangshan Gao, Xingjun Ma, Jingyi Wang, Youcheng Sun, Bo Li, Shouling Ji, Peng Cheng, and Jiming Chen. 2022. VeriFi: Towards Verifiable Federated Unlearning. CoRR abs/2205.12709 (2022). https://doi.org/10.48550/arXiv.2205.12709 arXiv:2205.12709
Ginart et al. (2019) Tony Ginart, Melody Guan, Gerg Valiant, and James Zou. 2019. Making AI forget you: Data deletion in machine learning. In NeurIPS.
Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 249–256.
Graves et al. (2021) Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021. Amnesiac machine learning. In AAAI, Vol. 35. 11516–11524.
Gu et al. (2019) Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7 (2019), 47230–47244.
Guo et al. (2021) Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2021. FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 10333–10350.
Halimi et al. (2022) Anisa Halimi, Swanand Kadhe, Ambrish Rawat, and Nathalie Baracaldo. 2022. Federated Unlearning: How to Efficiently Erase a Client in FL? CoRR abs/2207.05521 (2022). https://doi.org/10.48550/arXiv.2207.05521 arXiv:2207.05521
He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In ICCV. 770–778.
Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. stat 1050 (2015), 9.
Hu et al. (2022) Hongsheng Hu, Zoran Salcic, Gillian Dobbie, Jinjun Chen, Lichao Sun, and Xuyun Zhang. 2022. Membership Inference via Backdooring. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, Luc De Raedt (Ed.). ijcai.org, 3832–3838. https://doi.org/10.24963/ijcai.2022/532
Hu et al. (2019) Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao, Jinwen Ma, Dongyan Zhao, and Rui Yan. 2019. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR.
Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021. Highly accurate protein structure prediction with AlphaFold. Nature 596, 7873 (2021), 583–589.
Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In ICML. PMLR, 1885–1894.
Koh et al. (2019) Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. 2019. On the Accuracy of Influence Functions for Measuring Group Effects. Advances in Neural Information Processing Systems 32 (2019), 5254–5264.
Kolouri et al. (2020) Soheil Kolouri, Nicholas A Ketz, Andrea Soltoggio, and Praveen K Pilly. 2020. Sliced cramer synaptic consolidation for preserving deeply learned representations. In ICLR.
Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444.
LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324.
Li et al. (2016) Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. 2016. Data poisoning attacks on factorization-based collaborative filtering. Advances in neural information processing systems 29 (2016).
Li et al. (2019) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In ICML. PMLR, 3925–3934.
Liu et al. (2021) Gaoyang Liu, Xiaoqiang Ma, Yang Yang, Chen Wang, and Jiangchuan Liu. 2021. FedEraser: Enabling Efficient Client-Level Data Removal from Federated Learning Models. In 29th IEEE/ACM International Symposium on Quality of Service, IWQOS 2021, Tokyo, Japan, June 25-28, 2021. IEEE, 1–10. https://doi.org/10.1109/IWQOS52092.2021.9521274
Liu et al. (2022) Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. 2022. The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retraining. In IEEE INFOCOM 2022 - IEEE Conference on Computer Communications, London, United Kingdom, May 2-5, 2022. IEEE, 1749–1758.
Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In ICCV.
Martens (2020) James Martens. 2020. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research 21, 1 (2020), 5776–5851.
McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273–1282.
Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54, 6 (2021), 1–35.
Mehta et al. (2022) Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. 2022. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10422–10431.
OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774
Pardau (2018) Stuart L Pardau. 2018. The California consumer privacy act: Towards a European-style privacy regime in the United States. J. Tech. L. & Pol’y 23 (2018), 68.
Schelter et al. (2021) Sebastian Schelter, Stefan Grafberger, and Ted Dunning. 2021. HedgeCut: Maintaining Randomised Trees for Low-Latency Machine Unlearning. In Proceedings of the 2021 International Conference on Management of Data. 1545–1557.
Schwarz et al. (2018) Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & compress: A scalable framework for continual learning. In ICML. PMLR, 4528–4537.
Sekhari et al. (2021) Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems 34 (2021).
Sommer et al. (2022) David Marco Sommer, Liwei Song, Sameer Wagh, and Prateek Mittal. 2022. Athena: Probabilistic Verification of Machine Unlearning. Proceedings on Privacy Enhancing Technologies 3 (2022), 268–290.
Voigt and Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10–5555.
Wang et al. (2022) Junxiao Wang, Song Guo, Xin Xie, and Heng Qi. 2022. Federated Unlearning via Class-Discriminative Pruning. In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 622–632. https://doi.org/10.1145/3485447.3512222
Wu et al. (2018) Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. 2018. Memory replay gans: Learning to generate new categories without forgetting. In NeurIPS, Vol. 31.
Wu et al. (2022c) Chen Wu, Sencun Zhu, and Prasenjit Mitra. 2022c. Federated Unlearning with Knowledge Distillation. CoRR abs/2201.09441 (2022). arXiv:2201.09441 https://arxiv.org/abs/2201.09441
Wu et al. (2022b) Ga Wu, Masoud Hashemi, and Christopher Srinivasa. 2022b. Puma: Performance unchanged model augmentation for training data removal. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8675–8682.
Wu et al. (2022a) Leijie Wu, Song Guo, Junxiao Wang, Zicong Hong, Jie Zhang, and Yaohong Ding. 2022a. Federated Unlearning: Guarantee the Right of Clients to Forget. IEEE Netw. 36, 5 (2022), 129–135. https://doi.org/10.1109/MNET.001.2200198
Yan et al. (2022) Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. 2022. ARCANE: An Efficient Architecture for Exact Machine Unlearning. In Proceedings of the 31st International Joint Conference on Artificial Intelligence.
Yuan et al. (2023) Wei Yuan, Hongzhi Yin, Fangzhao Wu, Shijie Zhang, Tieke He, and Hao Wang. 2023. Federated Unlearning for On-Device Recommendation. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 - 3 March 2023, Tat-Seng Chua, Hady W. Lauw, Luo Si, Evimaria Terzi, and Panayiotis Tsaparas (Eds.). ACM, 393–401. https://doi.org/10.1145/3539597.3570463