Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction

The proliferation of Large Language Models (LLMs) has ignited significant legal and ethical debates, particularly concerning copyright infringement. These models often do not document their training data sources, leading to disputes over unauthorized use of copyrighted material. For instance, lawsuits are piling up against OpenAI accused of training ChatGPT using articles and other books without permission¹¹1See, e.g., “The copyright lawsuits against OpenAI are piling up as the tech company seeks data to train its AI”, Jun 30, 2024. Similar accusations have been leveled at Meta or Google for allegedly using protected content²²2See, e.g., “Congress Senate Tech Companies Pay AI Training Data”, July 2, 2024. These issues underscore the societal, economic and legal implications of LLM training practices.

Recent surveys highlight the challenges of respecting copyright in AI training across different jurisdictions, such as the USA and France. In the USA, the use of copyrighted data is generally prohibited without the rights holder’s permission unless it falls under “fair use” [24]. For the culture and media sectors, LLM training could not be exempted from this limitation due to the conditions associated with it. More than twenty lawsuits are pending in the USA. In the European Union, the Directive 2019/790 on copyright and related rights in the digital single market introduces a “text and data mining” exception, for any purpose, that could correspond to the use of protected content for LLM training. However, its benefit is conditional on lawful access to copyrighted data and the absence of an opt-out by rights holders. However, not only do the training databases contain infringing works, but most of the rights holders have exercised their opt-out. However, the opacity of the process compromises the return to exclusive rights. So, to provide leverage, the AI Act (European Regulation 2024/1689), the first comprehensive regulation on AI, has required LLM providers, on the one hand, to put in place an internal policy aimed at respecting copyright and, on the other, to be transparent about the sources of training. The AI Office will provide a template on this point. The issue of knowledge of the use of content by the LLM is therefore crucial for rights holders.

Objective. From a technical perspective, determining whether a specific document was a member of the training set of a machine learning (ML) model based on the model’s output, is a Membership Inference Attacks (MIAs) problem, highlighted in 2017 by Shokri et al. [26]. However, the effectiveness of MIAs in the context of LLMs is subject to debate.

Limits of existing solutions. To determine whether a particular document has been used to train an AI model, MIAs rely on overfitting, which results in stronger predictions when applied to training data. While some research shows that MIAs can achieve high accuracy [19, 18], other studies [7, 6] question their validity due to inherent biases in the datasets of members and non-members used for their assessment. For example, biases like time shifts and n-gram overlaps can lead to over-interpretation of results [7]. Additionally, studies indicate that some biases can be exploited to make a “blind” classifier, without model access, more effective than MIAs [6]. This raises doubts about the robustness and practical relevance of current MIA techniques, and recent surveys like [13] point out the lack of rigor and practical relevance of current proposals.

Research question and contributions. This paper aims to address the problem of assessing MIA effectiveness on LLMs which do not disclose their full training datasets but where part of the training dataset can be inferred. The research question we seek to answer is: “How can we construct an unbiased dataset for evaluating MIAs on LLMs with partially inferable training sets?”

We propose and evaluate two approaches: (1) Creating datasets that are “non-biased” by design with respect to known biases, and (2) Constructing datasets that cannot be classified, ensuring fairer assessment. Our contributions can be summarized as follows:

• •

  We provide algorithms for constructing ex-post datasets of two types: $No-Ngram$ (“No N-gram bias”) and $No-Class$ (“non classifiable”), each designed to mitigate specific types of biases for MIA assessment.

• •

  We validate our algorithms and compare our proposed methods in the assessment of existing MIAs.

• •

  We demonstrate that neutralizing known biases (e.g., time shifts, n-gram biases) is insufficient for accurate MIA assessment; we also show that several existing MIAs, which are presumed to be effective, are not efficient when evaluated using non-classifiable datasets. For instance, our experiments show that the TPR@10%FPR and ROC AUC of the best performing MIA out of the 6 assessed drop by 40% and 14.3% respectively when evaluated on datasets produced with our approach rather than on randomly sampled ones.

Outline. Section 2 reviews related work and positions our research. Section 3 synthesizes our hypotheses and the addressed problem. Section 4 presents our proposed solutions and algorithms, detailing ex-post (i.e., a posteriori) construction of unbiased datasets. Section 5 provides a comparative experimental evaluation of the proposed solutions. Finally, Section 6 concludes the paper and suggests directions for future research.

MIA techniques, originally developed for machine learning classification algorithms [26], have recently been adapted to the context of LLMs. The baseline technique use likelihood-based metrics such as loss [31] and perplexity [2] to distinguish between members (documents which were used for LLM training) and non-members (not used in training). Several studies show that likelihood-based MIAs applied to LLMs are effective, with high AUC-ROC values. For example, [19] reports an AUC-ROC of $0.856$ for the Gutenberg dataset [23] (also used for evaluation in our paper). Other metrics, such as Min-k%Prob [25], are based on the premise that if the text has been read by the LLM, it is more likely to appear. More precisely, Min-k%Prob selects the k% tokens of the document with the minimum probabilities returned by the LLM and computes their average log-likelihood.

Neighboring-based MIAs calibrate the perplexity score using either neighboring models $\mathcal{M}^{\prime}$ or neighboring documents $D^{\prime}$. Classification between members and non-members is then obtained by comparing the likelihood of the target model $\mathcal{M}$ and document $D$ with that of the neighboring model $\mathcal{M}^{\prime}$ and document $D$, or the model $\mathcal{M}$ and neighboring documents $D^{\prime}$. Neighboring models hence assume access to a reference model trained on a disjoint data set drawn from a similar distribution, which is often unrealistic [7]. Neighboring documents [9] are more realistic but present slightly lower performance and additional difficulties in correctly setting noise parameters.

In cases where LLMs do not output likelihood information, complementary metrics can be used with acceptable performance penalty. For example, [14] proposes a MIA method called SaMIA, which measures the similarity between input samples from a document $D$ and the rest of the text in $D$ using ROUGE [16]. SaMIA demonstrates an AUC-ROC of $0.64$ on subsets of The Pile dataset [10].

Recent studies [7, 18] challenge the high-performance claims of MIAs on LLMs. They identify several biases that may skew results, such as timeshift between members and non-members, leading to different distributions of dates and word usage. Re-evaluating some MIAs on Pythia [1], trained on genuinely random train/test splits of The Pile [10] (and hence have no bias), shows decreased AUC-ROC measures, questioning the apparent success of some MIAs.

Some studies suggest that naive classifiers can distinguish members from non-members with good results based on these biases [6, 20]. These studies conclude that MIAs must be evaluated on random datasets taken from a same distribution. While this is possible with open LLMs like Pythia, which reveal their training and test data sources, it is not feasible for LLMs that do not disclose their sources (those of interest in copyright cases). As shown in the literature (see, e.g., [7, 18, 21]), the same MIAs yield different performance (accuracy and relevance) results on different LLMs/datasets. Therefore, the assessment of MIAs on open LLMs cannot be directly transposed to LLMs that do not disclose their training dataset. This confirms the need for techniques like the one we propose.

A technique inspired by the Regression Discontinuity Design from causal inference, originally used to study treatment effects based on a cutoff date, is proposed in [20]. However, documents added just before or after the cutoff date must be known and sufficiently numerous. Additionally, declared dates often deviate from reality [4], making this approach impractical.

Many other works are based on MIA attacks on LLMs but rely on different hypotheses, leading to solutions not applicable to our context. [15] introduces a framework using loss gap variation during fine-tuning to detect if a document has been seen, though this is not generalizable to initial training documents and also requires an assesment using unbiased datasets. Related works on copyright aspects include copyright issues in LLM outputs [22, 17] to reduce copyrighted text generation and protect users from potential plagiarism, and watermarking techniques [21, 29] for detecting violations in LLM pretraining data, or LLM fine-tuning data [30], but these works are not transposable to our context.

A Membership Inference Attack on a large language model $\mathcal{M}$ is a binary classification task aimed at determining whether a specific textual document $D$ was included in the training dataset $\mathcal{D}_{\text{train}}$ used to build $\mathcal{M}$. The goal of this attack is to design a function $MIA:\mathcal{D}\rightarrow\{0,1\}$ that can ascertain the truth value of $D\in\mathcal{D}_{\text{train}}$ for any document in the document space $\mathcal{D}$.

In the context of copyright checks, our goal is to detect ex-post potential violations involving protected texts in the LLM’s pretraining dataset. Our hypotheses H1 to H3 stem from this context, acknowledging that the LLM may try to obscure the use of these texts:

H1: Self-Assessment.

We assume that a reliable assessment of the MIA must be performed on the target LLM $\mathcal{M}$ itself. As shown in the literature (see, e.g., [7, 18]), the same MIAs yield different performance (accuracy and relevance) results on different LLMs/datasets.

H2: Partial Member Knowledge.

The training dataset $\mathcal{D}_{\text{train}}$ of $\mathcal{M}$ is partially inferable, i.e., a subset $\mathcal{D}^{known}_{\text{train}}$ of $\mathcal{D}_{\text{train}}$ can be inferred by an attacker. For example, it is know that OpenAI models like GPT-4 have memorized some precise collections of copyright protected books [3].

H3: Bias Recognition.

A subset $\mathcal{D}^{known}_{nm}$ of non-members (i.e., $\mathcal{D}^{known}_{nm}\subset\mathcal{D}\backslash\mathcal{D}_{\text{train}}$) is (obviously) known. Traditionally, such a subset is constructed by considering documents not available at the time the target LLM was released. This creates inherent biases in the ex-post context, where members and non-members are not drawn from the same random distribution.

We address the problem of producing datasets of members $\mathcal{D}^{Nob}_{train}\subset\mathcal{D}^{known}_{train}$ and non-members $\mathcal{D}^{Nob}_{nm}\subset\mathcal{D}^{known}_{nm}$ which aim to minimize bias (hence the name “Nob” for Non-biased). These datasets ensure a reliable evaluation of MIAs on LLMs while satisfying these three hypotheses.

In this section, we present our approach to identifying and mitigating specific bias in the construction of datasets used for the assessment of MIAs. Our methodology operates in two phases: first, addressing bias caused by low n-gram overlap, which has been shown to significantly affect the assessment of MIAs [7]; and second, mitigating additional biases that go beyond n-gram overlap.

In this section, we apply and evaluate our proposal with reference to the Gutenberg dataset. Experimental settings are described in Sec. 5.1, detailing how the candidate datasets are constructed to avoid a priori bias, how bias are assessed, as well as the MIAs and LLMs assessed on the datasets produced following our proposal. In spite of known members and non-members being constructed to circumvent bias, Sec. 5.2 shows that random samples exhibit n-gram bias. Such bias are addressed in Sec 5.3 by producing datasets following the $No-Ngram$ algorithm (Alg. 1), which still exhibit residual bias exploitable by an agnostic classifier. Section 5.4 assesses the last pair of datasets produced following the $No-Class$ algorithm (Alg. 2). Finally, Sec. 5.5 presents the assessment of MIAs using the produced datasets and discusses the impact of bias in the evaluation of MIAs. All the code, resulting analysis and dataset are available online⁴⁴4https://github.com/ceichler/MIA-bias-removal.

As LLMs are trained leveraging myriads of data items, including copyrighted ones, it is key to ascertain whether a piece of data has been used in this process. Yet, the effectiveness of MIAs has been recently questioned, due to the existence of biases in datasets constructed ex-post. This work introduces $Nob-MIAs$, a set of algorithms to build unbiased datasets, thus setting a more solid ground for MIA assessment. Our experiments on the Gutenberg dataset confirms that our approach significantly reduces bias (e.g., an 84% reduction of difference in n-gram overlap distribution) and impacts on MIA evaluation, with TPR@10%FPR and ROC AUC of the best-performing MIA (out of 6) decreasing by 40% and 14.3% respectively, compared to evaluations on randomly sampled datasets.

This work opens several future research avenues, including extending the algorithms to detect and mitigate residual biases and applying this approach to non-textual MIAs, where ex-post dataset construction is also common.