Self-Correcting Self-Consuming Loops for Generative Model Training

Real curated datasets are costly to obtain, so there has been much interest in generating synthetic data as training data for various vision tasks. Azizi et al. (2023) demonstrates that text-to-image diffusion models such as Imagen (Saharia et al., 2022) can generate synthetic examples that augment the ImageNet dataset for better image classification. He et al. (2023) studies how synthetic data from text-to-image models, when used exclusively, can be used as training data for image recognition tasks. Similarly, Tian et al. (2023) finds that using synthetic outputs from a text-to-image model results in contrastive models whose downstream performance rivals that of CLIP (Radford et al., 2021) on visual recognition tasks, including dense prediction. And the work in Jahanian et al. (2022) explored methods for multi-view representation learning by using the latent space of the generative models to generate multiple “views” of the synthetic data. The above works collectively provide evidence that some representation learning tasks, when trained on synthetic data from some given generative models, yield excellent results.

Another line of reseach investigates the use of synthetic data for training generative models. Shumailov et al. (2023) and Martínez et al. (2024) show that the use of model generated content in generative model training results in model degradation, likely because self-consuming loops remove low-density areas from the estimated probability manifold. Alemohammad et al. (2024) formalize three different kinds of self-consuming generative models: the fully synthetic loop, the synthetic augmentation loop, and the fresh data loop. In all of these loops, they iteratively re-train the model from scratch for every new generation. They empirically find that only the fresh data loop avoids model degradation.

Another recent work (Bertrand et al., 2024) considers the problem of iteratively fine-tuning in the context of synthetic augmentation loops. They find that self-consuming augmentation loops do not necessarily collapse, so long as the synthetic augmentation percentage is sufficiently low. The authors use techniques from the field of performative stability (Perdomo et al., 2020) to prove the existence of a convergence phenomenon in the space of model parameters. Our paper differs from prior work as we conduct analysis on self-consuming generative model training when the synthetic data can be optionally corrected. The correction can be performed with a human-in-the-loop, or by incorporating learned or programmed expert knowledge, as explored for natural language (Saunders et al., 2022; Welleck et al., 2023; Wu et al., 2023) and human motion (Yuan et al., 2023; Xu et al., 2023). We validate our theory with a practical self-correcting operations designed for image generation and human motion synthesis tasks.

We mostly follow the notation from (Bertrand et al., 2024), except for introducing the correction function $\pi_{\gamma}$. Let us denote by $p_{\mathrm{data}}$ the ground truth probability distribution that we want to train a generative model to estimate. Suppose we have some dataset $\mathcal{D}_{\mathrm{real}}=\{x_{i}\}_{i=1}^{n}$ sampled from $p_{\mathrm{data}}$. We write $\hat{p}_{\mathrm{data}}=(1/n)\sum_{i=1}^{n}\delta_{x_{i}}$. More generally, we use a hat to denote the empirical distribution over finitely many samples from the corresponding distribution.

Suppose that we have a class of generative models parameterized by $\Theta\subset\mathbb{R}^{d}$. We denote by $p_{\theta}$ a probability distribution in this class with model parameters $\theta\in\Theta$. We define the optimal model parameters within this class to be

where we break ties by minimizing $\|\theta^{\star}\|$. Typically, such optimal parameters yield a model $p_{\theta^{\star}}$ which closely approximates the oracle ground truth distribution $p_{\mathrm{data}}$, but doesn’t equal it exactly; accordingly, we define the Wasserstein-2 distance between the distributions to be

The model weights for the first generation are naturally defined according to the optimization

This corresponds to training on the finite subset $\mathcal{D}_{\mathrm{real}}$. Next, let us suppose that the model weights from generation $t$ are denoted $\theta_{t}^{n}$. We will formalize a procedure for updating these weights for the next generation to obtain $\theta_{t+1}^{n}$. For this, we need to define our correction function, and then we will use it to define the weight update.

To continue our discussion from before, our iterative weight update is defined as $\theta_{t+1}^{n}:=\pi_{\gamma}\mathcal{G}_{\lambda}^{n}(\theta_{t}^{n})$.

Note that we use an global maximization in (3) when defining the initial parameters $\theta_{0}^{n}$, but we use a local maximization when computing our parameter update in (5). This difference is analogous to the differences between how model weights update during initial training, where parameter updates are more global, and during fine-tuning, where parameter updates are more local.

In order to prove our main result, we need some regularity assumptions about the learning procedure. Informally speaking, we will assume that the class of generative models that we consider is smoothly parameterized by its model weights; the loss landscape is concave near the ideal model weights; and the class of generative models does an increasingly good job approximating the target data distribution as the dataset size increases. We formally quantify and state these hypotheses in Assumption 4.2.

In Assumption 4.2 (2), the notation “$\preceq$” corresponds to the Loewner order on symmetric matrices: we write that $A\preceq B$ if $B-A$ is positive semi-definite, and $A\prec B$ if $B-A$ is positive definite. In particular, Assumption 4.2 (2) implies that the matrix $\mathbb{E}_{x\sim p_{\text{data}}}\left[\nabla_{\theta}^{2}\log p_{\theta}(x)\right]|_{\theta^{\star}}$ is negative definite, and its largest eigenvalue is at most $-\alpha$. And Assumption 4.2 (3) mirrors the main assumption in (Bertrand et al., 2024); it is motivated by generalization bounds in deep learning, see e.g. (Jakubovitz et al., 2019; Ji et al., 2021). The interested reader can consult Appendix B for more details on this assumption.

We now have the language to state our main result, which essentially says that if the initial parameters $\theta_{0}$ are sufficiently close to the optimal model parameters $\theta^{\star}$, and if the augmentation percentage $\lambda$ is sufficiently small, then under iterative fine-tuning with correction, we can expect our subsequent model parameters to stay close to $\theta^{\star}$.

We prove Theorem 4.3 in Appendix A.

Note that Theorem 4.3 relies on the assumption that the initial model parameters $\theta_{0}$ are sufficiently close to the ideal model parameters $\theta^{\star}$, and also that the augmentation percentage $\lambda$ is sufficiently small. We hypothesize that these assumptions can be relaxed in the case where a correction function participates in the iterative fine-tuning procedure–intuitively, the correction function should compensate for errors that arise from $\theta_{0}^{n}$ being worse, as well as errors that arise from incorporating more synthetic data. We frame this in the following conjecture.

We provide empirical evidence for Conjecture 4.7 in Section 7 on the human motion synthesis task. In fact, Theorem 4.3 represents partial progress towards this conjecture. Namely, according to Theorem 4.3, for large correction strength $\gamma$, we can effectively choose a synthetic augmentation percentage that is twice as large as we would be able to without any correction, and still be able to meet the assumptions of the theorem. This is because $\lim_{\gamma\to\infty}\frac{1+\gamma}{2+\gamma}=1$, which is twice as large as the bound when $\gamma=0$.

For our generative model, we use the Human Motion Diffusion Model (MDM) (Tevet et al., 2023). This is a classifier-free diffusion-based generative model for the text-to-motion generation task, where the model receives as input a description of a motion sequence (e.g. “get down on all fours and crawl across the floor”), and outputs a sequence of skeleton poses which attempt to embody that prompt. Synthesizing human motion is challenging not only for the diverse and compositional text prompts, but also due to failure of physics obeying-ness (e.g. feet skating, floating, penetrating a surface), which is not explicitly enforced by deep generative models.

For our self-correction function, we use Universal Humanoid Control (UHC) (Luo et al., 2021), which is an imitation policy that operates inside the MuJoCo physics simulator (Todorov et al., 2012). Given an input sequence of humanoid skeleton poses, UHC attempts to imitate the motion sequence, constrained by the laws of physics imposed by the physics simulator, and it outputs a new motion sequence that is the closest possible approximation it can replace it with. For example, if an input motion sequence violates the laws of physics by having a foot penetrate through the floor, then the motion sequence output by UHC will attempt to remove that physically impossible artifact while maintaining the semantic integrity of the original input motion. We use VPoser (Pavlakos et al., 2019) and SMPL (Loper et al., 2015) to translate joint representations between the human motion generator and the physics simulator.

The physics simulator allows us to self-correct a synthesized motion automatically. Our underlying assumption is that by enforcing the physics obeying-ness (via the simulator) and closeness to the synthesized motion (via the imitation objective), the self-correction function would act as similar as an idealized corrector as possible.

We preprocess the MoVi (Ghorbani et al., 2021) subset of HumanML3D (Guo et al., 2022) using the official code implementation of HumanML3D. We filter out movements involving interactions with chairs, as UHC by default does not handle human-object interactions. We take as our train split the train split from HumanML3D, intersected with our filtered subset of MoVi, and likewise for the test split. This procedure yields a train set of size $n=2794$ and a test set of size $546$. We further randomly select a smaller training set of $n\in\{64,128,256\}$ examples, to simulate the more challenging scenario when the initial generative model is sub-optimal (due to data scarcity). The smaller data also enables us to explore larger synthetic augmentation percentage due to compute constraints. From here, the iterative re-training procedure follows Algorithm 1. We spell it out in this concrete experimental setup.

We first train on the ground truth train split until the model is nearly converged, using all the default hyperparameters from MDM. We evaluate and save this last checkpoint from generation $0$. From here, for each generation $t\in\{1,2,\dots,50\}$, we run three sets of experiments.

1. A.

   Baseline: fine-tune the latest checkpoint from generation $t-1$ for $m$ batches on ground truth dataset $\mathcal{D}_{\mathrm{real}}$.

2. B.

   Iterative fine-tuning: fine-tune the latest checkpoint from generation $t-1$ on $\mathcal{D}_{\mathrm{real}}\cup\mathcal{D}_{\mathrm{synth},t-1}$ for $m$ batches. Here, $\mathcal{D}_{\mathrm{synth},t-1}$ is a synthetic dataset of size $\lfloor\lambda\cdot n\rfloor$ generated from the checkpoint for generation $t-1$, using randomly chosen prompts from the train split.

3. C.

   Iterative fine-tuning with self-correction: fine-tune the latest checkpoint from generation $t-1$ on $\mathcal{\mathcal{missing}}D_{\mathrm{real}}\cup\mathrm{UHC}(\mathcal{D}_{\mathrm{synth},t-1})$ for $m$ batches. Here, $\mathrm{UHC}(\mathcal{D}_{\mathrm{synth},t-1})$ denotes a synthetic dataset of size $\lfloor\lambda\cdot n\rfloor$ generated from the latest checkpoint for generation $t-1$, using randomly chosen prompts from the train split, which is then corrected by UHC.

We experiment with synthetic augmentation percentages $\lambda\in\{0.05,0.10,0.15,0.20,0.25\}$ on the larger dataset; we set the number of batches seen during generation $0$ to be $3125$, and the number of batches seen for each later generation to be $m=625$. Separately, we experiment with synthetic augmentation percentages $\lambda\in\{0.25,0.50,0.75,1.00\}$ on the smaller datasets; we set the number of batches seen during generation $0$ to be $78*k$ for dataset size $64*k$, and the number of batches seen for each later generation $t>0$ to be $m=16$. We choose to control how many data points the model sees across each generation, rather than controlling some other quantity like the number of epochs, as this allows each experiment to compare against its baseline in a controlled way, which in turn allows them to compare against each other in a controlled way.

We compute every evaluation one time for each checkpoint using the evaluation script provided in the original MDM codebase. Regardless of the train split size, we perform sampling for evaluation using all 546 motion sequences from the test split, since the FID score is sensitive to generated dataset size. We use the same hyperparameters as those used for MDM, including batch size $64$, AdamW (Loshchilov & Hutter, 2019) with learning rate $1e-4$, and classifier-free guidance parameter $2.5$. And for UHC we used the uhc_explicit model for imitation.

Our experimental results confirm our theoretical results, that iterative fine-tuning with self-correction outperforms iterative fine-tuning without self-correction, in the sense that the graphs are generally more stable across generations, and approach better evaluation metric values. In particular, Theorem 4.3 and Corollary 4.5 claim that any amount of idealized self-correction will improve the stability bound during iterative fine-tuning. Our results in Figure 4 demonstrate that the FID score is lower and more stable across generations when applying self-correction, and generally higher and less stable than the baseline, where there is no self-consuming training at all. We conduct experiments across multiple seeds, and we find empirically that this general phenomenon holds consistently, where the self-correction technique consistently yields improved training dynamics over iterative fine-tuning with no correction. Graphs from these runs can be found in Appendix G.

Our experimental results also provide empirical evidence for Conjecture 4.7. Observe that in the baseline experiments in Figure 4, the FID score decreases across generations, which indicates that the initial model parameters $\theta_{0}^{n}$ are not that close to the optimal model parameters $\theta^{\star}$; additionally, the augmentation percentages considered in the graph are $25\%$ and $100\%$. Conjecture 4.7 claims that performing self-correction during iterative fine-tuning improves performance, even when the initial model weights are sub-optimal and simultaneously the synthetic augmentation percentage is large. This claim is confirmed by Figure 4. We direct the curious reader to Appendix F, where we present graphs for all of the above listed training set sizes and augmentation percentages, providing additional empirical evidence for Theorem 4.3, Corollary 4.5, and Conjecture 4.7.

We visually inspect the generated human motion sequences in order to analyze what concrete effect the self-correction has on iterative fine-tuning. We find that the correctness and diversity of synthesized motions are improved by the self-correction procedure, in agreement with our quantitative analysis in Subsection 7.4. We present snapshots of our synthesized motions in Figure 5, and we analyze the motions in the caption. In short, we find that physics-disobeying artifacts such as floor penetration or floating become more pronounced without the self-correction. We also find that in the model without self-correction, the humanoid sometimes performs movements completely unrelated to the prompt; our model with self-correction fixes these negative phenomena. We direct the curious reader to Appendix E, where we present more examples from our qualitative analysis, as well as our project webpage, where we provide side-by-side video comparisons.