A New Formulation of Lipschitz Constrained With Functional Gradient Learning for GANs

GANs are designed to sample a random variable $z$ from a known distribution $p_{z}$, approximating the underlying data distribution $p_{*}$. This learning process is modeled as a minimax game where the generator and discriminator are iteratively optimized, as introduced by Goodfellow et al. [1]. The generator produces samples that mimic the true data distribution, while the discriminator distinguishes between the generated data and real data samples. Despite numerous remarkable efforts that have been made [2, 3], the GANs learning still suffers from training instability and mode collapse.

Recently, Composite Functional Gradient Learning (CFG), as proposed in Johnson and Zhang [4], has gained attention. CFG utilizes a strong discriminator and functional gradient learning for the generator, leading to convergent GAN learning theoretically and empirically. However, we have observed that there are still various hyper-parameters in CFG that may significantly impact the GAN learning process. Despite the advancements in stability, one still needs to carefully set these hyper-parameters to ensure a successful and well-trained GAN. Properly tuning these parameters remains an essential aspect of achieving optimal performance in GAN training. This issue hinders the widespread adoption of the CFG method for training GAN in real-world and large-scale datasets. One of the most effective mechanisms for addressing this issue for GAN training is the Lipschitz constraint, with the $0$-centered gradient penalty introduced by  Mescheder et al. [5] and the $1$-centered gradient penalty introduced by Gulrajani et al. [3] being among the most well-known. Empirically, stable training of a GAN can result in more diverse synthesis results. However, the theoretical foundation for stable GAN training using the Lipschitz gradient penalty and generating diverse synthesis samples is still unclear. It is therefore important to develop a new theoretical framework that can account for the Lipschitz constraint of the discriminator and the diversity of synthesis samples.

To address these challenges, we propose Lipschitz Constrained Functional Gradient GANs Learning (Li-CFG), an improved version of CFG. We present a comparative analysis of our Li-CFG and CFG methods in Fig. 2 and Example. 1.1. Additionally, we emphasize the importance of diversity in synthetic samples through Fig. 1. Remarkably, as shown in Fig. 1, the horse images generated by Li-CFG display a more diverse range of gestures, colors, and textures in comparison to those produced by the CFG method. When synthetic samples lack diversity, they fail to capture essential image characteristics. While recent generative models have made notable progress in enhancing diversity, there remains a gap in providing comprehensive theoretical explanations for certain aspects.

Our key insight is the introduction of Lipschitz continuity, a robust form of uniform continuity, into CFG. This provides a theoretical basis showing that synthetic sample diversity can be enhanced by reducing the latent vector neighborhood size through a discriminator constraint. For simplicity and clarity, we will use latent N-size to denote the neighborhood size of the latent vector and constraint to refer to the discriminator constraint for the rest of the article. First, Li-CFG integrates the Lipschitz constraint into CFG to tackle instability in training under dynamic theory Second, we establish a theoretical link between the discriminator gradient norm and the latent N-size. By increasing the discriminator gradient norm, the latent N-size is reduced, thus enhancing the diversity of synthetic samples. Lastly, to efficiently adjust the discriminator gradient norm, we introduce a novel Lipschitz constraint mechanism, the $\boldsymbol{\varepsilon}$-centered gradient penalty. This mechanism enables fine-tuning of the latent vector neighborhood size by varying the hyper-parameter $\boldsymbol{\varepsilon}$. Through this approach, we aim to achieve more effective control over the discriminator gradient norm and further improve the diversity of generated samples.

We summarize the key contributions of this work as follows: (1) We introduce a novel Lipschitz constraint to CFG, the $\boldsymbol{\varepsilon}$-centered gradient penalty, addressing the hyperparameter sensitivity in regression-based GAN training. Our Li-CFG method enables stable GAN training, producing superior results compared to traditional CFG. (2) We present a new perspective on analyzing the relationship between the discriminator constraint and synthetic sample diversity. To the best of our knowledge, we are the first to explore this relationship, demonstrating that our $\boldsymbol{\varepsilon}$-centered gradient penalty allows for effective control of diversity during training. (3) Our empirical studies highlight the superiority of Li-CFG over CFG across a variety of datasets, including synthetic and real-world data such as MNIST, CIFAR10, LSUN, and ImageNet. In an ablation study, we show a trade-off between diversity and model trainability. Additionally, we demonstrate the generalizability of our $\boldsymbol{\varepsilon}$-centered gradient penalty across multiple GAN models, achieving better results compared to existing gradient penalties.

To enhance readability, we provide definitions for frequently used mathematical symbols in our theory. For CFG and dynamic theory, the symbols will be explained in the respective sections.

$D_{\psi}:\mathcal{Y}\rightarrow\mathcal{C}$ represents a discriminator model that maps elements from the image space $\mathcal{Y}$ to the probability of belonging to real data distribution or the generated data distribution, where the parameter of generator is denoted by $\psi$. $\mathcal{Y}$ is a sample drawn from real data distribution $p_{*}$. $\mathcal{C}$ consists of the scale value obtained from the outputs of the discriminator. The symbol $D$ in the CFG method, along with the notation $D_{\psi}$ used in both the gradient penalty and our neighborhood theory, all represent the discriminator.

We use $R$ to denote the gradient penalty. Furthermore, we use $R_{1}$ to denote the $1$-centered Gradient Penalty, $R_{0}$ to denote the $0$-centered Gradient Penalty and $R_{\boldsymbol{\varepsilon}}$ to denote the $\boldsymbol{\varepsilon}$-centered Gradient Penalty. $r_{R}$ stands for the latent N-size of the corresponding gradient penalty. The symbol $\hat{\epsilon}$ in the definition of our neighborhood method represents a small quantity in image space. It is distinct from the symbol $\boldsymbol{\varepsilon}$ used in our $\boldsymbol{\varepsilon}$-centered GP method.

To simplify the analysis of this problem, we first transform the discrete M into the continuous M. First, we transform the Eq. (1) into $G_{m+\delta}(Z)=G_{m}(Z)+\delta g_{m}\left(G_{m}(Z)\right)$ by setting $\eta_{m}=\delta$, where $\delta$ is a small time step. By letting $\delta\rightarrow 0$, we have a generator that evolves continuously in time $m$ that satisfies an ordinary differential equation

The goal is to learn $g_{m}:\mathbb{R}^{k}\rightarrow\mathbb{R}^{k}$ from data so that the probability density $p_{m}$ of $G_{m}(\boldsymbol{z})$, which continuously evolves by Eq. (1), becomes close to the density $p_{*}$ of real data as $m$ continuously increases. To measure the ‘closeness’, we use $L$ denotes a distance measure between two distributions:

where $\ell:R^{2}\rightarrow R$ is a pre-defined function so that $L$ satisfies $L(p_{m})=0$ if and only if $p_{m}=p_{*}$ and $L(p)\geq 0$ for any probability density function $p$.

From the above equation, we will derive the choice of $g_{m}(\cdot)$ that guarantees that transformation Eq. (1) can always reduce $L(\cdot)$. Let $p_{m}$ be the probability density of random variable $G_{t}(\boldsymbol{z})$. Let $\ell_{2}^{\prime}\left(\rho_{*},\rho\right)=$ $\partial\ell\left(\rho_{*},\rho\right)/\partial\rho$. Then we have

With this definition of $\frac{dL\left(p_{m}\right)}{dm}$, we aim to keep that $\frac{dL\left(p_{m}\right)}{dm}$ is negative, so that the distance $L$ decreases. To achieve this goal, we choose $g_{m}(x)$ to be:

where $s_{m}(x)>0$ is an arbitrary scaling factor. $\phi_{0}(u)$ is a vector function such that $\phi(u)=u\cdot\phi_{0}(u)\geq 0$ and $\phi(u)=0$ if and only if $u=0$. Here are two examples: $(\phi_{0}(u)=u,\phi(u)=\|u\|_{2}^{2})$ and $(\phi_{0}(u)=\operatorname{sign}(u),\phi(u)=\|u\|_{1})$.

With this choice of $g_{m}(x)$, we obtain

that is, the distance $L$ is guaranteed to decrease unless the equality holds. Moreover, this implies that we have $\lim_{m\rightarrow\infty}\int s_{m}(x)p_{m}(x)\phi\left(\nabla_{x}\ell_{2}^{\prime}\left(p_{*}(x),p_{m}(x)\right)\right)dx=0$. (Otherwise, $L\left(p_{m}\right)$ would keep going down and become negative as $m$ increases, but $L\left(p_{m}\right)\geq 0$ by definition.) For simplicity, we omit the subscript $m$ in the following empirical settings.

Let us consider a case where the distance measure $L(\cdot)$ is an $f$-divergence. With a convex function $f:\mathbb{R}^{+}\rightarrow\mathbb{R}$ such that $f(1)=0$ and that $f$ is strictly convex at $1,L\left(p_{m}\right)$ defined by

is called $f$-divergence. Here we focus on a special case where $f$ is twice differentiable and strongly convex so that the second order derivative of $f$, denoted here by $f^{\prime\prime}$, is always positive. For instance, when we consider the KL divergence, $f$ can be represented by $-\ln x$, in which case $f^{\prime}=-1/x$ and $f^{\prime\prime}=1/x^{2}$. On the other hand, if we consider the reverse KL divergence, $f$ can be represented as $x\ln{x}$, in which case $f^{\prime}=\ln{x}+1$ and $f^{\prime\prime}=1/x$.

With this definition of function $f$ and $L$ as $f$-divergence setting, the value of $g_{m}(x)$ should to be

where $s_{m}(x)>0$ is an arbitrary scaling factor, $\ell_{2}\left(\rho_{*},\rho\right)=\rho_{*}f\left(\rho/\rho_{*}\right)$, $\nabla_{x}\ell_{2}^{\prime}\left(p_{*}(x),p_{m}(x)\right)=f^{\prime\prime}\left(r_{m}(x)\right)\nabla r_{m}(x)$, $f^{\prime\prime}=1/x^{2}$ when $f$ is KL-divergence and $r_{m}(\boldsymbol{x})=\exp(-D(\boldsymbol{x}))\approx p_{m}(x)/p_{*}(x)=\tilde{r}_{m}(\boldsymbol{x})$ when $D(x)\approx\ln\frac{p_{*}(x)}{p_{m}(x)}$, which is the analytic solution of the CFG discriminator.

Empirically, we define the function in Eq. (4) as $g(\boldsymbol{x})=\delta(\boldsymbol{x})\nabla_{\boldsymbol{x}}D(\boldsymbol{x})$. The $\delta(\boldsymbol{x})$ can be computed as

where we have an arbitrary scaling factor $s_{m}(\boldsymbol{x})$, a KL-divergence function $f=-\ln\boldsymbol{x}$, $f^{\prime\prime}=1/x^{2}$ and $\tilde{r}_{m}(\boldsymbol{x})=\exp(-D(\boldsymbol{x}))$. Since $\tilde{r}_{m}(\boldsymbol{x})f^{\prime\prime}(x)>0$, it can be absorbed into the $s_{m}(x)$. So the value of $\delta(\boldsymbol{x})$ is always greater than 0. For simplicity, we directly regard $\delta(\boldsymbol{x})$ as the scaling factor and set it to a fixed value as a hyper-parameter. We also demonstrate the results of different $\delta(\boldsymbol{x})$ values in the Fig. 2.

When integrating the gradient penalty into the GAN model, it is expressed as a regularization term following the loss function of the GAN. The formulation is represented as

where $\lambda$ controls the importance of the regularization, $R(\theta,\psi)$ denotes the gradient penalty term, $f$ represents a general function from [3] with different forms in various GAN models. This specific type of loss function indicates that the gradient penalty can influence the variability of the discriminator’s gradient, thereby impacting the generation of synthetic samples by the generator.

We have clarified the differences between our method and AdvLatGAN-div and MSGAN as follows: Firstly, we note that MSGAN, AdvLatGAN, and our method are related to Eq. (10).

where $\boldsymbol{c}$ is the condition vector for generation, and $d_{\boldsymbol{I}}$ and $d_{\boldsymbol{z}}$ are the distance metrics in the target (image) space and latent space, respectively.

Secondly, MSGAN maximizes Eq. (10) to train the generator, encouraging it to synthesize more diverse samples. Essentially, maximizing Eq. (10) means that for two different samples $\boldsymbol{z_{1}}$ and $\boldsymbol{z_{2}}$ from the latent space, the distance between their corresponding synthesized outputs $G(\boldsymbol{z_{1}})$ and $G(\boldsymbol{z_{2}})$ in the target space should be as large as possible. This leads to an increase in the diversity of the generated samples. The AdvLatGAN-div method searches for pairs of $z_{t}$ that match hard sample pairs, which are more likely to collapse in image space. This search process is achieved by iteratively optimizing the latent samples $\boldsymbol{z}$ to minimize Eq. (10), similar to generating adversarial examples. Then, the objective is to maximize Eq. (10) by using these hard sample pairs $z_{t}$ in the latent space as input again to optimize the generator. This process helps to generate diverse samples in the image space and avoid mode collapse. Both MSGAN and AdvLatGAN-div, which utilize Eq. (10), are based on empirical observations and do not have a solid theoretical basis.

Unlike the previous two methods, we don’t employ Eq. (10) as an additional loss function or an optimal target for training our model. Instead, we use Eq. (10) as the fundamental explanation of our theory. We have improved upon Eq. (10) by using the CFG method. This has helped us establish a connection between the distance of various $\boldsymbol{z}$ from the latent space and the discriminator’s constraints. We have also explained how the Lipschitz constraint of the discriminator can enhance the generator’s diversity based on our new theory.

The explanation and ablation study of $\varepsilon^{\prime}$ are in the supplementary. The base setting is presented in Table. 1 and Table. 2.

We note that the inception score is limited, as it fails to detect mode collapse or missing modes. Apart from that, we found that it generally corresponds well to human perception.

In addition, we used Fréchet inception distance (FID). FID measures the distance between the distribution of $f(x_{*})$ for real data $x_{*}$and the distribution of $f(x)$ for generated data $x$, where function $f$ extract the feature of an image. One advantage of this metric is that it would be high (poor) if mode collapse occurs, and a disadvantage is that its computation is relatively expensive. However, FID does not differentiate the fidelity and diversity aspects of the generated images, so we used the Precision/Recall to diagnose these properties. A high precision value indicates a high quality of generated samples, and a high recall value implies that the generator can generate many realistic samples that can be found in the ”real” image distribution. In the results below, we call these metrics the (inception) score, the Fréchet distance, and the Precision/Recall.

In this paper, We provide a novel perspective to analyze the relationship between constraint and the diversity of synthetic samples. We assert that the constraint can effectively influence the latent N-size that is strongly associated with the mode collapse phenomenon. To modify the latent N-size efficiently, we propose a new form of Gradient Penalty called $\boldsymbol{\varepsilon}$-centered GP. The experiments demonstrate that our method is more stable and achieves more diverse synthetic samples compared with the CFG method. Additionally, our method can be applied not only in the CFG method but also in common GAN models. Inception score, Fréchet Distance, Precision/Recall, and visual quality of generated images show that our method is more powerful. In future work, we plan to investigate the metric function of the generator from KL diversity to the Wasserstein distance to achieve a more stable and efficient GAN architecture.

declaration - statement

• •

  Conflicts of interest/Competing interests

  Not applicable

• •

  Funding

  This work was funded by the National Natural Science Foundation of China U22A20102, 62272419. Natural Science Foundation of Zhejiang Province ZJNSFLZ22F020010.

• •

  Ethics approval

  Not applicable

• •

  Consent to participate

  Not applicable

• •

  Consent for publication

  Not applicable

• •

  Availability of data and material

  All the data used in our work are sourced from publicly available datasets that have been established by prior research. References for all the data are provided in our main document.

• •

  Code availability

  Our code is some kind of custom code and is available for use.

• •

  Authors’ contributions

  We list the Authors’ contributions as follows:

  Conceptualization: [Chang Wan], [Yanwei Fu], [Xinwei Sun];

  Methodology: [Chang Wan], [Xinwei Sun], [Ke Fan];

  Experiments: [Chang Wan], [Ke Fan], [Yunliang Jiang];

  Writing ‐ original draft preparation: [Chang Wan], [Yanwei Fu];

  Writing ‐ review and editing: [Xinwei Sun], [Zhonglong Zheng], [Minglu Li];

  Funding acquisition: [Yunliang Jiang], [Zhonglong Zheng];

  Resources: [Zhonglong Zheng];

  Supervision: [Minglu Li], [Yunliang Jiang];

  All authors read and approved the final manuscript.