Conditional Idempotent Generative Networks

While the initial version of the paper [10] does not provide insights on the model architectures used for the experiments the authors describe, the paper does discuss at length the loss function of an IGN from a theoretical perspective.

Let again $X$ be the feature space and $D$ be the data manifold. Let $F:X\longrightarrow X$ be the idempotent function we are trying to construct and whose parameters are being learned by the IGN.

The loss function is then composed of the following three terms:

• •

  (reconstruction) Real data samples $d$ from the data distribution $D$ should remain unchanged by the model: $F(d)=d$.

• •

  (idempotent) Any further application of the model beyond the first one should act like the identity function: $F(F(z))=F(z)$ for every $z$ in $X$.

• •

  (tightness) The previous two conditions should be satisfied while ensuring $F$ has as small a range as possible.

We encourage the interested reader to check out additional details in the original paper [10].

The theoretical framework of idempotent generative network is fundamentally architecture-agnostic. And yet, one needs to choose a model architecture in order to test the feasibility of the idea in practice.

In [10], the authors do not explicitly describe the architecture used in their experiments. Inspired by [8], we will leverage a GAN-like architecture.

In the classic setup of Generative Adversarial Networks, a generator and a discriminator compete in a zero-sum game: the generator creates synthetic data points resembling as much as possible a real dataset, while the discriminator attempts to distinguish the real data points from the synthetic ones. In particular, in the GAN framework the discriminator is fed synthetic data points created by the generator.

In the IGN setting, instead, the two models will work in the opposite order: the discriminator $d$ processes input data into a latent tensor, and the generator $g$ creates a synthetic data point from that latent tensor. Our idempotent generative network is then $F=g\circ d$.

Let $X$ be again the feature space and $D$ be the data manifold, immersed into $X$. Suppose that every data point in $D$ comes with a condition. This could for example be a label or a caption (in case $D$ represent images). Let $C$ be the ‘condition’ space, the space where conditions $c$ belong to.

We can now extend the paradigm behind idempotent generative network to consider conditioning: we have an augmented data manifold $\widetilde{D}$ immersed in $X\times C$, every point $(d,c)$ consisting of a data point $d$ and its condition $c$. We want to find an idempotent map

such that:

• •

  (reconstruction) $F(d,c)=(d,c)$ for every augmented data point $(d,c)\in\widetilde{D}$.

• •

  (idempotence) $F(F(x,c))=F(x,c)$ for every $(x,c)$ in $X\times C$.

• •

  (tightness) the range of $F$ is as small as possible, i.e. $F(X\times C)$ is a low-dimensional submanifold of $X\times C$.

As explained in [10], the tightness condition is crucial to avoid the model learning to replicate the identity function (which satisfies the idempotence and the reconstruction requirements).

We plan on using the model as follows: suppose we want to generate a sample with condition $c$. We randomly select a noisy sample $z\in X$, we calculate $F(z,c)\in X\times C$ and we extract the first component $\tilde{d}$ of $F(z,c)$.

Even a perfect model $F$ satisfying the three conditions in section 3.1 above is not guaranteed to return a data point $\tilde{d}$ related to condition $c$. Indeed, the idempotent function $F$ is not being explicitly constrained to return a data point $\tilde{d}$ related to condition $c$, just a data point in the extended data manifold.

In order to help the model learn the conditioning, we treat mismatched data points like noisy examples. A mismatched data point is a pair $(d,\hat{c})$ where $(d,c)$ belongs to the augmented data manifold $\widetilde{D}$ and $c\neq\hat{c}$. In other words, we tell the model that $(d,\hat{c})$ should not be part of the range. By treating mismatched data points like noisy samples, we teach the network to not act like the identity function on them.

Let $D_{c}=\left\{d\in D\,|\,(d,c)\in\widetilde{D}\right\}$ be the subset of the data manifold $D$ consisting of points with condition $c$. We are then teaching the model $F$ to have range contained in $\widetilde{D}$ (tightness and reconstruction conditions), but that $D_{c}\times{\hat{c}}$ should not be part of the range for any $\hat{c}\neq c$ (mismatched conditions). By exclusion this forces the model $F$ to further restrict its range to the unions of $D_{c}\times{c}$ as $c$ varies in $C$.

The loss function for conditional idempotent generative networks is then made of five terms:

1. 1.

   (reconstruction loss $L_{rec}$) Real data samples $(d,c)$ from the augmented data manifold $\widetilde{D}$ should remain unchanged by the model: $F(d,c)=(d,c)$.

2. 2.

   (idempotent on noisy samples $L_{idem}^{noise}$) For any noisy sample $z$ and any condition $c$, any further application of the model beyond the first one should act like the identity function: $F(F(z,c))=F(z,c)$.

3. 3.

   (tightness on noisy samples $L_{tight}^{noise}$) The idempotence condition on noisy samples and the reconstruction condition should be satisfied while ensuring $F$ has as small a range as possible.

4. 4.

   (idempotent on mismatched samples $L_{idem}^{mism}$) For a data point $(d,c)$ on the augmented data manifolds $\widetilde{D}$ and a condition $\hat{c}$ different from $c$, any further application of the model beyond the first one should act like the identity function: $F(F(d,\hat{c}))=F(d,\hat{c})$.

5. 5.

   (tightness on mismatched samples $L_{tight}^{mism}$) The idempotence condition on mismatched samples and the reconstruction condition should be satisfied while ensuring $F$ has as small a range as possible.

Each one of these conditions is implemented via the choice of a distance function $\delta$ on the space $X\times C$, which is used to compare a data point before and after the application of $F$.²²2For simplicity one usually picks the same distance function $\delta$ for all loss terms, although this is not theoretically necessary. For example the reconstruction loss is implemented as

The total loss is then obtained as a weighted sum of these five terms:

Each one of the five weights is a hyperparameter that can be tuned during training.

During training, every batch will then be composed of three types of data points:

• •

  true data points $(d,c)$ in the augmented data manifold $\widetilde{D}$.

• •

  noisy samples $(z,c)$ where $z$ is a randomly sampled from $X$ following a prior distribution.

• •

  mismatched samples $(d,\hat{c})$ for $(d,c)\in\widetilde{D}$ and $\hat{c}\neq c$.

The relative proportion of the three types of data points within a batch is a training hyperparameter.

We discuss some theoretical considerations regarding what metrics should be used to evaluate Conditional Generative Idempotent Networks.

At a high level, an appropriate metric should measure the following two dimensions:

• •

  (realism) The synthetic data point $F(z,c)$ generated from condition $c$ should indeed look like a sample from $D_{c}$.

• •

  (diversity) Synthetic data points $F(z,c)$ generated from the same condition $c$ as we vary the noisy input $z$ should space over as large a subset of $X$ as possible.

There is a fairly standard set of metrics typically used to measure these two dimensions for Generative Adversarial Networks, for example Inception Score (IS) and Fréchet Inception Distance (FID). These metrics have been adapted to the setup of conditional generative adversarial networks (see for example [1]).

More complex metrics (see for example [4]) have been suggested for conditional image generation models, especially if the generation is implemented via a diffusion process. In this latter scenario, human evaluation is most of the time still the most accurate performance metric.

All of these metrics are reasonable choices and the one to be preferred mostly depends on the specific use case. If the condition space $C$ is a finite set of classes, metrics based on the inception score and the Fréchet inception distance are probably adequate. If the condition space $C$ is a continuous manifold (for example, $c$ can be any English sentence), then human evaluation is likely to be preferred.

Let $E:C\longrightarrow\mathbb{R}^{S}$ be an embedding function, where we denote by $\mathbb{R}^{S}=\mathbb{R}^{s_{1}}\times\ldots\times\mathbb{R}^{s_{k}}$ the subspace corresponding to the non-channel dimensions of the feature space $X$.

Channel conditioning is implemented by concatenating $E(c)$ and $x$ along the channel dimension.

This is a very generic and flexible idea, with many different variations available:

• •

  We can build different embeddings $E_{i}(c)$ whose target space consists of the non-channel dimensions of the input $x_{i}$ to layer $i$ of the network, and then concatenate $E_{i}(c)$ to $x_{i}$.

• •

  We can build embeddings whose target space is multiple copies of $\mathbb{R}^{S}$, so that when concatenating along the channel diemnsion we give the model additional flexibility on learning from the condition.

In section 5.3 we describe our experiments on training a conditional idempotent generative network with channel conditioning on the MNIST dataset.

Let $S^{\prime}=(s_{1}^{\prime},\ldots,s_{k}^{\prime})$ be any $k$-tuple with $0<s_{i}$ for any $1\leq i\leq k$. Let $E:C\longrightarrow\mathbb{R}^{S^{\prime}}$ be an embedding function.

Filter conditioning is implemented by calculating cross correlation (or transposed cross correlation) between $x$ and $E(c)$ along the non-channel dimensions. This is inspired by the concept of correlation filters for object detection (see for example [2]).

One can think of filter conditioning as replacing a standard or transposed convolutional layer (where the model learns the kernel’s weights and those weights are shared across all inputs) by a simpler cross-correlation layer, where each input is (transpose) cross-correlated to its condition’s embedding, and the model learns the embedding layer’s weights.

This also is a very generic and flexible idea, with many different variations available:

• •

  We can choose different filter dimensions $S^{\prime}$ for the embedding’s target space - in fact, $S^{\prime}$ can be considered as a hyperparameter of the model.

• •

  We can replace any number of convolutional layers (of a base model architecture that we start with) with filter conditioning layers where the filter dimensions $S^{\prime}$ are the dimensions of the original convolutional layer.

• •

  We can concatenate the result of a standard convolutional layer with the result of a filter conditioning layer, assuming that two layers’ parameters (kernel’s shape, padding, stride, dilation) are identical.

The MNIST dataset from [12] consists of images with a single channel and a size of $28\times 28$ pixels, valued in $[0,1]$. We preprocess the dataset by transforming images into torch tensors, and then linearly rescaling pixels to be valued between $-1$ and $1$. We add some small random noise to the training data, to fight overfitting.

In the setup of the previous sections, we have then the feature space

with the condition space

being the label set of the MNIST dataset.

Noisy data points are sampled from $X$ according to a normal distribution centered at $0$ and with standard deviation $1$, and then clipped so that they belong to $\left[-1,1\right]^{1\times 28\times 28}$.

The distance function $\delta$ used to implement the loss function is

the $L^{1}$ norm of the difference between the feature space components. While this is not strictly speaking a distance on $X\times C$, it works for our purposes since the models we implement pass through the condition $c$ ‘as-is’ (meaning that $F(x,c)=\left(F_{1}(x,c),c\right)$), and for the purpose of loss calculation we only need to calculate the distance between $(x,c)$ and $F(x,c)$.

At a high level, the architecture of our Conditional Idempotent Generative Networks is composed of:

• •

  a discriminator with domain $X\times C$, outputting a pair of a latent tensor and the condition $c$;

• •

  a generator, with input the latent tensor and the condition $c$, and range $X\times C$.

The CIGN is then obtained by running the discriminator first and the generator second.

The architectures of the discriminator and the generator are heavily inspired by the DCGAN framework, in its pytorch implementation ([6]).

The discriminator is defined as a sequence of convolutional layers with:

• •

  an increasing number of channels, starting from a latent dimension $l_{dim}$ and doubling each layer, with the final latent tensor having size $(i_{dim},1,1)$.³³3We denote by $i_{dim}$ the intermediate dimension. We call this intermediate because this is obtained exactly halfway through the forward pass of the entire CIGN, when the latent tensor is outputted by the discriminator and passed to the generator.

• •

  decreasing non-channel dimensions, until the latent tensor has non-channel dimensions $(1,1)$ - that is to say, it is a 1-dimensional tensor.

The generator is defined by mirroring exactly the layers of the discriminator, but in the opposite order: a sequence of transpose convolutional layers with decreasing number of channels and increasing non-channel dimensions. The parameters of each convolutional layer of the discriminator (kernel size, stide, padding, dilation) are equal to those of the corresponding transpose convolutional layer of the generator.

Our main metric is the average Fréchet Inception Distance across all 10 classes. We remark that this is identical to the metric named WCFID in section 3.2 of [1].

For each class, we compare the 1000 samples available in the validation dataset from [12] to 1000 synthetic samples generated by the model under consideration.

This comparison is implemented via the pytorch-fid library (see [7]): this library leverages the ImageNet-v3 model to get intermediate embeddings of the images, which are used to estimate the parameters of the data distribution under a normality assumption.

The aforementioned implementation [7] allows the user to choose among four different embeddings (each one passing the image through a partial inception-v3 model [11] up to a certain layer), but one of them is not available to us due to the small size of the distributions we are comparing (just 1000 samples). The other three embeddings output respectively a $64$-dimensional tensor, a $192$-dimensional tensor and a $768$-dimensional tensor. We denote the corresponding metrics by $\mathrm{FID}_{64}$, $\mathrm{FID}_{192}$ and $\mathrm{FID}_{768}$.

We implement the channel conditioning approach from section 4.1 in its simplest form:

1. 1.

   we train one shared embedding layer

   $$E:C\longrightarrow\mathbb{R}^{\mathrm{emb_{dim}}},$$

2. 2.

   and then for each (possibly transpose) convolutional layer where the input $x_{i}$ has non-channel dimensions $(h_{i},w_{i})$, we train one linear layer

   $$L_{i}:\mathbb{R}^{\mathrm{emb_{dim}}}\longrightarrow\mathbb{R}^{h_{i}\cdot w_{i}},$$

   and we concatenate $L_{i}(E(c))$ to $x_{i}$ along the channel dimension, before the forward pass of the (possibly transpose) convolutional layer.

We implement the filter conditioning approach from section 4.2 in its simplest form:

1. 1.

   we train one shared embedding layer

   $$E:C\longrightarrow\mathbb{R}^{\mathrm{emb_{dim}}},$$

2. 2.

   for each (possibly transpose) convolutional layer where the standard DCGAN architecture in [6] has input $x_{i}$ with $c_{i}$ channels and a kernel of size $(h_{i},w_{i})$, we train one linear layer

   $$L_{i}:\mathbb{R}^{\mathrm{emb_{dim}}}\longrightarrow\mathbb{R}^{c_{i}\cdot h_{i}\cdot w_{i}},$$

3. 3.

   we calculate channel-wise (possibly transpose) cross-correlation between $x_{i}$ and $L_{i}(E(c))$,

4. 4.

   for each (possibly transpose) convolutional layer where the standard DCGAN architecture has $c_{in}$ input channels and $c_{out}$ output channels, we train a channel mixer linear layer

   $$C_{i}:\mathbb{R}^{c_{in}}\longrightarrow\mathbb{R}^{c_{out}},$$

   which we apply to the output of the cross-correlation from the previous step.

In appendices B.1 and B.2 we display samples of the images generated by the best experiments.

In this section we detail the metrics calculated on our trained experiments.

As discussed in section 5.2, our metric is the average Fréchet Inception Distance (FID) across the ten classes. For each trained experiment and each class, we compare the $1000$ samples on the MNIST validation dataset with $1000$ samples generated by the trained experiment. We also report in table 3 minimum and maximum FID across all ten classes.

Essentially all metrics point to the large experiment trained with channel conditioning as the best experiment of the four. On the other hand, that experiment has about 4x as many trainable parameters as the next two experiments (the small experiment trained with channel-conditioning and the large experiment trained with filter conditioning).

The metrics for the small experiment with filter conditioning (the smallest model of all, with only  616k trainable parameters) are clearly better than the small experiment with channel conditioning.

The metrics for the large experiment with filter conditioning are unusually off. Human review of the synthetic images generated by this model suggests that it tends to create ‘thick font’ images (see appendix B.2) which is likely the reason for the large FID values and therefore poor results. We have not investigated why this experiment tends to generate ‘thick font’ images.

In conclusion, we believe that the jury is still out regarding which of the conditioning mechanism is better - the best performing experiment is the large model with channel conditioning, but the small filter conditioning model clearly outperforms a slightly larger model trained with channel conditioning.