Explicit Use of Fourier Spectrum in Generative Adversarial Networks

Neural Networks, also known as Artificial Neural Networks (ANNs), are a set of algorithms in machine learning. The recent deep learning revolution in computer science owes its existence to these frameworks of algorithms. They got their name on their design to mimic the structure of human neurons in the brain. The most simple structure of neural networks, known as Multi-Layer Perceptron (MLP), consists of a layer of nodes (neurons) as the input layer, one or more hidden layers, and an output layer connected with a designed architecture in a layer-wise manner. Values of the nodes in each layer can be seen as a vector and determined by three elements. The first element is the vector of the value of the previous layer. Second, a weight function linearly transforms the previous layer’s outputs, represented as a matrix. Finally, a non-linear activation function adding non-linearity to the structure. Figure 2.1 shows the architecture a simple MLP with two hidden layers, and activation function $\sigma$.
NNs function upon training over a large amount of data, changing their wight matrices gradually with a process called backpropagation to improve their accuracy. The popularity of the NNs is owing to their quick performance and superb accuracy over data-driven tasks such as prediction, classification, and regression.

To measure the performance of the neural network models on supervised tasks, researchers use various metrics. For the classification, we have precision, recall, accuracy, F1-score, etc.; for regression tasks, we have mean square error, mean absolute deviation, etc. However, all of them need a labeled dataset out of hand for any unsupervised task, including GANs. Defining a metric for unsupervised tasks is more difficult. Clustering tasks, for example, are usually evaluated using the Silhouette Coefficient. It measures how similar an item in the cluster is to the other items in the same cluster. GANs case is different and even trickier; there is no label in hand, and there is no ground truth other than some samples from the real dataset. The loss for each neural network is not in any way an indicator of how successful the model is in generating a distribution close to the original sampled distribution. They only show how good is each module (Generator or Discriminator) in fooling the other one. Finding an excellent metric to evaluate the GANs performance has been a topic of controversy for a while now [46]. Nonetheless, for quantitative measurement of their success, there are two widely adopted metrics, the Inception Score (IS) [47] and Fréchet Inception Distance (FID) [48].

Decomposing an image into different size scales with a basis function is a prominent idea in mathematics and engineering. The idea is to write any spatial signal as a sum of orthogonal basis functions to get a new representation based on the different resolutions of the original signal. This representation is known as frequency representation. Most of the decomposition techniques, including the ones described in this thesis, are bijective mappings, i.e., there is a unique representation in the frequency domain for each spatial signal. Arguably frequency domain’s theoretical significance comes from the Convolution Theorem for standard Linear Time Invariant systems. The interactions of such systems with their input and output signals formulate by the convolution operator. The Fourier basis diagonalizes convolution, hence replace it with a simple element-wise multiplication in the frequency domain. The importance does not only summarize in theoretical works. The frequency-domain has opened up various tools for engineers. For audio signals applying filters to remove noise, detect different voices, or equalize the voices of different musical instruments are a small portion of many applications of frequency representations. In the case of the image spectrum, although the information is not apparent by the look of the spectrum, it contains valuable data related to the rate of change in moving to neighbor pixels, the lines and curves and edges of the objects, and noise in the images. It can also tell us how much information there is in a particular resolution, which is important when an image contains chaotic structures. There are helpful filters in noise reduction, edge detection, and many others operating on the images’ frequency domain.
A few series and transforms have been introduced to change the representation of spatial periodic and non-periodic signals to a frequency space, such as Fourier transforms and series, Laplace transforms, and Z-transform. In this work, we adopted the arguably most famous decomposition technique, Discrete Fourier Transform (DFT).

In machine learning theory, supervised learning is usually formalized by assuming that labels $y$ are generated by an unknown function $f$, such that $y_{i}=f(x_{i})$. Also, let us assume that the data points and labels come from an i.i.d sampling of the underlying data. $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{N}$, where $N$ here is the number of observations. The learning objective here is to estimate the function $f$ as best as possible from a set of parameterized functions we have, known as a hypothesis class, $\mathcal{H}=\{f_{\theta\in\Theta}\}$. Neural network architecture can be seen as an instance of such parameterized hypothesis classes with parameters $\theta\in\Theta$ as the networks’ weights. The ideal method to evaluate the chosen function from the hypothesis class is the expected loss defined as follow:

$\tilde{f}\in\mathcal{H}$ is the chosen function, and $P$ is the underlying distribution of the data. $L(.,.)$ is the suitable loss for our task; some examples can be square loss and binary cross-entropy. Thus, to have successful learning, one needs to consider a notion of regularity in choosing functions $\tilde{f}$. This idea of regularization is known as the inductive bias, which can be applied via various methods, explained further in this section.
To have a successful learning scheme, the other important and even more prominent property is that the hypothesis class should be dense enough to have the capacity to approximate functions over current large-scale, high-quality datasets. However, in the case of neural networks, it has never been a problem. Even a simple two-layer perceptron, $f(\mathbf{x})=\mathbf{c}^{\intercal}\sigma(\mathbf{A}\mathbf{x}+\mathbf{b})$ shown to be dense in the space of continuous functions on $\mathbb{R}^{d}$.
To formalize the regularity notion into the learning scheme, we can get help from complexity measure $c\mathrel{\mathop{\mathchar 58\relax}}\mathcal{H}\to\mathbb{R}_{+}$, now the problem can be seen as:

The realizability assumption is considered for this formulation for convenience. In other words, now we are looking for the least complex (more regular) function among our hypothesis class. For the complexity measure, different norms of the working space are a well-known option used in learning approaches such as logistic regression or SVMs. In neural networks, network weights are a proper choice for complexity measurements. i.e. $c(f_{\theta})=c(\theta)$. While explicitly applying regularities to the empirical loss is a common, well-defined approach known as "Structural Risk Minimization," the Implicit approaches are getting more and more attention of researchers [64, 65].
As the dimension of the space goes higher, finding suitable regular functions will get more challenging as well. In the case of the long-established regularity notion, 1-Lipschitz functions. The number of observations needed for the desired loss will grow exponentially. The good news is that real-world applications such as computer vision tend to work with functions that inherently possess some spatial structures in their physical domains. The geometric deep learning objective is to exploit structures as a notion of regularity further to reduce the search space for the most proper function.

Symmetries on a domain or a signal are transformations that leave certain properties of the domain of the signal, or the signal itself, unchanged. Real-world applications are full of these operations. In computer vision’s object detection tasks, shift or rotation are examples of transformations that leave the type of object invariant. If there is an image of a rhinoceros, the shifted or rotated rhinoceros should still be classified as a rhinoceros.
The symmetries of the domain $\Omega$, and signal $\mathcal{X}(\Omega)$ operating on it present a constraint on the function $f$ operating on those signals. It turns out that these symmetries are powerful prior to consider as inductive bias. They will reduce the search space $\mathcal{H}(\mathcal{X}(\Omega))$ to only the functions satisfying the invariancy. Here we explain the two groups of functions we use their symmetric properties in this thesis, invariants and equivariants.

This section describes how we can apply the blueprint to a homogeneous grid space, where images are defined as signals over this space. Understanding this design is essential for the cause of this thesis since designing the Fourier layer for the discriminator of the GAN uses the same method with a similar approach.
Let’s assume that we have a one dimensional grid for simplicity. in this grid each element $\mathbf{x}_{u}$ has a right and a left neighbor $\mathbf{x}_{u-1},\mathbf{x}_{u+1}$. Using the blueprint we can design the equivariant function $\mathbf{F}$ with a local aggregation function $\phi$ operating over a grid element and its neighbors $\phi(\mathbf{x}_{u-1},\mathbf{x}_{u},\mathbf{x}_{u+1})$. Choosing $\phi(\mathbf{x}_{u-1},\mathbf{x}_{u},\mathbf{x}_{u+1})=\theta_{-1}\mathbf{x}_{u-1}+\theta_{0}\mathbf{x}_{u}+\theta_{1}\mathbf{x}_{u+1}$ yields the function $\mathbf{F}$ as the matrix product:

In the machine learning, using this form of diagonal matrix is known as weight sharing. This matrix in an example of a circulant matrix, which is essentially a vector $\mathbf{\boldsymbol{\theta}}=(\theta_{0},\dots,\theta_{n-1})$ append to itself after circular shifts. $\mathbf{C(\boldsymbol{\theta})}=(\theta_{u-v\>mod\>n})$. product of a vector with a circulant matrix is equivalent with cyclic discrete convolution.

In machine learning and signal processing, the above $\boldsymbol{\theta}$ is known as the filter, and its weights/values are subject to the learning process. Circulant matrices have commutativity property, which means their product is commutative, and it is also circulant. More mathematically $\mathbf{C(\boldsymbol{\theta})C(\boldsymbol{\eta})}=\mathbf{C(\boldsymbol{\eta})C(\boldsymbol{\theta})}$ If we choose $\boldsymbol{\theta}=(0,1,0,\dots,0)^{\intercal}$ the resulted circulant matrix $\mathbf{S}$, shift vectors one position right (circularly) and we call it the shift matrix. From the properties of the convolution, and circulant matrices, we get the desired shift equivariance:

An interesting fact is that the other way around is also true. A matrix is circulant if it commutes with the shift. In other words, convolutions are the only linear operators keeping the equivariancy.

Recent state-of-the-art GAN architectures used for vision tasks utilized the convolutional neural networks (CNN) structure for their generator and discriminator modules [3, 4, 54]. Regardless of the network architecture and the loss function difference, they all need to map from a low dimensional latent space to a high dimensional high-resolution image space. To this end different GANs used different upsampling approaches like bi-linear [3, 4, 54], transposed convolution [2, 66, 67] or nearest neighbor [68, 69]. In the simple case, when the up-sampler scales the image by a factor of $m=2$; it inserts a zero between all the pixels in each row/column. Then it applies the convolution to interpolates the inserted zeros with suitable values. A simple formulation of the last two is shown in Figure 5.1.

[20] first analyzed the frequency consequences of the GAN’s up-sampling modules. Properties of the DFT (Discrete Fourier Transform) reveal that zero-insertion in an image will affect the spectrum in a way that looks like multiple copies of the original spectrum attached together. The proof for the one-dimensional case is as follows, and it is easily extendable to the two-dimensional case. Given $x(n),n=0,...,N_{1}$ as the input signal to the up-sampler and $X(k),k=0,...,N-1$ as its DFT. By inserting the zeros we’ll get $x^{\prime}(n),n=0,...,2N-1$ where $x(2n)=x(n)$ and $x(2n+1)=0$ for $n=0,...,N-1$. Now the DFT of the $x^{\prime}(n)$ will be $X^{\prime}(k)$. For $k<N$,

for $k\geq N$, let $k^{\prime}=k-N$, thus $k^{\prime}=0,...,N-1$ then,

The above shows that there will be two copies of the spectrum of the $X(k)$ in $X^{\prime}(k)$ first at $[0,...,N-1]$ and second at $[N,...,2N-1]$. This transformation introduces high-frequency components to the spectrum. [70] discussed that these high-frequency components play a significant role in a checkerboard artifact phenomenon seen in the outputs of generative convolutional networks. Figure 5.2 demonstrate the high-frequency components in the spectrum as well as the checkerboard artifacts. In the next step of the up-sampling, when the convolution occurs, we have neither a guarantee for the learned convolution kernel to be a suitable low-pass filter nor a sampling strategy to avoid such high-frequency artifact appear in the subsequent layer output. Thus, it can not fade out the newly introduced high-frequency components.

Maybe the most famous property of the Fourier domain in signal processing arises from the convolution theorem. That is to say, the convolution operation in the spatial domain is equivalent to element-wise multiplication in the Fourier domain. It is important since the interactions of linear time invariant (LTI) signals and systems can be described with the convolution operation. This property is derivable with properties of symmetry, and circulant matrices explained in section 4.3. To that end, we need to remember that diagonalizable matrices share common eigenvectors (with different eigenvalues) iff they mutually commute. Since circulant matrices satisfy these conditions, we can calculate the eigenvectors of one of them. Shift matrix $\mathbf{S}$ is a convenient choice here, whose eigenvectors luckily happens to be the DFT basis:

Arranging the $\boldsymbol{\phi}$ vectors into a $n\times n$ matrix gives us $\boldsymbol{\Phi}=(\boldsymbol{\phi}_{0},\dots,\boldsymbol{\phi}_{n-1})$. Matrix multiplication with $\boldsymbol{\Phi}$ yield the inverse DFT and with its complex conjugate $\boldsymbol{\Phi^{*}}$ the DFT. All circulant matrices share this eigenvectors, so the Fourier transform of the filter is the eigenvector matrix $\mathcal{F}(\boldsymbol{\theta})=\boldsymbol{\Phi}^{*}\boldsymbol{\theta}$ for the circulant matrix $\boldsymbol{C}(\boldsymbol{\theta})$. Thus, we can write $\boldsymbol{C}(\boldsymbol{\theta})$ in diagonalized form to obtain the convolution theorem:

where $\odot$ is element-wise multiplication. we use the above mentioned characteristic of circulant matrices to provide a proof of work for our frequency domain architecture in the next section.

[20] shed light on the dissimilarities between authentic image spectrum and the images generated by convolutional networks. These disparities mostly show themselves in the forms of high-frequency patches in the generated images spectrum, discussed by [70] as responsible for the checkerboard artifacts showed in them. Knowing that Fourier is a bijective mapping, any apparent difference in the spectrum demonstrates a flaw in learning the original distribution, which is the main objective of training GANs as a generative model. In this work, we are developing a new architecture for the discriminator to address the spectral difference where the discriminator gets the images in the Fourier and the spatial domains. We then experimentally demonstrate that our model can enhance the learned distribution, emphasizing the importance of the spectrum of images on the learning process.

The most straightforward way for solving dissimilarities in the spectrum is to add frequency information to the discriminator. [5, 55, 56] have already proved that feeding information to the discriminator can boost the performance of the GANs for other applications. In this case, the network would better detect high-frequency components’ differences in the generated images by providing the discriminator with frequency information. Thus, it pushes the generator to produce images matching the actual image distribution’s spatial and frequency representation. Equation (5.3) shows the new loss function for the GAN incorporating the Fourier transform $\mathcal{F}(.)$ of the images directly in the loss.

For preprocessing the Fourier information [50] discusses the following 1D representation of Fourier transform via azimuthal integration over radial frequencies, practical enough to point out the dissimilarities of the spectrum and spatial domain. Assuming that images are $M\times M$.

Equation (5.4) can be seen as a coordinate conversion from Cartesian to polar and then getting the mean intensity over the radial distance. It is also used in [71] as the input to the frequency module. Despite its popularity due to more straightforward computation in 1D, performing average operation will cause loss of information in FFT amplitudes and completely disregard the phase information, which proved to be an essential part of the spectrum [72, 73]. In this work, we use both amplitude and phase of the image as the input to our frequency module to fully exploit the spectrum information.

At a high level, our work consists of two sections. First, we introduce a high-level architecture with unary losses (Figure 5.3) to incorporate frequency information into the GAN architecture in a guided manner. It will result in a discriminator more sensitive to frequency discrepancies, which pushes the generator to produce images more realistic in the frequency domain. Second, we devise a Frequency module to use in our architecture. This module’s task is to get the frequency spectrum and output a realness score. The frequency module uses a base layer we named EV-Freq which is designed based on the blueprint provided by geometric deep learning.

Before evaluating any result, we need to ensure that the model converges. Otherwise, the results can not serve as an indication of the superiority of our model. In GAN tasks, unlike conventional tasks, the objective in training is not improving variables such as decreasing a loss. We care about a gradual and continuous improvement in both agents involved in the process, meaning generator and discriminator. If any agents get too strong with respect to the other, the balance is disturbed, and the model will fail. This multi-agent scheme made GANs more susceptible to the instability of training. In fact, GANs are famous for being hard to train due to instability, and there have been several works to make them more stable. In this section, we provide empirical substantiations that our extension to the GAN framework will not jeopardize the stability of the baseline. We further discuss how each of the spatial and frequency modules acts separately convergence-wise and how they contribute to the convergence of the discriminator in total.
When networks are out of balance, we see a fast convergence in one of them, showing that the converged network has won the competition. It can happen when one network is inherently more potent. To address this kind of problem, we update the weaker network more frequently. In our cases, the baseline updates the discriminator 5 times for each generator update, and our extension uses the same ratio without instability. The other cause of failure in GANs is mode collapse. A mode collapse refers to a generator model that can generate one or a small subset of different outcomes or modes. It happens when the generator finds a way to produce plausible outcomes for the discriminator without imitating the distribution. Mode collapse is easily detectable by looking at the generated images since they come from a small set of images and are not diverse. In the learning curve, it shows itself as an oscillatory behavior in the generator loss (and, in some cases, discriminator loss).
Figures 6.1 and 6.2 show the Discriminator’s total loss, sum of the discriminator’s fake and real losses, and generator’s loss together for CIFAR100 and stl10 datasets. The curves are in a balanced manner showing fair competition while converging to the same loss. There is no instability in the curves, whether a fast convergence or an oscillatory behavior. For a deeper look, each discriminator loss consists of 3 other losses coming from the spatial and frequency modules and one joint loss from the J module. Figures 6.3 and 6.4 show the derailed losses for the mentioned datasets. Frequency loss has a lower value compared to the spatial loss in CIFAR100 and greater in the stl10. It shows that the frequency domain is as useful as the spatial domain to detect the fake images for the discriminator. The joint loss trains to gives most of its attention to the more functional loss. The power of unitary modules emerges here. While spatial and frequency losses equally contribute to the $\frac{1}{3}$ of the discriminator loss, the joint loss which is responsible for the last $\frac{1}{3}$ of the total loss, automatically learns to adopts its weights for better use of each domain.
Figures 6.5 and 6.6 show the raw scores used by discriminator for classification. The real scores are expected to be above zero and fake scores below zero. Although joint score mostly follows the spatial scores, frequency scores show fewer fluctuations and well above and below the 0 threshold, making it a valid feature for classification.
To recapitulate, frequency features showed to be a good representative for fake and real images in the discriminator and made the discriminator more powerful. The Generator also shows the capacity to be pushed along with the frequency extension in the discriminator without losing stability.

We use the conventional methods, Frechet Inception Distance and Inception Score explained in the section 2.3 for quantitative comparisons in the spatial domain. FID is calculated over 50000 samples of fake and real images. Inception score is computed over 50000 samples from the fake images with 10 splits. The evaluation is against the baseline (SNGAN) [sngan] and the SSDGAN [71] model. SSDGAN, the same as our model, uses frequency domain to enhance the discriminator. They use the azimuthal Integral of the amplitude as an input for their frequency module, which is a fully connected layer. Their model disregards the phase information of the FFT completely. While the amplitude of the FFT carries the intensity of each frequency, Phase information contains the location of these frequencies. To a human eyes phase information looks more important; hence we have not set them aside in our model. The other drawback of their model is using the azimuthal integration over FFT. By binning the same frequency resolutions, their model will lose the amplitude location information of the FFT. On the other hand, Ours takes the FFT as it is without any information loss as input.
table 6.10 and 6.11 demonstrate the quantitative comparison. Our model outperforms all other models except the inception score for stl-10, which is less but comparable to SSDGAN. However, SSDGAN has some major flaws, which we fully describe in the next section. Note that for the FID, the lower, the better, and for inception score, the higher, the better.

Figure 6.7 and 6.8 show some examples of the images created based on CIFAR100 and stl10 datasets respectively. We have not cherry picked the images and they are random images generated by the generator of our model.

Unlike the spatial domain, Evaluation in the frequency domain does not have a well-established method. To illustrate how generated images are similar to the real ones in the frequency-domain power spectrum is a good candidate. To further show the difference, we use the power spectrum distance between real images and those produced by the GAN models, normalized by the spectra of real images. (used in [77])

We also introduce the 2D frequency amplitude gap, which uses the average distance between the expected 2D-frequency gap between the amplitude of the authentic images and synthetic images normalized by those of authentic images.

Figure 6.9 shows the power spectrum of our model with real images and the other two models in the stl10 dataset. FreqGAN has the closest distance from the real image. It is more clear looking at the normalized power spectrum distance in figure 6.10. The ideal generator should have a constant zero distance throughout all the frequency bins. To our surprise, despite getting the azimuthal integration of the magnitude, which is essentially a variation of the power spectrum itself, SSDGAN has even poorer performance than the baseline SNGAN in the power spectrum. We believe this is owing to fully connected architecture. The fully connected neural networks have a dense hypothesis class in the frequency domain; therefore, estimating the best function even if the neurons are not many is hard for the neural network. In addition, the lack of unitary modules will cause an uncontrolled effect of these not-well-trained elements in the training process. FreqGAN, contrastingly, shows an almost flat curve over all frequency bins. addressing the high-frequency discrepancies in the SNGAN.
Figure 6.11 measures the same idea in a more direct way with the normalized 2d frequency spectrum gap. The color map shows the difference; the darker the pixel, the more significant the difference between the fake and authentic images. FreqGAN here shows the best performance overall again, while SSDGAN shows inferior performance than the baseline.
For further discussion, we believe the difference of our work will show itself for higher resolution photos more significantly. The high-resolution photos suffer more from high-frequency discrepancies, and our model shows a spectacular result in addressing them. In our plans for future work, we have in mind to further analyze the relation of resolution with the performance of our model.

We used the powerful tool of geometric deep learning to design our architecture for the frequency domain. In this section, we discuss its performance over the fully connected architecture and regular CNN. We trained 3 trails of neural networks for each architecture. The fully connected version is the same as the work done by SSDGAN [71]. The CNN version is equipped with spectral normalization as well to keep the comparison fair. Tabel 6.12 shows that GDL based architecture outperforms other models by a significant margin.

This thesis brought up current flaws in the CNN-based image generation techniques (such as generative adversarial networks), which made them easily distinguishable from authentic images. Then we investigate the possible justification for such flaws both mathematically and empirically. We found the main reason in high-frequency discrepancies between the spectrum of synthetic and authentic images. We showed that this disparity has its roots in the transconvolution layers of CNNs. This work proposed a solution for the GANs. An additional path for the discriminator, processing the spectrum of the images. Knowing the frequency representation of the images, the discriminator is equipped with direct access to detect problematic high-frequency elements. On the other side of the story, the generator adopts its weight to reduce the high-frequency elements, resulting in more realistic, hard to distinguish images.
Different datasets exhibit different distributions in their spatial and frequency features. Some are easier to detect in frequency, some in spatial. The proposed unitary modules in this work adjust the amount of attention the model gives to frequency or spatial domains based on their importance in the data. This scheme in leading the optimization not only resulted in better outputs but also stabilized the training.
This work empirically investigates different architectural designs for the frequency path of the discriminator, including fully connected and CNN schemes, and comes up with a brand new architecture EV-Freq. EV-Freq has systematically designed based on the geometric deep learning blueprint. It utilizes the physical constraints of shift invariancy in the images as an inductive bias to reduce the search space and improve the neural network’s estimation ability on the spectrum of the corresponding images. This thesis provides complete theoretical analysis and proofs for the newly design architecture.
We evaluate our extension to the GANs framework over two datasets, CIFAR100 and stl10, in spatial and frequency domains. In the conventional spatial domain, it outperformed the baseline by a significant margin. Our model also outperformed the other frequency-driven effort in improving GANs, SSDGAN. FreqGAN achieved an outstanding performance in the frequency domain, generating images almost identical to real images concerning the power spectrum. GANs are notorious for training instability. A section of this work focused on empirical substantiations that the proposed extension would not jeopardize the stability of the baseline.

The future steps for the current research are listed as follow:

• •

  This work improves the quality spectrum characteristics of the latent distribution of the generator by improving the discriminator power in frequency. Yet, it does not propose any direct change in the generator’s architecture. Investigating new designs for the generator to facilitate adopting with the frequency features is one of our prominent future lines of research

• •

  EV-freq introduced a shift equivariant function operating on frequency domain; however, these classes of functions keep the phase information intact, in the future steps, we continue our search to find functions operating on the phase of FFT spectrum to utilize them in the detection process of the discriminator.

• •

  Current work uses the FFT of the grayscaled version of the images. In other words, it just takes advantage of the brightness frequency in the images. For future steps, we intend to extend our work to a colored version of the image utilizing the color frequency information in our model

• •

  New PyTorch update introduces complex autograd. Since back-propagation is now available for complex numbers, we can readily work with complex losses. It promises a new architectural design with no need for IFFT at the end of the block, which can improve our performance in the spectral domain and improve the memory usage and computation complexity.