Mixture Density Conditional Generative Adversarial Network Models (MD-CGAN)

Generative Adversarial Networks (GANs) have been one of the many breakthroughs in Deep Learning methods in recent years. Several different variations of the model have been introduced since the method was first introduced in [1]. One of the most popular variations of this work is that of the Conditional Generative Adversarial Network (CGAN) [2] in which the generator and discriminator (we briefly review the GAN process in Section 2) are both conditioned on some observed information. In the application of time series forecasting, future values are conditioned on information observed from the past, either from the time series itself, some set of associated exogenous data, or a combination of the two. This conditioning addition to the GAN formalism makes the CGAN approach particularly useful as a foundational model for time series prediction. Most applications of (C)GANs, however, have been within computer vision and, to a lesser extent, in natural language processing and simulation models [3, 4, 5].

The literature on the application of any form of GAN model to problems associated with time series is, to date, limited. However, some recent literature shows the potential usefulness of the method. For example, the work reported in [6] applies a recurrent GAN to generate realistic, synthetic, medical data series and in [7] a (standard) GAN model is used to generate realistic financial asset prices and analyse their distributions.

In [8] a GAN is used to forecast high-frequency stock data and in [9] GANs are used to generate missing values in incomplete time series. We note that the GAN models used in all these applications make point estimates for the forecast. Although a perfectly valid approach, and one that has a long history in time series forecasting, we argue that probabilistic forecasts are a prerequisite in many application domains, in which knowledge of the predictive uncertainty is as vital as the prediction itself. In this work, we expand on the CGAN algorithm to allow a full predictive probability distribution, rather than a point value. To obtain richer predictive densities we model the posterior distribution using a finite Gaussian Mixture model (GMM). Although we find only occasional evidence that such non-Gaussian predictions offer significant benefits, we note that producing them is not much more costly than single Gaussian predictive distributions, and so present our approach as a more general multi-component model.

The goal of the GAN model is to estimate a generative model using an adversarial process [1]. This is achieved by simultaneously training two models. Firstly, a generative model $G$, that in the case of data forecasting learns past patterns in the data and infers the predictive values. Secondly, a discriminative model $D$, that determines how likely a sample was to originate from the ‘true’ training data, compared to being sampled from the generator. The generator is hence matched against an adversary, the discriminator (whose goal is to detect the difference between a true data sample and one created by the generator). Components of the model are then trained (via an optimization process) to maximize the probability of the discriminator being unable to distinguish true from generated data samples. Typically, including the approach we take here, the generator and the discriminator are both constructed as multilayer perceptrons, with stochastic gradient methods being employed to obtain optimization.

We start by formulating the definitions which we use in common across the GAN models we test. We consider a time series, $y_{t}$. Our aim is to estimate the forecast of some $y_{t^{\prime}>t}^{f}$, conditioned on a set of observations which we denote $\mathbf{x}_{t}$. The inputs to the generator network are $\mathbf{x}_{t}$ and $\mathbf{z}_{g}$, where $\mathbf{z}_{g}$ is a collection of samples from a normal distribution, $p(z_{n})_{g}=\mathcal{N}(0,\mathrm{var}_{\mathrm{data}})$. During model training, the output from the generator, $y_{t^{\prime}}^{f}$, as well as the true forecast sample $y_{t^{\prime}}$, are fed to the discriminator, whose role is to discriminate between them i.e. to identify $y_{t^{\prime}}^{f}$ as the ‘fake’ sample.

In an unconditioned GAN model, there is no control over the data that is generated. In the CGAN model, in contrast, by conditioning the model on additional information, it is possible to direct the data generation process [2]. Schematics of the GAN and CGAN models are depicted in Figure 1.

As with the GAN and CGAN methods, we consider a time series, $y_{t}$. Our aim now is to infer the posterior distribution over some $y_{t^{\prime}>t}$, conditioned on the set of observations, $\mathbf{x}_{t}$. In order to form the posterior distribution we model the full conditional density $p(y_{t^{\prime}}|\mathbf{x}_{t})$ as an adversarial network. To achieve this we use a Mixture Density Network (MDN) model [15] for the generator $G$. The inputs to the generator network are as per the CGAN approach, $\mathbf{x}_{t}$ and $\mathbf{z}_{g}$, where $\mathbf{z}_{g}$ is, as before, a collection of samples from a normal distribution, $p(z_{n})_{g}=\mathcal{N}(0,\mathrm{var}_{\mathrm{data}})$. The outputs of $G_{t^{\prime}}(\mathbf{x}_{t},\mathbf{z}_{g})$ are now, however, the parameters of the Gaussian mixture posterior over the forecast. This mixture has mixing coefficient, standard deviation and mean for the $i$-th component denoted as $\alpha_{i}$, $\sigma_{i}$, and $\mu_{i}$ respectively. As first proposed in [15], we achieve this by using latent variables $\mathbf{s}=\{\mathbf{s}_{\alpha},\mathbf{s}_{\sigma},\mathbf{s}_{\mu}\}$, conditioned on the inputs. The mapping from $[\mathbf{x}_{t},\mathbf{z}_{g}]\mapsto\mathbf{s}\mapsto\{\alpha_{i},\sigma_{i},\mu_{i}\}$ is modelled via our network. As the mixings must satisfy $\sum_{i}\alpha_{i}=1$, we map $\mathbf{s}_{\alpha}$ to $\mathbf{\alpha}$ via the softmax function, where $\mathbf{\alpha}_{i}=\frac{\exp({s}_{\alpha,i})}{\sum\limits_{i^{\prime}}\exp({s}_{\alpha,i^{\prime}})}$. The elements of $\mathbf{\sigma}$ are strictly positive so we adopt, $\sigma_{i}=\exp(s_{\sigma,i})$. Finally the means can be mapped directly from the latent variables, hence $\mu_{i}=s_{\mu,i}$. Schematically, the MD-CGAN method is depicted in Figure 2.

The above formalism allows us to directly model the predictive likelihood conditioned on an input, and the likelihood of $G$, conditioned on the observations $\mathbf{x}_{t}$ and samples $\mathbf{z}_{g}$ as:

where $m$ is the number of mixture components.

As in the CGAN model, the discriminator, $D$, is also conditioned on $\mathbf{x}_{t}$. The input to the discriminator model is, by design, $\mathbf{x}_{t}\sqrt{2\pi}\sigma_{a}\mathcal{L}(y_{t^{\prime}})$, where $\sigma_{a}$ is the standard deviation of the set of observed $y_{t}$ ¹¹1We note that the GAN approach is not sensitive, within reason, to this value (it is, in effect, a constant in the update equations) and we discuss its setting later in the paper.. For true values of $y_{t^{\prime}}$, $\sqrt{2\pi}\sigma_{a}\mathcal{L}(y_{t^{\prime}})$ is maximized. The generator tries to ‘fool’ the discriminator by generating $G_{t^{\prime}}$ such that the $\sqrt{2\pi}\sigma_{a}\mathcal{L}(G_{t^{\prime}})$ is maximized. The loss function for the generator, $L_{G}$ is as in Equation 2, The discriminator network, on the other hand, attempts to differentiate between true $y_{t^{\prime}}$ values and the pseudo-values created by the generator. The loss function for the discriminator, $L_{D}$ is as in Equation 3. We note that the lowest value of the discriminator loss is achieved when $\sqrt{2\pi}\sigma_{a}\mathcal{L}(y_{t^{\prime}})$ is maximal (unity) and $\mathcal{L}(G_{t^{\prime}}(\mathbf{x}_{t},\mathbf{z}_{g}))$ is minimal (zero).

Our algorithm thus follows the steps in Algorithm 1 below.

In this paper we present the MD-CGAN model, which offers extensions to the CGAN [2] methodology, particularly to allow for GAN inference of a (multi-modal) posterior distribution over forecast values. In the experiments considered, we find the MD-CGAN approach outperforms other methods on all datasets in which noise is prevalent, including all the financial time series investigated, over long term forecast horizons. As a GAN model, our approach retains adversarial robustness in forecasting, which we find is most notable when noise is extensively present in data. Our method is thus particularly well suited to dealing with financial data. Furthermore, MD-CGAN can effectively estimate a flexible posterior distribution, in contrast to standard GAN models which (almost without exception) produce point value outputs. Exploiting this rich, multi-model, posterior distribution is not reported in detail here but will feature in follow-up work. In summary, the MD-CGAN model combines the advantageous features of both flexible probabilistic forecasting and GAN methods. We see this as a particularly useful approach for dealing with time series in which noise is significant and for providing robust, long-term forecasts beyond simple point estimates.