Fully Embedded Time-Series Generative Adversarial Networks

The authors declare no known competing interests in the content of this manuscript.

Funding for this work was partially provided by the Collaborative Sciences Center for Road Safety (CSCRS), as well as the University of Tennessee, Knoxville.

Consider the random vector $X\in\mathcal{X}$, where individual instances are denoted by $x$. We operate in a discrete time setting where fixed time intervals exist between samples $X_{t}$, forming sequences of length $T$, such that $(X_{1},...,X_{T}):=X_{1:T}\in\mathcal{X}^{T}$. We note that $T$ may be a constant value or may be a random variable itself, and that the proposed methodology does not differ in either case. The representative dataset of length $N$ is given by $\mathcal{D}=\{x_{n,1:T}\}_{n=1}^{N}$. For convenience, the subscript $n$ is omitted in further notation.

The data distribution $p(X_{1:T})$ describes the sequences, such that any possible conditional $p(X_{t}\mid X_{t-i})$ for $t\leq T$ and $t-i\geq 0$ is absorbed into $p(X_{1:T})$. Thus, the overall learning objective is to produce the model distribution $\hat{p}(X_{1:T})$,

$\displaystyle\min_{\hat{p}}D(p(X_{1:T})\mid\mid\hat{p}(X_{1:T}\ ))$

(1)

where $D$ is some measure of divergence between the two distributions. We note here that RCGAN [1] applies adversarial loss that minimizes this divergence directly. In the following works, higher quality data is generated by supplementing this loss function with a supervised loss that is more stable [5], or by avoiding adversarial loss altogether [12]. While the adversarial autoencoder in Section 3.2 is intended to supplement and further stabilize training as well, we also highlight the additional ability of the $\operatorname{FAT}$ operator to stabilize the minimization of this objective on reconstructions in Section 3.3.

In order to represent complex temporal relationships, we would like to leverage the ability to encode the entire time-series as a low dimensional vector in a seq2seq style model. We introduce the random variable $Z\in\mathcal{Z}$ as an intermediate encoding for producing $\hat{p}(X_{1:T})$. We also introduce $\eta_{1:T}$ as a random noise vector from a known distribution $p_{\eta}$ sampled independent of time. Let $p_{z}(Z)$ be a known prior distribution, the encoder function $e:(\mathcal{X}^{T},\mathcal{\eta}^{T})\rightarrow\mathcal{Z}$ have the encoding distribution $q(Z\mid X_{1:T},\eta_{1:T})$, and the generator (or decoder) function $g:(\mathcal{Z},\mathcal{\eta}^{T})\rightarrow\mathcal{X}^{T}$ have the decoding distribution $\hat{p}(X_{1:T}\mid Z,\eta_{1:T})$.

Also, we can define an ideal mapping function that maps sequences to vectors $M:(\mathcal{X}^{T},\eta^{T})\rightarrow\mathcal{M}$, such that $M_{X}=M(X_{1:T},\eta_{1:T})$ and $p_{M}(M_{X})=p(X_{1:T})p_{\eta}(\eta_{1:T})$. The aggregated posterior can now be defined,

$\displaystyle q(Z_{X})=\int_{M_{X}}q(Z\mid X_{1:T},\eta_{1:T})p_{M}(M_{X})dM_{X}$

(2)

Similar to Eq. 1, encoder training occurs by matching this aggregated posterior distribution $q(Z_{X})$ to an arbitrary prior distribution $p_{Z}(Z)$.

$\displaystyle\min_{q}D(p_{z}(Z)\mid\mid q(Z_{X}))$

(3)

The posterior distribution $q(Z_{X})$ is equivalent to the universal approximator function in [3], where the noise $\eta_{t}$ exists to provide stochasticity to the model in the event that not enough exists in the data $\mathcal{D}$ for $q(Z_{X})$ to match $p_{z}(Z)$ deterministically.

The generator is trained through reconstructing the data distribution $p(X_{1:T})$ while being conditioned on the encodings.

$\displaystyle\min_{\hat{p}}D(p(X_{1:T})\mid\mid\hat{p}(X_{1:T}\mid Z_{X},\eta_{1:T})))$

(4)

If divergence is minimized in Eq. 3 and Eq. 4, then the encoder is able to perfectly replicate the prior distribution, and the generator is perfectly able to reconstruct $X_{1:T}$, resulting in a complete model of $p(X_{1:T})$ when the prior distribution $p_{z}(Z)$ is sampled and decoded at inference time.

In our formulation, the generator model minimizes two measures of divergence. First, there is the direct matching of the target distribution through adversarial training, characterized by Eq. 1. Then, there is the secondary reconstruction loss from the intermediate encodings, characterized by Eq. 4. Reconstructions of time series data are denoted $\bar{x}_{1:T}=g(e(x_{1:T}))$. The simplest loss function to enforce reconstruction of the original data might be,

$\displaystyle\mathcal{L}=\mathbb{E}_{x_{1:T}\sim p}(\sum_{t}\|x_{t}-\bar{x}_{t}\|^{2})$

(5)

where this objective is minimized in tandem by the generator and the encoder. This may prove to be challenging as our generator model is not an auto regressive model. It takes the intermediate encodings as input at every timestep, as shown in Fig. 1. This reduces the possibility of a vanishing gradient to the encoder itself, and also prevents the generator from forgetting the initial encoding for long sequences. This does, however, make learning through reconstruction more difficult due to the inability to incorporate teacher forcing methods. In addition, time-series reconstruction with real-valued data faces the optimization problem of large local minimums, where the model may collapse into producing only mean values across any one time-series or the mean of the entire dataset. To alleviate these problems, we propose the solution of only applying reconstruction loss at one time instance $t=\tau$ in the sequence, instead of the entire time series at once. Thus, the reconstruction loss,

$\displaystyle\mathcal{L}_{recon}=\mathbb{E}_{x_{1:T}\sim p}(\|x_{\tau}-\bar{x}_{\tau}\|^{2})$

(6)

trains the generator in a supervised manner. The question remains which time instance $\tau$ to choose. We propose a simple solution. Prior to training, we define a real-valued threshold $\epsilon$, such that any error that has compounded to produce the reconstruction $\bar{x}_{t}$ is acceptable so long as $\|x_{t}-\bar{x}_{t}\|^{2}<\epsilon$. In this way, time-series reconstructions are progressively learned from short-term to long-term, instead of all at once. To this end, the First Above Threshold (FAT) operator is introduced. This operator takes as input a sequence $l_{1:T}$ and a threshold $\epsilon$, such that the minimum value for $t$ is returned where $l_{t}>\epsilon$. In the event that $l_{t}<\epsilon\,\forall\,t\in T$, $\operatorname{FAT}_{t}(l_{1:T},\epsilon)=\operatorname*{arg\,max}_{t}(l_{1:T})$. With this newly defined operator,

$\displaystyle\tau=\operatorname{FAT}_{t}(\|x_{t}-\bar{x}_{t}\|^{2},\epsilon)$

(7)

defines $\tau$, thus defining the complete form of Eq. 6. The benefits are two fold. First, a progressive learning approach stabilizes the early portion of training, as the objective function may be less likely to become stuck at a local minimum while trying to encode and reconstruct long sequences all at once. Second, updating the parameters corresponding to only one time instance at a time has a regularizing effect by providing a more granular gradient that is less likely to interfere with the adversarial training for both the encoder and generator. The combination of both of these effects are demonstrated in Fig. 2, as the training dynamics of the model are compared between using the reconstruction loss of Eq. 5 and Eq. 6. We show that the application of the objective in Eq. 6 actually minimizes the loss of Eq. 5 more effectively than applying it directly in the sines dataset. The efficacy of the $\operatorname{FAT}_{t}$ operator can be expected to grow with longer, more complex sequences.

Time series data collected in the physical world such as sensor measurements will have some degree of stochasticity. We cannot replicate this stochasticity using a reconstruction objective alone. To this end, we introduce the feature space discriminator $d_{x}:\mathcal{X}^{T}\rightarrow\mathcal{Y}^{T}$ which maps sequences to classifications, such that $y_{1:T}=d(x_{1:T})$ and $\hat{y}_{1:T}=d(\bar{x}_{1:T})$. This loss is applied to the reconstructions $\bar{x}_{1:T}$, such that the objective can be minimized both through the encoder and the generator. In the Least Squares GAN form, the objective function of the discriminator described by,

		$\displaystyle\mathcal{L}_{dx}=\frac{1}{2}\mathbb{E}_{x_{1:T}\sim p}(\sum_{t}\|1-y_{t}\|^{2})+\frac{1}{2}\mathbb{E}_{\bar{x}_{1:T}\sim\hat{p}}(\sum_{t}\|\hat{y}_{t}\|^{2})$		(8)
		$\displaystyle\mathcal{L}_{fx}=\frac{1}{2}\mathbb{E}_{\bar{x}_{1:T}\sim\hat{p}}(\sum_{t}\|1-\hat{y}_{t}\|^{2})$		(9)

We now introduce the encoding discriminator $d_{z}:\mathcal{Z}\rightarrow\mathcal{Y_{Z}}$ such that $y_{z}=d_{z}(z)$ and $\hat{y}_{z}=d_{z}(z_{x})$. Also in the LSGAN form,

		$\displaystyle\mathcal{L}_{dz}=\frac{1}{2}\mathbb{E}_{z\sim p_{z}}(\|1-y_{z}\|^{2})+\frac{1}{2}\mathbb{E}_{z_{x}\sim q}(\|\hat{y_{z}}\|^{2})$		(10)
		$\displaystyle\mathcal{L}_{ez}=\frac{1}{2}\mathbb{E}_{z_{x}\sim q}(\|1-\hat{y_{z}}\|^{2})$		(11)

describe the objective functions for the discriminator and encoder, respectively.

Putting everything together, there are three measures of divergence we will minimize with our complete model. The adversarial training between the objective functions described by Eq. 8 and Eq. 9 minimize Eq. 1 directly, where $D$ is the $\chi^{2}$-divergence in the LSGAN formulation. The adversarial training described by the objective functions Eq. 10 and Eq. 11 apply to the divergence of Eq. 3, also minimizing $\chi^{2}$-divergence of the intermediate encodings. Finally, Eq. 6 corresponds to Eq. 4, minimizing the Kullback–Leibler (KL) divergence through maximum likelihood (ML) supervised training. In total, all parameter optimization occurs through,

		$\displaystyle\min_{e}\min_{g}(\lambda_{1}\mathcal{L}_{recon}+\mathcal{L}_{ez}+\lambda_{2}\mathcal{L}_{fx})$
		$\displaystyle\min_{d_{z}}(\mathcal{L}_{dz})$
		$\displaystyle\min_{d_{x}}(\mathcal{L}_{dx})$

thus describing the complete objective of the FETSGAN architecture.

The hyperparameters of the model are $\lambda_{1}$, $\lambda_{2}$, and $\epsilon$. Additionally, while not described in notation, the dimensionality of $p_{z}$ and $p_{\eta}$ are also parameters of the model. In terms of tuning these values, $p_{z}$ should generally correspond to the estimated complexity of the input signals while $p_{\eta}$ should correspond to the complexity of the noise. Overestimating these values may simply lead to more complex encodings than are required, and potentially more unstable learning in the embedding space described by Eqs. 10 and 11. $\lambda_{1}$ and $\lambda_{2}$ depend primarily on the scaling of the data. In our experiments, all data is normalized between $(-1,1)$. Finally, $\epsilon$ should be chosen based on the model’s use-case regarding what amount of error is acceptable in reconstruction. In other words, this value represents the amount of error tolerable for the signal, where any error under this value can be considered stochastic noise subject to adversarial feature space learning. All hyperparameters are fairly robust. To demonstrate this, the parameters are fixed for all experiments. The primary parameters are $\lambda_{1}=10$, $\lambda_{2}=1$, and ${\epsilon}=0.1$. The prior distribution $p_{z}$ and the noise distribution $p_{\eta}$ contain four dimensions, and both are sampled from $p_{z},p_{\eta}\sim\mathcal{U}(-1,1)$. All models are trained with the Adam optimization strategy [31], where the learning rate for the generator, encoder and both discriminators is $0.001$. These learning rates decay exponentially in the last $10\%$ of epochs. The full model implementation in Pytorch and instructions for experimental reproduction are provided in the link. ¹¹1https://github.com/jbeck9/FETSGAN

For our experiments, we compare the performance of our model primarily against TimeGAN [5] and RCGAN [1]. As a baseline comparison, we have also included a purely autoregressive method that was trained using only teacher forcing. The methods of [11, 12, 13] are omitted at the time of writing due to a lack of available implementations for reproduction. Due to those limitations, we limit the scope of our conclusions to time series models with adversarial learning applied directly to the feature space, or low dimensional encodings of the feature space.

The efficacy of our model is shown along three dimensions. First, we demonstrate the qualitative similarity between the original data and our generated data in Section 4.2. Then, we demonstrate the unique ability of our method to interpret the prior sampling distribution as a way of providing selective samples that are similar in style to specific samples from the original dataset in Section 4.3. Finally, we demonstrate that under strenuous classification and prediction tasks, our method holds state of the art performance among adversarial learners that apply directly on the feature space for time series data in Section 4.4.

We use three primary datasets for analysis. First, we generate a one dimensional sines dataset of length $T=100$, where the amplitude, frequency, and phase for each sequence are sampled from a uniform distribution. We also use the six-dimensional historical Google stocks dataset, containing stock price information from 2004 to 2019 of various lengths $T$. This dataset is provided directly in the code repository. Finally, we use 6 dimensions of the UCI Appliances energy dataset [32] and real-valued traffic and weather data from the UCI Metro Interstate dataset [33].

Visualizing synthetic data generated from an adversarial learning process is important to analyze the extent of mode collapse that may have occurred. This task is tricky for time series data, as it is possible that temporal mode collapse could exist, but be obscured if the temporal dimension is flattened for analysis. In the case of the sines dataset, we can reduce the sequences to a single dimension by simply capturing the dominant frequency in the sequence, using the Discrete Fourier Transform (DFT). Here, the dominant frequency is given by $\operatorname*{arg\,max}_{f}DFT(x_{1:T})$. Since the original data consists of a sine wave with a single frequency, this provides a valid analysis of the entire sequence. The amplitude and phase of the corresponding dominant frequency are taken as well. A histogram can then be produced, comparing the distribution of frequencies, amplitudes, and phases in each dataset. For the stocks and energy datasets, we settled for a visual comparison in the flattened temporal dimension using TSNE visualization [34], repeating the procedure reported in [5]. The results of these visualizations are shown in Fig. 3. While the results for TimeGAN generally match what is shown in [5], FETSGAN demonstrates a substantial improvement over TimeGAN, RCGAN, and the baseline autoregressive models in matching the underlying distribution for all datasets used.

There is an obvious use case that at inference time, perhaps there is a need to produce a specific style of data, or to sample from a specific portion of the data distribution $p(X_{1:T})$. Because our model forces dimensionality reduction through an encoder that is forced to match $q$ to $p_{z}$, we can leverage spatial relationships in the latent space $z$ to selectively sample from the prior distribution $p_{z}$ at inference time. This allows us to produce synthetic data that retains a specific style. To demonstrate this, three sine waves were taken from the data, corresponding to sequences $x=sin(2\pi ft)$ with $f=2,5,8$. These sequences were then encoded as $z_{x}=e(x)$. Finally, new sequences were generated by adding noise to these encodings, such that $x_{\eta}=g(z_{x}+\eta)$, where $\eta\sim\mathcal{N}(0,0.1)$. The results in Fig. 4 show that, as expected, spatial relationships are maintained in the latent space $z$. We are able to produce synthetic sine waves that maintain the close relationship to the anchor point $x$ they were sampled near, allowing the ability to produce synthetic sequences within an expected range of style.

To compare quantitative performance between models, we apply two testing metrics, discriminative score, and predictive score. Discriminative score is measured by training an ad hoc RNN classifier to discriminate between real dataset and a static synthetic dataset generated by each model. The best model will have the lowest score $0.5-pred$, corresponding to how far the predictions were below the decision boundary, where a score of $0$ corresponds to indistinguishable data. In the case of prediction, the “Train on Synthetic, Test on Real” (TSTR) approach is used [1]. A simple RNN is trained as a forecasting model, predicting 1, 3, and 5 steps into the future, given the sequence $x_{1:t}$ for any valid step where $t+step\leq T$ on synthetic data. Then, MAE prediction error from the trained model is measured on the real dataset. The best model is the one which produces the lowest prediction error on real data. Whenever the model under test calls for an RNN style network, a Gated Recurrent Unit (GRU) of 64 cells and 3 layers was used. For each architecture, 3 models were trained, and sampled 5 times. Thus, 15 tests were conducted for each value in Table 1. Training for all models occurred under 1000 epochs using the Adam optimizer with a learning rate of $0.001$, including the classification and prediction models. Variations of FETSGAN are included to analyze sources of gain. FETSGAN-FAT removes the $\operatorname{FAT}_{t}$ operation by replacing Eq. 6 with Eq. 5. FETSGAN-FD trains without the feature space discriminator $d_{x}$. The complete model scores best in totality, and the variations of the FETSGAN show statistically significant improvement over RCGAN and TimeGAN in all cases. We note that in the case of the noisy and lengthy Energy dataset, only the complete FETSGAN model was able to produce data realistic enough to completely fool the ad-hoc discriminator with a score under $0.1$ for all experiments. The number of trainable parameters for each model is also shown in Table 1, which was fixed for all experiments.

Fully embedding time-series data using an adversarial autoencoder ensures the synthetic data produced more closely matches the full distribution of the target data, and this auto-encoding reconstruction has a regularizing effect on the adversarial training. However, this also limits the potential applications of this methodology. Data such as video or language tokens are too highly dimensional to be reliably reduced to vector embeddings. As such, the realistic use cases are limited to real-valued time-series signals with relatively low dimensionality. The architecture chosen in this work was motivated in part by the fact that it is not auto-regressive in nature, and thus does not suffer from exposure bias. Limiting exposure bias would allow for more robust use of auto-regressive methods such as attention-based transformers, and constitutes a promising pathway for future work.

While the applications of our work may not reach the scope and magnitude of other synthetic data generation models such as large language models (LLMs), they are similar with respect to the ethical considerations highlighted in [20]. Our methodology can straightforwardly be used to produce synthetic datasets where privacy is a concern within the original data. The primary responsibility for using this methodology and those like it is to ensure it is made explicitly clear that the data produced is synthetic. Additionally, the creator of the model should thoroughly ensure it sufficiently matches the distributions of the target data before applying it to problems with real-world impact.

The authors declare no known competing interests in the content of this manuscript. Funding for this work was partially provided by the Collaborative Sciences Center for Road Safety (CSCRS), as well as the University of Tennessee, Knoxville.

All the data used for experimentation was publicly available, and contains no sensitive or personal information of any kind. No original datasets were produced through this research. All datasets are either provided through citation, or provided directly at the linked repository.