SimPSI: A Simple Strategy to Preserve Spectral Information
in Time Series Data Augmentation

Most of the real-world time series data consists of real values. Using the conjugate symmetry property of the Fourier transform for real-valued time series, we take the first half of the spectrum, in which the dimensions of the spectrum reduce to $\{x_{f},x^{\prime}_{f},\tilde{x}_{f}\}\in\mathbb{C}^{C\times(\lfloor L/2\rfloor+1)}$ while the dimensions of time series change to $\{x_{t},x^{\prime}_{t},\tilde{x}_{t}\}\in\mathbb{R}^{C\times L}$. The dimension of the preservation map also reduces to $P\in\mathbb{R}^{\lfloor L/2\rfloor+1}$.

Given an input spectrum $x_{f}\in\mathbb{C}^{C\times L}$, we concatenate real and imaginary parts of $x_{f}$ into a channel dimension, which the dimension changes to $x_{f}\in\mathbb{R}^{2C\times L}$. Then, $x_{f}$ is fed into a two-layer transformer encoder to capture the underlying context of spectral representation. The output of the last layer is averaged over the channel dimension to aggregate the channel information and passes through the sigmoid function to make the values between 0 and 1 as follows:

To train the preservation map generator $G$, we introduce a preservation contrastive loss $\mathcal{L}_{pc}$. Assume that an input spectrum $x_{f}\in\mathbb{C}^{C\times L}$, augmented spectrum $x^{\prime}_{f}\in\mathbb{C}^{C\times L}$, and corresponding spectrum-preservative map $P_{sp}$ are given. We define an information-preserved spectrum $\tilde{x}^{+}_{f}=(\mathbf{1}_{C}\cdot P_{sp}^{T})\odot x_{f}+(\mathbf{1}_{C}\cdot(\mathbf{1}_{L}-P_{sp})^{T})\odot x^{\prime}_{f}$ and a spectrum that preserves the inverted preservation map $\tilde{x}^{-}_{f}=(\mathbf{1}_{C}\cdot(\mathbf{1}_{L}-P_{sp})^{T})\odot x_{f}+(\mathbf{1}_{C}\cdot P_{sp}^{T})\odot x^{\prime}_{f}$. Then, the classifier $\hat{p}$ should predict $\tilde{x}^{+}_{t}$ better than $\tilde{x}^{-}_{t}$. Furthermore, if we define a randomly-preserved spectrum $\tilde{x}^{rnd}_{f}=(\mathbf{1}_{C}\cdot n_{f}^{T})\odot x_{f}+(\mathbf{1}_{C}\cdot(\mathbf{1}_{L}-n_{f})^{T})\odot x^{\prime}_{f}$ where $n_{f}\in\mathbb{R}^{L}$ is a random noise sampled from $U(0,1)$, then the prediction score of $\tilde{x}^{rnd}_{t}$ by the classifier $\hat{p}$ should be in between those of $\tilde{x}^{+}_{t}$ and $\tilde{x}^{-}_{t}$. We notate $\tilde{x}^{\{+,-,rnd\}}_{t}$ and $\tilde{x}^{\{+,-,rnd\}}_{f}$ as Fourier transform pairs of time series and corresponding spectrum. These constraints can be formulated as follows:

We then define the corresponding classification loss using the cross-entropy loss for these three predictions, $\mathcal{L}_{cl}^{+}=\mathcal{L}_{ce}(\hat{p}(y|\tilde{x}^{+}_{t}),y)$, $\mathcal{L}_{cl}^{rnd}=\mathcal{L}_{ce}(\hat{p}(y|\tilde{x}^{rnd}_{t}),y)$, and $\mathcal{L}_{cl}^{-}=\mathcal{L}_{ce}(\hat{p}(y|\tilde{x}^{-}_{t}),y)$ where $y$ is the label of the input time series $x_{t}$, and translate the constraints into an objective function as follows:

where $\beta_{1}$ and $\beta_{2}$ are hyperparameters satisfying $\beta_{1}<\beta_{2}$.

We use two objective functions for model training, classification loss $\mathcal{L}_{cl}$ for classifier training and the preservation contrastive loss $\mathcal{L}_{pc}$ for preservation map generator training. We separate the training procedure, updating the classifier $\hat{p}$ by $\mathcal{L}_{cl}$ with the preservation map generator $G$ froze, and then updating the preservation map generator $G$ by $\mathcal{L}_{pc}$ with the classifier $\hat{p}$ froze. It can be formulated as follows:

where $\theta_{G}$ and $\theta_{p}$ are the parameters of the preservation map generator $G$ and the classifier $\hat{p}$, respectively. Note that $\theta_{G}$ does not descend towards the gradient of $\mathcal{L}_{cl}$. This prevents $G$ from learning undesirable local minima, such as returning the uniform scalar value across different frequencies or the same map across different samples, in which the preservation map is not adaptive to the input time series but acts as a uniform band-pass filter. Also, the preservation map generator $G$ is removed during inference, so $G$ updated by the classification loss might interrupt classifier training.

We verified if the proposed method improves the performance of existing data augmentation techniques by capturing important spectral regions and preserving them. To do that, we devised a simulation where information is carried on a set of known frequencies. Inspired by the wireless communication domain (Ryu and Choi 2023), we constructed a synthetic dataset by modulating a sequence of random bits into the corresponding frequencies of a signal, called frequency shift keying (FSK), and the task is demodulating it. We used 8 and 32 different frequencies for modulation (FSK8 and FSK32), and each dataset consists of 2,304 training signals, 288 validation signals, and 288 testing signals. We chose ResNet1D (Ramjee et al. 2020) as a baseline network, which was given a 128-length modulated signal and returned a 32-length M-ary (M=8, 32) sequence. We used the Adam optimizer (Kingma and Ba 2015) with the learning rate $10^{-3}$, and the networks were trained for 50 epochs. The training was performed on a single NVIDIA RTX A6000 GPU. Appendix A provides more details about our experimental setup.

The performance improvement of random augmentations by the proposed method on the FSK32 dataset is described in Table 1. The accuracies of Jittering, Scale-Shift-Jittering, and Frequency masking were increased by 1.5%, 1.4%, and 1.5%, respectively, using the spectrum-preservative map.

We then verified whether the learned preservation map genuinely preserves the informative frequency components during augmentation. We displayed learned preservation maps of representative samples from the FSK8 test set in Fig. 4. We could observe eight equally-spaced frequencies that were preserved the most during Frequency masking. It perfectly matches the data generation process since we used those eight frequencies for signal modulation. The other frequencies did not contain information and showed a preservation value of around 0.5, meaning those components barely attributed to achieving Eq. (6).

In the HAR dataset (Reyes-Ortiz et al. 2012), data is collected by the accelerometer and gyroscope of a smartphone mounted on a waist and sampled at 50 Hz, which aims to classify human activities. Following the data preprocessing in (Eldele et al. 2021), an input time series has a length of 128 and nine channels. The dataset consists of 7,352 training samples and 2,947 test samples labeled with six classes. We chose a 3-layer CNN model for classification, which was used in (Wang, Yan, and Oates 2017; Eldele et al. 2021; Zhang et al. 2022), and additionally included a 2-layer LSTM model and a 2-layer Transformer model for further verifications. We used the Adam optimizer with the learning rate $10^{-3}$, and the networks were trained for 100 epochs. We adhered to configurations in (Eldele et al. 2021), and the training was performed on a single NVIDIA RTX A6000 GPU. Appendix B and C provide more details about our experimental setup.

We compared the performance of the model with and without SimPSI to evaluate the impact of SimPSI on recognition accuracy. To inspect its impact thoroughly, we performed experiments on three perspectives: different random augmentations, model architectures, and distortion magnitudes.

The performance increase of data augmentations by SimPSI on the HAR dataset is described in Table 1. We also tested an intuitive method that mixes the original data and its augmented view with a random preservation map sampled from $U(0,1)$. The accuracy of Jittering is enhanced by 0.2% using the magnitude spectrum, while the random preservation map decreases it by 0.4%. The accuracy of Scale-Shift-Jittering is improved by 0.9% using the magnitude spectrum, and Frequency masking is improved by 1.0% using all the proposed preservation maps. Appendix D provides more experimental results, and Appendix E provides the training time cost of the preservation maps.

Using Jittering, we compared three types of networks for time series classification: CNN, LSTM, and Transformer (Table 2). SimPSI increased the area under the precision-recall curve (AUPRC) of a 3-layer CNN by 0.1, while it increased the AUPRC of a 2-layer LSTM by 1.5 and a 2-layer Transformer by 2.2.

Using Permutation, we also tested SimPSI while varying the distortion magnitude of data augmentation. We changed the maximum number of segments of Permutation and compared the accuracy with and without SimPSI (Fig. 5). SimPSI consistently improved the performance of Permutation regardless of its distortion strength, alleviating the performance drop while the number of segments increased. Specifically, comparing the performance at 10 and 12 segments, Permutation alone dropped the accuracy by 0.6, while Permutation with SimPSI dropped it by 0.3.

We used the SleepEDF dataset (Goldberger et al. 2000) for classifying sleep stages from Electroencephalogram (EEG) signals sampled at 100 Hz. We followed the data preprocessing in (Eldele et al. 2021), where the input has a length of 3,000 and a single channel. The dataset comprises 35,503 training samples and 6,805 test samples labeled with five classes. We chose a 3-layer CNN model for classification, also used in the human activity recognition experiments. We used the Adam optimizer with the learning rate $10^{-3}$, and the networks were trained for 40 epochs. We adhered to configurations in (Eldele et al. 2021) for other details. The training was performed on a single NVIDIA RTX A6000 GPU.

The performance improvement of data augmentations by SimPSI on the SleepEDF dataset is summarized in Table 1. SimPSI increased the detection accuracy of Jittering by 0.7% using the saliency map, Scale-Shift-Jittering by 0.7% using the magnitude spectrum, and Frequency masking by 1.0% using the spectrum-preservative map. We note that the spectrum-preservative map outperformed the random preservation map regardless of the baseline augmentation techniques, supporting the effectiveness of the information-preserving approach.