MTSA-SNN: A Multi-modal Time Series Analysis Model Based on Spiking Neural Network
^††thanks: * These authors contributed equally to this work. ^††thanks: † Corresponding Author.

Modelling and forecasting time series data is a valuable task in various domains. It has evolved significantly, transitioning from traditional methods to deep learning techniques, resulting in improved prediction accuracy and relevance over time.

Initially, time series forecasting relied on traditional approaches such as the ARIMA model [5] and Fourier analysis [6]. ARIMA, which includes auto-regressive (AR) and moving average (MA) components with differencing (I) to address non-stationarity, had challenges related to parameter selection and model identification. Fourier analysis was used for frequency domain analysis to identify periodic and seasonal patterns in the data.

Later, deep learning methods such as RNN and LSTM emerged to handle temporal dependencies [7]. LSTM, an improved version of RNN, performed better with long sequences due to its enhanced memory and forgetting mechanisms, becoming the preferred model for many time series problems. Nonetheless, they encountered challenges related to gradient vanishing and exploding when handling extended sequences, which restricted their practicality.

In contrast to single-modal time series forecasting, multi-modal time series forecasting leverages multiple data sources, such as text, images, and sensor data, to capture a broader perspective, enabling a wider range of pattern and trend recognition. This approach offers benefits like information synthesis, complementarity of different data types, model robustness, and improved generalization. Multi-modal deep learning models use CNN and BiLSTM to extract features from multi-modal time series data. Ensemble models, including probabilistic time series prediction based on Hidden Markov Models [8] and stacked ensembles, have been used to enhance accuracy and reduce overfitting.

Specific algorithms, including interpretable ML models and multi-modal meta-learning techniques [9], have been applied in diverse use cases, ranging from early Parkinson’s disease detection to time series regression tasks. These applications highlight their potential in various domains, reflecting the diversity and complexity of time series modelling and forecasting. They underscore the evolving methods and technologies that offer robust tools for a broad spectrum of application scenarios.

Multi-modal time series models struggle with complex, irregularly and non-uniformly sampled data due to their continuous computations, difficulty in handling event-driven data patterns, and high computational complexity. However, Spiking Neural Networks (SNNs) hold promise in mitigating these challenges. SNNs, a unique class of neural networks that communicate using discrete spike signals in a continuous-time framework [10], are capable of emulating biological neural systems’ sparsity and encoding temporal information [11]. SNNs find practical application in various time series prediction scenarios, including financial time series forecasting, time series classification [3], and real-time online time series prediction [4].

SNNs pose challenges due to their complex neurons and non-differentiable pulse-based operations. Choosing a multi-modal time series model depends on the problem and data characteristics. The multi-spike network SNN variant is useful for financial time series prediction. Therefore, you should select the most appropriate model based on the problem and data characteristics.

SNNs (Spiking Neural Networks) present challenges due to the complexity of their neurons and the non-differentiable nature of pulse-based operations, making training complex. The choice of a multi-modal time series model should depend on the problem and data characteristics. For instance, a variant like multi-spike networks has proven valuable in time series prediction, especially for non-stationary data [12]. Thus, selecting the right model should align with the problem and data intricacies.

In summary, the proposed MTSA-SNN model efficiently encodes multimodal information into spikes. It utilizes a spike-based cooperative learning module to effectively map and integrate complex spike information. This method provides an accurate and practical event-driven approach that addresses the analysis of complex and non-stationary temporal information, demonstrating strong performance across multiple time series datasets.

The SNN Image Encoder is a component that processes time-series image information into pulse representations and extracts features. This encoder alternates between the Feature Extraction (FM) module and the Leaky Integrate-and-Fire (LIF) SNN module. Visual information initially passes through the SNN layer to be transformed into a unified and compatible pulse signal format, making it suitable for subsequent network operations. The FM module further performs feature extraction on the visual information converted into pulse signals, including operations such as convolution and pooling. After feature extraction, the pulse signal $\hat{S}_{1}$ is then passed to the pulse co-learning module.

The SNN Series Encoder is another modality encoder used for pulse-coding and feature extraction of temporal data sequences. These sequence data initially pass through the SNN layer and are then transformed into pulse signals. The network employs alternating operations between mapping layers and neurons. Neurons receive pulse information from the previous layer and membrane potential from the preceding time step in the sequence. By introducing this self-feedback mechanism, the pulse network can utilize membrane potential information from the previous time step to influence the calculations at the current time step. Consequently, the encoder is better equipped to capture the temporal correlations and dynamic changes in time-series data. The pulse information encoded through sequence encoding is $\hat{S}_{2}$.

Due to the strong temporal information processing capabilities of SNN, we employ the Leaky Integrate-and-Fire (LIF) model to describe the neural dynamics of multi-modal information. The following formula can represent the dynamic equation for the LIF model under continuous-time sequences:

$V(t)$ is a membrane potential function concerning time $t$. $V_{\text{rest}}$ represents the resting membrane potential of the neuron. $\tau_{m}$ is a constant that characterizes the charging and discharging rate of the neuron’s membrane potential. $I(t)$ is the synaptic pulse input function. $R$ denotes the membrane’s responsiveness to input currents.

When the membrane potential $V(t)$ exceeds the threshold potential $V_{th}$, the neuron is activated and triggers a spike, denoted as $H(t)$. $\Theta(x)$ is the Heaviside step function, which is 1 when $x\geq 0$ and 0 otherwise. $V_{\text{th}}$ represents the threshold potential. $V_{\text{reset}}$ is the reset potential, to which the membrane potential is reset when the neuron is activated.

A neuron receives multiple pulse signals. Their effects are not independent but accumulate within the neuron, leading to a sustained change in membrane potential. By controlling the pulse frequency and timing, neurons can integrate and encode input information over time. Assuming that N neurons generate multiple pulses at different time points, these pulse timings can be represented by a series of time sequences $\{t_{1}^{(i)},t_{2}^{(i)},....,t_{j}^{(i)}\}$. The cumulative effect of multiple pulses can be expressed as $\displaystyle P(t)=\sum_{i=1}^{N}\sum_{j=1}^{j}f(t-t_{j}^{(i)})$.

$f(t)$ represents the Dirac Delta function, signifying the generation of a pulse at the firing time. $P(t)$ is the output of the cumulative effect of multiple pulses, which corresponds to the pulse output of the encoder $\hat{S}_{1}$ & $\hat{S}_{2}$. Algorithm 2 is the workflow of the SNN encoder.

The pulse signals extracted from different encoders are first subjected to normalization and mapping operations before input into a unified pulse co-learning module. The pulse signals $\hat{S}_{1}$ and $\hat{S}_{2}$ obtained from two heterogeneous spaces are then transformed from the time domain to the frequency domain through Fourier transformation. Fourier transformation $FT(s)$ decomposes the signal into different frequency components, which aids in analyzing the frequency domain characteristics of different modal signals.

To better integrate and align the information from two different signal spaces, we introduce a joint learning function denoted as $\Psi$. This function aims to adjust the feature representations of the signals, mapping the signals from space $D_{i}$ and space $D_{t}$ to a common frequency domain space. During the training process, this function is continuously adjusted to make the pulse information in different modalities more consistent, achieving effective fusion and alignment of heterogeneous signals. $J_{align}$ denotes the fusion of pulse information in the joint learning space. $D_{j}$ is the dimension of joint learning space, where data from different modalities coexist in a shared representation.

We introduce a more effective pulse-based joint weight allocation mechanism (JWAM). This mechanism involves mapping the similarity results in $sim$ of multi-modal pulse signals into different spatial dimensions of the probability distribution matrix ($P_{mtsa}$). The similarity probability distribution is adaptively adjusted based on the features of each modality and their relative importance to achieve information fusion. $P_{mtsa}$ integrates information from various modalities, providing a quantitative method for scoring cross-modal information representation. $sim$ is a metric function used to measure the similarity between two pulse information representations in heterogeneous spaces. This function employs the Euclidean distance calculation method to assess the similarity between different modality representations. $\sigma^{2}$ is used to adjust the sensitivity of the similarity measurement function. It is worth noting that it can dynamically adapt based on the distribution information of different modality features, enhancing the robustness and adaptability of similarity measurements.

Furthermore, matrix transformations of the information in the joint space are utilized to adjust pulse signals. This operation aims to optimize the feature space while taking into consideration information from different modalities in order to better accommodate the characteristics of pulse sequences from other modalities. Additionally, we interact this process with cross-modal probability distributions to obtain the pulse fusion representation denoted as $J_{fusion}$. This can be expressed as:

The Output layer is responsible for two major tasks: predicting and classifying information from multi-modal time series pulse fusion data. It employs network layer techniques such as residual connections and ReLU to transform the fused information into a common format, making it available for various downstream tasks.

To effectively address the non-stationary, non-linear characteristics and constraints in multi-scale feature analysis of time-series data, we employ the wavelet transform analysis method. Wavelet transform possesses exceptional time-frequency locality and multi-scale analysis capabilities, making it more suitable for capturing local features of signals at different time and frequency scales. The MTSA-SNN network based on wavelet transform can capture richer feature representations, endowing it with a significant advantage in handling non-stationary signals, extracting critical signal features, and analyzing signals across multiple scales.

MTSA-SNN employs wavelet transform to decompose input signals into four subbands: LL, LH, HH and HL, which represent distinct signal characteristics in terms of different frequencies and spatial scales. This multi-scale and multi-frequency analysis approach equips the MTSA-SNN model with a comprehensive understanding of multimodal data, enhancing its learning capabilities. As illustrated in Fig. 3 and Fig. 4, the temporal visualizations of these four subbands in the ETT and stock prediction datasets demonstrate the effectiveness of this multi-scale analysis.

Fig. 2 depicts the pulse network outputs based on the MIT-BIH dataset with different processing methods. It is evident that the pulse output subjected to wavelet transform more accurately captures the features of multimodal signals, resulting in a more stable and effective neural activation.

We conduct experimental evaluations for classification and regression tasks on two traditional time series datasets, MIT-BIH Arrhythmia (MIT-BIH)[13] and Electricity Transformer Temperature (ETT)[14]. Additionally, we perform a market forecasting analysis on relevant stock indices of the Chinese stock market from June 6, 2013, to June 6, 2023, covering a ten-year period, focusing on the opening and closing prices.

MTSA-SNN demonstrates remarkable performance advantages in the field of biological time-series data analysis. The experimental results in Table I demonstrates that our model has achieved advanced performance in the detection of cardiac arrhythmias in multimodal electrocardiogram data. With a dataset classification accuracy of 98.75%, MTSA-SNN markedly outperforms previous leading algorithms. This is attributable to the effective simulation of the neural signal conduction process in biological systems through MTSA-SNN’s pulse-based fusion approach, resulting in significant performance advantages.

In addition, our method exhibits outstanding performance in various prediction tasks, including transformer temperature monitoring and stock market forecasting. Analyzing the results presented in Table II, our model demonstrates the lowest MAE and MSE across four different time steps in the ETT dataset. Furthermore, in Table III, MTSA-SNN achieves remarkably low errors of 0.96 and 1.15 in the stock market price prediction task compared to traditional time-series prediction models such as LSTM and XGBoost. MTSA-SNN, by converting complex and diverse multimodal time series data into a pulse-based representation, significantly enhances the model’s predictive and analytical capabilities regarding time-series information.

We conduct a comprehensive ablation study to evaluate different components of the MTSA-SNN model. As shown in Fig. 5, we present pulse signal output heatmaps for different components at the same time step using the MIT-BIH dataset. The brightness of the colours in the figure represents the activation levels of neurons. In comparison to the activation patterns from single-modal encoders, the joint learning module of MTSA-SNN activates more neurons, thus enriching the representation of temporal information. Furthermore, the application of wavelet transform enhances the representation of temporal information within the MTSA-SNN. This suggests that joint learning of pulses effectively balances multi-modal pulse signals and fuses them together. Simultaneously, wavelet transformation contributes to enhancing the representation of temporal information in the pulse network.

In addition, we analyze the spectral information of the waveform plots during the training process of the single-modal encoder and the joint learning module. In Fig. 6, the horizontal axis represents the time steps, while the vertical axis represents the amplitude. This indicates that the MTSA-SNN model effectively integrates and analyzes multi-modal signals while enhancing the overall robustness of the model.