Multi-Tier Platform for Cognizing Massive Electroencephalogram

To reveal information in both time- and space-domains, tier 1 of the proposed platform generates a conventional log-power spectrogram for each input EEG sample Phan et al. (2021). The procedure of the spectrogram transformation first applies the Discrete Fourier Transform with a moving window to the input signal, which is given by:

$$X(k)=\sum_{n=0}^{N-1}x(n)\omega(n)e^{-\frac{2\pi i}{N}kn},$$

(1)

where $N$ is the length of Hamming window function $\omega(n)$. $k$ is the frequency. A log-power representation is then defined by:

$$S_{\log}(k)=\log(|X(k)|),\quad\forall t.$$

(2)

Elements of the spectrogram represent the intensities at different frequency resolutions of corresponding time duration. Each spectrogram viewed as a time-frequency matrix is then normalised into a grey-scale intensity image scaled between [0, 1]:

$$S(k)=\frac{S_{\log}(k)-\min(S_{\log}(k))}{\max(S_{\log}(k)-\min(S_{\log}(k))}.$$

(3)

$S(k)$ is cut sequentially for every second to yield episodes named as truncated spectrograms $S(k,t)$, where $t$ denotes the truncation for $t$-th second. The truncated spectrograms with analog intensity are then fed into the SNN tier 2 for further pre-coding the ST information.

To tackle the non-differentiable issue of SNN activation units, we follow the well-known iterative leaky integrate-and-fire (LIF) model Wu et al. (2019). LIF can be succinctly described by the following relation by focusing on one timestep $t+1$ and the $n+1$-th layer of the network:

$$u_{t+1,n+1}=\tau u_{t,n+1}(1-o_{t,n+1})+\sum_{j}w_{j,n}o^{j}_{t+1,n},$$

(4)

where the subscript $t$ denotes $t$-th timestep and $n$ denotes $n$-th layer within the architecture. Hence $u_{t+1,n+1}$ is the membrane potential of neurons at the $n+1$-th layer and timestep $t+1$. $\tau$ is the decay rate of membrane potential. $o^{j}_{t+1}$ denotes the $j$-th firing spike with weight $w_{j,n}$ from the previous layer at time step $t+1$ Given the above information, $\sum_{j}w_{j}o^{j}_{t+1,n}$ is the pre-synaptic input and its information carrier is accumulated from the $n$-th layer. Hence, the neuron will output a fire spike and $u_{t+1}$ is the reset to $u_{0}$ ($u_{0}=0$) ,when $u_{t+1}$ reaches the firing threshold $u_{th}$:

$$o_{t+1,n+1}\begin{cases}1&\text{ if }u_{t+1,n+1}>u_{th}\\ 0&\text{ otherwise }\end{cases}$$

(5)

The spike fired by $o_{t+1,n+1}$ will propagate forward and activate next layer’s neurons.

Conventionally, SNN iterates $t$ times over each sample $s$ to produce a time-indexed sequence of samples and errors $(s_{1},\dots,s_{t})$, $(\epsilon_{1},\dots,\epsilon_{t})$. The output sequence $(s_{1},\dots,s_{t})$ is unsaved and typically only the average $\bar{s}$ is computed. The errors $(\epsilon_{1},\dots,\epsilon_{t})$ are averaged to yield the timestep-average ${\epsilon}$ for backpropagating ${\epsilon}$ of sample $s$. However, one cannot backtrack individual contribution of each timestep from the average of all timesteps. Moreover, EEG data are inherently time-indexed, which renders such iterates ambiguous to apply. We propose a pseudo-SNN to handle the episode-wise truncated spectrograms $S(k,t)=(\tilde{s}_{1},\dots,\tilde{s}_{t})$ that contain spatial information for every second, i.e. $\tilde{s}_{t}$ contains information from the $t$-th second. Every $\tilde{s}_{i},1\leq i\leq t$ is then fed into the SNN layer for $1$ rather than $t$ times forward pass. The resultant snapshot $z_{i}$ and error $\tilde{\epsilon}_{i}$ are appended to the sequence $(z_{1},\dots,z_{i})$ and $(\tilde{\epsilon}_{1},\dots,\tilde{\epsilon}_{i})$, respectively. Here, $\tilde{\epsilon}_{i}$ denotes error associated with $\tilde{s}_{i}$ rather than with $i$-th iteration of whole spectrogram $s$. The sequence of errors are averaged to yield sample-average error $\tilde{\epsilon}$ for backpropagation.

We argue that leveraging the truncated spectrograms has at least three advantages over the conventional method:

1. 1.

   since EEG data are inherently time-indexed, timestep-average can be regarded as a special case of sample-average of truncated spectrograms by not performing truncation and increasing the number of forward passes.

2. 2.

   spike snapshots allow for more efficient and finer-grind feature extraction, significantly boosting performance of downstream tasks.

3. 3.

   pre-coding phase based on the inherent EEG time naturally models the cumulate/fire process of SNN neurons. We conjecture such similarity might be essential for improved feature distillation and performance.

The second and last advantages are demonstrated by our experimental results in Sections 4.1 and 4.2, respectively .

At tier 2 of the proposed architecture, we independently pre-code the analog features into a spike snapshot for each truncated spectrogram. The sequence of spike snapshots can be viewed as a matrix $Z=[z_{1},\dots,z_{t}]$ to be fed into the latter ANN layer for refining SNN spike features. Matrix $Z$ preserves the spanning topology obtained from the SNN layer tier 2 and might benefit significantly the latter construction of feature subspaces, which stands as a sharp contrast to the conventional method that outputs an average vector $\bar{s}$. From an EEG perspective, maintaining $Z$ helps dynamically capture and accumulate the ST characteristics. To better excavate information associated with snapshots, a transpose operation is required for latter processing in tier 4.

Feature outputs from conventional SNN are fed into an ANN layer (or a subsequent softmax layer) for downstream tasks such as classification. However, to do so the feature output should be compressed, e.g., reshaping to a vector for input to a fully connected layer. Such compression would destroy the independence between snapshots. Hence, a network that can individually process and summarize snapshots is called for. Specifically, the recently popular attention-based mechanism excavates relevant relationship from input matrices Dosovitskiy et al. (2021). As such, an attention layer naturally suits our platform for processing and tracking individual contribution from the sequence of truncated spectrograms, while each truncated spectrogram is accepted by a sub-network to maintain the feature space.

The input of tier 4 is the linear projection of transposed input $Z^{T}$ onto higher dimensional feature spaces. The attention layer calculates the relevance of rows of $Z^{T}$ and maps the relevance to the ground truth by three matrices: query $Q$, key $K$, and value $V$. Specifically, the relevance is computed as:

$$A=\sigma(\frac{QK^{T}}{\sqrt{d}})\cdot V,$$

(6)

where $\sigma(\cdot)$ denotes row-wise softmax function. $\sqrt{d}$ denotes a normalization-like scale that is applied to each $Q$-$K$ computation. The result matrix $A$ records the relevance score calculated as the weighted average of the rows of $V$, with the weights corresponding to the softmax probabilities. Motivated by the recent success in visual tasks, tier 4 comprises an addition task-based indicator, such as class-token referring to Dosovitskiy et al. (2021) for details, for further decision-making such as classification.

In this section we show the results for the largest SHHS and Sleep-EDF datasets and present complete results in Appendix B due to the page limit.

SHHS Dataset. We first compare the proposed platform against existing work that also solely leverages EEG data Eldele et al. (2021); Karimzadeh et al. (2018). From Table 1 it is visible that the proposed platform significantly outperformed those existing methods in terms of classification accuracy for all stage. Compared to Eldele et al. (2021) that leveraged CNN to extract spatio-temporal features, the precision, recall, F1 (Pre, Re, F1 in short) of the proposed platform for the stage wake has been greatly improved to $0.95,0.94,0.94$, respectively. Similar trend holds also for stage N2 or N3 as well. The superior performance is consistent with our conjecture that the platform and truncated spectrograms enabled better capturing the relative phase of spatio-temporal rhythms. Such rhythms were proved by experiments to be not only complementary to time- and space-domain features but also essential in that they constitute a complete description for the underlying neuronal activity. The improvement is significant even compared with other methods exploiting multi-source signals such as Pathak et al. (2021); Phan et al. (2021); Fernandez-Blanco et al. (2020); Biswal et al. (2018).

Compared with Pathak et al. (2021), the proposed platform attained lower Pre, Re, F1 scores for the rapid eye movement (REM) stage. This was due to the electrooculogram (EOG) signal used in their study: it is generally received that EOG signal directly reflects the characteristics of REM. Classification for the N1 stage is difficult even for human experts. On this problem, Biswal et al. (2018) achieved higher scores than all other methods. We conjecture this was due to the large training set they used: their training set contained 16,000 subjects.

Sleep-EDF Dataset. Similar conclusion holds also for the Sleep-EDF dataset: the proposed platform attained highest overall accuracy. For existing works exploiting sole EEG data Fiorillo et al. (2021); Qu et al. (2020), significantly higher scores for all stages except N1 were achieved. The slightly better performance of Korkalainen et al. (2020); Phan et al. (2019) for stage REM was due to their additional EOG signal as introduced above. As a summary, it might be safe to put that the proposed method established a new EEG-based SOTA on the SHHS dataset. For the Sleep-EDF dataset, the proposed platform achieved better results than existing multi-source methods and hence established an overall SOTA.

Representation Ablation Results. Recall that the ablation study choices and their abbreviations were defined in Representation Ablation, Section 3. The results are plotted in Figure 2. By comparing the proposed method (Proposal) with space-domain truncated spectrograms (SDTS), we see Proposal consistently outperformed SDTS by a large margin, demonstrating the time-domain truncated spectrograms successfully matched the firing process of SNN neurons. Result of replacing the input data representation from truncated spectrograms to spike train is shown as SpikeTrain, which outperformed other methods except Proposal, demonstrating our representation can better extract underlying characteristics. This result is consolidated by NoTS which fed the entire EEG spectrogram instead of truncated spectrograms, incurring around 30% accuracy drop. The averaging of SNN output turned out to not destructive for the temporal information as can be seen from the curves of ATDTS, which performed the worst in all cases.

Architecture Ablation Results. The ablation study choices and their abbreviations were defined in Architecture Ablation, Section 3. The results are plotted in Figure 3. Proposal again consistently outperformed other methods. However, it is interesting to see the classic and simple method NoTier2 achieved accuracy only 10% worse than Proposal and slightly outperformed LSTM. This suggests that combining the binary spike sequence with our proposed platform might be a promising future direction. On the other hand, choices without the SNN layer SpikeTrainNoTier2, SG, NEO performed significantly poorer, even at the presence of spike detection mechanism.

Illustrating the correspondence between input and output is enlightening for further understanding the mechanism by which the proposed method cognizes brain activities. We feed EEG spectrograms into the proposed platform and plot the resulting intensity output in Figure 4, motivated by Chefer et al. (2021). It is visible from the figure that the proposed method successfully captured the highlights in EEG and represented them as shaded peaks, in which the peaks were the relevance to the result perceived by the model, and colors from deep to shallow represented the extent of match to the ground truth. The correspondence shown in Figure 4 partly illustrated the superior performance of the proposed method.