A Novel RL-assisted Deep Learning Framework
for Task-informative Signals Selection and Classification for Spontaneous BCIs

Notably, this module is flexible with many kinds of network architectures, varying from the existing ones in the literature to newly customized networks. In our experiments, we exploit the existing CNN architectures, namely, ShallowNet [7], DeepNet [7], EEGNet [6], and MSNN [18]. Basically, these architectures were proposed by different research groups and presented their superiority or validity in their respective experiments over various datasets. In the following, we denote an embedding network for feature representation as $\phi(\cdot;\theta_{\phi})$ with a tunable parameters $\theta_{\phi}$.

We introduce a learnable agent that adaptively and automatically selects task-relevant feature vectors of EEG signals over time in a trial without supervision, as there is no explicit way of observing such information in a trial. For the feature vectors $\bm{\phi}=\left\{\phi_{1},\dots,\phi_{T^{\prime}}\right\}\in\mathbb{R}^{D\times T^{\prime}}$ of the input signals, where $D$ is the dimension of feature vectors, we devise a method for automatic selection of signal segments over time $t\in\mathcal{S}$, $\mathcal{S}\subset\left\{1,\dots,T^{\prime}\right\}$, such that the selected feature vectors $\left\{\phi_{t}\right\}_{t\in\mathcal{S}}$ carry the most information related to the user’s intention, induced by means of MI. However, as MI involves an internal cognitive process in a brain, and thus there are no clear labels, i.e., informative or non-informative, for signals at which timepoints they actually include the intention-related information.

Here, we formulate the problem of informative feature vectors selection of signals in a Markov decision process [22] and devise an RL-assisted module to enhance the MI-EEG classification performance. Specifically, an agent interacts with the environment defined with a given MI-EEG trial via a sequence of states (defined with the set of feature vectors represented by an embedding network $\phi$), actions (selection or rejection), and rewards (effects of making specific actions, i.e., decisions) over time, as illustrated in Fig. 3(b).

In order to demystify our method, we define states, actions, and rewards as follows:

After selecting informative feature vectors by our agent over time in a trial, the aggregated vector representation of those is then fed into a densely-connected layer $\rho(\cdot;\theta_{\rho})$ for decision-making. As for the aggregation, we again introduce the mean average of feature vectors in Eq. (4), also called as the GAP [24]. In the viewpoint of BCI, the GAP layer can be understood as a means of emphasizing an important spectral range and its neighboring region for each of the feature dimension. Using the aggregated feature vector $\mathrm{AGG}\left(\left\{\phi_{i}\right\}_{i\in\mathcal{S}_{T^{\prime}}}\right)$, the classifier outputs a class label $\hat{\mathbf{y}}$ of the input EEG trial.

To jointly optimize the embedding network, the policy and value networks of an agent module, and a classifier, the proposed framework involves two types of learning schemes, i.e., supervised learning and reinforcement learning. We combine these two learning strategies in our network optimization.

First, the embedding network $\phi$ and a classifier $\rho$ are pre-trained in a supervised manner without the agent module by minimizing a cross-entropy loss. After pre-training, the actor and critic networks in an agent module are trained to select task-informative features by interacting with the environment. Initially, the agent takes the feature vectors $\bm{\phi}$ represented by the pre-trained embedding network. Thus, the agent basically starts from the more learned position in a parameter space, rather than a random initial point, thereby training parameters $\phi_{\pi}$ and $\phi_{v}$ faster and more robustly.

The model parameters updating is alternated between (i) the agent module and (ii) the other two modules of feature representation and classification. As the agent is directed to find more informative features by being iteratively updated, the embedding network and the classifier can also focus on the task-oriented feature learning, and thus can be better generalized in a more reliable way.

To optimize the sequential actions, we update the trainable parameters of the actor network $\theta_{\pi}$ and the critic network $\theta_{v}$ by performing a gradient ascent in regard to maximization of the expected total return $\mathbb{E}[R_{t}]$ ($t=1,...,T^{\prime}$). Basically, the actor parameters $\theta_{\pi}$ are learned in the direction of $\nabla_{\theta_{\pi}}\log\pi(a_{t}|s_{t};\theta_{\pi})\cdot R_{t}$ [22]. However, although the updating direction is an unbiased estimate of $\nabla_{\theta_{\pi}}\mathbb{E}[R_{t}]$, we need to reduce the variance of this estimate by introducing another value, called advantage, [14]. The advantage $A_{t}$ is calculated as follows:

By applying the advantage function to the gradient estimation, we define a loss for an actor network as follows:

Meanwhile, the value estimation function $V(\cdot;\theta_{v})$ approximates the expected return for the given state $s_{t}$, i.e., $V(s_{t};\theta_{v})=\mathbb{E}[R_{t}|s_{t}]$. Owing to the fact that we cannot directly know a value of a specific state, the value estimation function is optimized by a bootstrapping method [22]. According to its definition, the current state value estimation $V(s_{t};\theta_{v})$ should be equal to the summation of the current reward and the next state value estimation $r_{t}+\gamma V(s_{t+1};\theta_{v})$, thus its training loss is defined as follows:

The complete pseudo-algorithm to train all the networks in our framework is presented in Algorithm 1.

We used a publicly available big KU-MI dataset [4]¹¹1Available at http://gigadb.org/dataset/100542., which consists of left-hand and right-hand MI tasks. MI samples were acquired across two sessions from 54 healthy subjects, recorded from 62 Ag/AgCl electrodes according to the standard 10-20 system, and sampled with 1000Hz. Each MI class of the dataset contains 50 trials with a 4-second length. For preprocessing, following [9, 4], we downsampled EEG trials to 100Hz and then applied a band-pass filtering between 8 and 30Hz, including both $\mu$ and $\beta$ bands, and segmented from 1 sec to 3.5 sec (250 timepoints). Finally, we selected 20 electrodes (FC-1/2/3/4/5/6, C-1/2/3/4/5/6/z, and CP-1/2/3/4/5/6/z) over the sensory-motor cortex areas.

In order to empirically prove the validity of our proposed method, we compare with the existing subject-dependent and subject-independent methods in performance. By following the recent work of [9], we set the subject-dependent and subject-independent scenarios as follows:

While training our proposed framework in Fig. 3(a), we set a mini-batch size of 5, an exponentially decreasing learning rate with an initial value of 0.003 and a decreasing ratio of 0.001 per epoch, an RMSProp optimizer [25], and a Xavier initializer [26]. For the embedding and classification modules in our framework, we used the existing network architectures of [7, 6, 8]. Briefly, Shallow ConvNet [7] is composed of two convolutional layers, a temporal convolutional layer and a spatial convolutional layer with a square activation function for embedding in a feature space. Deep ConvNet [7] has a temporal convolutional layer, a spatial convolutional layer, and following three temporal convolutional layers with an exponential linear unit (eLU) activation function for feature representation. EEGNet [6] consists of a spectral convolutional layer, a spatial depthwise convolutional layer [19], and a temporal separable convolutional layer [19] with an eLU activation function for spatio-temporal feature representation. Finally, for the MSNN [8], a spectral convolution and three residually connected temporal separable convolutional layers and spatial convolutional layers with a leaky ReLU function were used as the embedding part. However, in order for better integration with our proposed agent module for signal segments selection, we made a slight modification in the architecture of Shallow ConvNet, Deep ConvNet, and EEGNet by replacing the last feature output layer (i.e., average pooling in Shallow ConvNet and EEGNet, max pooling in Deep ConvNet) with a GAP layer. In this reason, in the following, we differentiate those networks by naming with ‘original’ and ‘modified’ networks. In regard to the classification module, we utilized the above-mentioned networks’ densely-connected layers, respectively. As for the SSFR [9], because it was designed for energy map-based feature representation, rather than the spatio-temporal features, we did not consider it to apply in our framework.

In a pre-training phase for the embedding and classification networks, we set the number of epochs $n_{\mathrm{pre}}$ by 10. In regard to the total return $R_{t}$ estimation, a discount factor $\gamma$ of 0.95 was used. For the actor and critic networks, we designed densely-connected layers with a softmax and a sigmoid activation functions for their output layers, respectively. During training, we also applied an elastic net regularizer with the coefficients of $\ell_{1}$ and $\ell_{2}$ as 0.01 and 0.001, respectively.

We implemented all the models considered in our experiments, except for the linear models and SSFR as their performances were taken from [9], by Tensorflow 2 [27] and trained on a single Titan RTX GPU on Ubuntu 18.04.

In order to quantitatively validate the effectiveness of our proposed framework, we conducted a two-tailed Wilcoxon’s signed-rank test among the original deep models, their modified ones, and the counterpart agent-involved models. The results are plotted in Fig. 4, which state the statistical significance of our proposed agent module with its superiority in classification accuracy. In detail, for Shallow ConvNet [7], EEGNet [6], and MSNN [8], the proposed framework showed statistical significance with $p$-values of $<0.05$, $<0.05$, and $<0.01$, respectively. From this statistical comparison, it is reasonable to say that our proposed framework, specifically the agent module, played an important role to enhance the classification accuracy across all subjects. Additionally, we also compared performance of MSNN [8] and MSNN combined with our agent module (MSNN+AM) in the subject-independent scenario, and obtained the result that our method was statistically better with $p<0.05$ than the original model in classification accuracy.

In Fig. 5, we visualized the spectrogram (via short-time Fourier transform: STFT) of the C3/C4 channel signals in randomly selected trials from two subjects and the respective action sequences made by our agent module plugged in MSNN. In a coupled-consideration of the power spectrum and the the agent’s action of selection, we could observe their positive relations in the sense that the selected signal segments showed high spectral power in the neighbors of the $\mu$ and $\beta$ bands. Note that basically, the timepoints of an agent’s view in our framework ($1<t<T^{\prime}$) is different from the original input timepoints ($1<t<T$) due to a series of convolution operations in the embedding module. For intuitive interpretation of the agent’s action, we estimated and aligned the agent’s timepoints to the input timepoints by reversely computing the corresponding points in the input space.

In the meantime, for more neurophysiological inspection, in Fig. 6, we also visualized topographic maps of full signal segments in a trial and the signal segments selected by our proposed framework for the same trials in Fig. 5. Remarkably, topographic maps based on only the selected signal segments showed more clear and localized ERD/ERS pattern than those from the full signals. In particular, the selected signal segments have more prominent ERD patterns at around the C4 channel in the $\beta$-rhytm than the full signal segments in the Subject #2. When referring to the spectrogram of that subject in Fig. 5(a), it seemed there was no evident spectral power in the $\beta$-range over the full signal segments in a trial. However, after selecting the task-informative signal segments, we could observe a meaningful and distinguishable local pattern at the C4 channel in the $\beta$-range. Similarly, in the spectrogram of the full signals in a trial for the Subject #39 in Fig. 5(b), there seemed less prominent local activations in the $\mu$-range, thus no localized ERD/ERS pattern in Fig. 6(b). However, after selecting the task-relevant signal segments and plotting the corresponding topographic map, it was then observable a localized ERD/ERS at around the C3 channel. Based on these results, we empirically conclude that our agent module combined with MSNN in our proposed framework is capable of finding MI-relevant EEG signal segments, thus better learning MI-related feature representations and classifier enhancing the MI classification accuracy. Note that there there was no explicit guide or information for our agent to learn such neurophysiological knowledge.