Comparison of Classification Algorithms Towards Subject-Specific and Subject-Independent BCI

Electroencephalography (EEG) pattern recognition is an essential component of Brain Computer Interface (BCI) systems. In BCI systems, the patterns in recorded EEG signals are identified, classified, and finally translated to a control signal to be sent to an external device. In particular, motor imagery (MI) EEG-based BCI systems are believed to show the brain activity in regions that are involved in movement imagination, without real limb execution [1, 2, 3]. This application is beneficial to the neuro-rehabilitation of a large community of paralyzed patients whose corticospinal track is blocked.

The EEG temporal and spatial characteristics are correlated with an individual’s physical and mental state and may vary between subjects. Hence, subject-specific (SS) BCI models are designed and trained per subject, which makes SS BCI designs time consuming and inconvenient. To overcome the limitations of BCI, subject-independent (SI) BCI designs were introduced and received attention recently. The purpose of SI models is to train a general model that can be used by new subjects with little or no experimental calibration and model parameter adaptation.

Although MI EEG-based BCI is a major research area, it is not yet fully understood, possibly due to the lack of reliable data as well as complicated data collection and model training. Studies on SS BCI designs show that EEG classification on MI data has considerably lower accuracy than some other BCI tasks, such as P300, ERP, or VEP [4, 5]. For example, a recent study [6] reported a mean classification accuracy of $71.1\%$ for MI data, but $96.7\%$ and $95.1\%$ for the ERP and SSVEP paradigms. Mean accuracies of only $60.4\%$, $70.0\%$, and $67.4\%$ for MI EEG classification tasks were reported in [7, 8, 9], respectively. This can be partially attributed to the inability of the target users in operating the MI-related BCI system, also known as ”BCI-illiteracy” [7, 8, 10, 11]. The recent study in [6] reported an average BCI-illiteracy of 53.7% for the MI BCI paradigm, but 11.1% and 10.2% for ERP and SSVEP, respectively. BCI illiteracy can increase the number of outliers considerably, resulting in unreliable data.

On the other hand, SI BCI studies have reported variable classification accuracies. In [12, 13], maximum mean accuracies of $67\%$ and $72\%$ were reported, while in [14, 15] a comparison of the SS and SI BCI approaches reported an accuracy of $73\%$ and $64\%$ for the former, whereas the latter achieved $77\%$ and $68\%$ accuracy. The SI accuracy in [15] was increased to $84\%$ by considering various modifications, such as trial removal and partial inclusion of new user data in the training procedure.

Low classification accuracy can result in an inappropriate control signal and consequently make the MI EEG-based BCI system fail in either the SS or SI paradigms. BCI-illiteracy, lack of required experimentation, high between-subject variability, and small sample sizes can significantly contribute to low classification accuracy. Small sample size require the use of simple/linear statistical machine learning (ML) methods with few degrees of freedom [16]. However, even with simple ML methods, small sample size can pose issues due to the curse of dimensionality phenomenon [17, 18] and limitations in test-train split of the data, among many other issues [19, 20]. In addition, well-known methods like cross-validation may become inadmissible due to their high variance in MI EEG data-poor environments [21, 22, 23].

In this paper, we performed an empirical study of the classification accuracy of MI EEG-based SS and SI BCI designs, using Linear Discriminant Analysis (LDA), Support Vector Machines (SVM), Classification and regression tree (CART), and k-Nearest Neighbors (KNN), while investigating the effect of varying training sample size on classification accuracy. The rest of this paper is organized as follows. The basic classification methods and performance evaluation are described in Section II. Section III contains the experimental setups and results, which are discussed in Section IV. Finally, Section V contains conclusions.

We report in this section the results of a comprehensive empirical comparison of the previous classification algorithms on the MI EEG data described earlier.

Similarly to [6], to account for the transition time, we removed the first 1s and last 0.5s of the MI segments of the EEG signals and performed the rest of the analysis on the middle 2.5 seconds. The signals were filtered using a Butterworth band-pass filter with lower and upper cut-off frequencies of 3 and 35 Hz, respectively. The p-value threshold for the t-test was set to $0.05$ for the SS model, which resulted in various numbers of selected features per subject. This threshold was set on $0.005$ for the SI experiments, which resulted in total of 29 features.

We performed several classification experiments using the LDA, RB-SVM, CART, and kNN classification algorithms. We set the minimum leaf size to 3 for CART and $k=3$ for kNN. Classifier generalization ability was evaluated by randomly splitting the available data into 50% training data $S_{N}$ and 50% testing data $S_{M}$, and using a test-set error estimator, which is the relative accuracy of the trained classifier on the testing set. In our SI design, the total sample size was $800$, so that the testing sample size was $400$, which is enough to guarantee excellent test-set error estimation accuracy [32].

For the training step, $n\leq 400$ sample points were randomly drawn from $S_{N}$. For each value of $n$, we trained the classifiers on the selected data, calculated the test set error on $S_{M}$, and finally computed the percent prediction accuracies as $100(1-\mbox{test set error})\%$. The classification performances was examined with $n\in\{10,15,20\}$ for the SS and $n\in\{100,150,200,250,300,350,400\}$ for the SI models.

Randomly drawn training samples introduces a randomness factor to the classifiers’ training and prediction performance. To take into account this randomness factor, we repeated each experimental case 100 times. The SS models’ performance are presented in Table I and Figures 3 and 5. Table II shows the mean (std) percent accuracy of the SI models, whereas figure 6 compares the mean accuracy of the four classifiers for varying training sample sizes, ranging from $n=100$ to $n=400$. Figure 7 displays box plots of classifier accuracy for SI and SS approaches and compares the mean accuracy and standard deviation for varying training sample size.

Our experiments lead to several interesting conclusions. For SI design, as displayed in figure 6, in the small sample-size region ($100\leq n\leq 200$), LDA was the best performing classifier. CART beats all other classifiers if $200\leq n\leq 400$ samples are provided. SVM was the most robust (lowest std) in both SS and SI paradigms. The steep improvement in SVM accuracy leads to the hypothesis that it may achieve the highest accuracy if the training sample size is large enough.

For the SS design, as illustrated by Figures 3 - 5 and Table I, increasing sample size improves the classification performance in terms of mean prediction accuracy, although it may be impracticable to collect many EEG recordings per subject. While SVM was the best performing classifier for only a few subjects (S3, S5, and S11 when n=20), the other three classifiers compete for the highest SS accuracy. This behaviour suggests a hypothesis that a hybrid model may be beneficial for SS BCI systems, where a specific classification algorithm is selected according to the maximum accuracy of the test set for that single individual.

The side-by-side box plots in figures 7 show that, for all classification algorithms, in contrast to the SI models, the variance of the trained SS classifiers is not reduced with an increment in training sample size. Indeed, the variance is increased noticeably in CART and SVM. This variability is expected at least due to the high between-subject differences in EEG signals, and suggests that a trained BCI may not be appropriate for a totally new user. It is worthy to note that the study in [15] also demonstrated improved accuracy by including a few samples from new users in training a SI BCI.

Overall, the results indicate that SS designs, and possibly a hybrid SS model, may be more appropriate for personalized MI BCI if collecting enough samples from a specific subject is feasible. In case a recording session is inconvenient for a subject, particularly for mentally or physically challenged people, a SI BCI may be used. Such SI BCI should be trained on a sufficiently large training data set that is collected from a large group of subjects covering between-subject variability. Comparing the box plots, it can be seen that classification mean accuracy in SI BCI even with a relatively large sample size ($n=400$) is close to that of SS BCI with $n=20$. However, the low variance of SI BCI acccuracy may make it more applicable for a multi-user system.

This paper presents our evaluation of the subject-specific (SS) and subject-independent (SI) paradigms using two popular classification algorithms in MI BCI design, LDA and SVM, in addition to the non-parametric algorithms k-NN and CART. Our results show that for SI BCI, LDA beats other classification algorithms in the presence of small sample size. CART outperforms the rest for a relatively large sample size, whereas we hypothesize that SVM achieves the highest accuracy with a sufficiently large sample size. It is important to stress that generalization performance should be evaluated on a relatively large and independent test set, which was done here.

For SS models, although LDA and CART performed well for majority of subjects, the best classifier performance depends on the subject, which suggests the applicability of a hybrid model. As was demonstrated in[33] for gender-specific BCIs, considering the subject gender may improve the performance of SI designs. We leave the investigation of gender-specific SI BCI performance to future work. Another interesting research problem is to investigate the EEG recording channels with the least discriminating information, for example by using a sequential backward search. In addition, deep learning methods, particularly convolutional neural networks (CNNs), are able to process brain maps as input images, but may need larger sample size for training a network with good generalization abilities. A future experiment may study the performance of CNNs for SI BCI considering varying sample sizes.