Understanding Brain Dynamics for Color Perception using Wearable EEG headband

Muse headband consists of four channels/electrodes namely AF7 and TP9 on the left and AF8 and TP10 on the right. These are named and positioned according to the International 10-20 System ²²2https://en.wikipedia.org/wiki/10%E2%80%9320_system_(EEG), as shown in Figure 1. The sampling rate of Muse is 256Hz. The data from all channels was collected. There were eight subjects (aged 18-30yrs) who participated in the visual experiment. The experiment was conducted using the University of Nottingham’s Psychopy 3 (23, ) toolbox. Five trials each four minutes long were conducted for each participant at different times. In each trial, a color from RGB was shown in a random order, twenty times each, for a period of two seconds each, such that a black color was shown for two seconds between each of the RGB colors to provide a baseline to the experiment. The experiment was conducted in a dark room and the subjects were told to do minimum facial movements and eye blinks. A similar protocol has been followed in previous experiments too (9, ; 30, ; 31, ). The time period of the stimulus or the main color was kept small to only capture the effect of color on the cortical excitability. The data from Muse headband was collected using Muse SDK³³3https://choosemuse.com/development/ and a third party application for Muse called Mind Monitor⁴⁴4https://mind-monitor.com/. The Mind Monitor application indicates potential jaw clenches and eye blinks in the EEG data. We used this capability as a marker to get the starting timestamp of the data. The experiment started with a jaw clench which was captured by Mind Monitor and from that time stamp, the data was separated according to the color stimuli. The architecture we used is shown in Figure 2 and the detailed experimental setup is shown in Figure 3.

The raw EEG data is generally very noisy and it needs to be cleaned and pre-processed in order to remove artifacts from it. In our methodology, we cleaned the data in two steps. Firstly we analyzed the data using Matlab’s EEG lab software (18, ) and labeled any visible unwanted spikes and noise manually from the data. Secondly, we divided the data into small time windows of 50 ms and computed the variance of data in each window, if it was more than a selected threshold then the time window was flagged. We also examined the individual subject’s data and used the trial that has a minimum number of jaw clenches and eye blinks for further experimentation.

Feature extraction is a very vital part of our problem. The use of raw EEG data did not give good results in our experiment and so we used Time-Frequency analysis to find frequency band coefficients that were most relevant for our problem i.e. Alpha coefficients(8-12Hz) and Beta coefficients(13-30Hz). In past works, (19, ; 24, ) Discrete wavelet transform(DWT) has been used to extract the frequency bands of interest. However in our case, we were not interested in all the frequency bands, instead, we only considered alpha and beta bands. The use of DWT would have given us an improper breakdown of bands with the Alpha band in the range of 8-16Hz and beta in range of 16-32Hz and therefore to avoid this we used Continuous wavelet transform method as done in (25, ; 26, ) to extract the bands of interest. The mother wavelet that we used is the Morlet wavelet. The morlet wavelet has a peak in the center after which it tapers to the edges. The complex Morlet wavelet can be obtained by the convolution of a Gaussian with a sine wave and it is represented by the following equation:

where $t$ is time, A=$(\sigma_{t}\sqrt{\pi})^{-1/2}$, where $\sigma_{t}$ is duration of the wavelet and $f$ is the frequency of wavelet.

We extracted the power of alpha and beta bands from our EEG signal by convolution of Morlet wavelets of frequencies ranging from 8Hz to 30Hz along the whole signal at each time point. This was done with the help of Fast Fourier Transform (FFT). For each frequency, we first performed FFT of the signal and then the FFT of the Morlet wavelet. We then performed the convolution of the two transformed signals and applied Inverse Fourier transform to get the time-domain representation of data. The magnitude of the complex transformed signal was then extracted and it was squared to obtain the power across all time points. The important thing here is that we rejected the imaginary part as it gave us the phase information and the real part just gave us the band-passed signal but what we were more interested in was the power therefore we extracted the magnitude of the complex signal. We got a spectrogram like representation of the power of the signal, with columns denoting the time points and rows denoting the frequencies from 8Hz to 30Hz. Figure 4 shows the spectrograms for RGB colors. The Morlet wavelet helped to reduce edge artifacts and noise from the data. It also helped to obtain a balance in temporal precision and frequency precision. The sampling rate of the signal and the Morlet wavelet must be the same in order to perform convolution. Figure 5 shows the EEG data with and without the application of the Morlet wavelet. We removed the flagged artifacts that we got in Data cleaning step after applying the wavelet transform. This step was done after the transform was applied so that we did not reduce the points below the sampling frequency of 256Hz. The features were extracted from the remaining data. The features accounted for the spectral, correlation as well as statistical properties of the data which were normalized using z-score. We experimented with different time windows of length 100ms, 200ms, 500ms, 1000ms. Each window was taken with a 50% overlap with the next window. Each window was used to extract a single row of features vector. The next feature vector was obtained by moving the window half of its length.

The process of feature selection is important because it has many advantages like reduced training times, simplified and interpretable models, reduced chances of overfitting i.e. lesser variance and less impact of the curse of dimensionality. We performed feature selection/dimensionality reduction by two different methods. Firstly we used the Forward Feature selection technique which is a supervised approach and secondly, we used Autoencoders (33, ) which is an unsupervised approach for feature reduction. We elaborate on them in the following subsection.

We applied ML models on the 86 features that we extracted by the procedure explained in Section 3.3. The classification task was done in two folds. We first considered the data from individual subjects and applied models to that data to perform intra-subject classification for which we achieved an accuracy of 80.6%. Intra-subject classification helped us to study subject-specific differences of the EEG reactivity patterns. Then we considered the combined data from all the subjects and performed inter-subject classification and got an accuracy of 58.1%. The inter-subject case helped us to make a more generalized model. The classification was done in two ways and their performances have been compared. We performed classification using the original feature set as well as the reduced feature set from forward selection and autoencoders for both intra-subject and inter-subject. Below we elaborate on the models that we have used along with the chosen hyperparameters. We tuned the hyperparameters using Grid Search. The range of hyperparameters chosen was based on previous works (19, ; 21, ).

The accuracy score in our problem was calculated as :

In equation 8, $\hat{y_{i}}$ is the predicted value of the i-th sample and $y_{i}$ is the corresponding true value and $1(x)$ is the indicator function. $n_{samples}$ is the total number of samples. The accuracy indicates the samples that were correctly classified from all the samples.

ROC-AUC stands for Receiver operator characteristics- Area under the curve, it basically calculates the area under the receiver operator curve.The ROC curve is created by plotting the true positive rate ($\text{TPR}=\frac{TP}{TP+FN}$) against the false positive rate ($\text{FPR}=\frac{FP}{TN+FP}$) at various threshold settings. We find the area under the curve to evaluate our model. Since our problem is multiclass therefore we computed the average AUC of all possible pairwise combinations of classes using equation 5 as suggested in (37, ).

where $c$ is the number of classes and $\text{AUC}(j|k)$ is the AUC with $j$ as the positive class and $k$ as the negative class and $\text{AUC}(k|j)$ is vice versa. In general, $\text{AUC}(j|k)\neq\text{AUC}(k|j))$ in the multiclass case.

The Matthews correlation coefficient (27, ) is used to evaluate the quality of binary and multiclass classifications. The MCC is a kind of correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. In the multiclass case, Matthews correlation coefficient can be defined in terms of a confusion matrix $C$ for $K$ classes. The MCC for multiclass as suggested in (38, ) is calculated as follows:

where $t_{k}=\Sigma_{i}^{K}C_{ik}$ is the number of times class $k$ truly occurred, $p_{k}=\Sigma_{i}^{K}C_{ki}$ is the number of times class $k$ predicted, $c=\Sigma_{k}^{K}C_{kk}$ is the total number of samples correctly predicted and $s=\Sigma_{i}^{K}\Sigma_{j}^{K}C_{ij}$ the total number of samples.

Table 2 shows the accuracy score by using all features. We saw that the Random Forest algorithm performed the best and Neural Network and Gradient Boosting classifier also showed comparable results. The highest accuracy for an individual was 70.2% which was reasonably better than the accuracy of random guess i.e. 33%. The inter-subject accuracy of 56.8% was also very promising considering the fact that we applied leave one subject out cross-validation in this case. The results were better for intra-class classification which means that a customized model could be trained on an individual’s data and then it can be used for predictions for a particular subject rather than using data from different people which might also cause privacy issues.

In Table 3 we see the ROC-AUC scores. The highest average score of 0.851 was achieved by the Random Forest classifier, the average score of 0.803 was also better than an AUC score of 0.5 in the case of a random classifier.

The MCC scores by using all features are in Table 4. A MCC score of 0 means that the classifier is predicting randomly, in our case the highest MCC score was 0.523 which was much higher than a random prediction.

We saw significant improvement in the results with the use of forward feature selection which is a supervised feature selection technique. The irrelevant and noisy features were removed and the feature set was reduced to 10. This methodology helped us to curb the overfitting issue too and thus the performance on the validation set improved. In Table 5 we see the average accuracy scores by using the top 10 features. There was an increase in average accuracy by nearly 10% and we got the highest accuracy of almost 80.6% which was much better than any other previous approaches that have been used for EEG classification using RGB colors using wearable devices. The average subject accuracy of 72% showed that the classifier performed well for all the subjects. The average accuracy increased by 9.5%. In this case, also the results of intra-subject classification were better than that of inter-subject classification. The inter-subject classification accuracy improved by 1.3%. Random Forest algorithm had given us the best results in this case too with Neural network and Gradient Boost with comparable performance.

We see in Table 6 the AUC scores after forward feature selection. The best AUC score increased by 0.037 and the average AUC score has increased by 0.054. The MCC scores with forward feature selection are in Table 7 which also increased. Thus forward feature selection not only made our architecture efficient computationally but also increased the overall performance of the architecture. In fact, we got the best accuracy of 80.6% with the use of the Random Forest classifier with forward selection.

We applied autoencoder to observe how an unsupervised feature reduction technique would work on our data. With the autoencoder, a reduced feature set of 10 was obtained. Using this reduced feature set as input to the ML models, we achieved a lower average CV accuracy in comparison to classification using forward feature selection. Therefore autoencoders are not recommended for our application. Table 8-10 show the metrics achieved with the use of autoencoders.

The final proposed model for our application is that of Random Forest classifier with forward feature selection. In Table 11 we compare our results with previous efforts that have been done to classify EEG signals on basis of color stimuli using wearable EEG devices. In Figure 10 we show the ROC curve for the proposed model with AUC-ROC score of individual classes for all the subjects where 0 represents red, 1 represents green and 2 represents blue.