Raccoons vs Demons:
multiclass labeled P300 dataset

Firstly Brain-Computer Interfaces (BCI) were developed mostly with disabled patients in mind (e.g. [Bla+04], [Ric+13], [Hof+08]). Nowadays there are numerous attempts to employ BCI solutions for healthy people using noninvasive electrodes (see [Mar+12], ).

More than that BCI numerously applied to recreational activities such as games [And+16], [Kap+13]

Another crucial thing for further BCI development is processing and visualization instruments. At the beginning every laboratory used own programming environments and data format, some even used proprietary platforms. Happily at the moment we have fully open sourced systems based on Python programming language like MNE [Gra+13], PyRiemann [CBA13] and MOABB [JB18], also EDF data format [KO03] to store EEG/MEG data specifically.

Essentially in an oddball paradigm we have to solve 2 different but connected classification tasks:

1. 1.

   Binary

   Classification of a single epoch to contain P300 or not (in other words to be target or empty). In this case each epoch is classified independently. It is suitable for online (calibration then test on the same person) training because only 250 epochs are enough to train such a model. Another noticeable trait of this type of problem is imbalanced training and testing sets because each stimulus is activated the same number of times while only one of them is target. So we end up with $1$ to $(s-1)$ class ratio where $s$ is the number of stimuli. For this task accuracy metric is not sufficient because of class imbalance so we track precision, recall, f1 and ROC AUC in all experiments.

2. 2.

   Multiclass

   Classification of the stimulus chosen by user. As long as we have several stimuli (usually 5 to 7) it’s a multiclass problem. Here we consider several epochs at a time (usually equal number from each stimulus) and have prior knowledge that only one stimulus could be the target thus others are empty. In this case classification is more balanced because the participant chooses different stimuli each time (in fact it depends on visualization setup). Usually this problem is solved by summation of probabilities of binary problem by stimulus and choosing the one with highest sum. Regarding metrics this task is well described by accuracy.

Let’s introduce some terminology helpful in further narration (for illustration see 3:

• •

  Round - sequence of epochs with one target where each stimulus was activated once (and produced one epoch), so this is quantum of multiclass prediction. Usually stimuli are activated in random order.

• •

  Act - sequence of rounds with one target after which decision on target stimulus is made. Usually the number of sessions is from 5 to 10 in our experiments.

• •

  (Game) Record - sequence of acts obtained from one person during one recording session (one game).

• •

  Dataset - collection of records sharing the same visual activation and hardware setup.

As a main metrics we use multiclass accuracy because in a final gaming environment this is how user measures performance. Binary task metrics are measured in all experiments as well and in many cases higher binary metrics means higher multiclass accuracy.

We divide EEG signal handling to two principal steps:

1. 1.

   Preprocessing - which includes signal processing stuff and some non-statistical permutations e.g. clipping, epochs slicing, etc. At the end of this step we obtain epochs with labels (either binary or multiclass).

2. 2.

   Classification - applying statistical classifiers to predict labels having epoch data

For preprocessing step we search for optimal hyperparameter values by cross-validation. The reason for this seemingly trivial step is lack of motivation in choosing such parameters. Although many researches report similar filtering (e.g. [Con+11], [BC14], [Gug+09], [RG08], [Kay+18]) they don’t describe motivation for choosing one or another downsampling factor, filtering band, clipping as well as epoch duration. Below we report results of classification pipeline with varying parameters and choose ones that minimize final epoch representation without metrics degradation.

For this step we use predefined set

Binary epochs classification is a common task for BCI and main approaches are well described in [Lot+18]. In recent years Deep Learning emerged in all fields, so EEG analysis is not an exception and we have beautiful overviews of [CHC19] and [Roy+19].

Our preprocessing is quite standard:

1. 1.

   Decimation with an anti-aliasing filter from SciPy package [Vir+20]

2. 2.

   Bandpass filtering with Butterworth IIR

3. 3.

   Clipping off scale samples

4. 4.

   Channelwise standard scaling (subtracting mean and dividing by std for all values of each channel)

Each test were conducted with other parameters fixed at some sane level (such that qualitative result persisted)

Having robust binary target-empty epoch classifier (trained for each person separately) we may proceed to aggregating these predictions to a final multiclass problem solution.

As a standard approach we consider taking epochs grouped by activated stimulus, predicting their probabilities to be target and sum these probabilities. This way we obtain "activation score" for each stimulus. Then multiclass prediction consists in choosing the stimulus with the highest activation score. We call it baseline multiclass classification.

In gaming environment (which is our case) misclassification cost is high because person feels frustrated and snatched. So we have decided to introduce some confidence score by which we could reject potentially wrong predictions. In case of rejection "misfire" animation is played and we ask the user to repeat the choice procedure.

Some attempts to fix classification errors were taken. For example in [Mar+12] competition EEG signal itself used to predict if SVM classifier were right or not.

Introduction of multiclass labels allows us to incorporate prior knowledge of choosing only one stimulus per session. This could efficiently be used in Neural Networks architecture development as well as in Bayesian methods based approaches.