High Performance P300 Spellers Using GPT2 Word Prediction With Cross-Subject Training

In spoken English, it is well established that vowels occur more frequently than consonants [22]. Hence, from a timeline perspective of when they get highlighted in the flashboard scanning order, it is only natural that they occur earlier than some relatively rare consonants such as ${}^{\prime}x^{\prime}$ or ${}^{\prime}z^{\prime}$. We will now describe how to perform such trade-offs using multi-level language models.

Smoothing techniques [16], allow transitioning across models when OOV words are encountered. Consequentially, complex models can be blended in with simpler models (Figure 2). As an example, consider the word ’cat’ and suppose ’ca’ has already been spelled. Assume that ’cat’ is not present in the language model.

Hence, the probability of ’t’ given that ’ca’ has already been spelled (represented by the notation, p(‘t’—’ca’) is zero. Smoothing updates this OOV probability by considering other contexts where ’a’ and ‘t’ have co-occurred through p(’t’/’a’). Multiple smoothing techniques were considered, and Knesser-Ney smoothing was chosen for its superior OOV [26] performance. Also note that in general, the more sophisticated the language model, the better the conditional probability can be predicted [23]. Language models such as the one used contain a large vocabulary, which results in high prediction accuracy even as character length increases.

Knesser-Ney smoothing: In the following terminology, Model(’*’) denotes the occurrence frequency of the characters ’*’. For example, biword_model(’the quick’) is the frequency of the word ’the quick’ in the biword language model. Also, as an example of conditional probability syntax, p_biword (’c’ — ’the qui’) is the biword model probability of the character ’c’ being next given that the partial phrase ’the qui’ has been decoded.

where $L_{1}$is the normalizing constant equal to the number of distinct letters that follow ’the qui’ in the biword model divided by the biword model count of ’the qui’. If biword_model (’the qui’ ) = 0, $L_{1}$ =1 to take care of out of vocabulary (OOV) conditions.

where unigram_model(”) = total model entries and $d_{1},d_{2},d_{3},d_{4}$ are tunable parameters with $0\leq d_{1},d_{2},d_{3},d_{4}\leq 1$. All these d values were set to the commonly used mid-range default of 0.5. Just as $L_{1}$ is a biword model normalizing constant in Equation 1 whose derivation was described earlier with an example, $L_{2},L_{3},L_{4}$ are also normalizing constants similarly obtained from the corresponding lower order language models. More specifically, in the example used, $L_{2}$ is the normalizing constant equal to the number of distinct letters that follow ’qui’ in the word model divided by the word model count of ’qui’. Further, for satisfying OOV conditions, if word_model (’qui’) = 0, $L_{2}$ =1. If trigram_model (’ui’) = 0, $L_{3}$= 1 and if bigram_model (’i’) = 0, $L_{4}$ = 1 thereby accounting for the values of $L_{1}$…$L_{4}$ in all OOV cases. In summary, through Equations 1-5, the most complex model in Figure 2 falls back to lower-level models with OOV characters.

The step-wise linear discriminant analysis (SWLDA) as described in Speier [11] et al. and originally used in [3] was employed as the classifier. SWLDA utilizes a discriminant function, which is determined at the training step [28]. The SWLDA classifier was trained using one of two methods: Across subject cross validation (ASCV) (also known as leave-one-subject-out cross validation) and within-subject cross validation (WSCV). Though both use SWLDA, note that these two techniques fundamentally differ in how the entire subject responses are split between what was used for training and was used for testing.
a) ASCV: Out of $N$ subjects, one subject was chosen to be the test subject, while the other $N-{\it 1}$ subjects’ data were used to train the SWLDA classifier. The SWLDA classifier then attempted to intake the test subject’s data and spell out the target phrase. Note that this technique finally produces one classifier table for all $N$ subjects.
b) WSCV: In the WSCV, 2-fold cross-validation using SWLDA was performed on each individual subject EEG data (unlike ASCV) resulting in a unique classifier output for each subject. More specifically, the output phrase contained at least $20$ characters and had 120 flashes for every character that was split into three sets. During the first testing iteration, the first set was chosen to be the test phrase, while the other two sets were used to train the SWLDA classifier. On completion, the same procedure would repeat, but with the second and third set being the test phrase during the second and third trial respectively. This method would then be replicated on all the subjects.

During training, class labels were predicted using ordinary-least-squares. Next, the forward step-wise analysis adds the most important features, and the backward analysis step determines and removes the least important features. This process repeats until the number of features remains constant after a set number of iterations or until the number of required features is met. The final selected features are then added to the discriminant function. The score for flash $i$ for character $t$, $y_{t}^{i}$ , can then be computed as the dot product of the feature weight vector with the features from that trial’s signal.

The flashboard is a central part of a P300-based BCI system. In this section, we will describe schemes that organize the highlighted flashboard characters based on their occurrence probabilities. Note that these schemes are mapped virtually onto conventional static flashboards, whereby only the highlighted set of characters are modified and not the underlying flashboard itself. In other words, the flashboard is constant through the flashing sequence, though the highlighted characters can be arbitrarily chosen. By doing so, this readily overlays onto any flashboard in current use

a) Sequential flashboard: The flashboard highlighting is virtually modified (as shown in Figure 3) such that each set of highlighted characters are picked in a frequency weighted rather than alphabetical manner. More specifically, higher frequency characters are highlighted earlier, whereby the initial flashes are more likely to select the target character. Results from these modifications will be described in section 3 and discussed in section 4.
b) Diagonal flashboard: Consider the diagonal design of Figure 4 where the most likely highlighted characters are organized along diagonals. In the figure, note that the most likely letters e, t, a, i, n, o are on the main diagonal followed by the next likely letters on upper and lower diagonals and so on. This design forces characters that appear in similar contexts to flash separately, thus making them easier to distinguish. Once again, this virtual technique can also easily overlay on to static alphabetically arranged flashboards. Gains from these schemes are provided in section 3.

c) Huffman flashboard: Huffman codes [20] are popular in lossless data compression and used in prominent multimedia standards such as JPEG and MP3. The fundamental idea is to use fewer bits to represent the most frequent characters and more bits to represent less frequent characters. By doing so, the overall number of bits required to represent the complete data is minimized, approaching information theoretical performance bounds. Note that different frequency tables would result in different Huffman codes. This section deals with adapting this approach to P300 spellers. For example, suppose we had only $5$ symbols. Huffman scanning descends through a probabilistically organized binary tree (as illustrated in Figure 5) based on the prior decoded character.

One requirement of Huffman flashing is that the determination of whether the subject responded positively to the highlighted flashboard characters must be feedback to the flashing unit to help it identify the next set of flashing characters. For feasibility, the overall decoder latency should be small, which creates a limitation for its practical use.

This section describes modifications to adapt the P300 BCI scanning to optimize performance.
a) Weighted scanning order: Conventional random and deterministic flashing are straightforward schemes, where the flashing set of characters is either picked deterministically or round-robin. In contrast, in the proposed weighted flashing enhancement to conventional flashing, the highlighted set is picked based on that set’s probability as opposed to the random scheme where the sets are assumed to be equi-probable. The idea is that the more likely characters are highlighted earlier.
b) Dynamic stopping: Once a character is determined at the decoder, continuing the flashing creates an unnecessary reduction in speed, which is avoided by using a “dynamic stopping” strategy. Flashing of the current character is terminated and the next character flash is begun.

Flashboard efficiency can be increased by replacing unlikely characters with word choices.
a) Dijkstra’s Algorithm: We used Dijkstra’s algorithm [19] for word prediction. This was implemented in the form of

a trellis similar to dynamic programming techniques as shown in Figure 6. As shown in the figure, every node of a trellis stage has $M$ incoming possibilities (where $M$ is the number of unique characters in the flashboard) out of which only the one with the highest probability is retained. In doing so, the complexity was reduced, as every possibility did not have to be exhaustively evaluated. Further, as more stages were incorporated into the trellis, longer-length word choices are obtained while resulting in higher evaluation time and complexity. For each branch at a given stage, probabilities were determined by using the frequency count of branch characters given prior characters using the language model along with smoothing. At every stage, the highest probability completions then provided the most likely word completions.
b) GPT2: A large unsupervised transformer [23],[24] based model released by OpenAI, GPT2 is an enhanced deep neural network follow-on to the earlier GPT model and is powerful enough to generate coherent passages of text. In contrast to earlier models, GPT2 uses “attention mechanisms” to focus on segments of input text it thinks are the most relevant. In the context of BCI, it can become a sophisticated tool that predicts the subsequent word in a partially formed sentence given prior words. Though GPT2 is powerful, it does not have the ability of OOV prediction. However, it can be combined with Dijkstra’s algorithm (as shown in Figure 7) to account for OOV words, thereby creating a layered approach similar to character-based smoothing algorithms. More specifically, in case GPT2 word choices result in an empty or partially complete set of choices, one can then move to Dijkstra’s to predict OOV words.

A simple decoding scheme based on thresholding is used in this work to deduce the character or word. First, the score for each flash is computed by, $y_{t}^{i}={\mathbf{w}}{\bf.z_{t}^{i}}$ where the feature weight vector is determined by feature extraction as described in the previous subsection. With the assumption that the distributions were Gaussian [11], the “attended” and “non-attended” signals (note that this terminology refers to highlighted and non-highlighted characters on the flashboard) were found and given by

where ${\mathbf{A}_{t}^{i}}$ is the set of characters illuminated for the $i^{th}$ flash for character $t$ in the sequence. $\mu_{a},\sigma_{a}^{2},\mu_{n},\sigma_{n}^{2}$ are the means and variances of the distributions for the attended and non-attended flashes, respectively.

where $P(x_{t}|x_{t-1},..,x_{0})$ is the prior probability of a character given the history, $f(y_{t}^{i}|x_{t})$ are the pdf’s from Equation 6, and $Z$ is a normalizing constant. The language model is used to derive the prior as per

where $Count(x_{t},x_{t-1},..,x_{0})$ is the number of occurrences of $x_{t}x_{t-1}..x_{0}$ in the corpus. A threshold probability, $P_{Thresh}$, is then set to determine when a decision should be made. The program flashes characters until either $max_{x_{t}}\,P(x_{t}|{\bf y_{t}},x_{t-1},..,x_{0})\geq P_{Thresh}$ or the number of sets of flashes reaches a predetermined maximum value. The classifier then selects the character that satisfies $arg\,max_{x_{t}}\,P(x_{t}|{\bf y_{t}},x_{t-1},..,x_{0})$.

The subject can correct typing errors by invoking the backspace, which is provided in the flashboard. As a result, all the typing errors do get fixed, making the overall error rate zero. Alternatively, one can have an autocorrect wherein typed words are checked for presence in a dictionary and erred words are replaced by closest most likely words. This technique is faster, but some mistyped words are likely to either be mis-corrected or left uncorrected, which would result in a non-zero final error rate. While a vast array of more powerful decoding schemes involving hidden Markov models, machine learning, and particle filters [11] can be used, note that these enhancements only complement the techniques introduced in this paper and do not deter the results.

Data for offline analyses were obtained from 78 healthy volunteers with normal or corrected to normal vision between the ages of 20 and 35. All data were acquired using GTEC amplifiers, active EEG electrodes, and electrode cap (Guger Technologies, Graz, Austria); sampled at 256 Hz, referenced to the left ear; grounded to AF_Z; and filtered using a band-pass of 0.1 – 60 Hz. The electrode set consisted of 32 channels placed according to a previously published configuration [25]. The system used a 6×6 character grid, row and column flashes, and a stimulus duration of 100 msec and an interstimulus interval of 25 ms for a stimulus onset asynchrony of 125 msec. Statistical variation from such small data sets would limit the confidence in the results. As a result, a large and meaningful dataset, the text of the “Declaration of Independence” (DOI) is used as the simulation data [30]. The full length of this document is 7,892 characters after removing all punctuations and special characters. The DOI was chosen as the simulation dataset as it is concise, well-known, and contains words of varying length, including a number of out-of-vocabulary (OOV) words that stress the simulation. Furthermore, to keep the simulation accurate, numerous subject responses are used to provide a wide range of brain activity.

While running the simulations with the DOI dataset, the “attended” and “unattended” scores (the terminology refers to whether the desired character is present or absent in the highlighted flashboard character) for a given subject were chosen from the lab results of that subject. The lab results are sampled by a two state Markov chain which models the “attended” and the “unattended” states. Thus, there are four possible sampling combinations depending on whether the current and previous state was “attended” or “unattended”. When a character or word is decoded, it is compared to the intended character or word. In the case of an error, a “Backspace” is initiated as the subsequent character. Following this, the deleted character/word is flashed again. The net effect of this is to “Undo” errors, thereby making the final error rate zero. However, note that this could potentially even require multiple attempts and a long follow-up time. One could significantly constrain the follow-up time and tolerate the residual error, and this evaluation could be a possible future study.

Information Transfer Rate (ITR) is a crucial metric in BCI evaluation, quantifying the speed and accuracy of communication [4]. The metric is appealing for several reasons: it is derived from information theory principles, it combines the competing statistics of speed and accuracy, and it reduces the performance to an information transfer problem that can be compared across applications [31]. It is typically calculated using the formula: $\frac{log_{2}(N+1)}{T+P_{f}(T_{r}+T_{c})}\times 60\;bits/minute$ where $N$ is the number of characters in the speller (grid size), $T$ is the total time taken to complete the task (including pauses, corrections, etc.). Note that $P_{f}$ is the probability of incorrectly selecting a character, $T_{r}$ is the time required for a single character selection, and $T_{c}$ is the time taken for error corrections.

A second metric we used is the retry rate (or error rate) which measures the number of backspaces initiated as a fraction of the overall communicated characters. Note that when a character or word is decoded in error, the next character is assumed to be a backspace. The objective being to attempt mimicking a subject, correcting the misinterpreted error or word. This process repeats until the character/word is decoded correctly resulting in the error rate for that character or word going to 0. However, when the simulation of a particular character or word requires more than 75 flashboard scans, it is abandoned and an ITR of 0 (100% error rate) assigned for that particular character or word and the simulation moves on to the next character or word.

Some errors (e.g., obvious typos) do not affect the readability of text and could therefore be allowed to remain in the final output without affecting the meaning. Allowing these errors to remain could then potentially result in a more efficient system because it would not waste time forcing the user to make these corrections. Language models could also be used to automatically correct some of these errors [11],[12] if they are not corrected manually.

Both the Shapiro-Wilk test [34] and the Kolmogorov-Smirnov test [34] on the results comprehensively rejects normality $(p<10^{-3})$. Consequentially, use non-parametric tests for hypothesis testing. The Kruskal-Wallis test [34] was performed on the data, $(p<10^{-6})$ rejecting the null hypothesis of each scheme’s ITR, error rate originating from the same distribution. Hence, the Wilcoxon signed rank test [34] for paired samples is then conducted to determine the statistical significance.