Concurrent Linguistic Error Detection (CLED) for Large Language Models

As shown in Fig. 1, the OPUS-MT [13] model employs the standard Transformer structure with an encoder and a decoder, followed by a prediction layer. The encoder consists of N layers, and each layer is made of two sub-layers, Self-Attention (SA) and Feed-Forward Network (FFN). The decoder also consists of N layers; however, different from the encoder, there is a cross-attention (CA) sub-layer between the SA and the FFN in each layer of the decoder. The prediction layer consists of a linear operation and a log softmax operation. The details of the encoder and decoder are introduced next.

For Natural Language Processing (NLP) applications, the input X of the encoder is a M×W embedding matrix for the M words (or tokens) in the input text. Each row is the combination of the token embedding of a word and the position embedding, whose dimensions are both 1×W. In the SA sub-layer, the Multi-Head self-attention (MH-SA) performs the dot-product attention for a query matrix Q, a key matrix K, and a value matrix V as

where Q, K and V are the linear transformations of X as $Q=XW_{Q}^{T}+B_{Q}$, $K=XW_{K}^{T}+B_{K}$ and $V=XW_{V}^{T}+B_{V}$, respectively, and $W_{Q}$, $W_{K}$, and $W_{V}$ are the weight matrices, $B_{Q}$, $B_{K}$, and $B_{V}$ are the bias vectors. As shown in the upper left corner of Fig. 1, the MH-SA mechanism includes H ($H=W/d_{h}$) heads, and the output of the i-th head $head_{i}\left(X\right)$ is expressed as

The outputs of all heads are then concatenated as a M×W matrix, and merged by an W×W SA output matrix $W_{O}$ as

whose dimension is still M×W. After residual addition with the input X and layer normalization, the input to the FFN sub-layer becomes $Y=norm\left(X+SA\left(X\right)\right)$.

The Feed Forward (FF) block is composed of two fully connected layers and a non-linear activation GeLU between them, so the output of the FF block is expressed as

where $W_{F1}$ and $W_{F2}$ are the weight matrices for the two fully-connected layers, respectively, and $B_{F1}$ and $B_{F2}$ are the corresponding bias vectors. Finally, after residual addition and layer normalization, the output of one layer in the encoder becomes $Z=norm\left(Y+FF\left(Y\right)\right)$, which has the same dimensions as the input matrix X (M×W), therefore it can be fed as input to the next encoder layer.

The decoder generates the representation of the (n+1)-th token based on the first n predicted tokens $X_{n}$, which is a n×W token embedding matrix. The SA in the decoder is almost the same as in the encoder, but it applies a mask matrix to mask the information after the current token. The processing of the CA is like as SA; the only difference is that K and V are generated based on the encoder output Z rather than the output of SA. The FFN in the decoder is the same as in the encoder.

The OPUS-MT model includes 6 encoder layers and 6 decoder layers (N = 6), and each MH-SA includes 8 heads (H = 8). The embedding dimension is W = 512. The total number of parameters in OPUS-MT is approximately 78 million, so needing a memory of 156 million bytes when the parameters are stored in half-precision floating-point format.

The second model considered in this paper is T5 [3], a larger model that is a variant of the standard Transformer. As shown in Fig. 2, T5 introduces the following four main improvements over the original transformer.

1. 1.

   The position information is not included in the input embedding matrix but it is merged in the operation of MH-SA in the form of a relative position bias ($B_{RP}$). Therefore, Eqn. (1) becomes

   $$MHSA\left(Q,K,V\right)=softmax\left(QK^{T}+B_{RP}\right)\times V.$$

   (5)

2. 2.

   Layer normalization after the residual addition in SA, CA, or FFN in the standard Transformer is moved to the input of those sub-layers.

3. 3.

   The linear transformation in T5 removes all bias vectors.

4. 4.

   The FF block in T5 is made of three fully connected layers, and the operation is changed to

   $$FF\left(Y\right)=\left(GeLU\left(YW_{F1}^{T}\right)\odot\left(YW_{F2}^{T}\right)\right)W_{Fo}^{T},$$

   (6)

   where $W_{F1}$, $W_{F2}$ and $W_{Fo}$ are the weight matrices for the three fully connected layers respectively, and $\odot$ denotes the Hadamard product.

The standard T5 model includes 12 encoder layers and 12 decoder layers (N = 12), and each MH-SA includes 12 heads (H = 12). The embedding dimension is W = 768. Then the total number of parameters in T5 is approximately 247 million, which needs a memory of 494 million bytes for the half-precision floating point format.

In many cases, error detection can be performed concurrently with the system operation [15]. For example, in the Fast Fourier Transform (FFT), error detection can be performed by checking that the energy of the inputs is the same as the energy of the outputs [19]. Similar schemes that exploit the relations between inputs and outputs, have been proposed for other signal processing blocks [14]. For machine learning systems, the use of a concurrent classifier has been recently proposed [17]. This scheme (referred to as Concurrent Classifier Error Detection (CCED)) relies on monitoring a number of internal nodes of the machine learning model and feeding their values to a classifier that detects errors as illustrated in Fig. 3. The scheme has been evaluated for BERT when it is used for emotion classification or to locate the answer to a question in a given text. The nodes monitored in the CCED scheme of [17] were the softmax values, which are used to select the emotion or the position of the answer in the text. The use of internal nodes is needed in systems that produce outputs with no inherent structure or properties that can be used for error detection. For example, in emotion classification, the selected emotion provides little information on whether the system is in error. Instead, the distribution of the softmax values among the possible emotions can exhibit different patterns when there is an error, such that the concurrent classifier can detect the error. This is the case for BERT as shown in the evaluation results presented in [17].

However, when the models are used to generate text as it is commonly found in many modern applications such as ChatGPT, error detection based on the softmax values has some intrinsic limitations. First, some of the models are closed and do not expose the softmax values. For example, OpenAI only provides the values of the top five tokens¹¹1see https://platform.openai.com/docs/api-reference/chat/create which limits the use of CCED. A second limitation is that CCED would be significantly more complex because 1) the number of tokens is typically in the order of tens of thousands, so the concurrent classifier will no longer be simple, and 2) it must be run per token so for an interaction with the model, as an example answering a question, hundreds of runs of a more complex concurrent classifier will be needed. An interesting observation is that when language models are used to generate text, their outputs can be used for error detection and thus, the CCED scheme can be extended to work directly on the output of the system. This makes error detection significantly simpler and more widely applicable to models for which there is no access to internal nodes and reduces the complexity and overhead of the error detection process.

The objective of the error model is to capture the impact of transient soft errors on the language models. This is motivated by the importance of radiation-induced soft errors in computing components such as memories, GPUs, or FPGAs [20],[21]. Soft errors can affect both sequential logic elements flipping typically one of the bits stored and also combinational logic elements that can propagate the error until a sequential element catches an incorrect value [22]. In the first case, simulation of the errors, for example in the parameters of the model, can be easily performed by flipping one of the bits. However, the simulation of the effect of errors on combinational logic is more complex because it requires access to the detailed design of the component, for example, a GPU on which the model is run. This is typically not feasible and if it were, it would make the results specific to the platform evaluated.

Therefore, based on the practical restrictions of simulating errors on combinational logic and also to make the evaluation independent of the specific components when implementing the LLMs, we consider a single-bit error in one of the parameters of the LLM. This model captures the effect of soft errors on the model parameters when stored inside a computing unit, for example in a cache or a register file inside a GPU. The effects of soft errors in combinational elements are not fully captured, but to some extent, we can expect them to be similar to errors on the model parameters for floating point representations for which errors on the most significant bits of the exponent are critical [12]. Hence, the error model enables us to explore the effects of transient soft errors during the execution of an LLM in a general computing platform.

Since the errors considered in this paper are soft errors, they do not induce any permanent damage to the circuit functionality. We also assume them to occur inside the computational unit (for example a GPU or a TPU). Therefore, once an error is detected when executing the language model, it can be corrected by running the same input text a second time as that implies that the LLM parameters are reloaded from memory and any previous intermediate value is removed. Therefore, the CLED scheme proposed in this paper can correct the detected errors by re-computation. Note that differently from the standard re-computation that must rerun all input texts, CLED only runs a second time for input texts for which an error has been detected.

1) Dataset and evaluation metric

To evaluate the performance of T5 summarization, the test set of CNN Daily Mail²²2Further details on the datasets used for T5 and OPUS-MT are provided in the evaluation section. is used and the Rouge1 score is applied as the performance metric, which is calculated as

where, the denominator is the total number of 1-grams, and the numerator is the number of common 1-grams between the reference summary and generated summary. When there are no errors, the Rouge1 for T5 summarization is 0.4081.

2) Distribution of weights in the T5 model

To study the impact of errors on different bits, we first analyze the distribution of all weights in T5 that are publicly available, including $W_{Q}$, $W_{K}$, $W_{V}$, $W_{O}$, $W_{Fi1}$, $W_{Fi2}$ and $W_{Fo}$ of the encoder and those weight matrices in the decoder. Based on our analysis, the distribution of the weights for different layers and types is almost the same, and the average value of each bit for single precision weights is plotted in Fig. 4. Based on this figure, we can get three observations: (a) the 1st exponent bit is highly likely to be 0; (b) the 2nd to 6th exponent bits are highly likely to be 1; (c) the average value of other bits is approximately 0.5. This means that the exponents bits have little variability and only the mantissa bits change frequently.

3) General impact of errors on different bit positions

For a floating-point representation, a 0 to 1 flip of the i-th exponent bit increases the value of the weight by a factor of $2^{128}/2^{i-1}$, so errors on the upper exponent bits have a larger impact. In this section, we perform error injection experiments to evaluate the effect of flips on different bit positions. In the test for a specific bit position (e.g., x), we first randomly select one parameter, flip bit-x of this parameter, and then run the model on the test set. If at least one of the summaries is different from the error-free one, then the error Rouge1 score is recorded. The process is repeated 1000 times, and the average Rouge1 value is used as the significance of bit-x. The results are shown in Fig. 5. The flips of the 1st exponent bit introduce a severe performance degradation, and the flips of the 2nd to 5th exponent bits introduce a slight performance loss. Actually, all 0 to 1 flip of the first 5 exponent bits cause an obvious performance loss (as shown in Table I). As the 2nd to 5th exponent bits are 0 for only 1% of the cases, the overall performance loss seems small. In general, these results show that the first 5 exponent bits are the CBs for T5.

The experiment results show that in both the encoder and decoder errors on CBs of $W_{Q}$ and $W_{K}$ have negligible impact on the model performance due to the softmax operation, and those on the 1st exponent bit of $W_{F1}$ only introduce a few grammar errors in the output summary due to the GeLU operation. Therefore, the performance degradation mainly comes from the 0 to 1 flip of CBs of $W_{V}$, $W_{O}$, $W_{F2}$ and $W_{Fo}$.

4) Analysis of the impact of errors on CBs

The effect of errors on the CBs of $W_{V}$, $W_{O}$, $W_{F2}$ and $W_{Fo}$ are similar, and the results for different CBs are listed in Table II, in which the typical wrong outputs are also provided; the error on the 1st exponent bit will result in a fixed output that is irrelevant to the input text. Errors on the 2nd to 4th exponent bits also result in irrelevant summaries, but the outputs change for different inputs; errors on the 5th exponent bit only result in grammar errors in the output text. A discussion of the causes of these error characteristics is presented next.

• •

  Errors on the 1st exponent bit introduce abnormally large values (positive or negative) in the intermediate outputs, which may cause overflow when performing layer normalization. Since the T5 model does not have bias parameters, the overflow causes a 0 output of the layer normalization. Irrespective of which layer the errors occur (encoder or decoder), such 0 outputs force the final results to be fixed and unrelated to the input text.

• •

  Errors on the 2nd to 5th exponent bits do not cause overflow in the layer normalization operation, but the wrong values in the intermediate results cause changes in the final logits scores for the word selection in the prediction layer. For errors on the 2nd, 3rd, and 4th exponent bits of some large weights, the logits scores change significantly, so that the output is irrelevant to the input text and looks random. However, for the 4th exponent bits of some small weights and the 5th exponent bits of the weights, errors only introduce small changes to the logits scores, which would then cause some grammar errors in the final output. For example, the errors on the 5th exponent bit may cause repeating words in the text as shown in Table II.

1) Dataset and evaluation metric

We use the IWSLT2017 dataset [23] in this section, and the BLEU score is used to evaluate the translation performance of OPUS-MT, which is defined as

where $p_{n}$ is the accuracy of n-grams, $count_{output}^{n}$ and $count_{ref}^{n}$ represent the number of n-grams in the output translation and the reference translation, respectively, $count_{match}^{n}$ is the number of common n-grams in both the output translation and the reference translation, BP is the penalty factor, c is the sentence length of the output translation, and r is the sentence length of the reference translation. When there are no errors, the BLEU score is 0.2186.

2) Identification of CBs for OPUS-MT model

The weight distribution of the OPUS-MT model is first analyzed, and the average values of different bits of the weights are shown in Fig. 6. As we can see, the results are similar to T5, but they also reveal two differences: (a) the 1st exponent bit is always 0; (b) the 2nd to 4th exponent bits are always 1.

Then we perform error injection experiments to evaluate the impact of errors on different bit positions, and the average BLEU scores over 1000 runs are shown in Fig. 6. Only errors on the 1st exponent bit introduce a severe performance loss, while those on other bits do not affect the model performance. These results are expected because errors on the (upper) exponent bits lower than the 1st bit only decrease the weights, and thus, they do not affect the model performance. Therefore, the 1st exponent bit is the only CB for the OPUS-MT model.

3) Analysis of the impact of errors on the CB

Similar to the results for T5, the effects on errors on $W_{Q}$, $W_{K}$ and $W_{f1}$ are reduced by softmax and GeLU, the performance loss mainly comes from the errors on $W_{V}$, $W_{O}$ and $W_{f2}$. However, different from the cases for T5, the layer normalization in OPUS-MT also includes bias, so the layer output becomes a fixed matrix (with each row equal to the bias vector) instead of a 0 matrix. Therefore, if the errors occur in the encoder part, the translation results are determined by the bias vector of the layer norm, and they are not related to the input text. Since the layer norm bias is different for different layers, the wrong translation results also change for different layers, but it is fixed for different inputs. The errors in a layer of the decoder part also produce a fixed layer output matrix with each row becoming the bias vector, and finally generating a wrong translation result that is irrelevant to the input text. However, a difference is encountered, namely the fixed layer output is only used for the computation of Q of the next layer, and the computations for K and V of the following layers are based on the correct results from the encoder. Therefore, the final results are different for different inputs. Table III shows some examples of the wrong translation results for different layers in the encoder and decoder parts. The results are meaningless and include a repeat of fixed words or symbols. This provides the reasons for the BLEU score for errors on the 1st exponent bit being 0 in Fig. 7.

The main motivation for the development of the proposed CLED scheme is to provide error detection mechanisms that

1. 1.

   They operate on models for which there is no access to the internal nodes.

2. 2.

   They are widely applicable to language models regardless of their implementation details.

To the best of the authors’ knowledge, there is no concurrent error detection scheme that meets these requirements, CCED [17] meets partially 2) but it does not fulfill 1). As discussed in the introduction, a trivial but at the same time key observation is that when language models are used to generate text, we expect the text to be valid. This implies that words must be correct and the sentences and constructions must follow the rules of the language. Intuitively, errors that cause the model to malfunction are likely to output meaningless text or at least incorrect text as discussed in the previous section. As an example, this is an excerpt of the output of T5 with an error injected in one of its parameters: “Satoisesaias,mass of a customiseaed toeasaiesaado hudness in a shader in hiuued tuuuiut tiuisuaiui”; it can be seen that the text is clearly incorrect. This example and the ones presented in the previous section suggest that it may be possible to detect errors based only on the features of the output text to also enable the protection of closed models and applicable to models regardless of their implementation details.

The proposed CLED scheme is illustrated in Fig. 8. It operates on the output text of the LLM and therefore, it does not require any modifications to the model, or access to any internal nodes. This makes CLED attractive to protect models that are provided as a black box to which no modification can be made. The error detection is performed in two steps.

Initially, relevant linguistic features are extracted from the output text and fed to another classifier that performs error detection. Since error detection is based solely on the properties of the text, the approach should be applicable to newer LLMs. This is very interesting as the pace at which models are introduced is hectic and having a error detection scheme that is general and can be applied to any LLM facilitates the fast protection of LLMs as they are introduced.

Then, once an error is detected, as it is a soft error that does not affect the component functionality, the model is run a second time for the same input text. This implies that the LLM parameters are reloaded from external memory and previous results, or intermediate values are removed. So unless there is a second soft error (which is not likely as soft errors are rare events), the occurrence of two errors on consecutive runs is extremely unlikely, i.e., the second execution is usually error-free. Therefore, if CLED detects an error on the second run, it is assumed that it is a false positive and the text generated is thus assumed to be correct; this is like in [17].

The overheads introduced by the proposed scheme are thus from the extraction of the linguistic features, the concurrent classifier and the re-computations performed for false positives. Given the large size of the LLMs, the relative burden of the linguistic features and the concurrent classifiers is negligible, so the main overhead is introduced by the re-computations induced by the false positives.

The main elements of the proposed CLED scheme are the linguistic features and the concurrent classifier, both are briefly discussed in the next subsections.

Languages have spelling rules that restrict the possible characteristics of a valid text. In English, for example, the same consonant cannot be tripled, digraphs such as ‘uu’ and ‘aa’ are extremely rare, and ‘ii’ never occurs [24] (pp. 132 and 449). Similarly, in Spanish, only six consonants can appear in a word’s final position [25], and in German, sequences of fricatives (such as /vz/) are not allowed [26]. These rules can potentially be used to detect invalid texts that can be a sign of an error, because the models have been trained to produce valid texts. This seems to be confirmed by looking at the examples of texts generated by the models with an error shown in Tables II and III, invalid words are provided as well as character sequences in many cases.

The presence of invalid words is not the only linguistic information that can be used to detect errors. There are also patterns that valid texts follow, for example, the distribution of the part of speech words [27] is typically within some ranges, and not all words are for example nouns or verbs. These types of patterns can also be used to detect anomalous text caused by errors. Some works have proposed grammaticality when analyzing sentences to detect errors; for example, they extract the grammatical category of words with Part-of-Speech (POS) techniques and train models using the sequences of grammatical categories as features [28]. Other works demonstrate that the frequency of grammatical categories allows training simple classifiers with high accuracy, such as the one developed by Mendhakar and Darshan to differentiate between fiction and non-fiction texts [29]. However, these methods are not capable of detecting semantic errors and can classify sentences like “I play basketball” (correct) as well as “I play melon” (incorrect) as valid. This highlights the inadequacy of probabilistic grammatical models to detect semantic incongruencies, a problem already described by linguists [30].

Therefore, based on the previous discussion, two types of features are used as inputs to the concurrent classifier of the proposed CLED scheme: linguistic rules and linguistic patterns. The first type of feature checks the restrictions that are defined by the language. The second type of feature captures patterns that are common in the language, for example, a text in which the relative frequency of consonants is ten times larger than the number of vowels is also likely to be invalid due to an error. These features used can be adapted to better capture the impact of errors on the output text; some examples of features are:

1. 1.

   Restriction: number of capital letters in the middle of words.

2. 2.

   Restriction: number of sequences of the same letter repeating three or more times.

3. 3.

   Pattern: average word length.

4. 4.

   Pattern: frequency of vowels or consonants.

5. 5.

   Pattern: distribution of grammatical categories (i.e., verbs, nouns, determiners, conjunctions.).

Ideally, a common set of features can be used for different LLMs and applications, but depending on the performance of the classifier, the features to use may have to be adjusted to better capture the error patterns caused by the soft error in a specific LLM and application. In the proposed design, the same set of features is used for both models (OPUS-MT and T5) and applications (translation and summarization) to illustrate the general applicability of the linguistic features. This facilitates the protection of LLMs because it makes feature extraction independent of the model application.

Table IV shows the linguistic features used for the concurrent classifier. The first three are rules as they correspond to letter sequences that are invalid while the rest of the features correspond to patterns that are expected to be within a range of values in valid texts. As discussed before, these features will be used both for OPUS-MT and T5 to illustrate that a common set of features can be used for different models and applications.

As the number of linguistic features is small and they are also simple, a basic binary classifier should be sufficient to detect the errors. In the proposed design, a Random Forest classifier³³3https://scikit-learn.org/1.3/modules/generated/sklearn.ensemble.RandomForestClassifier.html is used as in [17]. Different combinations of models and features have been tested. The model’s hyperparameters have been configured using cross-validation and recall as the metric to evaluate the model to reduce false negatives. This process has been performed independently for OPUS-MT and T5; the same set of linguistic features and the same type of classifier (a Random Forest) is used for both models but trained with the relevant dataset in each case.

The T5 model is used to summarize news taken from the CNN Daily Mail dataset⁴⁴4CNN Daily Mail dataset: https://huggingface.co/datasets/cnn_dailymail/viewer/3.0.0/test. First, a subset of 11,490 news from such dataset has been selected and run to generate error-free summaries. Then an error has been injected in a randomly selected bit of a randomly selected parameter of the T5 model and the summaries for the same 11,490 news have been generated. If at least one of them was different from the error-free summary, then the error was considered relevant, otherwise if the error had no effect on any of the summaries, the error is assumed to be irrelevant⁵⁵5This is commonly the case when the error changes one of the least significant bits of a parameter.. The process has been repeated until 100 relevant errors were identified. Then a dataset has been built with the 11,490 error-free summaries and 11,490 erroneous summaries from the 100 relevant errors. The dataset is available online⁶⁶6Single-bit error dataset: https://doi.org/10.5281/zenodo.10647624. These error-free and erroneous texts are used as the dataset on which the concurrent classifier is trained and evaluated. The 22,980 samples are split, with 80% allocated for training, and 20% for testing.

Different classifiers with multiple hyperparameters configurations have been tested. The best model has been achieved with a Random Forest comprised of 50 trees maximum depth of 10 levels for each tree, with the square root of the number of features as the maximum number of features considered for each split, gini as the impurity criterion for splitting, with bootstrapping of samples, a minimum of 2 samples per leaf, and requiring at least 10 samples for a node split. Table V shows the resulting hyperparameters used to train a model that fits the error detection problem, with the mentioned features, and that avoids overfitting. The results on the test dataset reveal an accuracy of 93% and a recall of 93% with a false negative rate of 11% and a false positive rate of 2%.

The results are summarized in Fig. 9. A change of the decision threshold of the model allows for control of the overhead of re-computation, i.e., the unnecessary re-computations run when an error has been detected but the prediction was wrong (measured by the false positive rate). Moreover, it demonstrates the influence of this re-computation rate on error non-detection, quantified as false negatives. The results show that most of the errors, close to 90%, can be detected even with a very low, for example, 1%, recomputation overhead. The overhead of detecting additional errors grows significantly and to detect an additional 4% of the errors, the recomputations increase by more than 5 times. Therefore, the designer can trade-off detection effectiveness for recomputations depending on the system requirements and restrictions. Interestingly, since this change only implies changing the decision threshold, the same LLM can be run with different levels of protection using the proposed CLED. For example, when used to translate medical texts, we can allow for more than 20% recomputations to prioritize error detection than when used to translate news (at 1% overhead).

The relative complexity of the main model and the concurrent classifier is evaluated in terms of their average run time per sample in the dataset. The results are summarized in Table VI where the times for the concurrent classifier include the linguistic feature extraction and the Random Forest classifier. The overhead for concurrent error detection is small even when taking into account that the main model was run on a GPU (NVIDIA GeForce RTX 4080) and the concurrent classifier on a standard CPU (Intel Xeon E5-2630 v2 @2.6ghz and 32GB RAM). The runtime for the concurrent classifier is at least one order of magnitude below the main model. It is interesting to note that the complexity of the main model increases continuously, for example, most state-of-the-art LLMs have at least several billion parameters, so significantly more than OPUS-MT and T5. Instead, the complexity of the proposed CLED is independent of the LLM size as it operates directly on the output text. Therefore, the relative overhead of the concurrent classifier will be even lower for more advanced LLMs.

As a second case study, a different model (OPUS-MT) and application (translation) has been considered. The IWSLT17 translation dataset [23] has been used to select 8,549 samples of Chinese-to-English translation and OPUS-MT has been first run with no errors. Then the same process as for T5 was performed to identify 100 relevant errors and build a dataset with 8,459 error-free and 8,459 erroneous samples. The 17,098 samples are split, with 80% allocated for training, and 20% for testing. Again, the dataset is published online⁷⁷7Single-bit error dataset: https://doi.org/10.5281/zenodo.10647624.

Similarly to T5, a Random Forest binary classifier has been trained using the same features (Table IV) and hyperparameter configuration. As a result, a model with an accuracy of 95% and a recall of 95% was obtained, with a false negative rate of 9% and a false positive rate of 1%. Fig. 9 shows the error detection rate for different overhead rates (false positives). The results demonstrate that the use of Random Forest and the selected features enable the training of a concurrent classifier for error detection with a high accuracy and low false negative rate for different LLMs and applications. This confirms our intuition that the linguistic features can to some extent be general and used for different LLMs. As previously discussed, this makes CLED more attractive as the feature extraction and classifier can be the same and when changing LLM or application, the designer only needs to train the concurrent classifier with the relevant dataset to achieve an effective protection.