Evolutionary optimization of contexts for phonetic correction in speech recognition systems

Automatic speech recognition (ASR) systems are of great relevance in academic and business environments due to the ease of interaction they offer. There has been a growing interest in investigating these systems, which have migrated from probabilistic models to deep neural network systems [5] that have become the standard for professional audio-to-text transformation applications. Deep neural network systems for audio-to-text transformation often use an acoustic model to perform recognition at a first level and are later passed to language models for correction [9]. Commercial services generally operate as a black box, making it difficult for the user to modify language models.

While ASRs generally perform well, they often run into problems when used to recognize specific language domains, so post-processing techniques become relevant [4].

Many of the post-processing and correction tasks of these systems use a context, understood as a set of words, phrases, and expressions related to the particular domain that it is desired to recognize. Some of them have mechanisms to provide a context with which they improve the recognition of certain words and phrases. However, in many cases, it is not enough to significantly improve their performance.

Two particularly interesting topics are the generation of contexts and the phonetic representation for correction. Research has been carried out in this regard; however, there is a lack of experimentation related to the optimal context generation and phonetic representation’s joint operation.

This article presents a method for generating contexts using genetic algorithms to correct the output of the Google speech-to-text processing system. Next, the error correction process’s comparison of different critical strategies is carried out: representation of the sentence to be corrected, candidate selection, and comparison metrics.

The article is structured as follows: Section 2 describes the background to the problem and related work; Section 3 presents the methodology used for the investigation; In section 4, the experimental work carried out is described, the results of which are shown in section 5 and finally in section 6 the conclusions are provided along with some ideas to develop as future work.

The error correction algorithms in ASR systems have been approached from different perspectives, including phonetics. Kondrak [11] proposes an algorithm to calculate a metric of phonetic similarity between segments using multivalued articulatory phonetic characteristics. The Kondrak algorithm combines sets of edit operations and local and semi-global alignment models to calculate a set of near-optimal alignments.

Pucher et al. [16] present word confusion matrices using different measures of phonetic distance. The metrics presented are based on the minimum editing distance between phonetic transcriptions and the distances between hidden Markov models. His research shows a correlation between edit distance and word confusion in ASR systems, so these types of corrections become useful for rectifying recognition errors.

In [2] the problem of using the editing distance to compare strings in languages like Korean, where characters represent syllables instead of letters, is highlighted. This is reflected in the fact that substituting one syllable for another provides the same value regardless of the difference between its letters. The traditional solution uses hybrid metrics between characters and syllables; however, the authors argue that this approach does not satisfactorily solve the problem, so they propose an editing distance based on phonemes as a solution.

Droppo and Acero [9] use the phonetic editing distance to incorporate a third element of correction to ASR systems. They incorporate this distance to learn the relative probability of phonetic recognition strings, given an expected pronunciation. This strategy considers the context of the transcripts, changing the probability of correction depending on the words before and after.

Bassil and Semaan [4] employ a post-processing strategy for error correction in ASR systems. The presented method for detecting word errors uses a candidate generation algorithm and a context-sensitive error correction algorithm. The authors report a significant reduction in system errors.

In [7] phonetic correction strategies are used to correct the errors generated by an ASR system. In the cited work, the system’s transcript is transformed into a representation in International Phonetic Alphabet (IPA) format. A sliding window algorithm is used to select candidate phrases for correction according to the words provided, in context and distance to its phonetic representation in IPA format. The authors report an improvement in 30  % of the phrases recognized by the Google service.

An important component for the phonetic correction algorithm is the context used to construct the candidate phrases, so solutions capable of finding optimal configurations among vast search spaces are needed.

Genetic algorithms are stochastic search algorithms based on biological evolution principles and emulate the process through genetic operators applying recombination, mutation, and natural selection in a population [13, 18]. They have been applied to solve complex combinatorial problems, and the results show that they constitute a powerful and efficient strategy when used correctly. [18].

These types of algorithms have been used to analyze a large number of problems, including knapsack [18], process scheduling problems, the traveling salesperson [13], search for functions for symbolic regression [1], Gaussian kernel functions for sentiment analysis [17], among others.

When using genetic algorithms to solve a problem, possible solutions are expressed as a chain of symbols called a chromosome, where each symbol is a gene. From an initial generation of individuals, the processes of selection, mutation, recombination, and evaluation are iteratively executed, combining the individuals’ genes to produce new variations. Each individual is evaluated according to a function called fitness that describes how well they perform to solve the problem.

The correction algorithm uses the components of a context (set of words and phrases belonging to the application domain) to detect possible errors in the recognition and correct the transcript of an automatic speech recognition system. It comprises three main elements: phonetic representation, candidate phrase generator for correction, and editing distance metric. As a measure of evaluation of the results, the WER metric (Word Error Rate) was used, which is defined as follows:

where $S$ is the number of substitutions, $D$ the number of deletions, $I$ the number of insertions required to transform the hypothetical phrase into the actual phrase, and $N$ the number of words in the actual phrase.

The present work experiments were carried out using 451 phrases transcribed by the Google speech recognition system. This same corpus was used in [7], where details of the said corpus’ collection process and format can also be obtained.

Fig.  1 (a) shows the variation in the average WER obtained in the experiments grouped by the representation mode with different distance thresholds. The horizontal lines correspond to the WER obtained with the transcription of the basic STT (33.7 %) and the contextual STT (31.1 %). The effect of reducing the WER is observed when transforming the plain text into IPA, especially around the value 0.4 for the threshold, where a minimum average WER of 27.8 % is reached. On the other hand, the representation in DM produced similar results to the other methods for small values of the threshold, however around 0.3, its corrective capacity decreases consistently. The DMV version sustains its performance on par with plain text up to a threshold of 0.4.

In Fig.  1 (b), the average WER obtained by the different experimental configurations grouped by the candidate phrase generation algorithm is observed. The graph shows a better performance of the incremental comparison variants, either by size in letters or syllables, compared to the pivot window. The minimum average WER (29.4 %) is reached with a threshold of 0.4 for the LET method. The SYL version shows very similar results; however, the computational cost of processing is higher.

The best results were obtained with the IPA representation configuration and the LET candidate selection method. The WER obtained from the basic transcript decreased from 33.7 % to 28.1 %, and for the contextual transcript, it decreased from 31.1 % to 27.3 %. This configuration presented a global reduction in the relative WER of 19.3 %. Concerning the three distance metrics evaluated, the difference in the results was practically nil.

The results obtained from the experimentation with the genetic algorithms for the optimization of the context using the experimental configuration described in the section 4.2, indicate that an average error was obtained in the fifth execution of the experiment of 26.5 % , which started with an average error of 31.2 % and decreased to an average of 24.9 % in generation 100.

In Fig.  2 the average errors of the 100 generations in the fifth run of the experiment are shown. The best context found with this strategy contained 64 unigrams and 117 bigrams with a total WER of 24.7 %.

In the final phase of the experimentation described in section 4.3, two baselines are obtained to execute the correction process. The WER obtained when comparing the actual sentences spoken with the basic transcription was 32.0 %, while incorporating the genetically generated context, the WER was considerably reduced to 23.2 %. This result allows us to see the impact that an optimized context has on the language model used by Google by reducing the relative WER by 27.3 %.

Fig.  3 shows the correction algorithm results using the IPA representation with the context $C_{m}$. Starting from the basic STT, the minimum WER is obtained with a threshold of 0.4 and the LET selection process. With this configuration, the total WER is reduced from 32.0 % to 26.6 %, representing a reduction of 16.9 % in the WER relative. When starting the correction process from the transcription of the contextual STT, the WER decreased from 23.2 % to 21.0 %, reaching a 9.5 % improvement in the relative WER.

Similarly, Fig.  4 shows results using the genetically generated context $C_{g}$ as input to the correction algorithm. Using this context, a reduction of the minimum WER when using the WIN candidate generation procedure is observed. However, it does not seem to generate good results with the LET method. The absolute minimum WER for the basic STT is 25.3 %, thus reducing the relative WER of 21.0 %. Starting from the contextual STT, a minimum of 20.1 % is reached, representing a reduction of 13.6 % in relative WER.

From the results obtained in the experimentation, the phonetic correction algorithm’s usefulness is shown to reduce errors in the transcription of Google, both in its basic and contextual versions. It is observed that the best configuration for the algorithm is obtained using IPA as phonetic representation and incremental selection by letters, managing to reduce the relative WER by 19.0 %.

Similarly, we can mention that genetic algorithms are an efficient alternative for generating contexts since they managed to reduce the WER of Google basic transcription from 32.0 % to 23.2 %. The context was shown to have a crucial value in the performance of the algorithm.

The best results were obtained from the combination of the phonetic correction with the evolutionary optimization of the context, achieving a reduction of the absolute WER of 11.9 % by decreasing it from 32.0 % to 20.1 %, representing an improvement in relative WER of 37.2 %.

The fact that both the phonetic correction algorithm and the evolutionary context optimization are independent of the system used for the transcription and application domain means that the strategy presented can be extended to different ASR systems and application domains.

The algorithms presented throughout this article can take advantage of a priori knowledge of the application domain to mitigate the cold start problem. The above is because if the initial transcripts are not available, a context generated with human knowledge of the domain can be used, as in [7], which can be complemented with genetic algorithms as information is collected about interactions with actual users of the system.

Among future research lines, it is necessary to validate the results with a corpus of different application domains; in addition, experimentation using weighted editing costs considers phonetic characteristics of Spanish and the original audio such as noise, duration, the energy of the signal, among others. Another line of research is the comparison with deep learning algorithms since the problem of error correction in ASR systems can be considered a translation of erroneous transcripts to correct transcripts, so algorithms Machine translation can be helpful.