Automatic Speech Recognition Benchmark for Air-Traffic Communications

The communication methods between pilots and Air-Traffic Controllers (ATCos) have remained almost unchanged for many decades, where the ATCo’s main task is to transfer spoken guidance to pilots during all flight phases (e.g. approach, landing or taxi) and at the same time providing safety, reliability and efficiency. This task has shown to be extremely stressful and highly voice demanding because of the impact a small mistake can make. Several attempts towards increasing the confidence and reducing the workload of pilot-controller communication have been pursued in the past, including experiments with Automatic Speech Recognition (ASR). Initially, due to budget and scarcity of computing power, previous work targeted isolated word recognition, or ‘voice activity detection’ but currently most of the works performs ASR on nearly real time. Military applications were one of the first attempts involving engines for command-related ASR; Beek et al. [1] contrast the main ASR techniques with its relevance to military applications such as speaker verification, commands recognition and system control of aircraft. They remarked that pilot-ATCo communications have a very limited word set -vocabulary-, speaker-dependent issues and environmental noises that need to be addressed to produce a sufficiently-reliable system. Initially, the integration of ASR technologies in ATCo started in the late 80s’ with Hamel et al. report [2]; but lately, ASR technologies has been successfully deployed on ATC training simulators. For example, Matrouf et al. [3] proposed a user-friendly and robust system to train ATCos based on hierarchical frames and history of dialogues -context-dependent-. Similarly, DLR [3], MITRE [4] and more recently UPM-AENA [5] under the INVOCA project proposed akin training systems.

One of the current limitations in developing highly-accurate ASR engines for ATCo communications is the lack of available databases; likewise, generate the transcriptions of such data is extremely costly. Typically a raw ATCo-pilot voice communication recording of one hour -including silences- requires between eight to ten man-hours of transcription effort [6] (mainly as it requires highly trained participants, often active or retired ATCos). Afterwards, usually only 10 to 15 minutes speech segments of ATCo is obtained from 1h recording (after removing silence segments). Hence, it would take approximately one man-week work to get an hour of ATCos without silences [5, 6].

Currently, several researchers [7] and the International Civil Aviation Organization (ICAO) determined that the air-traffic is expected to grow about 3 to 6 percent yearly at least until 2025. Consequently, it has been seen a huge investment of the European Union (EU) to address the ATCos workload and development of ASR engines for field pilot-ATCos communication, but not only for training purposes. Three recent projects financed by the EU on the scope of ASR for ATCo communications are MALORCA¹¹1MAchine Learning Of speech Recognition models for Controller Assistance, http://www.malorca-project.de/wp/, ATCO2²²2AuTomatic COllection and processing of voice data from Air-Traffic COmmunications, https://www.atco2.org/ and HAAWAII³³3Highly Automated Air traffic controller Workstations with Artificial Intelligence Integration. MALORCA project (together with AcListant⁴⁴4Active Listening Assistant, www.AcListant.de) demonstrated that ASR tools can reduce ATCos workload [8] and increase the efficiency [9], where also it addressed the lack of transcribed air traffic speech data using semi-supervised training (similar to other ASR related tasks such as speech recognition applied for under-resourced languages [10, 11]) to decrease Word Error Rates (WER) and command error rates [12, 13]. We set as baseline word error rates the results from [12, 13] for two proposed train/test sets. ATCO2 ongoing project aims at developing a unique platform to collect, organize and pre-process air-traffic speech data from air space available either directly through publicly accessible radio frequency channels (such as LiveATC [14]), or indirectly from ANSPs. One of the current challenges of ASR engines for ATCo communications is the changing ATCos accent and vocabularies across different airports; hence, ATCO2 will develop a robust methodology capable of minimize their impact on the system. In this work, we present the first results based on six ATC in-domain databases which, to the authors’ knowledge, is the first time that such quantity of command-related databases (spanning more than 170 hours) have been used during the training phase. First, we explore transfer learning from a Deep Neural Network (DNN) system trained on an Out-Of-Domain (OOD) corpus, then we contrast the results with the state-of-the-art ASR chain recipes (from Kaldi’s toolkit [15]) such as TDNNF and CNN+TDNNF. Secondly, given the high likelihood of Out-Of-Vocabulary (OOV) words due to the intrinsic changing behaviour of the air-space, we tested Byte-Pair Encoding (BPE) based system, which stands as a technique capable of recognise OOV words.

Even though obtaining a full ATCo-pilot communication system goes far beyond of only ASR tasks, we convey in this study a benchmark of experiments going from transfer learning (from an OOD corpus) and adaptation with partial or complete in-domain command-related databases to BPE algorithms and end-to-end TDNNF models. Additionally, this study does not plan to describe in detail intrinsic characteristics of the ATCos communication such as noise (e.g. cockpit noise), speaker’s accent, ontology or transcription procedure; but rather, we show technical details and results about ASR in the ATM area. In Section 2 we define the corpus and data preparation used in the proposed benchmark experiments. Section 3 reviews the experimental setup such as lexicon, language and acoustic modelling. Then, Section 4 presents the main results. Finally, Section 5 concludes the paper and proposes the roadmap that ASR systems for ATCo communications should head.

Diverse studies conclude that almost 80% of all pilot radio messages contain at least one error and 30% of the incidents are accounted by miss-communications (and up to 50% in the terminal manoeuvring area) [16]. Therefore, ASR stands as a adequate solution, which cannot afford to be trained and tested ‘on-the-fly’ in real operational environment, but it is required to build the best possible system before its deployment. With this intention, we use the state-of-the-art ASR engines that are based on deep neural networks like Time-Delay Neural Networks (TDNN) and Convolutional Neural Network (CNN). These models are known as ‘data-hungry’ algorithms; in fact, state-of-the-art ASR systems need to be trained on large amount of data to achieve and acceptable operational performance. Sadly, there is a lack of such databases in the ATM world. One of our main contribution is to solve this problem employing partly-in-domain or ‘command-related’ databases, which retain similar phraseology and structure.

The results (Table 3) are split into four blocks. First, TDNNF-B is trained on an OOD set consisting of 1500 hours. This is our base model to perform transfer learning. Second, TDNNF-B model is adapted to the different ATC datasets (using TDNNF-B as initialization) i.e. Train1, Train2 and Tr1+Tr2. Third, we compare WERs for TDNNFs without transfer learning. Finally, we present results on a CNN+TDNNF chain model and TDNNF trained with BPE units. We kept the same hyper-parameters across all the experiments in order to make fair comparisons between models. The base model performs poorly on the ATC data. This is not surprising as Librispeech and Commonvoice are both read speech with mostly clear audio. The ATC data is more noisy, the speakers talk much quicker, and the accents are stronger. Despite the significant difference in domains, the pretraining still helps when the target dataset is not too large, as can be seen when comparing the first two rows of the TDNNF-TF-B and the TDNNF models in Table 3 (trained on Train1 or Train2). Note the sometimes large differences in performance of the models trained on either Train1 or Train2 can be explained by whether the accent(s) in the test set were also present in the training set. Once the target domain dataset becomes large enough, we do not see the benefit of pretraining (see the last row of the TDNNF-TF-B and the TDNNF models). The last block of experiments provides a broader cover of different DNN architectures and techniques on our proposed ASR benchmark for ATC communications. There is no clear winner. The CNN+TDNNF system yielded a new baseline of 5% WER for Atcosim, showing a relative improvement on WERs of 16.7% and 3.9% when compared to TDNNF-TF-B and TDNNF. The best model for Vienna dataset was TDNNF trained on Tr1+Tr2 and scored with a 4-gram LM, whereas for Prague it was TDNNF with 6-gram and lexicon based on BPE. Compared to previous experiments on MALORCA [12, 13], our approach yields 29.8% and 37.9% relative WER improvement for Vienna and Prague.

We further investigated why the BPE model does significantly better on the Prague test set, and found that the difference in performance is entirely explained by reduced deletions (five times more deletions in TDNNF and CNN+TDNNF than TDNNF-BPE system). The word-based model is obviously not able to recognize OOV words, which is the primary reason for the deletion errors. The OOV rates on Prague, Vienna, Airbus and Atcosim test sets are 3.3%, 1.1%, 0.0% and 0.1%. This shows that the BPE system is capable of recognizing OOVs and thereby improving performance; although, it does come at a cost since the BPE models also perform significantly worse on some test sets. Further investigation is required to understand the differences in performance between word and sub-word (BPE) based systems. We noticed that the BPE based model performs better on foreign words (even when the word-based model includes these words in its lexicon). We attributed it to the character-based lexicon system, which generalizes better to foreign languages which are not closely related to English. The Atcosim baseline WER is presented in [33], schiving 8.5% absolute WER when performing n-best list re-ranking using syntactic knowledge. In our case, we obtain first 63.4% WER with TDNNF-B and an improvement to 8.1% absolute WER when training only on Train1 set. An additional 10% relative WER improvement can be obtained if employing transfer learning (i.e. TDNNF-TF-B + Train1), reaching 7.3% absolute WER. An additional 28% relative WER improvement is achieved when training TDNNF on Tr1+Tr2, which is a bigger train set. Finally, with the intention to explore different DNN architectures we were able to further reduce the absolute WER to 5.0% when using a CNN+TDNNF system trained on Tr1+Tr2, accounting to 3.8% relative improvement from TDNNF.

Finally, we are currently exploring new hybrid frameworks for acoustic modeling –of ATCo data– such as Pkwrap [34]. Pkwrap ⁵⁵5https://github.com/idiap/pkwrap is a simple wrapper that is useful to train acoustic models in PyTorch using Kaldi’s LF-MMI training framework.

This work introduce state-of-the-art DNN architectures to the area of ASR for air-traffic communications. We tested several DNN architectures, amount of training data and transfer learning across the given experiments in order to reasonably compare their performance. To the author’s knowledge, this is the first study employing six air-traffic command-related databases spanning more than 176 hours of speech data that are strongly related in both, phraseology and structure to ATCos-pilots communications, therefore dealing with the burden of lack of databases that many previous studies have quoted. Specifically, we have shown that using in-domain ATC databases, even if not from the same country/airport, the system is capable to yield a 29.8% and 37.9% relative WER improvement for Vienna and Prague approaches. Also, we reported new baselines for Vienna, Prague and Atcosim test sets. In general, our proposed ASR engine for ATCo speech achieves an averaged WER of 7.75% across four test sets with different accent; the given results suggest that systems trained on cross-accent data helps in the overall system’s performance, rather than limiting the amount of data to single-accent datasets. Finally, one of the main outcomes of this research was the results on byte-pair encoding with Prague approach, reaching 5.0% WER. We advise that future research should further explore BPE units for ATC.