Improving callsign recognition with air-surveillance data in air-traffic communication

Communication between pilots and Air-Traffic Controllers (ATCo) is mostly voice-based in order to reduce possible distraction of pilots during the flight and to accelerate information exchange. The voice-based approach, however, makes communication potentially more error-prone because of the amount and speed of information transmission, environmental noise and variability of accents due to the international setting. These factors make the task extremely stressful, pilots and ATCos must always be highly concentrated to guarantee a high level of accuracy to provide safety.

Technical support of pilot-controller communication can increase the level of confidence of correct command perception and reduce the workload. Recent progress in Automatic Speech Recognition (ASR) including online recognition of continuous speech allows new perspectives in improving communication methods between pilots and ATCos [1].

A possible direction to improve the recognition quality is using contextual information, e.g. surveillance data. One of the key parts of ATCo commands is a callsign — a unique identifier for aircraft, of which the first part is an abbreviation of airline name and the last part is a flight number that contains a digit combination and may also incorporate an additional character combination, e.g. TVS84J. The callsign data comes from the radar in a compressed form (first column in Table 1), so before using the data for ASR, all callsigns are to be expanded to word sequences (second column in Table 1). The compressed form often allows more than one possible realisation in the ATCos’ speech: For example, DLH5KX can be expanded as ‘hansa five kilo x-ray’ or ‘lufthansa five kilo x-ray’. Since we can not say which particular expansion is true for an utterance callsign, it is important to take all expansion variants into account.

At a certain time point, only few aircraft are usually in the radar zone which means only a limited number of callsigns can be referred to in the ATCo communications. If a recognised callsign does not match any callsign registered by radar at the same time point, it means that there is no corresponding aircraft in the air space and the recognised command is invalid. In the proposed methods, we dynamically introduce surveillance information to increase the probability of recognising those callsigns which are present in the air space at the moment of utterance. In general terms, this research is centered on integrating contextual knowledge during ASR decoding. Nevertheless, previous research has shown that this technique could be also integrated during semi-supervised learning i. e., contextual semi-supervised learning [2]. The rest of the paper is organised as follows: first, we overview previous research done on the n-gram boosting; then, we give some theoretical background and describe two investigated approaches in more details; then, we present the data and the experiment set up; finally, we report the results and summarise our observations and ideas in the discussion.

As a callsign is a sequence of words, using contextual information to improve recognition of callsigns is a task of boosting certain n-grams. When information about currently registered callsigns is available from the surveillance data, it can be used to increase the probability of those particular callsign n-grams in the recognition.

Using contextual information for ATCo ASR has been already used in some previous studies [3, 4]. In [4], the weighted Levenstein distance is applied as post-processing to overcome the problem of variability of ATCO commands. In [3], a grammar with all semantic concepts of ATC embedded in XML annotation tags is used. After decoding, lattice hypotheses are rescored by adding an additional knowledge source component to the cost function. The knowledge-based rescoring penalises hypotheses which are invalid in the context, e.g. callsigns not registered in the air space. Although this approach helps to considerably increase the recognition accuracy, its limitation is that it deals only with concepts and callsigns which are annotated and included into the grammar. Those n-grams that do not appear in the grammar can not be extracted and evaluated.

In this paper, we propose boosting callsign n-grams by dynamically modifying their weights and adding them, if necessary, in the weighted finite-state transducer (WFST). Similar FST-based approaches with adjusted weights have been applied in different scenarios, for example, improving recognition of named entities (NEs) such as contacts, locations, films, etc. in order to adapt the recognition of voice-driven assistants to users [5, 6, 7]. In [6], the authors use class-based Language Model (LM) and bias towards the whole class in a particular context. In [7], potential named entities are firstly detected and tagged with a class, then, the input word lattice is composed with ‘tagging’ FSTs generated for each utterance with a tagged entity. At the final step, contextually relevant NEs are suggested for given phoneme hypotheses inside the time interval corresponding to a tagged path. Approaches in both papers aim to boost a certain pre-defined semantic class and then to take the best recognition candidate within this class. In the situation with callsigns, however, we do not have different pre-defined classes; a list of callsigns to be boosted varies between utterances and should be updated every time.

The approach proposed in [5], is probably the closest to our lattice rescoring method: a generalised composition is applied to combine baseline LM and n-gram WFSTs generated with the most frequent n-grams from recent news. The main difference is that they perform composition of two FSTs not at the rescoring step but in the first pass decoding as well. As this method achieved good results in recognition of recently frequent in news n-grams, we decided to adapt it for detecting callsign n-grams. In addition to the rescoring method, we also compare the proposed method to the other one involving model modifications.

We address the question of how to boost callsigns by investigating two approaches: 1) grammar modification with boosting n-grams in G.fst and dynamic graph composition, and 2) adjusting n-grams weights with lattice rescoring. The former method involves adaptation of the decoding graph itself, when in the latter approach, modifications are applied to the ASR outputs. We compare two methods to each other, as well as to the combination of both. We show that boosting several probable callsigns improves the recognition of a true one. To learn how well the methods work relatively to the ideal scenario, when the correct callsign is known, results of boosting surveillance callsigns are also compared to those achieved with boosting only the ground truth callsign, in our case available from the manual transcriptions.

In a hybrid ASR system the different knowledge sources are represented as WFSTs, which are composed together in the final decoding graph [8]:

$$HCLG=H\circ C\circ L\circ G$$

(1)

where $H,C,L$ and $G$ represent the Hidden Markov Models, context dependency, lexicon and LM (AKA grammar) correspondingly.

The operation of composition maps sequences from input transducers in a way that an output from one transducer should be an input to another one, which allows combining different levels of representation. In other words, with composition one can integrate into a system information from additional knowledge sources.

The complete $HCLG$ graph, however, is very large and complicated and its modification, ideally in real-time, is difficult. Modifying the $G$ is an easier and more flexible approach. The G-boosting method is possible because of new techniques that allow for on-the-fly composition of the $HCL$ and a modified $G$ [9, 10]. For all operations on WFSTs we use the wrapper released as part of [11] in GitHub¹¹1https://github.com/idiap/icassp-oov-recognition.

In Table 3, the results of the experiments are presented with utterance WER, callsign WER and accuracy of callsign recognition. Combination of both methods achieves the best callsign recognition results for all test sets. G-boosting and lattice rescoring together help to gain up to 4.6% of absolute (i.e. from 28.0 to 23.4%), or 16.5% of relative improvement of utterance WER, as well as up to 28.4% of absolute improvement in the accuracy of callsigns recognition (see Table 3). Table 4 gives some examples of improvement where airlines names and callsign word sequences are detected correctly comparing to the baseline predictions.

As previously discussed, the surveillance data typically provides information about a few callsigns registered in the air space at a certain timestamp. In order to compare our results to the same setup tested with only a ground truth callsign for each utterance, along with using FSTs biased to all registered callsigns, we repeated the experiments with FSTs biased to a single ground truth callsign. As information about the only correct callsign is not available in the reality, this is considered as an oracle (upper bound) performance. Even if the ground truth scores are always noticeably better, systems boosted for all surveillance data callsigns achieve comparable results.

The number of callsigns extracted from radar and used to boost the ASR system per utterance can affect the performance of the methods because by boosting more n-grams it is easier for the system to pick the wrong one in the end. We investigated this effect. The test sets have different numbers of boosted n-grams, from 5 to 29 (see Table 1), but even with many boosted callsigns the recognition accuracy goes considerably up comparing to the baseline.

Our methods also demonstrate consistent results for data of different quality. The level of noise in the recordings of LiveATC and Malorca test sets are very different, as well as the WERs achieved by their baseline systems. Nevertheless, we see considerable improvement for all test sets and the general tendency stays the same.

In addition to lattice rescoring similar to the one described in [5], we also compare it to adjusting weights inside of a grammar and composing it after with the rest of the graph in a dynamic mode. This method is convenient because it does not require an additional step of rescoring and the generated hypotheses already include biased n-grams. The main inconvenience of G-boosting approach is increasing memory consumption. One has to create a new modified G.fst every time when new information is available. At the same time, the old G.fst is usually kept for the next updates. That doubles the memory usage.

N-gram adaptation language modelling typically involves interpolation of in-domain and out-domain LMs. The strong side of the proposed methods is their flexibility. They adapt current contextual information to improve the recognition of target n-grams without any additional in-domain LMs. This allows using new contextual data per each particular utterance.

We investigated two methods of integrating contextual radar data in order to dynamically improve the recognition of callsigns per utterance. The first approach involves modifying weights for callsign n-grams in the grammar component of the decoding graph ($G$) followed by the dynamic lookahead decoding. In the second approach, callsign n-grams are boosted by composing the decoded lattices (lattices generated in the first pass decoding and converted to WFSTs) and contextual WFSTs biased to the callsign sequences registered in the surveillance data.

The best results are achieved with the combination of both methods. The same tendency of improvement is noticeable in all test sets and in the recordings of both lower and higher quality with the relative improvement of callsign recognition varying from 45.2 to 74.2% depending on the test set.

Introduction of contextual information considerably improves recognition of callsigns and, therefore, recognition of ATCo commands. As a noisy environment leading to lower recognition accuracy is often a reality in pilot-ATCo communication, the proposed methods and their combination will definitely improve the recognition of the key information in ATCo speech.