Defining maximum acceptable latency of AI-enhanced CAI tools

Recent years have seen an increasing number of studies around the design of computer-assisted interpreting tools with integrated automatic speech processing and their use by trainees and professional interpreters. This paper discusses the role of system latency of such tools and presents the results of an experiment designed to investigate the maximum system latency that is cognitively acceptable for interpreters working in the simultaneous modality. The results show that interpreters can cope with a system latency of 3 seconds without any major impact in the rendition of the original text, both in terms of accuracy and fluency. This value is above the typical latency of available AI-based CAI tools and paves the way to experiment with larger context-based language models and higher latencies.

Automatic speech recognition (ASR) has been regarded as a technology “with considerable potential for changing the way interpreting is practiced” [23]. In particular, ASR has been proposed as a means to overcome the shortcomings in current implementations of computer-assisted interpreting (CAI) tools, such as the inherent difficulty to manually lookup terms while interpreting. By integrating AI-based features in the interpreter workstation, such as the real-time automatic suggestion of numbers and other problem triggers [13], and by integrating advancements in extractive and predictive algorithms based on machine learning [28], the cognitive load required by the use of CAI tools may be reduced, offering new opportunities for interpreters to improve their performance. First empirical research seems to point in this direction [8, 14].

AI-enhanced CAI tools are generally based on the concatenation of several modules that may comprise a speech-to-text transcription engine, a parsing module to identify the units of interest, for example numbers, terminology or proper names, and a visualisation component for the human-machine interaction. Other implementations may adopt an end-to-end approach, making the cascading architecture obsolete. Such tools can be deployed on the edge, running completely on the user’s device, or on the web, hosted on servers and accessed through a Web Browser. Generally speaking, CAI tools present suggestions with a certain amount of delay with respect to the original speech. This delay is due to the architectural design of the tool and its components, for example the ASR latency or the computation time needed to inference a language model, but also to the intrinsic nature of the task, which may require a certain amount of linguistic context in order to make an informed decision on what to show to the final user. In a time-sensitive activity such as simultaneous interpreting, system latency may require interpreters to adopt new interpreting strategies to successfully integrate the suggestions into their rendition. If latency, however, exceeds a certain threshold, interpreters may not be able to integrate the suggestions, or may do this at the expenses of fluency, cohesion, or even accuracy. In this case, the use of such high-latency tool would become detrimental for the user-machine interaction, for the interpreter’s cognitive load, and finally for the quality of the rendition.

Not much is known about this maximum acceptable latency threshold, i.e. the maximum ear-voice-span (EVS) that interpreters can cope with in order to successfully integrate external suggestions without impacting the overall interpreting performance. Not only this value may vary among subjects, but it may also be different from the one reported in experimental analysis of interpretation performed without the use of a CAI tool.

Our hypothesis is that displaying suggestions within the average EVS reported in literature should not negatively impact the interpreter rendition, even when this is ‘forced’ upon the interpreter by the latency with which the tool generates such suggestions. To our knowledge this is the first study that tries to empirically answer this question. On the one hand, knowledge about this threshold is crucial to understand if current AI-based CAI implementations can already meet interpreters expectations or if major efforts should be placed in further reducing system latency. On the other hand, an acceptable higher latency could allow the integration of more time-demanding NLP features, such as the automatic prediction of difficult parts of the speech [[, eg.]]vogler_lost_2019.

The reminder of this paper is organized as follows. Section 3 introduces the related work in the area of CAI tools and the empirical experiments conducted on them so far. Section 4 describes the data and the methodology adopted in this experiment. Section 5 introduces the evaluation framework. Section 6 presents the results. Finally, section 7 discusses the conclusion and the outlook.

Recent years have seen an increasing number of studies around the design of AI-based computer-assisted interpreting tools and their use by trainees and professional interpreters.

Computer-assisted interpreting tools are digital devices designed to support the interpreter in different steps of their work, from preparation to the very act of interpreting. They have been proposed by several researchers in the past 20 years or so [29, 24, 27], but it is only recently that there has been a surge in interest among the community’s members.

Thanks to new advances in artificial intelligence, ASR has reached a quality level that makes it suitable for integration into supportive technologies. By automating and extending the query system of CAI tools, this integration may solve the shortcomings of traditional tools [[, eg.]]fantinuoli_interpretbank._2016, fantinuoli_computer-assisted_2017, hansen-schirra_nutzbarkeit_2012 and extend the features available in an interpreters’ workstation, for example automatically suggesting translations of specific terms as well as transcribing numbers and proper names in real-time.

Over the years, a handful of empirical studies have been carried out to test the feasibility of the human-machine interaction in the simultaneous modality. They have focused in particular on the effectiveness of ASR-support during the interpretation of numbers [9, 8, 14], one of the problem triggers of simultaneous interpreting identified in literature [4, 15, 17, 26]. In order to measure the impact on the quality of the rendition, these studies have used either mock-up systems with a very short latency [9, 6] or real-life tools with a reported latency of under 2 seconds [8, 14].

From an interpreter perspective, the system latency impacts the ear-voice span (EVS) of interpreters since it forces the interpreter to wait a certain amount of time before being able to integrate the suggestions into their rendition. The EVS is the amount of time that separates the words uttered by the speaker in the source speech and the equivalent rendition in the target speech uttered by the interpreter [23]. Research in this field has a “long-standing tradition” [7] and EVS is considered an essential variable that can potentially impact interpreters’ performance [2, 16, 17, 20] such that it continues to be an object for analysis and assessment.

Previous research has found that the variable limits of interpreters’ EVS are generally attributable to two main factors: input-related factors, which might also enable interpreters to consciously regulate their lag, and personal factors, e.g. short term memory-related factors, that influence the maximum capacity of the memory [26, 1, 19]. Interpreters are free to regulate - at least to a certain degree - their EVS depending on the features of the speech segment they are translating, increasing or decreasing it consciously as a strategy [1, 10].

Measuring EVS is not immediate and easy, due to the fact that grammar, word order and syntactic structure might be different in the two languages [26]. Average EVS reported in literature is between 2 and 3 seconds [2, 19, 20], with a peak of 10 seconds [22]; if measured in words, a mean ranging from 5 to 10 words has been reported [25] .

The analysis of the collected data has been performed on two different levels, namely on a stimuli-based and on a segment-based level.

The aim of the stimuli-based evaluation is to assess the accuracy level achieved by the interpreters in rendering the units of interest in the target language, depending on the variable latency of the mockup system. Within this level of assessment, the accurate rendition of the units of interest is evaluated considering the following components: the numerical information comprising its order of magnitude, and, if present, the relevant referent. A number of parameters were collected for the evaluation: presence of the number in the rendition, accuracy of the number, presence of the referent, accuracy of the referent, pronunciation disfluency. Numbers and referents that were approximated, generalized or omitted were considered to be errors. Moreover, lexical, syntactic, phonological or articulation mistakes were also classified as errors.

The segment-based evaluation assesses the quality of the entire segments in which the numbers are embedded. This evaluation focuses on two different parameters. The first is the accuracy of the target segment, measured in terms of the following linguistic aspects: faithfulness, grammatical correctness, completeness, logical cohesion, consistency, plausibility. The second parameter is the listener’s perception in terms of delivery flow. This evaluation level focuses on the following paralinguistic aspects: perception of the interpreters’ voice and rhythm, eloquence, presentation, prosody and communicative effectiveness. The segment-based evaluation has been performed using a Likert scale (1 to 5) by three different evaluators.

There are three major limitations in this study that could be addressed in future research. Firstly, the level of interpretation proficiency of the participants selected for this experiment is biased towards the lower end. It is reasonable to assume that a randomized sample in terms of proficiency, for example including also professional interpreters, may increase the maximum acceptable latency measured in the experiment. Secondly, the experiment focused only on a single language pair. Because of intrinsic variations among languages, EVS and different strategies adopted by interpreters on the basis of the language combination may influence the threshold under scrutiny. Finally, variations in speech complexity, for example in terms of syntactic structures, delivery speed etc. have not been taken into consideration in our study. However, they may have a considerable impact on interpreters cognitive load and consequently on their ability to successfully integrate suggestions and may require a dynamic adaptation of latency according to this variable.

In this paper we presented the results of an empirical experiment aimed at measuring the maximum acceptable latency of an automatic suggestion feature for simultaneous interpretation. The results seem to suggest that interpreters are able to integrate suggestions by ad-hoc extending their ear-voice-span to 2 seconds without compromising the quality of their rendition and to 3 seconds without any major disruption. A further extension of the system latency seems to induce a consistent reduction of precision in the use of such suggestions and in the emergence of information losses in the overall rendition. This is in line with our original hypothesis that the system latency should not exceed the average interpreter’s EVS. Within the latency threshold outlined in the experiment, next generation AI-enhanced CAI tools could be able to accommodate more complex and context-based NLP-features without a significant risk of impairing the usability of the tool.