Show Me the World in My Language: Establishing the First Baseline for
Scene-Text to Scene-Text Translation

Machine Translation has shown remarkable growth in the last few years, partly attributed to the adoption of neural models [2, 30, 6, 34, 7]. In parallel, substantial advancements have also been made in speech-to-speech translation [24, 11, 17, 12] where the goal is to develop systems that are capable of accurately interpreting spoken language in one dialect and seamlessly translating it into another while preserving the voice of the original speaker, thus enabling effective cross-lingual communication in real-time. Drawing inspiration from these research directions, we present an analogous problem in the scene text domain, namely “scene-text to scene-text translation” or, in short, “visual translation”. The visual translation task aims to translate text present in images from a source language to the target language while preserving the visual characteristics of the text and background as illustrated in Fig. 1. Visual Translation has extensive applications, e.g., transforming the travel experience by allowing tourists to instantly understand sign boards in foreign languages and enabling seamless interaction with the visual world without language barriers.

By drawing parallels with the speech-to-speech translation approaches, which comprise three components: automatic speech recognition (ASR), text-to-text machine translation (MT), and text-to-speech (TTS) synthesis, we propose a visual translation baseline that integrates scene-text recognition (STR), text-to-text machine translation (MT), and scene-text synthesis (STS). This cascaded system offers practical advantages over an end-to-end approach, as fully supervised end-to-end training necessitates a substantial collection of source and target scene text pairs, which can be challenging to obtain compared to parallel text pairs for MT or image-text pairs for STR. As STR and MT models are extensively explored in the literature, and several off-the-shelf methods are available, we prioritize enhancing the performance of the STS model. To this end, we extend a popular SRNet architecture [31] by decoupling background and foreground generation. For background generation, we employ a diffusion-based model using ControlNet [33] to generate a text-erased image from an input containing scene texts. We further modify SRNet so that it only focuses on foreground generation on a plain background. Once foreground and background are independently generated, we blend them into the scene image. To improve the quality of visual translation further, we propose a set of design enhancements such as using regular expressions to filter special strings, grouping words and translating them together, and a planning strategy to blend the translated text in the scene image appropriately.

We extensively evaluate the proposed baselines for Hindi-to-English and English-to-Hindi visual translation using our new automatic and user evaluation metrics. While the baselines show promising results, the problem remains far from solved and requires further research.

We make the following contributions: (i) We study the under-explored task of visual translation that aims to translate text in images to a target language while preserving its font, style, position, and background. To the best of our knowledge, the comprehensive study of this problem has largely been unexplored in the existing literature. (ii) We introduce a generic cascaded approach for visual translation, and we design a set of baselines using state-of-the-art approaches for scene text recognition, machine translation, and scene text synthesis and their task-specific design enhancements. (iii) Training a visual translation model with real-world images is challenging due to the lack of large-scale paired scene text images in different languages. Therefore, we use synthetic images for training. We present a method to generate paired images with words sharing the same visual properties, creating VT-Syn, a synthetic dataset of 600K paired visually diverse English-Hindi scene-text images. To evaluate performance on real images, we provide extensive annotations of translated text from three users. These benchmark datasets will support future research in visual translation. (iv) Due to the lack of principled evaluation metrics for visual translation tasks in the literature, we propose a set of automatic and user evaluation metrics. We believe these metrics will help track the progress of visual translation tasks effectively.

Machine Translation: It is a well-studied area [5, 10, 26, 9, 30, 6] that aims to convert a text from its source language to a target language. Current state-of-the-art models for machine translation are deep-learning based [9, 30, 6]. In the speech domain, Speech-to-Speech Translation (S2ST) aims to translate speech from one language to another while preserving the speaker’s voice and accent [11, 17, 12]. Inspired by these works, we focus on text translation in the visual modality, which brings newer research challenges, such as preserving font properties and integrity of the image background, which need to be addressed to produce visually appealing translations.

Translation of Text in Images: Recent years have seen growing interest in translating text within images, both in research and commercial domains. Current works primarily focus on recognition and translation methods for scene text [20, 16], prioritizing accurate translation without addressing visually consistent text generation. A popular commercial product – Google lens ¹¹1https://lens.google/#translate also falls into this category. These approaches often resort to simply overlaying translated text on source images. While some studies explore end-to-end methods for text translation in images that generate text directly in pixel space [22, 15], they typically deal with limited visual diversity in document-style images with plain backgrounds and fixed fonts – without tackling the complexities of scene text that we aim to address. The closest solution to our problem is Google Translate for images²²2https://translate.google.com/?op=images, a commercial product for visual translation of diverse scene text. However, its underlying technology remains proprietary and closed-source. We emphasize the need for the research community to study this problem openly, establish proper open-source solutions, create a benchmark, and define evaluation criteria – goals we pursue in this paper. Moreover, we observe that Google Translate still lacks translation quality and often fails to produce visually consistent results for complex cases, underlining the potential for better approaches. In Section 6.2, we provide a qualitative comparison between our work and Google Translate.

The task of Visual Translation can be reduced to a sequence of sub-tasks: locating and reading text in scene images, translating the text into the target language, and generating the final image containing the translated scene text. Motivated by this observation, we propose a cascaded approach to visual translation by combining models for (i) scene text recognition, (ii) machine translation, (iii) scene-text synthesis, and (iv) seamlessly blending the generated scene text into the image. These sub-tasks are well-explored independently in computer vision literature; thus, we benefit from the availability of trained models. Further, such an approach can perform generation at the word or phrase level, which can help preserve the consistency of non-text regions in the image. The outline of our cascaded baseline is illustrated in Fig. 2 and a detailed illustration is provided in Fig. 3. We describe each module in detail in the following sections.

Training and evaluation of a visual translation baseline require real-world images in the form of $(I,I^{\prime})$ where $I$ and $I^{\prime}$ are visually identical images containing corresponding scene text in two different languages with matching font and style. However, such instances are not easily available in the real world. We mitigate this data scarcity challenge by directly synthesizing the desired data: we generate paired scene-text images that are (i) identical in the image background and (ii) matching in font and style. A few examples are shown in Fig. 4.

In generating synthetic samples, we use a large corpus of words in both languages, as well as a diverse collection of fonts. To simulate real-world scene text, we also render the images on natural backgrounds, as well as vary the orientation, positioning, and size of the scene text in images. A more detailed procedure for generating the synthetic data is provided in Section 4.1.

Popularly, scene-text synthesis methods aim to generate this image using a single end-to-end trained model [31, 27, 14, 18]. However, in our empirical results, we consistently observed two frequent limitations with such methods: (i) incomplete erasure of source text in the generated image and (ii) unintended alterations to the background (non-text) regions. These errors often resulted in a "patchy" effect, among other unnatural artifacts, when the generated images were integrated into the full scene.

To address these issues, we propose a decoupled approach to scene-text synthesis, consisting of three independent steps: background generation and foreground generation, followed by a composition step. We collectively term this approach SRNet++ – an enhancement to the original SRNet architecture [31]. The proposed SRNet++ works as follows: (a) Background Generation: In this step, we employ a diffusion-based model using the ControlNet architecture to generate a text-erased image from an input containing scene texts using a publicly available implementation [29]. The model is conditioned on a binary text-masked image. It takes the full-sized source image and a mask image indicating detected text regions as input, producing a full-size image with text regions erased. These text-erased regions are then cropped to obtain clean background images. (b) Foreground Generation: In this, we modify the SRNet [31] architecture to generate only the foreground text information on a plain background. The model takes a source word crop and target text (rendered as black text on a gray background) as input, generating colored foreground scene text on a gray as output. During training, the model is optimized to generate both the target text and a skeletal image of the text. This model is trained from scratch using synthetic data. (c) Composition Step: This step combines the generated background and foreground images for each word. We apply Otsu’s method [25] to the foreground image to obtain a thresholded binary mask, which is used to extract the foreground text region. The extracted text is then composited onto the background image. This approach results in a clear, smooth image that maintains visual consistency with the source, avoiding the jagged-edge artifacts that can occur with simple overlay methods. The background generation model utilizes pre-trained weights, while the foreground generation model is trained from scratch on our synthetic dataset. In Section 6, we compare the decoupled approach of our proposed SRNet++ with direct scene-text generation methods of MOSTEL [27] and SRNet [31].

Once we obtain all the target word images through this process, we compose them onto the full-sized input image at their respective positions. This final composition step yields the complete visually translated image.

The problem of visual translation has not been comprehensively studied in the literature. Therefore, no benchmark dataset currently exists for its comprehensive investigation. To fill this gap, we present the following datasets:

Evaluating visual translation methods is complex, even more so than evaluating machine translation. While the evaluation of machine translation has been a longstanding research area in the NLP community, recent research has saturated the use of metrics such as BLEU, METOER, and ROUGE; visual translation poses several additional challenges. Unlike machine translation, which is typically evaluated for a sentence or paragraph of text, in visual translation, one has to evaluate the correctness of translation for a single word or a small set of words or phrases. Further, It requires not only assessing the linguistic accuracy of the translation but also ensuring the preservation of background and font properties.

In this work, we propose automatic and user evaluation as follows:

In this section, we comprehensively evaluate scene-text to scene-text translation baseline approaches discussed in Section 3.2 using both automatic and user evaluation metrics proposed in Section 5. We use VT-Real introduced in Section 4 for all our evaluation.

The automatic evaluation results are reported in Table 1. We observe that SRNet++ clearly emerges as the best scene text synthesis approach as compared to other existing architectures. The proposed design enhancements also significantly boost translation quality while maintaining nearly identical perceptual quality. We also observe that usage of IndicTrans2 as a translator consistently leads to an increase in translation quality. The state-of-the-art scene text recognition approaches are as good as ground truth annotations (Oracle) in the case of detecting and recognizing English text.

We further perform a rigorous user study using metrics presented in Section 5.2. As discussed in this section, we have collected user feedback from four qualified users and report mean scores of TQ, R, PQ, and SSP in Table 2. These scores nearly align with observations made via automatic evaluation. We also observe that there is significant room for improvement on all these metrics, indicating the challenge associated with the task.

We have presented a comprehensive study for the task of visual translation by proposing a series of baselines that utilize state-of-the-art approaches and their enhancements across various modules. Our baselines demonstrate promising results for translating scene text images between English and Hindi. However, it is evident that visual translation remains challenging, and addressing all of its complexities extends beyond the scope of this single paper. We hope that introducing this task, along with the dataset, baseline, and performance metrics, will inspire the research community to develop advanced models for visual translation.

This work was partly supported by MeitY, Government of India under NLTM-Bhashini.