On decoder-only architecture for speech-to-text and large language model integration

In recent times, the large language models (LLMs) have showcased remarkable achievements across various natural language benchmarks, encompassing question answering, machine translation, language understanding and more [1, 2, 3, 4, 5]. By employing a Transformer-based architecture [6] and training to anticipate forthcoming tokens within a sequence, this language model excels in contextual learning abilities. Not only does this significantly enhance its modeling prowess, but more importantly, it enables seamless user interaction that effectively connects cutting-edge research with real-world applications.

As speech represents the most innate and instinctive mode of human communication, integrating speech and LLMs will further boost the user experience of human-machine interaction. Based on this intuition, several attempts in combining speech signals and large language models were carried out [7, 8, 9, 10]. Among them, the cascaded approach is the most straightforward solution. In these systems, the speech signal is firstly transformed into word tokens through existing automatic speech recognition (ASR) [11] models, and LLM processes the recognized words for downstream tasks. Later, inspired by the integration of image information to LLMs [12, 13, 14, 15], researchers also explored the deep combination of speech signals [9, 10, 16, 17, 18, 19]. In [16], the authors proposed to jointly model the speech and text tasks through a unified decoder only network. Similarly, in [19], the authors proposed to optimize the audio token conversion module together with a off-the-shelf LLM. Instead of word pieces, discrete tokens of speech representation from a self-supervised model are used in [17].

While there have been promising outcomes, several crucial challenges regarding the integration of speech and LLMs still require further exploration. Initially, aligning the two modalities (speech and text) using a pretrained LLM poses challenges due to the typically longer sequence length of speech signals compared to text sequences. Moreover, given the costly nature of training LLMs, finding ways to minimize the overall integration cost while maintaining exceptional performance continues to be a challenging task. More importantly, considering the remarkable success of the LLMs, it is crucial to explore the untapped potential of using a decoder-only model [3, 16, 20, 21, 22] as the backbone network architecture for speech to text processing.

In this study, we aim to tackle the aforementioned challenges by exploring an efficient end-to-end integration of speech and language models. Our approach involves designing a simple yet effective architecture where a large language model that operates on text also incorporates acoustic embeddings. This integration enables the LM to condition its transcription or translation of the acoustic information. More specifically, our proposed method utilizes a pre-existing LLM and incorporates a acoustic feature compressor and an acoustic encoder introducing only a small number of free parameters. Diverging from previous approaches that convert speech into discretized tokens, our model directly maps the continuous representation of speech into the semantic space defined by the LM. During the processing stage, the speech feature is initially compressed by the acoustic compressor to reduce the sequence length. Subsequently, the acoustic encoder transforms the compressed speech signal into continuous vectors in the same semantic space of the text that can be consumed by the LLM. The final output is generated through the decoding process of the LLM.

We thoroughly investigate various practical aspects of our proposed model, such as selecting the appropriate acoustic compressor, attention mask, and fine-tuning methods. Additionally, we apply the proposed model to the task of translating speech in 13 different languages into English (EN) text and compare its performance against a strong baseline on CoVoST dataset. Finally, we demonstrate that the decoder-only model, even trained from scratch using only speech-text paired data, exhibits significant potential and several advantages over the commonly employed encoder-decoder architecture in speech processing. In this work, our contribution can be summarized as follows:

• •

  We introduce an efficient end-to-end integration method called Speech-LLaMA, which effectively integrates existing text-based large language models with speech processing. We have achieved substantial improvements in translation performance compared to strong baselines on various speech translation (ST) tasks.

• •

  We investigate various practical aspects of the proposed speech-LLM integrations that are crucial for enhancing performance. These aspects include acoustic compression of the acoustic feature, attention mask selection, and fine-tuning strategy.

• •

  On large, diverse and real-world data, we show that the decoder-only architecture can be as competitive as the encoder-decoder architecture for speech-to-text tasks. We show that decoder-only to also be more parameter efficient.

Our model aims at integrating speech signals into large language models, as well as relates to Connectionist Temporal Classification (CTC) feature length compression and low-rank adaptation (LoRA). We discuss these topics in the following.

In this work, we design an architecture named Speech-LLaMA where a text-LLM can also accept acoustic embedding as well as text as conditional prompts for text generation. By converting the speech input to a sequence of acoustic embeddings within the same space of the text embeddings, in the aspect of both length and semantics, the pre-trained text LLM can leverage its in-context learning capacity to absorb the speech signal and output corresponding text for speech translation task.

Overall, given the text prompt $\mathbf{p}$ and audio signals $\mathbf{x}$, the generation of the corresponding text sequence $\mathbf{y}=\{y_{0},y_{1},\cdots,y_{N-1}\}$ with a text-LLM is formulated as:

$$p(\mathbf{y}|\mathbf{p},\mathbf{x};\Theta_{\text{LLM}})=\prod_{n=0}^{N-1}p(y_{n}|\mathbf{y}_{<n},\mathbf{p},\mathbf{x};\Theta_{\text{LLM}})$$

(1)

where $\mathbf{y}_{<n}$ indicates the generated text sequence before $y_{n}$.

Overview Our proposed neural model consists of three distinct parts: a pre-trained text neural LLM, an audio encoder and a CTC compressor, as shown in Figure 1. The text-LLM in our case is a LLaMA-7B [14] but this method can be generalized to LLMs of any scale. The CTC compressor reduces the sequence length of the input speech filter-bank to match the length of the text, and the audio encoder transforms the compressed speech signal into continuous vectors in the LLM’s semantic space.

CTC compressor Different from the prior work that trained the CTC compressor jointly with the main task [23], our CTC compressor is a pre-trained module, aiming to match the audio and the text duration to the same scale by selecting the representative frames from the audio signal. In this work, we explore two ways to reduce the sequence length of the acoustic features in the CTC compressor: “blank-removal” and “frame-averaging”. For “blank-removal”, we simply discard all the frames that predicted the blank symbol according to the distribution of the CTC posteriors. On the other hand, for “frame-averaging”, we average the hidden states of consecutive frames without blank frames removed, once their CTC predictions belong to the same class.

Audio encoder The audio encoder is used to bridge representations generated from the CTC compressor to the text embeddings of the text-LLM. This module is designed to be relatively small in size and is initialized with random weights. During the fine-tuning process, the audio encoder is optimized to effectively integrate the audio information within the LLM, enhancing the overall performance of the system. Different from the methods in  [7, 19], where the audio encoder is trained to firstly map the speech signal into discrete tokens, which is then consumed by LLM, the proposed audio encoder is directly optimized to map the compressed acoustic signal to the continuous semantic space of LLM, allowing a deep integration between the audio encoder and the language model.

Instruct learning For each training sample, we prepend a text prompt that briefly describes the task, e.g., “$\mathtt{audio}$ $\Rightarrow$ $\mathtt{English}$” and “$\mathtt{transcribe}$ $\mathtt{the}$ $\mathtt{audio}$ $\mathtt{into}$ $\mathtt{English}$”. The text prompt are sampled from a pre-defined list, where some prompts contains the source language ID following the format “$\mathtt{translate}$ $\mathtt{[source]}$ $\mathtt{audio}$ $\mathtt{into}$ $\mathtt{English}$”. During evaluation, we fix the text prompt as “$\mathtt{translate}$ $\mathtt{the}$ $\mathtt{audio}$ $\mathtt{into}$ $\mathtt{English}$” for all testing samples.

LoRA fine-tuning On top of the proposed model, we apply the LoRA to four attention matrices in each layer of the LLaMA Transformer (e.g., $W_{q},W_{k},W_{v},W_{o}$). To stabilize the training, we adopt a two-stage training scheme which means we train the audio encoder firstly with the CTC compressor and LLaMA frozen and then introduce LoRA to the well-trained model and perform the second stage optimization. The entire system is still trained with cross-entropy loss between the LLM output and the reference transcription sequence on the same training data.

From-scratch training To further explore the potential of decoder-only architecture as a foundational architecture for speech modeling, we also include a “from-scratch” training of a decoder-only architecture. Here, we replace the text prompt, audio encoder, and CTC compressor with a randomly initialized convolutional 2D encoder. We also replace the pretrained LLaMA network with a much smaller randomly initialized autoregressive network. This architecture is shown in Figure 2. We add an $\langle\text{SOS}\rangle$ token at the end of the acoustic sequence to indicate the starting of the generation. In this case, the generation of the text sequence $\mathbf{y}$ with a decoder-only model is conditioned purely on audio signal $\mathbf{x}$ and previously generated text sequence $\mathbf{y}_{<n}$:

$$p(\mathbf{y}|\mathbf{x};\Theta_{\text{DEC}})=\prod_{n=0}^{N-1}p(y_{n}|\mathbf{y}_{<n},\mathbf{x};\Theta_{\text{DEC}})$$

(2)

where $\Theta_{\text{DEC}}$ refers to the parameters of the decoder model.

The speech translation (ST) task [25, 26, 27, 28, 29] has been chosen as the primary evaluation benchmark for assessing the proposed methods. In this task, the goal is to develop a system that can accurately translate spoken language from 13 source languages to English.

The results of the experiments are presented in Table 1, where several observations can be gleaned.

In this work, we propose a method to infuse an off-the-shelf large language model with acoustic information. The proposed model presents a deep integration between the audio with the LLM by directly mapping acoustic representation into the semantic space of LLM. We also explore several practical aspects of the proposed model for better performance including compression of the acoustic feature, attention mask design and adapter fine-tuning. We show that on a 13 language to English speech translation task, the proposed model significantly outperforms a strong sequence-to-sequence baseline model. We also show that the decoder-only architecture, trained from scratch, can achieve comparable performance with around 40% fewer parameters, which verifies the potential of decoder-only models for general speech-to-text modeling.