Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Visual Encoder - We utilize the visual component of CLIP, specifically the ViT-B/32 model, as the visual encoder, which converts incoming images to a feature sequence of length 49 for the image patches and a global semantic feature. Mini-Omni2 concatenates these to form a raw feature sequence of length 50, employing a single-layer LlamaMLP(Touvron et al., 2023) as the vision adapter.

Audio Encoder - In the encoder section, we continue our previous work by using the Whisper-small model as the audio encoder. We opted not to adopt a token-in-token-out modeling approach for audio input and output for two reasons. (i) Strong semantic alignment in speech recognition. The Whisper model, proposed by OpenAI, is trained on thousands of hours of datasets, demonstrating exceptional robustness. Furthermore, we unexpectedly found that Mini-Omni exhibits an understanding of Chinese data, despite not being trained on any Chinese datasets. We believe this is due to the Whisper model’s capability to automatically align audio from different languages, tones, and noise levels that convey the same meaning, thereby enabling the model to focus on the user’s intention. (ii) Unstable open-source audio tokens. We observed a phenomenon where a) the audio loss of Mini-Omni2 remains high during training, and b) the tokens for a segment of audio can vary significantly based on the content at both ends. We argue that tokens are insufficient for reliably conveying the content of speech input, as evidenced by the poor performance of ASR comparing to semantic features like Whisper.

Language Model - Mini-Omni2 uses the Qwen2-0.5B base version as its foundational language model. We have ported the Llama-based Qwen2 model using the LitGPT(AI, 2023) training framework, employing the configuration of the 0.5B model as the base language model. For the parallel generation of the multi-layer codebook shown in Figure 3, we expanded the vocabulary of the Qwen2 model by adding 7 × 4160 sub-LM-heads, as illustrated in Figure 4, resulting in a vocabulary size of 181,120.

Multimodal Modeling - Consider $Y=(y_{i}\in\mathcal{V}_{\text{txt}}\mid i=1,\ldots,t_{\text{txt}})$ as a text utterance from a vocabulary $\mathcal{V}_{\text{txt}}$ with length $t_{\text{txt}}$. The probability of $Y$ can be expressed as $p(Y)=\prod_{i=1}^{t_{\text{txt}}}p(y_{i}\mid y_{1},\ldots,y_{i-1})$. Now, when dealing with a continuous speech signal, we can convert it into discrete speech tokens (dst), represented as $D=(d_{i}\in\mathcal{V}_{\text{dst}}|i=1,\cdots,t_{\text{dst}})$ using a audio tokenizer. In this context $\mathcal{V}_{\text{dst}}$ is the vocabulary of discrete speech tokens. These discrete speech tokens can be treated as spoken language within $\mathcal{V}_{\text{dst}}$ and modeled in a manner similar to text. We combine text and speech in a new vocabulary $\mathcal{V}_{\text{voxt}}$ by $\mathcal{V}_{\text{voxt}}=\mathcal{V}_{\text{txt}}\cup\mathcal{V}_{\text{dst}}$. Additionally, we introduce visual features $V\in\mathcal{F}_{\text{vis}}$, where $\mathcal{F}_{\text{vis}}$ represents the continuous features extracted from the image. Therefore, we can model the probability of both speech, text, where $Z=(z_{i}\in\mathcal{V}|i=1,\cdots,t)$. This probability is expressed as $p(Z)=\prod_{i=1}^{t}p(z_{i}\mid z_{1},\cdots,z_{i-1},V)$, where $Z$ represents discrete speech tokens $D(\mathcal{V}=\mathcal{V}_{\text{dst}})$, text tokens $Y(\mathcal{V}=\mathcal{V}_{\text{txt}})$, and continuous video features $V(\mathcal{F}=\mathcal{F}_{\text{vis}})$, or various combinations of $Y$, $D$, and $V$. For the audio and text tokens generated simultaneously, the negative log-likelihood loss can be formulated as in Equation (1).

where $T$, $A$ are the text-audio output pairs in the training corpus $C$, and $m$ is the number of training examples. $X_{j}$ and $V_{j}$ is the input condition of the $j$-th example, $n_{j}$ is the max number of tokens of sample $T_{j}$ and $A_{j}$, and $T_{i,j}$ and $A_{i,j}$ represent the $i$-th text token and audio token of the $j$-th sample.

Multi-modal token-Mixed Input - The modeling of input and output tokens for some of the model’s main tasks is illustrated in Figure 3. In this section, we will discuss the model’s inputs and outputs. Since the model incorporates multiple LM-heads, it generates multiple sequences in an auto-regressive manner. As a result, the model also takes multiple sequences as inputs. The input sequences can include a mixed input from a minimum of one modality to a maximum of three modalities. In this subsection, we will discuss the methods for modality mixing.

• •

  Visual-[Audio|Text] Input Our experiments indicate that the Transformer architecture is easier to train and generates more natural responses when auto-regressive tasks are connected with semantic information. Therefore, as shown in Figure 3 (a), we first place the visual features processed by the vision adapter, followed by the Whisper features processed by the audio adapter. Finally, at the position where a response needs to be generated auto-regressively, we place a special token for the response. The total length is approximately 50(CLIP feature length) + $L_{a}$(Whisper feature length).

• •

  Single Modality Input Single-modal inputs may consist of visual, speech, or text inputs. We place the features of both visual and audio modalities across layers 1 to 7. These features will be replicated to enhance their prominence when averaged across all layer features. Notably, when only a single modality’s features are input without the control of a special token, the default tasks are image caption, speech-to-text question answering, and text-to-text question answering.

Text-Audio Parrallel Decoding In Mini-Omni2, we essentially retain the output strategy of Mini-Omni, employing the Text-Instruct Delay Parallel Decoding algorithm to enhance audio generation. This approach utilizes text-audio parallel decoding to simultaneously generate audio and text tokens, leveraging text-to-speech synthesis for real-time output. We continue the parallel generation method introduced by MusicGen(Copet et al., 2024), utilizing SNAC as the audio encoder, which comprises seven complementary token layers. In a single step, we generate eight tokens, including text, while maintaining a one-step delay between layers. Furthermore, we incorporate a Batch approach that involves two samples: one requiring both text and audio responses and the other necessitating a text-only response. By discarding the text token from the first sample and embedding the output from the second sample into the first, we effectively transfer the model’s text-based capabilities to audio tasks, significantly enhancing reasoning abilities with minimal resource overhead. We have provided detailed explanations of the specific technical details in the Mini-Omni(Xie and Wu, 2024).

Overall, we have introduced our modeling approach for three-modal inputs and two-modal outputs within a single model. Through these methods, the model can accomplish eight reasonable multi-modal tasks, with some of the primary tasks illustrated in Figure 3, showcasing all the multilayer tokens generated during a single inference.

In this section, we will introduce the training phase of the Mini-Omni2 model. The overall training process of Mini-Omni2 is illustrated in Figure 5. The training process is divided into three stages, with multitask training employed in each stage. In the figure, except for Stage 1, a foundational text-to-text task is additionally incorporated but not explicitly depicted. We categorize the entire training process into three stages:

• •

  Multimodal Encoder Adaptation In the first stage, we employ a rapid, small-scale training focused solely on the weights of the linear layer connecting the language model and the encoder. The objective of Stage 1 is to ensure that the multi-modal features received by the model closely resemble the characteristics of text tokens as represented in the model’s embedding layer. We believe this approach offers two primary advantages: 1. It allows the model to concentrate on logical reasoning in modality-specific question answering during subsequent training. 2. It minimizes the parameter changes in the language model’s core that would otherwise result from adapting to other modalities.

• •

  Modality Alignment In stage 2, the primary task of the model training is to transfer the question-answering ability based on text input to question-answering abilities based on images and audio. In this step, the adapters trained in stage 1 are temporarily frozen, and the weights of the language model are involved in the training. At this stage, all tasks do not involve audio responses. For tasks like image-based and audio-based QA, only text-based responses are generated to establish the model’s foundational logical capabilities. The speech output is simply an extension of this logical ability into different modalities.

• •

  Post training In Stage 3, the task of the model is to extend the output modality to include audio response generation. As shown in Figure 5, the model will be trained on all tasks from Stage 1 and Stage 2, with audio token outputs for all question-answering tasks. Additionally, the model will learn interruption mechanism, an algorithm introduced in the next section.

A real-time conversation model needs to have duplex capability in order to enable more flexible interactions. However, this interruption mechanism should not be a simple VAD (Voice Activity Detection)-based one, but rather a system that can determine whether the user intends to interrupt the model. Additionally, the model’s ability should be highly robust, capable of handling various external situations (e.g., noise, other conversations, and unrelated sounds). We explore this functionality with command-based task, where the model will stop talking immediately when user speaks "Stop Omni". Furthermore, this approach can be naturally extended to incorporate more sophisticated semantic interruption mechanisms through the development of more contextually appropriate interruption datasets.

Background Noise Selection: (1) We randomly utilized a variety of speech recognition samples from the Libri-tts dataset as the original human noise data samples. (2) We employed samples from the MUSAN(Snyder et al., 2015) dataset, which includes music, human voices, white noise, and urban noise.

Semantic Interruption Construction: We synthesized "Stop Omni" phrases with random voice timbres, which were subsequently mixed with noise. The specific data construction methods are introduced in the next section.

Combining the aforementioned data, the model will receive long sequences of data containing "Stop Omni" phrases amidst various noises. The model will generate two types of state tokens in real time: irq and n-irq, representing the intention of the user to interrupt and not to interrupt, respectively. During inference, when the model output irq token, it will stop the generating process and start to listen the new question. For this task, we use tokens as input to enhance the model’s real-time processing capabilities.

The training data for the Mini-Omni2 model is primarily sourced from five components, as shown in Table 1. (1) Textual Question-Answering Data: Throughout all training stages, whenever the language model weights were unfrozen for training, textual question-answering data was included to maintain the model’s reasoning ability. We used the first 1.5 million question-answer pairs from the Open-Orca dataset. (2) Speech Recognition Data: Speech recognition data was used to continuously maintain the model’s semantic understanding of external spoken input. We primarily utilized the LibriTTS, VCTK, and Multilingual LibriSpeech datasets. (3) Spoken Question-Answering Data: We did not use a standalone spoken dataset; instead, synthetic data was employed for training. The spoken question-answering data was derived from the Moss-002-sft dataset. (4) Image Question-Answering Data: We used 400,000 samples (caption and instruction) from the ALLaVA-4V dataset. (5) Voice Assistant Data: To make the model’s responses more aligned with the style of voice assistants, we continuously used the VoiceAssistant-400K dataset introduced in Mini-Omni.

The Mini-Omni2 model completed all training steps on eight A100 GPUs. During the adapter training stage, learning rates ranged from 2e-5 to 1e-3, while training the language model used learning rates between 2e-6 and 2e-4. The final fine-tuning was conducted with learning rates ranging from 2e-6 to 2e-5. A cosine scheduler was employed, with 1,500 warm-up steps and a global batch size of 192. Each stage was trained for one epoch using the full dataset. The scale of the vision and audio encoders was described earlier, and the language model used was the Qwen2-0.5B base model. All model adapters used Llama-MLP with an intermediate size of 4,864.

Spoken Dialogue Data: We used our speech recognition dataset as a random voice timbre library. To ensure robust training, a random sample from this dataset was selected as a voice prompt for the input of all spoken dialogue data, and CosyVoice(Du et al., 2024) was employed for zero-shot speech synthesis. For the output of all question-answering data, the same voice timbre was used from an internal TTS system.

Interruption Data: First, the noise data is stream-encoded and decoded to simulate real-time streaming input to the model. Then, a random segment of the noise data is extracted. At the end of this segment, a "Stop Omni" phrase is inserted, generated with a random voice timbre in the same manner as the dialogue data. Finally, an additional "tail" of 0-10 seconds is appended to the end of this segment. In terms of labeling, all data before the tail is labeled as "n-irq", while the tail segment is labeled as "irq", indicating that the model should be interrupted.

Currently, we provide the accuracy of Mini-Omni2 in speech recognition to evaluate the model’s speech understanding ability, and we present some practical cases. For model experience and more cases, please follow our github repositories.

According to the speech recognition results in Table 2, it can be observed that the accuracy of Mini-Omni2 shows a slight decline after adding the visual modality compared to Mini-Omni. This phenomenon may be attributed to the relative reduction in the proportion of data. Moreover, in comparison with the decoder of the whisper module employed by the model, the Mini-Omni2 model outperforms Whisper on the librispeech-other dataset. This demonstrates that our training process has enhanced the robustness of the model in speech recognition.

Here we present some use cases from Mini-Omni2.