SpeechX: Neural Codec Language Model as a Versatile Speech Transformer

Generative models based on a language modeling approach using autoregressive Transformers, also known as decoder-only Transformers, have garnered significant success in various application domains. Notable examples of such models include the GPT series [1, 2] and DALL-E [30]. The autoregressive approach has also been extended to the audio and speech domains. AudioLM [4] and MusicLM [10] are pioneering efforts that exploit multiple types of tokens, each with a distinct time scale and degree of semantic granularity, allowing for hierarchical token generation. This hierarchical structure, comprising both coarse and fine-grained tokens, enables the synthesis of sounds with both nuanced details and long-term regularities.

For zero-shot TTS, VALL-E [16] and SPEAR-TTS [17] employ the autoregressive Transformers by representing textual (semantic) and acoustic tokens as a single data stream. This approach enables the models to perform zero-shot speaker adaptation, facilitating the generation of TTS voices that mimic a specific person’s voice. It was demonstrated that these models could perform zero-shot TTS from speech clips as short as three seconds. A notable advantage of these autoregressive speech generation models is their ability to perform TTS without requiring a separate duration model. This streamlined architecture simplifies the training process and potentially offers increased flexibility needed to subsume various speech generation tasks. For this reason, we opt to build our SpeechX models by using autoregressive Transformers.

Several papers have recently reported efforts in developing audio-text-based speech generation models that support zero-shot TTS and several related tasks. These tasks include voice or style conversion (Make-A-Voice [18], NaturalSpeech2 [31], and Voicebox [15]), speech editing (Mega-TTS [21] and Voicebox), and denoising (NaturalSpeech2 and Voicebox). Voicebox has showcased noteworthy advancements by facilitating a multitude of tasks through its masked speech prediction principle. Nevertheless, its capabilities are still limited to clean speech generation alone, falling short of effectively dealing with noisy speech or encompassing conventional audio enhancement tasks such as noise suppression and target speaker extraction.

In this study, we deal with both clean and noisy speech and unify the generation and transformation tasks. To accomplish this, we extend VALL-E by performing multi-task learning with task-dependent prompts. The resulting model, which we call SpeechX, exhibits versatility in various speech processing tasks. The model excels not only in speech generation tasks like zero-shot TTS and speech editing but also performs effectively in enhancement tasks such as noise suppression and target speaker extraction. It also realizes novel capabilities, such as editing spoken content while retaining the background noise or effectively leveraging transcriptions for enhancement tasks.

Fig. 1 illustrates an overview of the SpeechX architecture. Building upon the principles introduced in VALL-E, SpeechX employs a neural codec language model based on Transformers. The model learns to perform conditional generation of a neural code sequence, denoted as $\mathcal{O}$, based on two input prompts: textual prompt $\mathcal{T}$ and acoustic prompt $\mathcal{A}$. The neural codes may also be referred to as acoustic tokens.

The textual prompt $\mathcal{T}$ is a sequence of phonemes obtained by applying grapheme-to-phoneme conversion²²2https://github.com/Kyubyong/g2p to an input text. The textual prompt conveys the semantic information, and thus it is called semantic tokens. Conversely, the acoustic prompt $\mathcal{A}$ encapsulates the acoustic information of an input speech signal. It is obtained by converting the input audio into a sequence of acoustic tokens with an encoder of the neural codec model. Furthermore, to specify the task to be executed, or equivalently the desired output, we incorporate additional tokens in the acoustic prompt. The details will be explained in Section III-C. The output $\mathcal{O}$ is a sequence of neural codes of the desired signal, which is then translated into a waveform signal with the codec decoder.

We use EnCodec [32] as the neural codec model, following the prior work. EnCodec is based on an encoder-decoder architecture with $L$ quantization layers. In our experiments, we use $L=8$ to be consistent with the configuration of [16]. Each layer of the EnCodec model produces discrete codes consisting of 1024 entries at a sampling rate of 75 Hz.

We emphasize that the proposed simple architecture capitalizes on the end-to-end modeling capability of the neural language modeling approach. In contrast to other zero-shot TTS or speech generation methods, this approach eliminates the need for a separate model, such as a speaker embedding model or a duration model, apart from the neural codec model. This key property allows SpeechX to acquire knowledge of diverse tasks with varying requirements and input-output relationships, thereby facilitating a versatile and highly extensible speech generation process.

As with VALL-E [16], SpeechX makes use of auto-regressive (AR) and non-auto-regressive (NAR) Transformer models. Specifically, the AR model is used to output the neural codes corresponding to the first quantization layer of EnCodec. On the other hand, the NAR model generates the neural codes of all the layers above the first layer, namely the second through eighth layers. Combining the AR and NAR models provides a reasonable trade-off between generation flexibility and inference speed, as discussed in [16].

Let output $\mathcal{O}$ be specifically represented as matrix $\mathbf{O}=[o_{t,l}]\in\mathbb{N}^{T\times L}$, where $o_{t,l}$ represents the code for the $l$-th codec layer at time frame $t$ and it can take one of the 1024 values. The output sequence length is denoted by $T$. The AR model comprises a stack of Transformer decoder layers [33] and is optimized by minimizing the negative log-likelihood of the first layer code of the desired output, which is defined as follows:

where $\mathbf{o}_{<t,1}=[o_{1,1},\cdots,o_{t-1,1}]$, while $\theta_{\textit{AR}}$ represents the AR Transformer model parameters. Different embedding projections are applied to the textual and acoustic tokens, and they are superimposed by sinusoidal positional embeddings.

Note that the AR model in SpeechX is conditioned on three elements: the acoustic prompt $\mathcal{A}$, the textual prompt $\mathcal{T}$, and the past acoustic history $\mathbf{o}_{<t,1}$. This formulation differs from that of VALL-E, where the AR model is conditioned only on $\mathcal{T}$ and $\mathbf{o}_{<t,1}$ (Eq. (1) of [16]), and the audio prompt is represented as part of $\mathbf{o}_{<t,1}$ during inference. This difference provides a practical benefit during the inference time of zero-shot TTS, where SpeechX no longer requires a transcription of the audio prompt. More specifically, in the case of VALL-E inference, we need to construct $\mathcal{T}$ from the concatenation of the transcription of the audio prompt and the text prompt. The audio is then generated by setting the codec sequence of the audio prompt to $\mathbf{o}_{<t,1}$. In contrast, for SpeechX inference, $\mathcal{T}$ is simply the text prompt. The model can generate the codec sequence without requiring the transcription of the audio prompt.

After obtaining the first layer codes with the AR model, the NAR model is used to generate the $l$-th layer codes based on the text and acoustic prompts as well as the output codes for the first $l-1$ layers, which have already been produced. The model is used repeatedly for $l=2,\cdots,8$. Since we use the same NAR model for the remaining seven layers, the NAR model is trained to minimize the following negative log-likelihood function:

where $\theta_{\textit{NAR}}$ represents the NAR model parameters, while $\bm{o}_{:,l}$ denotes the entire sequence of $o_{t,l}$ for the $l$th layer, and $\bm{o}_{:,<l}=[\bm{o}_{:,1},\cdots,\bm{o}_{:,l-1}]$. In this formulation, in order for the single NAR model to process each of the seven layers, the acoustic tokens from the first to $(l-1)$th layers, $\textbf{o}_{:,<l}$, are embedded and summed up.

SpeechX aims to handle multiple tasks with one model. To this end, we adopt task-based prompting, as illustrated in Table I and explained in detail below.

Noise suppression is a task of extracting clean speech signal $s$ from its noise-corrupted observation $s+n$, where $n$ denotes the noise. For the noise suppression task, we incorporate a special token, denoted as <ns>, to form the acoustic prompt, resulting in $\mathcal{A}=[\texttt{<ns>},\mathrm{C}(s+n)]$. Here, $\mathrm{C}(\cdot)$ denotes the function used to convert an audio signal into a neural codec token sequence. While the textual prompt $\mathcal{T}$ is supposed to be provided by a user as a reference transcription, we let the use of the textual prompt be optional to accommodate the scenario where the human transcription is unavailable. The desired output is the acoustic token sequence of the clean audio, $\mathrm{C}(s)$.

Speech removal involves removing speech from a noisy speech signal while preserving the background noise. It is useful for removing only unwanted speech from recordings. To address this task, we employ a special token, <sr>, to construct the acoustic prompt as $\mathcal{A}=[\texttt{<sr>},\mathrm{C}(s+n)]$. The desired output is the acoustic token sequence of the noise signal, $\mathrm{C}(n)$. As in the case of noise suppression, the textual prompt can be omitted.

Target speaker extraction aims at isolating clean speech $s_{1}$ of a target speaker from a mixture of $s_{1}$ and interfering speech $s_{2}$ from a secondary speaker. The target speaker is identified through a short enrollment audio $s^{\prime}_{1}$ of that individual, where we assumed three seconds for the enrollment. For this task, we form the acoustic prompt by concatenating the acoustic tokens extracted from the enrollment audio, $\mathrm{C}(s^{\prime}_{1})$, and those of the mixed speech, $\mathrm{C}(s_{1}+s_{2})$, with a task-specifying token, denoted as <tse>. That is, we have $\mathcal{A}=[\mathrm{C}(s^{\prime}_{1}),\texttt{<tse>},\mathrm{C}(s_{1}+s_{2})]$. The desired output is $\mathrm{C}(s_{1})$. As with the previous tasks, the inclusion of the textual prompt is optional.

Zero-shot TTS aims to generate a speech signal $s^{\prime}$ by leveraging both the provided input text and an enrollment speech $s$. The goal is to ensure that the speech characteristics of $s^{\prime}$ closely resemble those of $s$, while also accurately reflecting the input text. For this task, we employ the acoustic tokens extracted from the enrollment audio, denoted as $\mathrm{C}(s)$, as the acoustic prompt. The model generates acoustic tokens for the synthesized speech, $\mathrm{C}(s^{\prime})$, based on the input text. These acoustic tokens are then converted into the corresponding waveform.

Clean speech editing is defined as modifying a segment of input speech to align with an input text. Let $s$ denote the input speech signal to be edited. We divide $s$ into three distinct portions, $s_{\rm pre}$, $s_{\rm mid}$, and $s_{\rm post}$, with $s_{\rm mid}$ being the target segment for editing, without loss of generality ($s_{\rm pre}$ and $s_{\rm post}$ can be empty). We construct the acoustic prompt as $[\mathrm{C}(s_{\rm pre}),\texttt{<soe>},\texttt{<mask>},\texttt{<eoe>},\mathrm{C}(s_{\rm post})]$, where new tokens <soe>, <mask>, <eoe> are introduced to specify the task and the speech segment designated for editing. The desired output is a sequence of neural codes, $[\mathrm{C}(s_{\rm pre}),\mathrm{C}(s_{\rm edit}),\mathrm{C}(s_{\rm post})]$, where the spoken content of $[s_{\rm pre},s_{\rm edit},s_{\rm post}]$ matches the input text. The speaker characteristics of $s_{\rm edit}$ must be consistent with those of $s_{\rm pre}$ and $s_{\rm post}$.

Noisy speech editing, in contrast, operates on noisy speech as input, aiming to modify the speech content within a segment while keeping the underlying background noise intact. Therefore, this task would be more challenging than the clean speech editing task because the model needs to distinguish between speech and noise during the editing process. To accomplish this objective, it is crucial to provide the model with the complete input speech signal instead of masking out the segment for editing with a <mask> token. Therefore, we construct the acoustic prompt as $[\mathrm{C}(s_{\rm pre}+n_{\rm pre}),\texttt{<soe>},\mathrm{C}(s_{\rm mid}+n_{\rm mid}),\texttt{<eoe>},\mathrm{C}(s_{\rm post}+n_{\rm post})]$, with the subscripts corresponding to pre, mid, or post as previously defined. The desired output comprises a sequence of neural codes, $[\mathrm{C}(s_{\rm pre}+n_{\rm pre}),\mathrm{C}(s_{\rm edit}+n_{\rm mid}),\mathrm{C}(s_{\rm post}+n_{\rm post})]$. This formulation makes it clear that the model must transform $s_{\rm mid}$ into $s_{\rm edit}$ based on the text input while retaining $n_{\rm mid}$.

In practical speech editing scenarios, the input text is often obtained by first applying automatic speech recognition (ASR) to the input speech and then having a user edit the transcription. In such situations, it is simple to identify the positions at which <soe> and <eoe> must be inserted. Also, it is noteworthy that, in clean speech editing, the use of <mask> allows the model to adaptively change the output speech length in such a way that the output speech sounds natural in terms of speaking speed.

The outlined task-based prompting strategy equips the SpeechX model with the ability to uniquely decide the desired output during inference. This approach enables flexibility for incorporating additional tasks. Adding new tasks entails integrating corresponding prompting schemes and continuing model training from an existing checkpoint, where only embeddings for newly introduced task-specific tokens are randomly initialized. This can be performed without changing the underlying model architecture.

During training, we randomly sample the task for each model update at an equal probability. This is intended to ensure the model does not unduly favor any particular tasks. For noise suppression, speech removal, and target speaker extraction tasks, we include the textual prompt at a 50% probability so that the model equally experiences both text and text-less scenarios.

To help the model to acquire basic generation capabilities, we first train the model only for zero-shot TTS and then continue the training process using all the tasks to perform multi-task learning. In other words, we initialize the model with an existing VALL-E model checkpoint. Precisely speaking, the SpeechX model trained solely for zero-shot TTS exhibits slight divergence from VALL-E. This difference arises from the fact that the former explicitly incorporates a distinct enrollment audio, originating from the same speaker, for each training sample, while the latter does not. Nevertheless, for the sake of simplicity, we refer to this initialization approach as VALL-E initialization. When starting the multi-task training stage, randomly initialized embeddings are appended for the special tokens related to the task-dependent prompts. This two-stage training strategy substantially enhances performance across all tasks, as evidenced by our experimental results.

Zero-shot TTS: For each test sample, we used the reference transcription to create the textual prompt. The acoustic prompt was generated by randomly choosing another utterance of the same speaker and extracting a 3-second-long clip.

Noise suppression: We mixed each test sample with a noise sample randomly picked from the MUSAN dataset [34] at a signal-to-noise ratio (SNR) which was randomly determined from the range between 0 dB and 20 dB. The task was to recover the uncorrupted speech from the noisy speech. The acoustic prompt was obtained by applying EnCodec to the noisy signal. Regarding the textual prompt, we considered both text-less (i.e., using no semantic prompt) and text-guided noise suppression, where we used the reference transcription for the text-guided setting.

Target speaker extraction: We mixed each test sample with an utterance of a different speaker at a signal-to-interference ratio (SIR) which was randomly determined from the range between 0 dB and 20 dB. Also, we randomly chose one or more other utterances of the same speaker to create a 3-second-long enrollment clip to help models identify who the desired speaker is. Both the mixed and enrollment signals were used to derive the acoustic prompt as described in Section III-C. The task was to recover the original uncorrupted speech of the target speaker. As with the noise suppression task, we considered both text-less and text-guided settings.

Clean speech editing: For each test sample, we randomly selected a period of length between 10% and 50% of the whole utterance. We replaced the speech of the selected period with another randomly chosen speech sample of the same speaker. Given the partially replaced, speaker homogeneous speech and the reference transcription, the task was to generate a speech signal that follows the transcription without changing the speaker characteristics and the unreplaced portion of the input signal. In our experiments, we used the correct <soe> and <eoe> locations based on the knowledge of the replaced segment.

Noisy speech editing: We added a randomly picked MUSAN noise sample to each test sample of the clean speech editing task. The SNR was chosen from the range of 0 dB to 20 dB. Given the noise-corrupted partially replaced speech and the reference transcription, the task was to generate a noisy speech signal that follows the transcription without changing the background noise, the speaker characteristics, and the unreplaced portion of the input speech.

Speech removal: The same dataset was used as the one used for noise suppression. Given a noisy speech signal, the task was to extract the noise signal by removing the speech. We considered only the textless case. Consequently, the input exclusively comprised the acoustic prompt corresponding to the noisy speech.

For consistency and reproducibility, we opted to use objective metrics for individual tasks as described below.

Word error rate (WER): We employed the WER as a metric to evaluate the fidelity of the generated audio in adhering to the provided transcription. The ASR system utilized for our experiments was NeMo’s stt_en_conformer_transducer_large model³³3https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge, which is based on the Conformer Transducer architecture [35]. We selected this particular ASR model based on its superior stability and robustness against noise and processing artifacts in comparison to other publicly available ASR models, as was observed during our preliminary experiments. Robustness in ASR is particularly crucial for tasks such as noise suppression and noisy speech editing. The WER metric was employed across all tasks, with the exception of speech removal.

Speaker similarity score (SIM): The speaker similarity score served as a metric to assess the coherence of the generated speech in relation to the speaker’s characteristics. This score was calculated as the cosine similarity between the speaker embeddings of the generated speech and the desired speech signals. The computation of speaker embeddings was performed using NeMo’s TitaNet-Large⁴⁴4https://huggingface.co/nvidia/speakerverification_en_titanet_large. We employed the original audio data instead of utilizing an EnCodec-processed signal for the speaker similarity measurement to capture and reflect any potential speech deformation effects that may arise due to the use of the EnCodec model. SIM was used in zero-shot TTS, clean speech editing, and noisy speech editing.

DNSMOS: For evaluation in the noise suppression and target speaker extraction tasks, we utilized DNSMOS [36], a well-established model-based metric for predicting the perceived quality of acoustically corrupted speech⁵⁵5https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS. Specifically, we employed the OVRL score from the DNSMOS P.835 model. To evaluate the performance of target speaker extraction, we employed a personalized DNSMOS model, which was tailored for this particular task and is available on the same webpage.

Perceptual Evaluation of Speech Quality (PESQ): For the noise suppression and target speaker extraction tasks, we also utilized PESQ [37]. Unlike DNSMOS, PESQ is an intrusive metric that necessitates the clean reference signals. Consequently, PESQ is expected to assess the fidelity of the generated audio with respect to the original clean data.

Mel-cepstral distortion (MCD): MCD⁶⁶6https://pypi.org/project/pymcd is a metric used to quantify the dissimilarity between two sequences of mel cepstra. We employed this metric to objectively measure the speech removal accuracy by comparing the estimated noise with the ground truth noise audio.

We sourced clean speech data from LibriLight, comprising 60 thousand hours of untranscribed English reading speech from over 7,000 speakers [29], as was performed in the zero-shot TTS experiment using VALL-E [16]. To meet the specific training requirements for each task, data simulation was performed by following the methods employed for creating the evaluation data, as elaborated below. Note that, as discussed in Section III-D, we formed individual training mini-batches based on randomly selected tasks for each iteration.

For the noise suppression and speech removal tasks, we mixed the clean speech with noise samples from the DNS challenge corpus [38] at SNRs between -5 dB and 20 dB. Our models were trained to recover the acoustic tokens of the clean speech and noise for noise suppression and speech removal, respectively. For the target speaker extraction task, we mixed the individual clean speech samples with those of other randomly chosen speakers with SIRs ranging from -5 dB to 20 dB. Regarding clean speech editing, for each clean utterance, we randomly selected a subsegment of length ranging from 10% to 70%, and then substituted it with another audio segment from the same speaker with different content. We saved the start and end times of the replaced segment, which were used to insert the <soe> and <eoe> tokens to the correct positions in the acoustic prompt during training. Furthermore, to create training samples for noisy speech editing, we added noise samples used in the noise suppression task to the partially replaced clean audio. As a result, we obtained pairs of noisy partially replaced speech and the corresponding original noisy speech, which served as the training data for the noisy speech editing task. The SNR range used for noisy speech editing training was also from $-5$ dB to $20$ dB.

Since LibriLight does not provide reference transcriptions, we adopted a pseudo-labeling approach to derive the semantic prompts, i.e., the phoneme sequences of the individual training samples, by following [15, 16]. Specifically, we transcribed the LibriLight training data with an off-the-shelf Kaldi model that was trained on the 960-hour Librispeech data with 3x speed perturbation⁷⁷7https://kaldi-asr.org/models/m13.

Both the SpeechX AR and NAR models share the same Transformer architecture, featuring 12 layers, 16 attention heads, an embedding dimension of 1024, a feed-forward layer dimension of 4096, and a dropout rate of 0.1.

We conducted experiments employing two initialization methods: random initialization and VALL-E initialization (refer to Section III-D for details). In the random initialization scenario, we trained the SpeechX model for 800k iterations. The model optimization utilized the AdamW optimizer, with the learning rate undergoing a warm-up phase for the initial 32K updates, peaking at $5\times 10^{-4}$, before transitioning into a linear decay phase. Conversely, with VALL-E initialization, we opted for 400k iterations, as the initial model already underwent zero-shot TTS training over 400k iterations. In this instance, the learning rate scheduler was retained, but the warm-up period was shortened to the first 20k updates.

We employed expert models for different tasks to establish comparison baselines. For zero-shot TTS, we utilized VALL-E by following the model configuration outlined in the original paper [16]. For the noise suppression task, we employed a non-causal Deep Complex Convolutional Recurrent Network (DCCRN) [39], which is a widely recognized model for noise suppression. Our training data for DCCRN came from Microsoft’s internal dataset, and we further fine-tuned the model using the ASR objective based on the training recipe of [40]. For target speaker extraction, we leveraged VoiceFilter [22], employing a bidirectional LSTM configuration. We relied on a publicly available implementation of VoiceFilter⁸⁸8https://github.com/Edresson/VoiceSplit. Finally, for speech editing, we employed A³T [20] as the baseline. The implementation of A³T that we used is also publicly accessible⁹⁹9https://github.com/richardbaihe/a3t.