AMSS-Net: Audio Manipulation on User-Specified Sources with Textual Queries

In recent days, social media applications have attracted many users to create, edit, and share their audio, audio-visual, or other types of multimedia content. However, it is usually hard for non-experts to manipulate them, especially when they want to edit only the desired objects. For image manipulation, fortunately, recently proposed methods such as image inpainting (Yu et al., 2018), style transfer (Zhu et al., 2017), and text-guided image manipulation (Li et al., 2020; Liu et al., 2020) enables non-expert users to edit the desired objects while leaving other contents intact. These machine learned-based methods can reduce human labor for image editing and enable non-experts to manipulate their image without prior knowledge of tools that are often complicated to use.

On the other hand, little attention has been given to machine learning methods for automatic audio editing. It is challenging to edit specific sound objects (e.g., decrease the volume of cicada buzzing noise) with limited tools in the given audio. Considering that audio editing usually requires expert knowledge of audio engineering or signal processing, we explore a deep learning approach in conjunction with textual queries to lessen audio editing difficulty.

This paper addresses Audio Manipulation on Specific Sources (AMSS), which aims to edit only desired objects that correspond to specific sources, such as vocals and drums, according to a given description while preserving the content of sources that are not mentioned in the description. We formally define AMSS and a structured query language for AMSS in §2. AMSS can be used for many applications such as video creation tools making audio editing easy for non-experts. For example, users can decrease the volume of drums by typing simple textual instructions instead of time-consuming interactions with digital audio workstations.

Although many machine learning approaches have been proposed for audio processing (Ramírez and Reiss, 2018; Martínez Ramírez and Reiss, 2019; Martínez Ramírez et al., 2020; Steinmetz and Reiss, 2021; Wright et al., 2020; Mimilakis et al., 2016; Jansson et al., 2017; Meseguer-Brocal and Peeters, 2019; Choi et al., 2020a; Wisdom et al., 2020; Samuel et al., 2020), to the best of our knowledge, there is no existing method that can directly address AMSS (see §3). This paper proposes a novel end-to-end neural network that performs AMSS according to the given textual query. Designing a neural network for AMSS is straightforward if the sources of a given mixture track are observable. However, we assume that they are not observable because most audio data does not provide them in general. In the assumed environment, modeling AMSS is very challenging because a sound object (e.g., a sample in a wave, a frequency bin in a spectrogram) is ‘transparent’((Wyse, 2017)); a pixel in an image usually corresponds to only a single visual object, whereas a sound object carries information of multiple sources, as shown in Figure 1. Thus, we need different approaches for AMSS from the existing image manipulation techniques.

To address this challenge, we propose a neural network called AMSS-Net that extracts a feature map containing latent sources (see §5.4.1) from the given mixture audio and selectively manipulates them while preserving irrelevant latent sources. We describe the AMSS-Net architecture in §5.

Another challenge is that existing datasets cannot be directly used for supervised training AMSS-Net. If a training dataset of triples $\{(A^{(i)},A^{\prime(i)},S^{(i)})\}_{i=1}^{N}$, where $S^{(i)}$ is an AMSS description, $A^{(i)}$ is a mixture, and $A^{\prime(i)}$ is the manipulated audio according to $S^{(i)}$ is provided, we can train a neural network $net$ in a supervised manner by minimizing $\sum_{i=1}^{N}loss(net(A^{(i)},S^{(i)}),A^{\prime(i)})$, where $loss$ is a distance metric such as $L_{2}$. Unfortunately, there were no datasets currently available that directly target AMSS. To address this issue, we propose a training framework for AMSS (see §4) that uses a ‘source observable multi-track dataset’ such as Musdb18 (Rafii et al., 2017). To generate an AMSS triple on-the-fly, we apply audio transformations onto specific sources of a given multi-track using common methods from Digital Signal Processing (DSP) libraries.

We summarize our contributions as follows:

• •

  Our work is a pioneer study on selective audio manipulation.

• •

  We propose a supervised training framework for AMSS based on source observable multi-track datasets and DSP libraries together with evaluation benchmarks for AMSS.

• •

  We propose AMSS-Net, which performs multiple tasks and outperforms baselines on several tasks.

Since the past decade, data-driven approaches for audio manipulation have been active research fields. Meanwhile, there were virtually no studies that directly addressed AMSS. An exception was an early algorithmic method (Avendano, 2003), which proposed a framework that identifies and controls volumes of desired sources based on digital signal processing. Unlike (Avendano, 2003), this paper presents a machine learning-based approach for various AMSS tasks as well as volume controls. We review the literature related to machine learning-based studies for audio manipulation.

Audio Effect Modeling: As described in (Steinmetz and Reiss, 2021), audio effects are used to modify perceptual attributes of the given audio signal, such as loudness, spatialization, and timbre. Recently, several methods (Ramírez and Reiss, 2018; Martínez Ramírez and Reiss, 2019; Martínez Ramírez et al., 2020; Steinmetz and Reiss, 2021; Wright et al., 2020) have been proposed for audio effect modeling with deep neural networks. (Ramírez and Reiss, 2018) proposed a convolutional network performing equalization (i.e., an audio effect that changes the harmonic and timbral characteristics of audio signals). (Martínez Ramírez and Reiss, 2019; Martínez Ramírez et al., 2020) proposed convolutional and recurrent networks for nonlinear audio effects with Long and Short-Term Memory (Hochreiter and Schmidhuber, 1997), such as distortion and Leslie speaker cabinet. (Steinmetz and Reiss, 2021) presented an efficient neural network for modeling an analog dynamic range compressor enabling real-time operation on CPU.

All the methods above assume an audio input with a single source, while our method assumes a mixture of multiple sources as input. Also, they are task-specific (the trained model provides only one audio effect), while our approach can perform several manipulations with a single model.

Automatic Mixing: Audio mixing is a task that combines multi-track recordings into a single audio track. Before taking the sum of signals, mix engineers usually apply various audio signal processing techniques such as equalization and panning for a better hearing experience. Some approaches have attempted to automate audio mixing with deep neural networks. For example, (Martínez Ramírez et al., 2021a) trained end-to-end neural networks for automatic mixing of drums, where the audio waveform of individual drum recordings is the input of the model and the post-produced stereo mix is the output. (Steinmetz et al., 2020) proposed networks based on temporal dilated convolutions to learn neural proxies of specific audio effects and train them jointly to perform audio mixing. These methods mainly focus on developing expert-type mixing models that combine individual sources into a mixture track, regardless of user control or input for the desired type of transformation. On the other hand, AMSS-Net aims to manipulate user-specified sources in a desired manner, preserving others, where individual sources are not observable.

Unlike (Martínez Ramírez et al., 2021a, b), (Matz et al., 2015) assumed an environment where individual sources are not provided as input. They proposed an algorithmic framework that automatically remixes early jazz recordings, which are often perceived as irritating and disturbing from today’s perspective. It first decomposes the given input into individual tracks by means of acoustic source separation algorithms and remixes them using automatic mixing algorithms. (Mimilakis et al., 2016) proposed a replacement of the source separation and the mixing processes by deep neural networks. The environment assumed in (Mimilakis et al., 2016) is the most similar to ours. However, (Mimilakis et al., 2016) mainly focuses on remixing to change the audio mixing style, while this paper focuses on modifying specified sources in the desired manner. Thus, (Mimilakis et al., 2016) cannot directly address the AMSS problem.

Source Separation: Our work is also related to deep learning-based source separation methods (Jansson et al., 2017; Choi et al., 2020b; Yin et al., 2019; Liu and Yang, 2019; Meseguer-Brocal and Peeters, 2019; Choi et al., 2020a; Samuel et al., 2020). While early methods separate either a single source (Jansson et al., 2017; Choi et al., 2020b; Yin et al., 2019) or multiple sources once (Liu and Yang, 2019), conditioned source separation methods (Meseguer-Brocal and Peeters, 2019; Choi et al., 2020a; Samuel et al., 2020) isolate the source specified by an input symbol. A conditioned source separation task can be viewed as an AMSS task where we want to simply mute all the unwanted sources. We propose a method that can perform several AMSS tasks as well as source separation tasks using a single model, and controlled with a textual query.

We can design an algorithmic framework for AMSS using an existing separation method as a preprocessor, similar to (Matz et al., 2015). Such a method can perform some AMSS tasks by separating the individual sources and applying the appropriate DSP functions to target sources. However, source separation methods also generate minor artifacts, which are not present in the original source. Although they are negligible in each source, they can be large enough to be perceived when we sum the separated results for AMSS.

Instead of such a hybrid framework, we propose an end-to-end neural network since our approach can model more complex transformations such as dereverberation, which cannot be easily modeled with DSP algorithms. Still, our model must have the ability to extract the target sources to be manipulated. Inspired by recent deep learning-based methods, we design a novel model that extracts latent sources, selectively manipulates them, aggregates them, and outputs a mixture that follows a textual query.

The concept of latent source has been introduced in recent source separation methods (Wisdom et al., 2020; Choi et al., 2020a). (Wisdom et al., 2020) have trained their model to separate the given input into a variable number of latent sources, which can be remixed to approximate the original mixture. By carefully taking the weighted sum of separated latent sources, we can extract the desired source, such as clean speech. (Choi et al., 2020a) also use the concept of latent source for conditioned source separation. They proposed the Latent Source-Attentive Frequency Transformation (LaSAFT) method, which extracts the feature map for each latent source and takes the weighted sum of them by using an attention mechanism.

We also use the concept of latent sources in AMSS-Net, where each latent source deals with a more detailed aspect of acoustic features than a symbolic-level source (e.g., ‘vocals’). Similar to (Choi et al., 2020a), we assume that a weighted sum of latent sources can represent a source, while (Wisdom et al., 2020) assumed that latent sources are independent. Unlike previous works, our approach is based on channel-level separation as described in §5.4.1. Each decoding block of AMSS-Net extracts latent source channels from the given feature map so that each channel can be correlated to a latent source.

We propose a novel training framework that uses a multi-track dataset. It generates an AMSS triple $(A,A^{\prime},S)$ on the fly by applying DSP library transformations or audio effects onto target sources of a given multi-track audio file. For instance, suppose that we have a randomly generated query string $S=$‘apply lowpass to drums’ using the CFG and a multi-track that consists of three sources, namely, a vocal track $a_{0}$, a bass track $a_{1}$, and a drum track $a_{2}$. Our training framework takes the linear sum (i.e., $A=\sum a_{j}$) to generate $A$ to generate corresponding input $A$. For the target audio $A^{\prime}$, it computes $\sum_{j\notin\tau}a_{j}+\sum_{j\in\tau}f(a_{j})$, where $\tau$ and $f$ is the set of target sources and DSP function described in $S$, respectively. Our framework applies a Low-pass Filter (LPF) to $a_{2}$, and takes the sum for $A^{\prime}$ as follows: $A^{\prime}=a_{0}+a_{1}+LPF(a_{2})$. By doing so, it can generate an AMSS triple on-the-fly for a given description $S$.

For audio restoration tasks such as dereverberation, we swap A and A’. For example, the framework applies reverb to $a_{2}$, takes the sum for $A^{\prime}=a_{0}+a_{1}+reverb(a_{2})$, and returns $(A^{\prime},A,S)$ instead of $(A,A^{\prime},S)$ for the description “remove reverb from drums.”

The training framework has a set $\mathcal{G}$ of triple generators. A generator $g\in\mathcal{G}$ has a subset of CFG for text query generation and the corresponding DSP function $f$, which is used for computing $A^{\prime}$. $g$ also has an indicator that describes whether $g$ is for applying or removing the effect. For a multi-track $\{A_{j}\}_{j=1}^{n}$ and a set $\mathcal{G}$ of triple generators, our framework generates an AMSS triple by using Algorithm 1, where $n$ denotes the number of sources.

Our AMSS-Net takes as input an audio track $A$ with a text description $S$ and generates the manipulated audio track $\hat{A^{\prime}}$ as shown in Figure 2. It consists of two sub-networks, i.e., a Description Encoder $\textrm{E}_{desc}$ and a Spectrogram Encoder-Decoder Network $SEDN$. $\textrm{E}_{desc}$ extracts word features $w\in\mathbb{R}^{L\times E}$ from $S$, where $E$ denotes the dimension of the word features and $L$ denotes the number of words, to analyze the meaning of $S$. $SEDN$ takes as input word features $w$ and the complex-valued spectrogram $A_{spec}$ of the input audio $A$. Conditioned on the word feature $w$, it estimates the complex-valued spectrogram $\hat{A^{\prime}}_{spec}$, from which $\hat{A^{\prime}}$ can be reconstructed using iSTFT. We train the AMSS-Net by minimizing the $L_{2}$ loss between the ground-truth spectrogram $A^{\prime}_{spec}$ of $A^{\prime}$ and the estimated $\hat{A^{\prime}}_{spec}$.

In this paper, we define a novel task called AMSS. We propose AMSS-Net, which generates feature maps in which each channel deals with a latent source and selectively manipulates them while preserving irrelevant features. AMSS-Net can perform several AMSS tasks, unlike previous models such as LaSAFT-GPoCM-Net. The experimental results show that AMSS-Net outperforms baselines on several tasks. Future work will extend it to provide more complex AMSS tasks such as distortion and reverberation by adopting state-of-the-art methods such as Generative Adversarial Networks (GAN).