Hidden Echoes Survive Training in Audio To Audio Generative Instrument Models

We restrict the focus of our work to audio to audio models, in which a neural network is trained on a corpus and it synthesizes outputs in the style of the corpus. For instance, one could train such a model on a corpus of violins and feed it singing voice audio to create a “singing violin.” The first such technique we use, Differentiable Digital Signal Processing (DDSP) [12] has the simplest architecture out of all of the models. We use the version from the original paper in which the encoder is fixed as a 2 dimensional representation of pitch and loudness, respectively. These dimension are then fed to a decoder network which learns an additive and subtractive synthesizer to best match the training data for a particular pitch/loudness trajectory. The only thing we change is that we use the more recent PESTO [25] instead of CREPE [20] for efficiency, and we use a 3D latent space of loudness, pitch, and pitch confidence. In the end, our DDSP models have $\approx$5 million parameters.

The second most complex model we use is “RAVE” [5], which is a two-stage model that first learns a general audio autoencoder and then improves this autoencoder with generative adversarial training. We use Rave V2, which has $\approx$32 million parameters, and we use snake activations and train with compression augmentation.

The most complex model we use is “Dance Diffusion,” which uses a vanilla diffusion network [28] with attention to progressively denoise outputs from a completely random input. To condition a style transfer to sound more like a particular input $x$, one can jump-start the diffusion process with a scaled $x$ and some added AWGN noise with standard deviation $\eta\in[0,1]$. The closer $\eta$ is to 1, the more the output will take on the character of the corpus on which Dance Diffusion was trained. We use $\eta=0.2$ in all of our experiments, and we use a 81920 sample size, which means the receptive field spans $\approx$1.86 seconds, and the denoising network has $\approx$222 million parameters.

Given a discrete audio “carrier waveform” $x$, audio watermarking techniques hide a binary payload in a watermarked waveform $\hat{x}$ so that $x$ and $\hat{x}$ are perceptually indistinguishable. The original echo hiding paper by [17] accomplishes this by creating two waveforms $x_{0}$ and $x_{1}$, each with a single echo;

where $\alpha<1$ trades off perceptibility and robustness of the watermark, and $\delta_{0},\delta_{1}\leq$ 100 samples at a 44.1khz sample rate. These waveforms are then mixed together in windows to create $\hat{x}$ according to the payload; where $x_{0}$ is fully mixed at the center of a window if the payload contains a 0 at that moment and $x_{1}$ is fully mixed in if the payload contains a 0. For a window of 1024 samples, for instance, this amounts to $\approx$43 bits per second at 44.1khz. Because the echoes are at such a small shift, temporal aliasing of human hearing makes them less noticeable. Furthermore, since convolution in the time domain is multiplication in the frequency domain, the logarithm of the magnitude of the DFT of a window additively separates the frequency response of the echo from the frequency response of $x$. Therefore, the so-called “cepstrum” of a windowed signal $x_{w}$:

yields a signal in which a single echo is a high peak, which is referred to as the “cepstrum” $c$ ⁴⁴4[17] note that it is more mathematically correct take the complex logarithm of the DFT before taking the inverse DFT, and they further enhance with an autocorrelation. But we found better results with the traditional cepstrum.. Thus, to decode the payload from the watermarked signal, one computes $c$ on each window and infers a 0 if $c[\delta_{0}]>c[\delta_{1}]$ or a 1 otherwise.

Since we seek to hide echoes in the training data for generative models, it is unlikely that the models we train will synthesize the windows in the same order they occur in the training set. Therefore, we do away with the windowing completely and instead hide the same echo $\delta$ in the entire audio clip of each waveform in the training data. We then examine the cepstrum $c$ of an entire clip that comes out of our models. To score the cepstrum value at $\delta$ in a loudness-independent way, we compute the z-score at each lag $i$ as follows. First, let $\mu_{c}^{a,b}[i]$ be the mean of $c$ on the interval $[a,b]$, excluding $i$:

and let $\sigma_{c}^{a,b}[i]$ be the analogous standard deviation:

then we define the z-score as:

A model trained on data watermarked with echo $\delta$ works well if $z^{a,b}_{\delta}[\delta]>z^{a,b}_{i}[i],i\neq\delta$. In our experiments, we use $\alpha=0.4$, $\delta\in\{50,76,76,100\}$, $a=25$, and $b=125$. Henceforth, we will assume those parameters and simply refer to these numbers as “the z-scores $z$.” Figure 1 shows example cepstra from clips created with different RAVE[5] models trained on the GuitarSet [32] dataset, with various echoes $\delta$. The peaks and z-scores show that the models reproduce the echoes they were trained on. We will evaluate this more extensively in the experiment section (Figure 4).

Though we have found single echoes to be robust, the information capacity is low. Supposing we use echoes between 50 and 100 at integer values, we can store at most $\approx$5.7 bits of information in a single dataset. To increase the information capacity, we also explore followup work on “time-spread echo hiding” [22] that hides an entire pseudorandom binary sequence $p$ with $L$ bits by scaling, time shifting, and convolving it with the carrier signal $x$:

where, to maintain perceptual transparency, $\alpha$ is generally significantly smaller than it is for a single echo; we use $\alpha=0.01$. To uncover the hidden pattern, one computes the cepstrum $c$ according to Equation 2, and then does a cross-correlation of $c$ with $(2p-1)$ to obtain a signal $c^{*}$. If the echo pattern is well preserved, then $c^{*}[\delta]>c^{*}[i\neq\delta]$.

As in the original echo hiding paper, this work hides $p$ at different offsets $\delta$ in two different signals for hiding a 1 or a 0, but, once again, we hide the same time spread echo at the same lag $\delta=75$ for the entire clip in the training data of our models. We then compute the $z$-score $z_{c^{*}}^{a,b}$ on $c^{*}$ on the model outputs using an equation analogous to Equation 5, though we set $a=3,b=L+\delta$, and we also exclude the samples of $c^{*}$ 3 to the left and 3 to the right when computing $\mu_{c^{*}}^{a,b}$ and $\sigma_{c^{*}}^{a,b}$. Overall, we create 8 different versions of each training set we have, each embedded with a different time spread echo pattern of length $L=1024$. Furthermore, we ensure that the 28 pairwise Hamming distances between the 8 time spread patterns are approximately uniformly distributed between 0 and 1024. Figure 2 shows an example of a style transfer on a model trained on data with the first time spread pattern embedded in all of the training data.

Note that followup work by [33] suggests ensuring that the time spread echo patterns don’t have more than two 0’s or two 1’s in a row, which skews the perturbations in $\hat{x}$ to less perceptible higher frequencies. In this case, one can also compute an enhanced cross-correlation signal as $c^{*}[n]-0.5c^{*}[n-1]-0.5c^{*}[n+1]$. Though we ensured that our time spread echo patterns satisfied this property, we did not find an improvement in our experiments, so we stick to the original cross-correlation z-score.

For these experiments, we train each architecture on each of the original VocalSet, GuitarSet, and Groove datasets, as well as on each of these datasets with an embedded echo of 50, 75, 76, and 100. Figures 3, 5, and 6 show distributions of z-scores for models trained with an echo of 75 and tested with the corresponding stems. Figure 4 shows the mean and standard deviation of z-scores for the MUSDB18-HQ clips over all architectures over all instruments over all echoes. The echoes are quite robust over all architectures. The only weakness is a mixup of the adjacent echoes 75 and 76 for the Dance Diffusion models.

Next, we train RAVE and DDSP on 8 time spread echo patterns embedded in each dataset. We omit fully training dance diffusion with these patterns due to computational constraints and poorer results. Once again, we compute z-scores on the outputs of multiple random clips from the 100 examples in the MUSDB18-HQ training set. To quantify the extent to which each model captures the time spread pattern, we compute z-scores on the $c^{*}$ correlating with the original pattern $p$ on the output cepstra, and we also compute z-scores after correlating with a perturbed version $p^{\prime}$ of $p$ with an increasing number of bits randomly flipped. To quantify how the z-scores change, we compute an ROC curve, where the true positives are z-scores correlating to $p$, and the false positives are correlating to the perturbed versions $p^{\prime}$.

Figure 7 shows an example of this evaluation on a Rave model trained on the first time spread echo pattern embedded in the Groove dataset. The right plot shows the corresponding ROC curves for 512 bits flipped at different durations, as well as ROC curves where the false positives are z-scores in a clean model correlating $p$. Figure 8 shows the AUROC for all 8 pseudorandom patterns when comparing to 512 random bits flipped and when comparing to the clean model. Inter-model comparisons of z-scores are more challenging for the Rave models compared to single echoes due to the variation in embedding strength from model to model. However, within each model we always get a positive slope in AUROC vs bits flipped, and we can always tell the difference with the clean model. This indicates that the correct echo patterns survive training.

We use the train/test/validation set from Groove, and we create our own train/test/validation set for VocalSet (we omit GuitarSet in this experiment because it’s too small). We then embed echoes in the test set and fine tune the corresponding Dance Diffusion trained on the clean training sets, using the validation set to make sure we’re not overfitting. This represents a more realistic scenario than training such a large model from scratch with the same echo in the entire dataset. Figure 9 shows the results, which show some initial promise.

In realistic applications of audio to audio style transfer, it is common to treat the result as a stem and mix it in with other tracks. Hence, we perform a cursory experiment to see the extent to which the synthesized echoes survive mixing and demixing. We use the “hybrid Demucs” algorithm [9] to demix the audio. This demixing model was trained on (among other data) the MUSDB18-HQ training set, so we switch the inputs to the 50 clips from the MUSDB18-HQ test set.

To create our testing data, for each architecture, we input the drums stem to the model trained on Groove with a 50 sample echo, the “other” stem to the model trained on the GuitarSet data with a 75 sample echo, and the vocals stem to the model trained on VocalSet with a 100 echo. We then mix the results together with equal weights and demix them with Demucs into the drums, vocals, and “other” track. Finally, we compute z-scores on each demixed track at echoes of 50, 75, 76, and 100. Figure 10 shows the results. The trends are similar to the overall single echo z-scores in in Figure 4, albeit with slightly weaker z-scores. Still, all of the correct echoes pop out in their corresponding tracks.

Data augmentation is often important to train generalizable models. One form of data augmentation commonly used in audio is pitch shifting. Unfortunately, classical watermarks are known to be quite vulnerable to pitch shifting attacks [19]. Echo hiding is no exception; a shift in pitch up by a factor of $f$ will shift the echo down by a factor of $f$; therefore, we would expect degraded results in the presence of pitch shifting augmentation. To quantify this, we design an experiment training RAVE on the Guitarset data embedded with a single echo at 75 samples, for varying degrees of pitch augmentation, and we test on the MUSDB18-HQ dataset as before. Pitch shifting is disabled by default in RAVE, but when it is enabled, it randomly pitch shifts a clip 50% of the time with simple spline interpolation at the sample level. We modify the RAVE code to use higher quality pitch shifting with the Rubberband Library [6], and we enable a variable probability for pitch shifting. When pitch shifting happens for a clip in a batch, we pick a factor uniformly at random in the interval $[0.75,1.25]$. Figure 11 shows z-scores for training RAVE with an increasing probability of pitch shift augmentation, along with AUROC scores using the clean model to generate the false positive distribution. As expected, the results degrade with increasing amounts of pitch shifting, though for the default value of 50% pitch shifting, the z-scores are still quite far from the clean distribution. Surprisingly, even at 90% pitch shifting, the z-scores are still significant.

We perform a preliminary experiment tagging a dataset with two different echoes depending on timbre: we tag all but one of the males in VocalSet with a 50 echo and all but one of the females in the dataset with a 75 echo. As Figure 12 shows, when we test with the remaining male and female, the z-scores of the corresponding echoes are higher.