TriAAN-VC: Triple Adaptive Attention Normalization
for Any-to-Any Voice Conversion

Encoder. Each encoder layer consists of a convolution block and Instance Normalization (IN). Speaker Attention (SA) is used only in the speaker encoder. The convolution block is designed as a residual block comprising two convolution layers with a kernel size of 3 and a stride of 1.

Bottleneck layer. After the encoder process, f0 of the source speaker $x_{f0}\in\mathbb{R}^{T}$ is used to represent the pitch. Given $x^{\prime}_{c}\in\mathbb{R}^{C\times T}$ is the output of the content encoder, we apply a Gated Recurrent Unit (GRU) layer on the concatenated outputs between $x^{\prime}_{c}$ and $x_{f0}$. Before the decoder, the initial converted representation is generated by applying Dual Adaptive Normalization (DuAN), a conversion method described in Section 2.4, to the content and speaker representations.

Decoder. Each decoder layer contains a convolution block, the same as that of encoders, and TriAAN block. TriAAN block conducts conversion using the content feature from the previous layer and gathered feature maps from the speaker encoder layers. Finally, the outputs of the decoder are refined by GRU layers and PostNet [18] to predict the log mel-spectrogram $\hat{y}\in\mathbb{R}^{M\times T}$, where $M$ is the number of mel bins.

DuAN. Inspired by adaptive attention normalization in image style transfer [20], we design DuAN to extract detailed speaker features and to conduct conversion. DuAN represents attention-based statistics of layer-wise speaker features $F_{s}^{l}$. Given $x_{c}\in\mathbb{R}^{T\times C}$ is the content feature from the previous layer and $\mathcal{N}(\ \cdot\ )$ is the normalization function, the attention weight $\alpha\in\mathbb{R}^{T\times T}$, attention-weighted mean $M\in\mathbb{R}^{T\times C}$, and variance $Var\in\mathbb{R}^{T\times C}$ are defined as follows:

$W_{q,k,v}\in\mathbb{R}^{C\times C}$ denotes each weight for linear transformation. $\alpha$, obtained by the normalized feature, represents the similarity between the content and speaker features. Furthermore, $Var$ is calculated using the expectation of variables and the square of the variable expectations. By applying $\alpha$, the weighted mean and variance contain detailed speaker features, that is per-point statistics. To prevent biased results with excessively detailed speaker features, we take the time-wise average of $M$ and $Var$, followed by applying a square root on $Var$ to obtain the standard deviation $S\in\mathbb{R}^{C}$. The converted representation $x^{\prime}_{c}\in\mathbb{R}^{T\times C}$, obtained by adaptive normalization and the content feature, is defined as $x^{\prime}_{c}=\mathrm{IN}(x_{c})S+M$. To perform it in terms of channel and time, we separate the adaptive normalization process depending on $\mathcal{N}(\ \cdot\ )$ function (i.e., IN and TIN), making two converted representations.

GLAN. To represent the global speaker information, we utilize all feature maps $F_{s}^{1:L}$ from the speaker encoder. For the content feature $x_{c}\in\mathbb{R}^{C\times T}$ in GLAN, we apply a convolution layer to the channel-wise concatenation of two converted representations from DuAN. We obtain layer-wise concatenated means $\mu\in\mathbb{R}^{L\times C}$ and standard deviations $\sigma\in\mathbb{R}^{L\times C}$ from $F_{s}^{1:L}$, defined as $\mu=[\mathrm{avg}(F_{s}^{1});...,;\mathrm{avg}(F_{s}^{L})]$ and $\sigma=[\mathrm{std}(F_{s}^{1});...,;\mathrm{std}(F_{s}^{L})]$. To extract core statistics from global speaker features (i.e., $\mu$ and $\sigma$) for adaptive normalization, we adopt self-attention pooling [21], which emphasizes important speaker statistics. The attention pooling process used to obtain the weighted mean $\mu^{\prime}\in\mathbb{R}^{C}$ and standard deviation $\sigma^{\prime}\in\mathbb{R}^{C}$ is as follows:

$\alpha_{\mu,\sigma}\in\mathbb{R}_{\mu,\sigma}^{L}$ and $W_{\mu,\sigma}\in\mathbb{R}_{\mu,\sigma}^{C\times C}$ denote attention weights and each weight for transformation. Then, adaptive normalization as in DuAN is applied with $\mu^{\prime}$ and $\sigma^{\prime}$ for conversion.

To train TriAAN-VC, we combine reconstruction loss and siamese loss. Reconstruction loss is $L1$ loss between the ground truth mel-spectrogram $y\in\mathbb{R}^{M\times T}$ and predicted mel-spectrogram $\hat{y}\in\mathbb{R}^{M\times T}$. $y$ is extracted from the raw audio using a mel-spectrogram transformation, and $\hat{y}$ is predicted by the proposed model when input features $x\in\mathbb{R}^{H\times T}$ are CPC features of the raw audio. For siamese loss, $L1$ loss is applied between $y$ and $\hat{y}_{siam}\in\mathbb{R}^{M\times T}$, where $\hat{y}_{siam}$ is predicted by the model with $x$ augmented by time masking. Given $L1$ loss is $loss(y,\hat{y})=||y-\hat{y}||_{1}/T$, the combined loss is as follows:

By calculating the additional loss with $\hat{y}_{siam}$, the robustness and consistency of the model can be improved. In particular, since time masking removes content information during training, the loss with the siamese branch makes the model robust for maintaining content information.

Further experiment. As ablation studies, we analyze the contributions of the proposed components. As listed in rows 1-3 of Table 2, the use of each CPC feature and siamese loss contributed significantly to the performance gain of WER and SV. In rows 4-6 of Table 2, we excluded one of the components of TriAAN-VC without siamese loss. The results suggested that SA particularly contributed to the improvement of WER, and TriAAN block was the crucial component for the performance gain of SV. Although each component contributed to the performance gain in WER or SV, they also suffered from the trade-off problem, implying all the components are necessary to mitigate the trade-off problem. In addition to one-shot VC, TriAAN-VC was effective in multi-utterance scenarios. Under the multiple utterance setting using more than one target utterance, TriAAN-VC with CPC improved its performance by about 4% and 5% on WER and SV, compared to the one-shot VC results.