Multi-Scale Sub-Band Constant-Q Transform Discriminator
for High-Fidelity Vocoder

In this section, the strengths of CQT will be exhibited by introducing its design idea. We will see how CQT owns a flexible time-frequency resolution and why it can model pitch-level information better.

Following [18], the CQT $\boldsymbol{X}^{cq}(k,n)$ can be defined as:

$$\begin{split}\boldsymbol{X}^{cq}(k,n)=\sum_{j=n-\lfloor N_{k}/2\rfloor}^{n+\lfloor N_{k}/2\rfloor}x(j)a_{k}^{\ast}(j-n+N_{k}/2),\end{split}$$

(1)

where $k$ is the index of frequency bin, $x(j)$ is the $j$-th sample point of the analyzed signal, $N_{k}$ is the window length, $a_{k}(n)$ is a complex-valued kernel, and $a_{k}^{\ast}(n)$ is the complex conjugate of $a_{k}(n)$.

The kernels $a_{k}(n)$ can be obtained as:

$$\begin{split}a_{k}(n)=\frac{1}{N_{k}}w\left(\frac{n}{N_{k}}\right)e^{-i2\pi n\frac{Q_{k}}{N_{k}}},\end{split}$$

(2)

where $w(t)$ is the window function, and $Q_{k}$ is the constant Q-factor:

$$Q_{k}\overset{\textit{ref.}}{=}\frac{f_{k}}{\Delta f_{k}}=(2^{\frac{1}{B}}-1)^{-1},$$

(3)

where $f_{k}$ is the center frequency, $\Delta f_{k}$ is the bandwidth determining the resolution trade-off, and $B$ is the number of bins per octave.

Notably, for STFT, the $\Delta f_{k}$ is constant, meaning the time-frequency resolution is fixed for all frequencies. However, for CQT, its main idea is to keep $Q_{k}$ constant. As a result, the low-frequency bands will have a smaller $\Delta f_{k}$, bringing a higher frequency resolution, which could model the pitch information better. Besides, the high-frequency bands will have a bigger $\Delta f_{k}$, bringing a higher time resolution, which could track fast-changing harmonics variations better.

In Eq. (2), the window length of the $k$-th frequency bin, $N_{k}$, can be obtained as:

$$N_{k}=\frac{f_{s}}{\Delta f_{k}}=\frac{f_{s}}{f_{k}}\cdot(2^{\frac{1}{B}}-1)^{-1}$$

(4)

where $f_{s}$ is the sampling rate, and $f_{k}$ is defined as:

$$\begin{split}f_{k}=f_{1}\cdot 2^{\frac{k-1}{B}},\end{split}$$

(5)

where $f_{1}$ is the lowest center frequency, which is set to 32.7 Hz (C1) in our study.

To capture the information under more diverse time-frequency resolutions, we leverage the multi-scale idea [8, 15] and adopt sub-discriminators on CQTs with different overall resolution trade-offs.

Given Eq. (3) and (5), we can observe that the bandwidth $\Delta f_{k}$, which determines the resolution trade-off, is dependent on the number of bins per octave $B$. In other words, we can set the different $B$ to obtain the different resolution distributions. Based on that, we follow [15, 16] to apply three sub-discriminators with $B$ equals 24, 36, and 48, respectively.

As two sides of a coin, although the dynamic bandwidth $\Delta f_{k}$ brings flexible time-frequency resolution, it also brings the unfixed window length $N_{k}$. As a result, the kernels $a_{k}(n)$ in different frequency bins are not temporally synchronized [20]. The CQT spectrogram with such artifacts has been visualized in the bottom right of Fig. 1.

To alleviate this problem, [20] designs a series of kernels that are temporally synchronized within an octave. This algorithm has also been used in toolkits like librosa [21] and nnAudio [22]. However, such an algorithm only makes the $a_{k}(n)$ of intra-octave temporally synchronized but leaves those of inter-octave unsolved. During experiments, we found that just using CQT spectrograms with such a bias could even hurt the quality of vocoders (Section 3.4).

Based on that, we utilize the philosophy of representation learning and design the Sub-Band Processing (SBP) module to address this problem further. In particular, the real or imaginary part of a CQT spectrogram will first be split into sub-bands according to octaves. Then, each band will be sent to its corresponding convolutional layer to get its representation. Finally, we concatenate the representations from all bands to obtain the latent representation of the CQT spectrogram. In the upper right of Fig. 1, it can be observed that our proposed SBP successfully learns the temporally synchronized representations among all the frequency bins.

Our proposed discriminator can be easily integrated with existing GAN-based vocoders without interfering with the inference stage. We take HiFi-GAN [8] as an example. HiFi-GAN has a generator $G$ and multiple discriminators $D_{m}$. The generation loss $\mathcal{L}_{G}$, and discrimination loss $\mathcal{L}_{D}$ are defined as, $\mathcal{L}_{G}=\sum_{m=1}^{M}[\mathcal{L}_{adv}(G;D_{m})+2\mathcal{L}_{fm}(G;D_{m})]+45\mathcal{L}_{mel},\mathcal{L}_{D}=\sum_{m=1}^{M}[\mathcal{L}_{adv}(D_{m};G),$ where M is the number of discriminators, $D_{m}$ denotes the m-th discriminator, $\mathcal{L}_{adv}$ is the adversarial GAN loss, $\mathcal{L}_{fm}$ is the feature matching loss, and $\mathcal{L}_{mel}$ is the mel spectrogram reconstruction loss. Among these losses, only $\mathcal{L}_{fm}$ and $\mathcal{L}_{adv}$ are related to our discriminator. Thus, just adding $\mathcal{L}_{adv}(G;D_{\text{MS-SB-CQT}})+2\mathcal{L}_{fm}(G;D_{\text{MS-SB-CQT}})$ to $\mathcal{L}_{G}$ and $\mathcal{L}_{adv}(D_{\text{MS-SB-CQT}};G)$ to $\mathcal{L}_{D}$ can integrate the proposed discriminator in the training process.

Dataset  The experimental datasets contain both speech and singing voices. For the singing voice, we adopt M4Singer [23], PJS [24], and one internal dataset. We randomly sample 352 utterances from the three datasets to evaluate seen singers and leave the remaining for training (39 hours). 445 samples from Opencpop [25], PopCS [26], OpenSinger [27], and CSD [28] are chosen to evaluate unseen singers. For the speech, we use the train-clean-100 from LibriTTS [29] and LJSpeech [30]. We randomly sample 2316 utterances from the two datasets to evaluate seen speakers and leave the remaining for training (about 75 hours). 3054 samples from VCTK [31] are chosen to evaluate unseen speakers.

Implementation Details  The CNN in SBP uses a Conv2D with kernel size of (3, 9). The CNNs in each Sub-Discriminator consist of a Conv2D with kernel size (3, 8) and 32 channels, three Conv2Ds with dilation rates of [1, 2, 4] in the time dimension and a stride of 2 over the frequency dimension, and a Conv2D with kernel size (3, 3) and stride (1, 1). For CQT, the global hop length is empirically set to 256, and the waveform will be upsampled from $f_{s}$ to $2f_{s}$ before the computation to avoid the $f_{max}$ of the top octave hitting the Nyquist Frequency.

Evaluation Metrics  For objective evaluation, we use Perceptual Evaluation of Speech Quality (PESQ) [32] and Mel Cepstral Distortion (MCD) [33] to evaluate the spectrogram reconstruction. We use F0 Root Mean Square Error (F0RMSE) and F0 Pearson Correlation Coefficient (FPC) for evaluating pitch stability. The Mean Opinion Score (MOS) and Preference Test are used for subjective evaluation. We invited 20 volunteers who are experienced in the audio generation area to attend the subjective evaluation. Each setting in the bellowing MOS test has been graded 200 times, and each pair in the preference test has been graded 120 times.

To verify the effectiveness of the proposed discriminator, we take HiFi-GAN as an example and enhance it with different discriminators. The results of the analysis-synthesis are illustrated in Table 1. Regarding singing voice, we can observe that: (1) both HiFi-GAN (+C) and HiFi-GAN (+S) perform better than HiFi-GAN, showing the importance of time-frequency-representation-based discriminators [15]; (2) HiFi-GAN (+C) performs better than HiFi-GAN (+S) with a significant boost in MOS, showing the superiority of our proposed MS-SB-CQT Discriminator; (3) HiFi-GAN (+S+C) performs best both objectively and subjectively, which shows that different discriminators will have complementary information for each other, confirming the effectiveness of jointly training. A similar conclusion can be drawn for the unseen speaker evaluation of speech data.

To further explore the specific benefits of using the CQT-based and the STFT-based discriminators jointly, we conducted a case study (Fig. 2). Notably, in the displayed high-frequency parts, STFT has a better frequency resolution, and CQT has a better time resolution. Regarding the frequency parts in the rectangle, it can be observed that: (1) HiFi-GAN with MS-STFT Discriminator (Fig. 2(b)) can reconstruct its frequency accurately but cannot track the changes due to insufficient time resolution; (2) HiFi-GAN with MS-SB-CQT Discriminator (Fig. 2(c)) can track the harmonics, but the frequency reconstruction is inaccurate due to the low-frequency resolution; (3) Integrating those two combines their strengths and thus achieve a better reconstruction quality (Fig. 2(d)).

To verify the generalization ability of the proposed MS-SB-CQT Discriminator, besides HiFi-GAN, we also conduct experiments under MelGAN²²2https://github.com/descriptinc/melgan-neurips/ [6] and NSF-HiFiGAN³³3https://github.com/nii-yamagishilab/project-NN-Pytorch-scripts. Note that NSF-HiFiGAN is one of the state-of-the-art vocoders for singing voice [34]. It combines the neural source filter (NSF) [14] to enhance the generator of HiFi-GAN. The experimental results are presented in Table 2.

It is illustrated that: (1) In general, the performance of MelGAN and NSF-HiFiGAN can be improved significantly by jointly training with MS-SB-CQT and MS-STFT Discriminators, with both objective and subjective preference tests confirming the effectiveness; (2) In particular, MelGAN tends to overfit the low-frequency part and ignore mid and high-frequency components, resulting in audible metallic noise. After adding MS-STFT and MS-SB-CQT Discriminators, it could model the global information of spectrogram better⁴⁴4We show the representative cases on the demo page., bringing in significantly better MCD and PESQ. Although the low-frequency-related metrics worsen, the preference test shows that the overall quality has remarkably increased; NSF-HiFiGAN can synthesize high-fidelity singing voices. However, it still lacks frequency details. Adding MS-STFT and MS-SB-CQT Discriminators tackles that problem⁴, making synthesized samples closer to the ground truth. Subjective results with a higher preference percentage also demonstrate the effectiveness.

As introduced in Section 2.3, we propose the Sub-Band Processing module to obtain the temporally synchronized CQT latent representations. To verify the necessity of it, we conduct an ablation study that removes the SBP module from the proposed MS-SB-CQT Discriminator. We adopt Opencpop [25] as the experimental dataset. We randomly selected 221 utterances for evaluation and the remaining for training (about 5 hours).

In Tabel 3, we can see that HiFi-GAN can be enhanced successfully by our proposed MS-SB-CQT Discriminator. However, just applying the raw CQT to the discriminator (MS-CQT) would even harm the quality of HiFi-GAN. We speculate this is because the temporal desynchronization in inter-octaves of the raw CQT would burden the model learning. Therefore, it is necessary to adopt the proposed SBP module for designing a CQT-based discriminator.