An Investigation of Time-Frequency Representation Discriminators for High-Fidelity Vocoder

Short-Time Fourier Transform (STFT) [33] converts a time-domain signal $x$ into its TFR, as:

where $k$ is the index of frequency bin, $N_{k}$ is the window length, $x(t)$ denotes the $t$-th sample point, $w(t)$ is the window function, and $Q_{k}$ is the Q-factor, which is defined as,

where $f_{k}$ is the center frequency, $\Delta f_{k}$ is the bandwidth. The center frequency $f_{k}$ can be obtained as:

where $N_{fft}$ is the number of FFT bins. The bandwidth $\Delta f_{k}$, which determines the TF resolution trade-off, is defined as:

where $f_{s}$ is the sampling rate.

Although STFT can be easily implemented to model a speech signal, it also has drawbacks when encountering expressive speech or singing voice due to its characteristics (Table I), which are listed as follows:

• •

  Fixed TF resolution: In STFT, the window length $N_{k}$ is fixed, bringing the $\Delta f_{k}$ a constant (Eq. 4). This means the TF resolution is fixed for all frequency bins. Thus, STFT is not a good candidate to model signals that require a dynamic resolution for different frequency bands.

• •

  Limitation in harmonics modeling: The center frequency $f_{k}$s are linearly distributed in STFT (Eq. 3), which is incompatible with signals made up of harmonic frequency components that require geometrically distributed center frequencies for accurate modeling.

• •

  Inaccurate modeling of short-time transients: According to the Gibbs phenomenon [38], when converging a square wave using the decomposition basis $e^{-i2\pi nk}$ in STFT, no matter how many harmonics are used, a non-neglectable approximation error will always exist, meaning a poor modeling ability in short-time transients.

Constant-Q Transform (CQT) [34] converts a time-domain signal $x$ into its TFR with a constant Q-factor, as:

where $a_{k}(n)$ is a complex-valued convolutional kernel, and $a_{k}^{\ast}(n)$ is the complex conjugate of $a_{k}(n)$. $\lfloor\cdot\rfloor$ denotes rounding down. The kernels $a_{k}(n)$ can be obtained as:

where the quality factor $Q_{k}$s are defined constantly to conform with the human auditory system [39, 40]:

where $B$ is the number of bins per octave. The center frequencies $f_{k}$ in CQT are defined as:

where $f_{1}$ is the lowest center frequency.

Compared with STFT, CQT overcomes the fixed TF resolution and the linearly-distributed center frequencies (Table I), resulting in the following advantages:

• •

  Dynamic TF resolution: CQT utilizes a constant Q-factor (Eq. 7) to obtain a dynamic TF resolution across different frequency bins. Thus, as illustrated in Fig. 1, the low-frequency bands will have a smaller $\Delta f_{k}$, bringing a higher frequency resolution, which could model the F0 more accurately; The high-frequency bands will have a bigger $\Delta f_{k}$, bringing a higher time resolution, which could track fast-changing harmonics variations better.

• •

  Enhanced capability in harmonic modeling: For a better modeling ability with the sound that is made up of harmonic components, CQT utilizes a series of geometrically distributed center frequencies (Eq. 8). In our study, we set $f_{1}$ to 32.7 Hz (C1) to ensure the center frequencies conform with the notes in Western Music.

Continuous Wavelet Transform (CWT) [36] converts a time-domain signal $x$ into its TFR with shifting and scaling, as:

where $a_{k}$ is the scaling factor, $\psi(n)$ is the mother wavelet, and $\frac{1}{|a_{k}|^{\frac{1}{2}}}\psi(\frac{j-n}{a_{k}})$ is the scaled child wavelet.

By taking the FT between the mother wavelet and the child wavelet, the bandwidth $\Delta f_{k}$ can be obtained as:

where $\Delta f$ is the bandwidth of the mother wavelet. The relationship between the scaling factor $a_{k}$ and the center frequency $f_{k}$ is defined as:

thus also obtaining a constant Q-factor:

Compared with STFT, CWT tackles the issue brought by the constant TF resolution and the fixed decomposition basis (Table I), bringing the following benefits:

• •

  Dynamic TF resolution: CWT owns a constant Q-factor (Eq. 12), which brings a dynamic TF resolution (Fig. 1).

• •

  More diverse TFRs: In CWT, the decomposition basis $\psi(n)$ is a variable. Hence, it is possible to decompose a signal with different $\psi(n)$s for more diverse TFRs. Specifically, to model phase information, this study employs the Complex Morlet Wavelet (CMOR) [41] and the Complex Gaussian Wavelet Family (CGAU) [42].

• •

  Enhanced capability in modeling of short-time transients: The energy-centralized characteristic of wavelet basis also guarantees CWT a better modeling ability in short-time transients. Fig. 2 compares the decomposition basis in STFT and CWT when modeling a square wave. Wavelet basis can achieve a reconstructed signal with the same number of components but a significantly smaller error in the places where the ”step transients” occur due to its ”harsher shape,” hence showing a better modeling ability in short-time transients.

Multi-Scale Processing applies Sub-Discriminators operating on diverse TFRs to reduce the bias brought by the fixed pattern of a specific TF analysis technique, which has been widely used [24, 23, 5, 25, 26]. Motivated by the concept of Multi-Scale Processing, we apply Multi-Resolution Processing to both CQT- and CWT-based discriminators and propose Multi-Basis Processing for CWT-based discriminators only.

As two sides of a coin, although the dynamic $\Delta f_{k}$ brings flexible TF resolution, it also brings the unfixed $N_{k}$, which can be obtained from Eq. (2), (4), and (7):

Thus, in a fixed time step $t$, the kernels $a_{k}(t)$ in different frequency bins will convolute different amounts of sample points of the original signal as shown in Eq. 5. This means the convolutional kernels are not temporally synchronized [35] and will cause artifacts in the resulting spectrogram visualized in the bottom right of Fig. 3.

To alleviate this problem, [35] designs a series of temporally synchronized kernels within each octave. This algorithm has also been widely used in toolkits like librosa [44] and nnAudio [45]. However, such an algorithm only makes the $a_{k}(t)$ of intra-octave temporally synchronized but leaves those of inter-octave unsolved. Training our neural vocoder using features with such a bias will cause a burden to the adversarial training process and harm the resulting synthesis quality.

Based on that, we utilize the philosophy of representation learning and design the Sub-Band Processor (SBP) module to address this problem further (Fig. 3). In particular, the real or imaginary part of a CQT spectrogram will first be split into sub-bands according to different octaves. Then, each band will be sent to a layer to get its representation. Finally, we concatenate the representations from all bands to obtain the latent representation of the CQT spectrogram. In the upper right of Fig. 3, it can be observed that the SBP module successfully learns the desired representation that is temporally synchronized among all the frequency bins.

Although the variable wavelet basis $\psi(n)$ brings better TFR diversity and modeling ability in short-time transients, it also has drawbacks. Specifically, the characteristics of wavelet basis bring the requirement of ”unit hop length” [46] for alleviating information loss and maintaining good invertibility. Thus, the resulting CWT spectrogram from an audio signal of shape (1, T) will have a shape of (Scales, T). Such a large spectrogram will be incompatible with deep learning applications with limited GPU memory.

To reduce the memory complexity, instead of directly utilizing the raw CWT spectrogram, we use the learnable compressed latent representation obtained via our designed Temporal Compressor (TC) module (Fig. 4). Specifically, the real or imaginary part of the CWT spectrogram will first be sent scale-wise to a series of Conv2D layers, which use temporal-overlapped convolutional windows to maintain a good continuity between frames, to get its temporal-compressed representation. Then, we concatenate the representations from all scales to obtain the resulting latent representation of the CWT spectrogram. In the upper right of Fig. 4, it can be observed that our proposed TC module successfully compressed the spectrogram while maintaining the overall energy distribution.

Our proposed discriminators can be easily integrated with any GAN-based vocoders without interfering with the inference stage. We take HiFi-GAN [24] as an example. HiFi-GAN has a generator $G$ and multiple discriminators $D_{m}$. The generation loss $\mathcal{L}_{G}$, and discrimination loss $\mathcal{L}_{D}$ are as follows:

where M is the number of discriminators, $D_{m}$ denotes the m-th discriminator, $\mathcal{L}_{adv}$ is the adversarial GAN loss, $\mathcal{L}_{fm}$ is the feature matching loss, and $\mathcal{L}_{mel}$ is the mel spectrogram reconstruction loss.

Among these losses, only $\mathcal{L}_{fm}$ and $\mathcal{L}_{adv}$ are related to our discriminator. Suppose we want to integrate a new discriminator $D_{\text{new}}$ in the training process, just adding $\mathcal{L}_{adv}(G;D_{\text{new}})+2\mathcal{L}_{fm}(G;D_{\text{new}})$ to $\mathcal{L}_{G}$ and $\mathcal{L}_{adv}(D_{\text{new}};G)$ to $\mathcal{L}_{D}$ can finish the integration process.

To verify the effectiveness of the proposed discriminators, we take HiFi-GAN with MPD and MSD as the baseline model and enhance it with different discriminators. The results of the analysis-synthesis are illustrated in Table III.

Regarding singing voice, we can observe that: (1) HiFi-GAN (+S), HiFi-GAN (+C), and HiFi-GAN (+W) all outperform HiFi-GAN both subjectively and objectively, confirming the importance of the extra adversarial losses in the frequency domain [32]; (2) Both HiFi-GAN (+C) and HiFi-GAN (+W) outperform the HiFi-GAN (+S) objectively and subjectively, illustrating the effectiveness of utilizing TFRs with dynamic TF resolution; (3) HiFi-GAN (+C) outperforms HiFi-GAN (+W) objectively especially on F0-related metrics, showing the effectiveness of the pitch-aware center frequency distribution. HiFi-GAN (+W) outperforms HiFi-GAN (+C) subjectively, showing the effectiveness of the diverse energy-centered wavelet basis; (4) HiFi-GAN (+S+C+W) outperforms both objectively and subjectively on seen singers while holding better objective results with a similar subjective score on unseen singers, confirming the effectiveness of joint training.

Regarding speech, we can observe that: (1) For seen speakers, HiFi-GAN (+S+C+W) performs best objectively with a similar MOS score with HiFi-GAN (+W) (best) and HiFi-GAN (+C) (second best), illustrating the effectiveness of our proposed methods. (2) For unseen speakers, HiFi-GAN (+S+C+W) performs both objectively and subjectively best, indicating the enhanced generalization ability via utilizing different TFR-based discriminators. (3) Specifically, HiFi-GAN (+C) holds a comparatively higher MOS score in seen speakers but a lower MOS score in unseen speakers. We speculate that although CQT has a dynamic TF resolution which brings a better modeling ability, its generalization ability is degraded due to the incompatibility between the pitch-aware center frequency and speech (non-musical) signal.

To further explore the specific benefits of using the STFT-based, CQT-based, and CWT-based discriminators jointly, we conducted a case study as illustrated in Fig. 5. Notably, in the displayed low-frequency parts, STFT has a better time resolution, while CQT and CWT have a better frequency resolution. It can be observed that: (1) Regarding TF resolution (upper rectangles), HiFi-GAN with MS-SB-CQT Discriminator or with MS-TC-CWT Discriminator (Fig. 5(b), Fig. 5(c)) can reconstruct its frequency accurately but ”flattened” harmonic component due to the insufficient time resolution, while HiFi-GAN with MS-STFT Discriminator (Fig. 5(a)) can model the transients in the harmonic component but at the expanse of inaccurate pitch information due to the insufficient frequency resolution; (2) Additionally (bottom rectangles), HiFi-GAN with MS-TC-CWT Discriminator (Fig. 5(c)) can achieve a glitch-free spectrogram, showing its effectiveness in modeling short-time transients. (3) Integrating those three discriminators combines their strengths and thus achieves a better-reconstructed spectrogram (Fig. 5(d)), showing the effectiveness of joint training.

To verify the generalization ability of the complementary role between the STFT-, CQT-, and CWT-based discriminators, we also conduct experiments under NSF-HiFiGAN, BigVGAN, and APNet, as presented in Table IV.

It is illustrated that: (1) In general, the performance of NSF-HiFiGAN, BigVGAN, and APNet can be improved significantly by jointly training with MS-SB-CQT, MS-TC-CWT, and MS-STFT Discriminators, with both improved objective evaluation scores and subjective preference tests confirming the effectiveness; (2) Regarding NSF-HiFiGAN, although it can synthesis high-fidelity singing voices, it still lacks the modeling abilities of high-frequency band details. Adding adversarial losses based on diverse TFRs alleviates that problem, making synthesized samples closer to the ground truth. Subjective results with a higher preference percentage demonstrate the effectiveness; (3) For BigVGAN, although using the Snake activation function enhanced the generalization ability, it also brings artifacts in phase modeling. Since all MS-STFT, MS-SB-CQT, and MS-TC-CWT Discriminators utilize the phase information, the extra adversarial losses alleviate this problem. Improved subjective scores with significantly higher preference percentages illustrated the effectiveness; (4) As for APNet, although maintaining all the operations in the frame level significantly reduces the memory and computational costs, the difficulty in modeling phase information also brings quality degradation with metallic sound, especially regarding those Aperiodic Parts. Adding MS-STFT, MS-SB-CQT, and MS-TC-CWT Discriminators alleviates that problem, bringing in significantly better FPC and F0-RMSE. We believe this is because the reduction of the metallic sound greatly boosts the performance of the F0-detection algorithm. The subjective evaluation with a higher preference score also confirms its effectiveness. Representative cases regarding these findings can be found on our demo page\@footnotemark.

We propose the Sub-Band Processor module to obtain the temporally synchronized CQT latent representations, and the Multi-Basis Processing technique reduces the bias and thus improves the effectiveness further. We conduct an ablation study of those two methods to verify their necessity. The results of the analysis-synthesis are illustrated in Table V.

Regarding the CQT-based discriminators, we can see that our proposed MS-SB-CQT Discriminator significantly outperforms the one without the Sub-Band Processor. We speculate this is because the convolutional kernel in the Conv2D layer cannot handle properly the temporal desynchronization in the inter-octaves parts of the CQT spectrogram in the initial stage, which would cause a burden and eventually harm the overall audio quality. Regarding the CWT-based discriminators, we can see that our proposed MS-TC-CWT Discriminator performs better than the one that utilizes only one wavelet (we use the CMOR wavelet in the experiment), showing the effectiveness of utilizing different wavelet bases.

We also conducted a case study on the learned representation in the MS-SB-CQT and MS-TC-CWT Discriminators. Regarding the representation in MS-TC-CWT Discriminator, only compression is learned. Regarding the representation in MS-SB-CQT Discriminator, however, apart from the ability to obtain a temporally synchronized spectrogram, it also learns to apply dynamic attention to different frequency bands (Fig. 6).

Notably, among these three vocoders, HiFi-GAN performs averagely across all frequency bands; NSF-HiFiGAN performs better in the low-frequency bands due to its glitch-free ability and accurate pitch modeling but worse in the high-frequency bands due to the aliasing effects; BigVGAN performs better in the high-frequency bands due to its aliasing-free ability but worse in the low-frequency bands due to glitches. It can be observed that: (1) All the SBP modules trained with different generators learned to obtain the temporally synchronized representation (blue rectangle) compared with the ground truth CQT spectrogram; (2) The SBP module trained with NSF-HiFiGAN and BigVGAN learn to mask the low-frequency band and high-frequency band individually(red rectangle). We speculate it is to ignore the frequency band where the generator already has a good synthesis quality and to focus on the frequency bands with a poor reconstruction quality to discriminate whether the audio is generated or not, which indicates the SBP module learns applying dynamic attention to different frequency bands.