Lifter training and sub-band modeling for computationally efficient and high-quality voice conversion using spectral differentials

Let $\textrm{\boldmath$F$}^{(\mathrm{X})}=[{\textrm{\boldmath$F$}^{(\mathrm{X})}_{1}}^{\top},...,{\textrm{\boldmath$F$}^{(\mathrm{X})}_{t}}^{\top},...,{\textrm{\boldmath$F$}^{(\mathrm{X})}_{T}}^{\top}]^{\top}$ be a complex frequency spectrum sequence obtained by applying the short-time Fourier transform (STFT) to an input speech waveform, where $t$ represents the frame index and $T$ is the total number of frames. For the simplicity, we now focus on the frame $t$. A low-order real cepstrum $\textrm{\boldmath$C$}_{t}^{(\mathrm{X})}$ can be extracted from $\textrm{\boldmath$F$}_{t}^{(\mathrm{X})}$ [14]. The DNNs then estimate a real cepstrum of differential filter $\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ from $\textrm{\boldmath$C$}_{t}^{(\mathrm{X})}$. The loss function for $t$ is calculated as $L_{t}=(\textrm{\boldmath$C$}_{t}^{(\mathrm{Y})}-\hat{\textrm{\boldmath$C$}}_{t}^{(\mathrm{Y})})^{\top}(\textrm{\boldmath$C$}_{t}^{(\mathrm{Y})}-\hat{\textrm{\boldmath$C$}}_{t}^{(\mathrm{Y})})$, where $\hat{\textrm{\boldmath$C$}}_{t}^{(\mathrm{Y})}$ is a real cepstrum of converted speech given as $\hat{\textrm{\boldmath$C$}}_{t}^{(\mathrm{Y})}=\textrm{\boldmath$C$}_{t}^{(\mathrm{X})}+\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$, and $\textrm{\boldmath$C$}_{t}^{(\mathrm{Y})}$ is a real cepstrum of the target speech. The DNNs are trained to minimize the loss function for all time frames represented as follows:

$$L=\frac{1}{T}\sum_{t=1}^{T}L_{t}.$$

(1)

$\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ is estimated with the DNNs. After the high-order components of the cepstrum are padded with zeros, $\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ is multiplied by a time-independent lifter $\textrm{\boldmath$u$}_{\rm{min}}$ for a minimum-phase filter. The complex frequency spectrum of differential filter $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$ can be obtained by taking the inverse discrete Fourier transform (IDFT) of the liftered cepstrum. The lifter $\textrm{\boldmath$u$}_{\rm{min}}$ is represented as follows [15]:

$$\textrm{\boldmath$u$}_{\rm{min}}(n)=\begin{cases}1&\left(n=0,n=N/2\right)\\ 2&\left(0<n<N/2\right),\\ 0&\left(n>N/2\right)\end{cases}$$

(2)

where $N$ is the number of frequency bins of the DFT. A differential filter in the time domain $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ is obtained by applying the IDFT to $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$. The tap length of $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ is equal to $N$. To reduce the computational cost of convolution operation, we can truncate $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ with a fixed tap length $l$ ($l<N$). We define the $l$-tap truncated filter as $\textrm{\boldmath$f$}_{t}^{(l)}$. Although filter truncation can efficiently reduce the computational cost, $\textrm{\boldmath$f$}_{t}^{(l)}$ degrades converted-speech quality.

Our lifter-training method trains not only DNNs but also a lifter to avoid converted-speech quality degradation caused by filter truncation. Let $\textrm{\boldmath$u$}=[u_{1},...,u_{c}]^{\top}$ be a time-independent trainable lifter, where $c$ is the dimension of the real cepstrum. The filter-truncation process with $l$ is integrated into training as shown in Fig. 2.

As we described in Section 2.1, the DNNs estimate $\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ from $\textrm{\boldmath$C$}_{t}^{(\mathrm{X})}$. Then $\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ is multiplied by the trainable lifter $u$, and the complex frequency spectrum of the differential filter $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$ is obtained from the IDFT of $\textrm{\boldmath$C$}_{t}^{(\mathrm{D})}$ and exponential calculation. The differential filter in the time domain $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ is obtained by applying the IDFT to $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$. $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ is truncated to $\textrm{\boldmath$f$}_{t}^{(l)}$ by applying a window function $w$ given as Eq. (3):

	$\displaystyle\textrm{\boldmath$f$}_{t}^{(l)}$	$\displaystyle=$	$\displaystyle\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}\cdot\textrm{\boldmath$w$},$		(3)
	$w$	$\displaystyle=$	$\displaystyle\left[\overset{0\mathrm{th}}{1},\cdots,\overset{(l-1)\mathrm{th}}{1},\overset{l\mathrm{th}}{0},\cdots,\overset{(N-1)\mathrm{th}}{0}\right]^{\top}.$		(4)

By using the DFT again, a complex spectrum of the $l$-tap truncated differential filter $\textrm{\boldmath$F$}_{t}^{(l)}$ can be obtained. A complex spectrum of converted speech $\hat{\textrm{\boldmath$F$}}_{t}^{(\mathrm{Y})}$ is obtained by multiplying $\textrm{\boldmath$F$}_{t}^{(\mathrm{X})}$ by $\textrm{\boldmath$F$}_{t}^{(l)}$, and the real cepstrum of converted speech $\hat{\textrm{\boldmath$C$}}_{t}^{(\mathrm{Y})}$ is extracted from $\hat{\textrm{\boldmath$F$}}_{t}^{(\mathrm{Y})}$. The parameters of the DNNs and the lifter are jointly trained to minimize the same loss function as Eq. (1). Since all processes of this method are differentiable, the training can be done by back-propagation [16].

In the conversion process, the trained DNNs and lifter estimate $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$. $\textrm{\boldmath$f$}_{t}^{(\mathrm{D})}$ is obtained by applying the IDFT to $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$, and $\textrm{\boldmath$f$}_{t}^{(l)}$ is obtained by truncating with $l$. We can obtain the converted speech waveform by applying $\textrm{\boldmath$f$}_{t}^{(l)}$ to the source speech waveform.

We now describe our sub-band processing method for full-band extension of the conventional method. When the bandwidth of spectral-differential VC is extended to 48 kHz, conversion cannot be performed well due to the large fluctuations in the wider-band components. To avoid errors in the high-frequency components, our method involves sub-band processing for filtering separately for each frequency band. We apply the differential filter in a frequency region lower than 8 kHz, and do not apply it to a frequency region higher than 8 kHz. In practice, conversion of only the low-frequency band is possible by 1) subtracting 1.0 from the absolute value of $\textrm{\boldmath$F$}_{t}^{(\mathrm{D})}$, 2) applying the sigmoid function that transitions 1 to 0 at around 8 kHz, and 3) adding 1.0 again.

With the conventional method, the cepstrum is multiplied by the lifter coefficient to determine the shape of the filter so that the phase is minimized. Although the shape of the differential filter changes due to truncation, it is transformed to compensate for the effect of the truncation by applying the Hilbert transform using the lifter trained with the proposed lifter-training method. As a result, our lifter-training method can reduce the calculation amount while suppressing converted-speech quality degradation caused by the filter truncation. Figure 3 shows the cumulative power distribution of the differential filter with the conventional method ($l=512$) and proposed lifter-training method ($l=32$). The values on the vertical axis are normalized with the cumulative total. The power is concentrated in the region of tap length 0 to 100. Figure 4 also shows the difference between the lifter trained with the proposed method ($l=64$) and that for minimum phasing.

As explained in Section 1, liftering-based phase estimation requires only small computation. Since our lifter-training method adopts the same estimation as the conventional method, there is no increase in computational cost of phase estimation.

We applied our lifter-training method to VC, i.e., speaker conversion. We expect that this method can be applied to other tasks processed by filtering, e.g., source separation and speech enhancement.

We also discuss our sub-band processing method. Since the characteristics of a speech waveform vary significantly from band to band, it is effective to process the waveform separately for each band. In sub-band WaveNet [17], the speech waveform is divided into several bands and down-sampled, and the waveform in each band is processed separately. In our proposed sub-band processing method using full-band speech, the waveform of each frequency band is not down-sampled. In implementing real-time conversion, the computational cost of filtering can be reduced to $1/3$ by dividing the frequency domain into three bands and down-sampling the waveform of each band.

We built two intra-gender VC: for female-to-female (f2f) and male-to-male (m2m) conversion. The source and target speakers in female-to-female conversion were stored in the JSUT corpus [18] and Voice Actress Corpus [19], respectively. Those in male-to-male conversion were stored in the JVS corpus [20]. We used 100 utterances (approx. 12 min.) of each speaker, and the numbers of utterances for training, validation, and test data were 80, 10, 10, respectively.

We used narrow-band speech (16 kHz) and full-band speech (48 kHz) for the evaluation. In the narrow-band case, the window length was 25 ms, frame shift was 5 ms, the fast Fourier transform (FFT) length was 512 samples, and number of dimensions of the cepstrum was 40 (0th-through-39th). In the full-band case, the window length and frame shift were the same as those in the narrow-band case, but the FFT length was 2048 samples, and number of dimensions of the cepstrum was 120 (0th-through-119th). For pre-processing, the silent intervals of training and validation data were removed, and the lengths of the source and target speech were aligned by dynamic time warping.

The DNN architecture of the acoustic model was multi-layer perceptron consisting of two hidden layers. The numbers of each hidden unit were 280 and 100 in the narrow-band case, and 840 and 300 in the full-band case. The DNNs consisted of a gated linear unit [21] including the sigmoid activation layer and tanh activation layer, and batch normalization [22] was carried out before applying each activation function. Adam [23] was used as the optimization method. During training, the cepstrum of the source and target speech was normalized to have zero mean and unit variance. The batch size and number of epochs were set to 1,000 and 100, respectively. The model parameters of the DNNs used with the proposed lifter-training method were initialized with the conventional method. The initial value of the lifter coefficient was set to that of the lifter for minimum phasing. In the narrow-band case, the learning rates for the conventional and proposed lifter-training methods were set to 0.0005 and 0.00001, respectively. In the full-band case, the learning rates for the conventional and proposed lifter-training methods were set to 0.0001 and 0.000005, respectively.

The proposed lifter-training method was evaluated using both narrow-band (16 kHz) and full-band (48 kHz) speech. The truncated tap length $l$ for the narrow-band case was 128, 64, 48, and 32, and that for the full-band case was 224 and 192. On the other hand, the proposed sub-band processing method was evaluated using only the full-band speech.

We compared root mean squared error (RMSE) with the proposed lifter-training and conventional methods when changing $l$. The RMSE is obtained by taking the squared root of Eq. (1). Figure 5 shows a plot of the RMSEs in male-to-male and female-to-female VC using narrow-band speech (16 kHz). The proposed lifter-training method achieved higher-precision conversion than the conventional method for all $l$. The differences in the RMSEs between the proposed and conventional methods also tended to become more significant when $l$ is smaller. This result indicates that the proposed lifter-training method can reduce the effect of filter truncation.