Audio Spotforming Using Nonnegative Tensor Factorization with Attractor-Based Regularization

In this study, we consider the scenario depicted in Fig. 1. The target and interference sources are located in the same direction relative to each microphone array. Although the BF in each microphone array can enhance the target source, the interference sources in the same direction are also enhanced. However, because the residual interference sources are different for each BF output, estimation of the target source becomes feasible by extracting only the common components among the BF output signals.

Let $\mathsf{X}^{(a)}\in\mathbb{C}^{I\times J\times M}$ denote the time-frequency components of the multichannel observed signal captured by $a$-th microphone array, with its elements defined as $x_{i,j,m}^{(a)}$, where $i=0,1,\cdots,I-1$, $j=0,1,\cdots,J-1$, $a=0,1,\cdots,A-1$, and $m=0,1,\cdots,M-1$ are the indices of frequency bins, time frames, microphone arrays, and microphones within each array, respectively. In spotforming using distributed microphone arrays, BF is initially applied within each microphone array as $\bm{Y}^{(a)}=f_{\theta_{a}}(\mathsf{X}^{(a)})$, where $\bm{Y}^{(a)}\in\mathbb{C}^{I\times J}$ represents the output spectrogram of the BF operation and $\theta_{a}$ denotes a specific look direction. We also define the tensor for $(\bm{Y}^{(a)})_{a=0}^{A-1}$ as $\mathsf{Y}\in\mathbb{C}^{I\times J\times A}$, with its elements denoted as $y_{i,j}^{(a)}$. The objective of spotforming is to estimate the components of the target source from $\mathsf{Y}$ that correspond to the common components of the BF outputs $(\bm{Y}^{(a)})_{a=0}^{A-1}$.

In the conventional method [5], NMF extracts common components. Let $\bm{C}^{(\mathrm{conv})}\in\mathbb{R}_{\geq 0}^{I\times N}$ denote an input matrix of NMF, with its elements defined as $c_{i,n}^{(\mathrm{conv})}$, where $n=0,1,\cdots,N-1$ is the index of columns. This matrix is constructed by concatenating the amplitude spectrograms $(|\bm{Y}^{(a)}|)_{a=0}^{A-1}$ in the time-frame dimension, as shown in Fig. 2 (a), namely,

where $|\cdot|$ for matrices denotes an element-wise absolute operation and $N=AJ$. Matrix $\bm{C}^{(\mathrm{conv})}$ is decomposed using NMF as follows:

where $\bm{T}\in\{\bm{T}\in[0,1]^{I\times K}|\sum_{i}t_{i,k}=1\}$ and $\tilde{\bm{V}}\in\mathbb{R}_{\geq 0}^{N\times K}$ are the basis and activation matrices, respectively, and $k=0,1,\cdots,K-1$ is the index of NMF basis vectors. The optimization problem of (2) is defined as follows:

where $\mathcal{D}$ is a divergence function.

After the estimation of $\bm{T}$ and $\tilde{\bm{V}}$, a binary mask matrix $\tilde{\bm{H}}\in\{0,1\}^{J\times K}$ is calculated as

where $\tilde{h}_{j,k}$ is an element of $\tilde{\bm{H}}$ and $\tau\geq 0$ is a thresholding value. This binary mask defines the component with an activation greater than $\tau$ in all the microphone arrays as the common target source component and sets it to unity.

The spectrogram of the target source $\hat{\bm{S}}^{(a)}\in\mathbb{C}^{I\times J}$ can be estimated by a Wiener filter using the binary mask $\tilde{\bm{H}}$ as¹¹1The method proposed in [5] does not apply (5), but directly uses the estimated NMF model for obtaining $\hat{\mathbf{s}}^{(a)}$ with a phase recovery technique. Since (5) slightly improves the performance, we employ (5) in this paper.

where $\hat{s}^{(a)}_{i,j}$ is the $(i,j)$-th element of $\hat{\bm{S}}^{(a)}$. To obtain the time domain signals $\hat{\mathbf{s}}^{(a)}\in\mathbb{R}^{L}$, an inverse short-time Fourier transform (STFT) is applied to $\hat{\bm{S}}^{(a)}$, where $L$ is the signal length of $\hat{\mathbf{s}}^{(a)}$. Finally, a delay-and-sum operation using $(\hat{\mathbf{s}}^{(a)})_{a=0}^{A-1}$ is performed for further enhancement of the target source signal.

The NMF model in Fig. 2 (a) lacks model interpretability because of the absence of explicit modeling of the relationship between each basis vector in $\bm{T}$ and each of the BF outputs $(\bm{Y}^{(a)})_{a=0}^{A-1}$. Consequently, it is difficult to regularize $\bm{T}$ or $\bm{V}$ to enhance discrimination between the target and interference source components. Moreover, the hyperparameters $K$ and $\tau$ must be appropriately tuned depending on the observed signal in advance, which increases the difficulty of putting the conventional method into practical use.

To address these shortcomings, we propose a novel spotforming method that utilizes an NTF to model the spectrograms of the BF outputs (Fig. 2 (b)). The proposed method allocates $K$ basis vectors in $\bm{T}$ to each microphone array using an allocation matrix $\bm{Z}$. Furthermore, we introduce an attractor-based regularization into $\bm{Z}$ to allocate each basis vector automatically assign/allocate to the corresponding BF outputs. This significantly increases the interpretability of the model and makes basis vectors more discriminative than the conventional method. This regularization automatically optimizes the number of basis vectors for the target source, resulting in robust spotforming against hyperparameter settings.

In the proposed method, the input tensor $\mathsf{C}^{(\mathrm{prop})}\in\mathbb{R}_{\geq 0}^{A\times I\times J}$ of the NTF is defined as follows (Fig. 2 (b)):

where $c^{\mathrm{(prop)}}_{a,i,j}$ is an element of $\mathsf{C}^{(\mathrm{prop})}$. In contrast to (1), the three-dimensional tensor $\mathsf{C}^{(\mathrm{prop})}$ maintains the physical dimensions of the microphone array, frequency bins, and time frames. Then, $\mathsf{C}^{(\mathrm{prop})}$ is decomposed into three matrices, i.e., the allocation matrix $\bm{Z}=[\bm{z}_{0},\bm{z}_{1},\cdots,\bm{z}_{K-1}]\in\{\bm{Z}\in[0,1]^{A\times K}|\sum_{a}z_{a,k}=1\}$, the basis matrix $\bm{T}=[\bm{t}_{0},\bm{t}_{1},\cdots,\bm{t}_{K-1}]$, and the activation matrix $\bm{V}\in\mathbb{R}_{\geq 0}^{J\times K}$, as

where $z_{a,k}$ and $v_{j,k}$ are the elements of $\bm{Z}$ and $\bm{V}$, respectively, and $\bm{z}_{k}$ and $\bm{t}_{k}$ are the column vectors of $\bm{Z}$ and $\bm{T}$, respectively.

Matrix $\bm{Z}$ allocates $K$ basis vectors to $A$ microphone arrays to approximate input tensor $\mathsf{C}^{(\mathrm{prop})}$. Because the target source components are commonly included across all BF outputs $(|\bm{Y}^{(a)}|)_{a=0}^{A-1}$, such basis vectors should be allocated to all microphone arrays. Therefore, if $\bm{t}_{k}$ represents the target source, $\bm{z}_{k}$ should be $\bm{z}_{k}\approx[1/A,1/A,\cdots,1/A]^{\mathrm{T}}$. In contrast, if $\bm{t}_{k}$ corresponds to the other (interference) sources, $\bm{z}_{k}$ should be a one-hot vector. This allocation can be interpreted as a partitional clustering of the basis vectors. Although such optimization can be facilitated by (7) and the low-rank approximation property in NTF, we introduce a new attractor-based regularization to enhance the aforementioned partitional clustering further.

The optimization problem of the proposed method is formulated as follows:

where $\mu\geq 0$ is the weight coefficient, and the regularization term is defined as

Also, $p_{a,b}$ is an element of $\bm{p}_{b}$, $b=0,1,\cdots,B-1$ is the index of attractor vectors $(\bm{p}_{0},\bm{p}_{1},\cdots,\bm{p}_{B-1})$, and $B=A+1$. The set $\mathbb{P}$ encompasses $B$ attractor vectors for each class; $\bm{p}_{0}$ corresponds to the target-source class, while $\bm{p}_{1},\bm{p}_{2},\cdots,\bm{p}_{B-1}$ correspond to the other interference-source classes related to each microphone array. $b_{k}$ calculated by (11) corresponds to the index of the nearest attractor vector (class) from the current allocation vector $\bm{z}_{k}$. Thus, the regularization term in (9) forces $\bm{z}_{k}$ to be closer to the nearest attractor vector, $\bm{p}_{b_{k}}$, emphasizing source clustering of the basis vectors $(\bm{t}_{k})_{k=0}^{K-1}$. This can further affect the basis matrix, resulting in more discriminative basis vectors. In addition, this regularization automatically classifies $K$ basis vectors into target and interference sources. Consequently, the optimal number of basis vectors for the target source is estimated jointly during the optimization.

After the optimization of $\bm{Z}$, $\bm{T}$, and $\bm{V}$, a binary vector $\bm{h}=[h_{0},h_{1},\cdots,h_{K-1}]^{\mathrm{T}}\in\{0,1\}^{K}$ is calculated as follows:

where $h_{k}=1$ indicates that $\bm{t}_{k}$ corresponds to the target source. In contrast to (4), (12) is independent of the time frame $j$.

Similar to (5), the spectrogram of the target source is obtained using the following Wiener filter:

The other post-processing steps are the same as those used in the conventional method.

The cost function in (8) can be minimized using a majorization-minimization (MM) algorithm [7]. In this study, we use a generalized Kullback–Leibler divergence

in (3), (8), and (11), which provides better performance in many audio source separation tasks, e.g., [8]. We define the cost function in (8) as $\mathcal{J}$. By applying Jensen’s inequality, we obtain the majorization function $\mathcal{J}^{+}\geq\mathcal{J}$ as follows:

where $\overset{c}{=}$ denotes equality up to a constant and $\alpha_{a,i,j,k}>0$ is an auxiliary variable that satisfies $\sum_{k}\alpha_{a,i,j,k}=1$. The equality $\mathcal{J}^{+}=\mathcal{J}$ holds if and only if

From $\partial\mathcal{J}^{+}/\partial z_{a,k}=0$, we obtain

Thus,

By substituting (16) into (17), we obtain the update rule for $\bm{Z}$ as

Note that index $b_{k}$ must always be updated by (11) before updating $\bm{Z}$. Similar to (18), the update rules for $\bm{T}$ and $\bm{V}$ can be derived as follows:

To ensure $\textstyle\sum_{a}z_{a,k}=1$ and $\textstyle\sum_{i}t_{i,k}=1$, we apply the normalization of $\bm{z}_{k}$ and $\bm{t}_{k}$ after (18) and (19), respectively, such that the cost function does not change by scaling each column of $\bm{V}$.

For the convergence of the proposed optimization algorithm, we can state the following theorem. This ensures a theoretical nonincrease in the cost function in the proposed method.

A spotforming experiment was conducted to validate the proposed method. To simulate the recording environment illustrated in Fig. 3, we used a two-dimensional image method implemented in Pyroomacoustics [9]. We simulated two reverberation times for each environment, $T_{60}=0$ and $T_{60}=256$ ms, resulting in four recording conditions. The speech signals listed in Table I, randomly selected from the LibriTTS [10] dataset, were used as the dry sources. These dry sources were normalized to have uniform signal energies before the room impulse responses were convoluted. The effect of background noise was not considered in this experiment.

We applied a minimum variance distortionless response (MVDR) BF to each observed signal $\mathsf{X}^{(a)}$ of the microphone array and obtained enhanced signals $(\bm{Y}^{(a)})_{a=0}^{A-1}$. The target steering vectors and noise covariance matrices for each MVDR BF were set to their oracle values calculated from impulse responses. This condition simulated that BF preprocessing provides ideal performance, and the net performances of the conventional and proposed methods were compared. All microphones were synchronized in this experiment.

As an evaluation criterion, we used the source-to-distortion ratio (SDR) [11] of the target source, a common score reflecting the total source separation quality. Because the NMF and NTF results depend on the initial random values of the parameters, we used 10 random seeds. We calculated the average SDR scores and their standard deviations. The other conditions are listed in Table II.

Figs. 4 and 5 show the results of two- and three-microphone-array cases, respectively. The hyperparameter $\tau$ for the conventional method was set to 12 patterns, and the three better conditions were shown in Figs. 4 and 5. These results confirm that the proposed method outperforms the conventional method in all cases. Moreover, the proposed method maintained better performance when $K$ increased, while the performance of the conventional method degraded for a large value of $K$ in some conditions of $\tau$.

The hyperparameter $\mu$ in the proposed method also affected the performance. Fig. 6 shows the performance behavior of the proposed method with various settings of $\mu$. The proposed method achieved optimal performance when we set $\mu$ to a certain large value, e.g., $\mu\geq 100$. Strong regularization with (9) results in a hard classification of the basis vectors $(\bm{t}_{k})_{k=0}^{K-1}$, namely, each allocation vector $\bm{z}_{k}$ coincides with one of the attractor vectors $(\bm{p}_{b})_{b=0}^{B-1}$. This phenomenon was consistently confirmed under other values of $K$ and $T_{60}$. Thus, such optimization tends to provide a better spotforming performance for the proposed method. Thanks to this property, we can robustly obtain better results by using a certain large value of $\mu$ and $K$.