A deep neural network framework for dynamic multi-valued mapping estimation and its applications ^††thanks: This work is supported by HKRGC GRF (Project ID: 14305919).

Our objective is to develop a deep neural network framework to solve the optimization problem 5. To achieve this, we parameterize the mappings $f$, $p$, and $z$ using deep neural networks $G_{\theta},P_{\varphi}$ and $E_{\phi}$ respectively. The network structures are illustrated in the Fig. 2. The framework for formulating the DMM is shown in Fig. 3.

The deep neural network $G_{\theta}$ represents the bi-variate mapping $f$. In other words, we have $f(x,c_{j})=G_{\theta}(x,c_{j})$. It is depicted within the region bounded by the red dotted boundary in Fig. 2. The input $x\in X$ is passed through an embedding network, which generates a latent representation $l$. This latent representation captures the meaningful features of $x$. Subsequently, the latent representation, along with the codebook $C=\{c_{1},c_{2},...,c_{N}\}$, is fed into another deep generative network that produces plausible outputs ${f(x,c_{1}),f(x,c_{2}),...,f(x,c_{N})}$. Here, we assume that the discrete codebook $C$ captures shared label information across different $x\in X$. It is worth noting that different parameters $\theta$ of the deep neural network result in different one-to-$N$ mappings. Within this framework, we can effectively search for the optimal $f$ by optimizing $\theta$.

Similarly, the deep neural network $P_{\varphi}$ represents the bi-variate mapping $p$. In other words, $p(c_{j}|x)=P_{\varphi}(c_{j},x)$. It is shown within the region bounded by the green dotted boundary in Fig. 2. The input $x\in X$ is first passed through an embedding network, which produces a latent vector $h$. By performing a matrix multiplication $Ah$ and applying a softmax operation, we obtain a probability vector $p=(p(c_{1}|x),p(c_{2}|x),...,p(c_{N}|x))$. The output of $P_{\theta}$ estimates the probability $p(c_{j}|x)$ for each plausible output $f(x,c_{j})$. The choice of the fixed matrix $A$ is crucial for accurately predicting uncertainty estimation, and we will discuss this in detail later. In addition, we fuse the embedding networks for $G_{\theta}$ and $P_{\theta}$ as one named $E_{\theta}$. This reduces the parameters of the model and improves the efficiency of training. For simplification, we abbreviate $G_{\theta}(E_{\theta}(x),c_{j})$ and $P_{\varphi}(c_{j},E_{\theta}(x))$ to $G_{\theta}(x,c_{j})$ and $P_{\varphi}(c_{j},x)$ respectively.

To solve the optimization problem 5, we introduce the cluster mapping $z$. The cluster mapping takes $x\in X$ and an associated plausible output $y\in Y$ from the dataset as the input and output a vector, which is of the same dimension of the code in the codebook. To parameterize $z$, we utilize another deep neural network $E_{\phi}$. In other words, $z(x,y)=E_{\phi}(x,y)$. As shown within the region bounded by the blue dotted boundary in Fig. 2, $x\in X$ and $y\in Y$ are fed into an embedding network to output a vector $E_{\phi}(x,y)$. With $E_{\phi}(x,y)$, we can find the code in $C$ closest to $E_{\phi}(x,y)$ that solves 4.

Under this setting, all mappings $f$, $p$, and $z$ to be optimized are parameterized using deep neural networks. Therefore, they can be optimized by training the deep neural networks to obtain the optimal parameters that minimize a loss function defined by the energy functional in our optimization problem.

A crucial component of our proposed framework is the use of a discrete codebook $C=\{c_{1},c_{2},...,c_{N}\}\subset\mathbb{R}^{m}$ to represent a one-to-$N$ mapping. For each $x\in X$, we can obtain $N$ plausible outputs: $f(x,c_{1})$, $f(x,c_{2})$, …, $f(x,c_{N})$. Additionally, we can estimate the corresponding probabilities associated with each output: $p(c_{1},x)$, $p(c_{2},x)$, …, $p(c_{N},x)$.

It is important to highlight that for our framework to effectively represent a DMM, the following condition must hold for any $j\neq k$: if both $p(c_{j},x)$ and $p(c_{k},x)$ are non-zero, then $f(x,c_{j})\neq f(x,c_{k})$. In other words, each code in the codebook should be associated with a distinct output for every $x\in X$. The choice of the codebook is therefore crucial in enforcing this requirement.

In our framework, for every $x\in X$, a code $c_{j}\in C$ corresponds to a plausible output $G_{\theta}(x,c_{j})$. It is important to note that $c_{j}\in\mathbb{R}^{m}$, where usually the dimension $m$ of $c_{j}$ is significantly smaller than the dimension of the output space $Y$. Consequently, the generator produces plausible outputs $G_{\theta}(x,c_{j})\in Y$ that have a much higher dimension. The separability of the codes $c_{j}$ encourages the separability of the corresponding outputs $f(x,c_{j})$. Conversely, if two codes $c_{j}$ and $c_{k}$ are close to each other in the codebook $C$, the plausible outputs $f(x,c_{j})$ and $f(x,c_{k})$ will also be close to each other. This can hinder the capability of our framework to effectively represent a DMM.

In practice, although the discrete codebook inherently lends itself to modeling multi-modal label data $\{(x_{i},y_{i}^{k})_{k=1}^{\mathcal{N}_{i}}\}_{i=1}^{T}$, the occurrence of similar codes within the codebook during continuous updates can lead to highly similar results. To prevent this repetition and ensure diversity in the generated outcomes, it becomes crucial to maximize the separation between each code in the codebook $C$.

To maximize the separation between each code in the codebook, our strategy is to reduce the mutual correlation among vectors in the codebook. For this purpose, we introduce the following covariance loss.

To understand the meaning of the covariance loss, observe that all diagonal entries of $\mathcal{X}^{T}\mathcal{X}\in\mathbb{R}^{N\times N}$ are equal to 1. Thus, $\mathcal{X}^{T}\mathcal{X}-I_{N}$ has 0 on its diagonal. Each non-diagonal entry of $\mathcal{X}^{T}\mathcal{X}-I_{N}$ represents the inner product between two distinct data points in $\mathcal{X}$. By applying a threshold $\tau$ to $T^{\tau}(\mathcal{X}^{T}\mathcal{X}-I_{N})$, we retain only the entries that exceed $\tau$. These entries signify pairs of data points that are close to each other, as their inner product surpasses the threshold. The covariance loss measures the mean squared sum of all non-zero entries in $T^{\tau}(\mathcal{X}^{T}\mathcal{X}-I_{N})$. Minimizing the covariance loss $L_{cov}^{\tau}$ aims to encourage a greater separation between the data points in $\mathcal{X}$. By this loss function, we are only concerned with those vectors that are not almost orthogonal, i.e., inner products greater than the threshold. It can help us focus on preventing the occurrence of similar codes within the selected codes. In particular, when $L_{cov}^{\tau}=0$, it implies that the inner product between any pair of data points in $\mathcal{X}$ is below the threshold $\tau$. This indicates that the separability of each pair of data points in $\mathcal{X}$ satisfies a set tolerance, promoting greater distinctiveness among the data points.

Using $L_{cov}^{\tau}$, we can encourage the separation of the codes in the codebook $C=\{c_{1},c_{2},...,c_{N}\}$. To achieve this, we first normalize the codes in the codebook to $\widetilde{C}=\{\frac{c_{1}}{|c_{1}|},\frac{c_{2}}{|c_{2}|},...,\frac{c_{N}}{|c_{N}|}\}$. By minimizing the covariance loss $L_{cov}^{\tau}(\widetilde{C})$, we can effectively separate the normalized codes in the codebook $C$. This, in turn, promotes the separability of the corresponding plausible outputs $f(x,c_{j})$ in the output space $Y$, ensuring a diverse set of generated results for each input $x$.

Furthermore, choosing suitable parameters for the size of the codebook $N$ and the dimensionality of the codes $m$ is also important. For a set of unit vectors $\{v_{1},...,v_{N}\}$ in $\mathbb{R}^{m}$, if $N\leq m$, it is possible for their inner products to be all equal to zero, ensuring perfect orthogonality. However, in practical applications, the hyperparameters $N$ and $m$ of the codebook may require a broader range of choices to adapt to different datasets. It is not always the case that $N\leq m$. In situations where $N>m$, we require the variables in the codebook to be ”almost orthogonal.” The Kabatjanskii-Levenstein bound for almost orthogonal vectors tao2013almostorthogonal helps us to decide on appropriate choices for the values of $N$, $m$, and the threshold $t$ in such cases. This theorem provides theoretical guidance on the trade-offs between the codebook size $N$, the code dimensionality $m$, and the degree of orthogonality required, allowing us to configure these parameters effectively for different datasets.

For the special case when the hyperparameter $A=1/2$, we have the following theorem tao2013almostorthogonal :

Therefore, in our case, we set the hyperparameter $A=1/2$. Then, the threshold $\tau$ is chosen as $1/2m^{-1/2}$. As a condition on the hyperparameter choices, we must have $N<2m$, where $N$ is the size of the codebook and $m$ is the dimensionality of the codes. In our work, we carefully select the values of $N$ and $m$ such that the separability tolerance can be achieved using this threshold. With these parameters, we will incorporate the thresholded covariance loss $L_{cov}^{\tau}$ in the overall loss function for training the deep neural network to solve the optimization problem in 5. Our experimental results demonstrate the powerful impact of the covariance loss in improving the separability and diversity of the generated outputs.

In practice, the number of codes $N$ is often larger than the actual number of plausible outputs. This means that $\mathcal{N}_{x}\ll N$ for all $x\in X$, resulting in some codes being unused and left idle. Initially, the codes are configured such that the inner products between distinct pairs are less than the threshold. However, when updating the codes that have been used, they tend to accumulate in similar positions, reducing their separability. Minimizing the covariance loss plays a crucial role in addressing this issue. When minimizing the covariance loss, only the active codes are modified, while the inner products involving non-active codes remain at 0 even after applying the threshold function $T^{\tau}$. This ensures that the focus is on adjusting the positions of the codes that are actually contributing to the plausible outputs, enhancing their separability and promoting diversity in the output space.

Another important challenge to address is that when solving the optimization problem 5, the following condition 4 has to be considered:

During the training process, a data-label pair $(x_{i},y_{i}^{k})$ is randomly sampled from the unorganized dataset ${(x_{i},y_{i}^{k})}_{k=1}^{\mathcal{N}_{i}}$, and fed into the embedding network $E_{\phi}$. Subsequently, the nearest code $c$ is selected from the codebook for the corresponding feature $E_{\phi}(x_{i},y_{i}^{k})$, as follows:

The rationale behind this approach is that a learnable codebook, while exhibiting a slower learning capability, is required to capture more information from the data. Therefore, we propose to update the codebook $C$ using a dictionary learning algorithm, Vector Quantization (VQ) van2017neural , which computes the exponential moving average of the corresponding embedding network’s outputs. Our ablation analysis 10 has shown that a learnable codebook outperforms a fixed codebook.

Another crucial component in our framework is the estimation of the probability associated with each plausible output. In practice, the ground truth probability of each plausible output associated with an input is not known. Our goal is to estimate these probabilities from the sample training dataset $\{\{(x_{i},y_{i}^{k})\}_{k=1}^{\mathcal{N}_{i}}\}_{i=1}^{T}$. For each input $x_{i}$, recall that the collection of sampled plausible outputs $\{y_{i}^{k}\}_{k=1}^{\mathcal{N}_{i}}$ can contain repeated values. For example, when labeling lesions of a medical image, different medical experts might provide the same label, resulting in repeated labels in the dataset. The more repetitions of a particular label, the higher the estimated probability associated with that plausible output. The intuition behind this approach is that the frequency of a plausible output in the sample dataset can serve as a proxy for its true probability. The more often a plausible output appears, the more likely it is to be the correct label for the given input. By leveraging this idea, we can estimate the probability distribution over the plausible outputs for each input, even though the ground truth probabilities are not known.

In our framework, the probability $p:C\times X\to[0,1]$ is parameterized by a deep neural network $P_{\varphi}$. Given an input $x_{i}$ from the training dataset, it is fed into the network to obtain a feature vector $h\in\mathbb{R}^{m}$. Next, we compute a vector $\tilde{p}_{i}\in\mathbb{R}^{N}$ as $\tilde{p}_{i}=Ah$, where $A\in\mathbb{R}^{N\times m}$ is a suitable matrix. We then pass $\tilde{p}_{i}$ through a softmax operation to obtain the probability vector $p_{i}$, where:

The choice of the matrix $A$ is crucial in this setup. The value of $p_{ij}$ is related to the inner product between the $j$-th row of $A$ and the vector $\tilde{p}_{i}$. Specifically, $p_{ij}$ will be larger if $\tilde{p}_{i}$ is closer to the $j$-th row of $A$. One possible approach is to learn the matrix $A$ during the training process, allowing the network to discover the optimal projection that captures the relationship between the input features and the plausible output probabilities.

However, when training on an imbalanced dataset, the learnable vectors of the minority classes may collapse, a phenomenon known as “minority collapse”. To alleviate this issue, we define $A$ using the simplex equiangular tight frame (ETF) papyan2020prevalence . This results in a fixed ETF classifier, as proposed in yang2022we .

The simplex equiangular tight frame (ETF) is formally defined as follows:

All vectors in a simplex ETF have an equal $l_{2}$ norm and the same pair-wise angle, i.e.: $m_{i}^{T}m_{j}=\frac{N}{N-1}\delta_{i,j}-\frac{1}{N-1},\quad\forall i,j\in{1,2,...,N}$ where $\delta_{i,j}$ equals 1 when $i=j$ and 0 otherwise. The pair-wise angle $-\frac{1}{N-1}$ is the maximal equiangular separation of $N$ vectors in $\mathbb{R}^{m}$ papyan2020prevalence .

We can then define $A$ whose $j$-th row is given by $m_{j}$. To find the optimal $p$, we optimize the parameters of $P_{\varphi}$ given a training dataset $\{\{(x_{i},y_{i}^{k})\}_{k=1}^{\mathcal{N}_{i}}\}_{i=1}^{T}$ to minimize $E_{3}$ in our optimization problem 5. In particular, the first term in $E_{3}$ involves $P_{\varphi}$:

Therefore, $P_{\varphi}$ can be optimized by minimizing the following cross-entropy loss:

where $M=[m_{1},...,m_{N}]\in\mathbb{R}^{m\times N}$ is the fixed ETF classifier generated by Definition 3.

To solve the optimization problem 5, our framework is reduced to finding optimal parameters $\theta,\varphi$, and $\phi$, which minimizes $E_{3}$. More specifically, $E_{3}$ can now be written as follows:

The parameters of the deep neural network can then be optimized by stochastic gradient descent through backward propagation.

We describe the numerical algorithms in detail. Several techniques, such as simple gradient approximation, stop-gradient operation, and exponential moving average, are used in our numerical algorithms.

Firstly, the simple gradient approximation is used during the update of $\phi$. Since the encoder $E_{\phi}$ receives no gradient from the reconstruction loss, we assign it the gradient of the code $c(x_{i},y_{i}^{k})$, abbreviated as $c$ here. Thus, with the input data pair $(x_{i},y_{i}^{k})$, we have:

Here, the first equation approximates the gradient of the loss function with respect to the encoder parameters $\phi$. The second equation computes the gradient with respect to the generator parameters $\theta$, which includes both the reconstruction loss and the cross-entropy loss. The third equation computes the gradient with respect to the code matrix $C$.

Note that the selected codes receive no gradient from the reconstruction loss and regularization loss, but instead from the covariance loss. In addition, the codes are also updated by the exponential moving average van2017neural as follows:

where $\{e_{1},e_{2},...,e_{count}\}$ is a set of the embedding features that are closest to code $c$. $0<\kappa<1$ is the decay coefficient.

During the testing process, the probabilities predicted by $P_{\theta}$ are not exactly equal but very close to 0. A small threshold $\epsilon=1e-5$ is set to eliminate results with extremely low probability.

The details of the numerical algorithms for the training and testing processes are described in Algorithms 1 and 2, respectively.