Learning with Noisy Ground Truth: From 2D Classification to 3D Reconstruction

Supervised Classification. Considering a supervised classification problem with $K$ classes, suppose $\mathcal{X}\in\mathbb{R}^{d}$ be the input space, $\mathcal{Y}\in\{0,1\}^{K}$ is the ground-truth label space in an one-hot manner. In practice, the joint distribution $\mathcal{P}$ over $\mathcal{X}\times\mathcal{Y}$ is unknown. We have a training set $D=\{(\bm{x}_{i},\bm{y}_{i})\}^{N}_{i=1}$ which are independently sampled from joint distribution $\mathcal{P}$. Assume a mapping function class $\mathcal{F}$ wherein each $f:\mathcal{X}\rightarrow\mathbb{R}^{K}$ maps the input space to $K$-dimensional score space, we seek $f^{*}\in\mathcal{F}$ that minimizes an empirical risk $\frac{1}{N}\sum_{i=1}^{N}\ell(\bm{y}_{i},f(\bm{x}_{i}))$ for a certain loss function $\ell$.

To alleviate the impact of noisy labels in training data, existing work Bootstrap (Reed et al., 2015a) proposes to generate soft target by interpolating between the original noisy distributions and model predictions by $\beta\bm{\bar{y}}+(1-\beta)\bm{p}$, where $\beta$ weights the degree of interpolation. Thus the cross-entropy loss using Bootstrap becomes

A static weight (e.g. $\beta=0.8$) is applied as an approximate measure for the correction of a hypothetical noisy label. Another work M-correction (Arazo et al., 2019b) makes $\beta$ dynamic for different samples, i.e., using a noise model to individually weight each sample.

$w_{i}$ is dynamically set to posterior probability conditioned on loss value and the Gaussian Mixture Model (GMM) is estimated after each training epoch using the normalized cross entropy loss for each sample $i$. Thus, correct samples rely on their given label $\bar{y}_{i}$ ($w_{i}$ is large), while incorrect ones let their loss being dominated by their class prediction $z_{i}$ or their predicted probabilities $p_{i}$ (1 - $w_{i}$ is large). In early mature stages of training the CNN model should provide a good estimation of the true class for noisy sample, shown in Fig 2 when epoch = 45.

Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) has made a breakthrough in novel view synthesis, which is capable of synthesizing photo-realistic images at arbitrary views. NeRF represents a 3D scene as a continuous radiance field, and synthesizes images with differentiable volumetric rendering. Recently, 3D Gaussian Splatting (Kerbl et al., 2023) has emerged as another promising paradigm for novel view synthesis, which synthesizes the image by rendering a set of learnable Gaussian points. Despite great development in novel view synthesis, the aforementioned methods are limited to static scenes with constant light conditions. They are struggling with multi-view images that contain distractors.

Existing novel view synthesis methods represent a 3D scene as a parameterized model $\mathbf{S}_{\theta}$, while facilitating a differentiable rendering technique $\mathcal{R}(\cdot,\cdot)$ to render an image $\hat{\mathbf{I}}\in\mathbb{R}^{{H}\times{W}}$ as following:

where $\pi$ is the image pose. Concretely, Neural Radiance Fields (Mildenhall et al., 2020) leverages the Multilayer Perceptron (MLP) and volumetric rendering, whereas 3D Gaussian Splatting (Kerbl et al., 2023) is built upon a collection of learnable Gaussian points and splatting rasterization. However, both of them optimize a $\mathcal{L}_{2}$ loss between the ground truth image and rendered image:

where $\hat{\mathbf{I}}_{r}$ and $\mathbf{I}_{r}$ are the rendered color and ground truth color at pixel $r$, respectively. In the training of NeRF and 3DGS with multi-view images containing distractors (noise), we observed the similar phenomenon of memorization effect as in classification task. This phenomenon refers to the parameters initially fitting purity and excluding distractor pixels in the early stages of training, gradually overfitting to distractor pixels in the later stages, as evidenced by the rendered image and histogram in Fig. 3 (a). It degrades the quality of rendered image in novel views. Most of the 3D reconstruction methods do not consider the case of distractor noise, while the most related work is RoubstNeRF (Sabour et al., 2023).

To improve it, we can simply generate a dynamic weight $\mathbf{M}_{r}$ to indicate the distractor pixels similar as Equation (3). Then we analogy noisy ground truth color $\mathbf{I}_{r}$ to noisy ground truth label $\bar{y}_{i}$ and rendered color $\hat{\mathbf{I}}_{r}$ to predicted probabilities $p_{i}$. Then the loss of new mask-NeRF is

Similar to the M-correction in classification task, leverages the memorization effect to effectively prevent distractors from overfitted, maintaining the position of the red bars unchanged as much as possible. Similar to classification, early mature stage (5k steps) enables a clear distinction between purity and distractor pixels. Subsequently, we can promote the fitting of purity pixels and suppress the learning of distractor pixels throughout the Mip-NeRF 360 training process to easily achieve sharp and clean view rendering in a noisy input scenario. For estimation of $\mathbf{M}_{r}$, we can use the GMM or other unsupervised model to fit the loss distribution to differentiate the clean pixels from distractor (noisy) pixels. One case results (BabyYoda) with GMM on RoubustNeRF dataset (Sabour et al., 2023) are shown in Fig. 3 (b). More qualitative comparison results with existing methods are shown in Fig. 4. This research direction is barely explored in 3D reconstruction area, more work need to be done to make the algorithm more robust in the near future.

Consider a learning task $T$, LNGT deals with a dataset $\bar{D}=\{\bar{D}_{\text{train}},D_{\text{test}}\}$ consisting of a noisy training set $\bar{D}_{\text{train}}=\{(\bm{x}_{i},\bar{y}_{i})\}^{N}_{i}$, and a clean testing set $D_{\text{test}}$. Let $p(\bm{x},y)$ be the ground truth joint probability distribution of input $\bm{x}$ and output $y$, $\bar{p}(\bm{x},\bar{y})$ be the corrupted joint probability distribution of input $\bm{x}$ and output $\bar{y}$. Let $\hat{h}$ be the optimal hypothesis from $\bm{x}$ to $y$. LNGT learns to discover $\hat{h}$ by fitting $\bar{D}_{\text{train}}$ and testing on $D_{\text{test}}$. For clarity, we assume $\bar{D}_{\text{c\_train}}$ be a set of clean training samples (i.e. inputs with correct labels) and $\bar{D}_{\text{m\_train}}$ be the mislabeled training samples (i.e. inputs with wrong labels). We have $\bar{D}_{\text{train}}=\bar{D}_{\text{c\_train}}\cup\bar{D}_{\text{m\_train}}$. Note that $\bar{D}_{\text{c\_train}}$ and $\bar{D}_{\text{m\_train}}$ are imagination sets which are unobservable. We define them only for clear explanations.

To approximate $\hat{h}$, the LNGT model determines a hypothesis space $\mathcal{H}$ of hypotheses $h(\cdot;\theta)$’s, where $\theta$ denotes all the parameters used by $h$. A LNGT algorithm is an optimization strategy that searches $\mathcal{H}$ to find the $\theta$ that parameterizes the best $h^{*}\in\mathcal{H}$. The LNGT performance is measured by a loss function $\ell(\cdot,\cdot)$ defined over the prediction $h(\bm{x};\theta)$ and the observed output $y$ over the test set.

Given a hypothesis $h$, we want to minimize its expected risk $R$, which is the loss measured with respect to $p(\bm{x},y)$. Specifically,

As $p(\bm{x},y)$ is unknown, similar to regular machine learning tasks, the empirical risk, i.e., the average of sample losses over the noisy training set $\bar{D}_{\text{train}}$ of $N$ samples,

is usually used as a proxy for $R(h)$, leading to empirical risk minimization (Mohri et al., 2018). However, in this case, minimizing $R_{N}(h)$ usually leads to an estimation of $\bar{p}(\bm{x},\bar{y})$, which is completely different from $p(\bm{x},y)$. Therefore, directly training models without any adjustment has been observed to lead to poor generalization performance (Zhang et al., 2018a). Here we can decouple the $R_{N}(h)$ into $R_{N_{\text{c}}}(h)$ and $R_{N_{\text{m}}}(h)$. Since $\bar{D}_{\text{c\_train}}$ is a set of clean samples, finding a hypothesis that only minimizes $R_{N_{\text{c}}}(h)$ rather than $R_{N_{\text{m}}}(h)$ leads to a better estimation of $p(\bm{x},y)$. For better illustration, let

• •

  $\hat{h}=\arg\min_{h}R(h)$ be the function that minimizes the expected risk;

• •

  $h^{*}=\arg\min_{h\in\mathcal{H}}R(h)$ be the function in $\mathcal{H}$ that minimizes the expected risk;

• •

  $h_{N}=\arg\min_{h\in\mathcal{H}}R_{N}(h)$ be the function in $\mathcal{H}$ that minimizes the empirical risk;

• •

  $h_{N_{\text{c}}}=\arg\min_{h\in\mathcal{H}}R_{N_{\text{c}}}(h)$ be the function in $\mathcal{H}$ that only minimizes the empirical risk of $\bar{D}_{\text{c\_train}}$ rather than $\bar{D}_{\text{m\_train}}$ (Assume it is achievable state during learning).

As $\hat{h}$ is unknown, one has to approximate it by searching some $h\in\mathcal{H}$. $h^{*}$ is the best approximation for $\hat{h}$ in $\mathcal{H}$. $h_{N}$ is the best hypothesis in $\mathcal{H}$ obtained by minimizing the whole empirical risk $R_{N}(h)$, while $h_{N_{\text{c}}}$ is the optimal hypothesis in $\mathcal{H}$ that only minimizes $R_{N_{\text{c}}}(h)$. For simplicity, we assume that $\hat{h}$, $h^{*}$, $h_{N}$, and $h_{N_{\text{c}}}$ are unique. The total error can be decomposed as

where the expectation is with respect to the random choice of $\bar{D}_{\text{train}}$. The approximation error $\mathcal{E}_{\text{app}}(\mathcal{H})$ measures how close the functions in $\mathcal{H}$ can approximate the optimal hypothesis $\hat{h}$. The estimation error $\mathcal{E}_{\text{est}}(\mathcal{H},N_{\text{c}})$ measures the effect of minimizing the clean empirical risk $R_{N_{\text{c}}}(h)$ instead of the expected risk $R(h)$ within $\mathcal{H}$. The fitting error $\mathcal{E}_{\text{fit}}(\mathcal{H},N,N_{\text{c}})$ measures the effect of minimizing the full empirical risk $R_{N}(h)$ instead of only the clean empirical risk $R_{N_{\text{c}}}(h)$.

As can be observed, the total error is influenced by $\mathcal{H}$ (hypothesis space), $N$ (number of samples in $\bar{D}_{\text{train}}$) and $N_{\text{c}}$ (number of samples in $\bar{D}_{\text{c\_train}}$). A special case is when $N=N_{\text{c}}$, the LNGT reduces to regular learning problem.

Therefore, reducing the total error can be attempted from the perspectives of (1) data, which provides $\bar{D}_{\text{train}}$ and $\bar{D}_{\text{c\_train}}$; (2) model, which determines $\mathcal{H}$; and (3) algorithms, which searches for the optimal hypothesis $h_{N_{\text{c}}}$ that only fits $\bar{D}_{\text{c\_train}}$.

In LNGT, the model would easily fit all noisy samples. The empirical risk $R_{N}(h)$ may then be far from being a good approximation of the expected risk $R(h)$, and the resultant empirical risk minimizer $h_{\text{N}}$ overfits. Indeed, this is the core issue of LNGT, i.e., the empirical risk minimizer $h_{\text{N}}$ is no longer reliable. Therefore, LNGT is much harder. A comparison of learning with clean data and noisy data is shown in Figure 5. Compared to learning with clean data, both the estimation error and fitting error of LNGT increase.

Similar to learning with clean data, the estimation error can be reduced by increasing the number of samples. Therefore, some methods use prior knowledge to augment $\bar{D}_{\text{train}}$. For example, Mixup (Zhang et al., 2018b) constructs virtual training samples by linearly combining two random samples’ features and labels. (Nishi et al., 2021) evaluated multiple augmentation strategies and found that using one set of augmentations for loss modeling tasks and another set for learning is the most effective in LNGT. MixNN (Lu and He, 2021) dynamically mixes the sample with its nearest neighbors to generate synthetic samples for noise robustness. Other methods leverage the unlabeled data to improve the performance of LNL. For instance, (Garg et al., 2021) augmented the training data with random labeled data and provided a theoretical analysis that ensures the true risk is lower. (Iscen et al., 2022) utilized the unlabeled data to enforce the consistency of model predictions, resulting in improving the performance. Combined with Curriculum Learning, more complex Mixup-based methods (Li et al., 2020a; Cordeiro et al., 2023; Nagarajan et al., 2024) have been proposed recently.

These methods aim to prevent the model from overfitting to mislabeled samples.

Some methods focus on correcting the noisy labels, so that the model is gradually refined. (Reed et al., 2015b) proposed a bootstrapping method which modifies the loss with model predictions. (Ma et al., 2018) improved the bootstrapping method by exploiting the dimensionality of feature subspaces to dynamically reweigh the samples. (Arazo et al., 2019a) improved bootstrapping using a dynamic weighting scheme through unsupervised learning techniques. PLC (Zhang et al., 2020) progressively corrects the labels when the prediction confidence over a dynamic threshold. SELC (Lu and He, 2022) gradually corrects noisy labels by ensemble predictions.