3DEgo: 3D Editing on the Go!

As discussed above, differing from IN2N [11] that incorporates edited images gradually over several training iterations, our approach involves editing the entire dataset at once before the training starts. We desire 1) each edited frame, $E_{i}$ follows the editing prompt, $\mathcal{T}$, 2) retain the original images’ semantic content, and 3) the edited images, $\{E_{1},E_{2},\ldots,E_{N}\}$ are consistent with each other.

As discussed above, we incorporate IP2P [4] as a 2D editing diffusion model. The standard noise prediction from IP2P’s backbone that includes both conditional and unconditional editing is given as,

where $s_{f}$ and $s_{\mathcal{T}}$ are image and textual prompt guidance scale.We suggest enhancing the noise estimation process with our autoregressive training framework. Consider a set of $w$ views, represented as $W=\{E_{n}\}_{n=1}^{w}$. Our goal is to model the distribution of the $i$-th view image by utilizing its $w$ adjacent, already edited views. To achieve this, we calculate image-conditional noise estimation, $\varepsilon_{\theta}(e_{t},E,\varnothing_{\mathcal{T}})$ across all frames in $W$. The equation to compute the weighted average $\bar{\varepsilon}_{\theta}$ of the noise estimates from all edited frames within $W$, employing $\beta$ as the weight for each frame, is delineated as follows:

Here, $E_{n}$ represents the $n$-th edited frame within $W$, and $\beta_{n}$ is the weight assigned to the $n$-th frame’s noise estimate. The condition that the sum of all $\beta$ values over $w$ frames equals 1 is given by as, $\sum_{n=1}^{w}\beta_{n}=1$. This ensures that the weighted average is normalized. As we perform 2D editing without any pose priors, our weight parameter $\beta$ is independent of the angle offset between the frame to be edited, $f_{n}$ and already edited frames in $W$. To assign weight parameters with exponential decay, ensuring the closest frame receives the highest weight, we can use an exponential decay function for the weight $\beta_{n}$ of the $n$-th frame in $W$. By employing a decay factor $\lambda_{d}$ (0 < $\lambda_{d}$ < 1), the weight of each frame decreases exponentially as its distance from the target frame increases. The weight $\beta_{n}$ for the $n$-th frame is defined as, $\beta_{n}=\lambda_{d}^{w-n}$. This ensures the, $E$ closest to the target, $f$ ($n=1$) receives the highest weight. To ensure the sum of the weights to 1, each weight is normalized by dividing by the sum of all weights, $\beta_{n}=\frac{\lambda^{w-n}}{\sum_{j=1}^{w}\lambda^{w-j}}$.This normalization guarantees the sum of $\beta_{n}$ across all $n$ equals 1, adhering to the constraint $\sum_{n=1}^{w}\beta_{n}=1$.

Our editing path is determined by the sequence of frames from the captured video. Therefore, during the editing of frame $f_{n}$, we incorporate the previous $w$ edited frames into the set $W$, assigning the highest weight $\beta$ to $E_{n-1}$. Using Eq. 2 and Eq. 3, we define our score estimation function as following:

where $\gamma_{f}$ is a hyperparameter that determines the influence of the original frame undergoing editing on the noise estimation, and $\gamma_{E}$ represents the significance of the noise estimation from adjacent edited views.

Projecting KEA (see Section 3.2) into 3D Gaussians, $\mathcal{H}$, using $M$ for KEA identity assignment, is essential for accurate editing. Therefore, we introduce a vector, $m$ associated with the Gaussian point, $h=\{\mu,\Sigma,c,\alpha,m\}$ in the 3D Gaussian set, $\mathcal{H}_{i}$ of the $i_{th}$ frame. The parameter $m$ is a learnable vector of length 2 corresponding to the number of labels in the segmentation map, $M$. We optimize the newly introduced parameter $m$ to represent KEA identity during training. However, unlike the view-dependent Gaussian parameters, the KEA Identity remains uniform across different rendering views. Gaussian KEA identity ensures the continuous monitoring of each Gaussian’s categorization as they evolve, thereby enabling the selective application of gradients, and the exclusive rendering of targeted objects, markedly enhancing processing efficiency in intricate scenes.

Next, we delve into the training pipeline inspired by  [8, 3] in detail which consists of two stages: (i) Relative Pose Estimation, and (ii) Global 3D Scene Expansion.

To begin the training process, , we randomly pick a specific frame, denoted as $E_{i}$. We then employ a pre-trained monocular depth estimator, symbolized by $\mathcal{D}$, to derive the depth map $D_{i}$ for $E_{i}$. Utilizing $D_{i}$, which provides strong geometric cues independent of camera parameters, we initialize 3DGS with points extracted from monocular depth through camera intrinsics and orthogonal projection. This initialization step involves learning a set of 3D Gaussians $\mathcal{H}_{i}$ to minimize the photometric discrepancy between the rendered and current frames $E_{i}$. The photometric loss, $\mathcal{L}_{rgb}$, optimize the conventional 3D Gaussian parameters including color $c$, covariance $\Sigma$, mean $\mu$, and opacity $\alpha$. However, to initiate the KEA identity and adjust $m_{g}$ for 3D Gaussians, merely relying on $\mathcal{L}_{rgb}$ is insufficient. Hence, we propose the KEA loss, denoted as $\mathcal{L}_{KEA}$, which encompasses the 2D mask $M_{i}$ corresponding to $E_{i}$. We learn the KEA identity of each Gaussian point during training by applying $\mathcal{L}_{KEA}$ loss ($\mathcal{L}_{KEA}$). Overall, 3D Gaussian optimization is defined as,

where $\mathcal{R}$ signifies the 3DGS rendering function. The photometric loss $\mathcal{L}_{rgb}$ as introduced in [17] is a blend of $\mathcal{L}_{1}$ and D-SSIM losses:

$\mathcal{L}_{KEA}$ has two components to it. (i) 2D Binary Cross-Entropy Loss, and (ii) 3D Jensen-Shannon Divergence (JSD) Loss, and is defined as,

Let $\mathcal{N}$ be the total number of pixels in the $M$, and $\mathcal{X}$ represent the set of all pixels. We calculate binary cross-entropy loss $\mathcal{L}_{BCE}$ as following,

where $M(x)$ is the value of the ground truth mask at pixel $x$, indicating whether the pixel belongs to the foreground (1) or the background (0). The sum computes the total loss over all pixels, and the division by $\mathcal{N}$ normalizes the loss, making it independent of the image size. A rendering operation, denoted as $\mathcal{R}(\mathcal{H}_{i},m)(x)$, produces $m_{\mathcal{R}}$ for a given pixel $x$, which represents the weighted sum of the vector $m$ values for the overlapping Gaussians associated with that pixel. Here, $m$ and $m_{\mathcal{R}}$ both have a dimensionality of 2 which is intentionally kept the same as the number of classes in mask labels. We apply softmax function on $m_{\mathcal{R}}$ to extract KEA identity given as, $\text{KEA Identity}=\text{softmax}(m_{\mathcal{R}})$. The softmax output is interpreted as either 0, indicating a position outside the KEA, or 1, denoting a location within the KEA.

To enhance the accuracy of Gaussian KEA identity assignment, we also introduce an unsupervised 3D Regularization Loss to directly influence the learning of Identity vector $m$. This 3D Regularization Loss utilizes spatial consistency in 3D, ensuring that the Identity vector, $m$ of the top $k$-nearest 3D Gaussians are similar in feature space. Specifically, we employ a symmetrical and bounded loss based on the Jensen-Shannon Divergence,

Here, $S$ indicates the softmax function, and $m^{\prime}_{z}$ represents the $z^{th}$ Identity vector from the $Z$ nearest neighbors in 3D space.

In this optimization step, we keep the attributes of ${\mathcal{H}_{i}}^{*}$ fixed to distinguish camera motion from other Gaussian transformations such as pruning, densification, and self-rotation. Applying the above 3DGS initialization to sequential image pairs enables inferring relative poses across frames. However, accumulated pose errors could adversely affect the optimization of a global scene. To tackle this challenge, we propose the gradual, sequential expansion of the 3DGS.

As illustrated above, beginning with frame $E_{i}$, we initiate with a collection of 3D Gaussian points, setting the camera pose to an orthogonal configuration. Then, we calculate the relative camera pose between frames $E_{i}$ and $E_{i+1}$. After estimating the relative camera poses, we propose to expand the 3DGS scene. This all-inclusive 3DGS optimization refines the collection of 3D Gaussian points, including all attributes, across $I$ iterations, taking the calculated relative pose and the two observed frames as inputs. With the availability of the next frame $E_{i+2}$ after $I$ iterations, we repeat the above procedure: estimating the relative pose between $E_{i+1}$ and $E_{i+2}$, and expanding the scene with all-inclusive 3DGS.

To perform all-inclusive 3DGS optimization, we increase the density of the Gaussians currently under reconstruction as new frames are introduced. Following [17], we identify candidates for densification by evaluating the average magnitude of position gradients in view-space. To focus densification on these yet-to-be-observed areas, we enhance the density of the universal 3DGS every $I$ step, synchronized with the rate of new frame addition. We continue to expand the 3D Gaussian points until the conclusion of the input sequence. Through the repetitive application of both frame-relative pose estimation and all-inclusive scene expansion, 3D Gaussians evolve from an initial partial point cloud to a complete point cloud that encapsulates the entire scene over the sequence. In our global optimization stage, we still utilize the $\mathcal{L}_{KEA}$ loss as new Gaussians are added during densification.

This equation calculates the Chamfer distance by summing the squared Euclidean distances from each parameter in $h_{i}$ to its closest counterpart in $h_{j}$, and vice versa, thereby quantifying the similarity between the two Gaussians across all included parameters such as color, opacity, etc. Combining all the loss components results in the total loss function during scene expansion,

where $\lambda_{rgb}$, $\lambda_{KEA}$, $\lambda_{ipc}$ and $\lambda_{pc}$ act as weighting factors for the respective loss terms.