Few-Shot Single-View 3-D Object Reconstruction with Compositional Priors

Recently a focus in 3D reconstruction from single or multiple view has been on finding better shape representations and alternative decoder models to the typically used 3D discretized set of voxels and corresponding 3D CNN decoder. [2, 7, 32, 34, 35], one that can permit more efficient learning and generation. These include point clouds [4], meshes [30], signed distance transform based representations [19, 16]. Indeed there is not an agreed upon canonical 3D shape representation and decoder structure for use with deep learning models. However, [28, 16] has shown that although many of these models improve over each other they do not beat naive baselines such as nearest neighbors.

Few-shot learning has become a highly popular research topic in computer vision and machine learning [22, 6]. Typically research focuses on the classification task, with few works investigating more complex problems such as segmentation [26] or object detection. Existing methods comprise two categories: meta-learning/meta-gradient based approaches [5], and metric-learning [21, 27]. The former aims to teach models to adapt quickly, in a few gradient updates, to new unseen classes, while the latter learns a distance metric such that the distance of a query image to the few annotated examples of the same class is minimal.

Consider an encoder-decoder framework involving

• •

  an encoder $E_{I}$ that takes a 2D image, $I$, and outputs its embedding, $e_{I}$;

• •

  a category specific shape embedding, $e_{S}$;

• •

  a decoder $D$ that takes the image and shape embeddings and outputs the reconstructed 3D shape, $\bar{S}$, in the form of a voxelized grid.

This relationship is formally expressed by

This model can be trained using a binary cross entropy loss on the voxel occupancy confidence $p_{i}$ for voxel $i$ in the output grid given

In the rest of the text, we drop $S$ for notational simplicity. For base class training, $E_{I},E_{S}$, and $D$ are learned by minimizing (2). For inference on examples of new classes, $e_{S}$ is computed and fed along with extracted image embedding as input to the trained network.

In [29] (illustrated in Figure 2 (top)), $e_{S}$ is computed with a shape encoder $E_{S}$ that takes a category-specific shape prior $S^{p}_{c}$; $S^{p}_{c}$ is either a randomly selected shape from the set of training shapes associated with class $i$, or the average of all training shapes for a class in voxel space. Providing an explicitly defined single shape or average shape as a shape prior has severe limitations. Such a prior cannot account for intra-class variability and is therefore intrinsically sub-optimal in settings with more than one new examples are considered. We propose to address this issue by learning a global class embedding (GCE), $e_{S}^{i}$, that captures the “essence” of object class-$i$. The GCE is built using all available shapes for a particular class, we expect a high-dimensional latent representation to be much more successful in capturing nuances (like intra-class variability) than simple shape averaging.

Our framework is illustrated in Figure 2 (bottom). The encoder $E_{I}$ and decoder $D$ are trained jointly with the base class embeddings $e_{S}^{i}$ on the set of base classes, minimizing (2). For novel classes with a small training set $\{(I_{i},S_{i})\}_{i=1}^{N}$, all model parameters of $E_{I}$ and $D$ are fixed, and class specific embeddings $e_{S}^{i}$ are obtained by solving

using all of the available training images. We note that, due to our few-shot set-up, this optimization problem can be solved in few iterations since it only involves a small set of parameters ($e_{S}^{i}$) and a small amount of new samples. By construction, this model does not lose performance on base classes and can continually add multiple new classes, while at the same time learning implicit shape prior representation to guide the decoding process.

Finally we note that we use concatenation at the first stage of $D$ to combine $e_{I}$ and $e_{S}$, instead of the element-wise sum of $e_{I}$ and $e_{S}$ used in [29].

GCE allows us to exploit all available, class specific training shapes to learn a representative shape prior. However, the learned global embeddings do not explicitly exploit similarities across different classes, which may result in sub-optimal, and potentially redundant representations.

However, it does not explicitly exploit similarities across shapes, a strategy which would allow increased robustness in the lowest data regimes. We introduce a novel strategy to address this issue, attempting to learn compositional representations between classes which we call Compositional Global Class Embeddings (CGCE). This model is illustrated in Figure 3.

Our objective is to explicitly encourage the model to discover shared concepts among shapes which can be reused across shapes. Taking inspiration from work on compressing word embeddings [25], we propose to decompose our class representation into a linear combination of vectors that are shared across classes. More specifically, we learn a set of $M$ codebooks (or embedding tables), with each codebook $C_{i}$ comprising $m$ individual embedding vectors (or codes) $C_{i}=\{e_{i,1},\dots,e_{i,m}\}$, where $e_{i,m}\in\mathcal{R}^{D}$. Intuitively, each codebook can be interpreted as the representation of an abstract concept which can be shared across multiple classes.

For each class $i$, a learned attention vector $\boldsymbol{\alpha_{i}}$ selects the most relevant code(s) from each codebook. The codes of all codebooks are then combined to yield an final embedding: $e_{S}^{i}=\sum_{k=1}^{M}\sum_{j=1}^{m}a_{i}^{k,j}c_{k,j}$ where $a_{i}^{j,k}$ corresponds to scalar attention on the code $j$ at codebook $k$, while $c_{j,k}$ corresponds to the $j^{th}$ code of codebook $k$. During base class training we learn both $\boldsymbol{\alpha}_{i}$ and $c_{j,k}$. We highlight that codebooks are shared across classes and therefore need only be trained on base classes. As a result, $\boldsymbol{\alpha}_{i}$ constitutes the only class specific variable to fine-tune on novel classes:

Since we would like each codebook to assign a discrete attribute to each class, we aim for the model to select few codes in a given codebook. We thus use a form of attention that relies on the sparsemax [14] operator. Specifically each codebooks attention is given by $a_{c}^{k}=\textsc{sparsemax}(w_{c}^{k})$, where $w_{c}^{k}$ is a learned parameter and the sparsemax operator gives an output that sums to $1$, but will typically attend to just a few outputs.

Another inductive bias we can add is the use of multi-scale structure in the shape construction. An elegant way to do this is by applying the conditional batchnorm technique [20] to the 3D decoder model. Conditional batch normalization replaces the affine parameters in all batch-norm layers with embeddings at each layer. Since 3D decoders have an inherent multi-scale structure that gradually creates more refined areas in the output, each layer’s batch-norm parameters can be seen as defining the class conditional structure at different scales. As in GCE when receiving novel classes we learn the conditional batchnorm parameters for the new class, keeping all other weights frozen. We refer to this approach as Multi-scale Conditional Class Embeddings (MCCE).

In this section, we introduce and discuss three simple baselines that are used in our experiments. First, we consider an oracle nearest neighbor (ONN) [28] baseline. Given a query 3D shape, ONN exhaustively searches a shape database for the most similar entry with respect to a given metric, (Intersection-Over-Union/IoU in this case). Although this method cannot be applied in practice, it provide an upper bound on how well a retrieval method can perform on the task. With ONN, we aim to show that unlike the standard paradigm [28], the few shot generalization benchmark cannot be solved with such a naive baseline.

We further consider a zero-shot (ZS) baseline, and an all-shot (AS) baseline. For the ZS baseline, we train encoder decode model as described in Eq. (1) and use to infer 3D shapes for novel classes, without using the category-specific shape prior $e_{S}$. We expect this to give a lower bound of performance, since it does not make any use of shape prior information. For the AS baseline, we merge the base class and novel class datasets, obtaining $D_{A}=D_{b}\bigcup D_{n}$. We train the model on this joint dataset and then test on examples from only the novel classes. We expect that this baseline will set an upper bound on the performance of the vanilla encoder-decoder architecture, since the model also has access to the examples from the novel classes in $D_{n}$.

For our experiments we use the ShapeNetCore_v1.0 [1] dataset and the few-shot generalization benchmark of [29]. As in [29] we use the following 7 categories as our base classes: plane, car, chair, display, phone, speaker, table. We also use following 10 categories as our novel classes: bench, cabinet, lamp, rifle, sofa, watercraft, knife, bathtub, guitar, laptop. Note that we have added additional categories to the standard benchmark, for a more extensive evaluation. Out data comes in the form of pairs of $128\times 128$ sRGB images rendered using Blender [3], and $32\times 32\times 32$ voxelized representations obtained using Binvox [18, 17]. Each 3D model has 24 associated images, rendered from random viewpoints. For evaluation we use the standard Intersection over Union (IoU) score to compare predicted shapes $\bar{S}$ to ground truth shapes $S$: $\text{IoU}=|S\cap\bar{S}|/{|S\cup\tilde{S}|}$.

All methods are trained on the 7 base classes except for the AS-baseline which is trained on all 17 categories. All methods share the same 2D encoder and 3D decoder architectures. We use the same 2D encoder as in [23, 29], a ResNet [8] that takes a $128\times 128$ image as input, and outputs a 128-dimensional embedding. Our 3D decoder consists of 7 convolutional layers, followed by batch-normalization, and ReLU activations. For training, we use the same 80-20 train-test split as in R2N2 [2, 29]. Unless otherwise stated, we use $l_{r}=0.0001$ as the learning rate and ADAM [11] as the optimizer. All networks are trained with binary cross entropy on the predicted voxel presence probabilities in the output 3D grid.

We first illustrate that naive baselines perform poorly in a few-shot setting [29] which requires good generalization about shapes. We do this by performing a comparison to two baseline methods shown in Table 1. The first is a zero-shot (ZS) baseline, where a model is trained on base classes and used directly to infer shapes of novel classes. The all-shot (AS) baseline is trained on all classes using the full data (which has 1000-8000 shapes per class). We can expect these two baselines to be upper and lower bounds on the performance of the few-shot generalization benchmark. We use the nearest neighbor oracle described in Sec. 3.4 for several possible levels. We observe that the full nearest neighbor oracle outperforms the deep learning model trained on all the data (as observed in [29]) indicating the task does not require generalization. On the other hand in the few shot case we see that performance is often below that of even the ZS baseline, indicating this task is appropriate for evaluating model generalization.

Note that the ZS benchmark in Table 1 already obtains relatively high performance on select classes (sofa and cabinet), we hypothesize this is due to similarity towards base classes. In order to best evaluate these methods we take the novel classes and attempt to compute a similarity metric of new classes towards base classes. We do this by finding the average IOU of the nearest neighbor from base classes. Specifically for each shape in the novel class we compute $max_{j\in BaseShapes}IOU(S_{novel}^{i},S_{base}^{j})$ for all novel shapes $i$ in a given class and average across the class to obtain the proximity scores. The proximity scores for different classes are illustrated in Table 2. We observe in Figure 4 that indeed the higher performing zero-shot cases correspond to close proximity classes, we thus use the proximity criteria as an additional consideration when comparing methods. Furthermore to extend the number of distant classes to the base class set, we select 4 additional far proximity classes from ShapeNet than the original benchmark used in [29], these are included in green in Table 2.

We now evaluate the methods discussed in Sec 3. We evaluate the three methods on the 1-shot baseline in Table 3. We report both the IOU as well as the relative improvement over the ZS baseline. Note that as the ZS-baseline provides strong performance for easy classes, the average IOU is dominated by these, thus relative improvement is a more meaningful metric for aggregation across classes. Observe that GCE improves performance over the method of [29], particularly in cases where the classes are distant, obtaining $45\%$ improvement over Zero-Shot overall compared to [29]. CGCE and MCCE give a more drastic improvement in the performance obtaining $54\%$ and $52\%$ respectively by adding multiscale and compositional priors to the simple class prior of [29].

As discussed in Sec. 3 our approach based on global conditional embeddings is able to capture intra-class variability and thus extends more naturally beyond the 1-shot setting. In Table 4 we evaluate the compositional method (which performs best in 1-shot evaluation) on 10 and 25-shot settings and compare to [29]. We observe that similar to the 1-shot case most methods do not improve much the performance for close-proximity cases. However, for more distant classes we can see substantially performance improvement (sometimes 200$\%$+ in IOU). In Table 4 we observe the increased performance of CGCE versus [29] for increasing shots. Indeed for larger number of observations the method is able to obtain larger performance gains.