Automatic Reverse Engineering: Creating computer-aided design (CAD) models from multi-view images

Photogrammetry is frequently deployed as an image-based technique to measure three-dimensional shapes using inexpensive cameras. The most common monocular approaches are based on the Structure from Motion (SfM) method first described in [35]. Here, the software is provided with several images from different perspectives and then computes a point-cloud of the object of interest.

Automatically extracting a CAD model from a point-cloud is however not straight-forward. For example, the professional AutoCAD software can import but not post-process point clouds as of today [3]. Thus far, CAD model creation mostly remains a manual task.

Kim et al. [15] proposed 3D registration of given CAD model using the iterative closest point (ICP) method. Budroni et al. [5] have demonstrated the fitting of planar surfaces to point clouds for reconstructing of 3D models of interior rooms. More recently, Lui [19] proposed automatic reverse-engineering of CAD models from points clouds by iteratively fitting primitive models based on the RANSAC algorithm. In conclusion, there are few existing approaches which are however domain-specific. Instead, a neural-network based approach might generalize better in the long term.

Detection of 3D objects from multiple view perspectives has been addressed by Rukhovich et al. [33]. Similar to [39], they used a fully convolutional network. Notably the number of monocular images in their multi-view input can vary from inference to inference, offering high versatility. This is achieved by extracting features with conventional a Convolutional Neural Network (CNN), followed by pooling and back-projecting into a 3D volumetric space. In this space, bounding boxes are predicted by three-dimensional convolutions.

For 3D surface reconstruction, deep learning models have been suggested for different kinds of object representations, including point clouds [1, 12, 8, 48, 49, 6, 21], triangle meshes [38, 10, 23, 26], voxel grids [18, 42, 47], cubic blocks [46], parametric surfaces [34, 40, 13, 16, 17, 45], and signed distance fields (SDFs) [28, 14]. The majority of the studies above (e.g. [1, 26, 28, 14]) use auto-encoders, with a feature bottleneck between an encoder and a decoder stage. This network architecture also allows to simplify training by separating the two stages. To date, discrete CAD models have not been investigated for 3D surface representation.

A 3D multi-view CNN (MVCNN) for object classification was introduced by Su et al. [36]. They provide a certain number of images taken of the object as input to a common CNN and pool the extracted features using an element-wise maximum operation. The pooled information is then processed by a second CNN and a final prediction is made. Notably they conclude that inputting 12 evenly spaced perspectives offers the best trade-off between prediction accuracy and memory as well as time resources.

Their concept as been studied for classifying 3D shapes from point clouds [22, 31]. In general, working with MVCNNs seems to be a viable approach for extracting information from 3D scenes. Leal et al. [37] compared different 3D shape classifiers, identifying MVCNNs as superior to other methods due to better generalizability and outperforming several point-based and voxel-based approaches. Consequently, this approach will be followed in this work.

While MVCNNs showed good results for classification tasks, the simple pooling methods (e.g. element-wise max-pooling [36]) might allow a single view to overrule all other views. Geometric information not visible in some images might be lost for a 3D reconstruction task. Hence, we alternatively consider Recurrent CNNs as a more preservative information extractor.

Zreik et al. [50] used a Recurrent Neural Network (RNN) for spacial aggregation of extracted information from 3D angiography images after pre-processing by a 3D-CNN. Liu et al. [20] combined a traditional 2D CNN backbone and an RNN to synthesize multi-view features for a prediction of plant classes and conditions. After extensive experiments, they conclude that a combination of MobileNet as a backbone and a Gated Recurrent Unit (GRU) delivers the best trade-off of classification accuracy and computational overhead. Hence, GRUs will be evaluated in this study for multi-view pooling.

Even though most methods described above generate three-dimensional data, none of them directly attempts CAD file generation by means of generating a construction sequence comparable to manual CAD design. Thus their resulting models cannot easily be modified by an average user. However, recent research has started to address the direct generation of parametric 2D CAD models:

Willis et al. [41] first proposed generative models for CAD sketches, producing curve primitives and explicitly considering topology. SketchGen [27] generates CAD sketches in a graph representation, with nodes representing shape primitives and edges embodying the constraints. Similarly, Ganin et al. [9] utilized off-the-shelf data serialization protocols to embed construction sequences parsed from the online CAD editor Onshape [24].

DeepCAD by Wu et al. [44] was the first approach going beyond the 2D domain of CAD sketch generation. They formulated CAD modeling as a generation of command sequences, specifically tailored as an input to a transformer-based auto-encoder. The publicly available Onshape API was used to build a large dataset of 3D object models for training.

Each object is represented by a CAD sequence, consisting of three common types of commands: (1) Creation of a closed curve profile ("sketch") on a 2D plane, (2) 3D extrusions of such sketches and (3) boolean operations between the resulting 3D objects. Each of the CAD commands supports a number of parameters, which may be a mixture of continuous and discrete values. To conform with their neural network, Wu et al. sort each command’s parameters into a generalized parameter vector and all continuous parameters are quantized to 8-bits. The maximum number of commands in a given CAD construction sequence was limited to 60, corresponding to the longest sequence length in the dataset.

These CAD sequences are processed by an auto-encoder, trained to compress a given CAD model into a latent vector (dimension of 256) and then to reconstruct the original model from that embedding. This means, a random but valid CAD object can be constructed using a given 256-dimensional latent vector. In this work, chose the decoder part of DeepCAD as the generative stage of our new model as introduced next.

We introduce a novel network architecture for end-to-end generation of CAD models from multiple input images. The network is composed of three stages: (1) a CNN encoder backbone to extract information from each input image individually, (2) a pooling network that aggregates this information into a common latent vector, and (3) a generative decoder network constructing the output CAD sequences. This network structure is illustrated in Figure 1.

Considering its successful track record in object detection and classification as well as its small size, we chose the residual network architecture (ResNet) [11] as our encoder backbone. As the visual complexity of our input images is relatively low, we assumed that a smaller, more shallow variant of the network should suffice. Thus only its smallest two variants were evaluated, namely ResNet-18 and ResNet-34. The input image size is adjustable by means of ResNet’s adaptive average pooling layer. In this work, we used 128x128 monochrome as well as 224x224 RGB input images. The output of the last fully connected layer, a vector of fixed length 512, is fed into the pooling network. All input views are processed by the backbone network individually but share the same parameters.

The task of the multi-view pooling stage is to combine the information from multiple views. We evaluated two different network architectures during the experiments: (a) a simple feed-forward fully connected network (FCN) as a baseline model and (b) a gated recurrent unit (GRU). Following [7] and [20], we assume that a recurrent pooling approach should perform favorable, even though its training is inherently more challenging [29] because of the possible vanishing and exploding gradient problems.

The FCN pooling network concatenates the outputs of all backbone CNNs and propagates them through a numbers of layers (1 to 6 layers were evaluated) of linearly decreasing size with a final layer size of 256. This forms the latent vector compatible to the subsequent DeepCAD decoder network.

Unlike the FCN pooling which processes all input views simultaneously, the alternative GRU pooling receives the input views from the backbone CNN sequentially one after the other. This makes it more suitable for varying numbers of images. For evaluation of the GRU pooling stage, we tested different numbers of layers (1 to 8) of identical dimension, different temporal pooling strategies (mean, max, last) and different layer dimensions (64, 128, 256, 512, 1024, 2048). A single fully connected layer is used to achieve the latent vector size of 256.

Both pooling network variants use rectified linear units (ReLU) as their non-linear activation function in all layers except the last. The final layer generates the latent vector. Here the hyperbolic tanget function ($tanh$) is utilized as it provides output in the range $[-1,1]$ as required for the DeepCAD decoder network.

The final stage of the ARE-Net is formed by the decoder from the DeepCAD library [43] which generates CAD construction sequences from the 256-dimensional latent vector.

Training was performed in two stages: First, the full DeepCAD auto-encoder was pre-trained as described in [44]. After this training, the final latent vector of each CAD object from the training set was saved. Second, simulated image views were rendered from the ground truth CAD sequences and used to train our backbone and multi-view pooling networks. As the loss function, we used the mean-squared error between the predicted latent vectors of the simulated images and the ground-truth latent vectors from the first training stage. We employed the ADAM-optimizer, using 10 epochs during hyper-parameter optimization and 140 epochs for the final model.

Training images were generated from the DeepCAD dataset consisting of 178,238 CAD models. From each CAD sequence, a 3D mesh object and two different projection datasets were generated: (1) A simple dataset of 128x128 grayscale images from 10 fixed and evenly spaced view angles as shown in Figure 2. (2) A complex dataset of 256x256 RGB images with random but uniform object color from 24 randomly spaced viewing angles. In the second dataset the photogrammetry ground-plane from [4] was used as a base on which each model rests. It is composed of non-repeating patterns and is used as a turntable for real objects during the final real world test. The intention is to provide the model with additional information on orientation and scale of the objects, otherwise lost due to the random viewing angles.

For training on the simple dataset, all 10 images were used. When training on the complex dataset, a random selection of 5 to 20 images was chosen. To allow an unbiased comparison of our network to former work by the DeepCAD researchers, the same training-, validation- and testing-split (90%-5%-5%) used in [44] was applied.

Our model contains several hyper-parameters requiring optimization. General parameters are the learning rate, drop out ratio, weight decay and the number of ResNet backbone layers. The parameters of the two pooling networks are the number and dimensions of layers. For the GRU network, the temporal pooling strategy (mean, max, last) also needed investigation. In order to identify suitable hyper-parameters such as network attributes and training parameters which remain constant during any given training run an incremental experimentation procedure is followed. For hyper-parameter optimization, the Optuna library [25] was used. It allows for efficient search through the high dimensional search space and automatically records results and useful statistics.

To compare the accuracy of the predicted CAD models, three different metrics were employed: The command accuracy $ACC_{cmd}$ measures the agreement of the predicted CAD command type $\hat{t}_{i}$ with the ground truth command type $t_{i}$ for a CAD construction sequence of $N_{c}$ steps:

While $ACC_{cmd}$ measures that fraction of correct commands, the correctness of the continuous parameters of each command shall also be evaluated. The parameter accuracy $ACC_{param}$ quantifies the agreement of a predicted 8-bit CAD parameter $\hat{p}_{i,j}$ to its ground-truth counterpart $p_{i,j}$. Only correctly predicted commands $N_{c2}\leq N_{c}$ were evaluated and a threshold of $\eta=3$ was used, as suggested in [44]:

For geometric evaluation of the 3D model, the so-called Chamfer Distance $CD$ was used [30, 2]. It computes the shortest distance of one point $x$ on the surface $S_{1}$ of the predicted object to the closest point $y$ on the surface $S_{2}$ of the ground-truth object. This is carried out in both directions. In this work, 2000 surface points were evaluated per model.

As no other method generating CAD models is known to us, comparison is performed using the following two methods: (1) The original DeepCAD auto-encoder is fed with ground-truth CAD-sequences to encode a latent vector and decoded again. The predicted CAD sequence is then evaluated by the accuracy metrics described above. By using loss-less input CAD sequences, this approach represents the ideally achievable results in our comparison and will be referred to as the “baseline”.

(2) For a more realistic comparison, the PointNet++ encoder [32] was evaluated as a state-of-the-art method. Point-clouds were sampled from the ground-truth 3D objects. The PointNet++ encoder was used to map the point-clouds into a latent vector and then processed by the Deep-CAD decoder as proposed by [44].

For an initial assessment of the performance of our method on real world images, two test objects were chosen: a cardboard box representing a very simple case and a camera mount as a more complex example. Both are intentionally of uniform color to match the simulated objects seen during training. The objects were placed on a paper version of the photogrammetry ground plane. Then 20 pictures from varying perspectives were taken by a smartphone camera while changing the inclination angle relative to the ground plane and rotating a turntable underneath the object. The image background behind the objects was then cropped away manually. All pictures were sized down to 224x224 pixels and passed into the Automatic Reverse Engineering Network (ARE-Net) with GRU pooling as trained on the simulated complex dataset.