NutritionVerse-Thin: An Optimized Strategy for Enabling Improved Rendering of 3D Thin Food Models

A significant problem that arises when trying to create a 3D model of a thin object is image capture from different viewpoints since the object is unable to stand vertically without external support. Providing the necessary support obscures much of the article and stands out in the background, preventing quality data collection of the object itself. To alleviate this problem and collect multi-view perspectives without obstruction, we first attach the object to a thread using hot glue and next suspend the thread from a platform. We then rotate the thread (and thus the object) to capture images of the object from all 360 degrees.

Another issue is the existence of inconsistent lighting which results in irregularities in surface brightness. To minimize interference and ensure consistency, we suspend the object directly (45 cm) below a light source that shines perpendicular to the ground and dim all other light sources in the vicinity as seen in Fig. 2. A Google Pixel 7 Pro phone is used to record a video of the rotating item; the item always remains centered in the frame. We position the camera parallel to the item at around 10-15 cm from the item as these close-up shots help capture maximum detail, thus optimizing render quality. We gradually move the phone down and subsequently up (while maintaining distance and centering) to capture the top and bottom of the article. The suggested video length is 1:20 as this duration is short enough to avoid lengthy preprocessing times while allowing for competent multi-view data collection.

Approximately 200 frames are then sampled from the video at equal intervals using OpenCV [6]. The backgrounds of the sampled images are removed using Rembg and image masks are generated [7]. The images are downscaled to a resolution of (512x512) to reduce strain on the GPU during the mesh extraction process. The preprocessed images are then inputted to COLMAP [8] which produces intrinsic and extrinsic camera parameters of the accepted images [9]. We find that performing the more comprehensive exhaustive feature matching (compared to sequential matching) produces better results without imposing a significant time delay due to our small image set. Finally, the pose information, images, and image masks are passed into a thin object-optimized differentiable reconstruction method [5] for model training and mesh generation.

We first investigated instant-ngp [10] for rendering of thin objects but found that the outputted mesh had an irregular/rough surface and contained numerous holes as seen in Fig. 1. To address the poor rendering results of existing methods, we propose an extended training strategy for optimizing the differentiable reconstruction method proposed in [5] to handle thin objects. More specifically, we conduct training with a reduced tetrahedral grid resolution, a large Laplacian value, and strong signed distance field regularization to enforce greater mesh smoothness. While these constraints do come at the expense of minor detail loss, they regularize the surface geometric shape of items and result in significant reductions in the jaggedness of the mesh. An additional benefit is the elimination of ghost effects as seen in Fig. 3. Finally, we recommend a lower training resolution and number of iterations as these significantly lower computational costs without incurring any noticeable differences in render quality.