An Interactive Image-based Modeling System

As an important supplement to traditional computer-aided 3D modeling technology, image-based modeling technology has made great progress in recent years. The so-called image-based modeling (IBM) technology is to recover the geometry and surface texture and other attribute information of objects from real images through technical means, so that the created models are more visually realistic.

In IBM technology, there are two main methods: 1) Geometric reconstruction method based on discrete points. It uses the corresponding points on multiple images and adopts the principle of stereo vision to restore the three-dimensional coordinates of the points, and then establishes a polygonal mesh model through the topological connection between these points. The advantage of this method is that it can theoretically reconstruct any shape of geometry; the disadvantage is that in practical applications, for objects with complex geometry and texture, there is a problem of low reliability in automatically selecting and matching corresponding points between multiple images. However, manual matching of corresponding points has the problem of excessive interaction. 2) Modeling method based on predefined voxels. A three-dimensional geometric parameterized model of the target model is given through human-computer interaction, and then by establishing the corresponding point-line relationship between the three-dimensional model and the image and the constraint relationship of the three-dimensional geometry itself, the precise parameters of the geometric model are determined by the optimization method. The advantage of this method is that it utilizes the constraint relationship of the model itself, and the amount of interaction is smaller than the previous method; the disadvantage is that a parameterized model needs to be built in advance, which limits the geometry type of the recoverable target model.

We propose a progressive interactive image-based modeling method using a pre-defined voxel-based approach and incorporating ideas from discrete point methods. Using this modeling approach, we developed an IBM modeling system with high versatility, reliability, and ease of use.

Image-based modeling restores the 3D model based on the information of the image. Without loss of generality, we believe that the 3D model to be restored can be represented as a grid. The mesh model M can be represented by a quadruple (K, V, D, S) [9]. Among them, V is a set of vertex positions v1, . . . , vn, which defines the shape of the mesh; K is the connection relationship between V and defines the topology of the mesh; D defines the discrete surface of the model attributes, such as the surface material identification number; S defines the continuous attributes indicated by the model, such as color, normal, texture coordinates, etc. The goal of image-based reconstruction is to recover such a quaternary relation M in space from the image.

Hoppe proved that the conversion between two grids $M^{n}$ and $M^{0}$ in space only needs limited steps to realize [9]. We introduce such ideas into image-based modeling techniques and propose a progressive interactive image-based modeling method. We assume that the model to be recovered from several images $I_{j}(j=1,...,N)$ is $M^{\infty}$, and the vertex $v_{i}$ must satisfy the projection constraint: $m_{i}^{j}=sP_{j}v_{i}$. Among them, $m^{i}_{j}$ is the plane point on the image $I_{j}$, which is a known condition; $s$ is the projection coefficient; $P_{j}$ is the camera projection matrix corresponding to the image $I_{j}$ ($m^{j}_{i}$ and $v_{i}$ are expressed in the form of homogeneous coordinates). What needs to be solved is $P_{j}$ and $v_{i}$. In addition to the projection constraints, the constraints that can be used in the restoration process are: epipolar constraint between cameras $f(m_{i},m_{i}^{\prime})=0$, which defines the corresponding relationship between the plane projection points of the same space point in different images; The space constraints between vertices $g(v_{i0},...,v_{in})=0$, such as coplanarity, collinearity, and face-to-face parallel, face-to-face perpendicular constraints.

We represent all projection constraints, epipolar constraints and spatial constraints as a constraint relation group F. Theoretically, when there are enough constraints and certain conditions are met, the vertex set V can be recovered by solving F. In addition, if we simplify D and S to the texture and physical coordinates of the mesh surface, then when the topology K and vertex set V of the mesh are known, we can get the vertices of each face on the image plane according to the projection constraints. Through the inverse matrix of the projection matrix, we can inversely obtain the texture coordinates of the surface and the vertex. Therefore, the grid (K, V, D, S) can be obtained by the corresponding calculation of (K, F).

Our modeling system can be regarded as a state machine. The user applies different operators op to the state machine to change into different states by interacting with the system, where the state can be represented by $(K^{i},F^{i})$, and the system can pass the pair $(K^{i},F^{i})$ Calculations are made to obtain mesh models $M^{i}$, these $M^{i}(i=0,...,n)$ will approximate the final mesh $M^{\infty}$ (Figure 1).

The operators are divided into the following three categories according to the different imposed objects.

(1) K-op operators, that is, operators applied to topology K. It is proved in [9] that only one pair of reversible operators is needed for two grids Mn and M 0 in space: vertex splitting and edge merging. Considering that they are not conducive to intuitively controlling the grid during interaction, for this purpose, we define two pairs of equivalent reciprocal inverse operators as shown in Table 1.

In order to construct the mesh from an initial topology quickly, we set several initial topology construction operators ($K_{create}$) are defined to create predefined voxels.

(2) F-op operator, that is, the operator imposed on the constraint F. Two types of operators are defined: A, the projection constraint operator ($V_{proj}$), which specifies the correspondence between spatial points and pixels on the image to establish projection constraints; B, the spatial constraint operator ($V_{space}$), which specifies the spatial constraints between spatial points relation, including the corresponding inverse operator

(3) V-op operator, that is, the operator applied to the vertex set V. Although the system solves V through F, when the number of constraints in F is insufficient or when numerical iteration is used to solve V, and the initial value is not good, the system can be assisted in solving V by directly adjusting the positions of some vertices.

We take the modeling process of the top of a pavilion as an example to illustrate. First, according to the top shape, use the $K_{create}$ operator to create a cuboid voxel, and then through the operator $V_{proj}$, the user drags the cube to the corresponding points under different image projections to establish projection constraints. The system uses the edge vertical information of the cube and the existing entities in the scene to calculate the camera internal parameters of each image; extracts point pairs from the corresponding relationship between multiple images to establish epipolar constraints, and calculates the camera external parameters. According to the scene’s current projection constraints, spatial constraints (in hidden voxels) and updated camera parameters number, which computes the geometry of the model in the scene. Obviously, a cube cannot properly approach the top of the real scene, so the user can continue to use $E_{split}$ to split the edges of the cube and generate new vertices, as shown in Figure 2; The vertices need to be connected with $V_{connect}$ to complete the further approximation of the top shape of the pavilion. After each drag operation by the user, the system updates the model. Finally, the system automatically solves the texture and texture coordinates.

The system is divided into four modules: interface module, space modeling module, plane image processing module and optimization solution module. Figure 3 presents two examples of modeling by the system from multiple images.

We implement a progressive interactive image-based modeling approach with a corresponding system. Compared with the existing IBM method and system, the system better resolves the contradiction between interaction volume, reliability and flexibility of modeling, and creatively introduces the progressive IBM modeling method, which simplifies the modeling process. process and make it conform to human cognitive habits. The system has the following characteristics: progressive interactive solution mode; WYSIWYG interface environment; accurate camera calibration algorithm; texture editing function and accurate texture acquisition. Further work includes: on the basis of reconstructing geometry and restoring texture attributes, how to further restore the surface lighting attributes, and restore the light distribution and attributes in the scene.

There are some extended articles about applying physical lighting computation in various applications:

1. 1.

   Deep Learning-Based Monte Carlo Noise Reduction By training a neural network denoiser through offline learning, it can filter noisy Monte Carlo rendering results into high-quality smooth output, greatly improving physics-based Availability of rendering techniques [14], common research includes predicting a filtering kernel based on g-buffer [2], using GAN to generate more realistic filtering results [26], and analyzing path space features Perform manifold contrastive learning to enhance the rendering effect of reflections [4], use weight sharing to quickly predict the rendering kernel to speed up reconstruction [7], filter and reconstruct high-dimensional incident radiation fields for unbiased reconstruction rendering guide [13], etc.

2. 2.

   The multi-light rendering framework is an important rendering framework outside the path tracing algorithm. Its basic idea is to simplify the simulation of the complete light path illumination transmission after multiple refraction and reflection to calculate the direct illumination from many virtual light sources, and provide a unified Mathematical framework to speed up this operation [5], including how to efficiently process virtual point lights and geometric data in external memory [23], how to efficiently integrate virtual point lights using sparse matrices and compressed sensing [12], and how to handle virtual line light data in translucent media [11], use spherical Gaussian virtual point lights to approximate indirect reflections on glossy surfaces [10], and more.

3. 3.

   Automatic optimization of rendering pipelines Apply high-quality rendering technology to real-time rendering applications by optimizing rendering pipelines. The research contents include automatic optimization based on quality and speed [24], automatic optimization for energy saving [25, 27], LOD optimization for terrain data [17], automatic optimization and fitting of pipeline rendering signals [16], anti-aliasing [29], etc.

4. 4.

   Using physically-based process to guide the generation of data for single image reflection removal [15]; propagating local image features in a hypergraph for image retreival [1]; managing 3D assets in a block chain-based distributed system [19].