Deep Patch-based Human Segmentation

Surface mapping approaches solve the mapping or parameterization, ranging from local patch-like surfaces to global shapes. The Exponential Map is often used to parameterize a local region around a central point. It defines a bijection in the local region and preserves the distance with low distortion. Geodesic Polar Map (GPM) describes the Exponential Map using polar coordinates. [9, 19, 27, 24] implemented GPM on triangular meshes based on approximate geodesics. Exact discrete geodesic algorithms such as [31, 35] are featured with relatively accurate tracing of geodesic paths and hence polar angles. The common problem with GPM is that it easily fails to generate a one-to-one map due to the poor approximation of geodesic distances and the miscalculation of polar angles. To overcome the problem one needs to find the inward ray of geodesics mentioned in [21]. However, sometimes the local region does not form a topological disk and the tracing of the isocurve among the triangles is very difficult. To guarantee a one-to-one mapping in a local patch, one intuitive way is to adapt the harmonic maps or the angle-preserving conformal maps. A survey [10] reviewed the properties of these mappings. The harmonic maps minimize deformation and the algorithm is easy to implement on complex surfaces. However, as shown in [6, 8], in the discrete context (i.e. a triangle mesh) if there are many obtuse triangles, the mapping could be flipped over. [12, 15, 26, 29, 28] solved the harmonic maps on closed surfaces with zero genus, which is further extended to arbitrary-genus by [12, 20]. These global shapes are mapped to simple surfaces with the same genus. If the domains are not homeomorphous, one needs to cut or merge pieces into another topology [32, 7]. These methods are globally injective and maintain the harmonicity while producing greater distortion around the cutting points.

Inspired by current deep learning techniques, there have been a number of approaches attempting to extend these methods to handle the 3D human segmentation task. Limited by irregular domain of 3D surfaces, successful network architecture can not be applied straightforwardly. By leveraging Convolutional Neural Networks (CNNs), Guo et al. [13] initially handled 3D mesh labeling/segmentation in a learning way. To use CNNs on 3D meshes, they reshape per-triangle hand-crafted features (e.g. Curvatures, PCA, spin image) into a regular gird where CNNs are well defined. This approach is simple and flexible for applying CNNs on 3D meshes. However, as the method only considers per-triangle information, it fails to aggregate information among nearby triangles which is crucial for human segmentation. At the same time, Masci et al. [23] designed the network architecture, named GCNN (Geodesic Convolutional Neural Networks), so as to deal with non-Euclidean manifolds. The convolution is based on a local system of geodesic polar coordinates to parameterize a local surface patch. This convolution requires to be insensitive to the origin of angular coordinates, which means it disregards patch orientation. Following [23], anisotropic heat kernels were introduced in [5] to learn local descriptor with incorporating patch orientation. To use CNNs on surface setting, Maron et al. [22] introduced a deep learning method on 3D mesh models via parameterizating a surface to a canonical domain (2D domain) where the successful CNNs can be applied directly. However, their parameterization rely on the choice of three points on surfaces, which would involve significant angle and scale distortion. Later, an improved version of parameterization was employed in [14] to produce a low distortion coverage in the image domain. Recently, Rana et al. [16] designed a specific method for triangle meshes by modifying traditional CNNs to operate on mesh edges.

In this work, we address 3D human segmentation by assigning a semantic label to each vertex with the aid of its local structure (patch). Due to intrinsic irregularity of surfaces, traditional 2D CNNs can not be applied to this task immediately. To this end, we map a surface patch into a 2D grid (or image), in which we are able to leverage successful network architectures (e.g. ResNet [17], VGG [30]).

As shown in Fig. 1, for each vertex on a 3D human model, a local patch is built under geodesic measurement. We then convert each local patch into a 2D grid (or image) via a 3D-2D mapping step, to suit the powerful 2D CNNs. To preserve geometric information both locally and globally, we embed local and global shape descriptors into the 2D grid as input features. Finally, we establish the relation between per-vertex (or per-patch) feature tensor and its corresponding semantic label in a supervised learning manner. We first introduce the surface mapping step for converting a local patch into 2D grid in Section 3.2, and then explain the neural network and implementation details in Section 3.3.

Patch extraction. Given a triangular mesh $M$, we compute the local patch $P$ for each vertex $v\in M$ based on the discrete geodesic distance $d$ by satisfying $d(v_{p})<r_{p}$ for all $v_{p}\in P$. $r_{p}$ is an empirically fixed radius for all patches. Assume the area of $M$ is $\alpha$, $r_{p}=\sqrt{(\alpha/m)}$, where $m$ is set to $1000$ in this work. The geodesic distance $d$ is computed locally using the ICH algorithm [35] due to its efficiency and effectiveness.

Parameterization. There are two cases for parameterization in our context, whereby $P$ is a topological disk and otherwise. For the former case, we denote the 2D planar unit disk by $D$, we compute the harmonic maps $\mu:P\to D$ by solving the Laplace equations

with Dirichlet boundary condition

where $v_{i}$ is an interior vertex of $P$ (Eq. (1)) and $c_{ij}$ is the cotangent weight on edge $(v_{i},v_{j})$. In Eq. (2), $v^{\prime}_{k}$ ($k\in[1,m]$) belongs to the boundary vertex set of $P$. The boundary vertex set contains $m$ vertices, which are sorted in a clockwise order according to the position on the boundary of $P$. Suppose $(v_{i},v_{j})$ is an interior edge. $(v_{i},v_{j},v_{k})$ and $(v_{i},v_{j},v_{l})$ are two adjacent triangles, $c_{ij}$ is calculated as

where $\beta_{k}$ and $\beta_{l}$ is the angle between $(v_{i},v_{k})$ and $(v_{j},v_{k})$, and between $(v_{i},v_{l})$ and $(v_{j},v_{l})$, respectively.

There are cases where the local patch $P$ is not a topological disk and the harmonic maps can not be computed. In this case, we trace the geodesic paths for each $v_{p}\in P$, by reusing the routing information stored by the ICH algorithm when computing $d$. See Fig. 2 for illustration of parameterization. Similar to [23], we then obtain a surface charting represented by polar coordinates on $D$. We next perform an alignment and a $32\times 32$ grid discretization on $D$.

Alignment and grid discretization. The orientation of $D$ is ambiguous in the context of the local vertex indexing. We remove the ambiguity by aligning each patch with a flow vector field $\mathcal{\phi}$ on $M$. For each vertex $v\in M$ and its associated patch $P_{v}$, the flow vector $\mathcal{\phi}(v)$ serves as the reference direction of $P_{v}$ when mapping to $\mathbb{R}^{2}$. Fig. 2 illustrates the reference direction as an example. $\mathcal{\phi}$ is defined as a vector field flowing from a set of pre-determined sources $v_{s}\in M$ to the sinks $v_{t}\in M$. We initially solve a scalar function $u$ on $M$ using the following Laplace equation.

and the flow vector field is $\mathcal{\phi}=\nabla u$. We further calibrate the polar angles of $D$ with $\phi$. Considering the first adjacent edge $e_{base}$ around a source vertex $v_{s}$ as a base edge, BaseToRef is the angle between the projected flow vector $\phi^{\prime}(v_{s})$ and $e_{base}$. $\phi^{\prime}$ is the projected $\phi$ onto a random adjacent face of the base edge. From the harmonic maps $\mu$, we easily obtain the polar angles of the local, randomly-oriented polar coordinate system. The polar angles are represented by AxisToV for all $v\in D$ and AxisToBase for $e_{base}$. To align the local polar axis to the reference direction, the calibrated polar angle for all $v\in D$ is calculated as $\theta_{v}=\textit{AxisToV}-\textit{AxisToBase}-\textit{BaseToRef}$.

The grid with $32\times 32$ cells is embedded inside the calibrated $D$ such that $D$ is the circumcircle of the grid. We build a Cartesian coordinate system in $D$, and the origin is the pole in the polar system. The x-axis and the y-axis overlap the polar axis and $\theta=\pi/2$, respectively. The vertices and triangles on $D$ are converted to this Cartesian coordinate system. Some cells belong to a triangle if the cell centers are on the triangle. We compute the barycentric coordinates of each involved cell (center) with respect to the three vertices of that triangle. The barycentric coordinates will be used for calculating cell features based on vertex features later.

Shape descriptors. After generating $32\times 32$ grids (or images), we embed shape descriptors as features into them. The features of each cell are calculated with linear interpolation using the barycentric coordinates computed above. The descriptors include Wave Kernel Signature (WKS) [3], curvatures (minimal, maximal, mean, Gaussian) and average geodesic distance (AGD) [18]. We normalize each kind of descriptors on a global basis, that is, the maximum and minimum values are selected from the descriptor matrix, rather than simply from a single row or column of the matrix.

Neural network. As a powerful and successful network, we adopt VGG network architecture with $16$ layers (see Fig. 1) as our backbone in this work. The cross-entropy loss is employed as our loss function for the VGG16 net. It is worth noting, however, that the surface parameterization presented in this work is quite general in nature, being applicable to many other CNNs elsewhere in the literature.

Implementation details. We implement the VGG16 network in PyTorch on a desktop PC with an Intel Core i7-9800X CPU (3.80 GHz, 24GB memory). We set a training epoch number of $200$ and a mini-batch size of $64$. SGD is set as our optimizer and the learning rate is decreased from $1.0\times 10^{-3}$ to $1.0\times 10^{-9}$ with increasing epochs. To balance the distribution of each label, in the training stage we randomly sample $5,000$ samples per label in each epoch. Training takes about $3.5$ hours on a GeForce GTX 2080Ti GPU (11GB memory, CUDA 9.0).

Once the model is trained, we can infer semantic labels of a human shape in a vertex-wise way. Given a human shape, we first compute the involved shape descriptors for each vertex. For each vertex, we build a local surface patch and parameterize it into a 2D grid (or image) as described in Section 3.2. We embed all the shape descriptors into a 2D grid and feed it into our trained model for prediction.

In this work, we use dataset from [22] which consists of $373$ train human models from SCAPE [2], FAUST [4], MIT [33] and Adobe Fuse [1], and $18$ test human models from SHREC07 [11]. Some examples of our training dataset are shown in Fig. 3. For each human model, there are $8$ semantic labels (e.g., Head, Arm, Torso, Limb, Feet), as shown in Fig. 1. To represent geometric information of a human model both globally and locally, we concatenate a set of shape descriptors as input features: $26$ WKS features [3], $4$ curvature features ($C_{min}$, $C_{max}$, $C_{mean}$, $C_{gauss}$) and AGD [18].

To provide a fair comparison, we also evaluate our segmentation results in an area-aware manner [22]. For each segmentation result, the accuracy is computed as a weighted ratio of correctly labeled triangles over the sum of all triangle area. Therefore, the overall accuracy on all involved human shapes is defined as

where $N$ denotes the number of test human models and $A_{i}$ is the sum of triangle area of the $i$-th human model. $J_{i}$ is the set including the indices of correctly labeled triangles of the $i$-th human model and $a_{ij}$ represents the $j$-th triangle area of the $i$-th human model. Since we address the human segmentation task in a vertex-wise manner, the per-vertex labels need to be transferred into per-face labels for the quantitative evaluation. The face label is simply estimated by using a voting strategy among its three vertex labels. We immediately set the label with two or three vertices as the label on the face. We randomly select a vertex label as the face label, if three vertex labels are totally different.

In this section, we show the visual and quantitative results. As shown in Fig. 4, the top row lists several of our results in the test set, and the bottom row displays the corresponding ground-truth models. To further evaluate our method for 3D human segmentation, a quantitative comparison with recent human segmentation techniques are summarized in Table 1. As we can see from Table 1, our method achieves an accuracy of $89.89\%$, ranking the second place among all methods. Our approach is a bit inferior to the best method [14] which certainly benefits from its data augmentation strategy.

Besides the above results, we also evaluate different selection choices for input features. Table 2 shows that the input features including WKS, curvatures and AGD obtain the best performance, in terms of accuracy. Moreover, we evaluate the performance of two different neural networks in 3D human segmentation, as shown in Table 3. It is obvious that the VGG16 obtains a better accuracy than the ReseNet50, and we thus employ VGG16 as the backbone in this work.