Revisiting Lesion Tracking in 3D Total Body Photography

Shape correspondence between non-rigid surfaces represented as triangle meshes has been an active research topic in computer vision and computer graphics [58, 47, 11]. The shape correspondence problem for triangle meshes is finding a set of corresponding points between two meshes. For human shapes, priors of the human body are commonly applied, such as near-isometric deformation and local rigidity for limbs [66, 62, 6, 4].

Given a set of lesions detected in a mesh, we can construct a graph in which a node corresponds to a single lesion and an edge is a connection between a pair of lesions. The node attribute is the position of a lesion, and the edge attribute is the geodesic distance between a pair of lesions. Then, the problem of matching source and target lesions can be formulated as a (partial) graph-matching problem that maximizes the node-to-node and edge-to-edge affinity of the two graphs. Two-graph matching can be modeled as the quadratic assignment problem (QAP), and is known to be NP-hard.

Traditional approaches [30, 15, 9] aim to match graphs by maximizing quadratic objective functions. While effective for simple cases, these methods often struggle with complex graph structures. Some proposed approaches utilize relaxation strategies in graph matching to mitigate the hard combinatorial problem [29, 25, 51]. More recent methods explore hyper-graph matching represented by a tensor to encode the higher-order information, offering increased expressiveness but at the cost of higher computational complexity [14, 41, 64].

Learning-based methods have been shown to improve matching accuracy [61, 46, 31]. Wang et al. [61] presents a QAP network to solve the matching problem as a vertex classification task over the association graph whose nodes represent candidate correspondences between the two graphs and edge weights are induced by the affinity matrix built with the two graphs. Liao et al. [31] convert the problem of hypergraph matching into a node classification problem and develop a hypergraph neural network. Despite their promise, these methods often require exhaustively annotated datasets for training and struggle to generalize across different domains or datasets. Meanwhile, to address real-world scenarios involving noisy or incomplete data, some techniques are developed for partial graph matching and soft assignment [59, 16].

Despite the advances in graph matching, the solution itself (unlike a correspondence map) does not provide location information for unmatched lesions, an issues that is critical in clinical settings. Starting from coarse correspondence maps, we propose a framework that performs lesion assignment and maps lesions onto a template mesh that allows clinical verification.

Several works have been proposed for tackling the skin lesion tracking problem over the full body [27, 28, 53, 54]. Korotkov et al. [28] designed a TBP system with 21 high-resolution cameras and a turntable to track lesions. However, their method assumes the patient poses are the same across visits and heavily rely on calibrated camera poses for finding lesion correspondence. The work of Korotkov et al. [27] improved the earlier method but does not extend to multiple visits. Strzelecki et al. [54] developed a TBP system with one digital camera rotating and moving vertically around the subject. The camera captures 32 images for lesion detection and lesion matching [53] based on feature matching and triangulation. However, their method fails when the skin surface is inclined at an angle deviating significantly from 90^∘ with respect to the camera viewing direction. Overall, these methods are limited to a controlled environment and sensitive to camera perspectives and changes in body poses [22].

Recently, the concept of finding lesion correspondence using a 3D representation of the human body has been explored in [8, 65, 3, 22]. Zhao et al. [65], Bogo et al. [8], and Ahmedt-Aristizabal et al. [3] proposed to use a template mesh and rely on anatomical position defined on the template mesh for lesion matching. However, using the template-based correspondence map is insufficient to accurately match lesions. Furthermore, these methods have limited positional expression for lesions since they “snap” the location of a lesion to the nearest vertex. Huang et al. [22] proposed to improve the lesion correspondence localization accuracy using landmark-based correspondences refined by texture information. However, their method requires manual annotation of landmarks, is limited to the resolution of the mesh, and is sensitive to inconsistent texture between scans due to scanning artifacts. In this paper, we represent lesion positions using barycentric coordinates within a triangle, making our approach less sensitive to the resolution of the mesh and allowing us to achieve a higher matching accuracy. Additionally, we utilize lesion signals that are agnostic to the textured mesh assuming lesions are provided separately.

Given a template mesh $\mathcal{M}_{\mathcal{T}}$, source and target meshes $\mathcal{M}_{0}$, $\mathcal{M}_{1}$, and two sets of detected lesions $X_{0}\subset\mathcal{M}_{0}$, $X_{1}\subset\mathcal{M}_{1}$, we would like to find correspondence maps $\phi_{0}^{\mathcal{T}}:\mathcal{M}_{0}\rightarrow\mathcal{M}_{\mathcal{T}}$ and $\phi_{1}^{\mathcal{T}}:\mathcal{M}_{1}\rightarrow\mathcal{M}_{\mathcal{T}}$, and a matching matrix $\pi=\{0,1\}^{(|X_{0}|+1)\times(|X_{1}|+1)}$ minimizing an energy of consisting of two terms:

That is, we want a pair of corresponding source and target lesions to be close to each other while encouraging the correspondence matrix to be doubly stochastic. By adding a dummy lesion to each of the lesion sets in $\pi$, we allow the matching function to account for unmatchable lesions. Specifically, assuming a lesion in the source scan can be matched to at most one lesion in the target scan and vice versa, we also enforce:

(where $\pi_{i,0}=1$ indicates a match between the $i$-th lesion on the source and the target’s dummy lesion). Since the dummy lesions can be matched multiple times, the sums involving the dummy lesions can be greater than 1.

The template-based correspondence maps are coarse for two reasons. First, when fitting a template mesh to source/target scan, non-isometric deformation is present at locations near body joints and locations of soft tissues. Second, misalignment between the deformed template mesh and the input mesh occurs if the body pose of the input mesh is far from the canonical “T” pose. Since the coarse correspondence map relies on the nearest point on the registered template mesh to the query point, such a misalignment degrades the accuracy of the mapping. Consequently, a pair of corresponding points in the source and the target will not map to the same position on the template mesh. To refine the correspondence map, we transfer the texture and lesion signals of the source/target to the template mesh using the source/target-to-template correspondences. We then construct a vector field on the template mesh that aligns the transferred signals.

Given source/target correspondence maps $\phi_{i}^{\mathcal{T}}:\mathcal{M}_{i}\rightarrow\mathcal{M}_{\mathcal{T}}$, we expect lesions, $x_{0}\in X_{0}$ and $x_{1}\in X_{1}$ to be in correspondence if the geodesic distance between $\phi_{0}^{\mathcal{T}}(x_{0})$ and $\phi_{1}^{\mathcal{T}}(x_{1})$ is small. Conversely, we expect $x_{0}\in X_{0}$ (resp. $x_{1}\in X_{1}$) to be unmatched if the geodesic distance from $\phi_{0}^{\mathcal{T}}(x_{0})$ to $\phi_{1}^{\mathcal{T}}(x_{1})$ for all $x_{1}\in X_{1}$ (resp. from $\phi_{1}^{\mathcal{T}}(x_{1})$ to $\phi_{0}^{\mathcal{T}}(x_{0})$ for all $x_{0}\in X_{0}$) is large. We formalize these observations, expressing the assignment matrix $\pi\in\{0,1\}^{(|X_{0}|+1)\times(|X_{1}|+1)}$ as the minimizer of the energy:

where $D_{\mathcal{T}}:\mathcal{M}_{\mathcal{T}}\times\mathcal{M}_{\mathcal{T}}\rightarrow{\mathbb{R}}^{\geq 0}$ is the geodesic distance function on $\mathcal{M}_{\mathcal{T}}$. The assignment problem can be optimized through the Kuhn–Munkres algorithms [39]. We follow the implementation in Pygmtools [60] to solve the minimizer to Eqn. 8.

We extend the 3DBodyTex dataset [48, 49] by annotating lesion correspondence in every pair of meshes. Following the suggestion from a medical expert reported in [57], the labeling process is inclusive for anything that could be potentially considered a skin lesion (e.g. including freckles). The dataset is labeled by four experienced annotators using the point list picking function in CloudCompare [1]. It takes an annotator approximately 15 minutes to label one subject with 100 lesion pairs in two poses (meshes). We excluded subjects when fewer than 10 lesions were found.

The average number of annotated lesions is 129.6 ($\sigma=88.4$) across 198 subjects, totaling 25,666 lesions. We define the density of the annotated lesions as the number of neighboring lesions within a geodesic distance of 100 mm. Across all subjects, the average density is 8.5 with $\sigma=6.1$. In addition, the lesions are distributed with 12,846 lesions on the trunk, 3761 lesions on the upper right limb, 3617 lesions on the upper left limb, 2139 lesions on the lower left limb, 2291 lesions on the lower left limb, and 1012 lesions on the head. Overall, numerous skin lesion pairs are annotated with diversity in body shapes, sizes, poses, and anatomical variations. Therefore, the proposed dataset is suitable for the evaluation of skin lesion tracking that approximates real-world scenarios for TBP. The distribution of the lesion annotations can be found in the supplement.

We define a “challenging-pose” subset that consists of 6 subjects in which one of the two poses is challenging. This subset is separated from the entire dataset and evaluated individually. Furthermore, since lesion tracking is most valuable for patients with numerous and dense lesions, we define another subset of 35 subjects as a “numerous-lesions” set in which subjects are annotated with more than 200 lesions.

To evaluate the quality of the established correspondence maps, we measure the geodesic distance between a pair of source and target lesions mapped on the template mesh. We report 1) the average geodesic distance across all the annotated lesion pairs ($D_{\text{LP}}$) and 2) the subject-wise geodesic distance ($D_{\text{SW}}$) as the average geodesic distance of the annotated lesion pairs for the individual subject, and then aggregated across all the subjects. To interpret the geodesic distance between a pair of source and target lesions in a clinical application, a pair of lesions is successfully mapped if the distance between them is less than a threshold criterion. Using the threshold criterion of 10 mm (as in [22]), we measure the success rate for each subject as the percentage of the correctly mapped source and target skin lesions over the total number of annotated skin lesion pairs. We report the subject-wise success rate computed on a pair of meshes (for one subject) and averaged across paired meshes.

We compare our method to two baseline methods that rely solely on geometry to provide correspondence maps, SMPL-NICP [35] and DiffusionNet [52]. We note that DiffusionNet [52] is used as a feature extractor to transform each vertex into a higher-order embedding. Therefore, shape correspondence from DiffusionNet [52] belongs to the category of canonical embedding in shape correspondence (described in §2.1), relying on functional maps to compute correspondence between shapes assuming descriptor preservation over the underlying meshes. Table 1 shows the comparison of the established correspondence maps. For both $D_{\text{LP}}$ and $D_{\text{SW}}$, the correspondence maps from our method are substantially more accurate with a smaller variance compared to the baseline methods. As a result, the proposed method achieves a success rate of 89.9% ($\sigma=4.7\%$) at the 10-mm criterion, significantly higher than baseline methods. We note that the reported success rate from [22] is only 57% ($\sigma=14\%$), computed over 10 subjects and totaling around 200 lesions within our dataset.

Fig. 3 (a) shows the distribution of the geodesic distance between all the lesion pairs. Fig. 3 (b) shows the distribution of the subject-wise geodesic distance between lesion pairs. We observe that the proposed method effectively aligns lesion pairs closer and reduces the long-tail distribution as well. To further investigate the improvement of flow field refinement, we compare correspondence maps established by the proposed method and SMPL-NICP for an individual subject. Fig. 4 (a) shows the distribution of the geodesic distances for all the lesion pairs on the subject. In Fig. 4 (b), we observe that the flow field refinement is also effective in aligning lesion pairs that are originally far from each other (e.g. with a geodesic distance of more than 30 mm). Fig. 4 (c) visualizes the source and the target mesh of the subject, demonstrating challenges of non-isometric deformation due to soft tissue and differing poses. Fig. 4 (d) visualizes the texture signals transferred to the template mesh. In Fig. 4 (e), we show the source and the target lesions mapped onto the template mesh with (ours) and without (SMPL-NICP) flow field refinement. We observe that after the refinement source and target lesions are mapped more closely together, facilitating the task of lesion assignment.

We compare the proposed framework to existing lesion matching methods from Zhao et al. [65] and Ahmedt-Aristizabal et al. [3]. We note that the two methods essentially rely on the anatomical position of the source lesions and the target lesions that are mapped in the template mesh while differing in two ways. First, Ahmedt-Aristizabal et al. [3] use linear assignment, whereas Zhao et al. [65] resort to quadratic assignment using the preservation of geodesic distance between lesion pairs. Second, Ahmedt-Aristizabal et al. [3] selects LoopReg [7] for template registration, while Zhao et al. [65] selects 3D-CODED [18]. To eliminate the effect coming from different registering methods, we re-implement their methods using SMPL-NICP [35] for template registration, similar to what is used in our framework.

For each subject, we calculate the matching accuracy as the number of correctly matched lesions over the number of annotated lesion pairs. Table 2 shows the comparison of the matching accuracy evaluated with different subsets of the dataset. The matching accuracy of the proposed method is the highest for the entire dataset and the “numerous-lesions” subset. In particular, when only evaluating subjects within the “numerous-lesions” subset, we observe that the improvement from our method is more pronounced. In this subset, our method achieves a 98.2% ($\sigma=1.5\%$) matching accuracy over 35 subjects totaling 9772 lesion pairs, as compared to 92.2% ($\sigma=16.9\%$) and 96.0% ($\sigma=5.7\%$) for the methods of Zhao et al. [65] and Ahmedt-Aristizabal et al. [3], respectively. Therefore, the proposed method effectively pairs up skin lesions for subjects with numerous lesions, the most important benefit of using TBP for skin cancer. The matching accuracy for individual subjects within the “numerous-lesions” subset is shown in the supplement. We remark that the reported matching accuracy from Zhao et al. [65] is 83% ($\sigma=38\%$) conducted on 10 subjects and totaling around 200 lesion pairs. We also observe that the proposed method is more robust to differences in topology between source and target meshes. A subject with changes in topology between the two scans and the results can be found in the supplement.

For the “challenging-pose” subset, the approach from Zhao et al. [65] gives the highest matching accuracy, followed by our method. We note that the template-based correspondence map fails when the poses are challenging, either unseen in the training dataset (in SMPL-NICP [35]) or far from the template “T” pose. In this case, comparing the methods from Zhao et al. [65] and Ahmedt-Aristizabal et al. [3], the preservation of the geodesic distance between pairs of lesions helps to improve the matching accuracy beyond linear assignment.

We evaluate the robustness of the proposed framework to noisy lesion detection by independently taking out $p$% of lesions in the source and target, and then pairing up lesions. The evaluation is performed on the “numerous-lesions” subset. We compare the proposed framework to the approaches from Zhao et al. [65] and Ahmedt-Aristizabal et al. [3].

We compute precision, recall, and F1 scores between the source and target lesions for each subject and then report the average and standard deviation across subjects. The F1 score is the harmonic mean of precision and recall. Given predicted matrix $\pi_{pred}$ containing $k_{pred}$ matches, and ground truth matrix $\pi_{gt}$ containing $k_{gt}$ matches, denote $\odot$ as element-wise product, we have:

Fig. 5 compares the precision, recall, and F1 scores for the three methods under different noise levels. Compared to the baseline methods, the proposed method is consistently more robust to errors in lesion detection for a noise level smaller than 25%. We note that the reported recall for lesion detection by Zhao et al. [65] ranges from 78% to 96%, validating that the noise range in our experiment is practical. Furthermore, while the preservation of geodesic distance helps the lesion matching when correspondence maps fail to align lesions (in §4.3), it is notoriously sensitive to noise, which is corroborated in our findings.

The choice of the barycentric coordinate representation for lesions (and correspondence maps) tackles the limitation of inaccurate and resolution-dependent matching present in previous works [65, 3, 22]. As a comparison, on the entire dataset, the average matching accuracy across all the subjects using the vertex representation is 82.9% ($\sigma=11.2$), as compared to 98.3% ($\sigma=3.6$) evaluated with our re-implementation of approach from Ahmedt-Aristizabal et al. [3]. As expected, we observe a consistent decrease in matching accuracy as the lesion count increases due to inaccurate representation. A detailed comparison of the average matching accuracy for different numbers of lesions per subject can be found in the supplement. In addition, the selected representation is robust to the mesh resolution. Our evaluation is conducted on the low-resolution mesh from the 3DBodyTex dataset [48, 49] with 10K vertices on average, whereas both Zhao et al. [65] and Haung et al. [22] use high-resolution mesh with 300K vertices from the same dataset.

Our method achieves state-of-the-art matching accuracy without sacrificing efficiency. For each subject, our framework takes around 7 minutes, including the following components: 1) Coarse correspondence map (4 minutes). 2) Signal construction on template mesh (20 seconds). 3) Surface optical flow (150 seconds). 4) Lesion assignment (1 second). In particular, for the “numerous-lesion” subset, our method performs significantly better than the approach from Ahmedt-Aristizabal et al. [3] with an acceptable computation overhead of 3 minutes. On the other hand, the quadratic assignment used by Zhao et al. [65] is NP-hard.

Our dataset includes a variety of skin tones, notably including 5 subjects with dark skin tones. We observe that the number of skin lesions is relatively small for those subjects, with 26.6 ($\sigma=7.7$) lesions on average. Our method successfully pairs up all the lesions on the 5 subjects. Example results can be found in the supplement.

The proposed framework has several limitations. First, our method fails to accurately map the source and the target lesions on the template mesh within the 10-mm criterion if a large non-isometric deformation exists. Fig. 6 (a) visualizes a subject with large non-isometric deformation around the chest and the belly area. Fig. 6 (b) demonstrates the source and the target lesions imaged on the template mesh. Although the proposed vector-field-based refinement brings closer lesions in correspondence, the lesion pairs in the red box are still far from each other, with a geodesic distance of more than 50 mm. However, these lesions are correctly paired up in our lesion assignment step. Second, the proposed framework fails for some challenging poses when the template mesh is incorrectly registered to the input mesh. For example, Fig. 6 (c) shows an example where template-based registration fails. The registered template mesh flips the left and the right legs. As a result, our method cannot match lesions successfully for lesions on those incorrectly registered body regions. However, some challenging poses may not be common in clinical settings, especially considering that patients have to maintain the pose during the scan. Last, the proposed dataset exhibits limited annotations of distal extremities, such as lesions on fingers and toes. In particular, due to complex morphology, these anatomical regions are challenging for digital imaging. Consequently, the proposed framework may demonstrate reduced performance and reliability for lesions in these regions.