DeepSSM: A Blueprint for Image-to-Shape Deep Learning Models

Deep learning models are data-hungry, and data is usually scarce in medical imaging applications; therefore, data-augmentation methods are typically used to significantly increase the training sample size. For statistical shape modeling, the given population size is typically in the range of 100-200 samples; this is insufficient to train a CNN as the network will overfit. To handle this data scarcity issue, DeepSSM frameworks include a model-based data augmentation approach to generate augmented training samples that follow the statistics of the given population.

We start with a set of $N$ original images $\mathcal{I}=\{I_{1},...,I_{N}\}$ and obtain a point distribution model (PDM), also called a correspondence model, on these shapes. We use ShapeWorks [13] to generate the PDMs used in the experiments presented in this work. In a recent benchmarking study [23, 22], ShapeWorks was shown to discover clinically relevant shape differences. However, any set of correspondences (i.e., a PDM or even manual landmarks) can be utilized for the proposed model. This PDM will give us a set of surface correspondence points for the anatomy associated with each image. Let’s denote this set of original correspondence points $\mathcal{C}=\{C_{1},...,C_{N}\}$.

The first step in data augmentation is to define a generative model over the existing PDM space (also known as shape space). This allows us to sample plausible shapes from the distribution. In previous iterations of DeepSSM, we performed PCA on the shape space and modeled it parametrically using a Gaussian model [6] or a Gaussian mixture model (GMM) [8]. However, when the underlying distribution of the original shape space is nonlinear, the Gaussian model will ignore vital subtleties in the distribution. Also, the GMM model with too few clusters is prone to overestimating the variance and sampling implausible shapes.

To counteract these problems, we propose a non-parametric kernel density estimate (KDE) for modeling the PCA space of the correspondences. We found this to be a robust augmentation strategy that generates plausible shapes while still capturing all the subtleties of the original shape space. Augmentation is performed in PCA space and not directly on the shape space as it preserves the smoothness of the anatomy making it a more plausible shape without noise. Additionally, the augmentation is performed by utilizing all available PCA modes, i.e. N-1, which along with proving smoothing will also ensure that we do not destroy any inherent characteristic that shape distribution may exhibit. Let $\mathcal{Z}=\{Z_{1},...,Z_{N}\}$ represent the PCA scores for the correspondences, then we define $p_{\sigma}(Z)$ as:

where $L$ is the dimension of the PCA subspace. Here the kernel bandwidth, $\sigma$, is computed as the average nearest neighbor distance in the PCA subspace. In this case, when a new PCA score $Z_{s}\sim p_{\sigma}(Z)$ is sampled from the kernel of the $n^{th}$ training sample ($K_{n}^{\sigma}(Z)$), we can obtain the sampled correspondence, $C_{s}$, via PCA reconstruction. Once we have the sample correspondence representation, we need the associated image $I_{s}$ to complete the data augmentation process. We find the original correspondence, $C_{n}$, from whose kernel $C_{s}$ is sampled from and use the corresponding raw image $I_{n}$. Using the Thin plate spline (TPS) warp between $C_{n}\rightarrow C_{s}$ (serving as accurate landmarks on the anatomy surface), we warp the image $I_{n}\rightarrow I_{s}$. Using this technique, we can generate thousands of training data pairs of images and their corresponding PCA score vectors/correspondence points, while preserving the shape population statistics. The entire pipeline is depicted in Figure 2.

Using the above data augmentation techniques, we form the training data composed of (i) raw 3D images, (ii) correspondences associated with each scan, and (iii) the PCA scores associated with the correspondences. In the following section, we present the DeepSSM architecture and associated loss functions. We also introduce a novel alternate architecture that utilizes a non-linear representation of correspondences in place of the PCA scores.

We showcased two different architectures of the DeepSSM model; however, both these are trained by utilizing the $\mathbb{L}$-2 norm as training loss. Some anatomy datasets need a different loss. Specifically, when the given anatomy exhibits very subtle shape variations in localized regions, i.e., the variance of the particles across the data is only significant for a small subset of correspondences. Amidst this variance imbalance across the particles, if we utilize the $\mathbb{L}$-2 correspondence loss as given in Eq. 2, the network will degenerate to producing the mean shape as the output, independent of the input image. To handle such datasets, we utilize the focal loss [37], which was initially proposed for handling class imbalance in object detection tasks but have since been extended for regression applications [38]. For a pair of input ($x$) and prediction ($\hat{x}$) the focal loss is given as:

Here, $a$ and $c$ are hyperparameters where $a$ is a scaling factor and $c$ decides the threshold. If the particle difference is below $c$, the contribution of the particle to the overall loss is reduced. The scaling parameter $a$ is used to dampen or accentuate the loss based on the magnitude of $c$. This focal loss allows for better generalization on datasets with subtle variations and prevents the model from degenerating to learning the mean shape. To incorporate focal loss with Base-DeepSSM we modify the loss in Eq. 2 to the following:

For the TL-DeepSSM the focal loss is applied both for autoencoder training as well as training the latent space, and hence the loss function is modified as follows:

The hyperparamenter $c$ is estimated using a heuristic. We compute the absolute difference per point for each sample in the training data with respect to the mean shape, i.e., $S(i,j)=||C^{j}_{i}-\mu^{j}||$, for $i^{th}$ sample and $j^{th}$ correspondence. We choose $c$ as the score corresponding to the 90 percentile value. We set the scaling hyperparameter $a$ to 10 in all our experiments for particle based focal loss (eq. 8 and eq. 10), as in most cases $c$ lies between 1-2. The TL-DeepSSM with the focal loss, has focal loss applied at two different places and each of them will have its own a and c parameters. First, for the correspondences autoencoder, whose a and c parameters are discovered similarly to that of the base-DeepSSM, as the loss is applied on the correspondences. Second, for the latent space, we use the trained autoencoder and the latent space for the training data to get an estimate of c as described above, here the estimates of c are in orders of 10 for all dataset and hence we set our a as 1 for all cases.

Craniosynostosis is a morphological defect that affects infants, where one of the sutures of the skull is fused prematurely, and the subsequent brain growth leads to abnormal head shapes. In severe cases, craniosynostosis can lead to increased intracranial pressure, causing symptoms such as headaches, drowsiness, vision changes, and more. It can also cause several neurological problems. Metopic craniosynostosis is craniosynostosis where the metopic suture (also called frontal suture) is fused prematurely. It leads to an unnaturally triangular head shape, a morphological symptom known as trigonocephaly

[32]. Compensatory growth in metopic craniosynostosis also causes expansion at the back of the skull. Metopic craniosynostosis is difficult to identify with CT scans as the suture fuses typically at an early age, making it difficult to ascertain whether or not the suture fused prematurely. The current diagnosis protocol includes subjective observation of the head shape. The diagnosis affects the subsequent treatment protocol, namely to perform corrective craniofacial surgery or not. This treatment is risky by nature and requires an objective measure of the severity of the condition. This objective severity measure for craniosynostosis will guide the surgeons to make a better-informed diagnosis. In this section, we will utilize DeepSSM to construct a shape model on cranial data. The end goal is to quantify the deviation of a given shape diagnosed with metopic craniosynostosis from the population of normal cranial shapes.

Data Description and Processing: Cranial CT scans of infants between 5-15 months of age were acquired at UPMC Children’s Hospital of Pittsburgh. A subset of 28 scans were diagnosed with metopic craniosynostosis by a board-certified craniofacial plastic surgeon utilizing the CT images as well as a physical examination. The remaining 92 scans are CT scans of trauma subjects acquired as a by-product of standard clinical practice; however, these scans do not exhibit any deformities and can be treated as a set of control patients. All the data is IRB approved and acquired by utilizing standard de-identification and HIPAA protocols. Along with the CT scans, we have corresponding segmentations of the cranium. We separate this dataset into 105 training and 15 testing images. We process images and segmentations for the training data and utilize the ShapeWorks [13] package to generate the initial PDM on the data. To satisfy the GPU memory requirements, all of the images are downsampled isotropically by a factor of 4. This results in volume size of $69\times 51\times 66$ and resulting voxel spacing of 4mm in each direction. The PDM consists of 1024 3D points and uses 20 PCA components (capturing 99% of variability). We perform data augmentation and generate 5000 additional data points. The total 5000 augmented plus 105 original training images are divided into training and validation set with a 90%-10% split.

Training Specifics: We train the Base-DeepSSM with 20 PCA components (explaining 99% of data variability), using loss on correspondences (Eq. 2) for 100 epochs with early stopping with respect to validation loss. Similarly, we also train the Base-DeepSSM with focal loss as given by Eq. 8 for 100 epochs. Finally, we train a TL-DeepSSM with a bottleneck of 64 dimensions with and without the focal loss. We train the autoencoder for 5000 epochs, the T-flank for 100 epochs, and the joint model for 50 epochs. The values of the hyperparameter $c$ for the focal loss was discovered by the heuristic described in Section 3.3, and is 1.09 for particles (in base and TL DeepSSM) and 10.9 for TL latent space.

Evaluation and Analysis: The average RMSE and the per-point RMSE are given in Figure 4, and the average surface-to-surface distance for each test sample is provided via box-plots in Figure 6. These metrics, namely RMSE, per-point RMSE, and surface-to-surface distance are computed as described in the beginning of the Section 4. Qualitative results are showcased in Figure 5, the heatmap is the surface-to-surface distance between the original mesh and the mesh reconstructed from the Base-DeepSSM (without focal loss) predicted particles. From Figure 4 we can see that the prediction error is centered around the eye sockets or frontal ridge, however the average RMSE is well within the sub-voxel margin, as the input images have spacing of 4mm. We can see that both normal and metopic test scans capture the shape well and with a sub-voxel accuracy in most areas. The left-most metopic scan in Figure 5 is the result with the largest error, and the maximum error is localized in the region near the orbital rim. The oribital rim is a sharp and thin structure, and the original PDM on which DeepSSM is trained does not provide very accurate correspondences to the region. These inaccuracies persists in DeepSSM training and the trained model also exhibits this error in the localized area. This is a data problem, and the neural network does not learn the structures that are not well expressed by data. The improvement provided by focal loss is minimal both quantitatively as well as qualitatively, however, we can see from Figure 4 that the distribution of RMSE error is more spread out and less localized in specific regions. Because focal loss does not provide significant improvement on this dataset, we the Base-DeepSSM and TL-DeepSSM trained without focal loss for the downstream application and analysis.

Downstream Task – Severity Quantification: Our end goal is to utilize DeepSSM to provide clinicians with an end-to-end tool where they can input an image and acquire a severity score.

The severity score is designed to measure the abnormality of a subject with respect to the normal population (samples with a negative diagnosis for metopic craniosynostosis). We utilize an unsupervised learning method that models the variations in the normal population and learns the deviation between a test image and this population [48]. The method employs Mahalanobis distance/Z-score, which uses the variance in the normal population to normalize the multidimensional difference of the target subject. It can be regarded as a Z-score for multiple dimensions and effectively scales up rare characteristics in the shape statistics of the normal population.

The sample number is usually limited and much lower than the shape dimension; we account for the variability outside the sample subspace using Probabilistic Principal Component Analysis (PPCA). PPCA estimates the full rank covariance $S$ as given by Equations (13) and (14). Here, $D$ is the shape dimension, $L$ is the subspace dimension, $\bar{C}$ is the empirical mean, $U$ are the columns of eigenvectors and $\Lambda$ is the diagonal matrix of eigenvalues of $\frac{1}{N}\sum_{i}(C_{i}-\bar{C})(C_{i}-\bar{C})^{T}$. We can use the eigendecomposition $U_{S}\Lambda_{S}U_{S}^{T}=S$ to whiten a shape deviation, project back to the shape space and get the point-wise Mahalanobis Distance $\tilde{C_{i}}$ which explains the contributions across the shape domain to the final severity score (as shown in Equation (15)).

Evaluation result for the severity scores is shown in Table 2. For each shape model, we fitted PPCA (with 95% variance subspace) on 72 control scans and calculated the severity scores for the remaining 48 scans (these include 28 metopic and 20 normal scans). The latter group is provided with aggregated expert ratings of the degree of metopic craniosynostosis, collected through an online portal with 36 craniofacial experts. We displayed the 3d segmentation and asked the physicians to estimate the deformity on a 5 point Likert scale. We modeled the expert ratings by Latent trait theory [52, 40] which parameterized potential raters’ bias and inconsistency and obtained the aggregated latent severity using Maximum Likelihood Estimation.

We use both Pearson and Spearman correlation coefficients to measure the similarity between the predicted severity scores and the discovered latent trait using the expert ratings. The results presented in Table 2 suggest that both base- and TL-DeepSSM have severity scores that are highly correlated with the expert consensus. In addition, we also calculate the Area Under Curve (AUC) using the binary diagnosis to evaluate the efficacy of the predicted severity scores for classification. The AUC also suggest that DeepSSM based severity is a good indicator to identify scans with metopic craniosynostosis. Both AUC and correlation scores for the DeepSSM variants are comparable to results using ShapeWorks [13], hence verifying that DeepSSM can perform comparably with state-of-the-art PDM at this downstream task.

We displayed the point-wise Mahalanobis distance across the shape domain as is shown in Figure 7. The point-wise Mahalanobis distance is obtained by averaging the whitened deviations of all metopic scans. It is then projected onto the surface normals and visualized using the heap map on the top of the mean control shape in both top and side views. The heat maps show that the center of the frontal bone protrudes while the sides shrink; this morphological deviation showcases trigonocephaly, which is a clinically expected observation. Therefore, DeepSSM shape model captures the characteristic deformation pattern for metopic craniosynostosis.

Cam-type femoroacetabular impingement (cam-FAI) is characterized by an abnormal bone growth of the femoral head. The abnormal bone growth causes excessive friction between the hip socket(part of the large pelvis bone) and the femoral head [26]. This friction can cause eventual joint damage leading to pain and restricted movement, which is also a leading cause of hip osteoarthritis. Modeling the shape from a joint population of femurs diagnosed with and without CAM-FAI can characterize the local femoral head deformity (cam-lesion) with respect to the femoral shape statistics of healthy subjects. Such a characterization can provide surgeons an anatomical map of the deformity and therefore driving the surgical intervention. It can also help understand the pathology more intuitively, and correct for local abnormal morphology and preserve normal bone function. We intend to utilize DeepSSM models to achieve this characterization and compare it to the state-of-the-art PDM models, showcasing the efficacy of utilizing DeepSSM for such tasks while mitigating the need for segmentation and added pre-processing.

Data Description and Processing: For training the DeepSSM models, we have 49 CT images of the femur bone. Out of these 49 CT, 42 subjects are healthy and not diagnosed with any morphological abnormality in the femur bone, and we call these control scans. The remaining 7 scans are diagnosed with CAM-FAI. For the initial shape model, we use ShapeWorks [13] and generate 1024 correspondences on these shapes. Next, we apply data augmentation on these 49 scans producing 5000 additional data points. This combined 5049 scans are split into 90-10% split for training and validation. We utilize 20 PCA modes (capturing  99% variability) for training the DeepSSM. In addition, to the training and validation data, we also have 7 controls and 2 CAM-FAI scans reserved for testing the DeepSSM model. Each image is downsampled by a factor of 4 to fit the GPU memory requirements easily. This downsampling results in the 3D volumes with dimensions of $65\times 46\times 58$ voxels, with isotropic voxel spacing of 2mm.

Training Specifics: We train the femur model for the Base-DeepSSM and TL-DeepSSM with and without the focal loss. We train the baseline DeepSSM with 20 PCA modes without focal loss via Eq. 2 and with focal loss by Eq. 8. We train each of these models for 50 epochs with early stopping based on the validation loss. Finally, we train a TL-DeepSSM with and without the focal loss, where we train the autoencoder for 10000 epochs followed by training the T-flank for 50 epochs and joint training for 25 epochs. The values of the hyperparameter c for the focal loss was discovered by the heuristic described in Section 3.3, and is 1.32 for particles (in base and TL DeepSSM) and 6.3 for TL latent space.

Results and Analysis: The average and per-point RMSE are given in 4. Surface-to-surface distance is computed between the mesh reconstructed from the DeepSSM predicted correspondences and the mesh coming from ground-truth segmentation of the femur bone, and is given in Figure 6. We have two key observations, first, we notice that focal loss improves the RMSE scores marginally, however this improvement is crucial in capturing the subtle variability in femur anatomy. We also see from Figure 4 that the models with focal loss have a more diffused loss and is less persistent in greater trochanter area. Second, we notice that the TL variant outperform the Base-DeepSSM but not by a huge margin indicating that the PCA scores as a shape descriptor captures most of the variability. In this dataset we notice that the models trained without the focal loss degenerate to producing a similar predictions in areas which have large variations in shape and as they are very limited in number the loss is still low in value. However, this is meaningless as the areas with larger variance are the most important in characterizing the anatomy and we want DeepSSM to capture them. Qualitative results using the Base-DeepSSM with focal loss are shown in Figure 8, it showcases the surface-to-surface distance on testing images (both controls and CAM-FAI).

We notice that the error on the lesser trochanter and the CAM lesion is relatively high in some cases; this is due to two reasons. First, the plane at which the femurs are chopped off is arbitrary, and in some cases (both in training and testing images), the entire lesser trochanter is not captured. This inconsistency in cutting planes introduces a high and uncertain shape variation in that region that is hard for the network to learn. This is a data issue, and can be fixed by standardizing the scans fed as an input to the model. Second, the error in CAM lesion occurs due to data imbalance, i.e., there are many more controls than the CAM-FAI diagnosed femurs. This results in extreme cases of CAM (such as the second CAM case in Figure 8) not completely being captured by the DeepSSM model. DeepSSM model can only encode the variations that are present in training data, and if the dataset were larger and well balanced between two classes, the performance of the model would increase.

Downstream Analysis – Group Differences: As mentioned earlier, it is of interest to the clinical community to capture the statistical morphological difference between the CAM-FAI shape and the representative control femur bone shape. This characterization is better represented by models trained with focal loss and hence, for this downstream analysis we use base- and TL-DeepSSM trained with the focal loss. We construct two groups, one for controls and the other for pathology (cam-FAI), and compute the difference between their means ($\mu_{normal}$ and $\mu_{cam}$) and showcase this difference on a mesh. This is termed as a group difference [26]. We compute this group difference utilizing the DeepSSM predicted particles, as well as via ShapeWorks PDM model. The entire data is used (both testing and training) for these group differences, and the figures are shown in Figure 9. Each group difference showcases the difference going from pathological mean shape to control the mean shape and is overlayed on top of the mean pathological scan. We can see that the group differences for both the state-of-the-art PDM model and DeepSSM are very similar. Therefore showcasing that DeepSSM can be used to get correspondences without undergoing heavy pre-processing and segmentation steps, and these correspondences can be used to characterize the CAM deformity.

TL-DeepSSM predicts a latent space that captures non-linear variations, but at the same time, can also capture the morphological differences between normal and CAM-FAI femoral shape in its latent space. To show this we move along the line connecting mean normal ($\mu_{norm}$) and mean pathological ($\mu_{cam}$) in the latent space, and then at each sample we utilize the decoder($f_{\phi}$) to get the corresponding correspondences, i.e $C_{\lambda}=f_{\phi}((1-\lambda)\mu_{norm}+\lambda\mu_{cam})$. Figure 10 shows the difference between each of these samples and mean normal shape, i.e., $C_{0}-C_{\lambda}$, overlaid on $C_{\lambda}$. We observe the clinically expected outcome, as femur anatomy smoothly exhibiting inward motion around the CAM lesion as $\lambda$ increases.

Atrial Fibrillation (AF) is irregular heartbeat (arrhythmia) that can lead to several problems such as stroke, blood clots, and even cardiac arrest. Catheter ablation is a therapeutic procedure that is commonly utilized to treat AF; however, many patients can exhibit a recurrence of AF after ablation. This AF recurrence is demonstrated to be related to the shape of the left atrium [9, 8]. We utilize DeepSSM models and their predicted shape descriptors to validate this hypothesis and estimate AF recurrence based on the shape of the left atrium.

Data Description and Processing: For the left atrium (LA) experiments, we utilize 206 late gadolinium enhancement (LGE) MRI images from patients that have been diagnosed with atrial fibrillation (AF). These scans are acquired after the first ablation. Of the 206 scans, 122 are used in training, and additional 84 scans are kept as testing set curated carefully, ensuring even distribution of intensity and shape variation in the training versus testing data. A PDM of 1024 correspondences is generated using ShapeWorks [13] for the training set, then using this as a base shape model, correspondences are also found for the test set for evaluation. We extract 19 modes from the PDM via PCA, which captures  95% of the shape variability. We then run data-augmentation on the training set to create 5000 additional samples, 80% of which are used in training and 20% in validation. For training purposes, the MRIs are downsampled from $235\times 138\times 175$ (0.625mm isotropic voxel spacing) with a factor of 2, resulting in images of size $118\times 69\times 88$ (1.25mm voxel spacing).

Training Specifics: We train the Base-DeepSSM with and without the focal loss for 100 epochs with 19 PCA scores that capture ( 95%) data variability. We use early stopping to evaluate performance on the best epoch. For the TL-DeepSSM (with and without the focal loss), we train the autoencoder for 10000 epochs, followed by 100 epochs to train the latent space and 25 epochs for joint training. The autoencoder bottleneck of 64 was chosen for TL-DeepSSM. The hyperparameter c used in focal loss is 2.54 for the particle loss (base and TL-DeepSSM), and is 13.76 for TL-DeepSSM latent space loss.

Results and Analysis: This is a challenging dataset, firstly in terms of the intensity variations present in the LGE images (Figure 11), with some images being significantly degraded and noisy. Secondly, in terms of shape variation, the left atrium anatomy exhibits natural clustering based on the shape of the atrium and based on the different numbers of pulmonary veins and their orientation [28]. Additionally, determining where the pulmonary veins should end in a segmentation is arbitrary across samples and across annotators. Due to this highly non-linear shape distribution, the TL-DeepSSM outperforms the other variants considerably. The RMSE results are shown ion Figure 4, and the surface-to-surface results are shown in Figure 6. Similar to the craniosynostosis dataset, here the results with and without the focal loss does not make much difference quantitatively as well as qualitatively. The TL-network provides considerable improvement over the average RMSE as seen in Figure 4, with majority of the error still being concentrated around the pulmonary veins similar to that of Base-DeepSSM. Qualitative results are showcased in Figure 11, which depicts surface-to-surface distance between the ground truth mesh and the mesh reconstructed from the TL-DeepSSM predicted particles. We notice that the surface-to-surface distance shown in these scans is as large as 11 mm, which ideally is a significant error. However, this error is primarily localized in the pulmonary vein region that has inconsistent initial segmentation. The left atrium body, whose shape is indicative of AF recurrence, exhibits near sub-voxel accuracy.

Downstream Task - AF Recurrence Prediction: As the focal loss does not make much difference, this downstream analysis is performed using the base- and TL-DeepSSM trained without the focal loss. Left atrium shape is indicative of AF recurrence [26]. In our dataset, we are also provided with binary outcome data that indicate whether or not the patient had AF recurrence post-ablation. Using this binary information, we aim to train a supervised predictor to obtain AF recurrence probability from the shape descriptor. State-of-the-art PDM models provide us with correspondences, and an ideal choice of shape descriptor is to use PCA scores. However, not all the modes of PCA are useful in exhibiting anatomical changes that discriminate left atrium with and without AF recurrence. Hence, we perform feature trimming via a heuristic; we compute the absolute difference between the shape descriptor of the normal and AF recurrence left atrium and select ten dimensions with the highest magnitude. Using this trimmed PCA scores shape descriptor, we use a multi-layer perceptron for classification with 122 training and 84 testing data. We perform this for PCA scores from ShapeWorks [13] and Base-DeepSSM. Furthermore, we also perform the same analysis utilizing the latent space of TL-DeepSSM as the shape descriptor. Both these models used are the ones without the focal loss. The results are showcased in Table 3. We see that PDM and Base-DeepSSM perform similarly, indicating that the shape model from DeepSSM can be directly used for providing a useful shape-descriptor that performs comparably with the state-of-the-art PDM. We also see that TL-DeepSSM space outperforms the PCA scores space of other models; this can be attributed to the non-linear nature of the anatomical variation that is better captured by an autoencoder in place of PCA. Hence, the TL-DeepSSM learns a richer shape descriptor for the left atrium.

Since TL-DeepSSM produces a more discriminatory shape descriptor for AF recurrence prediction, we move along the latent space of the TL-DeepSSM similar to analysis performed in Section 4.2 in Figure 10. Reiterating, we find the mean of the normal left atrium and mean of left atrium diagnosed with AF recurrence in the latent space. We move along the line (using parameter $\lambda\in[0,1]$) joining these means and reconstruct the samples back to the correspondence space ($C_{\lambda}$). We take the difference of each of these samples from the normal mean and project the variation on the normal mean. Note that here we visualize $C_{\lambda}-C_{0}$ on $C_{0}$ shown in Figure 12, which is reverse of the visualization performed in Figure 10, this is done for ease of visualization. Both ways still demonstrate the shape variation from normal to pathological shape. From Figure 12 we see that as the $\lambda$ increases, i.e., moving towards pathology, the left atrium shows outward movement, indicating increased bulging of the left atrium as a morphological change observed in AF recurrence patients. This observation clearly shows the left atrium shape is more spherical in cases diagnosed with AF recurrence, this aligns with observations in previous studies [11, 10] that showcases that sphericity of the left atrium body is associated with recurrence, and the feature can be used to predict AF recurrence after ablation. Another work [53] observes a larger asymmetry in left atrium shapes of patients with AF recurrence, this also aligns with our observations in Figure 12 that exhibits more expansion on one side than the other. This alignment with previous works and their observations strengthens the argument that TL-DeepSSM’s latent space captures the anatomical variations that differentiate the left atrium shapes with and without AF recurrence after ablation, and can be effectively used as a predictor for the same.

We have proposed two different loss functions as well as two different architectures. Throughout our experimentation, we have discovered that the utility of these variants differs based on the data/application the model is being used for. This discussion sheds light on where each variant might be helpful over others.

As mentioned above, the choice for variant depends on the data or applications or both. Let us start by looking at applications of the shape descriptor. These can be broadly divided into two parts, (i) where the shape descriptor is being used as a feature for subsequent prediction/classification/clustering model, and (ii) where the feature is examined to understand the morphology of a sample/group of samples. In (i), we can use any shape descriptor, and it is on the subsequent model to extract the shape information. In case of (ii), the interpretability of the shape descriptor plays an important role, and PCA scores provide this, and hence, base DeepSSM will work better for such applications.

Now let us look at the data, and here is where most of the subjectivity comes into play. For most datasets, TL-DeepSSM outperforms the base model (Figure 4 and Figure 6). For datasets with limited variability in shapes and no natural clustering of morphology, the TL-DeepSSM performs marginally better quantitatively. This marginal difference does not affect the downstream application, and hence, it is better to use base DeepSSM, which provides us with a more interpretable shape descriptor. Whereas with anatomy like left atrium where we have highly non-linear shape variations, we see that the quantitative performance of TL-DeepSSM outshines other variants significantly. Furthermore, TL-DeepSSM’s shape descriptor performs better at downstream tasks, showcasing that TL-DeepSSM should be used with such datasets. In some other datasets, the variation across the population is localized in some small regions and the rest are mostly constant. Such a scenario occurs with femur dataset (Section 4.2). In such cases utilizing focal loss, break the network out of the local-minima of learning data mean or few cluster means.