Joint Landmark and Structure Learning for Automatic Evaluation of Developmental Dysplasia of the Hip

To take advantage of the spatial information between anatomical structures and landmarks, we propose a multi-task learning network that combines subtasks, including detection, segmentation of multiple anatomical structures, and prediction of multiple landmarks.

As the main basis for DDH measurement, the segmentation of anatomical structures requires a network framework with excellent performance. Recently, popular segmentation networks, such as U-net [34] and FCN [35], have been used to segment the hip image. It is also well known that cropping out the area containing the target structure can eliminate the interference of other structures, and the segmentation will be greatly improved [21]. It is similar to the instance segmentation in [36, 37, 38, 39, 40]. In this study, we adopt Mask R-CNN, which can simultaneously perform detection and segmentation tasks, as the basic network.

Although the positioning of three landmarks can be roughly estimated by an end-to-end landmark detection network[41, 42, 43, 44, 45], the detection performance may be degraded because of the large appearance variabilities of landmarks and relatively low quality of US images. The purpose of the multi-task is to allow the network to additionally learn the potential associated information of different structures in the image so that the positioning of landmarks and the segmentation of anatomical structures can be improved.

Our network is divided into two stages. The first step is to extract multi-scale features of the input images via the backbone, which is ResNet-101-FPN in our work. The output feature maps are then fed into the Region Proposal Network (RPN)[46] to determine the bounding boxes of the candidate regions. In the second step, the extracted feature maps go through a layer for extracting a small feature map from each Region of Interest, called RoIAlign, and the feature map is then assigned to multiple subtasks for target detection, segmentation, and landmark detection. The specific implementation of the first two tasks and the loss function $L_{maskrcnn}$ composed of classification $L_{cls}$, boundary box regression $L_{box}$ and segmentation $L_{mask}$, are based on the work in [32]. The loss function for landmark detection is softmax cross-entropy loss defined as:

where $y_{i}$ is the $i$-th landmark label (0 or 1) and ${y}^{\prime}_{i}$ is the prediction probability of the $i$-th label normalized by softmax.

As for the challenge that incomplete anatomical structures may lead to fragmental segmentations in the US image of the hip joint, as shown in Fig. 2(a) and Fig. 2(b), we propose to introduce the prior knowledge of anatomical structure to alleviate it. Since the shape of the same anatomical structure usually has a certain similarity, the shape information can be input into the network as the priors so that the network can still output the segmentation results that match the original shape in the case of partially missing anatomical structures.

Inspired by the curve similarity presented in [33], we propose a novel Shape Similarity (SS) loss to regularize the network and use the reference shape of the anatomical structure to improve the segmentation, see Fig. 4 for the pipeline. The binary prediction map is the output by the mask branch at first. Since the segmentation boundary is usually a closed curve, we divide it into two curves, such as the upper and the lower curves. The boundary of structure $j$ can then be approximately fitted with two cubic polynomials

where $a$ to $d$ are the coefficients of the equation. We use the least-squares method to find the coefficients of this equation and the solution is in the closed-form of

where $X$ and $Y$ are the abscissas and ordinate matrix of points on the curve, $\xi$ is the regularization weight, and $I$ is the identity matrix. The coefficients of (2) other than the constant term are extracted as the coefficient vector $F=\left[a,b,c\right]$ .

For a boundary of an anatomical structure, we obtain the coefficient vector $F_{1}$ from the mask and the coefficient vector $F_{2}$ from the ground truth by (2) and then calculate the cosine similarity of these two vectors by

where $\left\langle\cdot\right\rangle$ means the dot product operation and $\left|\cdot\right|$ means the length of the vector. If the two vectors tend to be similar, the value of $cs_{i}$ will be close to 1, otherwise, it is close to 0. Therefore, we define our SS loss on four structures of the hip joint as

SS loss encodes the shape prior by emphasizing the edge-wise consistency between the predicted mask and ground truth. During network training, $L_{SS}$ penalizes the large segmentation discrepancy which is often caused by the blurry edges or incomplete structures in hip joint US images. Our network is hence able to recover boundary predictions and alleviates the failure cases.

In the hip US image, three landmarks are essential for measuring angles, among which the bony rim is the most difficult to locate. The bony rim is located at the critical line where the low rim of the flat ilium transitions from hyperechoic to hypoechoic. However, in clinical practice, sometimes the flat ilium and lower limb are connected, resulting in the absence of a hypoechoic area, as shown in Fig. 2(c). In this case, it is important for the network to learn the positional relationship between the bony rim and the flat ilium, so that the bony rim could fall on the edge of the flat ilium segmentation as much as possible. We introduce the landmark-structure consistent prior to the network which enables the mask branch and the landmark branch to supervise each other and achieve better consistency.

The bony rim can be obtained by two methods from our network: one is inferred from the segmentation mask by the definition that the bony rim is located on the tail of the lateral edge of the flat ilium; the other is predicted by the landmark branch. The results of the two methods should be the same point. In other words, the distance between these two points is 0. We modify this distance as a loss function, named BR loss, to constrain both branches by extracting bony rim from the mask branches and the landmark branch respectively.

In the process of calculating BR loss, the coordinates of two bony rim landmarks obtained from the structure segmentation and the landmark detection are denoted as $m_{i}$ and $k_{i}$, respectively. Given the number of samples, we calculate the Euclidean distance between two points. The BR loss is defined as the $L_{2}$-norm given by

Our method automatically measures the diagnostic indicators of DDH in the prediction phase. The segmentation branches output four masks: flat ilium $S_{1}$, lower limb $S_{2}$, labrum $S_{3}$, and CO junction $S_{4}$; the landmark branch outputs primary positions of three landmarks: the bony rim $P_{k1}$, the lower limb point $P_{k2}$ and the labrum point $P_{k3}$. Taking into account the close correlation between the landmark and anatomical structure (see Fig. 1(a)), landmark positions predicted by the two strategies are averaged for better accuracy and robustness. The final positions of three landmarks are calculated by the formula

where $P_{si}$ is the landmark inferred from segmented structures.

With the three landmarks in hand, we can automatically measure the angles: alpha and beta (see Fig. 1(b)). The base line $L_{B}$ is the tangent of the flat ilium; the bony roof line $L_{1}$ is the tangent line to the outer contour of flat ilium at the lower limb point; the cartilage roof line $L_{2}$ is the line that connects the bony rim to the center of the labrum. The alpha angle is calculated between $L_{B}$ and $L_{1}$, and the beta is calculated between $L_{B}$ and $L_{2}$.

The final loss function of our method is defined as

where $\lambda$ is regularization weight. Fig. 3 shows the overall network architecture as well as the losses for each subtask.

Table 2 shows the mean and standard deviation (std) of angle errors between the predicted and the manual annotations. Thus, our method significantly performs better than manual plots by doctors and various other methods (Unet, PSPnet and DeeplabV3+). It reveals the efficacy of our multi-task learning framework for sharing feature information among different tasks. Besides, the average error is consistently reduced as more proposed components being incorporated into the Mask R-CNN. This verifies that our proposed techniques can well leverage the shape priors and landmark knowledge, which can help improve the system performance. After the addition of both losses and the operation of averaging landmarks obtained from the landmark and the mask branches, our method outperforms the Mask R-CNN significantly in terms of mean accuracy and stability. Especially, it achieves an improvement of 0.521 and 1.556 degrees in mean and standard deviation for the alpha angle.

The cumulative distributions of predicted angle errors are shown in Fig. 5. It can be seen that our method is the best in terms of mean accuracy, with 93.456% and 84.615% successful angle estimates for alpha and beta angles. Here we suppose that angle estimate with an error of fewer than 5 degrees are considered successful since it is close to a professional doctor’s performance. We also report that the proposed method achieves 0.407% and 1.215% poor estimates for alpha and beta angles (we consider that an angle estimate with an error of more than 10 degrees is poor), whereas the Mask R-CNN has poor rates of 3.239% and 2.444% for alpha and beta angles. In short, it can be concluded that the proposed method significantly improves accuracy and robustness.

Scatter plots of the predicted and true values of the alpha and beta angles are shown in Fig. 6. The Pearson correlation coefficient of the alpha and beta angles are 0.88 and 0.89, indicating that there is a strong correlation between the predicted and true values of the two angles. In addition, the two-sample t-test is used to compare the difference between the two values. The p-values of the alpha and beta angle are 0.28 and 0.11, respectively, with both of them larger than 0.05. Therefore, there is no significant difference between the predicted and the true value of the two angles.

In clinical practice, doctors focus more on the alpha angle because it determines the type of the hip joint, while the beta angle only determines the subclass. To prove the clinical effectiveness of our method, we evaluate the classification performance of automatic measurement on Graf’s categories. The hip joint images are classified into two categories based on the predicted alpha angle, considering that there are only two types of hip joints (type I and type II) in our dataset. The classification errors are summarized in Table 3. In terms of clinical decision support, great importance is given to the cutoff of $60^{\circ}$, distinguishing normal (type I) from dysplastic anatomy (type II) [13]. Results in Table 3 indicate that our method can reduce the error rate from 10.121% to 5.668%. We observe that the misclassifications are mainly concentrated in the range of $58^{\circ}<\alpha<62^{\circ}$. In these cases, an angle estimate error can lead to misclassification, and how to deal with it remains an open issue. Fig. 7 shows the angle measurements by our method and the labels. Fig. 7 (a)-(d) show the precise automatic measurement results. There is only a minor difference between our method and the manual plot for landmarks and auxiliary measurement lines. Fig. 7 (e) and (f) are the cases with errors of more than 5 degrees. The image with alpha angle error larger than 5° is shown in Fig. 7 (e). It is caused by the offset of the bony rim to the inner side of the ilium. In Fig. 7 (f), one case with the beta angle error larger than 5 is shown. It is mainly caused by the positioning deviation of the labrum point. Although the angle errors are more than 5 degrees in these two cases, it is noted that they do not influence the visual judgment by experts, and thus their Graf classification will remain the same.

In this study, we make qualitative and quantitative comparisons on the segmentation of four key anatomical structures. Table 4 and Table 5 quantitatively evaluate the effects of different network structures on the segmentation in terms of overall segmentation and edge accuracy, respectively.

As shown in Table 4 and Table 5, our method achieves comparable performance when compared with the SOTA segmentation networks. It can be observed in Table 4 that DeeplabV3+ and our method outperform the Unet and PSPnet, with each of them has its strengths. In specific, DeeplabV3+ performs best in the segmentation of flat ilium and CO-Junction, while our method segment the lower limb and labrum with the best performance. Besides, compared with the original Mask R-CNN, adding SS Loss can improve the segmentation results, especially for small objects such as the lower limb. Moreover, with the SS Loss and the landmark branch being incorporated together, the segmentation results are further improved in both overall segmentation and edge accuracy evaluation. It indicates that jointly learning anatomical structures and landmarks can refine the predictions of the Mask R-CNN. Specifically, compared with the plain Mask R-CNN, our method increases the DSC of the four structures by 3.184% on average and decreases the HD by an average of 12.706%. These results demonstrate that the proposed multi-task learning network significantly improves the overall performance of segmentation and accuracy of borders.

Fig. 8 shows three examples of visual comparisons on the segmentation of the labrum and lower limb, respectively. From Fig. 8 (a) and Fig. 8 (b), we can see that the plain Mask R-CNN predicts incomplete segmentation results when the target is blurred due to the poor image quality and imperfect standard plane. In contrast, our method can well circumvent this problem by introducing structure priors, so that the model can learn the relationships among anatomical structures successfully. These findings verify the efficacy of our method in segmenting key anatomical structures. In clinical diagnosis, sometimes all bony parts of the acetabular roof are hyperechoic (see Fig. 2 (c)). The flat ilium is connected with the lower ilium, which may cause over-segmentation or under-segmentation, and subsequently, the bony rim is difficult to be distinguished. Thanks to the landmark detection branch and the bony rim loss, our method can learn the structural information of different parts and the two structures can be well separated, as shown in Fig. 8 (c). In summary, the above results demonstrate that the proposed SS loss and BR loss are helpful to learn relationships among anatomical structures, predicting better segmentation results and bony rim locations.

Table 6 shows the mean and standard deviation distance of the predicted positions of three landmarks: the bony rim, the lower limb point, and the midpoint of the labrum. For the network without the landmark branch, the landmarks are determined only by the predicted mask, according to the location relationship among landmarks and anatomical structures as shown in Fig. 1(a). As shown in Table 6, our method achieved better performance than Unet, PSPnet, and DeeplabV3+ in localizing Bony Rim and Lower Limb Point. It is noted that although DeeplabV3+ performs best in detecting the landmark of the midpoint of the labrum (4.480), our method obtains a very closed result (4.563). Furthermore, compared with the original Mask R-CNN, adding SS Loss and the landmark branch can reduce the detection errors. It can be observed in ”-MASK+Landmark” and ”Our Method” rows that our method performs better than the results with landmark branch only. Our method leads the baseline by an average of 0.722 pixels in terms of mean distance, which indicates the superiority of our proposed multi-task learning system.

For the sake of consistency in predicting the Bony Rim position, we propose the BR loss. The BR loss is added to the network with the landmark branch but no SS loss. The quantitative improvement is recorded in Table 7. We evaluate it from the perspective of the segmentation of flat ilium, the localization error of bony rim, and the measurement error of beta angle since the BR loss only works on the bony rim and its related tasks. Experimental results show that BR loss improves the performance of multiple tasks. Especially in the measurement of beta angle, the error is reduced by 0.152°, and the relative error is reduced by 4.975%. BR loss has a great contribution to the optimization of network performance.

There are four parameters to control the weight of losses in 8. Our hypothesis is that, the shape similarity based SS loss is general and critical for the accurate segmentation of all the four anatomical structures in the DDH US image, It intuitively plays a more important role than the consistency based losses $L_{landmark}$ and $L_{BR}$. Therefore, we assigned 1.0 for $L_{SS}$, while assigned 0.5 for both $L_{landmark}$ and $L_{BR}$.

We further conducted experiments to verify our hypothesis and presented the influences of the parameters. As shown in Table 8, we tried several typical combinations of the coefficients. When comparing the four parameters, larger $\lambda$4 often leads to better alpha angle results. While larger $\lambda$2 and $\lambda$3 often contribute to slightly better beta angle results. Because the alpha angle is the main indicator for DDH evaluation, we finally take the coefficient setting in our paper as a compromise for alpha and beta angles.

We test the three types of images separately to check the stability and robustness of our method. According to the differences in image quality, the test data (247 images) are divided into three categories: good (123 images), medium (70 images), and poor (54 images), as exemplars shown in Fig.9. We tested our method on those three categories and showed the results in Table 9. As we can see, our proposed method presents a slight difference for images with different qualities. Good image quality contributes to better accuracy than poor images. When compared with the Mask R-CNN, our method shows superiority in keep high accuracy. This proves the advantages of our novel frameworks.

In addition, the dataset used in our experiment is from two types of US machines. To investigate whether our method performs significantly differently on two machines, we compare the experimental results in Table 10. It can be seen in Fig. 10 that there is a visual difference between the two types of machine images. The image of Machine 2 is clearer than that of Machine 1. However, the p-values of the alpha angle and the beta angle by the two-sample t-test are 0.118 and 0.152, which shows that there is no significant difference between the two machines. The deviation caused by different image qualities can be eliminated in our method, especially in the measurement of alpha and beta angles. To a certain extent, it indicates that our method is not affected by image quality in angle measurement, and has good robustness.

Moreover, the dataset of hip joints is divided into two types, type I and type II. The average test data for each fold contains 215 type I cases and 32 type II cases. Taking into account the imbalance in the number of images of the two types, we train the model by using two types of data together but evaluate the test results separately. As shown in Table 10, the p-values of the alpha angle and the beta angle by the two-sample t-test are 0.802 and 0.523, which shows that there is no significant difference between errors of alpha and beta angles of the two types of images. Please note that, due to the lack of sufficient positive cases (type III and type IV), our method is only validated on type I and II images. In the future, we will collect more positive cases to verify the reliability of our proposed method.