Shape-Aware Fine-Grained Classification of Erythroid Cells

WBC datasets. There are many available white blood cell image datasets for investigating the WBC classification and counting tasks rezatofighi2011automatic; mohamed2012efficient; labati2011all; zheng2018fast; naruenatthanaset2020red; kouzehkanan2021raabin. Here, we only describe three representative WBC datasets and highlight their difference with our BMEC; see Table 1. The LISC rezatofighi2011automatic is an early WBC dataset that includes the hematological images taken from peripheral blood of healthy subjects. The smears were stained by through the Gismo-right technique and a Sony Model No.SSCDC50AP camera was used to capture the observation of a Axioskope 40 microscope at 100X magnification. Then, 250 WBC images were extracted and labelled by one blood expert into five fine-grained categories. Similarly, BCCD mohamed2012efficient contains 349 WBC images that have been taken from peripheral blood and annotated by one expert. The smears were also stained by the Gismo-right technique and the observations of a regular light microscope at 100X were captured using a CCD color camera. Each WBC image extracted from the smear image was annotated by one expert into one of five categories. Raabin-WBC kouzehkanan2021raabin is a recent, large and public dataset of white blood cell images extracted from the normal peripheral blood samples. The peripheral blood smears were stained by the Giemsa technique and the images were taken using a Samsung Galaxy S5 camera to capture the observation of an Olympus CX18 microscope at 100X magnification. In total, 17,965 WBCs images are cropped from the smear images and labelled into five categories by two blood experts. Comparing to the white blood cells, different types of erythroid cells look more similar and the shape difference between the types is subtle. Therefore, it is more challenging for fine-grained erythroid cell classification, e.g., over 99% accuracy can be achieved on Raabin-WBC while only around 80% accuracy can be obtained on our BMEC. Due to the difficulty on distinguishing the erythroid cell types, we asked three hematologists to jointly label the BMEC images in a more rigorous manner.

Erythroid cell datasets. Compared to the WBC datasets, there are only a few open-source datasets containing the erythrocytes, while the data quality, quantity and variety are limited comparing to our BMEC. As shown in Table 1, CBC alam2019machine is a hybrid dataset which contains three types of blood cells: red blood cells, white blood cells, and platelets. The images in CBC were taken using a Nikon V1 camera to capture the observation of an ECLIPSE 50i microscope at 100X magnification, and then annotated by two experts. It focuses more on investigating the differences between the three types of blood cells rather than classifying red blood cells into fine-grained types. In addition, CBC only has 360 images which are difficult for training the deep neural networks. One most related dataset on erythroid cell classificaiton is the erythrocytesIDB gonzalez2014red which contains 629 images of individual erythroid cells extracted from 196 full filed peripheral blood images. A Kodak EasyShare V803 camera was used to capture the observation of a Leika microscope at 100X magnification. The erythrocytesIDB images are annotated by one expert into three categories based on the cell shape: circular, elongated and other shape. Comparing to BMEC, the shape differences for erythroid cells in erythrocytesIDB are more significant since they focus on the study in cell deformation caused by certain blood diseases. Hence, it’s much easier to achieve high accuracy (98% in gonzalez2014red) on erythrocytesIDB. In contrast, our BMEC provides a much larger number of images with minor shape differences, so that it can be used to train a more generalizable deep learning model for fine-grained erythroid cell classificaiton.

Along with the release of more open-source blood cell datasets, automatic cell classification and counting techniques have been proposed, including the traditional image processing and machine learning based methods, as well as the deep learning based frameworks.

Traditional methods. Generally, cell classification using traditional image processing and machine learning techniques involves image segmentation, feature extraction, classification, and counting steps. For example, Alomari et al. alomari2014automatic employed the Circular Hough Transform (CHT) in an iterative circle detection framework to segment, classify and count WBCs and RBCs. Such hough transform based shape detection methods are usually time-consuming and their detection and classification accuracy may be significantly reduced when the shape of the blood cells becomes more complex. Meanwhile, machine learning based methods aim to classify the blood cells based on some features (e.g., shape and color) extracted from the cell images. Lippeveld et al. lippeveld2020classification applied the traditional machine learning techniques such as random forest breiman2001random and gradient boosting classifier friedman2002stochastic to classify and count the stain-free WBCs images based on the size, location, texture, and signal strength feature. Tavakoli et al. tavakoli2021new proposed a new segmentation and feature extraction algorithm which computes three shape features and four novel color features for classifying WBC images. Although above traditional methods have achieved appealing results for WBC or RBC classification on certain datasets such as Raabin-WBC and LISC, they heavily rely on hand-crafted features which are hardly generalizable to more challenging tasks, e.g., fine-grained classification on our BMEC.

Deep learning based cell classification. Deep learning can automatically extract features by learning from a large amount of data to compute tasks such as classification and detection. To apply deep learning models for the cell classification, one way is to use models pre-trained on general large image datasets for feature extraction and perform particular traditional feature selection and classification methods on the smaller-scale WBC or RBC dataset. In tougaccar2020classification, features extracted from pre-trained AlexNet krizhevsky2012imagenet, GoogLeNet szegedy2015going and ResNet50 he2016deep models are filtered using the Maximal Information Coefficient and Ridge feature selection methods, and quadratic discriminant analysis was used as a classifier on a dataset of WBCs. Sahlol et al. sahlol2020efficient employed a statistically enhanced Salp Swarm Algorithmmirjalili2017salp to filter the features extracted from WBC images using the VGG simonyan2014very model pre-trained on ImageNet and performed the classification using a traditional decision tree.

The other way to apply deep learning for cell classification is using the transfer learning technique which starts from a pre-trained CNN model and performs model fine-tuning on a specific WBC or RBC dataset. For example, Alzubaidi et al. alzubaidi2020deep first trained a CNN model on a combination of three WBC datasets and then employed transfer learning to classify RBC images in erythrocytesIDB gonzalez2014red. Pasupa et al. pasupa2020convolutional exploited pre-trained ResNet he2016deep and DenseNet huang2017densely and performed fine-tuning on their RBC images. In addition, they have handled the class imbalance problem by incorporating the focal loss lin2017focal . Comparing to traditional machine learning techniques, deep learning based classification methods can achieve superior performance without manual feature design. However, it is difficult to either train a traditional classifier or perform transfer learning for the fine-grained classification task, especially when there is no sufficient finely annotated training data. We propose a novel shape-aware transfer learning framework that utilizes different types of pre-trained backbone models and fine-tunes the network on our large-scale BMEC for fine-grained erythroid cell classification.

Shape-aware cell classification. The cell’s morphological or shape characteristics such as the circularity, convexity and size have been investigated in traditional methods alomari2014automatic; lippeveld2020classification; tavakoli2021new for cell classification. For deep learning based methods, given only the original cell images, the networks can automatically learn some shape features zeiler2014visualizing which are generally more robust than the hand-crafted ones. However, the implicit learning of deep shape features requires a large amount of training data. Even when transfer learning is used, it may be still difficult for the network to learn the fine-grained shape variations. To explicitly incorporate the shape information in the network, Tavakoli et al. tavakoli2021generalizability passed two binary shape images along with the RGB images of WBCs to a customized CNN with five channels (three for RGB and two for shape masks) and showed the generalizability of the model can be improved when the shape masks are used. Similarly, we also explicitly integrate the shape information into the network. Instead of using the nucleus and ROC (Representative Of the Convex hull = convex hull - nucleus) images as in tavakoli2021generalizability, we choose convex hull as our shape mask image since we find the contour is more suitable to represent the shape of erythroid cells. Moreover, we design a novel shape attention module which computes a shape-attended image feature, to improve the learning of fine-grained shape-aware image features.

Our pipeline consists of four modules as shown in Figure 2: a cell shape extraction module, a dual-branch network, a shape attention module and a cell classification network. The shape extraction module transforms the RGB erythroid cell image into the binary shape mask. The RGB image and shape mask are separately passed to the dual-branch network to generate the raw and shape features. The shape attention module calculates the shape attention weights and generates a shape-attended image feature from the raw image feature. Then, the raw image feature, shape-attended image feature and the shape feature are aggregated to a fused shape-aware feature, which is passed to the lightweight classification network for prediction.

From Figure 1 and Table 2, it can be observed the shape information is crucial to distinguish different fine-grained types of erythroid cells, especially when the appearances of the cells are similar. Therefore, we explicitly extract the cell shape information as a binary shape mask and treat it as a prior knowledge (or inductive bias) for the classification model. We employ the cell image segmentation method proposed in tavakoli2021new to extract the shape mask for each BMEC cell image. The main steps of cell shape extraction are shown in Algorithm LABEL:alg:Framwork, while Figure 3 shows the corresponding intermediate and segmentation results. Three types of binary mask images are obtained: the convex hull, nucleus and cytoplasm. Figure 4 shows the image segmentation results of different types of erythroid cells.

Unlike tavakoli2021generalizability which utilized the nucleus and ROC (convex hull - nucleus) images as the shape images, we choose the convex hull image as our shape image. The main reason is tavakoli2021generalizability mainly focuses on the WBCs whose nucleuses are in apparently different shapes. However, for erythroid cells, their nucleuses are more intact and look quite similar comparing to WBCs (see Figure 4). Also, the cytoplasm (i.e., the ROC in tavakoli2021generalizability) images of the erythroid cells are noisy and not suitable to represent the cell shapes. In contrast, convex hull contains the key shape characteristics and it is more robust to noise comparing to the nucleus. We pass the binary image of convex hull to our network and also compute the shape attention to enhance the original image feature.

To explicitly integrate the shape information into the deep learning framework, we design a dual-branch network which encodes the RGB image and the binary shape mask (convex hull image) in two separate branches.

As shown in Figure 2, the top branch network takes the RGB image as input and extracts the raw feature using Swin-Transformer liu2021swin, the state-of-the-art backbone model for image feature extraction. Although the RGB image branch can learn the pixel intensity variations between different cells, it does not focus on the cell’s shape or structure characteristics. Therefore, we introduce a separate bottom branch network which employs a VGG19 he2016deep model to explicitly compute the shape feature from the binary shape mask image obtained from the cell shape extraction module. The raw image feature and the shape feature from two branches are integrated in the shape attention module (Section 4.4) and fused in the later stage. In Section 5.4, we show the shape branch is an effective complement to the RGB branch for fine-grained cell classification. In addition, our dual-branch network is a general design such that the backbone models in each branch can be replaced by other deep image feature extraction models. In current our experiments, we find using the combination of Swin-Transformer and VGG19 for the two branches can achieve the best performance (Table 4 and 5).

To further integrate the shape information to the extracted image feature, we propose a novel shape attention module which takes the raw image feature and shape feature as input and outputs a shape-attended image feature (see Figure 2 middle). First, the shape attention weights are computed from the binary shape mask image following the similar attention mechanism used in SENet JieHu2018SqueezeandExcitationN. Then, the raw image feature is multiplied by the shape attention weights to get the shape-attended feature. The mathematical formulation is as follows:

The last module of our pipeline is a conventional classification network which consists of two sets of fully connected, ReLu and Dropout layers (Figure 2 right). The raw image feature $F_{R}$, shape feature $F_{S}$ and shape-attended image feature $F_{SA}$ are concatenated to form an fused cell feature $F_{C}=\{F_{R},F_{SA},F_{S}\}$ for final classification. With the shape feature and shape attention explicitly considered, the shape-aware network can achieve superior performance comparing to only using the RGB feature for the fine-grained erythroid cell classification (see Table 6).

The network architecture in Figure 2 is implemented with the Timm rw2019timm and PyTorch paszke2017automatic. Both branch networks (Swin-Transformer, VGG19) are initialized with ImageNet pre-trained weights. Furthermore, the shape attention module and cell classification network are initialized with Kaiming initialization KaimingHe2015DelvingDI. We adopt the SGD optimizer with a momentum of 0.9, weight decay of 2e-5 and a cosine lr schedule with the warm-up strategy loshchilov2016sgdr. We set the warm-up learning rate to 1e-4, the maximum lr to 1e-3, and the minimum lr to 1e-4. Four data augmentation techniques are used: random crop, random horizontal flip with probability of 0.5, random vertical flip with probability of 0.5, and color jitter with factor of 0.4. The network is trained for 300 epochs with batch size of 32 on three NVIDIA GTX 3090 GPUs and the total training takes 3 hours.

Our dual-branch network allows flexible choices of different backbone models for the cell classification. To quantitatively evaluate the performance of different backbones for the two branches, we conducted classification experiments with various classic and SOTA image feature extraction models on our BMEC dataset and compared the results in Table 4 and 5. Since the RGB image in the top branch contains more color variations than the binary shape mask in the bottom branch, it is expected that a more complex backbone model is needed for the top branch, while a relative simple model may be sufficient for the bottom branch. Therefore, in Table 4, we fixed the bottom branch with a VGG19 which is a classic CNN based model and tested different backbones for the top branch. It is shown the SOTA Swin-Transformer model can produce the best accuracy and F1 score comparing to other models such as ResNet he2016deep, DenseNet huang2017densely or ResNest HangZhang2020ResNeStSN for extracting features from the RGB images. Similarly, we fixed the top branch network as Swin-Transformer and evaluated different models for the bottom branch. The results in Table 5 shows the VGG19 model is most suitable for the bottom branch, while one possible reason is that other models may underfit or overfit the binary shape mask images. Based on above experiments, we finally choose Swin-Transformer and VGG19 for the top and bottom branches, and our final model achieved 81.12% accuracy on BMEC. Note that the backbones can always be easily evaluated and replaced when newer and more powerful image models are proposed.

To further demonstrate the effectiveness of our shape-aware network, we conducted classification experiments on two other public WBCs datasets, LISC rezatofighi2011automatic and Raabin-WBC kouzehkanan2021raabin. We retrained our model using their training data and compared the testing results with the reported numbers in rezatofighi2011automatic and kouzehkanan2021raabin. As shown in Table 7 and 8, we achieved the best classification accuracy on both datasets, showing the generalizability of our method. In addition, the high accuracy (over 98.5%) on both WBC datasets indicates the classification of WBCs is relatively easier than the erythroid cells since the shape differences of WBCs are more prominent. We hope our BMEC dataset can inspire and facilitate more research on fine-grained classification and other learning tasks for erythroid cells.

To verify the effectiveness of the shape information, we performed an ablation study which compared our full dual-branch network with the RGB-only single branch network (see Table 6). Specifically, we disabled the bottom branch and only used the raw image feature for classification. In this case, the network becomes a conventional RGB image based classification model. When two branches are enabled, both the shape feature extracted from the binary shape mask and shape-attended image feature computed from the shape attention module are fused to the raw image feature for classification. For the single branch network, different backbone models are tested on the BMEC RGB images. For our dual-branch network, the VGG19 is used in each configuration so that we can compare the performance improvement for each case. From the results in Table 6, the dual-branch network can always outperform the single-branch, confirming the effectiveness of the shape information for improving the fine-grained cell classification.