SAIA: Split Artificial Intelligence Architecture for Mobile Healthcare Systems

Our proposed Split Artificial Intelligence Architecture (SAIA), as shown in Fig. 1, consists of four components (i.e., the data pre-processing interface, the embedded AI, the networked AI and the decision unit) that work synergistically between the client side and the server side. For each use case, SAIA has two phases: preparation and operation. In the preparation phase, four components would be prepared and trained accordingly: (i) the data pre-processing interface (including objection detection, semantic segmentation and feature extraction) (Section 2.2); (ii) the embedded AI contains certain lightweight classification classifier(s) (Section 2.3); (iii) the networked AI trains a multi-classifier fusion of several advanced DNN classifiers (Section 2.4); and (iv) the decision unit is a lightweight ML classifier that trains on a set of labeled meta data (Section 2.5).

In the operation phase, (i) the client (i.e., mobile or IoT devices) receives the data, passes the data through the data pre-processing interface (including objection detection, semantic segmentation and feature extraction); (ii) the client evaluates the data on the embedded AI (i.e., the lightweight classifier(s)), and produces the unlabeled meta data accordingly; (iii) the decision unit (DU) evaluates the meta data, if DU decides to keep the data on the client side, the testing result of the embedded AI is returned, otherwise, the data would be sent to the networked AI for further evaluation. Below presents the details about the design of each components.

In this component, we design and implement a set of objection detection, semantic segmentation and feature extraction algorithms that could fit on various image-based healthcare applications.

Objection detection: Since the medical images captured by the mobile and IoT devices usually contain complex background, it is of vital importance to separate the region-of-interest (ROI) from the background. We investigate two fast object detection approaches: (i) Faster R-CNN [23] and (ii) Single Shot Detector (SSD) [24]. We also conducted a preliminary experiment using both object detection approaches on the onychomycosis dataset [22] (more details about the dataset are described in Section 3.2), where we annotated 2,000 images that each contains a full-hand, and then trained and applied Faster R-CNN and SSD on the annotated images. In the result, Faster R-CNN obtains in a Jaccard Index of 99.6 in comparison with 98.2 obtained from SSD. Therefore, we decided to apply Faster R-CNN on the onychomycosis dataset (Section 3.2).

Semantic segmentation: We utilize Otsu’s thresholding segmentation algorithm [25] for the image semantic segmentation. By finding the optimal threshold from the histogram of pixel counts, the algorithm isolates the region-of-interest (ROI) (e.g., objects) from the complex backgrounds. For instance, it could separate the skin lesion from the normal skin and artifacts (hairs, badges and black borders). Compared with the other semantic segmentation approaches, such as U-Net CNN [26], Otsu’s thresholding segmentation algorithm not only has been shown to be effective in many medical image segmentation tasks [27, 28, 29], but also is more efficient in terms of storage usage, energy consumption and inference time while deploying on mobile and IoT devices.

Feature extraction: We design and apply different sets of feature extraction techniques to different healthcare applications. For instance, in skin lesion detection, melanoma (i.e., cancerous skin lesion) usually proliferates asymmetrically then appears as irregular shapes. Hence, the derived segmentation maps and corresponding gray-scale images are used to extract 9 structural features, including: asymmetric index [30], eccentricity, perimeter, max/min/mean intensity, solidity, compactness and circularity [31] . Furthermore, color variations are also effective characteristics to distinguish different types of skin lesions. Instead of using convention RGB color space, we employ CIELUV color space that enables us to well perceive the differences in colors. Besides, LUV color space also decouples the chromaticity (UV) and luminance (L), which yields invariant features in respect to the light condition. Instead of taking only statistics (mean, standard deviation, skewness and kurtosis) from LUV histogram [32], we utilize the whole distribution of colors , which are separated into 3 channels (i.e., L, U and V). As a result, $3\times 255$ color features would be generated from three normalized histograms. Moreover, we observe that the texture of lesions can also distinguish skin lesion types. Hence, local binary patterns (LBP) [33] analysis is applied to capture the textured information. We investigated several sets of radius and number of surrounding points then observed that using radius of 3 and 8 neighboring points yields the best performance of Embedded-AI with 26 textured features from each normalized LBP histogram. On the other hand, since the onychomycosis detection does not appear to be “shape-sensitive”, we only adopt the color-based features (i.e., LUV) and texture features (i.e., LBP) to it.

Since the client device has limited computational power, and limited battery life, we intend to utilize lightweight ML algorithms. The embedded AI solutions will be used to generate initial classification results; thus enabling embedded artificial intelligence. The use of lightweight algorithms would decrease the burden on battery life; which enables the operator’s equipment to last longer in contested environments. These algorithms also require less computational power, which results in producing intelligence in a more timely manner (than more complex algorithms).

These lightweight classification algorithms could be distance-based algorithms that are based on Euclidean or Manhattan distances, or it could be logistic regression. In both cases, the computation would be linear in the number of (reduced) features; and do not involve multiple layers of computations; thus, providing more timely intelligent results and consuming less power. Examples of such algorithms include:

Decision Tree (DT): DT [34] is a non-linear classifier. It is rule-based learning method that would construct a tree where the leaves represent class labels, and the branches represent conjunctions of features that lead to those class labels. The tree structure would depend on the algorithm and data used to generate it, but in certain situations, it might be lightweight and suitable for our embedded AI (e.g. if it was of linear complexity). The advantage is that it can handle non-linearly separable data (better than Logistic Regression).

Random Forests (RF): Decision tree classifier [34] usually yields high-variance and low-bias results, thus bagging (or bootstrap aggregation) is a remedy for such issues. RF [35] is a large collection of de-correlated trees, which are aggregated by taking their averages. Besides, trees generated in bagging is identically distributed, the expected value from this bagging set of trees is the same as the expectation of any tree in this set. Thus, variance reduction is only remedy of improvement.

Support Vector Machine (SVM): SVM [36] is a very popular non-linear classifier. It is a maximal margin classifier, meaning it tries to find a separating hyperplane that maximizes the margin between the different classes (while logistic regression, for example, tries to find any separating boundary). Using the Kernel trick, Kernel SVM can effectively discriminate between non-linearly separable classes, without incurring the cost of explicitly transforming the data to higher dimensions. A sample Kernel function is the RBF Kernel:

Kernel SVM requires storing a number of the training samples, called support vectors. Let this number be $n<<N$, and it is much smaller than the total number of training samples $N$. This means that Kernel SVM would have space and time complexity of $O(n\cdot M)$. While Kernel SVM is very useful in many applications, its training might not scale well with datasets that have large number of training samples (beyond tens of thousands).

Dropouts meet Multiple Additive Regression Trees (DART): DART [37] is an evolution of gradient boosting machine, that adopts the dropouts for regularization (preventing over-fitting) from deep neural networks. Boosted trees with XGboost [38] is one of the most well-performed learning structure, which results in a great number of winning solutions of data science competitions [39]. Apart from classification, boosting tree can be used in a wide range of problem such as regularized regression (Ridge and Lasso) [40], quantile regression [41] or survival analysis [42, 43]. Motivated by systems optimization and fundamental principle of machine learning, XGboost [38] is an efficient and flexible library with implementation of parallel tree boosting which enables fast and accurate results.

In our preliminary experiments, DART [37] outperforms the conventional multiple additive regression tree (MART) [44] and AdaBoost [45] in both datasets (i.e., skin lesion [46, 20, 47] and onychomycosis dataset [22]) in terms of both training time and accuracy. All the implementations regarding DART [37] in this work utilize both XGboost library for Python 3 [48] and Microsoft’s LightGBM framework [49] for gradient boosting machine.

Since the embedded AI algorithms might not handle non-linearly separable classification problems well, we aim to use more powerful algorithms in the networked AI. Such algorithms would include various advanced DNNs, and we design it as a multi-classifier fusion of those classifiers. These algorithms are more computationally intensive; but they can produce more accurate results; thus, providing more confident intelligence. This type of computation can be facilitated on the server side by using state-of-the-art big data technologies. Below presents the details of our multi-classifier fusion approach.

The core component of our proposed SAIA framework is the decision unit component, which controls whether a client-side captured sample (e.g., image) would be sent to the networked AI, or would be processed by the embedded AI. We adopt a meta-information based algorithm selection approach in the design of our decision unit component. In the training phase, we (i) use a set of meta-information generation samples (apart from the training/testing samples of the embedded AI and the networked AI) to generate a set of meta-information (e.g., various features directly extracted from each sample, and the soft predicted probabilities by the embedded AI for each sample), (ii) use our customized decision rule to generate the true label (i.e., “kept for the embedded AI” or “sent to the networked AI”) of each sample, and (iii) use the meta-information and true labels of those samples to train a lightweight binary classifier as the decision unit. In the testing phase, our framework extracts the same set of meta-information from each testing sample, and tests it through the pre-trained decision unit to determine whether sending the sample to the server or not.

To be simplified, we use the soft predicted probabilities provided by the embedded AI as the meta-information, and use gradient boosted trees to build the decision unit classifier. In this work, we adopt a basic decision rule: (i) if communication resources are available, we will send the meta-information of given sample to the decision unit, and then, if the embedded AI and the networked AI produce different predicted results (e.g., classes of a healthcare application) and the networked AI is correct, the sample will be sent to the server (i.e., using the networked AI), otherwise, it will be kept on the client (i.e., using the embedded AI); (ii) if the communication resources are not available, we will keep everything on the client (i.e., using the embedded AI).

Other than just using a “yes or no” binary decision, we also design the decision unit to utilize a parameter, namely $\epsilon$, to tune the criteria of how much data could be sent to the networked AI. To be specific, we propose a weighted loss function, where the gradients of the samples that should be “sent to the networked AI” are scaled by a parameter $\epsilon$. In the binary classification problem of the decision unit, let us denote the samples that should be “sent to the networked AI” as the positive class, i.e., $(y_{i}=1)$, and denote the samples that should be “kept for the embedded AI” as the negative class, i.e., $(y_{i}=0)$.

Then, the objective function of the gradient boosted trees at iteration $t$ is optimized by the simplified second-order approximation [38] of the original loss function, which is defined as below:

where $l(y_{i},\hat{y}^{(t-1)})$ is the cross-entropy loss function, $g_{i}=\partial_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}^{(t-1)})$ and $h_{i}=\partial^{2}_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}^{(t-1)})$ are the gradient and hessian statistic of the loss function and $\Omega(f_{t})$ is the penalized term. Our customized function $S(y_{i})$ is defined as below:

where $\epsilon$ is a predefined hyperparameter that could be adjusted to increase/decrease the expected true positive rate of the decision unit, so that to optimize the amount of data that should be sent to the server.

We implemented our embedded AI on a Google Pixel 4 XL smartphone that has a Qualcomm Snapdragon 855 chip-set, 6GB RAM and Android 10.0 OS. Our network-AI was implemented and performed on a server with Intel$\circledR$Core^TM [email protected] CPU, 128GB RAM and 4 GTX 1080Ti 11GB GPUs.

We investigated two popular benchmark healthcare image datasets in our experimental evaluation: (i) International Skin Imaging Collaboration Challenge 2019 (ISIC 2019) [46, 20, 47] and (ii) Onychomycosis dataset [22]. ISIC 2019 has training and testing sets with overall 33,569 images. Since the ground truth of the testing data was not available, we only employed its original training data in our evaluation. It contains 25,331 images of 8 skin lesion diseases (i.e., 8 classes): melanoma (4,522), melanocytic nevus (12,875), basal cell carcinoma (3,323), actinic keratosis (876), benign keratosis (2,624) dermatofibroma (239), vascular lesion (253) and squamous cell carcinoma (628). We randomly split 80%, 5% and 15% as training data, meta-information data and testing data, respectively. In order to have enough data to train the decision unit and base classifiers, data augmentation was applied to enlarge the training data and meta-information data by performing different rotation degrees (i.e., 90, 180 and 270), horizontal flipping and combinations of both. Thus, the training data and meta-information data became 81,020 and 10,400 in total, respectively. Regarding onychomycosis dataset, which contains 53,794 region-of-interest extracted abnormal (34,014) and normal (19,780) fingernail images. We split it into training/meta-information/testing by the ratio of 70%/10%/20%, respectively. Due to enough number of samples, we only performed horizontal flipping on the meta-information dataset. Note that our embedded AI and networked AI were trained on both original and augmented images.

Embedded AI. In the preparation of the Embedded AI, for each dataset, we trained three conventional machine learning classifiers (i.e., SVM, RF and DART). To figure out the optimal set of hyperparameters for each classifier, we performed 5-fold cross validation for each classifiers. Table 1 shows the performance (i.e., accuracy) of three classifiers on two datasets, where with the optimal settings, DART classifier outperforms SVM and RF classifiers on both datasets. Furthermore, while applying One-vs-All strategy for training SVM and RF classifiers in skin lesion dataset that has 8 classes, using SVM and RF result in considerably much larger model size than using DART, that consumes more storage space of the mobile and IoT devices. Also, since DART uses softmax function as the objective function, DART classifier directly provides the soft predicted probabilities of each sample, which would be taken as the meta-information to train our decision unit (as described in Section 2.5). Therefore, we decided to deploy the DART classifier as the embedded AI for both datasets in the rest of our experiments.

Networked AI We evaluated twelve different CNN architectures (as shown in Table 2) on the server side with pre-trained weights on ImageNet [50]. Different networks expect different input sizes: 331$\times$331 for PNASNet-5-Large and NASNet-A-Large; 320$\times$320 for ResNeXt101-32$\times$16d; 299$\times$299 for InceptionResNet-V2, Xception, Inception-V4 and Inception-V3; 224$\times$224 for SENet154, SE-ResneXt101-32, EfficientNet-B7, Dual Path Net-107$\times$4d and ResNet152. All the networks were fine-tuned in Pytorch, using SGD optimizer with learning rate 0.001 (degraded after 20 epochs by 0.1) and momentum 0.9. We stopped the training process either in 40 epochs or the validation accuracy failed to improve for over 7 consecutive epochs. To keep the same batch size 32 in each evaluation, and due to the memory constraint of single GPU, certain networks were trained parallelly with multiple GPUs: PNASNet-5-Large (4), NASNet-A-Large (4), ResNext101-32$\times$16d (4), SENet154 (2), EfficientNet-B7 (2) and Dual Path Net-107 (2). The performance result of each base CNN classifier on each dataset has been shown in Table 2. As described in Section 2.4, we utilize the multi-classifier fusion of those twelve advanced CNN architectures as our networked AI aiming to provide the SOTA performance of each dataset/application.

Decision Unit As presented in Section 2.5, we prepared the decision unit for both datasets accordingly. To evaluate the effectiveness of the tuning hyperparameter $\epsilon$, we use a discrete set of integers ranging from $0$ (i.e., meaning no decision unit deployed) to $100$ for $\epsilon$.

In this section, we compare the effectiveness of our proposed SAIA framework with three baselines: only using the embedded AI, only using the networked AI, and SAIA but with a randomized decision unit, that randomly determines whether sending a given sample to the server-side or not. All the experiments that utilized the randomized decision unit are performed 100 times and evaluated using the averaged results. Fig. 3a and Fig. 4a show that as we increase the value of $\epsilon$, more and more data would be sent to the networked AI. Also, the curves quickly converges as the $\epsilon$ increasing. Thus, one can tune $\epsilon$ based on the communication resource available to adjust how much data to be sent to the server-side for processing.

Fig. 3b and Fig. 4b illustrate that as we increase the value of $\epsilon$, the accuracies obtained by SAIA for both datasets are also increasing and quickly converged to the SOTA accuracy achieved by the networked AI. For instance, as shown in Fig. 3b, while $\epsilon=25$, SAIA achieves nearly the same accuracy as the networked AI (i.e., 90% vs. 90.6%), but only sends 70% of the samples to the networked AI (Fig. 3a). In Fig. 4b, while $\epsilon=17$, SAIA achieves exactly the same accuracy as the networked AI (i.e., 93.2%), but only sends around 80% of the samples to the networked AI (Fig. 4a).

Fig. 3c and Fig. 4c present the comparison among our proposed SAIA, SAIA with a randomized decision unit, only using the embedded AI and only using the networked AI. We observed that (i) our proposed SAIA consistently outperforms the SAIA with a randomized decision unit (other than while keeping all the data at the embedded AI or the network AI); (ii) as sending more samples to the server, compared with the SAIA with a randomized decision unit, the accuracy of our proposed SAIA framework quickly converges to the accuracy of the networked AI (i.e., while sending 75% samples of the skin lesion dataset, and while sending 80% samples of the onychomycosis dataset); (iii) while sending nearly half of the samples to the server, our proposed SAIA obtains the highest accuracy advantage over the SAIA with a randomized decision unit, e.g., as shown in Fig. 3c, while $\epsilon=10$, almost over half of the skin lesion samples (51%) were sent to the server, and the difference of the accuracies is 88.87% (ours) vs. 83.95% (randomized), and as shown in Fig. 4c, while $\epsilon=5$, 53% of the onychomycosis samples were sent to the server, and the difference of the accuracies is 92% (ours) vs. 86% (randomized). To summarize, our proposed SAIA could control how much data to be sent to the server based on the environment and accuracy requirement. Our framework could also achieve the same accuracy as processing on the server side, while sending much less data to the server side.

In this section, we evaluate the performance (i.e., accuracy) of our proposed decision unit (i.e., a lightweight binary classifier), and the effectiveness of the hyperparameter $\epsilon$ while influencing the performance of the decision unit. Fig. 5 and Fig. 6 illustrate the confusion matrices of the decision units while applying different $\epsilon$ for the skin lesion and onychomycosis datasets respectively. As we increase $\epsilon$, more samples that suppose to be sent to the server are determined by the decision unit to be sent to the server (i.e. the true positive rate increases). For instance, in Fig. 5, as $\epsilon$ increasing from $3$ to $10$, the ratio of data supposed to be sent to the server being sent to the server changed from 0.63 to 0.88. On the other hand, as $\epsilon$ increasing, more samples that suppose to be kept on the client are also determined by the decision unit to be sent to the server (i.e. the false positive rate also increases). For instance, in Fig. 5, as $\epsilon$ increasing from $3$ to $10$, the ratio of data supposed to be kept on the client being sent to the server changed from 0.26 to 0.44. Fig. 6 presents the same patterns. However, as illustrated in Fig. 8 and Fig. 9, as we increase $\epsilon$, even though both TPR and FPR are increasing, the TPR is always much higher than the FPR, and the average increasing speed of the TPR is always higher than that of the FPR.

In this section, we evaluate the efficiency of our proposed SAIA regarding to the elapsed time averaged over each sample. As illustrated in Fig. 8 and Fig 9, in our experiments on the skin lesion dataset, the elapsed time (second per sample) of the embedded AI and the networked AI are 0.308s and 2.51s respectively, and that of the onychomycosis dataset are 0.3s and 2.5s respectively. Fig. 8a and Fig 9a show that the elapsed time of SAIA system is linearly dependent on the percentage of data sent from the client to the server. Fig. 8b and Fig 9b illustrate that as we increase $\epsilon$, the elapsed time (second per sample) of SAIA would quickly converge to a constant value (e.g., for the skin lesion dataset, while $\epsilon=50$, the elapsed time of SAIA converges to around 1.89s; for the onychomycosis dataset, while $\epsilon=17$, the elapsed time of SAIA converges to around 2.11s). Furthermore, as presented in Fig. 8c and Fig 9c, while SAIA reach the same accuracy as the networked AI, SAIA has less elapsed times on both datasets (i.e., 1.89s vs. 2.51s on the skin lesion dataset, and 2.11s vs. 2.5 on the onychomycosis dataset). To summarize, even with enough communication resource, by applying the decision unit, our system does not have to send all the data to the server-side, while obtaining the same accuracy as sending all the data to the server-side, and much less processing time compared with the networked AI.

Many real-world applications (e.g., mobile healthcare, smart home, wearable technologies) require to collect and analyze the data on mobile and IoT devices. Hence, compact DNNs have been proposed to conduct inference on such devices. For instance, SqueezeNet [61] obtains AlexNet [62] level of accuracy with 50x fewer parameters and less than 0.5MB model size, by downsampling the data using $1\times 1$ convolution filters. MobileNet [63, 64, 65] proposes a useful building block, “inverted residual block” into its design of DNNs, that significantly reduces computation complexity without accuracy loss, compared with traditional DNN models. YOLO, a state-of-the-art, real-time object detection system, is designed by using customized architecture, that only has one forth operations of VGG16 [66]. EfficientNet [57] is one of the state-of-the-art DNN models recently proposed for execution on mobile and IoT devices, that uniformly scales each dimension (e.g., width, depth and resolution) of DNN models with a fixed set of scaling coefficients. Although the compact DNNs could dramatically reduce it computation complexity, the overall performance of compact DNN model still would not be as good as the more advanced models deployed on the server side, that could also be the ensemble/fusion of several well-trained DNN models

DNN model compression techniques [11, 12, 13, 67, 68, 69, 70, 71] have been proposed to reduce the size and computation workload of DNN models running on the mobile and IoT devices. For instance, Knowledge distillation [11, 12, 13] has been proposed to compress a model by teaching a simplified student DNN model, step by step, exactly what to do using a complex pre-trained teacher DNN model, and then deploy the student DNN model on the mobile devices. Network pruning [72] has been proposed to trim the network connections within DNNs that have less influence on the inference accuracy. Data quantization [69] has been proposed to educe the number of bits to represent each weight value of DNN models. However, certain recent DNN models, such as MobileNet [65] and EfficientNet [57] are already very compact and hard to compress significantly. Deploying compressed DNN models on the mobile and IoT devices also cannot take advantage of the more advanced models deployed on the server side.

Split-DNN architectures [15, 16, 17, 73] have been proposed to offload the execution of complex DNN models to compute-capable servers from the mobile or IoT devices, where a DNN is split into head and tail sections, deployed at the client side and the server side, respectively. For instance, Osia et al. [73] proposes a hybrid architecture where a DNN model, that has previously been trained and fine-tuned on the cloud, would be split into two smaller neural networks: a feature extraction network that runs on the mobile or IoT devices, and a classification network that runs on the cloud system, and both neural networks on the local device and the cloud system would collaborate on running the original complex DNN model. Matsubara et al. [18] proposes a KD-based Split-DNN framework to reduce the communication cost between the client and the server. However, such approaches usually cannot fully rely on the client-side model, thus unable to work if the communication is impeded. However, none of such approaches directly address the communication bottleneck between the client and the server. Also, the existing approaches cannot adjust the AI usage on between the client and the serve depending on the device’s condition (e.g., storage size, power consumption and communication bandwidth).