AutoCl: A Visual Interactive System for Automatic Deep Learning Classifier Recommendation Based on Models Performance.

In recent years, we’ve seen an increase in the use of deep learning (DL) models to support data-driven decision making in various fields. such as speech recognition, computer vision, natural language processing, drug discovery, biomedical sciences, etc. However, despite the effectiveness of DL models, we still have a limited theoretical understanding of these models. Additionally, developing a DL model requires some highly technical expertise and domain knowledge. For example, to improve the performance of DL model, careful selection of model, layer, epoch, optimizer, and batch size is critical. Classification is one of the most important machine learning problems, hence there is a wide range of classification algorithms are available. However, in most real-world applications, the selection of a classification method for a new dataset or application area is still challenging task.

Numerous automated machine learning (AutoML) methods have been proposed by researchers to fill the gap in human knowledge by automating the DL model developing process. These systems often have steep learning curves due to manual input and a lack of performance visualization features [8]. As a result, these systems are most of the time inaccessible to the growing number of practitioners who lack the time or background to learn sophisticated tools [4]. Unlike other common AutoML methods, AutoCl is not a black-box optimization process. It can compare and visualize the performance of multiple classifier in class level and recommends best performing classifier with proper hyperparameter.

Most of the exisitng AutoML methods recommends single model as output where AutoCl not only recommends a best model but also provides multiple model output as well as gives the users hyperparameter tuning capability. So that it easier to compare the performance of multiple model in class level. Therefore, from the existing studies, it is observed that choosing a better model for performing any task is challenging without strong DL expertise and domain knowledge.

The main objective of this paper is to facilitate non-experts such as researchers and practitioners to choose the most appropriate deep learning classifier for satisfying their requirements. In this paper, we present AutoCl, an interactive deep learning classifier recommendation system as shown in Figure 1, where we visualize the performance ranking of multiple DL models and recommend best possible model with best hyperparameter settings. The contributions of this work include:

• •

  We demonstrate a suite of visualization tools that illustrate the performance of multiple classification algorithm.

• •

  We show how hyperparameter affects the classifier performance in class level.

• •

  Finally the system recommend a top performing classifier with proper hyperparameter.

The outline of this paper is organized as follows. In Section 2, we explore the related work, and in Section 3, we briefly discussed the system design and development to explore the performance of deep learning classifiers. Sections 4 to 6 illustrate the implementation and discussion to evaluate the performance. Finally, in Section 7, we provide a conclusion.

Building a deep learning classifier by non-experts is a challenging task, it’s also time consuming and error-prone procedure. The challenges motivated many researchers to come up with various AutoML systems to support automatic algorithm selection and Hyper-parameter tuning over the years. In those system they followed statistical approaches, decision trees, neural networks, and other computational intelligence techniques.

In a study Maher and Sakr proposed SmartML [7] a system for automatic classification algorithm selection and hyperparameter tuning using both meta-learning and Bayesian optimization. Their system limited to work on only 15 machine learning classifier. Auto-sklearn  [2] is another framework based on scikit-learn for automatic classification and regression task. But it only work for traditional ML algorithms (such as SVM and KNN).

Only a few deep learning framework are proposed for AutoML tasks. Auto-Keras [5] is one of them. It is built based on Keras which is another popular deep learning framework. Auto-Keras focuses on discovering neural network architectures that compete with human-experts. Finally it provides an appropriate deep learning model based on user input data. It also supports multi-modal and multi-task. In addition, very few studies have examined and compared the performance of diffrent AutoML systems [3] [1]. In general, these studies reveal that there is no obvious winner since there are always trade-offs that must be addressed and optimized based on the context of the problems and the users need.

To the best of our knowledge, this is the first work that automatically recommends best possible deep learning classifier with proper hyperparameter to the non-expert users interactively. It also enables user to tune diffrent hyperparameter settings and compare multiple models performance metrics on class level.

We design an interactive recommender system for non-expert users named AutoCl to automatically recommends DL classifier with proper hyperparameter settings. We describe how AutoCl processes data and recmommends best classifer based on user input data. The system can enables user to tune multiple hyperparameter settings to get optimum result. The system architecture of AutoCl shown in Figure 2 consists of two modules, i.e., (A) background unit and (B) interface unit.

A. Background Unit: The background unit of AutoCl works as a data processor. As the system evaluates multiple models and their metrics, it takes a dataset as an input, and after prepossessing the data, it generates evaluation metrics as output. First, the dataset goes through prepossessing, and it splits into training and testings subsets. It contains file indices, ground truth levels, or class information. The dataset is trained on multiple deep learning classifier and evaluated based on the testing dataset. The background unit calculates high-level statistics about the performance of each model to construct the performance result summary. For automatic recommendation task, AutoKeras [5] library run on multiple classifier and generates a summery of classifiers with best combination of hyperparameter. It do a neural architecture search (NAS) with network morphism based on Bayesian optimization to select a best classifier settings.

Finally, the classifier’s performance metrics and recommendation data are saved in a JSON file, and the data goes to the interface unit for visualization. The model’s performance data includes model name, hyperparameter settings such as layer structure, epoch number, batch size, optimizer, etc., and metrics such as accuracy, precision, specificity, sensitivity. It also includes list of top ranked classifiers with appropriate hyperparameter for recommendation task.

B. Interface Unit: AutoCl interface unit consists of three user interface (UI) modules, and it is responsible for visualizing all the results. As shown in Figure 1, modules are visualized as follows: A) This module enables users to choose and tune options such as hyperparameter and metrics. B) A bump chart performance ranking view showing the performance of multiple models in the class level. C) A recommendation view that automatically show the top ranked classifier with suggested hyperparameter settings. Users can adopt best possible classifier for their task suggested by the recommendation module. They can also tune and compare multiple classifier effectively and gather actionable insights determining which classifier to adopt using a combination of different hyperparameter.

In this section, we show how AutoCl can effectively support non-experts to choose best deep learning classifier for their work. Specifically, we demonstrate how multiple classifier performance comparison and tuning hyperparameter can support their exploration.

John, a student who is interested in image classification problem. For a study he need to select a image classifier. He wants to know, how an image classifier model classify images. He has done some data visualization before, but he doesn’t have much deep learning expertise. He finds CIFAR-10 dataset, which is publicly available [6]. It is a multiclass dataset containing 60000 32x32 color images. It has ten (10) classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Each class has exactly 6,000 images. There are 50,000 training images and 10,000 test images in the dataset.

He loads the dataset in AutoCl system and specify few classifier (CNN, VGG-16, AlexNet, DenseNet, and ResNet-50) and mentions the classes he want to predict. Once he run the AutoCl Background Unit, the system process the data and calculate necessary metrics and recommender data. Then the data shows in Interface Unit. He can see that the AutoCl recommender view suggest him top performing classifier with necessary hyperparameter. CNN performs better where ResNet50 performs poorly.

He also want to know which classifier provide better performance on class level. From the AutoCl Option Selection and Performance Ranking view 1(A-B), he sets different hyperparameter and observe the performance metrics such as accuracy precision instantly. He selects the epoch from option settings and discover that the performance varies for different epoch number. The check all the options settings and interactively discover and learnt how the classifier classifies the image in class level. After using the AutoCl dashboard, finally he come to a decision that CNN is the most performing classifier for his task on given CIFAR-10 dataset.

We conduct a two-stage evaluation study to assess the potential usability and usefulness of our system. In the first stage, we demonstrated a usage scenario as described in section 5. In the second stage, AutoCl is compared with three different recommendation system as shown in Table 1 where (Yes) and (No) indicates the presence and absence of the feature selections [7],  [2],  [5]. In short,  AutoCl is useful for recommending classifier and evaluating performance at the class levels to compare multiple classifier effectively.

In this paper, we presented AutoCl, an interactive recommender system to suggest best performing classifier to non-expert users. Although there are several cloud based and opensource AutoML system exits, those are either costly or not feature rich. Our system is helpful for non-experts to effectively select deep learning classifier for their work. It help them interactively tune hyperparameter and suggest best performing classifier. In the future, we will improve the system to be used as a visual interactive DL tool to update and learn additional models at the instance level with a higher accuracy rate. We will also incorporate our system for other deep learning task such as prediction, natural language processing etc.