Enabling Automated Machine Learning for Model-Driven AI Engineering

AutoML aims to automate the practice of ML for real-world applications. It offers end-to-end solutions to non-experts (and experts) in the field of ML in a more efficient manner, compared to the manual approach. The automation might be applied to any part of the ML pipeline: from data pre-processing to hyper-parameter optimization and model evaluation. Most existing AutoML solutions concentrate on one particular part of the ML pipeline [1]. In this work, we focus on ML model family/architecture selection, as well as hyper-parameter optimization for the ML model. To this aim, we construct a tree-structured search space by considering a number of ML methods that have proven useful for the selected use case scenario of the case study that will follow. The set of selected ML methods comprises (i) Decision Trees (DT), (ii) Random Forests (RF), (iii) Gated Recurrent Units (GRU), (iv) Long Short-Term Memories (LSTM), (v) Fully-Connected Neural Networks (FCNN), also known as Multi-Layer Perceptron (MLP), (vi) Denoising Autoencoders (DAE), (vii) Factorial Hidden Markov Models (FHMM), and (viii) Combinatorial Optimization (CO). Figure 1 illustrates the mentioned search space with these ML methods and the possible hyper-parameters that must be automatically tuned for each of them should they get selected by the AutoML engine.

We consider the possible choices for setting the hyper-parameters as listed below: (a) criterion $\in$ {MSE (Mean Squared Error), Friedman_MSE, MAE (Mean Absolute Error)}; (b) min sample split $\in$ Uniform [2,200]; (c) n estimators (i.e., number of estimators) $\in$ Uniform [5,100]; (d) optimizer $\in$ {Adam, Nadam, RMSprop}; (e) learning rate $\in$ {1e-2, 1e-3, 1e-4, 1e-5}; (f) loss function $\in$ {MSE, MAE}; (g) n layers (i.e., number of layers) $\in$ Uniform [5, 8]; (h) dropout probability $\in$ Uniform [0.1, 0.6]; (i) sequence length $\in$ {64, 128, 256, 512, 1024}.

To enable the automated selection of the best ML model architecture/family and the most appropriate hyper-parameters for the selected ML model given a specific use case and a dataset, we deploy Bayesian Optimization (BO). There exist various open-source libraries that offer BO. We pick the most widely-used library, namely Hyperopt [4]. This library requires a Tree-structured Parzen Estimator (TPE) model and provides distributed, asynchronous hyper-parameter optimization. Moreover, it supports search spaces with different types of variables (continuous and discrete), varied search scales (e.g., uniform vs. log-scaling), as well as conditional variables that are only meaningful for certain combinations of other variables [4].

We augment MDSE with AutoML to support software developers who might not be ML experts in creating smart software systems. We provide them with a Domain-Specific Modeling Language (DSML) that is based on the prior work ML-Quadrat [3], which supported automated source code and ML model generation for smart IoT services. In ML-Quadrat, the practitioners themselves had to specify the target ML model family/architecture, as well as the values of the corresponding hyper-parameters. However, to assist the software developers who use the DSML and have ML requirements, but may not necessarily possess sufficient ML knowledge and skills, we provide the above-mentioned AutoML prototype as open-source software on Github¹¹1https://github.com/ukritw/autonialm. Currently, it is tailored to the target use case domain of our case study that is described below. Using this prototype that has also a web-based Graphical User Interface (GUI), one can select the most appropriate setup for ML and feed it into the ML-Quadrat DSML. Last but not least, we implement a number of constraints and exception handling mechanisms, according to the API documentations of the target ML libraries, namely Scikit-Learn and Keras/TensorFlow, in the model-to-code transformations (code generators) of ML-Quadrat. If the AutoML mode is enabled, then we can enforce these constrains and allow them to make necessary changes to the existing user-specified parameters of the model instances. Otherwise, we only let them show warnings without overriding any user-specified option. For instance, if standardization of numeric values must have been done before training a certain ML model, but this is missing, we warn them while generating the implementation of the target IoT service out of the software model. Only in the case that AutoML is ON, then we additionally add the proper standardization technique in the data pre-processing phase of the ML pipeline.

To validate the proposed approach, we demonstrate our experimental results of a case study from the smart energy systems domain. Here is the use case scenario: Given an aggregate signal of the electrical energy consumption of a household over a period of time, we aim to disaggregate the signal into a number of individual loads. This is called Non-Intrusive (Appliance) Load Monitoring (NIALM/NILM) or energy disaggregation, and is similar to the single-channel source separation problem in Physics and Signal Processing, but generally more challenging. We conduct a benchmarking of a number of competing approaches to NIALM, using the REDD [5] and the UK-DALE [6] reference datasets. It transpires that simple ML models, such as DT that have not been studied can actually outperform some of the more complex deep learning ML model architectures that were proposed in the literature. Note that training a DT is much faster and requires much less amount of data and computational, as well as energy resources, compared to a deep learning model. This example shows the benefits of the proposed AutoML-based approach, even for the ML experts themselves, since they could avoid trial-and-error practices for selecting the ML model and tuning its hyper-parameters. In addition, another deep learning ML model, namely GRU, which is proposed by this work, outperforms the state of the art approaches. Figure 2 illustrates the experimental results of our benchmarking study with the following NIALM approaches: (i) CO [7], (ii) DAE [8], (iii) DT, (iv) Dictionary-based (our implementation of [9]), (v) DNN-HMM (our implementation of [10]), (vi) FCNN, (vii) FHMM [11], (viii) GRU, (ix) LSTM [8]. Note that applying DT, FCNN and GRU to this problem is our initiative. Table 1 shows the average error (MAE) for the mentioned approaches. Note that we used the data of $37$ days from house 1 of the REDD dataset [5] with a sampling rate of $0.05$ Hz.

We proposed a novel approach to enable AutoML in model-driven software and AI engineering of smart IoT services. We automated the ML model family/architecture selection and the hyper-parameter configuration parts of the ML pipeline. The deployed BO method selects the best options for the ML model and its hyper-parameters, such that the prediction results of the trained ML model outperform any manual practices, or at least perform as good as them²²2Obviously, this holds only for the supported ML models and methods by the AutoML engine..

This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.