\hrefhttps://www.kaggle.com/mi2datalab/mementomlMementoML: Performance of selected machine learning algorithm configurations on OpenML100 datasets

Finding optimal hyperparameters for the machine learning algorithm can often significantly improve its performance Probst et al. [2018]. But how to choose them in a time-efficient way? In this paper we present the protocol of generating benchmark data describing the performance of different ML algorithms with different hyperparameter configurations. Data collected in this way is used to study the factors influencing the algorithm’s performance.

This collection was prepared for the purposes of the study presented in the EPP paper Gosiewska et al. [2020]. We tested algorithms performance on dense grid of hyperparameters. Tested datasets and hyperparameters were chosen before any algorithm has run and were not changed. This is a different approach than the one usually used in hyperparameter tuning, where the selection of candidate hyperparameters depends on the results obtained previously. However, such selection allows for systematic analysis of performance sensitivity from individual hyperparameters.

This resulted in a comprehensive dataset of such benchmarks that we would like to share. We hope, that computed and collected result may be helpful for other researchers. This paper describes the way data was collected. Here you can find benchmarks of 7 popular machine learning algorithms on 39 OpenML datasets.

The detailed data forming this benchmark are available at: https://www.kaggle.com/mi2datalab/mementoml.

Kühn et al. [2018] introduced a benchmark of algorithms created for the OpenML repository. This dataset contains data about 6 algorithms written in R: glmnet, rpart, kknn, svm, ranger, xgboost. It also allows us to run some additional computations and obtain further results in a similar way.

The Smith et al. [2014] provides a MongoDB database that has got data on an instance level. It contains predictions made for every single instance in the considered datasets, information about algorithms and its hyperparameters. There is also a possibility to extend this benchmark.

mlpack benchmark Edel et al. [2014] contains data about performance of different algorithms in popular machine learning frameworks/libraries. It also provides comprehensive scripts for further evaluation.

We used several number of popular machine learning algorithms: gradient boosting on decision trees (catboost Prokhorenkova et al. [2017], gbm, xgboost), generalized linear models (glmnet Friedman et al. [2010]), $k$ nearest neighbours (kknn), random forests (randomforest Liaw and Wiener [2002], ranger Wright and Ziegler [2017]). All computations were made in R and all models were from R packages. Almost all of them are used through the mlr Bischl et al. [2016] framework. Only catboost was used directly, because is not included in mlr.

Benchmarks on predicted values were calculated with mlr function performance for all models except catboost. For that one were used measureACC and measureAUC.

Each dataset was divided into $20$ train/test bootstrap pairs. Each row is not guaranteed to be chosen exactly once to the test set or $19$ times to the train set because each pair was chosen independently in a bootstrap manner. Each considered model with a particular set of hyperparameters was $20$ times trained, one time on each train subset and evaluated on a corresponding test. There were two collated measures: ACC and AUC. Thus, for each model and each dataset should be $20\cdot|paramset|$ ACC and $20\cdot|paramset|$ AUC measures. However, not all computations finished and they are updated on a regular basis as they progress. Additionally, learning times were collected.

What is important in our approach, the first thing we did is arbitrary choosing datasets and randomly choosing parameters in a way described above in a table 1. Thus they had been all chosen and set before any calculations began and they were not updated meanwhile. Splits to train/test subsets are also constant for each dataset. This makes comparing results between different algorithms easier. We are trying to cover as much of parameter space as possible, including also parameters in subspaces that may not work well or even work at all. This should make possible researches in more broad-spectrum concerning hyperparameters. However, for practical reasons, we had to draw them only from finite ranges.

All hyperparameters were chosen with a fixed random seed, thus they can be reproduced at will. However, not all parameters were chosen in the same way. catboost parameters were drawn using a newer version of the script. In this updated version of the scipt, there is also a possibility to draw parameters for other algorithms.

Similarly, all algorithms had fixed random seeds before every single run (default seed parameter in a calculate function). Thus you can easily and independently reproduce only some of the results without the need to make all previous calculations. Some of the results were reproduced to ensure the proper functioning of the whole framework.

Moreover, we trace versions of the used software libraries.

It is much easier to make calculations by your own using our docker container. This container is a modified r-base image that installs all needed both linux and R packages, has a structure compliance with presumptions of a calculate.R script, has got a screen utility, copies train/test splits from ”datasets” directory and parameters from ”parameters”. If you want to add some new algorithm that is not present in mlr framework, you need to add an install statement in an install_packages.R script and add new function to calculate.R with compliant result vector.

To build docker just run:

sudo docker build .

Before you run docker ensure you have a directory for results in a host. results directory inside a container has to be matched with the host’s results directory.

To run a docker simply type:

sudo docker run -ti -v [host’s results absolute directory]:/results elo bash

Open R console after running this command, source calculate.R function from ”scripts” and run it with proper parameters.

All results will be saved to host’s result directory passed in a docker run command.

This work results in a comprehensive dataset with wide hyperparameter space covered. Dataset is ready to use, easily accessible and, if needed, allows further calculations in compliance form. It can be used in developing a branch of machine learning - meta-learning, finding the best defaults of hyperparameters and reasonable subspaces as well as discovering the impact and importance of particular ones.

This dataset can be found at: MementoML