After obtaining his BSc. in Artificial Intelligence and MSc. in Computer Science, Mike Huisman joined LIACS in March 2021 as a teaching Ph.D. candidate. He is part of the Reinforcement Learning Group at LIACS and has also joined the ADA group in November 2021.
His research is in the field of deep meta-learning, where the goal is to learn an efficient deep learning algorithm that can learn new tasks quickly (from a limited amount of data). This can democratize the use of deep learning by lowering the threshold for using powerful deep learning technologies in terms of required compute resources and data. He conducts his research under the supervision of Prof. Dr. A. Plaat and Dr. J.N. van Rijn.
On 29 July, I defended my bachelor thesis entitled “Parallel Algorithm Portfolios in Sparkle” after half a year of work following the project start in February 2021. For this project I was part of the ADA group and was supervised by Koen van der Blom and Holger Hoos.
In a parallel portfolio, a set of independent, non-communicating solvers solve problem instances in parallel. Within the thesis, insight is given into the design and implementation of parallel portfolios into Sparkle. Additionally, experiments show the validity of the implementation and the practical capabilities of parallel portfolios as implemented in Sparkle.
My work built on the existing platform created by the Sparkle development team. The thesis is of value to the platform since the state of the art is represented by a set of algorithms, and parallel algorithm portfolios allow for an easily accessible method to conduct experiments with such portfolios.
Below an example is outlined showing how parallel algorithm portfolios can be used in Sparkle. Once the platform is initialized, the needed components (problem instances, solvers) are added to Sparkle. Using these the portfolio can be constructed. With the constructed portfolio and a selection of instances, an experiment can be conducted. Finally, we generate a report that describes the used components and an empirical evaluation of the results from the experiment.
First results in the thesis suggest that parallelising up to 64 instances of a nondeterministic solver with different random seeds results in performance gains compared to lower numbers of instances. When using 128 solver instances performance is similar to 64 solver instances, and there no longer seems to be a benefit to using more solver instances. With 256 or more solver instances the overhead appears to increase and performance starts to drop again in this practical implementation.
If you are interested in the project, feel free to take a look at the bitbucket project page and try Sparkle for yourself.
Machine learning (ML) is used more and more in a wide range of applications, but there are not enough machine learning experts to properly support this growth. With automated machine learning (AutoML) the goal is to make ML easier to use. As a result, experts should be able to deploy more ML systems, and less expertise would be needed to work with AutoML than when working with ML directly.
AutoML adoption
For AutoML to have this effect, however, it first needs to be adopted in practice. To study adoption levels in practice, we held a survey among teams that engineer software with ML components. As it turns out, adoption is not actually very high. In the figure below, we can see that, depending on the AutoML technique, more that 20 or even 30% of the respondents did not adopt AutoML at all. In addition, another 50 to 60% do not adopt AutoML techniques completely.
Adoption per technique. The first bar shows adoption of the feature technique including ‘Implicit, e.g. deep learning’ answers, while in the second bar they are excluded.
What holds back adoption?
Although we were able to assess the adoption levels through the survey, it does not tell us why people do or do not adopt these AutoML techniques. To find out, we started holding interviews, and already gained some insights based on two initial interviews.
The people we talked to raised a number of usability concerns. Firstly, there is a perception that AutoML techniques are hard to use and will require a significant initial investment to learn how to use. Secondly, someone indicated that in practice AutoML systems sometimes fail to work on the used data, but do not always make clear what the problem is. Thirdly, a participant raised that it is difficult to predict a good run length for AutoML systems. Related to this, there were also concerns about the required computational resources to use AutoML.
Looking forward
Clearly, there is room to improve AutoML adoption, and there are still some issues to resolve that could help to increase adoption. To get a more crisp view of which issues are common, and which are more atypical we plan to hold more interviews. In addition to learning about common issues, we also hope to find out whether there are differences between organisation types. Government organisations dealing with citizen data may, for instance, be more hesitant to adopt AutoML for privacy or bias sensitive applications.
Paper
If you are interested in the details you can learn more by reading our paper “AutoML Adoption in ML Software“ which was published in the AutoML workshop at ICML 2021. Authors: Koen van der Blom, Alex Serban, Holger Hoos, Joost Visser. Link: https://ada.liacs.nl/papers/BloEtAl21.pdf
Many fields of computational science advance by improving the algorithms for solving key problems, such as SAT solving or binary classification. When comparing these algorithms, they are usually ranked based on some performance metric, with the algorithm that performs best over a set of problem instances getting the highest score. In other words, the generalist is considered to be the best algorithm.
Meta-algorithms
Meta-algorithmic techniques, such as automated algorithm selection, are able to provide great performance improvements over such a generalist approach. They do this by taking advantage of the complementary strengths of different algorithms, or in the case of algorithm configuration, of configurable or tunable algorithm components. Algorithms that are specially selected or configured for a specific subset of problem instances usually perform better, on this subset, than a generalist algorithm.
However, meta-algorithms are complex and difficult to use correctly. At the same time, when using them incorrectly, you may not benefit from their full potential, or in the extreme case, you may even obtain worse performance than without using them.
Sparkle
The Sparkle platform aims to make meta-algorithms more accessible to novice users. To achieve this, Sparkle implements standard protocols for algorithm selection and algorithm configuration that support their correct use. After an experiment, the results, as well as the problem instances, algorithms and protocol used, are presented in an automatically generated LaTeX/PDF document, parts of which can be easily copied and pasted into scientific papers.
As an example, we outline below how algorithm selection can be used in Sparkle. Once the platform is initialised, the components that we need (problem instances, solvers and a feature extractor) are added to Sparkle. Using these, the feature values for each instance, and the performance values of each solver on each instance can be computed. With those values a portfolio selector is constructed, which can be used to automatically select a solver for new problem instances. Finally, we generate a report that describes which components were used, which process Sparkle used to construct a portfolio selector with those components, and how large the marginal contribution of each solver is to this portfolio selector.
With the first version of Sparkle ready, we are working to improve it further by making things easier to use, and providing more meta-algorithmic techniques. For this we are supported by the software lab at LIACS, headed by Prof. Joost Visser. From this lab, Jérémie Gobeil helps to both improve Sparkle itself, as well as the development processes. Additionally, several students, including Jeroen Rook and Richard Middelkoop, also put work into Sparkle as part of their studies.
Besides working in the ADA research group, Richard studies Computer Science & Economics where his main interest lies in being able to apply codes to real-life applications. Therefore, he found the ADA group an excellent environment to work on his bachelor project.
Richards’ project is concerned with Parallel Algorithm Portfolios, and adding this new functionality to the Sparkle platform. The problems which can be solved by the functionality will encompass decision problems, optimisation problems with a single solution and optimisation problems with several possible solution. All being well, the project will showcase a practical application of Parallel Portfolios.
This blog post has originally been published on automl.org
By Koen van der Blom, Holger Hoos, Alex Serban, Joost Visser
In our global survey among teams that build ML applications, we found ample room for increased adoption of AutoML techniques. While AutoML is adopted at least partially by more than 70% of teams in research labs and tech companies, for teams in non-tech and government organisations adoption is still below 50%. Full adoption remains limited to under 10% in research and tech, and to less than 5% in non-tech and government.
Software engineering for machine learning
With the inclusion of ML techniques in software, the risks and requirements of the software also changes. In turn, the best ways to maintain and develop software with ML components is different from traditional software engineering (TSE). We call this new field software engineering for machine learning (SE4ML).
Based on a study of the academic and “grey” literature (the latter comprises blog articles, presentation slides and white papers), we identified best practices for SE4ML, and composed a recommended reading list on the topic. These practices include the use of AutoML techniques to improve the use of ML components in software. All practices are described in our practice catalogue. We encourage readers to have a look and to send us their suggestions for additions.
Figure 1: Engineering best practices for machine learning.
Having a solid overview of the best practices, we set up a survey to find out the extent to which these SE4ML practices have already been adopted. We asked teams working on software with machine learning components to which degree they followed each practice in their work. In the resulting data we saw that tech companies have the highest adoption of the new ML related best practices, while research labs have the highest adoption of AutoML. Overall, more practices are adopted by teams that are larger and more experienced, with practices originating from TSE seeing slightly lower adoption than the new ML specific practices.
Effects of best practice adoption
Besides the adoption of best practices, we also investigated the effects of the practices. This resulted in insights into which specific practices relate to which desired outcomes. For instance, we found that software quality is influenced most by continuous integration, automated regression tests, and static analysis. On the other hand, agility is primarily affected by automated model deployment, having a cohesive- multi-disciplinary team, and enabling parallel training experiments. With these insights into the effectiveness of different practices, we hope to increase practice adoption and improve the quality of software with ML components.
A recent line of research advancements in ML focuses on automated machine learning (AutoML), an area that aims to automate (parts of) the construction and use of ML pipelines to enable a wider audience to make effective and responsible use of ML, without needing to become an expert in the field. We took a closer look at our survey results and found that, compared to non-tech companies and governmental organisations, research labs and tech companies are ahead in the adoption of AutoML practices (see Figure 2).
Figure 2: AutoML adoption by organisation type.
While overall AutoML adoption is similar across continents, non-profit and low-tech organisations see higher adoption in Europe than in North America. We also found that teams with multiple years of experience are more likely to adopt AutoML techniques. Finally, across the board, there is significant room to increase adoption of AutoML, but this is especially true for non-tech companies and governmental organisations.
More detailed information on our findings regarding AutoML adoption can be found in our piece titled “The 2020 State of AutoML Adoption”.
Looking forward
Based on feedback from and interviews with participants, we recently revised our survey to learn more. Specifically, in our latest version of the survey, we ask in more detail about responsible use of ML and about the adoption of different AutoML techniques such as automated feature selection and neural architecture search. You can help us by taking the 10-minute survey and by spreading the word. If you want to stay up to date with our progress, keep an eye on our website.
Koen van der Blom joined the ADA research group as a post-doctoral researcher. From March 2019 onward he started working with Holger Hoos in the area of meta-algorithmics. One of the things he works on is a tool called Sparkle. Sparkle aims to make meta-algorithmics such as algorithm selection and configuration easier to use for a wide audience. Besides this, he is also interested in performance analysis and prediction. What can be said about the expected performance of a new instance, based on previously seen instances? And how can you compare performance in a fair way?
Before this, during his PhD, Koen worked on multi-objective mixed-integer evolutionary algorithms applied to early-stage building design under the supervision of Michael Emmerich, Hèrm Hofmeyer, and Thomas Bäck. He continues to be interested in these problems, and particularly when it comes to optimisation in mixed-integer spaces. Who knows, perhaps combining aspects from the old and new will yet lead to other exciting work.
ADA Research Group 