AutoML adoption in software engineering for machine learning

This blog post has originally been published on automl.org

By Koen van der Blom, Holger Hoos, Alex Serban, Joost Visser

In our global survey among teams that build ML applications, we found ample room for increased adoption of AutoML techniques. While AutoML is adopted at least partially by more than 70% of teams in research labs and tech companies, for teams in non-tech and government organisations adoption is still below 50%. Full adoption remains limited to under 10% in research and tech, and to less than 5% in non-tech and government.

Software engineering for machine learning

With the inclusion of ML techniques in software, the risks and requirements of the software also changes. In turn, the best ways to maintain and develop software with ML components is different from traditional software engineering (TSE). We call this new field software engineering for machine learning (SE4ML).

Based on a study of the academic and “grey” literature (the latter comprises blog articles, presentation slides and white papers), we identified best practices for SE4ML, and composed a recommended reading list on the topic. These practices include the use of AutoML techniques to improve the use of ML components in software. All practices are described in our practice catalogue. We encourage readers to have a look and to send us their suggestions for additions.

Figure 1: Engineering best practices for machine learning.

Having a solid overview of the best practices, we set up a survey to find out the extent to which these SE4ML practices have already been adopted. We asked teams working on software with machine learning components to which degree they followed each practice in their work. In the resulting data we saw that tech companies have the highest adoption of the new ML related best practices, while research labs have the highest adoption of AutoML. Overall, more practices are adopted by teams that are larger and more experienced, with practices originating from TSE seeing slightly lower adoption than the new ML specific practices.

Effects of best practice adoption

Besides the adoption of best practices, we also investigated the effects of the practices. This resulted in insights into which specific practices relate to which desired outcomes. For instance, we found that software quality is influenced most by continuous integration, automated regression tests, and static analysis. On the other hand, agility is primarily affected by automated model deployment, having a cohesive- multi-disciplinary team, and enabling parallel training experiments. With these insights into the effectiveness of different practices, we hope to increase practice adoption and improve the quality of software with ML components. 

A brief overview of additional survey results can be found in our piece titled “The 2020 State of Engineering Practices for Machine Learning”, and in our technical publication “Adoption and Effects of Software Engineering Best Practices in Machine Learning”.

How about automated ML?

A recent line of research advancements in ML focuses on automated machine learning (AutoML), an area that aims to automate (parts of) the construction and use of ML pipelines to enable a wider audience to make effective and responsible use of ML, without needing to become an expert in the field. We took a closer look at our survey results and found that, compared to non-tech companies and governmental organisations, research labs and tech companies are ahead in the adoption of AutoML practices (see Figure 2).

Figure 2: AutoML adoption by organisation type.

While overall AutoML adoption is similar across continents, non-profit and low-tech organisations see higher adoption in Europe than in North America. We also found that teams with multiple years of experience are more likely to adopt AutoML techniques. Finally, across the board, there is significant room to increase adoption of AutoML, but this is especially true for non-tech companies and governmental organisations.

More detailed information on our findings regarding AutoML adoption can be found in our piece titled “The 2020 State of AutoML Adoption”.

Looking forward

Based on feedback from and interviews with participants, we recently revised our survey to learn more. Specifically, in our latest version of the survey, we ask in more detail about responsible use of ML and about the adoption of different AutoML techniques such as automated feature selection and neural architecture search. You can help us by taking the 10-minute survey and by spreading the word. If you want to stay up to date with our progress, keep an eye on our website.

Sponsored Post Learn from the experts: Create a successful blog with our brand new courseThe WordPress.com Blog

Are you new to blogging, and do you want step-by-step guidance on how to publish and grow your blog? Learn more about our new Blogging for Beginners course and get 50% off through December 10th.

WordPress.com is excited to announce our newest offering: a course just for beginning bloggers where you’ll learn everything you need to know about blogging from the most trusted experts in the industry. We have helped millions of blogs get up and running, we know what works, and we want you to to know everything we know. This course provides all the fundamental skills and inspiration you need to get your blog started, an interactive community forum, and content updated annually.

PhD Student Yasha Pushak and Professor Holger Hoos Win Second Award for Ground-Breaking Research on Automated Algorithm Configuration

Just two years after receiving the best paper award at the 15th International Conference on Parallel Problem Solving from Nature (PPSN 2018), PhD student and Vanier Scholar Yasha Pushak and Professor Holger Hoos received another best paper award at the 22nd international Genetic and Evolutionary Computation Conference (GECCO 2020) for a line of research on automated algorithm configuration. Their second paper, titled “Golden Parameter Search: Exploiting Structure to Quickly Configure Parameters in Parallel” was the single winner of the ECOM Track at GECCO 2020, determined by voting of the audience.

Important and challenging computation problems are typically solved through the use of highly parameterized algorithms. Finding high-quality parameter settings for the algorithm can often improve their performance by several orders of magnitude. In the last two decades, automated algorithm configuration has emerged as a hot topic of research.

State of the art algorithm configurators rely on powerful search heuristics to explore the space of parameter configurations; however, they are typically quite costly, often requiring days or even weeks to find high-quality parameter configurations. These methods assume that finding high-quality parameter configurations is challenging. Consider, for example, the problem of trying to climb to the highest point of a rugged mountain with a large number of peaks in a dense fog: If you always climb up the steepest hill, you will likely get stuck at a peak that is far from the highest on the mountain. To avoid this, most configurators spend time exploring the entire “landscapes” of the algorithm configuration problem — including regions that don’t initially appear promising, just to be certain they haven’t missed anything. 

In their 2018 PPSN award-winning paper, “Algorithm Configuration Landscapes: More Benign than Expected?” Yasha Pushak and Holger Hoos hypothesized that the landscapes induced by algorithm configuration problems are likely much simpler and more structured than previously assumed. By way of analogy, consider the task of choosing a temperature at which to bake a cake: If the temperature is too high the cake will burn, but if it is too low the cake won’t bake. Therefore, finding the optimal temperature simply corresponds to trading off between “too high” and “too low”. Yasha Pushak and Holger Hoos developed a statistical procedure to test for similar structure in the response of algorithm parameters. They applied their procedure to a large set of parameters and were unable to reject their hypotheses in 99.5% of the cases they studied. Read more about algorithm configuration landscapes here.

Capitalizing on these insights, Yasha Pushak and Holger Hoos developed a new automated algorithm configuration procedure: Golden Parameter Search (GPS). GPS is the first of a new generation of automated algorithm configuration procedures designed to exploit structural properties of algorithm configuration landscapes. In their 2020 GECCO award-winning paper, they showed that GPS was often capable of finding similar-or better-quality parameter configurations using a fraction of the computational budget required by several other long-standing, state-of-the-art algorithm configuration procedures. Read more about GPS here.

Laurens Arp presents his work titled “Dynamic macro-scale traffic flow optimisation using crowd-sourced urban movement data” at IEEEMDM 2020

Traffic congestion tends to be bad for the environment, the economy, and above all: the drivers’ moods. As such, it is a worthwhile cause to pursue improvements for; in particular, being computer- and data scientists, using data-driven methods to try to alleviate this problem seemed a particularly exciting approach. This is just what we (Laurens Arp, Dyon van Vreumingen, Daniela Gahwens and Mitra Baratchi) published a paper for in the IEEE MDM 2020 proceedings. The project, which started as a course project for Mitra’s Urban Computing course at Leiden University, evolved first into a short paper submission to the Netmob Future Cities Challenge (FCC), and subsequently into a full paper submission to the MDM conference.

The main idea of the method proposed was to redistribute traffic by imposing external costs (or rewards) to specific road segments. We used a movement dataset for Tokyo, provided by Foursquare for the FCC, from which we could derive which proportion of drivers would want to go to and from specific parts of the city. By combining this with a macro-scale traffic flow model (Greenshields), we were able to compute the occupancy of specific roads and the total travel time this resulted in. This model could then be used as an objective function for black-box optimisation; we would optimise the cost parameters of road segments so that the optimal number of desired routes got redirected to alternative roads such that the overall traffic time was minimised.

The amount of improvement (under the Greenshields model) we were able to achieve was highly dependent on the number of cars we estimated would be on the road at the same time. The best improvement achieved was 13.15% for a little over 13 500 cars (925 hours), and the worst was 1.35% for just under 113 000 cars (437 hours). Interestingly, even the relatively modest improvements, occurring for large amounts of cars, could still be meaningful, because there are more drivers to benefit from them. We also added a few fairness analyses to the paper, the results of which did not seem to indicate any unfair disadvantage to individual drivers. You can find the pre-recorded conference presentation here.

We hope that our paper will be able to contribute to the perpetually on-going efforts worldwide of causing less damage to the environment, boosting the economy, and perhaps also helping some drivers’ moods.

New PhD student joining the ADA research group

The ADA research group welcomes Matthias König as a new PhD student.

Before joining ADA, Matthias wrote his master thesis on automated age estimation from unconstrained facial imagery under the supervision of Holger Hoos and Jan van Rijn, while doing an internship at PwC’s Data Analytics unit. He holds a master’s degree in Media Technology from Leiden University and, next to that, followed the Information Studies/Data Science master’s course at the University of Amsterdam.

Matthias’ research is concerned with detecting when (Auto-)AI systems are “out of their depth” and developing mechanisms to fill potential gaps in the training space of these systems. Broadly speaking, he is interested in making AI systems more reliable by finding ways to verify their predictions – especially in settings where ground truth is inaccessible. Hopefully, his work will make AI-based tools safer to use for both experts and non-experts and reduce uncertainty in the predictions made by AI systems.

A fairly long time ago in an office actually rather close by…

Koen van der Blom joined the ADA research group as a post-doctoral researcher. From March 2019 onward he started working with Holger Hoos in the area of meta-algorithmics. One of the things he works on is a tool called Sparkle. Sparkle aims to make meta-algorithmics such as algorithm selection and configuration easier to use for a wide audience. Besides this, he is also interested in performance analysis and prediction. What can be said about the expected performance of a new instance, based on previously seen instances? And how can you compare performance in a fair way?

Before this, during his PhD, Koen worked on multi-objective mixed-integer evolutionary algorithms applied to early-stage building design under the supervision of Michael Emmerich, Hèrm Hofmeyer, and Thomas Bäck. He continues to be interested in these problems, and particularly when it comes to optimisation in mixed-integer spaces. Who knows, perhaps combining aspects from the old and new will yet lead to other exciting work.

Laurens Arp has joined the ADA research group to perform his Master thesis project

How could AI techniques be used to best improve the living standards of people around the world? This is the main question of interest for Laurens Arp, a Master student who joined the ADA research group in November 2019. He is currently working on his Master Thesis under the supervision of Mitra Baratchi and Holger Hoos. The project is about the data-driven evaluation and optimization methods of geographical regions.

The main focus of the project will be on (spatial) representation learning. Current methods would not sufficiently address the problem yet, as most approaches will either focus too much on spatial structure instead of how this structure affects the features of a neighborhood, or are aimed too much at encoding similarity rather than interaction. Once a suitable representation has been found, machine learning and deep learning could be used to automatically learn the relationship between geographical features and the measures one might like to use to evaluate a region. If the resulting model is sufficiently accurate, it could then be used to rate the quality of region configurations generated by an optimization algorithm, allowing for the optimization of the development of the region.

20190802_212429

The ADA Research Group welcomes a new visitor!

Zhou joined the ADA Research Group in September 2019 as a visiting PhD student for a period of one year. He has started his PhD in March 2017 under the supervision of dr. Gangquan Si at School of Electrical Engineering, Xi’an Jiaotong University.

In his research he focuses on time series data mining techniques, including pre-processing, representation, classification, and prediction. His work has been successfully applied to a project titled “Research of Data Mining Technique and Development of Intelligent Data Management System for Electrical Equipment”, supported by China Southern Power Grid. In this project, he tries to make full use of massive data collected from various tests and online monitoring systems, aimed at providing accurate evaluation and prediction of electrical equipment status.

Zhou is currently working on online time series segmentation with the purpose of dimension reduction, especially on how to apply Automated Machine Learning methods for this task. During his visit, he will be closely supervised by dr. Mitra Baratchi and prof. Holger Hoos.

Image from iOS

Double win at ICT.OPEN 2019: Anna Louise and Can take home pitch prize and poster prize

This year, the ADA group was well represented at ICT.OPEN, the conference for ICT professionals in The Netherlands, which took place on Tuesday 19 and Wednesday 20 March.

Cooperation is key

Prof. Holger H. Hoos gave the keynote lecture on Tuesday morning, talking about how machine learning is transforming the way we do science. He argued that, while competition is important, cooperation is what ultimately makes us conquer the problems of our times.

Impression of Prof. Dr. Holger Hoos’s keynote lecture by Flatland.

Bob the builder

That evening Anna Louise Latour won the elevator pitch competition. In the final round, four PhD students competed to explain their research in under three minutes to a jury and the audience, using only a single prop to illustrate their story.

Anna Louise described how make optimal decisions under constraints and uncertainty, using an optimisation version of the Powergrid Reliability Problem as an example. During her pitch, she used a yellow Bob-the-builder helmet to distinguish between the user with the problem, and the Computer Scientist with the solution.

She impressed jury and audience with the clarity and content of her pitch, and with her charisma, ranking highest in the jury report and receiving three times as many audience votes as the runner-up.

Predictive power

The next day, we had another win: Can Wang was awarded second prize in the Commit2Data poster competition. Her poster explained how Automated Machine Learning can be used to predict the energy consumption of a household. “Pitching my poster to the jury was a bit out of my comfort zone, but I’m very happy that I’ve won!”, Can said afterwards.

Missed it?

For those of you who missed ICT.OPEN this year: make sure to join next year! In the mean time, enjoy NWO’s video of this year’s event:

Cooperation is key, revisited

Of course this outcome was also a team effort. ADA group members, as well as other people from LIACS, helped Can and Anna Louise by providing feedback on their posters and presentations, or being there to cheer.  Therefore, we celebrated as a team, with cakes 🙂

New master student joining ADA

Daniël Fokkinga joined the ADA research group as a Master’s student in September of 2018. His research is jointly supervised by Anna Louise Latour and Marie Anastacio and applies Automated Algorithm Configuration (AAC) on a pipeline for solving Stochastic Constraint Optimisation Problems (SCOPs).

An example of such a SCOP is the viral marketing problem. We are given a social network where an edge between two nodes represents a probabilistic communication relationship between two people in the network. We want to launch a new product by handing out a limited number of free samples to people in the network. We hope that they will like our product and spread the word to turn their acquaintances into buyers of our product. To whom in the network do we give the free samples, in order to maximise the expected number of people that will buy our product after they’ve heard about it from others? Who are the most influential people in our network?

Recently, a new pipeline for solving SCOPs was proposed. While it has shown its merit in a proof of principle, the different steps in the pipeline have not been optimised yet. While alternative design choices for elements in the pipeline have been proposed, their influence on the performance of the pipeline over a wide range of problems has not been extensively explored.

Daniël’s focus is on exposing these alternatives as parameters to optimise the pipeline for different applications, using Automated Algorithm Configuration.

Automated Algorithm Configuration allows a user to leave the tuning of parameters and thus the decision of the best approach to follow for parts of the full pipeline to a computer. Given a set of example instances, a configurator finds the combination of parameter values that is most likely to perform well on a new instance.

With this research Daniël hopes to show another successful application of Algorithm Configuration in a new field and improve the performance of previously developed algorithms.

BUSP6002

New guest researcher

Ada research group welcomes another guest researcher, Yi Chu. She will stay with Ada research group for a year.

Yi is a PhD candidate at the Institute of Computing Technology, Chinese Academy of Sciences. She received her master’s degree in computer technology from Beijing University on Posts and Telecommunications in 2014.
Yi’s research interests include heuristic algorithms for NP-hard problems and automatic algorithm design using optimization and learning techniques. Currently, she is working on using programming by optimization to improve the performance of heuristic algorithms for solving the maximum clique problem.Yi Chu-1709