Machine learning (ML) is used more and more in a wide range of applications, but there are not enough machine learning experts to properly support this growth. With automated machine learning (AutoML) the goal is to make ML easier to use. As a result, experts should be able to deploy more ML systems, and less expertise would be needed to work with AutoML than when working with ML directly.
AutoML adoption
For AutoML to have this effect, however, it first needs to be adopted in practice. To study adoption levels in practice, we held a survey among teams that engineer software with ML components. As it turns out, adoption is not actually very high. In the figure below, we can see that, depending on the AutoML technique, more that 20 or even 30% of the respondents did not adopt AutoML at all. In addition, another 50 to 60% do not adopt AutoML techniques completely.
What holds back adoption?
Although we were able to assess the adoption levels through the survey, it does not tell us why people do or do not adopt these AutoML techniques. To find out, we started holding interviews, and already gained some insights based on two initial interviews.
The people we talked to raised a number of usability concerns. Firstly, there is a perception that AutoML techniques are hard to use and will require a significant initial investment to learn how to use. Secondly, someone indicated that in practice AutoML systems sometimes fail to work on the used data, but do not always make clear what the problem is. Thirdly, a participant raised that it is difficult to predict a good run length for AutoML systems. Related to this, there were also concerns about the required computational resources to use AutoML.
Looking forward
Clearly, there is room to improve AutoML adoption, and there are still some issues to resolve that could help to increase adoption. To get a more crisp view of which issues are common, and which are more atypical we plan to hold more interviews. In addition to learning about common issues, we also hope to find out whether there are differences between organisation types. Government organisations dealing with citizen data may, for instance, be more hesitant to adopt AutoML for privacy or bias sensitive applications.
Paper
If you are interested in the details you can learn more by reading our paper “AutoML Adoption in ML Software“ which was published in the AutoML workshop at ICML 2021.
Authors: Koen van der Blom, Alex Serban, Holger Hoos, Joost Visser.
Link: https://ada.liacs.nl/papers/BloEtAl21.pdf
ADA Research Group 