Background. Due to the widespread adoption of Artificial Intelligence (AI)
and Machine Learning (ML) for building software applications, companies are
struggling to recruit employees with a deep understanding of such technologies.
In this scenario, AutoML is soaring as a promising solution to fill the AI/ML
skills gap since it promises to automate the building of end-to-end AI/ML
pipelines that would normally be engineered by specialized team members. Aims.
Despite the growing interest and high expectations, there is a dearth of
information about the extent to which AutoML is currently adopted by teams
developing AI/ML-enabled systems and how it is perceived by practitioners and
researchers. Method. To fill these gaps, in this paper, we present a
mixed-method study comprising a benchmark of 12 end-to-end AutoML tools on two
SE datasets and a user survey with follow-up interviews to further our
understanding of AutoML adoption and perception. Results. We found that AutoML
solutions can generate models that outperform those trained and optimized by
researchers to perform classification tasks in the SE domain. Also, our
findings show that the currently available AutoML solutions do not live up to
their names as they do not equally support automation across the stages of the
ML development workflow and for all the team members. Conclusions. We derive
insights to inform the SE research community on how AutoML can facilitate their
activities and tool builders on how to design the next generation of AutoML
technologies