1 research outputs found
METTLE: a METamorphic testing approach to assessing and validating unsupervised machine LEarning systems
Unsupervised machine learning is the training of an artificial intelligence
system using information that is neither classified nor labeled, with a view to
modeling the underlying structure or distribution in a dataset. Since
unsupervised machine learning systems are widely used in many real-world
applications, assessing the appropriateness of these systems and validating
their implementations with respect to individual users' requirements and
specific application scenarioscontexts are indisputably two important
tasks. Such assessment and validation tasks, however, are fairly challenging
due to the absence of a priori knowledge of the data. In view of this
challenge, we develop a amorphic esting approach to
assessing and validating unsupervised machine arning systems,
abbreviated as METTLE. Our approach provides a new way to unveil the (possibly
latent) characteristics of various machine learning systems, by explicitly
considering the specific expectations and requirements of these systems from
individual users' perspectives. To support METTLE, we have further formulated
11 generic metamorphic relations (MRs), covering users' generally expected
characteristics that should be possessed by machine learning systems. To
demonstrate the viability and effectiveness of METTLE we have performed an
experiment involving six commonly used clustering systems. Our experiment has
shown that, guided by user-defined MR-based adequacy criteria, end users are
able to assess, validate, and select appropriate clustering systems in
accordance with their own specific needs. Our investigation has also yielded
insightful understanding and interpretation of the behavior of the machine
learning systems from an end-user software engineering's perspective, rather
than a designer's or implementor's perspective, who normally adopts a
theoretical approach