Real-life applications, heavily relying on machine learning, such as dialog
systems, demand out-of-domain detection methods. Intent classification models
should be equipped with a mechanism to distinguish seen intents from unseen
ones so that the dialog agent is capable of rejecting the latter and avoiding
undesired behavior. However, despite increasing attention paid to the task, the
best practices for out-of-domain intent detection have not yet been fully
established.
This paper conducts a thorough comparison of out-of-domain intent detection
methods. We prioritize the methods, not requiring access to out-of-domain data
during training, gathering of which is extremely time- and labor-consuming due
to lexical and stylistic variation of user utterances. We evaluate multiple
contextual encoders and methods, proven to be efficient, on three standard
datasets for intent classification, expanded with out-of-domain utterances. Our
main findings show that fine-tuning Transformer-based encoders on in-domain
data leads to superior results. Mahalanobis distance, together with utterance
representations, derived from Transformer-based encoders, outperforms other
methods by a wide margin and establishes new state-of-the-art results for all
datasets.
The broader analysis shows that the reason for success lies in the fact that
the fine-tuned Transformer is capable of constructing homogeneous
representations of in-domain utterances, revealing geometrical disparity to out
of domain utterances. In turn, the Mahalanobis distance captures this disparity
easily.Comment: to appear in AAAI 202
