As the foundation of current natural language processing methods, pre-trained
language model has achieved excellent performance. However, the black-box
structure of the deep neural network in pre-trained language models seriously
limits the interpretability of the language modeling process. After revisiting
the coupled requirement of deep neural representation and semantics logic of
language modeling, a Word-Context-Coupled Space (W2CSpace) is proposed by
introducing the alignment processing between uninterpretable neural
representation and interpretable statistical logic. Moreover, a clustering
process is also designed to connect the word- and context-level semantics.
Specifically, an associative knowledge network (AKN), considered interpretable
statistical logic, is introduced in the alignment process for word-level
semantics. Furthermore, the context-relative distance is employed as the
semantic feature for the downstream classifier, which is greatly different from
the current uninterpretable semantic representations of pre-trained models. Our
experiments for performance evaluation and interpretable analysis are executed
on several types of datasets, including SIGHAN, Weibo, and ChnSenti. Wherein a
novel evaluation strategy for the interpretability of machine learning models
is first proposed. According to the experimental results, our language model
can achieve better performance and highly credible interpretable ability
compared to related state-of-the-art methods.Comment: Accepted at ACL 2023, Finding