Large language models (LMs) such as GPT-4 are very powerful and can process
different kinds of natural language processing (NLP) tasks. However, it can be
difficult to interpret the results due to the multi-layer nonlinear model
structure and millions of parameters. Lack of understanding of how the model
works can make the model unreliable and dangerous for everyday users in
real-world scenarios. Most recent works exploit the weights of attention to
provide explanations for model predictions. However, pure attention-based
explanation is unable to support the growing complexity of the models, and
cannot reason about their decision-making processes. Thus, we propose
LMExplainer, a knowledge-enhanced interpretation module for language models
that can provide human-understandable explanations. We use a knowledge graph
(KG) and a graph attention neural network to extract the key decision signals
of the LM. We further explore whether interpretation can also help AI
understand the task better. Our experimental results show that LMExplainer
outperforms existing LM+KG methods on CommonsenseQA and OpenBookQA. We also
compare the explanation results with generated explanation methods and
human-annotated results. The comparison shows our method can provide more
comprehensive and clearer explanations. LMExplainer demonstrates the potential
to enhance model performance and furnish explanations for the reasoning
processes of models in natural language