Quantifying Attention Flow in Transformers

In the Transformer model, "self-attention" combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating from different tokens gets increasingly mixed. This makes attention weights unreliable as explanations probes. In this paper, we consider the problem of quantifying this flow of information through self-attention. We propose two methods for approximating the attention to input tokens given attention weights, attention rollout and attention flow, as post hoc methods when we use attention weights as the relative relevance of the input tokens. We show that these methods give complementary views on the flow of information, and compared to raw attention, both yield higher correlations with importance scores of input tokens obtained using an ablation method and input gradients

트랜스포머의 어텐션 스코어 조작에 관한 연구

학위논문(석사) -- 서울대학교대학원 : 데이터사이언스대학원 데이터사이언스학과, 2023. 2. 이재진.Although Korean has distinctly different features from English, attempts to find a new Transformer model that more closely matches Korean by reflecting them are insufficient. Among the characteristics of the Korean language, we pay special attention to the role of postpositions. Agglutinative languages have more freedom in word order than inflectional languages, such as English, thanks to the postpositions. This study is based on the hypothesis that the current Transformer is challenging to learn the postpositions sufficiently, which play a significant role in agglutinative languages such as Korean. In Korean, the postpositions are paired with the substantives, so paying more attention to the corresponding substantives seems reasonable compared to other tokens in the sentence. However, the current Transformer learning algorithm has many limitations in doing so. Accordingly, it is shown that the performance of the natural language understanding (NLU) task can be improved by deliberatively changing the attention scores between the postpositions and the substantives. In addition, it is hoped that this study will stimulate the research on new learning methods that reflect the characteristics of Korean.한국어는 영어와 분명히 다른 특성을 갖고 있지만 이를 Transformer에 반영하여 한국어에 보다 부합하는 새로운 모델을 찾는 시도는 그리 충분하지 않다. 본 연구에서는 한국어 특성 중에 특히 조사의 역할에 주목한다. 조사 덕분에 영어와 같은 굴절어에 비해 문장 내 단어 순서의 자유도가 높은 교착어라는 특성을 반영하여 Transformer의 attention score 계산 방법의 변경을 제안한다. 본 연구는 한국어와 같은 교착어에서 매우 중요한 역할을 하는 조사가 현재의 Transformer에서는 충분히 학습되기 어렵다는 가설에 바탕을 둔다. 한국어에서 조사는 해당 체언과 쌍으로 묶이므로 문장 내의 다른 token에 비해 해당 체언을 좀더 attention하는 것이 타당해 보이지만 현재의 Transformer 학습 방법으로는 한계가 많다는 의미이다. 이에 조사-체언 간의 attention score를 인위적으로 변화시킴으로써 NLU(Natural Language Understanding) 관련 자연어 처리 task의 성능을 높일 수 있음을 보인다. 아울러 한글 특성을 반영한 새로운 학습 방법에 관한 연구에 자극이 될 수 있기를 기대한다.Chapter 1. Introduction 1 Chapter 2. Related work ５ Chapter 3. Korean and Transformer 7 Chapter 4. Methodology ９ Chapter 5. Results and Analysis 15 Chapter 6. Future work 20 Chapter 7. Conclusion 21 Bibliography 22 Abstract in Korean 26석

Conceptual challenges for interpretable machine learning

As machine learning has gradually entered into ever more sectors of public and private life, there has been a growing demand for algorithmic explainability. How can we make the predictions of complex statistical models more intelligible to end users? A subdiscipline of computer science known as interpretable machine learning (IML) has emerged to address this urgent question. Numerous influential methods have been proposed, from local linear approximations to rule lists and counterfactuals. In this article, I highlight three conceptual challenges that are largely overlooked by authors in this area. I argue that the vast majority of IML algorithms are plagued by (1) ambiguity with respect to their true target; (2) a disregard for error rates and severe testing; and (3) an emphasis on product over process. Each point is developed at length, drawing on relevant debates in epistemology and philosophy of science. Examples and counterexamples from IML are considered, demonstrating how failure to acknowledge these problems can result in counterintuitive and potentially misleading explanations. Without greater care for the conceptual foundations of IML, future work in this area is doomed to repeat the same mistakes

From Explainable to lnterpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?

Deep learning (DL) has substantially enhanced natural language processing (NLP) in healthcare research. However, the increasing complexity of DL-based NLP necessitates transparent model interpretability, or at least explainability, for reliable decision-making. This work presents a thorough scoping review of explainable and interpretable DL in healthcare NLP. The term “eXplainable and Interpretable Artificial Intelligence” (XIAI) is introduced to distinguish XAI from IAI. Different models are further categorized based on their functionality (model-, input-, output-based) and scope (local, global). Our analysis shows that attention mechanisms are the most prevalent emerging IAI technique. The use of IAI is growing, distinguishing it from XAI. The major challenges identified are that most XIAI does not explore “global” modelling processes, the lack of best practices, and the lack of systematic evaluation and benchmarks. One important opportunity is to use attention mechanisms to enhance multi-modal XIAI for personalized medicine. Additionally, combining DL with causal logic holds promise. Our discussion encourages the integration of XIAI in Large Language Models (LLMs) and domain-specific smaller models. In conclusion, XIAI adoption in healthcare requires dedicated in-house expertise. Collaboration with domain experts, end-users, and policymakers can lead to ready-to-use XIAI methods across NLP and medical tasks. While challenges exist, XIAI techniques offer a valuable foundation for interpretable NLP algorithms in healthcare

Interpretable by Design: Learning Predictors by Composing Interpretable Queries

There is a growing concern about typically opaque decision-making with high-performance machine learning algorithms. Providing an explanation of the reasoning process in domain-specific terms can be crucial for adoption in risk-sensitive domains such as healthcare. We argue that machine learning algorithms should be interpretable by design and that the language in which these interpretations are expressed should be domain- and task-dependent. Consequently, we base our model's prediction on a family of user-defined and task-specific binary functions of the data, each having a clear interpretation to the end-user. We then minimize the expected number of queries needed for accurate prediction on any given input. As the solution is generally intractable, following prior work, we choose the queries sequentially based on information gain. However, in contrast to previous work, we need not assume the queries are conditionally independent. Instead, we leverage a stochastic generative model (VAE) and an MCMC algorithm (Unadjusted Langevin) to select the most informative query about the input based on previous query-answers. This enables the online determination of a query chain of whatever depth is required to resolve prediction ambiguities. Finally, experiments on vision and NLP tasks demonstrate the efficacy of our approach and its superiority over post-hoc explanations.Comment: 29 pages, 14 figures. Accepted as a Regular Paper in Transactions on Pattern Analysis and Machine Intelligenc