Selective Attention for Context-aware Neural Machine Translation

Despite the progress made in sentence-level NMT, current systems still fall short at achieving fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentences as context and may not scale to entire documents. To this end, we propose a novel and scalable top-down approach to hierarchical attention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose single-level attention approaches based on sentence or word-level information in the context. The document-level context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depending on whether we use monolingual or bilingual context. Our experiments and evaluation on English-German datasets in different document MT settings show that our selective attention approach not only significantly outperforms context-agnostic baselines but also surpasses context-aware baselines in most cases.Comment: Accepted at NAACL-HLT 201

Environmental context influences visual attention to responsible drinking messages

Aims: Responsible drinking messages (RDMs) are used as a key tool to reduce alcohol-related harms. A common form of RDM is in a poster format displayed in places such as bars, bus stops and toilet cubicles. However, evidence for the effectiveness of RDMs remains limited. Moreover, it is not known how environmental contexts (e.g. the number of alcohol-related cues in the environment) impact how such RDMs are interacted with, nor how this in turn affects their efficacy. Methods: One hundred participants completed a pseudo taste preference task in either in a bar laboratory (alcohol cue rich environmental context) or a traditional laboratory. The walls of the laboratory displayed either RDM or control posters during this task and eye tracking was used to assess participant attention to the posters. Results: Participants looked at the RDM posters less in the bar laboratory where the environmental context is rich in alcohol cues compared to a traditional laboratory where alcohol cues are sparse. Neither poster type or environmental context affected the amount of 'alcohol' consumed and the amount of visual attention given to RDMs was unrelated to the amount of 'alcohol' consumed. Conclusions: These findings provide experimental evidence that RDMs do not influence drinking behaviour in the direction intended (reduced consumption in situ). In addition, locating RDMs in alcohol-cue rich environments may result in sub-optimal behavioural responses to the RDM materials (e.g. visual attention to content). To maximize the potential impact of RDMs, the optimal location for RDMs is in environments where pre-existing alcohol cues are sparse to non-existent. Short Summary: Responsible drinking messages (RDMs) aim to reduce alcohol consumption, however, the findings of this study show that they may not influence in situ consumption. These findings also suggest that the optimal location for RDMs is in environments with few or no other alcohol-related cues

First Steps Toward a Computational Theory of Autism

A computational model with three interacting components for context sensitive reinforcement learning, context processing and automation can autonomously learn a focus attention and a shift attention task. The performance of the model is similar to that of normal children, and when a single parameter is changed, the performance on the two tasks approaches that of autistic children

Attentive Convolution: Equipping CNNs with RNN-style Attention Mechanisms

In NLP, convolutional neural networks (CNNs) have benefited less than recurrent neural networks (RNNs) from attention mechanisms. We hypothesize that this is because the attention in CNNs has been mainly implemented as attentive pooling (i.e., it is applied to pooling) rather than as attentive convolution (i.e., it is integrated into convolution). Convolution is the differentiator of CNNs in that it can powerfully model the higher-level representation of a word by taking into account its local fixed-size context in the input text t^x. In this work, we propose an attentive convolution network, ATTCONV. It extends the context scope of the convolution operation, deriving higher-level features for a word not only from local context, but also information extracted from nonlocal context by the attention mechanism commonly used in RNNs. This nonlocal context can come (i) from parts of the input text t^x that are distant or (ii) from extra (i.e., external) contexts t^y. Experiments on sentence modeling with zero-context (sentiment analysis), single-context (textual entailment) and multiple-context (claim verification) demonstrate the effectiveness of ATTCONV in sentence representation learning with the incorporation of context. In particular, attentive convolution outperforms attentive pooling and is a strong competitor to popular attentive RNNs.Comment: Camera-ready for TACL. 16 page