Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists

Language Models (LMs) are important components in several Natural Language Processing systems. Recurrent Neural Network LMs composed of LSTM units, especially those augmented with an external memory, have achieved state-of-the-art results. However, these models still struggle to process long sequences which are more likely to contain long-distance dependencies because of information fading and a bias towards more recent information. In this paper we demonstrate an effective mechanism for retrieving information in a memory augmented LSTM LM based on attending to information in memory in proportion to the number of timesteps the LSTM gating mechanism persisted the information

Automatic Detection of Vague Words and Sentences in Privacy Policies

Website privacy policies represent the single most important source of information for users to gauge how their personal data are collected, used and shared by companies. However, privacy policies are often vague and people struggle to understand the content. Their opaqueness poses a significant challenge to both users and policy regulators. In this paper, we seek to identify vague content in privacy policies. We construct the first corpus of human-annotated vague words and sentences and present empirical studies on automatic vagueness detection. In particular, we investigate context-aware and context-agnostic models for predicting vague words, and explore auxiliary-classifier generative adversarial networks for characterizing sentence vagueness. Our experimental results demonstrate the effectiveness of proposed approaches. Finally, we provide suggestions for resolving vagueness and improving the usability of privacy policies.Comment: 10 page

Классификация последовательностей на основе коротких мотивов

Sequence classification problems often arise in such areas as bioinformatics and natural language processing. In the last few year best results in this field were achieved by the deep learning methods, especially by architectures based on recurrent neural networks (RNN). However, the common problem of such models is a lack of interpretability, i.e., extraction of key features from data that affect the most the model’s decision. Meanwhile, using of less complicated neural network leads to decreasing predictive performance thus limiting usage of state-of-art machine learning methods in many subject areas. In this work we propose a novel interpretable deep learning architecture based on extraction of principal sets of short substrings — sequence motifs. The presence of extracted motif in the input sequence is a marker for a certain class. The key component of proposed solution is differential alignment algorithm developed by us, which provides a smooth analog of classical string comparison methods such as Levenshtein edit distance, and Smith–Waterman local alignment. Unlike previous works devoted to the motif based classification, which used CNN for shift-invariant searching, ours model provide a way to shift and gap invariant extraction of motifs.Задачи, связанные с классификацией последовательностей символов некоторого алфавита, часто возникают в таких областях, как биоинформатика и обработка естественного языка. Методы глубокого обучения, в особенности модели на основе рекуррентных нейронных сетей, в последние несколько лет зарекомендовали себя как наиболее эффективный способ решения подобных задач. Однако существующие подходы имеют серьезный недостаток — низкую интерпретируемость получаемых результатов. Крайне сложно установить какие именно свойства входной последовательности ответственны за её принадлежность к тому или иному классу. Упрощение же таких моделей с целью повышения их интерпретируемости, в свою очередь, приводит к снижению качества классификации. Такие недостатки ограничивают применение современных методов машинного обучения во многих предметных областях. В настоящей работе мы представляем принципиально новую, интерпретируемую архитектуру нейронных сетей, основанную на поиске набора коротких подпоследовательностей — мотивов, наличие которых влияет на принадлежность последовательности к определенному классу. Ключевой составляющей предлагаемого решения является разработанный нами алгоритм дифференцируемого выравнивания, являющийся дифференцируемым аналогом таких классических способов сравнения строк, как редакционное расстояние Левенштейна и алгоритм Смита–Ватермана. В отличие от предыдущих работ, посвященных классификации последовательностей на основе мотивов, новый метод позволяет не только выполнять поиск в произвольной части строки, но и учитывать возможные вставки

ATMSeer: Increasing Transparency and Controllability in Automated Machine Learning

To relieve the pain of manually selecting machine learning algorithms and tuning hyperparameters, automated machine learning (AutoML) methods have been developed to automatically search for good models. Due to the huge model search space, it is impossible to try all models. Users tend to distrust automatic results and increase the search budget as much as they can, thereby undermining the efficiency of AutoML. To address these issues, we design and implement ATMSeer, an interactive visualization tool that supports users in refining the search space of AutoML and analyzing the results. To guide the design of ATMSeer, we derive a workflow of using AutoML based on interviews with machine learning experts. A multi-granularity visualization is proposed to enable users to monitor the AutoML process, analyze the searched models, and refine the search space in real time. We demonstrate the utility and usability of ATMSeer through two case studies, expert interviews, and a user study with 13 end users.Comment: Published in the ACM Conference on Human Factors in Computing Systems (CHI), 2019, Glasgow, Scotland U