46 research outputs found
Online Embedding Compression for Text Classification using Low Rank Matrix Factorization
Deep learning models have become state of the art for natural language
processing (NLP) tasks, however deploying these models in production system
poses significant memory constraints. Existing compression methods are either
lossy or introduce significant latency. We propose a compression method that
leverages low rank matrix factorization during training,to compress the word
embedding layer which represents the size bottleneck for most NLP models. Our
models are trained, compressed and then further re-trained on the downstream
task to recover accuracy while maintaining the reduced size. Empirically, we
show that the proposed method can achieve 90% compression with minimal impact
in accuracy for sentence classification tasks, and outperforms alternative
methods like fixed-point quantization or offline word embedding compression. We
also analyze the inference time and storage space for our method through FLOP
calculations, showing that we can compress DNN models by a configurable ratio
and regain accuracy loss without introducing additional latency compared to
fixed point quantization. Finally, we introduce a novel learning rate schedule,
the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate
to outperform other popular adaptive learning rate algorithms on a sentence
classification benchmark.Comment: Accepted in Thirty-Third AAAI Conference on Artificial Intelligence
(AAAI 2019
Retrieve-and-Read: Multi-task Learning of Information Retrieval and Reading Comprehension
This study considers the task of machine reading at scale (MRS) wherein,
given a question, a system first performs the information retrieval (IR) task
of finding relevant passages in a knowledge source and then carries out the
reading comprehension (RC) task of extracting an answer span from the passages.
Previous MRS studies, in which the IR component was trained without considering
answer spans, struggled to accurately find a small number of relevant passages
from a large set of passages. In this paper, we propose a simple and effective
approach that incorporates the IR and RC tasks by using supervised multi-task
learning in order that the IR component can be trained by considering answer
spans. Experimental results on the standard benchmark, answering SQuAD
questions using the full Wikipedia as the knowledge source, showed that our
model achieved state-of-the-art performance. Moreover, we thoroughly evaluated
the individual contributions of our model components with our new Japanese
dataset and SQuAD. The results showed significant improvements in the IR task
and provided a new perspective on IR for RC: it is effective to teach which
part of the passage answers the question rather than to give only a relevance
score to the whole passage.Comment: 10 pages, 6 figure. Accepted as a full paper at CIKM 201
Chatbot development to assist patients in health care services
Dissertação de mestrado integrado em Engenharia InformáticaDados de alta qualidade sobre tratamentos médicos e de informação técnica tornaram-se
acessíveis, criando novas oportunidades de E-Saúde para a recuperação de um paciente.
A implementação da aprendizagem automática nestas soluções provou ser essencial e
eficaz na elaboração de aplicações para o utilizador para aliviar a sobrecarga do sector
de saúde. Atualmente, muitas interações com os utentes são realizadas via telefonemas
e mensagens de texto. Os agentes de conversação podem responder a estas questões,
fomentando uma rápida interação com os pacientes.
O objetivo fundamental desta dissertação é prestar apoio aos pacientes, fornecendo
uma fonte de informação fidedigna que lhes permita instruir-se e esclarecer dúvidas
sobre os procedimentos e repercussões dos seus problemas de saúde. Este propósito foi
concretizado não apenas através de uma plataforma Web intuitiva e acessível, composta
por perguntas frequentes, mas também integrando um agente de conversação inteligente
para responder a questões.
Para este fim, cientificamente, foi necessário conduzir a investigação, implementação
e viabilidade dos agentes de conversação no domínio fechado para os cuidados de
saúde. Constitui um importante contributo para a comunidade de desenvolvimento de
chatbots, na qual se reúnem as últimas inovações e descobertas, bem os desafios actuais
da aprendizagem automática, contribuindo para a consciencialização desta área.High-quality data on medical treatments and facility-level information has become
accessible, creating new eHealth opportunities for the recuperation of a patient. Machine
learning implementation in these solutions has been proven to be essential and effective
in building user-centred applications to relieves the burden on the healthcare sector.
Nowadays, many patient interactions are handled through healthcare services via phone
calls and text message exchange. Conversation agents can provide answers to these
queries, promoting fast patient interaction.
The underlying aim of this dissertation is to assist patients by providing a reliable
source of information to educate themselves and clarify any doubts about procedures
and implications of their health issue. This purpose was achieved not only through
an intuitive and accessible web platform, with frequently asked questions, but also by
integrating an intelligent chatting agent to answer questions.
To this end, scientifically, it was necessary to conduct the research, implementation
and feasibility of closed-domain conversation agents for healthcare. It is a valuable
input for the chatbot development community, which assembles the latest innovations
and findings, as well as the current challenges of machine learning, contributing to the
awareness of this field
Neural Representations of Concepts and Texts for Biomedical Information Retrieval
Information retrieval (IR) methods are an indispensable tool in the current landscape of exponentially increasing textual data, especially on the Web. A typical IR task involves fetching and ranking a set of documents (from a large corpus) in terms of relevance to a user\u27s query, which is often expressed as a short phrase. IR methods are the backbone of modern search engines where additional system-level aspects including fault tolerance, scale, user interfaces, and session maintenance are also addressed. In addition to fetching documents, modern search systems may also identify snippets within the documents that are potentially most relevant to the input query. Furthermore, current systems may also maintain preprocessed structured knowledge derived from textual data as so called knowledge graphs, so certain types of queries that are posed as questions can be parsed as such; a response can be an output of one or more named entities instead of a ranked list of documents (e.g., what diseases are associated with EGFR mutations? ). This refined setup is often termed as question answering (QA) in the IR and natural language processing (NLP) communities.
In biomedicine and healthcare, specialized corpora are often at play including research articles by scientists, clinical notes generated by healthcare professionals, consumer forums for specific conditions (e.g., cancer survivors network), and clinical trial protocols (e.g., www.clinicaltrials.gov). Biomedical IR is specialized given the types of queries and the variations in the texts are different from that of general Web documents. For example, scientific articles are more formal with longer sentences but clinical notes tend to have less grammatical conformity and are rife with abbreviations. There is also a mismatch between the vocabulary of consumers and the lingo of domain experts and professionals. Queries are also different and can range from simple phrases (e.g., COVID-19 symptoms ) to more complex implicitly fielded queries (e.g., chemotherapy regimens for stage IV lung cancer patients with ALK mutations ). Hence, developing methods for different configurations (corpus, query type, user type) needs more deliberate attention in biomedical IR.
Representations of documents and queries are at the core of IR methods and retrieval methodology involves coming up with these representations and matching queries with documents based on them. Traditional IR systems follow the approach of keyword based indexing of documents (the so called inverted index) and matching query phrases against the document index. It is not difficult to see that this keyword based matching ignores the semantics of texts (synonymy at the lexeme level and entailment at phrase/clause/sentence levels) and this has lead to dimensionality reduction methods such as latent semantic indexing that generally have scale-related concerns; such methods also do not address similarity at the sentence level. Since the resurgence of neural network methods in NLP, the IR field has also moved to incorporate advances in neural networks into current IR methods.
This dissertation presents four specific methodological efforts toward improving biomedical IR. Neural methods always begin with dense embeddings for words and concepts to overcome the limitations of one-hot encoding in traditional NLP/IR. In the first effort, we present a new neural pre-training approach to jointly learn word and concept embeddings for downstream use in applications. In the second study, we present a joint neural model for two essential subtasks of information extraction (IE): named entity recognition (NER) and entity normalization (EN). Our method detects biomedical concept phrases in texts and links them to the corresponding semantic types and entity codes. These first two studies provide essential tools to model textual representations as compositions of both surface forms (lexical units) and high level concepts with potential downstream use in QA. In the third effort, we present a document reranking model that can help surface documents that are likely to contain answers (e.g, factoids, lists) to a question in a QA task. The model is essentially a sentence matching neural network that learns the relevance of a candidate answer sentence to the given question parametrized with a bilinear map. In the fourth effort, we present another document reranking approach that is tailored for precision medicine use-cases. It combines neural query-document matching and faceted text summarization. The main distinction of this effort from previous efforts is to pivot from a query manipulation setup to transforming candidate documents into pseudo-queries via neural text summarization. Overall, our contributions constitute nontrivial advances in biomedical IR using neural representations of concepts and texts
Learning to Answer Semantic Queries over Code
During software development, developers need answers to queries about
semantic aspects of code. Even though extractive question-answering using
neural approaches has been studied widely in natural languages, the problem of
answering semantic queries over code using neural networks has not yet been
explored. This is mainly because there is no existing dataset with extractive
question and answer pairs over code involving complex concepts and long chains
of reasoning. We bridge this gap by building a new, curated dataset called
CodeQueries, and proposing a neural question-answering methodology over code.
We build upon state-of-the-art pre-trained models of code to predict answer
and supporting-fact spans. Given a query and code, only some of the code may be
relevant to answer the query. We first experiment under an ideal setting where
only the relevant code is given to the model and show that our models do well.
We then experiment under three pragmatic considerations: (1) scaling to
large-size code, (2) learning from a limited number of examples and (3)
robustness to minor syntax errors in code. Our results show that while a neural
model can be resilient to minor syntax errors in code, increasing size of code,
presence of code that is not relevant to the query, and reduced number of
training examples limit the model performance. We are releasing our data and
models to facilitate future work on the proposed problem of answering semantic
queries over code