445 research outputs found
Large-Scale Educational Question Analysis with Partial Variational Auto-encoders
Online education platforms enable teachers to share a large number of
educational resources such as questions to form exercises and quizzes for
students. With large volumes of such crowd-sourced questions, quantifying the
properties of these questions in crowd-sourced online education platforms is of
great importance to enable both teachers and students to find high-quality and
suitable resources. In this work, we propose a framework for large-scale
question analysis. We utilize the state-of-the-art Bayesian deep learning
method, in particular partial variational auto-encoders, to analyze real-world
educational data. We also develop novel objectives to quantify question quality
and difficulty. We apply our proposed framework to a real-world cohort with
millions of question-answer pairs from an online education platform. Our
framework not only demonstrates promising results in terms of statistical
metrics but also obtains highly consistent results with domain expert
evaluation.Comment: 19 pages, 13 figure
Deep Learning based Recommender System: A Survey and New Perspectives
With the ever-growing volume of online information, recommender systems have
been an effective strategy to overcome such information overload. The utility
of recommender systems cannot be overstated, given its widespread adoption in
many web applications, along with its potential impact to ameliorate many
problems related to over-choice. In recent years, deep learning has garnered
considerable interest in many research fields such as computer vision and
natural language processing, owing not only to stellar performance but also the
attractive property of learning feature representations from scratch. The
influence of deep learning is also pervasive, recently demonstrating its
effectiveness when applied to information retrieval and recommender systems
research. Evidently, the field of deep learning in recommender system is
flourishing. This article aims to provide a comprehensive review of recent
research efforts on deep learning based recommender systems. More concretely,
we provide and devise a taxonomy of deep learning based recommendation models,
along with providing a comprehensive summary of the state-of-the-art. Finally,
we expand on current trends and provide new perspectives pertaining to this new
exciting development of the field.Comment: The paper has been accepted by ACM Computing Surveys.
https://doi.acm.org/10.1145/328502
Neural Graph Transfer Learning in Natural Language Processing Tasks
Natural language is essential in our daily lives as we rely on languages to communicate and exchange information. A fundamental goal for natural language processing (NLP) is to let the machine understand natural language to help or replace human experts to mine knowledge and complete tasks. Many NLP tasks deal with sequential data. For example, a sentence is considered as a sequence of works. Very recently, deep learning-based language models (i.e.,BERT \citep{devlin2018bert}) achieved significant improvement in many existing tasks, including text classification and natural language inference. However, not all tasks can be formulated using sequence models. Specifically, graph-structured data is also fundamental in NLP, including entity linking, entity classification, relation extraction, abstractive meaning representation, and knowledge graphs \citep{santoro2017simple,hamilton2017representation,kipf2016semi}. In this scenario, BERT-based pretrained models may not be suitable. Graph Convolutional Neural Network (GCN) \citep{kipf2016semi} is a deep neural network model designed for graphs. It has shown great potential in text classification, link prediction, question answering and so on. This dissertation presents novel graph models for NLP tasks, including text classification, prerequisite chain learning, and coreference resolution. We focus on different perspectives of graph convolutional network modeling: for text classification, a novel graph construction method is proposed which allows interpretability for the prediction; for prerequisite chain learning, we propose multiple aggregation functions that utilize neighbors for better information exchange; for coreference resolution, we study how graph pretraining can help when labeled data is limited. Moreover, an important branch is to apply pretrained language models for the mentioned tasks. So, this dissertation also focuses on the transfer learning method that generalizes pretrained models to other domains, including medical, cross-lingual, and web data. Finally, we propose a new task called unsupervised cross-domain prerequisite chain learning, and study novel graph-based methods to transfer knowledge over graphs
How to Do Machine Learning with Small Data? -- A Review from an Industrial Perspective
Artificial intelligence experienced a technological breakthrough in science,
industry, and everyday life in the recent few decades. The advancements can be
credited to the ever-increasing availability and miniaturization of
computational resources that resulted in exponential data growth. However,
because of the insufficient amount of data in some cases, employing machine
learning in solving complex tasks is not straightforward or even possible. As a
result, machine learning with small data experiences rising importance in data
science and application in several fields. The authors focus on interpreting
the general term of "small data" and their engineering and industrial
application role. They give a brief overview of the most important industrial
applications of machine learning and small data. Small data is defined in terms
of various characteristics compared to big data, and a machine learning
formalism was introduced. Five critical challenges of machine learning with
small data in industrial applications are presented: unlabeled data, imbalanced
data, missing data, insufficient data, and rare events. Based on those
definitions, an overview of the considerations in domain representation and
data acquisition is given along with a taxonomy of machine learning approaches
in the context of small data
Autoencoders for natural language semantics
Les auto-encodeurs sont des réseaux de neurones artificiels qui apprennent des représentations. Dans un auto-encodeur, l’encodeur transforme une entrée en une représentation, et le décodeur essaie de prédire l’entrée à partir de la représentation. Cette thèse compile trois applications de ces modèles au traitement automatique des langues : pour l’apprentissage de représentations de mots et de phrases, ainsi que pour mieux comprendre la compositionnalité.
Dans le premier article, nous montrons que nous pouvons auto-encoder des définitions
de dictionnaire et ainsi apprendre des vecteurs de définition. Nous proposons une nouvelle
pénalité qui nous permet d’utiliser ces vecteurs comme entrées à l’encodeur lui-même, mais
aussi de les mélanger des vecteurs distributionnels pré-entraînés. Ces vecteurs de définition
capturent mieux la similarité sémantique que les méthodes distributionnelles telles que
word2vec. De plus, l’encodeur généralise à un certain degré à des définitions qu’il n’a pas
vues pendant l’entraînement.
Dans le deuxième article, nous analysons les représentations apprises par les auto-encodeurs
variationnels séquence-à-séquence. Nous constatons que les encodeurs ont tendance à mémo-
riser les premiers mots et la longueur de la phrase d’entrée. Cela limite considérablement
leur utilité en tant que modèles génératifs contrôlables. Nous analysons aussi des variantes
architecturales plus simples qui ne tiennent pas compte de l’ordre des mots, ainsi que des mé-
thodes basées sur le pré-entraînement. Les représentations qu’elles apprennent ont tendance
à encoder plus nettement des caractéristiques globales telles que le sujet et le sentiment, et
cela se voit dans les reconstructions qu’ils produisent.
Dans le troisième article, nous utilisons des simulations d’émergence du langage pour
étudier la compositionnalité. Un locuteur – l’encodeur – observe une entrée et produit un
message. Un auditeur – le décodeur – tente de reconstituer ce dont le locuteur a parlé dans
son message. Nous émettons l’hypothèse que faire des phrases impliquant plusieurs entités,
telles que « Jean aime Marie », nécessite fondamentalement de percevoir chaque entité comme
un tout. Nous dotons certains agents de cette capacité grâce à un mechanisme d’attention,
alors que d’autres en sont privés. Nous proposons différentes métriques qui mesurent à quel
point les langues des agents sont naturelles en termes de structure d’argument, et si elles sont davantage analytiques ou synthétiques. Les agents percevant les entités comme des touts
échangent des messages plus naturels que les autres agents.Autoencoders are artificial neural networks that learn representations. In an autoencoder, the
encoder transforms an input into a representation, and the decoder tries to recover the input
from the representation. This thesis compiles three different applications of these models to
natural language processing: for learning word and sentence representations, as well as to
better understand compositionality.
In the first paper, we show that we can autoencode dictionary definitions to learn word
vectors, called definition embeddings. We propose a new penalty that allows us to use these
definition embeddings as inputs to the encoder itself, but also to blend them with pretrained
distributional vectors. The definition embeddings capture semantic similarity better than
distributional methods such as word2vec. Moreover, the encoder somewhat generalizes to
definitions unseen during training.
In the second paper, we analyze the representations learned by sequence-to-sequence
variational autoencoders. We find that the encoders tend to memorize the first few words
and the length of the input sentence. This limits drastically their usefulness as controllable
generative models. We also analyze simpler architectural variants that are agnostic to word
order, as well as pretraining-based methods. The representations that they learn tend to
encode global features such as topic and sentiment more markedly, and this shows in the
reconstructions they produce.
In the third paper, we use language emergence simulations to study compositionality. A
speaker – the encoder – observes an input and produces a message about it. A listener – the
decoder – tries to reconstruct what the speaker talked about in its message. We hypothesize
that producing sentences involving several entities, such as “John loves Mary”, fundamentally
requires to perceive each entity, John and Mary, as distinct wholes. We endow some agents
with this ability via an attention mechanism, and deprive others of it. We propose various
metrics to measure whether the languages are natural in terms of their argument structure,
and whether the languages are more analytic or synthetic. Agents perceiving entities as
distinct wholes exchange more natural messages than other agents
Scalable Population Synthesis with Deep Generative Modeling
Population synthesis is concerned with the generation of synthetic yet
realistic representations of populations. It is a fundamental problem in the
modeling of transport where the synthetic populations of micro-agents represent
a key input to most agent-based models. In this paper, a new methodological
framework for how to 'grow' pools of micro-agents is presented. The model
framework adopts a deep generative modeling approach from machine learning
based on a Variational Autoencoder (VAE). Compared to the previous population
synthesis approaches, including Iterative Proportional Fitting (IPF), Gibbs
sampling and traditional generative models such as Bayesian Networks or Hidden
Markov Models, the proposed method allows fitting the full joint distribution
for high dimensions. The proposed methodology is compared with a conventional
Gibbs sampler and a Bayesian Network by using a large-scale Danish trip diary.
It is shown that, while these two methods outperform the VAE in the
low-dimensional case, they both suffer from scalability issues when the number
of modeled attributes increases. It is also shown that the Gibbs sampler
essentially replicates the agents from the original sample when the required
conditional distributions are estimated as frequency tables. In contrast, the
VAE allows addressing the problem of sampling zeros by generating agents that
are virtually different from those in the original data but have similar
statistical properties. The presented approach can support agent-based modeling
at all levels by enabling richer synthetic populations with smaller zones and
more detailed individual characteristics.Comment: 27 pages, 15 figures, 4 table
Convolutional Neural Network Visualization for fMRI Brain Disease Classification Tasks
Neurologists face an increasingly overwhelming amount of data that they must use to determine diagnoses for patients with potential brain diseases. Our project aims to supplement the upcoming technology of automated brain disease classification using deep convolutional neural networks to develop a suite of visualization tools for these high dimensional deep convolutional neural networks. For this project we developed high quality visualizations for convolutional neural networks that classify fMRI brain scans. Visualizations such as those developed in this project could be used by doctors to identify which parts of the brain are indicators for mental conditions or by data scientists to understand more about how their networks work
Disentangled Representation Learning
Disentangled Representation Learning (DRL) aims to learn a model capable of
identifying and disentangling the underlying factors hidden in the observable
data in representation form. The process of separating underlying factors of
variation into variables with semantic meaning benefits in learning explainable
representations of data, which imitates the meaningful understanding process of
humans when observing an object or relation. As a general learning strategy,
DRL has demonstrated its power in improving the model explainability,
controlability, robustness, as well as generalization capacity in a wide range
of scenarios such as computer vision, natural language processing, data mining
etc. In this article, we comprehensively review DRL from various aspects
including motivations, definitions, methodologies, evaluations, applications
and model designs. We discuss works on DRL based on two well-recognized
definitions, i.e., Intuitive Definition and Group Theory Definition. We further
categorize the methodologies for DRL into four groups, i.e., Traditional
Statistical Approaches, Variational Auto-encoder Based Approaches, Generative
Adversarial Networks Based Approaches, Hierarchical Approaches and Other
Approaches. We also analyze principles to design different DRL models that may
benefit different tasks in practical applications. Finally, we point out
challenges in DRL as well as potential research directions deserving future
investigations. We believe this work may provide insights for promoting the DRL
research in the community.Comment: 22 pages,9 figure
- …