2 research outputs found
Dual Control Memory Augmented Neural Networks for Treatment Recommendations
Machine-assisted treatment recommendations hold a promise to reduce physician
time and decision errors. We formulate the task as a sequence-to-sequence
prediction model that takes the entire time-ordered medical history as input,
and predicts a sequence of future clinical procedures and medications. It is
built on the premise that an effective treatment plan may have long-term
dependencies from previous medical history. We approach the problem by using a
memory-augmented neural network, in particular, by leveraging the recent
differentiable neural computer that consists of a neural controller and an
external memory module. But differing from the original model, we use dual
controllers, one for encoding the history followed by another for decoding the
treatment sequences. In the encoding phase, the memory is updated as new input
is read; at the end of this phase, the memory holds not only the medical
history but also the information about the current illness. During the decoding
phase, the memory is write-protected. The decoding controller generates a
treatment sequence, one treatment option at a time. The resulting dual
controller write-protected memory-augmented neural network is demonstrated on
the MIMIC-III dataset on two tasks: procedure prediction and medication
prescription. The results show improved performance over both traditional
bag-of-words and sequence-to-sequence methods.Comment: 12 pages, 6 figure
Memory and attention in deep learning
Intelligence necessitates memory. Without memory, humans fail to perform
various nontrivial tasks such as reading novels, playing games or solving
maths. As the ultimate goal of machine learning is to derive intelligent
systems that learn and act automatically just like human, memory construction
for machine is inevitable. Artificial neural networks model neurons and
synapses in the brain by interconnecting computational units via weights, which
is a typical class of machine learning algorithms that resembles memory
structure. Their descendants with more complicated modeling techniques (a.k.a
deep learning) have been successfully applied to many practical problems and
demonstrated the importance of memory in the learning process of machinery
systems. Recent progresses on modeling memory in deep learning have revolved
around external memory constructions, which are highly inspired by
computational Turing models and biological neuronal systems. Attention
mechanisms are derived to support acquisition and retention operations on the
external memory. Despite the lack of theoretical foundations, these approaches
have shown promises to help machinery systems reach a higher level of
intelligence. The aim of this thesis is to advance the understanding on memory
and attention in deep learning. Its contributions include: (i) presenting a
collection of taxonomies for memory, (ii) constructing new memory-augmented
neural networks (MANNs) that support multiple control and memory units, (iii)
introducing variability via memory in sequential generative models, (iv)
searching for optimal writing operations to maximise the memorisation capacity
in slot-based memory networks, and (v) simulating the Universal Turing Machine
via Neural Stored-program Memory-a new kind of external memory for neural
networks.Comment: PHD Thesi