152,510 research outputs found
EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting
Deep learning inspired by differential equations is a recent research trend
and has marked the state of the art performance for many machine learning
tasks. Among them, time-series modeling with neural controlled differential
equations (NCDEs) is considered as a breakthrough. In many cases, NCDE-based
models not only provide better accuracy than recurrent neural networks (RNNs)
but also make it possible to process irregular time-series. In this work, we
enhance NCDEs by redesigning their core part, i.e., generating a continuous
path from a discrete time-series input. NCDEs typically use interpolation
algorithms to convert discrete time-series samples to continuous paths.
However, we propose to i) generate another latent continuous path using an
encoder-decoder architecture, which corresponds to the interpolation process of
NCDEs, i.e., our neural network-based interpolation vs. the existing explicit
interpolation, and ii) exploit the generative characteristic of the decoder,
i.e., extrapolation beyond the time domain of original data if needed.
Therefore, our NCDE design can use both the interpolated and the extrapolated
information for downstream machine learning tasks. In our experiments with 5
real-world datasets and 12 baselines, our extrapolation and interpolation-based
NCDEs outperform existing baselines by non-trivial margins.Comment: main 8 page
On Lightweight Privacy-Preserving Collaborative Learning for IoT Objects
The Internet of Things (IoT) will be a main data generation infrastructure
for achieving better system intelligence. This paper considers the design and
implementation of a practical privacy-preserving collaborative learning scheme,
in which a curious learning coordinator trains a better machine learning model
based on the data samples contributed by a number of IoT objects, while the
confidentiality of the raw forms of the training data is protected against the
coordinator. Existing distributed machine learning and data encryption
approaches incur significant computation and communication overhead, rendering
them ill-suited for resource-constrained IoT objects. We study an approach that
applies independent Gaussian random projection at each IoT object to obfuscate
data and trains a deep neural network at the coordinator based on the projected
data from the IoT objects. This approach introduces light computation overhead
to the IoT objects and moves most workload to the coordinator that can have
sufficient computing resources. Although the independent projections performed
by the IoT objects address the potential collusion between the curious
coordinator and some compromised IoT objects, they significantly increase the
complexity of the projected data. In this paper, we leverage the superior
learning capability of deep learning in capturing sophisticated patterns to
maintain good learning performance. Extensive comparative evaluation shows that
this approach outperforms other lightweight approaches that apply additive
noisification for differential privacy and/or support vector machines for
learning in the applications with light data pattern complexities.Comment: 12 pages,IOTDI 201
Private Model Compression via Knowledge Distillation
The soaring demand for intelligent mobile applications calls for deploying
powerful deep neural networks (DNNs) on mobile devices. However, the
outstanding performance of DNNs notoriously relies on increasingly complex
models, which in turn is associated with an increase in computational expense
far surpassing mobile devices' capacity. What is worse, app service providers
need to collect and utilize a large volume of users' data, which contain
sensitive information, to build the sophisticated DNN models. Directly
deploying these models on public mobile devices presents prohibitive privacy
risk. To benefit from the on-device deep learning without the capacity and
privacy concerns, we design a private model compression framework RONA.
Following the knowledge distillation paradigm, we jointly use hint learning,
distillation learning, and self learning to train a compact and fast neural
network. The knowledge distilled from the cumbersome model is adaptively
bounded and carefully perturbed to enforce differential privacy. We further
propose an elegant query sample selection method to reduce the number of
queries and control the privacy loss. A series of empirical evaluations as well
as the implementation on an Android mobile device show that RONA can not only
compress cumbersome models efficiently but also provide a strong privacy
guarantee. For example, on SVHN, when a meaningful
-differential privacy is guaranteed, the compact model trained
by RONA can obtain 20 compression ratio and 19 speed-up with
merely 0.97% accuracy loss.Comment: Conference version accepted by AAAI'1
- …