3 research outputs found
Sparse multiresolution representations with adaptive kernels
Reproducing kernel Hilbert spaces (RKHSs) are key elements of many
non-parametric tools successfully used in signal processing, statistics, and
machine learning. In this work, we aim to address three issues of the classical
RKHS based techniques. First, they require the RKHS to be known a priori, which
is unrealistic in many applications. Furthermore, the choice of RKHS affects
the shape and smoothness of the solution, thus impacting its performance.
Second, RKHSs are ill-equipped to deal with heterogeneous degrees of
smoothness, i.e., with functions that are smooth in some parts of their domain
but vary rapidly in others. Finally, the computational complexity of evaluating
the solution of these methods grows with the number of data points, rendering
these techniques infeasible for many applications. Though kernel learning,
local kernel adaptation, and sparsity have been used to address these issues,
many of these approaches are computationally intensive or forgo optimality
guarantees. We tackle these problems by leveraging a novel integral
representation of functions in RKHSs that allows for arbitrary centers and
different kernels at each center. To address the complexity issues, we then
write the function estimation problem as a sparse functional program that
explicitly minimizes the support of the representation leading to low
complexity solutions. Despite their non-convexity and infinite dimensionality,
we show these problems can be solved exactly and efficiently by leveraging
duality, and we illustrate this new approach in simulated and real data
Optimally Compressed Nonparametric Online Learning
Batch training of machine learning models based on neural networks is now
well established, whereas to date streaming methods are largely based on linear
models. To go beyond linear in the online setting, nonparametric methods are of
interest due to their universality and ability to stably incorporate new
information via convexity or Bayes' Rule. Unfortunately, when used online,
nonparametric methods suffer a "curse of dimensionality" which precludes their
use: their complexity scales at least with the time index. We survey online
compression tools which bring their memory under control and attain approximate
convergence. The asymptotic bias depends on a compression parameter that trades
off memory and accuracy. Further, the applications to robotics, communications,
economics, and power are discussed, as well as extensions to multi-agent
systems
Federated Classification using Parsimonious Functions in Reproducing Kernel Hilbert Spaces
Federated learning forms a global model using data collected from a
federation agent. This type of learning has two main challenges: the agents
generally don't collect data over the same distribution, and the agents have
limited capabilities of storing and transmitting data. Therefore, it is
impractical for each agent to send the entire data over the network. Instead,
each agent must form a local model and decide what information is fundamental
to the learning problem, which will be sent to a central unit. The central unit
can then form the global model using only the information received from the
agents. We propose a method that tackles these challenges. First each agent
forms a local model using a low complexity reproducing kernel Hilbert space
representation. From the model the agents identify the fundamental samples
which are sent to the central unit. The fundamental samples are obtained by
solving the dual problem. The central unit then forms the global model. We show
that the solution of the federated learner converges to that of the centralized
learner asymptotically as the sample size increases. The performance of the
proposed algorithm is evaluated using experiments with both simulated data and
real data sets from an activity recognition task, for which the data is
collected from a wearable device. The experimentation results show that the
accuracy of our method converges to that of a centralized learner with
increasing sample size