5,389 research outputs found
High-performance Kernel Machines with Implicit Distributed Optimization and Randomization
In order to fully utilize "big data", it is often required to use "big
models". Such models tend to grow with the complexity and size of the training
data, and do not make strong parametric assumptions upfront on the nature of
the underlying statistical dependencies. Kernel methods fit this need well, as
they constitute a versatile and principled statistical methodology for solving
a wide range of non-parametric modelling problems. However, their high
computational costs (in storage and time) pose a significant barrier to their
widespread adoption in big data applications.
We propose an algorithmic framework and high-performance implementation for
massive-scale training of kernel-based statistical models, based on combining
two key technical ingredients: (i) distributed general purpose convex
optimization, and (ii) the use of randomization to improve the scalability of
kernel methods. Our approach is based on a block-splitting variant of the
Alternating Directions Method of Multipliers, carefully reconfigured to handle
very large random feature matrices, while exploiting hybrid parallelism
typically found in modern clusters of multicore machines. Our implementation
supports a variety of statistical learning tasks by enabling several loss
functions, regularization schemes, kernels, and layers of randomized
approximations for both dense and sparse datasets, in a highly extensible
framework. We evaluate the ability of our framework to learn models on data
from applications, and provide a comparison against existing sequential and
parallel libraries.Comment: Work presented at MMDS 2014 (June 2014) and JSM 201
From Common to Special: When Multi-Attribute Learning Meets Personalized Opinions
Visual attributes, which refer to human-labeled semantic annotations, have
gained increasing popularity in a wide range of real world applications.
Generally, the existing attribute learning methods fall into two categories:
one focuses on learning user-specific labels separately for different
attributes, while the other one focuses on learning crowd-sourced global labels
jointly for multiple attributes. However, both categories ignore the joint
effect of the two mentioned factors: the personal diversity with respect to the
global consensus; and the intrinsic correlation among multiple attributes. To
overcome this challenge, we propose a novel model to learn user-specific
predictors across multiple attributes. In our proposed model, the diversity of
personalized opinions and the intrinsic relationship among multiple attributes
are unified in a common-to-special manner. To this end, we adopt a
three-component decomposition. Specifically, our model integrates a common
cognition factor, an attribute-specific bias factor and a user-specific bias
factor. Meanwhile Lasso and group Lasso penalties are adopted to leverage
efficient feature selection. Furthermore, theoretical analysis is conducted to
show that our proposed method could reach reasonable performance. Eventually,
the empirical study carried out in this paper demonstrates the effectiveness of
our proposed method
- …