3,471 research outputs found
Learnware: Small Models Do Big
There are complaints about current machine learning techniques such as the
requirement of a huge amount of training data and proficient training skills,
the difficulty of continual learning, the risk of catastrophic forgetting, the
leaking of data privacy/proprietary, etc. Most research efforts have been
focusing on one of those concerned issues separately, paying less attention to
the fact that most issues are entangled in practice. The prevailing big model
paradigm, which has achieved impressive results in natural language processing
and computer vision applications, has not yet addressed those issues, whereas
becoming a serious source of carbon emissions. This article offers an overview
of the learnware paradigm, which attempts to enable users not need to build
machine learning models from scratch, with the hope of reusing small models to
do things even beyond their original purposes, where the key ingredient is the
specification which enables a trained model to be adequately identified to
reuse according to the requirement of future users who know nothing about the
model in advance
Improving Heterogeneous Model Reuse by Density Estimation
This paper studies multiparty learning, aiming to learn a model using the
private data of different participants. Model reuse is a promising solution for
multiparty learning, assuming that a local model has been trained for each
party. Considering the potential sample selection bias among different parties,
some heterogeneous model reuse approaches have been developed. However,
although pre-trained local classifiers are utilized in these approaches, the
characteristics of the local data are not well exploited. This motivates us to
estimate the density of local data and design an auxiliary model together with
the local classifiers for reuse. To address the scenarios where some local
models are not well pre-trained, we further design a multiparty cross-entropy
loss for calibration. Upon existing works, we address a challenging problem of
heterogeneous model reuse from a decision theory perspective and take advantage
of recent advances in density estimation. Experimental results on both
synthetic and benchmark data demonstrate the superiority of the proposed
method.Comment: 9 pages, 5 figues. Accepted by IJCAI 202
Beyond shared memory loop parallelism in the polyhedral model
2013 Spring.Includes bibliographical references.With the introduction of multi-core processors, motivated by power and energy concerns, parallel processing has become main-stream. Parallel programming is much more difficult due to its non-deterministic nature, and because of parallel programming bugs that arise from non-determinacy. One solution is automatic parallelization, where it is entirely up to the compiler to efficiently parallelize sequential programs. However, automatic parallelization is very difficult, and only a handful of successful techniques are available, even after decades of research. Automatic parallelization for distributed memory architectures is even more problematic in that it requires explicit handling of data partitioning and communication. Since data must be partitioned among multiple nodes that do not share memory, the original memory allocation of sequential programs cannot be directly used. One of the main contributions of this dissertation is the development of techniques for generating distributed memory parallel code with parametric tiling. Our approach builds on important contributions to the polyhedral model, a mathematical framework for reasoning about program transformations. We show that many affine control programs can be uniformized only with simple techniques. Being able to assume uniform dependences significantly simplifies distributed memory code generation, and also enables parametric tiling. Our approach implemented in the AlphaZ system, a system for prototyping analyses, transformations, and code generators in the polyhedral model. The key features of AlphaZ are memory re-allocation, and explicit representation of reductions. We evaluate our approach on a collection of polyhedral kernels from the PolyBench suite, and show that our approach scales as well as PLuTo, a state-of-the-art shared memory automatic parallelizer using the polyhedral model. Automatic parallelization is only one approach to dealing with the non-deterministic nature of parallel programming that leaves the difficulty entirely to the compiler. Another approach is to develop novel parallel programming languages. These languages, such as X10, aim to provide highly productive parallel programming environment by including parallelism into the language design. However, even in these languages, parallel bugs remain to be an important issue that hinders programmer productivity. Another contribution of this dissertation is to extend the array dataflow analysis to handle a subset of X10 programs. We apply the result of dataflow analysis to statically guarantee determinism. Providing static guarantees can significantly increase programmer productivity by catching questionable implementations at compile-time, or even while programming
- …