236 research outputs found
Convex relaxation of mixture regression with efficient algorithms
We develop a convex relaxation of maximum a posteriori estimation of a mixture of regression models. Although our relaxation involves a semidefinite matrix variable, we reformulate the problem to eliminate the need for general semidefinite programming. In particular, we provide two reformulations that admit fast algorithms. The first is a max-min spectral reformulation exploiting quasi-Newton descent. The second is a min-min reformulation consisting of fast alternating steps of closed-form updates. We evaluate the methods against Expectation-Maximization in a real problem of motion segmentation from video data
Automatically Score Tissue Images Like a Pathologist by Transfer Learning
Cancer is the second leading cause of death in the world. Diagnosing cancer
early on can save many lives. Pathologists have to look at tissue microarray
(TMA) images manually to identify tumors, which can be time-consuming,
inconsistent and subjective. Existing algorithms that automatically detect
tumors have either not achieved the accuracy level of a pathologist or require
substantial human involvements. A major challenge is that TMA images with
different shapes, sizes, and locations can have the same score. Learning
staining patterns in TMA images requires a huge number of images, which are
severely limited due to privacy concerns and regulations in medical
organizations. TMA images from different cancer types may have common
characteristics that could provide valuable information, but using them
directly harms the accuracy. By selective transfer learning from multiple small
auxiliary sets, the proposed algorithm is able to extract knowledge from tissue
images showing a ``similar" scoring pattern but with different cancer types.
Remarkably, transfer learning has made it possible for the algorithm to break
the critical accuracy barrier -- the proposed algorithm reports an accuracy of
75.9% on breast cancer TMA images from the Stanford Tissue Microarray Database,
achieving the 75\% accuracy level of pathologists. This will allow pathologists
to confidently use automatic algorithms to assist them in recognizing tumors
consistently with a higher accuracy in real time.Comment: 19 pages, 6 figure
Classification Tree Pruning Under Covariate Shift
We consider the problem of \emph{pruning} a classification tree, that is,
selecting a suitable subtree that balances bias and variance, in common
situations with inhomogeneous training data. Namely, assuming access to mostly
data from a distribution , but little data from a desired
distribution with different -marginals, we present the first
efficient procedure for optimal pruning in such situations, when
cross-validation and other penalized variants are grossly inadequate.
Optimality is derived with respect to a notion of \emph{average discrepancy}
(averaged over space) which significantly relaxes a
recent notion -- termed \emph{transfer-exponent} -- shown to tightly capture
the limits of classification under such a distribution shift. Our relaxed
notion can be viewed as a measure of \emph{relative dimension} between
distributions, as it relates to existing notions of information such as the
Minkowski and Renyi dimensions.Comment: 38 pages, 8 figure
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
We study linear regression under covariate shift, where the marginal
distribution over the input covariates differs in the source and the target
domains, while the conditional distribution of the output given the input
covariates is similar across the two domains. We investigate a transfer
learning approach with pretraining on the source data and finetuning based on
the target data (both conducted by online SGD) for this problem. We establish
sharp instance-dependent excess risk upper and lower bounds for this approach.
Our bounds suggest that for a large class of linear regression instances,
transfer learning with source data (and scarce or no target data) is
as effective as supervised learning with target data. In addition, we show
that finetuning, even with only a small amount of target data, could
drastically reduce the amount of source data required by pretraining. Our
theory sheds light on the effectiveness and limitation of pretraining as well
as the benefits of finetuning for tackling covariate shift problems.Comment: 32 pages, 1 figure, 1 tabl
Transfer Learning for Contextual Multi-armed Bandits
Motivated by a range of applications, we study in this paper the problem of
transfer learning for nonparametric contextual multi-armed bandits under the
covariate shift model, where we have data collected on source bandits before
the start of the target bandit learning. The minimax rate of convergence for
the cumulative regret is established and a novel transfer learning algorithm
that attains the minimax regret is proposed. The results quantify the
contribution of the data from the source domains for learning in the target
domain in the context of nonparametric contextual multi-armed bandits.
In view of the general impossibility of adaptation to unknown smoothness, we
develop a data-driven algorithm that achieves near-optimal statistical
guarantees (up to a logarithmic factor) while automatically adapting to the
unknown parameters over a large collection of parameter spaces under an
additional self-similarity assumption. A simulation study is carried out to
illustrate the benefits of utilizing the data from the auxiliary source domains
for learning in the target domain
- …