The success of deep learning relies heavily on large and diverse datasets
with extensive labels, but we often only have access to several small datasets
associated with partial labels. In this paper, we start a new initiative,
"Label-Assemble", that aims to unleash the full potential of partially labeled
data from an assembly of public datasets. Specifically, we introduce a new
dynamic adapter to encode different visual tasks, which addresses the
challenges of incomparable, heterogeneous, or even conflicting labeling
protocols. We also employ pseudo-labeling and consistency constraints to
harness data with missing labels and to mitigate the domain gap across
datasets. From rigorous evaluations on three natural imaging and six medical
imaging tasks, we discover that learning from "negative examples" facilitates
both classification and segmentation of classes of interest. This sheds new
light on the computer-aided diagnosis of rare diseases and emerging pandemics,
wherein "positive examples" are hard to collect, yet "negative examples" are
relatively easier to assemble. Apart from exceeding prior arts in the ChestXray
benchmark, our model is particularly strong in identifying diseases of minority
classes, yielding over 3-point improvement on average. Remarkably, when using
existing partial labels, our model performance is on-par with that using full
labels, eliminating the need for an additional 40% of annotation costs. Code
will be made available at https://github.com/MrGiovanni/LabelAssemble