Representation learning has been increasing its impact on the research and
practice of machine learning, since it enables to learn representations that
can apply to various downstream tasks efficiently. However, recent works pay
little attention to the fact that real-world datasets used during the stage of
representation learning are commonly contaminated by noise, which can degrade
the quality of learned representations. This paper tackles the problem to learn
robust representations against noise in a raw dataset. To this end, inspired by
recent works on denoising and the success of the cosine-similarity-based
objective functions in representation learning, we propose the denoising
Cosine-Similarity (dCS) loss. The dCS loss is a modified cosine-similarity loss
and incorporates a denoising property, which is supported by both our
theoretical and empirical findings. To make the dCS loss implementable, we also
construct the estimators of the dCS loss with statistical guarantees. Finally,
we empirically show the efficiency of the dCS loss over the baseline objective
functions in vision and speech domains