We consider the problem of deep semi-supervised classification, where label information is obtained in the form of pairwise constraints. Existing approaches to this problem begin with a clustering network and utilize custom loss functions to encourage the learned representations to conform to the obtained constraints. We present a novel framework that seamlessly integrates pairwise constrained clustering, semi-supervised classification, and supervised classification. This approach leverages advances in unsupervised learning by jointly training a Siamese network and autoencoder to learn a representation that is amenable for both clustering and classification. The resulting framework outperforms existing approaches on common image recognition datasets
