Deep learning has demonstrated unreasonable effectiveness on several high dimensional regression and classification problems, far exceeding theoretical expectations. In this thesis, we analyze this phenomena from the perspective of approximation theory. Utilizing recent theoretical advances, we investigate whether and under what conditions deep networks can escape the curse of dimensionality, providing experimental evidence where the theory falls short. We use these insights to suggest new approaches to network design that is more in accordance with this theory, and give several examples of where such designs succeed
