A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold

Although Deep Learning (DL) has achieved success in complex Artificial Intelligence (AI) tasks, it suffers from various notorious problems (e.g., feature redundancy, and vanishing or exploding gradients), since updating parameters in Euclidean space cannot fully exploit the geometric structure of the solution space. As a promising alternative solution, Riemannian-based DL uses geometric optimization to update parameters on Riemannian manifolds and can leverage the underlying geometric information. Accordingly, this article presents a comprehensive survey of applying geometric optimization in DL. At first, this article introduces the basic procedure of the geometric optimization, including various geometric optimizers and some concepts of Riemannian manifold. Subsequently, this article investigates the application of geometric optimization in different DL networks in various AI tasks, e.g., convolution neural network, recurrent neural network, transfer learning, and optimal transport. Additionally, typical public toolboxes that implement optimization on manifold are also discussed. Finally, this article makes a performance comparison between different deep geometric optimization methods under image recognition scenarios.Comment: 41 page

Nonparametric Estimation in Random Coefficients Binary Choice Models

This paper considers random coefficients binary choice models. The main goal is to estimate the density of the random coefficients nonparametrically. This is an ill-posed inverse problem characterized by an integral transform. A new density estimator for the random coefficients is developed, utilizing Fourier-Laplace series on spheres. This approach offers a clear insight on the identification problem. More importantly, it leads to a closed form estimator formula that yields a simple plug-in procedure requiring no numerical optimization. The new estimator, therefore, is easy to implement in empirical applications, while being flexible about the treatment of unobserved heterogeneity. Extensions including treatments of non-random coefficients and models with endogeneity are discussed.Inverse problems, Discrete choice models

Grassmannian flows and applications to nonlinear partial differential equations

We show how solutions to a large class of partial differential equations with nonlocal Riccati-type nonlinearities can be generated from the corresponding linearized equations, from arbitrary initial data. It is well known that evolutionary matrix Riccati equations can be generated by projecting linear evolutionary flows on a Stiefel manifold onto a coordinate chart of the underlying Grassmann manifold. Our method relies on extending this idea to the infinite dimensional case. The key is an integral equation analogous to the Marchenko equation in integrable systems, that represents the coodinate chart map. We show explicitly how to generate such solutions to scalar partial differential equations of arbitrary order with nonlocal quadratic nonlinearities using our approach. We provide numerical simulations that demonstrate the generation of solutions to Fisher--Kolmogorov--Petrovskii--Piskunov equations with nonlocal nonlinearities. We also indicate how the method might extend to more general classes of nonlinear partial differential systems.Comment: 26 pages, 2 figure

Real Analysis, Harmonic Analysis, and Applications

The workshop focused on important developments within the last few years in real and harmonic analysis, including polynomial partitioning and decoupling as well as significant concurrent progress in the application of these for example to number theory and partial differential equations