Duality for Neural Networks through Reproducing Kernel Banach Spaces

Reproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in various areas of machine learning. Recently, Barron spaces have been used to prove bounds on the generalisation error for neural networks. Unfortunately, Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear coupling of the weights. This can be solved by using the more general Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces belong to a class of integral RKBS. This class can also be understood as an infinite union of RKHS spaces. Furthermore, we show that the dual space of such RKBSs, is again an RKBS where the roles of the data and parameters are interchanged, forming an adjoint pair of RKBSs including a reproducing kernel. This allows us to construct the saddle point problem for neural networks, which can be used in the whole field of primal-dual optimisation

Weighted p-regular kernels for reproducing kernel Hilbert spaces and Mercer Theorem

[EN] Let (X, Sigma, mu) be a finite measure space and consider a Banach function space Y(mu). Motivated by some previous papers and current applications, we provide a general framework for representing reproducing kernel Hilbert spaces as subsets of Kothe Bochner (vectorvalued) function spaces. We analyze operator-valued kernels Gamma that define integration maps L-Gamma between Kothe-Bochner spaces of Hilbert-valued functions Y(mu; kappa). We show a reduction procedure which allows to find a factorization of the corresponding kernel operator through weighted Bochner spaces L-P(gd mu; kappa) and L-P (hd mu; kappa) - where 1/p + 1/p' = 1 - under the assumption of p-concavity of Y(mu). Equivalently, a new kernel obtained by multiplying Gamma by scalar functions can be given in such a way that the kernel operator is defined from L-P (mu; kappa) to L-P (mu; kappa) in a natural way. As an application, we prove a new version of Mercer Theorem for matrix-valued weighted kernels.The second author acknowledges the support of the Ministerio de Economia y Competitividad (Spain), under project MTM2014-53009-P (Spain). The third author acknowledges the support of the Ministerio de Ciencia, Innovacion y Universidades (Spain), Agencia Estatal de Investigacion, and FEDER under project MTM2016-77054-C2-1-P (Spain).Agud Albesa, L.; Calabuig, JM.; Sánchez Pérez, EA. (2020). Weighted p-regular kernels for reproducing kernel Hilbert spaces and Mercer Theorem. Analysis and Applications. 18(3):359-383. https://doi.org/10.1142/S0219530519500179S359383183Agud, L., Calabuig, J. M., & Sánchez Pérez, E. A. (2011). The weak topology on q-convex Banach function spaces. Mathematische Nachrichten, 285(2-3), 136-149. doi:10.1002/mana.201000030CARMELI, C., DE VITO, E., & TOIGO, A. (2006). VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM. Analysis and Applications, 04(04), 377-408. doi:10.1142/s0219530506000838CARMELI, C., DE VITO, E., TOIGO, A., & UMANITÀ, V. (2010). VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES AND UNIVERSALITY. Analysis and Applications, 08(01), 19-61. doi:10.1142/s0219530510001503Cerdà, J., Hudzik, H., & Mastyło, M. (1996). Geometric properties of Köthe–Bochner spaces. Mathematical Proceedings of the Cambridge Philosophical Society, 120(3), 521-533. doi:10.1017/s0305004100075058Chavan, S., Podder, S., & Trivedi, S. (2018). Commutants and reflexivity of multiplication tuples on vector-valued reproducing kernel Hilbert spaces. Journal of Mathematical Analysis and Applications, 466(2), 1337-1358. doi:10.1016/j.jmaa.2018.06.062Christmann, A., Dumpert, F., & Xiang, D.-H. (2016). On extension theorems and their connection to universal consistency in machine learning. Analysis and Applications, 14(06), 795-808. doi:10.1142/s0219530516400029Defant, A. (2001). Positivity, 5(2), 153-175. doi:10.1023/a:1011466509838Defant, A., & Sánchez Pérez, E. A. (2004). Maurey–Rosenthal factorization of positive operators and convexity. Journal of Mathematical Analysis and Applications, 297(2), 771-790. doi:10.1016/j.jmaa.2004.04.047De Vito, E., Umanità, V., & Villa, S. (2013). An extension of Mercer theorem to matrix-valued measurable kernels. Applied and Computational Harmonic Analysis, 34(3), 339-351. doi:10.1016/j.acha.2012.06.001Eigel, M., & Sturm, K. (2017). Reproducing kernel Hilbert spaces and variable metric algorithms in PDE-constrained shape optimization. Optimization Methods and Software, 33(2), 268-296. doi:10.1080/10556788.2017.1314471Fasshauer, G. E., Hickernell, F. J., & Ye, Q. (2015). Solving support vector machines in reproducing kernel Banach spaces with positive definite functions. Applied and Computational Harmonic Analysis, 38(1), 115-139. doi:10.1016/j.acha.2014.03.007Galdames Bravo, O. (2014). Generalized Kӧthe $p$-dual spaces. Bulletin of the Belgian Mathematical Society - Simon Stevin, 21(2). doi:10.36045/bbms/1400592625Lin, P.-K. (2004). Köthe-Bochner Function Spaces. doi:10.1007/978-0-8176-8188-3Lindenstrauss, J., & Tzafriri, L. (1979). Classical Banach Spaces II. doi:10.1007/978-3-662-35347-9Meyer-Nieberg, P. (1991). Banach Lattices. Universitext. doi:10.1007/978-3-642-76724-1Okada, S., Ricker, W. J., & Sánchez Pérez, E. A. (2008). Optimal Domain and Integral Extension of Operators. doi:10.1007/978-3-7643-8648-1Zhang, H., & Zhang, J. (2013). Vector-valued reproducing kernel Banach spaces with applications to multi-task learning. Journal of Complexity, 29(2), 195-215. doi:10.1016/j.jco.2012.09.00

Solving Support Vector Machines in Reproducing Kernel Banach Spaces with Positive Definite Functions

In this paper we solve support vector machines in reproducing kernel Banach spaces with reproducing kernels defined on nonsymmetric domains instead of the traditional methods in reproducing kernel Hilbert spaces. Using the orthogonality of semi-inner-products, we can obtain the explicit representations of the dual (normalized-duality-mapping) elements of support vector machine solutions. In addition, we can introduce the reproduction property in a generalized native space by Fourier transform techniques such that it becomes a reproducing kernel Banach space, which can be even embedded into Sobolev spaces, and its reproducing kernel is set up by the related positive definite function. The representations of the optimal solutions of support vector machines (regularized empirical risks) in these reproducing kernel Banach spaces are formulated explicitly in terms of positive definite functions, and their finite numbers of coefficients can be computed by fixed point iteration. We also give some typical examples of reproducing kernel Banach spaces induced by Mat\'ern functions (Sobolev splines) so that their support vector machine solutions are well computable as the classical algorithms. Moreover, each of their reproducing bases includes information from multiple training data points. The concept of reproducing kernel Banach spaces offers us a new numerical tool for solving support vector machines.Comment: 26 page