1,095 research outputs found
Duality for Neural Networks through Reproducing Kernel Banach Spaces
Reproducing Kernel Hilbert spaces (RKHS) have been a very successful tool in
various areas of machine learning. Recently, Barron spaces have been used to
prove bounds on the generalisation error for neural networks. Unfortunately,
Barron spaces cannot be understood in terms of RKHS due to the strong nonlinear
coupling of the weights. This can be solved by using the more general
Reproducing Kernel Banach spaces (RKBS). We show that these Barron spaces
belong to a class of integral RKBS. This class can also be understood as an
infinite union of RKHS spaces. Furthermore, we show that the dual space of such
RKBSs, is again an RKBS where the roles of the data and parameters are
interchanged, forming an adjoint pair of RKBSs including a reproducing kernel.
This allows us to construct the saddle point problem for neural networks, which
can be used in the whole field of primal-dual optimisation
Can TM system form an unconditional basis for Banach spaces?
The research on the algorithm of analytic signal has received much attention
for a long time. Takenaka-Malmquist (TM) system was introduced to consider
analytic functions in 1925. If TM system satisfies hyperbolic inseparability
condition, then it is an orthogonal basis. It can form unconditional basis for
Hilbert space and Schauder basis for Banach space
. In characterizing a function space, a
necessary condition is whether the basis is unconditional. But since the
introduction of TM systems in 1925, to the best of our knowledge, no one has
proved the existence of a TM system capable of forming an unconditional basis
for Banach space . TM system has a simple and
intuitive analytical structure. Hence it is applied also to the learning
algorithms and systematically developed to the reproducing kernel Hilbert
spaces (RKHS). Due to the lack of unconditional basis properties, it cannot be
extended to the reproducing kernel Banach spaces (RKBS) algorithm. But the case
of Banach space plays an important role in machine learning. In this paper, we
prove that two TM systems can form unconditional basis for
Weighted p-regular kernels for reproducing kernel Hilbert spaces and Mercer Theorem
[EN] Let (X, Sigma, mu) be a finite measure space and consider a Banach function space Y(mu). Motivated by some previous papers and current applications, we provide a general framework for representing reproducing kernel Hilbert spaces as subsets of Kothe Bochner (vectorvalued) function spaces. We analyze operator-valued kernels Gamma that define integration maps L-Gamma between Kothe-Bochner spaces of Hilbert-valued functions Y(mu; kappa). We show a reduction procedure which allows to find a factorization of the corresponding kernel operator through weighted Bochner spaces L-P(gd mu; kappa) and L-P (hd mu; kappa) - where 1/p + 1/p' = 1 - under the assumption of p-concavity of Y(mu). Equivalently, a new kernel obtained by multiplying Gamma by scalar functions can be given in such a way that the kernel operator is defined from L-P (mu; kappa) to L-P (mu; kappa) in a natural way. As an application, we prove a new version of Mercer Theorem for matrix-valued weighted kernels.The second author acknowledges the support of the Ministerio de Economia y Competitividad (Spain), under project MTM2014-53009-P (Spain).
The third author acknowledges the support of the Ministerio de Ciencia, Innovacion y Universidades (Spain), Agencia Estatal de Investigacion, and FEDER under project MTM2016-77054-C2-1-P (Spain).Agud Albesa, L.; Calabuig, JM.; Sánchez Pérez, EA. (2020). Weighted p-regular kernels for reproducing kernel Hilbert spaces and Mercer Theorem. Analysis and Applications. 18(3):359-383. https://doi.org/10.1142/S0219530519500179S359383183Agud, L., Calabuig, J. M., & Sánchez Pérez, E. A. (2011). The weak topology on q-convex Banach function spaces. Mathematische Nachrichten, 285(2-3), 136-149. doi:10.1002/mana.201000030CARMELI, C., DE VITO, E., & TOIGO, A. (2006). VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM. Analysis and Applications, 04(04), 377-408. doi:10.1142/s0219530506000838CARMELI, C., DE VITO, E., TOIGO, A., & UMANITÀ, V. (2010). VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES AND UNIVERSALITY. Analysis and Applications, 08(01), 19-61. doi:10.1142/s0219530510001503Cerdà , J., Hudzik, H., & Mastyło, M. (1996). Geometric properties of Köthe–Bochner spaces. Mathematical Proceedings of the Cambridge Philosophical Society, 120(3), 521-533. doi:10.1017/s0305004100075058Chavan, S., Podder, S., & Trivedi, S. (2018). Commutants and reflexivity of multiplication tuples on vector-valued reproducing kernel Hilbert spaces. Journal of Mathematical Analysis and Applications, 466(2), 1337-1358. doi:10.1016/j.jmaa.2018.06.062Christmann, A., Dumpert, F., & Xiang, D.-H. (2016). On extension theorems and their connection to universal consistency in machine learning. Analysis and Applications, 14(06), 795-808. doi:10.1142/s0219530516400029Defant, A. (2001). Positivity, 5(2), 153-175. doi:10.1023/a:1011466509838Defant, A., & Sánchez Pérez, E. A. (2004). Maurey–Rosenthal factorization of positive operators and convexity. Journal of Mathematical Analysis and Applications, 297(2), 771-790. doi:10.1016/j.jmaa.2004.04.047De Vito, E., Umanità , V., & Villa, S. (2013). An extension of Mercer theorem to matrix-valued measurable kernels. Applied and Computational Harmonic Analysis, 34(3), 339-351. doi:10.1016/j.acha.2012.06.001Eigel, M., & Sturm, K. (2017). Reproducing kernel Hilbert spaces and variable metric algorithms in PDE-constrained shape optimization. Optimization Methods and Software, 33(2), 268-296. doi:10.1080/10556788.2017.1314471Fasshauer, G. E., Hickernell, F. J., & Ye, Q. (2015). Solving support vector machines in reproducing kernel Banach spaces with positive definite functions. Applied and Computational Harmonic Analysis, 38(1), 115-139. doi:10.1016/j.acha.2014.03.007Galdames Bravo, O. (2014). Generalized Kӧthe -dual spaces. Bulletin of the Belgian Mathematical Society - Simon Stevin, 21(2). doi:10.36045/bbms/1400592625Lin, P.-K. (2004). Köthe-Bochner Function Spaces. doi:10.1007/978-0-8176-8188-3Lindenstrauss, J., & Tzafriri, L. (1979). Classical Banach Spaces II. doi:10.1007/978-3-662-35347-9Meyer-Nieberg, P. (1991). Banach Lattices. Universitext. doi:10.1007/978-3-642-76724-1Okada, S., Ricker, W. J., & Sánchez Pérez, E. A. (2008). Optimal Domain and Integral Extension of Operators. doi:10.1007/978-3-7643-8648-1Zhang, H., & Zhang, J. (2013). Vector-valued reproducing kernel Banach spaces with applications to multi-task learning. Journal of Complexity, 29(2), 195-215. doi:10.1016/j.jco.2012.09.00
Solving Support Vector Machines in Reproducing Kernel Banach Spaces with Positive Definite Functions
In this paper we solve support vector machines in reproducing kernel Banach
spaces with reproducing kernels defined on nonsymmetric domains instead of the
traditional methods in reproducing kernel Hilbert spaces. Using the
orthogonality of semi-inner-products, we can obtain the explicit
representations of the dual (normalized-duality-mapping) elements of support
vector machine solutions. In addition, we can introduce the reproduction
property in a generalized native space by Fourier transform techniques such
that it becomes a reproducing kernel Banach space, which can be even embedded
into Sobolev spaces, and its reproducing kernel is set up by the related
positive definite function. The representations of the optimal solutions of
support vector machines (regularized empirical risks) in these reproducing
kernel Banach spaces are formulated explicitly in terms of positive definite
functions, and their finite numbers of coefficients can be computed by fixed
point iteration. We also give some typical examples of reproducing kernel
Banach spaces induced by Mat\'ern functions (Sobolev splines) so that their
support vector machine solutions are well computable as the classical
algorithms. Moreover, each of their reproducing bases includes information from
multiple training data points. The concept of reproducing kernel Banach spaces
offers us a new numerical tool for solving support vector machines.Comment: 26 page
- …