Replacing pooling functions in Convolutional Neural Networks by linear combinations of increasing functions

Traditionally, Convolutional Neural Networks make use of the maximum or arithmetic mean in order to reduce the features extracted by convolutional layers in a downsampling process known as pooling. However, there is no strong argument to settle upon one of the two functions and, in practice, this selection turns to be problem dependent. Further, both of these options ignore possible dependencies among the data. We believe that a combination of both of these functions, as well as of additional ones which may retain different information, can benefit the feature extraction process. In this work, we replace traditional pooling by several alternative functions. In particular, we consider linear combinations of order statistics and generalizations of the Sugeno integral, extending the latter’s domain to the whole real line and setting the theoretical base for their application. We present an alternative pooling layer based on this strategy which we name ‘‘CombPool’’ layer. We replace the pooling layers of three different architectures of increasing complexity by CombPool layers, and empirically prove over multiple datasets that linear combinations outperform traditional pooling functions in most cases. Further, combinations with either the Sugeno integral or one of its generalizations usually yield the best results, proving a strong candidate to apply in most architectures.Tracasa Instrumental (iTRACASA), SpainGobierno de Navarra-Departamento de Universidad, Innovacion y Transformacion Digital, SpainSpanish Ministry of Science, Spain PID2019-108392GB-I00Andalusian Excellence project, Spain PID2019-108392GB-I00Conselho Nacional de Desenvolvimento Cientifico e Tecnologico (CNPQ) PC095-096Fundacao de Amparo a Ciencia e Tecnologia do Estado do Rio Grande do Sul (FAPERGS) P18-FR-4961 301618/2019-4 19/2551-000 1279-

Generalizing max pooling via (a, b)-grouping functions for Convolutional Neural Networks

Financial support of Tracasa Instrumental (iTRACASA) and of the Gobierno de Navarra - Departamento de Universidad, Innovación y Transformación Digital, as well as that of the Spanish Ministry of Science (project PID2019-108392GB-I00 (AEI/10.13039/501100011033)) and the project PC095-096 FUSIPROD. T. Asmus and G.P. Dimuro are supported by the projects CNPq (301618/2019-4) and FAPERGS (19/2551-0001279-9). F. Herrera is supported by the Andalusian Excellence project P18-FR-4961. Z. Takáč is supported by grant VEGA 1/0267/21. Open access funding provided by Universidad Pública de NavarraDue to their high adaptability to varied settings and effective optimization algorithm, Convolutional Neural Networks (CNNs) have set the state-of-the-art on image processing jobs for the previous decade. CNNs work in a sequential fashion, alternating between extracting significant features from an input image and aggregating these features locally through “pooling” functions, in order to produce a more compact representation. Functions like the arithmetic mean or, more typically, the maximum are commonly used to perform this downsampling operation. Despite the fact that many studies have been devoted to the development of alternative pooling algorithms, in practice, “max-pooling” still equals or exceeds most of these possibilities, and has become the standard for CNN construction. In this paper we focus on the properties that make the maximum such an efficient solution in the context of CNN feature downsampling and propose its replacement by grouping functions, a family of functions that share those desirable properties. In order to adapt these functions to the context of CNNs, we present (a,b)-grouping functions, an extension of grouping functions to work with real valued data. We present different construction methods for (a,b)-grouping functions, and demonstrate their empirical applicability for replacing max-pooling by using them to replace the pooling function of many well-known CNN architectures, finding promising results.Andalusian Excellence P18-FR-4961Departamento de Universidad, Innovación y Transformación DigitalConselho Nacional de Desenvolvimento Científico e Tecnológico 301618/2019-4 CNPqFundação de Amparo à Pesquisa do Estado do Rio Grande do Sul 19/2551-0001279-9 FAPERGSMinisterio de Ciencia e Innovación AEI/10.13039/501100011033, PC095-096 FUSIPROD, PID2019-108392GB-I00 MICINNVedecká Grantová Agentúra MŠVVaŠ SR a SAV 1/0267/21 VEGAUniversidad Pública de Navarra UPN