Replacing pooling functions in Convolutional Neural Networks by linear combinations of increasing functions

Traditionally, Convolutional Neural Networks make use of the maximum or arithmetic mean in order to reduce the features extracted by convolutional layers in a downsampling process known as pooling. However, there is no strong argument to settle upon one of the two functions and, in practice, this selection turns to be problem dependent. Further, both of these options ignore possible dependencies among the data. We believe that a combination of both of these functions, as well as of additional ones which may retain different information, can benefit the feature extraction process. In this work, we replace traditional pooling by several alternative functions. In particular, we consider linear combinations of order statistics and generalizations of the Sugeno integral, extending the latter’s domain to the whole real line and setting the theoretical base for their application. We present an alternative pooling layer based on this strategy which we name ‘‘CombPool’’ layer. We replace the pooling layers of three different architectures of increasing complexity by CombPool layers, and empirically prove over multiple datasets that linear combinations outperform traditional pooling functions in most cases. Further, combinations with either the Sugeno integral or one of its generalizations usually yield the best results, proving a strong candidate to apply in most architectures.Tracasa Instrumental (iTRACASA), SpainGobierno de Navarra-Departamento de Universidad, Innovacion y Transformacion Digital, SpainSpanish Ministry of Science, Spain PID2019-108392GB-I00Andalusian Excellence project, Spain PID2019-108392GB-I00Conselho Nacional de Desenvolvimento Cientifico e Tecnologico (CNPQ) PC095-096Fundacao de Amparo a Ciencia e Tecnologia do Estado do Rio Grande do Sul (FAPERGS) P18-FR-4961 301618/2019-4 19/2551-000 1279-

Efficient Data Driven Multi Source Fusion

Data/information fusion is an integral component of many existing and emerging applications; e.g., remote sensing, smart cars, Internet of Things (IoT), and Big Data, to name a few. While fusion aims to achieve better results than what any one individual input can provide, often the challenge is to determine the underlying mathematics for aggregation suitable for an application. In this dissertation, I focus on the following three aspects of aggregation: (i) efficient data-driven learning and optimization, (ii) extensions and new aggregation methods, and (iii) feature and decision level fusion for machine learning with applications to signal and image processing. The Choquet integral (ChI), a powerful nonlinear aggregation operator, is a parametric way (with respect to the fuzzy measure (FM)) to generate a wealth of aggregation operators. The FM has 2N variables and N(2N − 1) constraints for N inputs. As a result, learning the ChI parameters from data quickly becomes impractical for most applications. Herein, I propose a scalable learning procedure (which is linear with respect to training sample size) for the ChI that identifies and optimizes only data-supported variables. As such, the computational complexity of the learning algorithm is proportional to the complexity of the solver used. This method also includes an imputation framework to obtain scalar values for data-unsupported (aka missing) variables and a compression algorithm (lossy or losselss) of the learned variables. I also propose a genetic algorithm (GA) to optimize the ChI for non-convex, multi-modal, and/or analytical objective functions. This algorithm introduces two operators that automatically preserve the constraints; therefore there is no need to explicitly enforce the constraints as is required by traditional GA algorithms. In addition, this algorithm provides an efficient representation of the search space with the minimal set of vertices. Furthermore, I study different strategies for extending the fuzzy integral for missing data and I propose a GOAL programming framework to aggregate inputs from heterogeneous sources for the ChI learning. Last, my work in remote sensing involves visual clustering based band group selection and Lp-norm multiple kernel learning based feature level fusion in hyperspectral image processing to enhance pixel level classification

Feature and Decision Level Fusion Using Multiple Kernel Learning and Fuzzy Integrals

The work collected in this dissertation addresses the problem of data fusion. In other words, this is the problem of making decisions (also known as the problem of classification in the machine learning and statistics communities) when data from multiple sources are available, or when decisions/confidence levels from a panel of decision-makers are accessible. This problem has become increasingly important in recent years, especially with the ever-increasing popularity of autonomous systems outfitted with suites of sensors and the dawn of the ``age of big data.\u27\u27 While data fusion is a very broad topic, the work in this dissertation considers two very specific techniques: feature-level fusion and decision-level fusion. In general, the fusion methods proposed throughout this dissertation rely on kernel methods and fuzzy integrals. Both are very powerful tools, however, they also come with challenges, some of which are summarized below. I address these challenges in this dissertation. Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher-dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the pre-computed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. These challenges are tackled by the introduction of many new algorithms in the following chapters. I also address MKL\u27s storage and speed drawbacks, allowing MKL-based techniques to be applied to big data efficiently. Some algorithms in this work are based on the Choquet fuzzy integral, a powerful nonlinear aggregation operator parameterized by the fuzzy measure (FM). These decision-level fusion algorithms learn a fuzzy measure by minimizing a sum of squared error (SSE) criterion based on a set of training data. The flexibility of the Choquet integral comes with a cost, however---given a set of N decision makers, the size of the FM the algorithm must learn is 2N. This means that the training data must be diverse enough to include 2N independent observations, though this is rarely encountered in practice. I address this in the following chapters via many different regularization functions, a popular technique in machine learning and statistics used to prevent overfitting and increase model generalization. Finally, it is worth noting that the aggregation behavior of the Choquet integral is not intuitive. I tackle this by proposing a quantitative visualization strategy allowing the FM and Choquet integral behavior to be shown simultaneously

Development of a sustainable groundwater management strategy and sequential compliance monitoring to control saltwater intrusion in coastal aquifers

The coastal areas of the world are characterized by high population densities, an abundance of food, and increased economic activities. These increasing human settlements, subsequent increases in agricultural developments and economic activities demand an increasing amount quantity of freshwater supplies to different sectors. Groundwater in coastal aquifers is one of the most important sources of freshwater supplies. Over exploitation of this coastal groundwater resource results in seawater intrusion and subsequent deterioration of groundwater quality in coastal aquifers. In addition, climate change induced sea level rise, in combination with the effect of excessive groundwater extraction, can accelerate the seawater intrusion. Adequate supply of good quality water to different sectors in coastal areas can be ensured by adoption of a proper management strategy for groundwater extraction. Optimal use of the coastal groundwater resource is one of the best management options, which can be achieved by employing a properly developed optimal groundwater extraction strategy. Coupled simulation-optimization (S-O) approaches are essential tools to obtain the optimal groundwater extraction patterns. This study proposes approaches for developing multiple objective management of coastal aquifers with the aid of barrier extraction wells as hydraulic control measure of saltwater intrusion in multilayered coastal aquifer systems. Therefore, two conflicting objectives of management policy are considered in this research, i.e. maximizing total groundwater extraction for advantageous purposes, and minimizing the total amount of water abstraction from barrier extraction wells. The study also proposes an adaptive management strategy for coastal aquifers by developing a three-dimensional (3-D) monitoring network design. The performance of the proposed methodologies is evaluated by using both an illustrative multilayered coastal aquifer system and a real life coastal aquifer study area. Coupled S-O approach is used as the basic tool to develop a saltwater intrusion management model to obtain the optimal groundwater extraction rates from a combination of feasible solutions on the Pareto optimal front. Simulation of saltwater intrusion processes requires solution of density dependent coupled flow and solute transport numerical simulation models that are computationally intensive. Therefore, computational efficiency in the coupled S-O approach is achieved by using an approximate emulator of the accompanying physical processes of coastal aquifers. These emulators, often known as surrogate models or meta-models, can replace the computationally intensive numerical simulation model in a coupled S-O approach for achieving computational efficiency. A number of meta-models have been developed and compared in this study for integration with the optimization algorithm in order to develop saltwater intrusion management model. Fuzzy Inference System (FIS), Adaptive Neuro Fuzzy Inference System (ANFIS), Multivariate Adaptive Regression Spline (MARS), and Gaussian Process Regression (GPR) based meta-models are developed in the present study for approximating coastal aquifer responses to groundwater extraction. Properly trained and tested meta-models are integrated with a Controlled Elitist Multiple Objective Genetic Algorithm (CEMOGA) within a coupled S-O approach. In each iteration of the optimization algorithm, the meta-models are used to compute the corresponding salinity concentrations for a set of candidate pumping patterns generated by the optimization algorithm. Upon convergence, the non-dominated global optimal solutions are obtained as the Pareto optimal front, which represents a trade-off between the two conflicting objectives of the pumping management problem. It is observed from the solutions of the meta-model based coupled S-O approach that the considered meta-models are capable of producing a Pareto optimal set of solutions quite accurately. However, each meta-modelling approach has distinct advantages over the others when utilized within the integrated S-O approach. Uncertainties in estimating complex flow and solute transport processes in coastal aquifers demand incorporation of the uncertainties related to some of the model parameters. Multidimensional heterogeneity of aquifer properties such as hydraulic conductivity, compressibility, and bulk density are considered as major sources of uncertainty in groundwater modelling system. Other sources of uncertainty are associated with spatial and temporal variability of hydrologic as well as human interventions, e.g. aquifer recharge and transient groundwater extraction patterns. Different realizations of these uncertain model parameters are obtained from different statistical distributions. FIS based meta-models are advanced to a Genetic Algorithm (GA) tuned hybrid FIS model (GA-FIS), to emulate physical processes of coastal aquifers and to evaluate responses of the coastal aquifers to groundwater extraction under groundwater parameter uncertainty. GA is used to tune the FIS parameters in order to obtain the optimal FIS structure. The GA-FIS models thus obtained are linked externally to the CEMOGA in order to derive an optimal pumping management strategy using the coupled S-O approach. The evaluation results show that the proposed saltwater intrusion management model is able to derive reliable optimal groundwater extraction strategies to control saltwater intrusion for the illustrative multilayered coastal aquifer system. The optimal management strategies obtained as solutions of GA-FIS based management models are shown to be reliable and accurate within the specified ranges of values for different realizations of uncertain groundwater parameters. One of the major concerns of the meta-model based integrated S-O approach is the uncertainty associated with the meta-model predictions. These prediction uncertainties, if not addressed properly, may propagate to the optimization procedures, and may deteriorate the optimality of the solutions. A standalone meta-model, when used within an optimal management model, may result in the optimization routine producing actually suboptimal solutions that may undermine the optimality of the groundwater extraction strategies. Therefore, this study proposes an ensemble approach to address the prediction uncertainties of meta-models. Ensemble is an approach to assimilate multiple similar or different algorithms or base learners (emulators). The basic idea of ensemble lies in developing a more reliable and robust prediction tool that incorporates each individual emulator's unique characteristic in order to predict future scenarios. Each individual member of the ensemble contains different input -output mapping functions. Based on their own mapping functions, these individual emulators provide varied predictions on the response variable. Therefore, the combined prediction of the ensemble is likely to be less biased and more robust, reliable, and accurate than that of any of the individual members of the ensemble. Performance of the ensemble meta-models is evaluated using an illustrative coastal aquifer study area. The results indicate that the meta-model based ensemble modelling approach is able to provide reliable solutions for a multilayered coastal aquifer management problem. Relative sea level rise, providing an additional saline water head at the seaside, has a significant impact on an increase in the salinization process of the coastal aquifers. Although excessive groundwater withdrawal is considered as the major cause of saltwater intrusion, relative sea level rise, in combination with the effect of excessive groundwater pumping, can exacerbate the already vulnerable coastal aquifers. This study incorporates the effects of relative sea level rise on the optimized groundwater extraction values for the specified management period. Variation of water concentrations in the tidal river and seasonal fluctuation of river water stage are also incorporated. Three meta-models are developed from the solution results of the numerical simulation model that simulates the coupled flow and solute transport processes in a coastal aquifer system. The results reveal that the proposed meta-models are capable of predicting density dependent coupled flow and solute transport patterns quite accurately. Based on the comparison results, the best meta-model is selected as a computationally cheap substitute of the simulation model in the coupled S-O based saltwater intrusion management model. The performance of the proposed methodology is evaluated for an illustrative multilayered coastal aquifer system in which the effect of climate change induced sea level rise is incorporated for the specified management period. The results show that the proposed saltwater intrusion management model provides acceptable, accurate, and reliable solutions while significantly improving computational efficiency in the coupled S-O methodology. The success of the developed management strategy largely depends on how accurately the prescribed management policy is implemented in real life situations. The actual implementation of a prescribed management strategy often differs from the prescribed planned strategy due to various uncertainties in predicting the consequences, as well as practical constraints, including noncompliance with the prescribed strategy. This results in actual consequences of a management strategy differing from the intended results. To bring the management consequences closer to the intended results, adaptive management strategies can be sequentially modified at different stages of the management horizon using feedback measurements from a deigned monitoring network. This feedback information can be the actual spatial and temporal concentrations resulting from the implementation of actual management strategy. Therefore, field-scale compliance of the developed coastal aquifer management strategy is a crucial aspect of an optimally designed groundwater extraction policy. A 3-D compliance monitoring network design methodology is proposed in this study in order to develop an adaptive and sequentially modified management policy, which aims to improve optimal and justifiable use of groundwater resources in coastal aquifers. In the first step, an ensemble meta-model based multiple objective prescriptive model is developed using a coupled S-O approach in order to derive a set of Pareto optimal groundwater extraction strategies. Prediction uncertainty of meta-models is addressed by utilizing a weighted average ensemble using Set Pair Analysis. In the second step, a monitoring network is designed for evaluating the compliance of the implemented strategies with the prescribed management goals due to possible uncertainties associated with field-scale application of the proposed management policy. Optimal monitoring locations are obtained by maximizing Shannon's entropy between the saltwater concentrations at the selected potential locations. Performance of the proposed 3-D sequential compliance monitoring network design is assessed for an illustrative multilayered coastal aquifer study area. The performance evaluations show that sequential improvements of optimal management strategy are possible by utilizing saltwater concentrations measurements at the proposed optimal compliance monitoring locations. The integrated S-O approach is used to develop a saltwater intrusion management model for a real world coastal aquifer system in the Barguna district of southern Bangladesh. The aquifer processes are simulated by using a 3-D finite element based combined flow and solute transport numerical code. The modelling and management of seawater intrusion processes are performed based on very limited hydrogeological data. The model is calibrated with respect to hydraulic heads for a period of five years from April 2010 to April 2014. The calibrated model is validated for the next three-year period from April 2015 to April 2017. The calibrated and partially validated model is then used within the integrated S-O approach to develop optimal groundwater abstraction patterns to control saltwater intrusion in the study area. Computational efficiency of the management model is achieved by using a MARS based meta-model approximately emulating the combined flow and solute transport processes of the study area. This limited evaluation demonstrates that a planned transient groundwater abstraction strategy, acquired as solution results of a meta-model based integrated S-O approach, is a useful management strategy for optimized water abstraction and saltwater intrusion control. This study shows the capability of the MARS meta-model based integrated S-O approach to solve real-life complex management problems in an efficient manner

Monitoring and Optimization of ATLAS Tier 2 Center GoeGrid

The demand on computational and storage resources is growing along with the amount of infor- mation that needs to be processed and preserved. In order to ease the provisioning of the digital services to the growing number of consumers, more and more distributed computing systems and platforms are actively developed and employed. The building block of the distributed computing infrastructure are single computing centers, similar to the Worldwide LHC Computing Grid, Tier 2 centre GoeGrid. The main motivation of this thesis was the optimization of GoeGrid perfor- mance by efficient monitoring. The goal has been achieved by means of the GoeGrid monitoring information analysis. The data analysis approach was based on the adaptive-network-based fuzzy inference system (ANFIS) and machine learning algorithm such as Linear Support Vector Machine (SVM). The main object of the research was the digital service, since availability, reliability and ser- viceability of the computing platform can be measured according to the constant and stable provisioning of the services. Due to the widely used concept of the service oriented architecture (SOA) for large computing facilities, in advance knowing of the service state as well as the quick and accurate detection of its disability allows to perform the proactive management of the com- puting facility. The proactive management is considered as a core component of the computing facility management automation concept, such as Autonomic Computing. Thus in time as well as in advance and accurate identification of the provided service status can be considered as a contribution to the computing facility management automation, which is directly related to the provisioning of the stable and reliable computing resources. Based on the case studies, performed using the GoeGrid monitoring data, consideration of the approaches as generalized methods for the accurate and fast identification and prediction of the service status is reasonable. Simplicity and low consumption of the computing resources allow to consider the methods in the scope of the Autonomic Computing component

Design of neuro-fuzzy models by evolutionary and gradient-based algorithms

All systems found in nature exhibit, with different degrees, a nonlinear behavior. To emulate this behavior, classical systems identification techniques use, typically, linear models, for mathematical simplicity. Models inspired by biological principles (artificial neural networks) and linguistically motivated (fuzzy systems), due to their universal approximation property, are becoming alternatives to classical mathematical models. In systems identification, the design of this type of models is an iterative process, requiring, among other steps, the need to identify the model structure, as well as the estimation of the model parameters. This thesis addresses the applicability of gradient-basis algorithms for the parameter estimation phase, and the use of evolutionary algorithms for model structure selection, for the design of neuro-fuzzy systems, i.e., models that offer the transparency property found in fuzzy systems, but use, for their design, algorithms introduced in the context of neural networks. A new methodology, based on the minimization of the integral of the error, and exploiting the parameter separability property typically found in neuro-fuzzy systems, is proposed for parameter estimation. A recent evolutionary technique (bacterial algorithms), based on the natural phenomenon of microbial evolution, is combined with genetic programming, and the resulting algorithm, bacterial programming, advocated for structure determination. Different versions of this evolutionary technique are combined with gradient-based algorithms, solving problems found in fuzzy and neuro-fuzzy design, namely incorporation of a-priori knowledge, gradient algorithms initialization and model complexity reduction.Todos os sistemas encontrados na natureza exibem, com maior ou menor grau, um comportamento linear. De modo a emular esse comportamento, as técnicas de identificação clássicas usam, tipicamente e por simplicidade matemática, modelos lineares. Devido à sua propriedade de aproximação universal, modelos inspirados por princípios biológicos (redes neuronais artificiais) e motivados linguisticamente (sistemas difusos) tem sido cada vez mais usados como alternativos aos modelos matemáticos clássicos. Num contexto de identificação de sistemas, o projeto de modelos como os acima descritos é um processo iterativo, constituído por vários passos. Dentro destes, encontra-se a necessidade de identificar a estrutura do modelo a usar, e a estimação dos seus parâmetros. Esta Tese discutirá a aplicação de algoritmos baseados em derivadas para a fase de estimação de parâmetros, e o uso de algoritmos baseados na teoria da evolução de espécies, algoritmos evolutivos, para a seleção de estrutura do modelo. Isto será realizado no contexto do projeto de modelos neuro-difusos, isto é, modelos que simultaneamente exibem a propriedade de transparência normalmente associada a sistemas difusos mas que utilizam, para o seu projeto algoritmos introduzidos no contexto de redes neuronais. Os modelos utilizados neste trabalho são redes B-Spline, de Função de Base Radial, e sistemas difusos dos tipos Mamdani e Takagi-Sugeno. Neste trabalho começa-se por explorar, para desenho de redes B-Spline, a introdução de conhecimento à-priori existente sobre um processo. Neste sentido, aplica-se uma nova abordagem na qual a técnica para a estimação dos parâmetros é alterada a fim de assegurar restrições de igualdade da função e das suas derivadas. Mostra-se ainda que estratégias de determinação de estrutura do modelo, baseadas em computação evolutiva ou em heurísticas determinísticas podem ser facilmente adaptadas a este tipo de modelos restringidos. É proposta uma nova técnica evolutiva, resultante da combinação de algoritmos recentemente introduzidos (algoritmos bacterianos, baseados no fenómeno natural de evolução microbiana) e programação genética. Nesta nova abordagem, designada por programação bacteriana, os operadores genéticos são substituídos pelos operadores bacterianos. Deste modo, enquanto a mutação bacteriana trabalha num indivíduo, e tenta otimizar a bactéria que o codifica, a transferência de gene é aplicada a toda a população de bactérias, evitando-se soluções de mínimos locais. Esta heurística foi aplicada para o desenho de redes B-Spline. O desempenho desta abordagem é ilustrada e comparada com alternativas existentes. Para a determinação dos parâmetros de um modelo são normalmente usadas técnicas de otimização locais, baseadas em derivadas. Como o modelo em questão é não-linear, o desempenho deste género de técnicas é influenciado pelos pontos de partida. Para resolver este problema, é proposto um novo método no qual é usado o algoritmo evolutivo referido anteriormente para determinar pontos de partida mais apropriados para o algoritmo baseado em derivadas. Deste modo, é aumentada a possibilidade de se encontrar um mínimo global. A complexidade dos modelos neuro-difusos (e difusos) aumenta exponencialmente com a dimensão do problema. De modo a minorar este problema, é proposta uma nova abordagem de particionamento do espaço de entrada, que é uma extensão das estratégias de decomposição de entrada normalmente usadas para este tipo de modelos. Simulações mostram que, usando esta abordagem, se pode manter a capacidade de generalização com modelos de menor complexidade. Os modelos B-Spline são funcionalmente equivalentes a modelos difusos, desde que certas condições sejam satisfeitas. Para os casos em que tal não acontece (modelos difusos Mamdani genéricos), procedeu-se à adaptação das técnicas anteriormente empregues para as redes B-Spline. Por um lado, o algoritmo Levenberg-Marquardt é adaptado e a fim de poder ser aplicado ao particionamento do espaço de entrada de sistema difuso. Por outro lado, os algoritmos evolutivos de base bacteriana são adaptados para sistemas difusos, e combinados com o algoritmo de Levenberg-Marquardt, onde se explora a fusão das características de cada metodologia. Esta hibridização dos dois algoritmos, denominada de algoritmo bacteriano memético, demonstrou, em vários problemas de teste, apresentar melhores resultados que alternativas conhecidas. Os parâmetros dos modelos neuronais utilizados e dos difusos acima descritos (satisfazendo no entanto alguns critérios) podem ser separados, de acordo com a sua influência na saída, em parâmetros lineares e não-lineares. Utilizando as consequências desta propriedade nos algoritmos de estimação de parâmetros, esta Tese propõe também uma nova metodologia para estimação de parâmetros, baseada na minimização do integral do erro, em alternativa à normalmente utilizada minimização da soma do quadrado dos erros. Esta técnica, além de possibilitar (em certos casos) um projeto totalmente analítico, obtém melhores resultados de generalização, dado usar uma superfície de desempenho mais similar aquela que se obteria se se utilizasse a função geradora dos dados

Proceedings. 25. Workshop Computational Intelligence, Dortmund, 26. - 27. November 2015

Dieser Tagungsband enthält die Beiträge des 25. Workshops „Computational Intelligence“ des Fachausschusses 5.14 der VDI/VDE-Gesellschaft für Mess- und Automatisierungstechnik (GMA) , der vom 26. – 27. November 2015 in Dortmund stattfindet