Computational intelligence techniques for maximum energy efficiency of cogeneration processes based on internal combustion engines

153 p.El objeto de la tesis consiste en desarrollar estrategias de modelado y optimización del rendimiento energético de plantas de cogeneración basadas en motores de combustión interna (MCI), mediante el uso de las últimas tecnologías de inteligencia computacional. Con esta finalidad se cuenta con datos reales de una planta de cogeneración de energía, propiedad de la compañía EnergyWorks, situada en la localidad de Monzón (provincia de Huesca). La tesis se realiza en el marco de trabajo conjunto del Grupo de Diseño en Electrónica Digital (GDED) de la Universidad del País Vasco UPV/EHU y la empresa Optimitive S.L., empresa dedicada al software avanzado para la mejora en tiempo real de procesos industriale

How important are activation functions in regression and classification? A survey, performance comparison, and future directions

Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification data sets such as the MNIST, CIFAR-10, and CIFAR- 100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, Pytorch, and JAX.Comment: 28 pages, 15 figure

On-the-fly synthesizer programming with rule learning

This manuscript explores automatic programming of sound synthesis algorithms within the context of the performative artistic practice known as live coding. Writing source code in an improvised way to create music or visuals became an instrument the moment affordable computers were able to perform real-time sound synthesis with languages that keep their interpreter running. Ever since, live coding has dealt with real time programming of synthesis algorithms. For that purpose, one possibility is an algorithm that automatically creates variations out of a few presets selected by the user. However, the need for real-time feedback and the small size of the data sets (which can even be collected mid-performance) are constraints that make existing automatic sound synthesizer programmers and learning algorithms unfeasible. Also, the design of such algorithms is not oriented to create variations of a sound but rather to find the synthesizer parameters that match a given one. Other approaches create representations of the space of possible sounds, allowing the user to explore it by means of interactive evolution. Even though these systems are exploratory-oriented, they require longer run-times. This thesis investigates inductive rule learning for on-the-fly synthesizer programming. This approach is conceptually different from those found in both synthesizer programming and live coding literature. Rule models offer interpretability and allow working with the parameter values of the synthesis algorithms (even with symbolic data), making preprocessing unnecessary. RuLer, the proposed learning algorithm, receives a dataset containing user labeled combinations of parameter values of a synthesis algorithm. Among those combinations sharing the same label, it analyses the patterns based on dissimilarity. These patterns are described as an IF-THEN rule model. The algorithm parameters provide control to define what is considered a pattern. As patterns are the base for inducting new parameter settings, the algorithm parameters control the degree of consistency of the inducted settings respect to the original input data. An algorithm (named FuzzyRuLer) able to extend IF-THEN rules to hyperrectangles, which in turn are used as the cores of membership functions, is presented. The resulting fuzzy rule model creates a map of the entire input feature space. For such a pursuit, the algorithm generalizes the logical rules solving the contradictions by following a maximum volume heuristics. Across the manuscript it is discussed how, when machine learning algorithms are used as creative tools, glitches, errors or inaccuracies produced by the resulting models are sometimes desirable as they might offer novel, unpredictable results. The evaluation of the algorithms follows two paths. The first focuses on user tests. The second responds to the fact that this work was carried out within the computer science department and is intended to provide a broader, nonspecific domain evaluation of the algorithms performance using extrinsic benchmarks (i.e not belonging to a synthesizer's domain) for cross validation and minority oversampling. In oversampling tasks, using imbalanced datasets, the algorithm yields state-of-the-art results. Moreover, the synthetic points produced are significantly different from those created by the other algorithms and perform (controlled) exploration of more distant regions. Finally, accompanying the research, various performances, concerts and an album were produced with the algorithms and examples of this thesis. The reviews received and collections where the album has been featured show a positive reception within the community. Together, these evaluations suggest that rule learning is both an effective method and a promising path for further research.Aquest manuscrit explora la programació automàtica d’algorismes de síntesi de so dins del context de la pràctica artística performativa coneguda com a live coding. L'escriptura improvisada de codi font per crear música o visuals es va convertir en un instrument en el moment en què els ordinadors van poder realitzar síntesis de so en temps real amb llenguatges que mantenien el seu intèrpret en funcionament. D'aleshores ençà, el live coding comporta la programació en temps real d’algorismes de síntesi de so. Per a aquest propòsit, una possibilitat és tenir un algorisme que creï automàticament variacions a partir d'alguns presets seleccionats. No obstant, la necessitat de retroalimentació en temps real i la petita mida dels conjunts de dades són restriccions que fan que els programadors automàtics de sintetitzadors de so i els algorismes d’aprenentatge no siguin factibles d’utilitzar. A més, el seu disseny no està orientat a crear variacions d'un so, sinó a trobar els paràmetres del sintetitzador que aplicats a l'algorisme de síntesi produeixen un so determinat (target). Altres enfocaments creen representacions de l'espai de sons possibles, per permetre a l'usuari explorar-lo mitjançant l'evolució interactiva, però requereixen temps més llargs. Aquesta tesi investiga l'aprenentatge inductiu de regles per a la programació on-the-fly de sintetitzadors. Aquest enfocament és conceptualment diferent dels que es troben a la literatura. Els models de regles ofereixen interpretabilitat i permeten treballar amb els valors dels paràmetres dels algorismes de síntesi, sense processament previ. RuLer, l'algorisme d'aprenentatge proposat, rep dades amb combinacions etiquetades per l'usuari dels valors dels paràmetres d'un algorisme de síntesi. A continuació, analitza els patrons, basats en la dissimilitud, entre les combinacions de cada etiqueta. Aquests patrons es descriuen com un model de regles IF-THEN. Els paràmetres de l'algorisme proporcionen control per definir el que es considera un patró. Llavors, controlen el grau de consistència dels nous paràmetres de síntesi induïts respecte a les dades d'entrada originals. A continuació, es presenta un algorisme (FuzzyRuLer) capaç d’estendre les regles IF-THEN a hiperrectangles, que al seu torn s’utilitzen com a nuclis de funcions de pertinença. El model de regles difuses resultant crea un mapa complet de l'espai de la funció d'entrada. Per això, l'algorisme generalitza les regles lògiques seguint una heurística de volum màxim. Al llarg del manuscrit es discuteix com, quan s’utilitzen algorismes d’aprenentatge automàtic com a eines creatives, de vegades són desitjables glitches, errors o imprecisions produïdes pels models resultants, ja que poden oferir nous resultats imprevisibles. L'avaluació dels algorismes segueix dos camins. El primer es centra en proves d'usuari. El segon, que respon al fet que aquest treball es va dur a terme dins del departament de ciències de la computació, pretén proporcionar una avaluació més àmplia, no específica d'un domini, del rendiment dels algorismes mitjançant benchmarks extrínsecs utilitzats per cross-validation i minority oversampling. En tasques d'oversampling, mitjançant imbalanced data sets, l'algorisme proporciona resultats equiparables als de l'estat de l'art. A més, els punts sintètics produïts són significativament diferents als creats pels altres algorismes i realitzen exploracions (controlades) de regions més llunyanesEste manuscrito explora la programación automática de algoritmos de síntesis de sonido dentro del contexto de la práctica artística performativa conocida como live coding. La escritura de código fuente de forma improvisada para crear música o imágenes, se convirtió en un instrumento en el momento en que las computadoras asequibles pudieron realizar síntesis de sonido en tiempo real con lenguajes que mantuvieron su interprete en funcionamiento. Desde entonces, el live coding ha implicado la programación en tiempo real de algoritmos de síntesis. Para ese propósito, una posibilidad es tener un algoritmo que cree automáticamente variaciones a partir de unos pocos presets seleccionados. Sin embargo, la necesidad de retroalimentación en tiempo real y el pequeño tamaño de los conjuntos de datos (que incluso pueden recopilarse durante la misma actuación), limitan el uso de los algoritmos existentes, tanto de programación automática de sintetizadores como de aprendizaje de máquina. Además, el diseño de dichos algoritmos no está orientado a crear variaciones de un sonido, sino a encontrar los parámetros del sintetizador que coincidan con un sonido dado. Otros enfoques crean representaciones del espacio de posibles sonidos, para permitir al usuario explorarlo mediante evolución interactiva. Aunque estos sistemas están orientados a la exploración, requieren tiempos más largos. Esta tesis investiga el aprendizaje inductivo de reglas para la programación de sintetizadores on-the-fly. Este enfoque es conceptualmente diferente de los que se encuentran en la literatura, tanto de programación de sintetizadores como de live coding. Los modelos de reglas ofrecen interpretabilidad y permiten trabajar con los valores de los parámetros de los algoritmos de síntesis (incluso con datos simbólicos), haciendo innecesario el preprocesamiento. RuLer, el algoritmo de aprendizaje propuesto, recibe un conjunto de datos que contiene combinaciones, etiquetadas por el usuario, de valores de parámetros de un algoritmo de síntesis. Luego, analiza los patrones, en función de la disimilitud, entre las combinaciones de cada etiqueta. Estos patrones se describen como un modelo de reglas lógicas IF-THEN. Los parámetros del algoritmo proporcionan el control para definir qué se considera un patrón. Como los patrones son la base para inducir nuevas configuraciones de parámetros, los parámetros del algoritmo controlan también el grado de consistencia de las configuraciones inducidas con respecto a los datos de entrada originales. Luego, se presenta un algoritmo (llamado FuzzyRuLer) capaz de extender las reglas lógicas tipo IF-THEN a hiperrectángulos, que a su vez se utilizan como núcleos de funciones de pertenencia. El modelo de reglas difusas resultante crea un mapa completo del espacio de las clases de entrada. Para tal fin, el algoritmo generaliza las reglas lógicas resolviendo las contradicciones utilizando una heurística de máximo volumen. A lo largo del manuscrito se analiza cómo, cuando los algoritmos de aprendizaje automático se utilizan como herramientas creativas, los glitches, errores o inexactitudes producidas por los modelos resultantes son a veces deseables, ya que pueden ofrecer resultados novedosos e impredecibles. La evaluación de los algoritmos sigue dos caminos. El primero se centra en pruebas de usuario. El segundo, responde al hecho de que este trabajo se llevó a cabo dentro del departamento de ciencias de la computación y está destinado a proporcionar una evaluación más amplia, no de dominio específica, del rendimiento de los algoritmos utilizando beanchmarks extrínsecos para cross-validation y oversampling. En estas últimas pruebas, utilizando conjuntos de datos no balanceados, el algoritmo produce resultados equiparables a los del estado del arte. Además, los puntos sintéticos producidos son significativamente diferentes de los creados por los otros algoritmos y realizan una exploración (controlada) de regiones más distantes. Finalmente, acompañando la investigación, realicé diversas presentaciones, conciertos y un ´álbum utilizando los algoritmos y ejemplos de esta tesis. Las críticas recibidas y las listas donde se ha presentado el álbum muestran una recepción positiva de la comunidad. En conjunto, estas evaluaciones sugieren que el aprendizaje de reglas es al mismo tiempo un método eficaz y un camino prometedor para futuras investigaciones.Postprint (published version

Discovering logical knowledge in non-symbolic domains

Deep learning and symbolic artificial intelligence remain the two main paradigms in Artificial Intelligence (AI), each presenting their own strengths and weaknesses. Artificial agents should integrate both of these aspects of AI in order to show general intelligence and solve complex problems in real-world scenarios; similarly to how humans use both the analytical left side and the intuitive right side of their brain in their lives. However, one of the main obstacles hindering this integration is the Symbol Grounding Problem [144], which is the capacity to map physical world observations to a set of symbols. In this thesis, we combine symbolic reasoning and deep learning in order to better represent and reason with abstract knowledge. In particular, we focus on solving non-symbolic-state Reinforcement Learning environments using a symbolic logical domain. We consider different configurations: (i) unknown knowledge of both the symbol grounding function and the symbolic logical domain, (ii) unknown knowledge of the symbol grounding function and prior knowledge of the domain, (iii) imperfect knowledge of the symbols grounding function and unknown knowledge of the domain. We develop algorithms and neural network architectures that are general enough to be applied to different kinds of environments, which we test on both continuous-state control problems and image-based environments. Specifically, we develop two kinds of architectures: one for Markovian RL tasks and one for non-Markovian RL domains. The first is based on model-based RL and representation learning, and is inspired by the substantial prior work in state abstraction for RL [115]. The second is mainly based on recurrent neural networks and continuous relaxations of temporal logic domains. In particular, the first approach extracts a symbolic STRIPS-like abstraction for control problems. For the second approach, we explore connections between recurrent neural networks and finite state machines, and we define Visual Reward Machines, an extension to non-symbolic domains of Reward Machines [27], which are a popular approach to non-Markovian RL tasks

Entwicklung eines auf Fuzzy-Regeln basierten Expertensystems zur Hochwasservorhersage im mesoskaligen Einzugsgebiet des Oberen Mains

People worldwide are faced with flood events of different magnitudes. A timely and reliable flood forecast is essential for the people to save goods and, more important, lives. The development of a fuzzy rule based flood forecast system considering extreme flood events within meso-scale catchments and with return periods of 100 years and more is the main objective of this work. Considering one river catchment extreme flood events are usually seldom. However, these data are essential for a reliable setup of warning systems. In this work the database is extended by simulations of possible flood events performing the hydrological model WaSiM-ETH (Water balance Simulation model ETH) driven by generated precipitation fields. The therefore required calibration of the hydrological model is performed applying the genetic optimization algorithm SCE (Shuffled Complex Evolution). Thereby, different SCE configuration setups are investigated and an optimization strategy for the Upper Main basin is developed in order to ensure reliable und satisfying calibration results. In this thesis the developed forecast system comprises different time horizons (3 days; 6, 12, and 48 hours) in order to ensure a reliable and continuous flood forecast at the three main gauges of the Upper Main river. Thereby, the focus of the different fuzzy inference systems lies on different discharge conditions, which together ensure a continuous flood forecast. In this work the performance of the two classical fuzzy inference systems, Mamdani and Takagi-Sugeno, is investigated considering all four forecast horizons. Thereby, a wide variety of different input features, among others Tukey data depth, is taken into consideration. For the training of the fuzzy inference systems the SA (Simulated Annealing) optimization algorithm is applied. A further performance comparison is carried out considering the 48 hour forecast behaviour of the two fuzzy inference systems and the hydrological model WaSiM-ETH. In this work the expert system ExpHo-HORIX is developed in order to combine the single, trained fuzzy inference systems to one overall flood warning system. This expert system ensures beside the fast forecast a quantification of uncertainties within a manageable, user-friendly, and transparent framework which can be easily implemented into an exiting environment.Menschen weltweit werden mit Hochwasserereignissen unterschiedlicher Stärke konfrontiert. Um Eigentum und, noch viel wichtiger, Leben zu retten, ist eine rechtzeitige und zuverlässige Hochwasserwarnung und folglich -vorhersage unerlässlich. Ziel dieser Arbeit ist es deshalb, ein auf Fuzzy-Regeln basiertes Hochwasserwarnsystem für mesoskalige Einzugsgebiete und die Vorhersage von extremen Hochwasserereignissen mit Wiederkehrperioden von 100 Jahren und mehr unter Berücksichtigung von Unsicherheiten zu entwickeln. Da extreme Hochwasserereignisse mit einer Jährlichkeit von 100 oder mehr Jahren in der Realität nicht in jedem Einzugsgebiet bereits beobachtet und aufgezeichnet wurden, ist eine Erweiterung der Datenbank auf Grund von Modellsimulationen zwingend notwendig. In dieser Arbeit werden hierzu das hydrologische Modell WaSiM-ETH (Wasserhaushalts-Simulations-Modell ETH) sowie von Bliefernicht et al. (2008) generierte Niederschlagsfelder verwendet. Die Kalibrierung des Modells erfolgt mit dem SCE (Shuffled Complex Evolution) Optimierungsalgorithmus. Um reproduzierbare Kalibrierungsergebnisse zu erzielen und die notwendige Kalibrierungszeit möglichst gering zu halten, werden unterschiedliche Optimierungskonfigurationen untersucht und eine Kalibrierungsstrategie für das mesoskalige Einzugsgebiet des Oberen Mains entwickelt. Um eine kontinuierliche und zuverlässige Vorhersage zu garantieren, ist die Idee entwickelt worden, Fuzzy-Regelsysteme für unterschiedliche Vorhersagehorizonte (3 Tage; 6, 12 und 48 Stunden) für die drei Hauptpegel des Oberen Mains aufzustellen, die im Zusammenspiel eine kontinuierliche Vorhersage sicher stellen. Der Fokus der 3-Tagesvorhersage liegt hierbei in der zuverlässigen Wiedergabe von geringen und mittleren Abflussbedingungen sowie der zuverlässigen und rechtzeitigen Vorhersage von Überschreitungen einer vordefinierten Meldestufe. Eine vorhergesagte Überschreitung der Meldestufe führt zu einem Wechsel der Vorhersagesysteme von der 3-Tages- zu der 6-, 12- und 48-Stundenvorhersage, deren Fokus auf der Vorhersage der Hochwasserganglinie liegt. In diesem Zusammenhang wird die Effizienz der beiden klassischen Regelsysteme,Mamdani und Takagi-Sugeno, sowie die Kombination unterschiedlicher Eingangsgrößen, unter anderem Tukey Tiefenfunktion, näher untersucht. Ein weiterer Effizienzvergleich wird zwischen den Mamdani Regelsystemen der 48-Stundenvorhersage und dem hydrologischen ModellWaSiM-ETH durchgeführt. Für das Training der beiden Regelsysteme wird der SA (Simulated Annealing) Optimierungsalgorithmus verwendet. Die einzelnen Fuzzy-Regelsysteme werden schließlich in dem entwickelten Hochwasserwarnsystem ExpHo-HORIX (Expertensystem Hochwasser - HORIX) zusammengefügt. Standardmäßig wird für jede Vorhersage die Niederschlagsunsicherheit auf Grund von Ensemble-Vorhersagen innerhalb ExpHo-HORIX analysiert und ausgewiesen. Im Hochwasserfall können für die stündlichen Fuzzy-RegelsystemeModellunsicherheiten des hydrologischenModells, das für die Generierung der Datenbank von Extremereignissen verwendet wurde, zusätzlich ausgewiesen werden. Hierzu müssen zusätzlich Ergebnisse der SCEM Analyse (Grundmann, 2009) vorliegen

Improving classification models with context knowledge and variable activation functions

This work proposes two methods to boost the performances of a given classifier: the first one, which works on a Neural Network classifier, is a new type of trainable activation function, that is a function which is adjusted during the learning phase, allowing the network to exploit the data better respect to use a classic activation function with fixed-shape; the second one provides two frameworks to use an external knowledge base to improve the classification results

Deep neural networks for video classification in ecology

Analyzing large volumes of video data is a challenging and time-consuming task. Automating this process would very valuable, especially in ecological research where massive amounts of video can be used to unlock new avenues of ecological research into the behaviour of animals in their environments. Deep Neural Networks, particularly Deep Convolutional Neural Networks, are a powerful class of models for computer vision. When combined with Recurrent Neural Networks, Deep Convolutional models can be applied to video for frame level video classification. This research studies two datasets: penguins and seals. The purpose of the research is to compare the performance of image-only CNNs, which treat each frame of a video independently, against a combined CNN-RNN approach; and to assess whether incorporating the motion information in the temporal aspect of video improves the accuracy of classifications in these two datasets. Video and image-only models offer similar out-of-sample performance on the simpler seals dataset but the video model led to moderate performance improvements on the more complex penguin action recognition dataset

North American Fuzzy Logic Processing Society (NAFIPS 1992), volume 2

This document contains papers presented at the NAFIPS '92 North American Fuzzy Information Processing Society Conference. More than 75 papers were presented at this Conference, which was sponsored by NAFIPS in cooperation with NASA, the Instituto Tecnologico de Morelia, the Indian Society for Fuzzy Mathematics and Information Processing (ISFUMIP), the Instituto Tecnologico de Estudios Superiores de Monterrey (ITESM), the International Fuzzy Systems Association (IFSA), the Japan Society for Fuzzy Theory and Systems, and the Microelectronics and Computer Technology Corporation (MCC). The fuzzy set theory has led to a large number of diverse applications. Recently, interesting applications have been developed which involve the integration of fuzzy systems with adaptive processes such a neural networks and genetic algorithms. NAFIPS '92 was directed toward the advancement, commercialization, and engineering development of these technologies

Designing Regularizers and Architectures for Recurrent Neural Networks

Cette thèse contribue a la recherche vers l'intelligence artificielle en utilisant des méthodes connexionnistes. Les réseaux de neurones récurrents sont un ensemble de modèles séquentiels de plus en plus populaires capable en principe d'apprendre des algorithmes arbitraires. Ces modèles effectuent un apprentissage en profondeur, un type d'apprentissage machine. Sa généralité et son succès empirique en font un sujet intéressant pour la recherche et un outil prometteur pour la création de l'intelligence artificielle plus générale. Le premier chapitre de cette thèse donne un bref aperçu des sujets de fonds: l'intelligence artificielle, l'apprentissage machine, l'apprentissage en profondeur et les réseaux de neurones récurrents. Les trois chapitres suivants couvrent ces sujets de manière de plus en plus spécifiques. Enfin, nous présentons quelques contributions apportées aux réseaux de neurones récurrents. Le chapitre \ref{arxiv1} présente nos travaux de régularisation des réseaux de neurones récurrents. La régularisation vise à améliorer la capacité de généralisation du modèle, et joue un role clé dans la performance de plusieurs applications des réseaux de neurones récurrents, en particulier en reconnaissance vocale. Notre approche donne l'état de l'art sur TIMIT, un benchmark standard pour cette tâche. Le chapitre \ref{cpgp} présente une seconde ligne de travail, toujours en cours, qui explore une nouvelle architecture pour les réseaux de neurones récurrents. Les réseaux de neurones récurrents maintiennent un état caché qui représente leurs observations antérieures. L'idée de ce travail est de coder certaines dynamiques abstraites dans l'état caché, donnant au réseau une manière naturelle d'encoder des tendances cohérentes de l'état de son environnement. Notre travail est fondé sur un modèle existant; nous décrivons ce travail et nos contributions avec notamment une expérience préliminaire.This thesis represents incremental work towards artificial intelligence using connectionist methods. Recurrent neural networks are a set of increasingly popular sequential models capable in principle of learning arbitrary algorithms. These models perform deep learning, a type of machine learning. Their generality and empirical success makes them an attractive candidate for further work and a promising tool for creating more general artificial intelligence. The first chapter of this thesis gives a brief overview of the background topics: artificial intelligence, machine learning, deep learning, and recurrent neural nets. The next three chapters cover these topics in order of increasing specificity. Finally, we contribute some general methods for recurrent neural networks. Chapter \ref{arxiv1} presents our work on the topic of recurrent neural network regularization. Regularization aims to improve a model's generalization ability, and is a key bottleneck in the performance for several applications of recurrent neural networks, most notably speech recognition. Our approach gives state of the art results on the standard TIMIT benchmark for this task. Chapter \ref{cpgp} presents the second line of work, still in progress, exploring a new architecture for recurrent neural nets. Recurrent neural networks maintain a hidden state which represents their previous observations. The idea of this work is to encode some abstract dynamics in the hidden state, giving the network a natural way to encode consistent or slow-changing trends in the state of its environment. Our work builds on a previously developed model; we describe this previous work and our contributions, including a preliminary experiment