Optimisation de réseaux de neurones à décharges avec contraintes matérielles pour processeur neuromorphique

Les modèles informatiques basés sur l'apprentissage machine ont démarré la seconde révolution de l'intelligence artificielle. Capables d'atteindre des performances que l'on crut inimaginables au préalable, ces modèles semblent devenir partie courante dans plusieurs domaines. La face cachée de ceux-ci est que l'énergie consommée pour l'apprentissage, et l'utilisation de ces techniques, est colossale. La dernière décennie a été marquée par l'arrivée de plusieurs processeurs neuromorphiques pouvant simuler des réseaux de neurones avec une faible consommation d'énergie. Ces processeurs offrent une alternative aux conventionnelles cartes graphiques qui demeurent à ce jour essentielles au domaine. Ces processeurs sont capables de réduire la consommation d'énergie en utilisant un modèle de neurone événementiel, plus communément appelé neurone à décharge. Ce type de neurone est fondamentalement différent du modèle classique, et possède un aspect temporel important. Les méthodes, algorithmes et outils développés pour le modèle de neurone classique ne sont pas adaptés aux neurones à décharges. Cette thèse de doctorat décrit plusieurs approches fondamentales, dédiées à la création de processeurs neuromorphiques analogiques, qui permettent de pallier l'écart existant entre les systèmes à base de neurones conventionnels et à décharges. Dans un premier temps, nous présentons une nouvelle règle de plasticité synaptique permettant l'apprentissage non supervisé des réseaux de neurones récurrents utilisant ce nouveau type de neurone. Puis, nous proposons deux nouvelles méthodes pour la conception des topologies de ce même type de réseau. Finalement, nous améliorons les techniques d'apprentissage supervisé en augmentant la capacité de mémoire de réseaux récurrents. Les éléments de cette thèse marient l'inspiration biologique du cerveau, l'ingénierie neuromorphique et l'informatique fondamentale pour permettre d'optimiser les réseaux de neurones pouvant fonctionner sur des processeurs neuromorphiques analogiques

Recurrent neural networks: methods and applications to non-linear predictions

This thesis deals with recurrent neural networks, a particular class of artificial neural networks which can learn a generative model of input sequences. The input is mapped, through a feedback loop and a non-linear activation function, into a hidden state, which is then projected into the output space, obtaining either a probability distribution or the new input for the next time-step. This work consists mainly of two parts: a theoretical study for helping the understanding of recurrent neural networks framework, which is not yet deeply investigated, and their application to non-linear prediction problems, since recurrent neural networks are really powerful models suitable for solving several practical tasks in different fields. For what concerns the theoretical part, we analyse the weaknesses of state-of-the-art models and tackle them in order to improve the performance of a recurrent neural network. Firstly, we contribute in the understanding of the dynamical properties of a recurrent neural network, highlighting the close relation between the definition of stable limit cycles and the echo state property of an echo state network. We provide sufficient conditions for the convergence of the hidden state to a trajectory, which is uniquely determined by the input signal, independently of the initial states. This may help extend the memory of the network and increase the design options for the network. Moreover, we develop a novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem. Our new method allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps. Exploiting the singular value decomposition of the vanishing factors in the gradient and random matrices theory, we find that the singular values have to be confined in a narrow interval and derive conditions about their root mean square value. Then, we also improve the efficiency of the training of a recurrent neural network, defining a new method for speeding up this process. Thanks to a least square regularization, we can initialize the parameters of the network, in order to set them closer to the minimum and running fewer epochs of classical training algorithms. Moreover, it is also possible to completely train the network with our initialization method, running more iterations of it without losing in performance with respect to classical training algorithms. Finally, it is also possible to use it as a real-time learning algorithm, adjusting the parameters to the new data through one iteration of our initialization. In the last part of this thesis, we apply recurrent neural networks to non-linear prediction problems. We consider prediction of numerical sequences, estimating the following input choosing it from a probability distribution. We study an automatic text generation problem, where we need to predict the following character in order to compose words and sentences, and a path prediction of walking mobile users in the central area of a city, as a sequence of crossroads. Then, we analyse the prediction of video frames, discovering a wide range of applications related to the prediction of movements. We study the collision problem of bouncing balls, taking into account only the sequence of video frames without any knowledge about the physical characteristics of the problem, and the distribution over days of mobile user in a city and in a whole region. Finally, we address the state-of-the-art problem of missing data imputation, analysing the incomplete spectrogram of audio signals. We restore audio signals with missing time-frequency data, demonstrating via numerical experiments that a performance improvement can be achieved involving recurrent neural networks

A probabilistic numerical method for optimal multiple switching problem and application to investments in electricity generation

In this paper, we present a probabilistic numerical algorithm combining dynamic programming, Monte Carlo simulations and local basis regressions to solve non-stationary optimal multiple switching problems in infinite horizon. We provide the rate of convergence of the method in terms of the time step used to discretize the problem, of the size of the local hypercubes involved in the regressions, and of the truncating time horizon. To make the method viable for problems in high dimension and long time horizon, we extend a memory reduction method to the general Euler scheme, so that, when performing the numerical resolution, the storage of the Monte Carlo simulation paths is not needed. Then, we apply this algorithm to a model of optimal investment in power plants. This model takes into account electricity demand, cointegrated fuel prices, carbon price and random outages of power plants. It computes the optimal level of investment in each generation technology, considered as a whole, w.r.t. the electricity spot price. This electricity price is itself built according to a new extended structural model. In particular, it is a function of several factors, among which the installed capacities. The evolution of the optimal generation mix is illustrated on a realistic numerical problem in dimension eight, i.e. with two different technologies and six random factors

Improving Access and Mental Health for Youth Through Virtual Models of Care

The overall objective of this research is to evaluate the use of a mobile health smartphone application (app) to improve the mental health of youth between the ages of 14–25 years, with symptoms of anxiety/depression. This project includes 115 youth who are accessing outpatient mental health services at one of three hospitals and two community agencies. The youth and care providers are using eHealth technology to enhance care. The technology uses mobile questionnaires to help promote self-assessment and track changes to support the plan of care. The technology also allows secure virtual treatment visits that youth can participate in through mobile devices. This longitudinal study uses participatory action research with mixed methods. The majority of participants identified themselves as Caucasian (66.9%). Expectedly, the demographics revealed that Anxiety Disorders and Mood Disorders were highly prevalent within the sample (71.9% and 67.5% respectively). Findings from the qualitative summary established that both staff and youth found the software and platform beneficial

The Impact of Digital Technologies on Public Health in Developed and Developing Countries

This open access book constitutes the refereed proceedings of the 18th International Conference on String Processing and Information Retrieval, ICOST 2020, held in Hammamet, Tunisia, in June 2020.* The 17 full papers and 23 short papers presented in this volume were carefully reviewed and selected from 49 submissions. They cover topics such as: IoT and AI solutions for e-health; biomedical and health informatics; behavior and activity monitoring; behavior and activity monitoring; and wellbeing technology. *This conference was held virtually due to the COVID-19 pandemic

Machine learning in analytical chemistry: applying innovative data analysis methods using chromatographic techniques

Dissertação de mestrado em Chemical Analysis and Characterisation Techniques Chemical SciencesScientific and technological advances allowed the extraction of a growing quantity of knowledge from the analysed samples by means of analytical techniques. Over the last few years, the dimensionality of data that the most recent analytical techniques produce is so high, that its analysis is now called megavariate analysis. Recently, the usage of machine learning tools in chemical data analysis have allowed the extraction of relevant information from samples at a level which, until then, would just not be possible. The objective of this work consists in classifying manufacturing conditions of printed circuit boards based on data acquired by SLE-HPLC-ESI-MS. As such, this dissertation is divided in two parts: the first synthesizes the work taken to assure the analytical method produces data with adequate quality in such a way the second part shows the development of predictive model using the previous acquired data. At the same time, a data augmentation technique which, to the best of our knowledge, constitutes the first time a data augmentation technique for classification problems using chromatographic data, has been developed. Best models’ results show precisions above 94% for all manufacturing conditions prediction. Moreover, the developed data augmentation technique reports superior performances when compared to three other data augmentation techniques. In summary, the results show that, besides distinguishing classes with different chemical compositions, it is possible to obtain information about which are the chemical compounds that differentiate the classes. This information might be of significant importance for areas such as quality control, food chemistry, botany and pharmaceutical industry.O constante avanço científico-tecnológico permitiu que, ao longo do último século, as técnicas de análise química extraíssem cada vez mais conhecimento das amostras analisadas. Nos últimos anos, a quantidade de dados que as mais recentes técnicas analíticas produzem possui uma dimensão tão elevada que a sua análise é denominada de análise megavariacional. Recentemente, a aplicação de ferramentas de machine learning em análises de dados químicos tem permitido extrair informação relevante das amostras analisadas que até recentemente não era possível. Com isto em mente, o objetivo deste trabalho consiste em classificar condições de manufatura de placas de circuito impresso tendo por base dados provenientes de análise por cromatografia líquida acoplada a espetrometria de massa com extração sólido-líquido. Desta forma, esta dissertação está dividida em duas partes: a primeira sintetiza o trabalho efetuado para garantir que o método de análise produz dados com qualidade adequada para que na segunda parte esses dados sejam usados para construir modelos preditivos. Paralelamente, foi desenvolvida uma técnica de aumento de dados que, até onde o nosso conhecimento vai, constitui a primeira técnica de aumento de dados desenvolvida para problemas de classificação com dados provenientes de análises cromatográficas. Os resultados dos melhores modelos mostram precisões superiores a 94% para a previsão de todas as condições de manufatura. Adicionalmente, a técnica de aumento de dados desenvolvida mostra desempenhos superiores comparativamente a outras técnicas de aumento de dados. Em síntese, os resultados obtidos indicam que, para além de distinguir classes com composições químicas diferentes, é possível adquirir informação sobre quais são os compostos químicos que distinguem as classes em estudo. Esta informação pode vir a ter uma importância significativa em áreas como controlo de qualidade, química alimentar e indústria fito-farmacêutica.Fundação para a Ciência e Tecnologia através do projeto POCI-01-0145-FEDER-029147 - PTDC/FIS-PAR/29147/2017 financiado por: OE/FCT, Lisboa 2020, Compete 2020 POCI, Portugal 2020 FEDE

Real-time Knowledge-based Fuzzy Logic Model for Soft Tissue Deformation

In this research, the improved mass spring model is presented to simulate the human liver deformation. The underlying MSM is redesigned where fuzzy knowledge-based approaches are implemented to determine the stiffness values. Results show that fuzzy approaches are in very good agreement to the benchmark model. The novelty of this research is that for liver deformation in particular, no specific contributions in the literature exist reporting on real-time knowledge-based fuzzy MSM for liver deformation

Use of mathematical methods in the resolution of chemical engineering problems

This thesis consists of a compendium of five works that illustrate the utilization of selected mathematical methods to solve specific chemical engineering problems. Hence, the thesis is intended to cover both, a review of fundamental mathematical procedures for the solution of models raised from chemical phenomena, and a demonstration of their effectiveness to obtain useful novel significant results. The opening paper explores diverse global optimization algorithms to adjust both kinetic constants and the binary interaction parameters (BIPs) for the Peng-Robinson equation of state to the experimental data. Those parameters are essential to determine the model raised from the supercritical transesterification of triolein with methanol to produce biodiesel, with CO2 as cosolvent, consisting of three reversible reaction in series. Here, a novel model merging the ordinary differential equations system raised from kinetic mechanism and the time-dependent thermodynamic state of the complex mixture is presented for diverse operating conditions. Among all results obtained, novel binary interaction coefficients for the intermediate reaction species (dioleins and monooleins) highlight. The second and fourth papers included in this thesis are aimed at the study of lanolin extraction from raw wool, using 5% ethanol in CO2. The former explores solid lanolin extraction under near-critical conditions by means of a mass-transfer model based on the shrinking-core concept, while the latter is addressed at the liquid lanolin supercritical extraction. Both models result in a partial differential equations (PDEs) system determined by the solubility of multiphasic lanolin, Henry-type partition coefficient and the lanolin mass transfer coefficient. Hence, in each paper the raised PDEs system is solved through a different method: in the second paper orthogonal collocation method is employed, while in the fourth paper finite differences method is used combined with the numerical integration of an expression previously obtained by means of the Laplace transform. Finally, an optimization procedure is used in order to fit the extraction parameters to the experimental data, achieving coherent results that agree well with those previously reported. Between the cases exposed, liquid lanolin extraction is significantly complex to model because of the diffusion phenomena that may occur inside the two lanolin fraction mixture added to the diffusion of solvent in the interphase. Therefore, in the third work a nonlinear autoregressive exogenous neural network model is designed to predict the outcoming extracted fraction of lanolin at diverse temperatures, pressures, solvent mass flow rates, wool packing densities and times. The problem with the scarce data available for training of the neural network is overcome by augmenting experimental data using an empirical Weibull function, which correctly predicts the lanolin breakthrough at the extractor exit. This hybrid Weibull - Neural Network algorithm results in a low prediction error and conform a powerful tool for optimizing operating conditions, proved by the fast convergence of genetic algorithm procedure. This thesis closes with Molecular Dynamics simulations for peptide-folding studies, followed by a Principal Component Analysis (PCA) and clustering analysis to understand the Free Energy Landscape of the peptide (FEL). Those methods are aimed at assessing the conformational profile of bombesin, a peptide with interest in drug design as a possible novel agonist and/or antagonist in the fight against cancer. Results suggest that the peptide adopts mainly helical structures at the C-terminus and, to a lesser extent, hairpin turn structures at the N-terminus. Those results agree with those available from NMR in a 2,2,2-trifluoroethanol/water (30% v/v), and point out a suitable a-helix conformation for binding where Trp8 and His12 interaction has a significant role.Aquesta tesi consta d'un compendi de cinc treballs que il·lustren la utilització de mètodes matemàtics per resoldre problemes específics d'enginyeria química. Per tant, la tesi està destinada a ser una revisió dels procediments matemàtics fonamentals per a la solució de models derivats de fenòmens químics i, a més, una demostració de la seva efectivitat per obtenir resultats útils i innovadors. L'article que obre la tesi explora diferents algoritmes d'optimització global per ajustar tant les constants cinètiques com el Paràmetres d'Interacció Binària (PIB) per a l'equació d'estat de Peng Robinsos a les dades experimentals. Aquests paràmetres són essencials per determinar el model derivat de la transesterificació supercrítica de la trioleïna amb metanol per produir biodièsel, amb CO2 com a cosolvent, que consisteix en tres reaccions reversibles en sèrie. Aquí, es presenta un nou model que fusiona el sistema d'EDOs derivat del mecanisme cinètic i l'estat termodinàmic de la barreja per a condicions de funcionament diverses. Entre tots els resultats obtinguts, destaquen els nous PIBs trobats per a les espècies de reacció intermèdies. El segon i quart treball inclosos en aquesta tesi estan destinats a l'estudi de l'extracció de lanolina de llana crua amb 5% d'etanol en CO2. El primer explora l'extracció de lanolina sòlida en condicions gairebé crítiques mitjançant un model de transferència de massa basat en el concepte del nucli minvant, mentre que el segon s'adreça al cas de l'extracció supercrítica de lanolina líquida. Ambdós models donen com a resultat un sistema d'EDPs determinat per la solubilitat de la lanolina multifàsica, el coeficient de partició de Henry i el coeficient de transferències de massa. Per tant, a cada article el sistema d'EDPs obtingut es resol mitjançant un mètode diferent: en el article s'utilitza un mètode de col·laboració ortogonal, mentre que en el quart s'utilitza el mètode de diferències finites combinat amb la integració numèrica d'una expressió obtinguda mitjançant la Transformada de Laplace. Finalment, es porta a terme una optimització per ajustar els paràmetres d'extracció a les dades experimentals, aconseguint resultats coherents que coincideixen amb els reportats anteriorment. Entre els casos expotsats, l'extracció de lanolina líquida és significativament complexa de modelar a causa dels fenòmens de difusió que es poden produir a l'interior de les dues fraccions de lanolina a més de la difusió del dissolvent en la interfase. Per tant, en el tercer treball es dissenya un model de xarxa neuronal exògena no lineal autoregressiva per predir la fracció extreta de lanlina a diverses temperatures, pressions, cabals de dissolvent, densitats d'empaquetament i temps. El problema derivat de l'escassetat de dades disponibles per a l'entrenament de la xarxa neuronal es supera amb l'augment d'aquestes mitjançant una funció de Weibull empírica, que prediu correctament l'avanç de la lanolina a la sortida de l'extractor. Aquest algoritme híbrid Weibull - xarxa neuronal resulta en un baix error de predicció i conforma una potent eina per optimitzar les condicions operatives, demostrada per la ràpida convergència de l'algoritme genètic utilitzat. Aquesta tesi tanca amb simulacions de Dinàmica Molecular per a l'estudi del plegament de pèptids seguint d'un Anàlisi de Components Principals (ACP) i del "clustering" per a l'anàlisi del Paisatge d'Energia Lliure (PEL). L'objectiu és avaluar el perfil conformacional de la bombesina, un pèptid amb interès en el disseny de fàrmacs com a possible nou agonista i/o antagonista en la lluita contra el càncer. Els resultats suggereixen que el pèptid adopta estructures helicoïdals principalment al extrem C, i també en menor mesura estructures de forquilla al extrem N. Aquests resultats coincideixen amb els disponibles de RMN en 2,2-trifluoroetanol/aigua (30% v/v) i indiquen una conformació d’hèlix a adequada per a la unió on la interacció Trp8 i His12 té un paper important