Continual Learning of Multi-modal Dynamics with External Memory

We study the problem of fitting a model to a dynamical environment when new modes of behavior emerge sequentially. The learning model is aware when a new mode appears, but it does not have access to the true modes of individual training sequences. We devise a novel continual learning method that maintains a descriptor of the mode of an encountered sequence in a neural episodic memory. We employ a Dirichlet Process prior on the attention weights of the memory to foster efficient storage of the mode descriptors. Our method performs continual learning by transferring knowledge across tasks by retrieving the descriptors of similar modes of past tasks to the mode of a current sequence and feeding this descriptor into its transition kernel as control input. We observe the continual learning performance of our method to compare favorably to the mainstream parameter transfer approach

Damage classification after the 2009 L'Aquila earthquake using multinomial logistic regression and neural networks

Post-earthquake surveys represent a fundamental tool for managing the emergency phase after a strong earthquake. In Italy, the evaluation of the post-earthquake functionality of ordinary buildings is based on the AeDES forms (Agibilitá e Danno nell'Emergenza Sismica, or equivalently, Rapid Post-Earthquake Damage evaluation forms). This form includes information on the building and records of the observed damage classified according to type and intensity in 60 subclasses. Based on the observed damage and expert judgment, the buildings are clustered into six risk classes, from A to F. The assigned class is used to calculate the maximum economic reimbursement owed for the reconstruction or repair of the building. However, often the cluster assignment is not entirely objective due to the inherent responsibility associated with a less conservative assessment. This paper uses the data from the 2009 L'Aquila earthquake to develop classification models based on multinomial logistic regression (MLR) and artificial neural networks (ANN) calibrated with data theoretically less influenced by personal biases. The proposed models, particularly the MLR, are intended to support the decision-making of the evaluation team in future updates of the AeDES forms. This approach cannot substitute expert evaluation, which is always necessary for complex scenarios but may mitigate the impact of subjectivity and can provide an indication of the expected outcome of the survey

Mitigation of Catastrophic Interference in Neural Networks and Ensembles using a Fixed Expansion Layer

Catastrophic forgetting (also known in the literature as catastrophic interference) is the phenomenon by which learning systems exhibit a severe exponential loss of learned information when exposed to relatively small amounts of new training data. This loss of information is not caused by constraints due to the lack of resources available to the learning system, but rather is caused by representational overlap within the learning system and by side-effects of the training methods used. Catastrophic forgetting in auto-associative pattern recognition is a well-studied attribute of most parameterized supervised learning systems. A variation of this phenomenon, in the context of feedforward neural networks, arises when non-stationary inputs lead to loss of previously learned mappings. The majority of the schemes proposed in the literature for mitigating catastrophic forgetting are not data-driven, but rather rely on storage of prior representations of the learning system. We introduce the Fixed Expansion Layer (FEL) feedforward neural network that embeds an expansion layer which sparsely encodes the information contained within the hidden layer, in order to help mitigate forgetting of prior learned representations. The fixed expansion layer approach is generally applicable to feedforward neural networks, as demonstrated by the application of the FEL technique to a recurrent neural network algorithm built on top of a standard feedforward neural network. Additionally, we investigate a novel framework for training ensembles of FEL networks, based on exploiting an information-theoretic measure of diversity between FEL learners, to further control undesired plasticity. The proposed methodology is demonstrated on a several tasks, clearly emphasizing its advantages over existing techniques. The architecture proposed can be applied to address a range of computational intelligence tasks, including classification problems, regression problems and system control

An Artificial Neural Network to Infer the Mediterranean 3D Chlorophyll-a and Temperature Fields from Remote Sensing Observations

Remote sensing data provide a huge number of sea surface observations, but cannot give direct information on deeper ocean layers, which can only be provided by sparse in situ data. The combination of measurements collected by satellite and in situ sensors represents one of the most effective strategies to improve our knowledge of the interior structure of the ocean ecosystems. In this work, we describe a Multi-Layer-Perceptron (MLP) network designed to reconstruct the 3D fields of ocean temperature and chlorophyll-a concentration, two variables of primary importance for many upper-ocean bio-physical processes. Artificial neural networks can efficiently model eventual non-linear relationships among input variables, and the choice of the predictors is thus crucial to build an accurate model. Here, concurrent temperature and chlorophyll-a in situ profiles and several different combinations of satellite-derived surface predictors are used to identify the optimal model configuration, focusing on the Mediterranean Sea. The lowest errors are obtained when taking in input surface chlorophyll-a, temperature, and altimeter-derived absolute dynamic topography and surface geostrophic velocity components. Network training and test validations give comparable results, significantly improving with respect to Mediterranean climatological data (MEDATLAS). 3D fields are then also reconstructed from full basin 2D satellite monthly climatologies (1998–2015) and resulting 3D seasonal patterns are analyzed. The method accurately infers the vertical shape of temperature and chlorophyll-a profiles and their spatial and temporal variability. It thus represents an effective tool to overcome the in-situ data sparseness and the limits of satellite observations, also potentially suitable for the initialization and validation of bio-geophysical models

Predictive modelling of global solar radiation with artificial intelligence approaches using MODIS satellites and atmospheric reanalysis data for Australia

Global solar radiation (GSR) prediction is a prerequisite task for agricultural management and agronomic decisions, including photovoltaic (PV) power generation, biofuel exploration and several other bio-physical applications. Since short-term variabilities in the GSR incorporate stochastic and intermittent behaviours (such as periodic fluctuations, jumps and trends) due to the dynamicity of atmospheric variables, GSR predictions, as required for solar energy generation, is a challenging endeavour to satisfactorily predict the solar generated electricity in a PV system. Additionally, the solar radiation data, as required for solar energy monitoring purposes, are not available in all geographic locations due to the absence of meteorological stations and this is especially true for remote and regional solar powered sites. To surmount these challenges, the universally (and freely available) atmospheric gridded datasets (e.g., reanalysis and satellite variables) integrated into solar radiation predictive models to generate reliable GSR predictions can be considered as a viable medium for future solar energy exploration, utilisation and management. Hence, this doctoral thesis aims to design and evaluate novel Artificial Intelligence (AI; Machine Learning and Deep Learning) based predictive models for GSR predictions, using the European Centre for Medium Range Weather Forecasting (ECMWF) Interim-ERA reanalysis and Moderate Resolution Imaging Spectroradiometer (MODIS) Satellite variables enriched with ground-based weather station datasets for the prediction of both long-term (i.e., monthly averaged daily) as well as the short-term (i.e., daily and half-hourly) GSR. The focus of the study region is Queensland, the sunshine state, as well as a number of major solar cities in Australia where solar energy utilisation is actively being promoted by the Australian State and Federal Government agencies. Firstly, the Artificial Neural Networks (ANN), a widely used Machine Learning model is implemented to predict daily GSR at five different cities in Australia using ECMWF Reanalysis fields obtained from the European Centre for Medium Range Weather Forecasting repository. Secondly, the Self-Adaptive Differential Evolutionary Extreme Learning Machine (i.e., SaDE-ELM) is also proposed for monthly averaged daily GSR prediction trained with ECMWF reanalysis and MODIS satellite data from the Moderate Resolution Imaging Spectroradiometer. Thirdly, a three-phase Support Vector Regression (SVR; Machine Learning) model is developed to predict monthly averaged daily GSR prediction where the MODIS data are used to train and evaluate the model and the Particle Swarm Algorithm (PSO) is used as an input selection algorithm. The PSO selected inputs are further transformed into wavelet subseries via non-decimated Discrete Wavelet Transform to unveil the embedded features leading to a hybrid PSO-W-SVR model, seen to outperform the comparative hybrid models. Fourthly, to improve the accuracy of conventional techniques adopted for GSR prediction, Deep Learning (DL) approach based on Deep Belief Network (DBN) and Deep Neural Network (DNN) algorithms are developed to predict the monthly averaged daily GSR prediction using MODIS-based dataset. Finally, the Convolutional Neural Network (CNN) integrated with a Long Short-Term Memory Network (LSTM) model is used to construct a hybrid CLSTM model which is tested to predict the half-hourly GSR values over multiple time-step horizons (i.e., 1-Day, 1-Week, 2-Week, and 1-Month periods). Here, several statistical, Machine Learning and Deep Learning models are adopted to benchmark the proposed DNN and CLSTM models against conventional models (ANN, SaDE-ELM, SVR, DBN). In this doctoral research thesis, a Global Sensitivity Analysis method that attempts to utilise the Gaussian Emulation Machine (GEM-SA) algorithm is employed for a sensitivity analysis of the model predictors. Sensitivity analysis of selected predictors ascertains that the variables: aerosol, cloud, and water vapour parameters used as input parameters for GSR prediction play a significant role and the most important predictors are seen to vary with the geographic location of the tested study site. A suite of alternative models are also developed to evaluate the input datasets classified into El Niño, La Niña and the positive and negative phases of the Indian Ocean Dipole moment. This considers the impact of synoptic-scale climate phenomenon on long-term GSR predictions. A seasonal analysis of models applied at the tested study sites showed that proposed predictive models are an ideal tool over several other comparative models used for GSR prediction. This study also ascertains that an Artificial Intelligence based predictive model integrated with ECMWF reanalysis and MODIS satellite data incorporating physical interactions of the GSR (and its variability) with the other important atmospheric variables can be considered to be an efficient method to predict GSR. In terms of their practical use, the models developed can be used to assist with solar energy modelling and monitoring in solar-rich sites that have diverse climatic conditions, to further support cleaner energy utilization. The outcomes of this doctoral research program are expected to lead to new applications of Artificial Intelligence based predictive tools for GSR prediction, as these tools are able to capture the non-linear relationships between the predictor and the target variable (GSR). The Artificial Intelligence models can therefore assist climate adaptation and energy policymakers to devise new energy management devices not only for Australia but also globally, to enable optimal management of solar energy resources and promote renewable energy to combat current issues of climate change. Additionally, the proposed predictive models may also be applied to other renewable energy areas such as wind, drought, streamflow, flood and electricity demand for prediction

CapsLoc3D: Point Cloud Retrieval for Large-Scale Place Recognition Based on 3D Capsule Networks

Point cloud-based place recognition can be used for global localization in large-scale scenes and loop-closure detection in simultaneous localization and mapping (SLAM) systems in the absence of GPS. Current learning-based approaches aim to extract global and local features from 3D point clouds to encode them as descriptors for point cloud retrieval. The key problems are that the occlusion of point clouds by dynamic objects in the scene affects the point cloud structure, a single perceptual field of the network cannot adequately extract point cloud features, and the correlation between features is not fully utilized. To overcome this, we propose a novel network called CapsLoc3D. We first use the static point cloud generation module to remove the occlusion effects of dynamic objects, and then obtain the point cloud descriptors by processing with the CapsLoc3D network which contains the point spatial transformation module, multi-scale feature fusion module, Capsnet module and a GeM Pooling layer. After validation using the Oxford RobotCar, KITTI, and NEU datasets, experiments show that our method performs better and also has good generalization performance and computational efficiency compared with current state-of-the-art algorithms

A probabilistic neural network computer vision system for corn kernel damage evaluation

An investigation was conducted to determine whether image processing and machine vision technology could be used for identification of the damage factor in corn kernels. Prominent types of corn kernel damage were found to be germ damage and blue-eye mold damage. A sample set containing 720 kernels with approximately equal numbers of blue-eye mold-damaged, germ-damaged, and sound kernels was obtained and evaluated by human inspectors and the computer vision system. While the computer vision system developed was slightly less consistent in classification than trained human inspectors, it did prove to be a promising step toward inspection automation;Two probabilistic neural network architectures were implemented. The first network, based on a universal smoothing factor algorithm, was used to segment the collected images into blue-eye mold-damaged, germ-damaged, sound germ, shadow in sound germ, hard starch, and soft starch areas. Morphological features from each of the segmented areas were then input to a second probabilistic neural network which used genetic algorithms to optimize a unique smoothing factor for each network input. Output of the second layer network was overall kernel classification of blue-eye mold-damaged, germ-damaged, and sound. Overall accuracy of classification on unseen images was 78%, 94%, and 93% for blue-eye mold-damaged, germ-damaged, and sound categories, respectively. Correct classification for sound and damaged categories on unseen images was 92% and 93%, respectively

Dynamic data driven investigation of petrophysical and geomechanical properties for reservoir formation evaluation

Petrophysical and geomechanical properties of the formation such as Young’s modulus, bulk modulus, shear modulus, Poisson’s ratio, and porosity provide characteristic description of the hydrocarbon reservoir. It is well-established that static geomechanical properties are good representatives of reservoir formations; however, they are non-continuous along the wellbore, expensive and determining these properties may lead to formation damage. Dynamic geomechanical formation properties from acoustic measurements offer a continuous and non-destructive means to provide a characteristic description of the reservoir formation. In the absence of reliable acoustic measurements of the formation, such as sonic logs, the estimation of the dynamic geomechanical properties becomes challenging. Several techniques like empirical, analytical and intelligent systems have been used to approximate the property estimates. These techniques can also be used to approximate acoustic measurements thus enable dynamic estimation of geomechanical properties. This study intends to explore methodologies and models to dynamically estimate geomechanical properties in the absence of some or all acoustic measurements of the formation. The present work focused on developing empirical and intelligent systems like artificial neural networks (ANN), Gaussian processes (GP), and recurrent neural networks (RNN) to determine the dynamic geomechanical properties. The developed models serve as a cost-effective, reliable, efficient, and robust methods, offering dyanmic geomechanical analysis of the formation. This thesis has five main contributions: (a) a new data-driven empirical model of estimating static Young’s modulus from dynamic Young’s modulus, (b) a new data-driven ANN model for sonic well log prediction, (c) a new data-driven GP model for shear wave transit time prediction, (d) a new dynamic data-driven RNN model for sonic well log reproduction, and (e) an assessment on the ANN as a reliable sonic logging tool