Recursive Principal Components Analysis

A recurrent linear network can be trained with Oja's constrained Hebbian learning rule. As a result, the network learns to represent the temporal context associated to its input sequence. The operation performed by the network is a generalization of Principal Components Analysis (PCA) to time-series, called Recursive PCA. The representations learned by the network are adapted to the temporal statistics of the input. Moreover, sequences stored in the network may be retrieved explicitly, in the reverse order of presentation, thus providing a straight-forward neural implementation of a logical stack

Short-Term Memory Optimization in Recurrent Neural Networks by Autoencoder-based Initialization

Training RNNs to learn long-term dependencies is difficult due to vanishing gradients. We explore an alternative solution based on explicit memorization using linear autoencoders for sequences, which allows to maximize the short-term memory and that can be solved with a closed-form solution without backpropagation. We introduce an initialization schema that pretrains the weights of a recurrent neural network to approximate the linear autoencoder of the input sequences and we show how such pretraining can better support solving hard classification tasks with long sequences. We test our approach on sequential and permuted MNIST. We show that the proposed approach achieves a much lower reconstruction error for long sequences and a better gradient propagation during the finetuning phase.Comment: Accepted at NeurIPS 2020 workshop "Beyond Backpropagation: Novel Ideas for Training Neural Architectures

Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory

The effectiveness of recurrent neural networks can be largely influenced by their ability to store into their dynamical memory information extracted from input sequences at different frequencies and timescales. Such a feature can be introduced into a neural architecture by an appropriate modularization of the dynamic memory. In this paper we propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. First, we show how to extend the architecture of a simple RNN by separating its hidden state into different modules, each subsampling the network hidden activations at different frequencies. Then, we discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies. Each new module works at a slower frequency than the previous ones and it is initialized to encode the subsampled sequence of hidden activations. Experimental results on synthetic and real-world datasets on speech recognition and handwritten characters show that the modular architecture and the incremental training algorithm improve the ability of recurrent neural networks to capture long-term dependencies.Comment: accepted @ ECML 2020. arXiv admin note: substantial text overlap with arXiv:2001.1177

Fault detection and monitoring system using enhanced principal component analysis for the application in wastewater treatment plant

Fault detection and monitoring is essentially important in wastewater treatment to ensure that safety, environmental regulations compliance, maintenance and operation of the Wastewater Treatment Plant (WWTP) are under control. Many researchers have developed methods in fault detection and monitoring such as fuzzy logic, parameter estimation, neural network and Principal Component Analysis (PCA). In studies involving data and signal model approach, PCA is the most appropriate method used in this work. Besides when using PCA, the dimensionality of the data, noise and redundancy can be reduce. However, PCA is only suitable for data with mean constant or steady state data. The use of PCA can also increase false alarm and produce false fault in a plant such as WWTP. Modifications of PCA need to be done to overcome the problems and hence, enhanced methods of PCA are proposed in this work. The enhanced methods are Multiscale PCA (MSPCA) and Recursive PCA (RPCA), which are appropriate for offline monitoring test and online monitoring test, respectively. To see the effectiveness of the methods, they were applied into the european Co-operation in the field of Scientific and Technical Research (COST) simulation benchmark WWTP. The results from the simulation plant were then applied in a real WWTP, IWK Bunus Regional Sewage Treatment Plant (RSTP). The data of WWTP involved are Dissolved Oxygen (DO), Biochemical Oxygen Demand (BOD), Chemical Oxygen Demand (COD) and Nitrate (SNO). In analysis for both plants, faults were detected when the confidence limit is over 95% and confidence limits in the range of 90-95% were considered for alarm region in the data, using Hotelling's T2 and residual. Finally, simulation results of the proposed methods were compared and it was found that the enhanced methods of PCA (MSPCA and RPCA) were able to reduce false alarm and false fault in the analysis of fault detection by 70% for steady state influence and dynamic influence and hence provides more accurate results in detecting faults in the process data

Efficient nonlinear Bayesian survey design using D-N optimization

A new method for fully nonlinear, Bayesian survey design renders the optimization of industrial-scale geoscientific surveys as a practical possibility. The method, DNoptimization, designs surveys to maximally discriminate between different possible models. It is based on a generalization to nonlinear design problems of the D criterion (which is for linearized design problems). The main practical advantage of DNoptimization is that it uses efficient algorithms developed originally for linearized design theory, resulting in lower computing and storage costs than for other nonlinear Bayesian design techniques. In a real example in which we optimized a seafloor microseismic sensor network to monitor a fractured petroleum reservoir, we compared DNoptimization with two other networks: one proposed by an industrial contractor and one optimized using a linearized Bayesian design method. Our technique yielded a network with superior expected data quality in terms of reduced uncertainties on hypocenter locations.</jats:p

Data Analytics and Performance Enhancement in Edge-Cloud Collaborative Internet of Things Systems

Based on the evolving communications, computing and embedded systems technologies, Internet of Things (IoT) systems can interconnect not only physical users and devices but also virtual services and objects, which have already been applied to many different application scenarios, such as smart home, smart healthcare, and intelligent transportation. With the rapid development, the number of involving devices increases tremendously. The huge number of devices and correspondingly generated data bring critical challenges to the IoT systems. To enhance the overall performance, this thesis aims to address the related technical issues on IoT data processing and physical topology discovery of the subnets self-organized by IoT devices. First of all, the issues on outlier detection and data aggregation are addressed through the development of recursive principal component analysis (R-PCA) based data analysis framework. The framework is developed in a cluster-based structure to fully exploit the spatial correlation of IoT data. Specifically, the sensing devices are gathered into clusters based on spatial data correlation. Edge devices are assigned to the clusters for the R-PCA based outlier detection and data aggregation. The outlier-free and aggregated data are forwarded to the remote cloud server for data reconstruction and storage. Moreover, a data reduction scheme is further proposed to relieve the burden on the trunk link for data uploading by utilizing the temporal data correlation. Kalman filters (KFs) with identical parameters are maintained at the edge and cloud for data prediction. The amount of data uploading is reduced by using the data predicted by the KF in the cloud instead of uploading all the practically measured data. Furthermore, an unmanned aerial vehicle (UAV) assisted IoT system is particularly designed for large-scale monitoring. Wireless sensor nodes are flexibly deployed for environmental sensing and self-organized into wireless sensor networks (WSNs). A physical topology discovery scheme is proposed to construct the physical topology of WSNs in the cloud server to facilitate performance optimization, where the physical topology indicates both the logical connectivity statuses of WSNs and the physical locations of WSN nodes. The physical topology discovery scheme is implemented through the newly developed parallel Metropolis-Hastings random walk based information sampling and network-wide 3D localization algorithms, where UAVs are served as the mobile edge devices and anchor nodes. Based on the physical topology constructed in the cloud, a UAV-enabled spatial data sampling scheme is further proposed to efficiently sample data from the monitoring area by using denoising autoencoder (DAE). By deploying the encoder of DAE at the UAV and decoder in the cloud, the data can be partially sampled from the sensing field and accurately reconstructed in the cloud. In the final part of the thesis, a novel autoencoder (AE) neural network based data outlier detection algorithm is proposed, where both encoder and decoder of AE are deployed at the edge devices. Data outliers can be accurately detected by the large fluctuations in the squared error generated by the data passing through the encoder and decoder of the AE

Reflectance spectrum analysis of mineral flotation froths and slurries

The global demand of mining products has increased during recent years, and there is pressure to improve the efficiency of mines and concentration processes. This thesis focuses on froth flotation, which is one of the most common concentration methods in mineral engineering. Froth flotation is used to separate valuable minerals from mined ore that has been crushed, mixed with water and ground to a small particle size. The separation is based on differences in the surface chemical properties of the minerals. Monitoring and control of flotation processes mainly relies on the on-line analysis of the process slurry streams. Traditionally, the analysis is performed using X-ray fluorescence (XRF) analyzers that measure the elemental contents of the solids in the slurries. The thesis investigates the application of visual and near-infrared (VNIR) reflectance spectroscopy to improve the on-line analysis of mineral flotation froths and slurries. In reflectance spectroscopy the sample is illuminated and the spectrum of the reflected light is captured by a spectrograph. The main benefits of VNIR reflectance spectroscopy with respect to XRF-based analysis are the relatively low cost of the equipment required and the easy and fast measurement process. As a consequence, the sampling rate of the reflectance spectrum measurement is radically faster than in the XRF analysis. Data-based modeling is applied to the measured VNIR spectra to calculate the corresponding elemental contents. The research is conducted at a real copper and zinc flotation process. The main results of the thesis show that VNIR reflectance spectroscopy can be used to measure temporal changes in the elemental contents of mineral flotation froths and slurries in the analyzed process. Especially the slurry measurements from the final concentrates provide accurate information on the slurry contents. A multi-channel slurry VNIR analyzer prototype is developed in this thesis. When combined with an XRF analyzer, it is able to measure the slurry lines with a very fast sampling rate. This considerably improves the monitoring and control possibilities of the flotation process. The proposed VNIR analyzer is adaptively calibrated with the sparse XRF measurements to compensate for the effect of changes in other slurry properties. The high-frequency slurry analysis is shown to reveal fast grade changes and grade oscillations that the XRF analyzer is unable to detect alone. Based on the new measurement, a plant-wide study of the harmful grade oscillations is conducted in order to improve the performance of the flotation process

Planification de mouvement pour systèmes anthropomorphes

L'objet de cette thèse est le développement et l'étude des algorithmes de planification de mouvement pour les systèmes hautement dimensionnés que sont les robots humanoïdes et les acteurs virtuels. Plusieurs adaptations des méthodes génériques de planification de mouvement randomisées sont proposées et discutées. Une première contribution concerne l'utilisation de techniques de réduction de dimension linéaire pour accélérer les algorithmes d'échantillonnage. Cette méthode permet d'identifier en ligne quand un processus de planification passe par un passage étroit de l'espace des configurations et adapte l'exploration en fonction. Cet algorithme convient particulièrement bien aux problèmes difficiles de la planification de mouvement pour l'animation graphique. La deuxième contribution est le développement d'algorithmes randomisés de planification sous contraintes. Il s'agit d'une intégration d'outils de cinématique inverse hiérarchisée aux algorithmes de planification de mouvement randomisés. On illustre cette méthode sur différents problèmes de manipulation pour robots humanoïdes. Cette contribution est généralisée à la planification de mouvements corps-complet nécessitant de la marche. La dernière contribution présentée dans cette thèse est l'utilisation des méthodes précédentes pour résoudre des tâches de manipulation complexes par un robot humanoïde. Nous présentons en particulier un formalisme destiné à représenter les informations propres à l'objet manipulé utilisables par un planificateur de mouvement. Ce formalisme est présenté sous le nom d'« objets documentés ». ABSTRACT : This thesis deals with the development and analysis of motion planning algorithms for high dimensional systems: humanoid robots and digital actors. Several adaptations of generic randomized motion planning methods are proposed and discussed. A first contribution concerns the use of linear dimensionality reduction techniques to speed up sampling algorithms. This method identifies on line when a planning process goes through a narrow passage of some configuration space, and adapts the exploration accordingly. This algorithm is particularly suited to difficult problems of motion planning for computer animation. The second contribution is the development of randomized algorithms for motion planning under constraints. It consists in the integration of prioritized inverse kinematics tools within randomized motion planning. We demonstrate the use of this method on different manipulation planning problems for humanoid robots. This contribution is generalized to whole-body motion planning with locomotion. The last contribution of this thesis is the use of previous methods to solve complex manipulation tasks by humanoid robots. More specifically, we present a formalism that represents information specific to a manipulated object usable by a motion planner. This formalism is presented under the name of "documented object"