Advances in Extreme Learning Machines

Nowadays, due to advances in technology, data is generated at an incredible pace, resulting in large data sets of ever-increasing size and dimensionality. Therefore, it is important to have efficient computational methods and machine learning algorithms that can handle such large data sets, such that they may be analyzed in reasonable time. One particular approach that has gained popularity in recent years is the Extreme Learning Machine (ELM), which is the name given to neural networks that employ randomization in their hidden layer, and that can be trained efficiently. This dissertation introduces several machine learning methods based on Extreme Learning Machines (ELMs) aimed at dealing with the challenges that modern data sets pose. The contributions follow three main directions.    Firstly, ensemble approaches based on ELM are developed, which adapt to context and can scale to large data. Due to their stochastic nature, different ELMs tend to make different mistakes when modeling data. This independence of their errors makes them good candidates for combining them in an ensemble model, which averages out these errors and results in a more accurate model. Adaptivity to a changing environment is introduced by adapting the linear combination of the models based on accuracy of the individual models over time. Scalability is achieved by exploiting the modularity of the ensemble model, and evaluating the models in parallel on multiple processor cores and graphics processor units. Secondly, the dissertation develops variable selection approaches based on ELM and Delta Test, that result in more accurate and efficient models. Scalability of variable selection using Delta Test is again achieved by accelerating it on GPU. Furthermore, a new variable selection method based on ELM is introduced, and shown to be a competitive alternative to other variable selection methods. Besides explicit variable selection methods, also a new weight scheme based on binary/ternary weights is developed for ELM. This weight scheme is shown to perform implicit variable selection, and results in increased robustness and accuracy at no increase in computational cost. Finally, the dissertation develops training algorithms for ELM that allow for a flexible trade-off between accuracy and computational time. The Compressive ELM is introduced, which allows for training the ELM in a reduced feature space. By selecting the dimension of the feature space, the practitioner can trade off accuracy for speed as required.    Overall, the resulting collection of proposed methods provides an efficient, accurate and flexible framework for solving large-scale supervised learning problems. The proposed methods are not limited to the particular types of ELMs and contexts in which they have been tested, and can easily be incorporated in new contexts and models

High-performance ensembles of the Online Sequential Extreme Learning Machine algorithm for regression and time series forecasting

Orientador: André Leon Sampaio GradvohlDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de TecnologiaResumo: As ferramentas baseadas em aprendizado de máquina têm sido utilizadas para previsão em séries temporais, devido à sua capacidade de identificar relações nos conjuntos de dados sem serem programadas explicitamente para isto. Algumas séries temporais podem ser caracterizadas como fluxos de dados, e consequentemente podem apresentar desvios de conceito, o que traz alguns desafios a mais para as técnicas tradicionais de aprendizado de máquina. A utilização de técnicas de aprendizado online, como os algoritmos e ensembles derivados do Online Sequential Extreme Learning Machine são adequados para previsão em fluxo de dados com desvios de conceito. No entanto, as previsões baseadas em fluxos de dados frequentemente possuem uma séria restrição relacionada ao tempo de execução dos algoritmos, devido à alta taxa de entrada das amostras. O objetivo deste trabalho foi verificar as acelerações no tempo de execução, proporcionadas pela aplicação de técnicas de computação de alto desempenho no algoritmo Online Sequential Extreme Learning Machine e em três ensembles que o utilizam como base, quando comparadas às respectivas abordagens convencionais. Para tanto, neste trabalho são propostas versões de alto desempenho implementadas em Linguagem C com a biblioteca Intel MKL e com o padrão MPI. A Intel MKL fornece funções que exploram os recursos multithread em processadores com vários núcleos, o que também expande o paralelismo para arquiteturas de multiprocessadores. O MPI permite paralelizar as tarefas com memória distribuída em vários processos, que podem ser alocados em um único nó computacional ou distribuídos por vários nós. Em resumo, a proposta deste trabalho consiste em uma paralelização de dois níveis, onde cada modelo do ensemble é alocado em um processo MPI e as funções internas de cada modelo são paralelizadas em um conjunto de threads por meio da biblioteca Intel MKL. Para os experimentos, foi utilizado um conjunto de dados sintético e outro real com desvios de conceito. Cada conjunto possui em torno de 175.000 instâncias contendo entre 6 e 10 atributos, e um fluxo online foi simulado com cerca de 170.000 instâncias. Os resultados experimentais mostraram que, em geral, os ensembles de alto desempenho melhoraram o tempo de execução, quando comparados com sua versão serial, com desempenho até 10 vezes mais rápido, mantendo a acurácia das previsões. Os testes foram realizados em três ambientes de alto desempenho distintos e também num ambiente convencional simulando um desktop ou um notebookAbstract: Tools based on machine learning have been used for time series forecasting because of their ability to identify relationships in data sets without being explicitly programmed for it. Some time series can be characterized as data streams, and consequently can present concept drifts, which brings some additional challenges to the traditional techniques of machine learning. The use of online learning techniques, such as algorithms and ensembles derived from the Online Sequential Extreme Learning Machine, are suitable for forecasting data streams with concept drifts. Nevertheless, data streams forecasting often have a serious constraint related to the execution time of the algorithms due to the high incoming samples rate. The objective of this work was to verify the accelerations in the execution time, provided by the adoption of high-performance computing techniques in the Online Sequential Extreme Learning Machine algorithm and in three ensembles that use it as a base, when compared to the respective conventional approaches. For this purpose, we proposed high-performance versions implemented in C programming language with the Intel MKL library and the MPI standard. Intel MKL provides functions that explore the multithread features in multicore CPUs, which expands the parallelism to multiprocessors architectures. MPI allows us to parallelize tasks with distributed memory on several processes, which can be allocated within a single computational node, or distributed over several nodes. In summary, our proposal consists of a two-level parallelization, where we allocated each ensemble model into an MPI process, and we parallelized the internal functions of each model in a set of threads through Intel MKL library. For the experiments, we used a synthetic and a real dataset with concept drifts. Each dataset has around 175,000 instances containing between 6 and 10 attributes, and an online data stream has been simulated with about 170,000 instances. Experimental results showed that, in general, high-performance ensembles improved execution time when compared with its serial version, performing up to 10-fold faster, maintaining the predictions' accuracy. The tests were performed in three distinct high-performance environments and also in a conventional environment simulating a desktop or a notebookMestradoSistemas de Informação e ComunicaçãoMestre em Tecnologi

Towards a more efficient and cost-sensitive extreme learning machine: A state-of-the-art review of recent trend

In spite of the prominence of extreme learning machine model, as well as its excellent features such as insignificant intervention for learning and model tuning, the simplicity of implementation, and high learning speed, which makes it a fascinating alternative method for Artificial Intelligence, including Big Data Analytics, it is still limited in certain aspects. These aspects must be treated to achieve an effective and cost-sensitive model. This review discussed the major drawbacks of ELM, which include difficulty in determination of hidden layer structure, prediction instability and Imbalanced data distributions, the poor capability of sample structure preserving (SSP), and difficulty in accommodating lateral inhibition by direct random feature mapping. Other drawbacks include multi-graph complexity, global memory size, one-by-one or chuck-by-chuck (a block of data), global memory size limitation, and challenges with big data. The recent trend proposed by experts for each drawback is discussed in detail towards achieving an effective and cost-sensitive mode

Efficient multitemporal change detection techniques for hyperspectral images on GPU

Hyperspectral images contain hundreds of reflectance values for each pixel. Detecting regions of change in multiple hyperspectral images of the same scene taken at different times is of widespread interest for a large number of applications. For remote sensing, in particular, a very common application is land-cover analysis. The high dimensionality of the hyperspectral images makes the development of computationally efficient processing schemes critical. This thesis focuses on the development of change detection approaches at object level, based on supervised direct multidate classification, for hyperspectral datasets. The proposed approaches improve the accuracy of current state of the art algorithms and their projection onto Graphics Processing Units (GPUs) allows their execution in real-time scenarios

SYSTEM-ON-A-CHIP (SOC)-BASED HARDWARE ACCELERATION FOR HUMAN ACTION RECOGNITION WITH CORE COMPONENTS

Today, the implementation of machine vision algorithms on embedded platforms or in portable systems is growing rapidly due to the demand for machine vision in daily human life. Among the applications of machine vision, human action and activity recognition has become an active research area, and market demand for providing integrated smart security systems is growing rapidly. Among the available approaches, embedded vision is in the top tier; however, current embedded platforms may not be able to fully exploit the potential performance of machine vision algorithms, especially in terms of low power consumption. Complex algorithms can impose immense computation and communication demands, especially action recognition algorithms, which require various stages of preprocessing, processing and machine learning blocks that need to operate concurrently. The market demands embedded platforms that operate with a power consumption of only a few watts. Attempts have been mad to improve the performance of traditional embedded approaches by adding more powerful processors; this solution may solve the computation problem but increases the power consumption. System-on-a-chip eld-programmable gate arrays (SoC-FPGAs) have emerged as a major architecture approach for improving power eciency while increasing computational performance. In a SoC-FPGA, an embedded processor and an FPGA serving as an accelerator are fabricated in the same die to simultaneously improve power consumption and performance. Still, current SoC-FPGA-based vision implementations either shy away from supporting complex and adaptive vision algorithms or operate at very limited resolutions due to the immense communication and computation demands. The aim of this research is to develop a SoC-based hardware acceleration workflow for the realization of advanced vision algorithms. Hardware acceleration can improve performance for highly complex mathematical calculations or repeated functions. The performance of a SoC system can thus be improved by using hardware acceleration method to accelerate the element that incurs the highest performance overhead. The outcome of this research could be used for the implementation of various vision algorithms, such as face recognition, object detection or object tracking, on embedded platforms. The contributions of SoC-based hardware acceleration for hardware-software codesign platforms include the following: (1) development of frameworks for complex human action recognition in both 2D and 3D; (2) realization of a framework with four main implemented IPs, namely, foreground and background subtraction (foreground probability), human detection, 2D/3D point-of-interest detection and feature extraction, and OS-ELM as a machine learning algorithm for action identication; (3) use of an FPGA-based hardware acceleration method to resolve system bottlenecks and improve system performance; and (4) measurement and analysis of system specications, such as the acceleration factor, power consumption, and resource utilization. Experimental results show that the proposed SoC-based hardware acceleration approach provides better performance in terms of the acceleration factor, resource utilization and power consumption among all recent works. In addition, a comparison of the accuracy of the framework that runs on the proposed embedded platform (SoCFPGA) with the accuracy of other PC-based frameworks shows that the proposed approach outperforms most other approaches

Air quality forecasting using neural networks

In this thesis project, a special type of neural network: Extreme Learning Machine (ELM) is implemented to predict the air quality based on the air quality time series itself and the external meteorological records. A regularized version of ELM with linear components is chosen to be the main model for prediction. To take full advantage of this model, its hyper-parameters are studied and optimized. Then a set of variables is selected (or constructed) to maximize the performance of ELM, where two different variable selection methods (i.e. wrapper and filtering methods) are evaluated. The wrapper method ELM-based forward selection is chosen for the variable selection. Meanwhile, a feature extraction method (Principal Component Analysis) is implemented in the hope of reducing the candidate meteorological variables for feature selection, which proves to be helpful. At last, with all the parameters being properly optimized, ELM is used for the prediction and generates satisfying results

Automatska klasifikacija slika zasnovana na fuziji deskriptora i nadgledanom mašinskom učenju

This thesis investigates possibilities for fusion, i.e. combining of different types of image descriptors, in order to improve accuracy and efficiency of image classification. Broad range of techniques for fusion of color and texture descriptors were analyzed, belonging to two approaches – early fusion and late fusion. Early fusion approach combines descriptors during the extraction phase, while late fusion is based on combining of classification results of independent classifiers. An efficient algorithm for extraction of a compact image descriptor based on early fusion of texture and color information, is proposed in the thesis. Experimental evaluation of the algorithm demonstrated a good compromise between efficiency and accuracy of classification results. Research on the late fusion approach was focused on artificial neural networks and a recently introduced algorithm for extremly fast training of neural networks denoted as Extreme Learning Machines - ELM. Main disadvantages of ELM are insufficient stability and limited accuracy of results. To overcome these problems, a technique for combining results of multiple ELM-s into a single classifier is proposed, based on probability sum rules. The created ensemble of ELM-s has demonstrated significiant improvement of accuracy and stability of results, compared with an individual ELM. In order to additionaly improve classification accuracy, a novel hierarchical method for late fusion of multiple complementary descriptors by using ELM classifiers, is proposed in the thesis. In the first phase of the proposed method, a separate ensemble of ELM classifiers is trained for every single descriptor. In the second phase, an additional ELM-based classifier is introduced to learn the optimal combination of descriptors for every category. This approach enables a system to choose those descriptors which are the most representative for every category. Comparative evaluation over several benchmark datasets, has demonstrated highly accurate classification results, comparable to the state-of-the-art methods

Evolutionary design of deep neural networks

Mención Internacional en el título de doctorFor three decades, neuroevolution has applied evolutionary computation to the optimization of the topology of artificial neural networks, with most works focusing on very simple architectures. However, times have changed, and nowadays convolutional neural networks are the industry and academia standard for solving a variety of problems, many of which remained unsolved before the discovery of this kind of networks. Convolutional neural networks involve complex topologies, and the manual design of these topologies for solving a problem at hand is expensive and inefficient. In this thesis, our aim is to use neuroevolution in order to evolve the architecture of convolutional neural networks. To do so, we have decided to try two different techniques: genetic algorithms and grammatical evolution. We have implemented a niching scheme for preserving the genetic diversity, in order to ease the construction of ensembles of neural networks. These techniques have been validated against the MNIST database for handwritten digit recognition, achieving a test error rate of 0.28%, and the OPPORTUNITY data set for human activity recognition, attaining an F1 score of 0.9275. Both results have proven very competitive when compared with the state of the art. Also, in all cases, ensembles have proven to perform better than individual models. Later, the topologies learned for MNIST were tested on EMNIST, a database recently introduced in 2017, which includes more samples and a set of letters for character recognition. Results have shown that the topologies optimized for MNIST perform well on EMNIST, proving that architectures can be reused across domains with similar characteristics. In summary, neuroevolution is an effective approach for automatically designing topologies for convolutional neural networks. However, it still remains as an unexplored field due to hardware limitations. Current advances, however, should constitute the fuel that empowers the emergence of this field, and further research should start as of today.This Ph.D. dissertation has been partially supported by the Spanish Ministry of Education, Culture and Sports under FPU fellowship with identifier FPU13/03917. This research stay has been partially co-funded by the Spanish Ministry of Education, Culture and Sports under FPU short stay grant with identifier EST15/00260.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: María Araceli Sanchís de Miguel.- Secretario: Francisco Javier Segovia Pérez.- Vocal: Simon Luca

Non-iterative and Fast Deep Learning: Multilayer Extreme Learning Machines

In the past decade, deep learning techniques have powered many aspects of our daily life, and drawn ever-increasing research interests. However, conventional deep learning approaches, such as deep belief network (DBN), restricted Boltzmann machine (RBM), and convolutional neural network (CNN), suffer from time-consuming training process due to fine-tuning of a large number of parameters and the complicated hierarchical structure. Furthermore, the above complication makes it difficult to theoretically analyze and prove the universal approximation of those conventional deep learning approaches. In order to tackle the issues, multilayer extreme learning machines (ML-ELM) were proposed, which accelerate the development of deep learning. Compared with conventional deep learning, ML-ELMs are non-iterative and fast due to the random feature mapping mechanism. In this paper, we perform a thorough review on the development of ML-ELMs, including stacked ELM autoencoder (ELM-AE), residual ELM, and local receptive field based ELM (ELM-LRF), as well as address their applications. In addition, we also discuss the connection between random neural networks and conventional deep learning