Novel Computationally Intelligent Machine Learning Algorithms for Data Mining and Knowledge Discovery

This thesis addresses three major issues in data mining regarding feature subset selection in large dimensionality domains, plausible reconstruction of incomplete data in cross-sectional applications, and forecasting univariate time series. For the automated selection of an optimal subset of features in real time, we present an improved hybrid algorithm: SAGA. SAGA combines the ability to avoid being trapped in local minima of Simulated Annealing with the very high convergence rate of the crossover operator of Genetic Algorithms, the strong local search ability of greedy algorithms and the high computational efficiency of generalized regression neural networks (GRNN). For imputing missing values and forecasting univariate time series, we propose a homogeneous neural network ensemble. The proposed ensemble consists of a committee of Generalized Regression Neural Networks (GRNNs) trained on different subsets of features generated by SAGA and the predictions of base classifiers are combined by a fusion rule. This approach makes it possible to discover all important interrelations between the values of the target variable and the input features. The proposed ensemble scheme has two innovative features which make it stand out amongst ensemble learning algorithms: (1) the ensemble makeup is optimized automatically by SAGA; and (2) GRNN is used for both base classifiers and the top level combiner classifier. Because of GRNN, the proposed ensemble is a dynamic weighting scheme. This is in contrast to the existing ensemble approaches which belong to the simple voting and static weighting strategy. The basic idea of the dynamic weighting procedure is to give a higher reliability weight to those scenarios that are similar to the new ones. The simulation results demonstrate the validity of the proposed ensemble model

Some New Results in Distributed Tracking and Optimization

The current age of Big Data is built on the foundation of distributed systems, and efficient distributed algorithms to run on these systems.With the rapid increase in the volume of the data being fed into these systems, storing and processing all this data at a central location becomes infeasible. Such a central \textit{server} requires a gigantic amount of computational and storage resources. Even when it is possible to have central servers, it is not always desirable, due to privacy concerns. Also, sending huge amounts of data to such servers incur often infeasible bandwidth requirements. In this dissertation, we consider two kinds of distributed architectures: 1) star-shaped topology, where multiple worker nodes are connected to, and communicate with a server, but the workers do not communicate with each other; and 2) mesh topology or network of interconnected workers, where each worker can communicate with a small number of neighboring workers. In the first half of this dissertation (Chapters 2 and 3), we consider distributed systems with mesh topology.We study two different problems in this context. First, we study the problem of simultaneous localization and multi-target tracking. Multiple mobile agents localize themselves cooperatively, while also tracking multiple, unknown number of mobile targets, in the presence of measurement-origin uncertainty. In situations with limited GPS signal availability, agents (like self-driving cars in urban canyons, or autonomous vehicles in hazardous environments) need to rely on inter-agent measurements for localization. The agents perform the additional task of tracking multiple targets (pedestrians and road-signs for self-driving cars). We propose a decentralized algorithm for this problem. To be effective in real-time applications, we propose efficient Gaussian and Gaussian-mixture based filters, rather than the computationally expensive particle-based methods in the existing literature. Our novel factor-graph based approach gives better performance, in terms of both agent localization errors, and target-location and cardinality errors. Next, we study an online convex optimization problem, where a network of agents cooperate to minimize a global time-varying objective function. Only the local functions are revealed to individual agents. The agents also need to satisfy their individual constraints. We propose a primal-dual update based decentralized algorithm for this problem. Under standard assumptions, we prove that the proposed algorithm achieves sublinear regret and constraint violation across the network. In other words, over a long enough time horizon, the decisions taken by the agents are, on average, as good as if all the information was revealed ahead of time. In addition, the individual constraint violations of the agents, averaged over time, are zero. In the next part of the dissertation (Chapters 4), we study distributed systems with a star-shaped topology. The problem we study is distributed nonconvex optimization. With the recent success of deep learning, coupled with the use of distributed systems to solve large-scale problems, this problem has gained prominence over the past decade. The recently proposed paradigm of Federated Learning (which has already been deployed by Google/Apple in Android/iOS phones) has further catalyzed research in this direction. The problem we consider is minimizing the average of local smooth, nonconvex functions. Each node has access only to its own loss function, but can communicate with the server, which aggregates updates from all the nodes, before distributing them to all the nodes. With the advent of more and more complex neural network architectures, these updates can be high dimensional. To save resources, the problem needs to be solved via communication-efficient approaches. We propose a novel algorithm, which combines the idea of variance-reduction, with the paradigm of carrying out multiple local updates at each node before averaging. We prove the convergence of the approach to a first-order stationary point. Our algorithm is optimal in terms of computation, and state-of-the-art in terms of the communication requirements. Lastly in Chapter 5, we consider the situation when the nodes do not have access to function gradients, and need to minimize the loss function using only function values. This problem lies in the domain of zeroth-order optimization. For simplicity of analysis, we study this problem only in the single-node case. This problem finds application in simulation-based optimization, and adversarial example generation for attacking deep neural networks. We propose a novel function value based gradient estimator, which has better variance, and better query-efficiency compared to existing estimators. The proposed estimator covers the most commonly used existing estimators as special cases. We conduct a comprehensive convergence analysis under different conditions. We also demonstrate its effectiveness through a real-world application to generating adversarial examples from a black-box deep neural network

A review of the application of the simulated annealing algorithm in structural health monitoring (1995-2021)

In recent years, many innovative optimization algorithms have been developed. These algorithms have been employed to solve structural damage detection problems as an inverse solution. However, traditional optimization methods such as particle swarm optimization, simulated annealing (SA), and genetic algorithm are constantly employed to detect damages in the structures. This paper reviews the application of SA in different disciplines of structural health monitoring, such as damage detection, finite element model updating, optimal sensor placement, and system identification. The methodologies, objectives, and results of publications conducted between 1995 and 2021 are analyzed. This paper also provides an in-depth discussion of different open questions and research directions in this area

Advanced Mobile Robotics: Volume 3

Mobile robotics is a challenging field with great potential. It covers disciplines including electrical engineering, mechanical engineering, computer science, cognitive science, and social science. It is essential to the design of automated robots, in combination with artificial intelligence, vision, and sensor technologies. Mobile robots are widely used for surveillance, guidance, transportation and entertainment tasks, as well as medical applications. This Special Issue intends to concentrate on recent developments concerning mobile robots and the research surrounding them to enhance studies on the fundamental problems observed in the robots. Various multidisciplinary approaches and integrative contributions including navigation, learning and adaptation, networked system, biologically inspired robots and cognitive methods are welcome contributions to this Special Issue, both from a research and an application perspective

Urban Overheating - Progress on Mitigation Science and Engineering Applications

The combination of global warming and urban sprawl is the origin of the most hazardous climate change effect detected at urban level: Urban Heat Island, representing the urban overheating respect to the countryside surrounding the city. This book includes 18 papers representing the state of the art of detection, assessment mitigation and adaption to urban overheating. Advanced methods, strategies and technologies are here analyzed including relevant issues as: the role of urban materials and fabrics on urban climate and their potential mitigation, the impact of greenery and vegetation to reduce urban temperatures and improve the thermal comfort, the role the urban geometry in the air temperature rise, the use of satellite and ground data to assess and quantify the urban overheating and develop mitigation solutions, calculation methods and application to predict and assess mitigation scenarios. The outcomes of the book are thus relevant for a wide multidisciplinary audience, including: environmental scientists and engineers, architect and urban planners, policy makers and students

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

Sensors Fault Diagnosis Trends and Applications

Fault diagnosis has always been a concern for industry. In general, diagnosis in complex systems requires the acquisition of information from sensors and the processing and extracting of required features for the classification or identification of faults. Therefore, fault diagnosis of sensors is clearly important as faulty information from a sensor may lead to misleading conclusions about the whole system. As engineering systems grow in size and complexity, it becomes more and more important to diagnose faulty behavior before it can lead to total failure. In the light of above issues, this book is dedicated to trends and applications in modern-sensor fault diagnosis

Adaptation and Stochasticity of Natural Complex Systems

The methods that fueled the microscale revolution (top-down design/fabrication, combined with application of forces large enough to overpower stochasticity) constitute an approach that will not scale down to nanoscale systems. In contrast, in nanotechnology, we strive to embrace nature’s quite different paradigms to create functional systems, such as self-assembly to create structures, exploiting stochasticity, rather than overwhelming it, in order to create deterministic, yet highly adaptable, behavior. Nature’s approach, through billions of years of evolutionary development, has achieved self-assembling, self-duplicating, self-healing, adaptive systems. Compared to microprocessors, nature’s approach has achieved eight orders of magnitude higher memory density and three orders of magnitude higher computing capacity while utilizing eight orders of magnitude less power. Perhaps the most complex of functions, homeostatis by a biological cell – i.e., the regulation of its internal environment to maintain stability and function – in a fluctuating and unpredictable environment, emerges from the interactions between perhaps 50M molecules of a few thousand different types. Many of these molecules (e.g. proteins, RNA) are produced in the stochastic processes of gene expression, and the resulting populations of these molecules are distributed across a range of values. So although homeostasis is maintained at the system (i.e. cell) level, there are considerable and unavoidable fluctuations at the component (protein, RNA) level. While on at least some level, we understand the variability in individual components, we have no understanding of how to integrate these fluctuating components together to achieve complex function at the system level. This thesis will explore the regulation and control of stochasticity in cells. In particular, the focus will be on (1) how genetic circuits use noise to generate more function in less space; (2) how stochastic and deterministic responses are co-regulated to enhance function at a system level; and (3) the development of high-throughput analytical techniques that enable a comprehensive view of the structure and distribution of noise on a whole organism level