Combined optimization algorithms applied to pattern classification

Accurate classification by minimizing the error on test samples is the main goal in pattern classification. Combinatorial optimization is a well-known method for solving minimization problems, however, only a few examples of classifiers axe described in the literature where combinatorial optimization is used in pattern classification. Recently, there has been a growing interest in combining classifiers and improving the consensus of results for a greater accuracy. In the light of the "No Ree Lunch Theorems", we analyse the combination of simulated annealing, a powerful combinatorial optimization method that produces high quality results, with the classical perceptron algorithm. This combination is called LSA machine. Our analysis aims at finding paradigms for problem-dependent parameter settings that ensure high classifica, tion results. Our computational experiments on a large number of benchmark problems lead to results that either outperform or axe at least competitive to results published in the literature. Apart from paxameter settings, our analysis focuses on a difficult problem in computation theory, namely the network complexity problem. The depth vs size problem of neural networks is one of the hardest problems in theoretical computing, with very little progress over the past decades. In order to investigate this problem, we introduce a new recursive learning method for training hidden layers in constant depth circuits. Our findings make contributions to a) the field of Machine Learning, as the proposed method is applicable in training feedforward neural networks, and to b) the field of circuit complexity by proposing an upper bound for the number of hidden units sufficient to achieve a high classification rate. One of the major findings of our research is that the size of the network can be bounded by the input size of the problem and an approximate upper bound of 8 + √2n/n threshold gates as being sufficient for a small error rate, where n := log/SL and SL is the training set

Metaheuristic design of feedforward neural networks: a review of two decades of research

Over the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era

Evolutionary optimization of sparsely connected and time-lagged neural networks for time series forecasting

Time Series Forecasting (TSF) is an important tool to support decision mak- ing (e.g., planning production resources). Artificial Neural Networks (ANN) are innate candidates for TSF due to advantages such as nonlinear learn- ing and noise tolerance. However, the search for the best model is a complex task that highly affects the forecasting performance. In this work, we propose two novel Evolutionary Artificial Neural Networks (EANN) approaches for TSF based on an Estimation Distribution Algorithm (EDA) search engine. The first new approach consist of Sparsely connected Evolutionary ANN (SEANN), which evolves more flexible ANN structures to perform multi-step ahead forecasts. The second one, consists of an automatic Time lag feature selection EANN (TEANN) approach that evolves not only ANN parameters (e.g., input and hidden nodes, training parameters) but also which set of time lags are fed into the forecasting model. Several experiments were held, using a set of six time series, from different real-world domains. Also, two error metrics (i.e., Mean Squared Error and Symmetric Mean Absolute Per- centage Error) were analyzed. The two EANN approaches were compared against a base EANN (with no ANN structure or time lag optimization) and four other methods (Autoregressive Integrated Moving Average method, Random Forest, Echo State Network and Support Vector Machine). Overall, the proposed SEANN and TEANN methods obtained the best forecasting results. Moreover, they favor simpler neural network models, thus requiring less computational effort when compared with the base EANN.The research reported here has been supported by the Spanish Ministry of Science and Innovation under project TRA2010-21371-C03-03 and FCT - Fundacao para a Ciencia e Tecnologia within the Project Scope PEst- OE/EEI/UI0319/2014. The authors want to thank specially Martin Stepnicka and Lenka Vavrickova for all their help. The authors also want to thank Ramon Sagarna for introducing the subject of EDA

Meta-Learning Evolutionary Artificial Neural Networks

In this paper, we present MLEANN (Meta-Learning Evolutionary Artificial Neural Network), an automatic computational framework for the adaptive optimization of artificial neural networks wherein the neural network architecture, activation function, connection weights; learning algorithm and its parameters are adapted according to the problem. We explored the performance of MLEANN and conventionally designed artificial neural networks for function approximation problems. To evaluate the comparative performance, we used three different well-known chaotic time series. We also present the state of the art popular neural network learning algorithms and some experimentation results related to convergence speed and generalization performance. We explored the performance of backpropagation algorithm; conjugate gradient algorithm, quasi-Newton algorithm and Levenberg-Marquardt algorithm for the three chaotic time series. Performances of the different learning algorithms were evaluated when the activation functions and architecture were changed. We further present the theoretical background, algorithm, design strategy and further demonstrate how effective and inevitable is the proposed MLEANN framework to design a neural network, which is smaller, faster and with a better generalization performance

Artificial Intelligence in Engineering Management

L

Determining uncertainties in AI applications in AEC sector and their corresponding mitigation strategies

The Artificial Intelligence (AI) methodologies and techniques have been used to solve a wide spectrum of engineering problems in Architectural, Engineering and Construction (AEC) industry with the aim of improving overall productivity and optimized decision throughout full project life cycle (planning, design, construction and maintenance). However, many AI applications are facing different limitations and constrains due to the lack of comprehensive understanding about the inherent uncertainty fundamentally and mathematically, hence the use of AI has not achieved a satisfactory level. It requires different actions to tackle different types of uncertainties which varies according to different types of applications. This paper therefore reviews 5 type of popular AI algorithms, including Primary Component Analysis, Multilayer Perceptron, Fuzzy Logic, Support Vector Machine and Genetic Algorithm; then examines how these artificial intelligence techniques can assist the decision-making process by mitigating uncertainty meanwhile achieving the expected high efficiency. The paper reviews each germane technique, mathematical explanation, analysis of reasons causing uncertainty, and concludes a set of guidelines and an application framework for optimizing their informed uncertainty for AEC applications. This work will pave the way for the fundamental understanding and in turn to provide a valuable reference for applying AI techniques in AEC sector properly to achieve better overall performance

AI based residential load forecasting

The increasing levels of energy consumption worldwide is raising issues with respect to surpassing supply limits, causing severe effects on the environment, and the exhaustion of energy resources. Buildings are one of the most relevant sectors in terms of energy consumption in the world. Many researches have been carried out in the recent years with primary concentration on efficient Home or Building Management Systems. In addition, by increasing renewable energy penetration, modern power grids demand more accurate consumption predictions to provide the optimized power supply which is stochastic in nature. This study will present an analytic comparison of day-ahead load forecasting during a period of two years by applying AI based data driven models. The unit of analysis in this thesis project is based on households smart meter data in England. The collected and collated data for this study includes historical electricity consumption of 75 houses over two years of 2012 to 2014 city of London. Predictive models divided in two main forecasting groups of deterministic and probabilistic forecasting. In deterministic step, Random Forest Regression and MLP Regression employed to make a forecasting models. In the probabilistic phase,DeepAR, FFNN and Gaussian Process Estimator were employed to predict days ahead load forecasting. The models are trained based on subset of various groups of customers with registered diversified load volatility level. Daily weather data are also added as new feature in this study into subset to check model sensitivity to external factors and validate the performance of the model. The results of implemented models are evaluated by well-known error metrics as RMSE,MAE, MSE and CRPS separately for each phase of this study. The findings of this master thesis study shows that the Deep Learning methods of FNN, DeepAR and MLP compared to other utilized methods like Random Forest and Gaussian provide better data prediction reslts in terms of less deviance to real load trend, lower forecasting error and computation time. Considering probabilistic forecasting methods it is observed that DeepAR can provide better results than FFNN and Gaussian Process model. Although the computation time of FFNN was lower than other