Regularization in Symbolic Regression by an Additional Fitness Objective

Symbolic regression is a method for discovering functions that minimize error on a given dataset. It is of interest to prevent overfitting in symbolic regression. In this work, regularization of symbolic regression is attempted by incorporating an additional fitness objective. This new fitness objective is called Worst Neighbors (WN) score, which measures differences in approximate derivatives in the form of angles. To compute the Worst Neighbors score, place partition points between each pair of adjacent data points. For each pair of data points, compute the maximum angle between the line formed by the pair of data points and the lines formed by adjacent partition points. The maximum of all these maximum angles is the Worst Neighbors score. This method differs from other attempts to regularize symbolic regression because it considers the behavior of the evolved function between data points. A high WN score indicates that the function has overfit the data. A low score could indicate either an underfit solution or a well fit solution. The error objective is used to make this distinction. Worst Neighbors can reduce overfitting in symbolic regression because it encourages functions that have a low error and a low Worst Neighbors score. The error objective helps stop the solutions from becoming underfit and the Worst Neighbors score helps stop the solutions from becoming overfit. To use Worst Neighbors for target functions of higher dimensions, select nearby points as neighbors and compute the Worst Neighbors score on the evolved function restricted to the plane formed by these neighbors and the output direction. For the one dimensional case, Worst Neighbors has promise in reducing error on unseen data when compared with Age-Fitness Pareto Optimization (AFPO). WN achieves a small decrease in testing error on several target functions compared to AFPO

Genetic programming with semantic equivalence classes

Ruberto, S., Vanneschi, L., & Castelli, M. (2019). Genetic programming with semantic equivalence classes. Swarm and Evolutionary Computation, 44(February), 453-469. DOI: 10.1016/j.swevo.2018.06.001In this paper, we introduce the concept of semantics-based equivalence classes for symbolic regression problems in genetic programming. The idea is implemented by means of two different genetic programming systems, in which two different definitions of equivalence are used. In both systems, whenever a solution in an equivalence class is found, it is possible to generate any other solution in that equivalence class analytically. As such, these two systems allow us to shift the objective of genetic programming: instead of finding a globally optimal solution, the objective is now to find any solution that belongs to the same equivalence class as a global optimum. Further, we propose improvements to these genetic programming systems in which, once a solution that belongs to a particular equivalence class is generated, no other solution in that class is accepted in the population during the evolution anymore. We call these improved versions filtered systems. Experimental results obtained via seven complex real-life test problems show that using equivalence classes is a promising idea and that filters are generally helpful for improving the systems' performance. Furthermore, the proposed methods produce individuals with a much smaller size with respect to geometric semantic genetic programming. Finally, we show that filters are also useful to improve the performance of a state-of-the-art method, not explicitly based on semantic equivalence classes, like linear scaling.authorsversionpublishe

Where Does the Density Localize? Convergent Behavior for Global Hybrids, Range Separation, and DFT+U

Approximate density functional theory (DFT) suffers from many-electron self- interaction error, otherwise known as delocalization error, that may be diagnosed and then corrected through elimination of the deviation from exact piecewise linear behavior between integer electron numbers. Although paths to correction of energetic delocalization error are well- established, the impact of these corrections on the electron density is less well-studied. Here, we compare the effect on density delocalization of DFT+U, global hybrid tuning, and range- separated hybrid tuning on a diverse test set of 32 transition metal complexes and observe the three methods to have qualitatively equivalent effects on the ground state density. Regardless of valence orbital diffuseness (i.e., from 2p to 5p), ligand electronegativity (i.e., from Al to O), basis set (i.e., plane wave versus localized basis set), metal (i.e., Ti, Fe, Ni) and spin state, or tuning method, we consistently observe substantial charge loss at the metal and gain at ligand atoms (ca. 0.3-0.5 e or more). This charge loss at the metal is preferentially from the minority spin, leading to increasing magnetic moment as well. Using accurate wavefunction theory references, we observe that a minimum error in partial charges and magnetic moments occur at higher tuning parameters than typically employed to eliminate energetic delocalization error. These observations motivate the need to develop multi-faceted approximate-DFT error correction approaches that separately treat density delocalization and energetic errors in order to recover both correct density and magnetization properties.Comment: 34 pages, 11 figure

Improving malware detection with neuroevolution : a study with the semantic learning machine

Project Work presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Knowledge Management and Business IntelligenceMachine learning has become more attractive over the years due to its remarkable adaptation and problem-solving abilities. Algorithms compete amongst each other to claim the best possible results for every problem, being one of the most valued characteristics their generalization ability. A recently proposed methodology of Genetic Programming (GP), called Geometric Semantic Genetic Programming (GSGP), has seen its popularity rise over the last few years, achieving great results compared to other state-of-the-art algorithms, due to its remarkable feature of inducing a fitness landscape with no local optima solutions. To any supervised learning problem, where a metric is used as an error function, GSGP’s landscape will be unimodal, therefore allowing for genetic algorithms to behave much more efficiently and effectively. Inspired by GSGP’s features, Gonçalves developed a new mutation operator to be applied to the Neural Networks (NN) domain, creating the Semantic Learning Machine (SLM). Despite GSGP’s good results already proven, there are still research opportunities for improvement, that need to be performed to empirically prove GSGP as a state-of-the-art framework. In this case, the study focused on applying SLM to NNs with multiple hidden layers and compare its outputs to a very popular algorithm, Multilayer Perceptron (MLP), on a considerably large classification dataset about Android malware. Findings proved that SLM, sharing common parametrization with MLP, in order to have a fair comparison, is able to outperform it, with statistical significance

Energy efficient enabling technologies for semantic video processing on mobile devices

Semantic object-based processing will play an increasingly important role in future multimedia systems due to the ubiquity of digital multimedia capture/playback technologies and increasing storage capacity. Although the object based paradigm has many undeniable benefits, numerous technical challenges remain before the applications becomes pervasive, particularly on computational constrained mobile devices. A fundamental issue is the ill-posed problem of semantic object segmentation. Furthermore, on battery powered mobile computing devices, the additional algorithmic complexity of semantic object based processing compared to conventional video processing is highly undesirable both from a real-time operation and battery life perspective. This thesis attempts to tackle these issues by firstly constraining the solution space and focusing on the human face as a primary semantic concept of use to users of mobile devices. A novel face detection algorithm is proposed, which from the outset was designed to be amenable to be offloaded from the host microprocessor to dedicated hardware, thereby providing real-time performance and reducing power consumption. The algorithm uses an Artificial Neural Network (ANN), whose topology and weights are evolved via a genetic algorithm (GA). The computational burden of the ANN evaluation is offloaded to a dedicated hardware accelerator, which is capable of processing any evolved network topology. Efficient arithmetic circuitry, which leverages modified Booth recoding, column compressors and carry save adders, is adopted throughout the design. To tackle the increased computational costs associated with object tracking or object based shape encoding, a novel energy efficient binary motion estimation architecture is proposed. Energy is reduced in the proposed motion estimation architecture by minimising the redundant operations inherent in the binary data. Both architectures are shown to compare favourable with the relevant prior art

Review of Deep Learning Algorithms and Architectures

Deep learning (DL) is playing an increasingly important role in our lives. It has already made a huge impact in areas, such as cancer diagnosis, precision medicine, self-driving cars, predictive forecasting, and speech recognition. The painstakingly handcrafted feature extractors used in traditional learning, classification, and pattern recognition systems are not scalable for large-sized data sets. In many cases, depending on the problem complexity, DL can also overcome the limitations of earlier shallow networks that prevented efficient training and abstractions of hierarchical representations of multi-dimensional training data. Deep neural network (DNN) uses multiple (deep) layers of units with highly optimized algorithms and architectures. This paper reviews several optimization methods to improve the accuracy of the training and to reduce training time. We delve into the math behind training algorithms used in recent deep networks. We describe current shortcomings, enhancements, and implementations. The review also covers different types of deep architectures, such as deep convolution networks, deep residual networks, recurrent neural networks, reinforcement learning, variational autoencoders, and others.https://doi.org/10.1109/ACCESS.2019.291220

Deep Semantic Learning Machine Initial design and experiments

Dissertation presented as the partial requirement for obtaining a Master's degree in Data Science and Advanced AnalyticsComputer vision is an interdisciplinary scientific field that allows the digital world to interact with the real world. It is one of the fastest-growing and most important areas of data science. Applications are endless, given various tasks that can be solved thanks to the advances in the computer vision field. Examples of types of tasks that can be solved thanks to computer vision models are: image analysis, object detection, image transformation, and image generation. Having that many applications is vital for providing models with the best possible performance. Although many years have passed since backpropagation was invented, it is still the most commonly used approach of training neural networks. A satisfactory performance can be achieved with this approach, but is it the best it can get? A fixed topology of a neural network that needs to be defined before any training begins seems to be a significant limitation as the performance of a network is highly dependent on the topology. Since there are no studies that would precisely guide scientists on selecting a proper network structure, the ability to adjust a topology to a problem seems highly promising. Initial ideas of the evolution of neural networks that involve heuristic search methods have provided encouragingly good results for the various reinforcement learning task. This thesis presents the initial experiments on the usage of a similar approach to solve image classification tasks. The new model called Deep Semantic Learning Machine is introduced with a new mutation method specially designed to solve computer vision problems. Deep Semantic Learning Machine allows a topology to evolve from a small network and adjust to a given problem. The initial results are pretty promising, especially in a training dataset. However, in this thesis Deep Semantic Learning Machine was developed only as proof of a concept and further improvements to the approach can be made