ReLEx: Regularisation for Linear Extrapolation in Neural Networks with Rectified Linear Units

Despite the great success of neural networks in recent years, they are not providing useful extrapolation. In regression tasks, the popular Rectified Linear Units do enable unbounded linear extrapolation by neural networks, but their extrapolation behaviour varies widely and is largely independent of the training data. Our goal is instead to continue the local linear trend at the margin of the training data. Here we introduce ReLEx, a regularising method composed of a set of loss terms design to achieve this goal and reduce the variance of the extrapolation. We present a ReLEx implementation for single input, single output, and single hidden layer feed-forward networks. Our results demonstrate that ReLEx has little cost in terms of standard learning, i.e. interpolation, but enables controlled univariate linear extrapolation with ReLU neural networks

{\sc CosmoNet}: fast cosmological parameter estimation in non-flat models using neural networks

We present a further development of a method for accelerating the calculation of CMB power spectra, matter power spectra and likelihood functions for use in cosmological Bayesian inference. The algorithm, called {\sc CosmoNet}, is based on training a multilayer perceptron neural network. We compute CMB power spectra (up to $\ell=2000$) and matter transfer functions over a hypercube in parameter space encompassing the $4\sigma$ confidence region of a selection of CMB (WMAP + high resolution experiments) and large scale structure surveys (2dF and SDSS). We work in the framework of a generic 7 parameter non-flat cosmology. Additionally we use {\sc CosmoNet} to compute the WMAP 3-year, 2dF and SDSS likelihoods over the same region. We find that the average error in the power spectra is typically well below cosmic variance for spectra, and experimental likelihoods calculated to within a fraction of a log unit. We demonstrate that marginalised posteriors generated with {\sc CosmoNet} spectra agree to within a few percent of those generated by {\sc CAMB} parallelised over 4 CPUs, but are obtained 2-3 times faster on just a \emph{single} processor. Furthermore posteriors generated directly via {\sc CosmoNet} likelihoods can be obtained in less than 30 minutes on a single processor, corresponding to a speed up of a factor of $\sim 32$. We also demonstrate the capabilities of {\sc CosmoNet} by extending the CMB power spectra and matter transfer function training to a more generic 10 parameter cosmological model, including tensor modes, a varying equation of state of dark energy and massive neutrinos. {\sc CosmoNet} and interfaces to both {\sc CosmoMC} and {\sc Bayesys} are publically available at {\tt www.mrao.cam.ac.uk/software/cosmonet}.Comment: 8 pages, submitted to MNRA

Infering Air Quality from Traffic Data using Transferable Neural Network Models

This work presents a neural network based model for inferring air quality from traffic measurements. It is important to obtain information on air quality in urban environments in order to meet legislative and policy requirements. Measurement equipment tends to be expensive to purchase and maintain. Therefore, a model based approach capable of accurate determination of pollution levels is highly beneficial. The objective of this study was to develop a neural network model to accurately infer pollution levels from existing data sources in Leicester, UK. Neural Networks are models made of several highly interconnected processing elements. These elements process information by their dynamic state response to inputs. Problems which were not solvable by traditional algorithmic approaches frequently can be solved using neural networks. This paper shows that using a simple neural network with traffic and meteorological data as inputs, the air quality can be estimated with a good level of generalisation and in near real-time. By applying these models to links rather than nodes, this methodology can directly be used to inform traffic engineers and direct traffic management decisions towards enhancing local air quality and traffic management simultaneously.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Genetic Classification of Populations using Supervised Learning

There are many instances in genetics in which we wish to determine whether two candidate populations are distinguishable on the basis of their genetic structure. Examples include populations which are geographically separated, case--control studies and quality control (when participants in a study have been genotyped at different laboratories). This latter application is of particular importance in the era of large scale genome wide association studies, when collections of individuals genotyped at different locations are being merged to provide increased power. The traditional method for detecting structure within a population is some form of exploratory technique such as principal components analysis. Such methods, which do not utilise our prior knowledge of the membership of the candidate populations. are termed \emph{unsupervised}. Supervised methods, on the other hand are able to utilise this prior knowledge when it is available. In this paper we demonstrate that in such cases modern supervised approaches are a more appropriate tool for detecting genetic differences between populations. We apply two such methods, (neural networks and support vector machines) to the classification of three populations (two from Scotland and one from Bulgaria). The sensitivity exhibited by both these methods is considerably higher than that attained by principal components analysis and in fact comfortably exceeds a recently conjectured theoretical limit on the sensitivity of unsupervised methods. In particular, our methods can distinguish between the two Scottish populations, where principal components analysis cannot. We suggest, on the basis of our results that a supervised learning approach should be the method of choice when classifying individuals into pre-defined populations, particularly in quality control for large scale genome wide association studies.Comment: Accepted PLOS On

Development of appropriateness explicit criteria for cataract extraction by phacoemulsification

BACKGROUND: Consensus development techniques were used in the late 1980s to create explicit criteria for the appropriateness of cataract extraction. We developed a new appropriateness of indications tool for cataract following the RAND method. We tested the validity of our panel results. METHODS: Criteria were developed using a modified Delphi panel judgment process. A panel of 12 ophthalmologists was assembled. Ratings were analyzed regarding the level of agreement among panelists. We studied the influence of all variables on the final panel score using linear and logistic regression models. The explicit criteria developed were summarized by classification and regression tree analysis. RESULTS: Of the 765 indications evaluated by the main panel in the second round, 32.9% were found appropriate, 30.1% uncertain, and 37% inappropriate. Agreement was found in 53% of the indications and disagreement in 0.9%. Seven variables were considered to create the indications and divided into three groups: simple cataract, with diabetic retinopathy, or with other ocular pathologies. The preoperative visual acuity in the cataractous eye and visual function were the variables that best explained the panel scoring. The panel results were synthesized and presented in three decision trees. Misclassification error in the decision trees, as compared with the panel original criteria, was 5.3%. CONCLUSION: The parameters tested showed acceptable validity for an evaluation tool. These results support the use of this indication algorithm as a screening tool for assessing the appropriateness of cataract extraction in field studies and for the development of practice guidelines

Neural Network Parameterizations of Electromagnetic Nucleon Form Factors

The electromagnetic nucleon form-factors data are studied with artificial feed forward neural networks. As a result the unbiased model-independent form-factor parametrizations are evaluated together with uncertainties. The Bayesian approach for the neural networks is adapted for chi2 error-like function and applied to the data analysis. The sequence of the feed forward neural networks with one hidden layer of units is considered. The given neural network represents a particular form-factor parametrization. The so-called evidence (the measure of how much the data favor given statistical model) is computed with the Bayesian framework and it is used to determine the best form factor parametrization.Comment: The revised version is divided into 4 sections. The discussion of the prior assumptions is added. The manuscript contains 4 new figures and 2 new tables (32 pages, 15 figures, 2 tables

Constraints on fNL from Wilkinson Microwave Anisotropy Probe 7-year data using a neural network classifier

We present a multi-class neural network (NN) classifier as a method to measure nonGaussianity, characterised by the local non-linear coupling parameter fNL, in maps of the cosmic microwave background (CMB) radiation. The classifier is trained on simulated non-Gaussian CMB maps with a range of known fNL values by providing it with wavelet coefficients of the maps; we consider both the HealPix (HW) wavelet and the spherical Mexican hat wavelet (SMHW). When applied to simulated test maps, the NN classfier produces results in very good agreement with those obtained using standard chi2 minimization. The standard deviations of the fNL estimates for WMAPlike simulations were {\sigma} = 22 and {\sigma} = 33 for the SMHW and the HW, respectively, which are extremely close to those obtained using classical statistical methods in Curto et al. and Casaponsa et al. Moreover, the NN classifier does not require the inversion of a large covariance matrix, thus avoiding any need to regularise the matrix when it is not directly invertible, and is considerably faster.Comment: Accepted for publication in MNRAS, 9 pages, 5 figures, 1 tabl

On the Bounds of Function Approximations

Within machine learning, the subfield of Neural Architecture Search (NAS) has recently garnered research attention due to its ability to improve upon human-designed models. However, the computational requirements for finding an exact solution to this problem are often intractable, and the design of the search space still requires manual intervention. In this paper we attempt to establish a formalized framework from which we can better understand the computational bounds of NAS in relation to its search space. For this, we first reformulate the function approximation problem in terms of sequences of functions, and we call it the Function Approximation (FA) problem; then we show that it is computationally infeasible to devise a procedure that solves FA for all functions to zero error, regardless of the search space. We show also that such error will be minimal if a specific class of functions is present in the search space. Subsequently, we show that machine learning as a mathematical problem is a solution strategy for FA, albeit not an effective one, and further describe a stronger version of this approach: the Approximate Architectural Search Problem (a-ASP), which is the mathematical equivalent of NAS. We leverage the framework from this paper and results from the literature to describe the conditions under which a-ASP can potentially solve FA as well as an exhaustive search, but in polynomial time.Comment: Accepted as a full paper at ICANN 2019. The final, authenticated publication will be available at https://doi.org/10.1007/978-3-030-30487-4_3