Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions

Quantification of the stationary points and the associated basins of attraction of neural network loss surfaces is an important step towards a better understanding of neural network loss surfaces at large. This work proposes a novel method to visualise basins of attraction together with the associated stationary points via gradient-based random sampling. The proposed technique is used to perform an empirical study of the loss surfaces generated by two different error metrics: quadratic loss and entropic loss. The empirical observations confirm the theoretical hypothesis regarding the nature of neural network attraction basins. Entropic loss is shown to exhibit stronger gradients and fewer stationary points than quadratic loss, indicating that entropic loss has a more searchable landscape. Quadratic loss is shown to be more resilient to overfitting than entropic loss. Both losses are shown to exhibit local minima, but the number of local minima is shown to decrease with an increase in dimensionality. Thus, the proposed visualisation technique successfully captures the local minima properties exhibited by the neural network loss surfaces, and can be used for the purpose of fitness landscape analysis of neural networks.Comment: Preprint submitted to the Neural Networks journa

Frequency Effects on Predictability of Stock Returns

We propose that predictability is a prerequisite for profitability on financial markets. We look at ways to measure predictability of price changes using information theoretic approach and employ them on all historical data available for NYSE 100 stocks. This allows us to determine whether frequency of sampling price changes affects the predictability of those. We also relations between price changes predictability and the deviation of the price formation processes from iid as well as the stock's sector. We also briefly comment on the complicated relationship between predictability of price changes and the profitability of algorithmic trading.Comment: 8 pages, 16 figures, submitted for possible publication to Computational Intelligence for Financial Engineering and Economics 2014 conferenc

First-principles calculation of DNA looping in tethered particle experiments

We calculate the probability of DNA loop formation mediated by regulatory proteins such as Lac repressor (LacI), using a mathematical model of DNA elasticity. Our model is adapted to calculating quantities directly observable in Tethered Particle Motion (TPM) experiments, and it accounts for all the entropic forces present in such experiments. Our model has no free parameters; it characterizes DNA elasticity using information obtained in other kinds of experiments. [...] We show how to compute both the "looping J factor" (or equivalently, the looping free energy) for various DNA construct geometries and LacI concentrations, as well as the detailed probability density function of bead excursions. We also show how to extract the same quantities from recent experimental data on tethered particle motion, and then compare to our model's predictions. [...] Our model successfully reproduces the detailed distributions of bead excursion, including their surprising three-peak structure, without any fit parameters and without invoking any alternative conformation of the LacI tetramer. Indeed, the model qualitatively reproduces the observed dependence of these distributions on tether length (e.g., phasing) and on LacI concentration (titration). However, for short DNA loops (around 95 basepairs) the experiments show more looping than is predicted by the harmonic-elasticity model, echoing other recent experimental results. Because the experiments we study are done in vitro, this anomalously high looping cannot be rationalized as resulting from the presence of DNA-bending proteins or other cellular machinery. We also show that it is unlikely to be the result of a hypothetical "open" conformation of the LacI tetramer.Comment: See the supplement at http://www.physics.upenn.edu/~pcn/Ms/TowlesEtalSuppl.pdf . This revised version accepted for publication at Physical Biolog

Dynamics of the Wang-Landau algorithm and complexity of rare events for the three-dimensional bimodal Ising spin glass

We investigate the performance of flat-histogram methods based on a multicanonical ensemble and the Wang-Landau algorithm for the three-dimensional +/- J spin glass by measuring round-trip times in the energy range between the zero-temperature ground state and the state of highest energy. Strong sample-to-sample variations are found for fixed system size and the distribution of round-trip times follows a fat-tailed Frechet extremal value distribution. Rare events in the fat tails of these distributions corresponding to extremely slowly equilibrating spin glass realizations dominate the calculations of statistical averages. While the typical round-trip time scales exponential as expected for this NP-hard problem, we find that the average round-trip time is no longer well-defined for systems with N >= 8^3 spins. We relate the round-trip times for multicanonical sampling to intrinsic properties of the energy landscape and compare with the numerical effort needed by the genetic Cluster-Exact Approximation to calculate the exact ground state energies. For systems with N >= 8^3 spins the simulation of these rare events becomes increasingly hard. For N >= 14^3 there are samples where the Wang-Landau algorithm fails to find the true ground state within reasonable simulation times. We expect similar behavior for other algorithms based on multicanonical sampling.Comment: 9 pages, 12 figure

A review of Monte Carlo simulations of polymers with PERM

In this review, we describe applications of the pruned-enriched Rosenbluth method (PERM), a sequential Monte Carlo algorithm with resampling, to various problems in polymer physics. PERM produces samples according to any given prescribed weight distribution, by growing configurations step by step with controlled bias, and correcting "bad" configurations by "population control". The latter is implemented, in contrast to other population based algorithms like e.g. genetic algorithms, by depth-first recursion which avoids storing all members of the population at the same time in computer memory. The problems we discuss all concern single polymers (with one exception), but under various conditions: Homopolymers in good solvents and at the $\Theta$ point, semi-stiff polymers, polymers in confining geometries, stretched polymers undergoing a forced globule-linear transition, star polymers, bottle brushes, lattice animals as a model for randomly branched polymers, DNA melting, and finally -- as the only system at low temperatures, lattice heteropolymers as simple models for protein folding. PERM is for some of these problems the method of choice, but it can also fail. We discuss how to recognize when a result is reliable, and we discuss also some types of bias that can be crucial in guiding the growth into the right directions.Comment: 29 pages, 26 figures, to be published in J. Stat. Phys. (2011

Designing labeled graph classifiers by exploiting the R\'enyi entropy of the dissimilarity representation

Representing patterns as labeled graphs is becoming increasingly common in the broad field of computational intelligence. Accordingly, a wide repertoire of pattern recognition tools, such as classifiers and knowledge discovery procedures, are nowadays available and tested for various datasets of labeled graphs. However, the design of effective learning procedures operating in the space of labeled graphs is still a challenging problem, especially from the computational complexity viewpoint. In this paper, we present a major improvement of a general-purpose classifier for graphs, which is conceived on an interplay between dissimilarity representation, clustering, information-theoretic techniques, and evolutionary optimization algorithms. The improvement focuses on a specific key subroutine devised to compress the input data. We prove different theorems which are fundamental to the setting of the parameters controlling such a compression operation. We demonstrate the effectiveness of the resulting classifier by benchmarking the developed variants on well-known datasets of labeled graphs, considering as distinct performance indicators the classification accuracy, computing time, and parsimony in terms of structural complexity of the synthesized classification models. The results show state-of-the-art standards in terms of test set accuracy and a considerable speed-up for what concerns the computing time.Comment: Revised versio

Semi-automatic selection of summary statistics for ABC model choice

A central statistical goal is to choose between alternative explanatory models of data. In many modern applications, such as population genetics, it is not possible to apply standard methods based on evaluating the likelihood functions of the models, as these are numerically intractable. Approximate Bayesian computation (ABC) is a commonly used alternative for such situations. ABC simulates data x for many parameter values under each model, which is compared to the observed data xobs. More weight is placed on models under which S(x) is close to S(xobs), where S maps data to a vector of summary statistics. Previous work has shown the choice of S is crucial to the efficiency and accuracy of ABC. This paper provides a method to select good summary statistics for model choice. It uses a preliminary step, simulating many x values from all models and fitting regressions to this with the model as response. The resulting model weight estimators are used as S in an ABC analysis. Theoretical results are given to justify this as approximating low dimensional sufficient statistics. A substantive application is presented: choosing between competing coalescent models of demographic growth for Campylobacter jejuni in New Zealand using multi-locus sequence typing data