Entropy and information in neural spike trains: Progress on the sampling problem

The major problem in information theoretic analysis of neural responses and other biological data is the reliable estimation of entropy--like quantities from small samples. We apply a recently introduced Bayesian entropy estimator to synthetic data inspired by experiments, and to real experimental spike trains. The estimator performs admirably even very deep in the undersampled regime, where other techniques fail. This opens new possibilities for the information theoretic analysis of experiments, and may be of general interest as an example of learning from limited data.Comment: 7 pages, 4 figures; referee suggested changes, accepted versio

The cytoplasm of living cells: A functional mixture of thousands of components

Inside every living cell is the cytoplasm: a fluid mixture of thousands of different macromolecules, predominantly proteins. This mixture is where most of the biochemistry occurs that enables living cells to function, and it is perhaps the most complex liquid on earth. Here we take an inventory of what is actually in this mixture. Recent genome-sequencing work has given us for the first time at least some information on all of these thousands of components. Having done so we consider two physical phenomena in the cytoplasm: diffusion and possible phase separation. Diffusion is slower in the highly crowded cytoplasm than in dilute solution. Reasonable estimates of this slowdown can be obtained and their consequences explored, for example, monomer-dimer equilibria are established approximately twenty times slower than in a dilute solution. Phase separation in all except exceptional cells appears not to be a problem, despite the high density and so strong protein-protein interactions present. We suggest that this may be partially a byproduct of the evolution of other properties, and partially a result of the huge number of components present.Comment: 11 pages, 1 figure, 1 tabl

Shannon Meets Carnot: Generalized Second Thermodynamic Law

The classical thermodynamic laws fail to capture the behavior of systems with energy Hamiltonian which is an explicit function of the temperature. Such Hamiltonian arises, for example, in modeling information processing systems, like communication channels, as thermal systems. Here we generalize the second thermodynamic law to encompass systems with temperature-dependent energy levels, $dQ=TdS+dT$, where $$ denotes averaging over the Boltzmann distribution and reveal a new definition to the basic notion of temperature. This generalization enables to express, for instance, the mutual information of the Gaussian channel as a consequence of the fundamental laws of nature - the laws of thermodynamics

Intrinsic limitations of inverse inference in the pairwise Ising spin glass

We analyze the limits inherent to the inverse reconstruction of a pairwise Ising spin glass based on susceptibility propagation. We establish the conditions under which the susceptibility propagation algorithm is able to reconstruct the characteristics of the network given first- and second-order local observables, evaluate eventual errors due to various types of noise in the originally observed data, and discuss the scaling of the problem with the number of degrees of freedom

Information theoretic approach to interactive learning

The principles of statistical mechanics and information theory play an important role in learning and have inspired both theory and the design of numerous machine learning algorithms. The new aspect in this paper is a focus on integrating feedback from the learner. A quantitative approach to interactive learning and adaptive behavior is proposed, integrating model- and decision-making into one theoretical framework. This paper follows simple principles by requiring that the observer's world model and action policy should result in maximal predictive power at minimal complexity. Classes of optimal action policies and of optimal models are derived from an objective function that reflects this trade-off between prediction and complexity. The resulting optimal models then summarize, at different levels of abstraction, the process's causal organization in the presence of the learner's actions. A fundamental consequence of the proposed principle is that the learner's optimal action policies balance exploration and control as an emerging property. Interestingly, the explorative component is present in the absence of policy randomness, i.e. in the optimal deterministic behavior. This is a direct result of requiring maximal predictive power in the presence of feedback.Comment: 6 page

Stability of the replica symmetric solution for the information conveyed by by a neural network

The information that a pattern of firing in the output layer of a feedforward network of threshold-linear neurons conveys about the network's inputs is considered. A replica-symmetric solution is found to be stable for all but small amounts of noise. The region of instability depends on the contribution of the threshold and the sparseness: for distributed pattern distributions, the unstable region extends to higher noise variances than for very sparse distributions, for which it is almost nonexistant.Comment: 19 pages, LaTeX, 5 figures. Also available at http://www.mrc-bbc.ox.ac.uk/~schultz/papers.html . Submitted to Phys. Rev. E Minor change

Dimensionality and dynamics in the behavior of C. elegans

A major challenge in analyzing animal behavior is to discover some underlying simplicity in complex motor actions. Here we show that the space of shapes adopted by the nematode C. elegans is surprisingly low dimensional, with just four dimensions accounting for 95% of the shape variance, and we partially reconstruct "equations of motion" for the dynamics in this space. These dynamics have multiple attractors, and we find that the worm visits these in a rapid and almost completely deterministic response to weak thermal stimuli. Stimulus-dependent correlations among the different modes suggest that one can generate more reliable behaviors by synchronizing stimuli to the state of the worm in shape space. We confirm this prediction, effectively "steering" the worm in real time.Comment: 9 pages, 6 figures, minor correction