The Stochastic complexity of spin models: Are pairwise models really simple?

Models can be simple for different reasons: because they yield a simple and computationally efficient interpretation of a generic dataset (e.g. in terms of pairwise dependences) - as in statistical learning - or because they capture the essential ingredients of a specific phenomenon - as e.g. in physics - leading to non-trivial falsifiable predictions. In information theory and Bayesian inference, the simplicity of a model is precisely quantified in the stochastic complexity, which measures the number of bits needed to encode its parameters. In order to understand how simple models look like, we study the stochastic complexity of spin models with interactions of arbitrary order. We highlight the existence of invariances with respect to bijections within the space of operators, which allow us to partition the space of all models into equivalence classes, in which models share the same complexity. We thus found that the complexity (or simplicity) of a model is not determined by the order of the interactions, but rather by their mutual arrangements. Models where statistical dependencies are localized on non-overlapping groups of few variables (and that afford predictions on independencies that are easy to falsify) are simple. On the contrary, fully connected pairwise models, which are often used in statistical learning, appear to be highly complex, because of their extended set of interactions

Understanding Neural Coding on Latent Manifolds by Sharing Features and Dividing Ensembles

Systems neuroscience relies on two complementary views of neural data, characterized by single neuron tuning curves and analysis of population activity. These two perspectives combine elegantly in neural latent variable models that constrain the relationship between latent variables and neural activity, modeled by simple tuning curve functions. This has recently been demonstrated using Gaussian processes, with applications to realistic and topologically relevant latent manifolds. Those and previous models, however, missed crucial shared coding properties of neural populations. We propose feature sharing across neural tuning curves, which significantly improves performance and leads to better-behaved optimization. We also propose a solution to the problem of ensemble detection, whereby different groups of neurons, i.e., ensembles, can be modulated by different latent manifolds. This is achieved through a soft clustering of neurons during training, thus allowing for the separation of mixed neural populations in an unsupervised manner. These innovations lead to more interpretable models of neural population activity that train well and perform better even on mixtures of complex latent manifolds. Finally, we apply our method on a recently published grid cell dataset, recovering distinct ensembles, inferring toroidal latents and predicting neural tuning curves all in a single integrated modeling framework

Should we increase instruction time in low achieving schools? Evidence from Southern Italy

AbstractThis paper investigates the short term effects of a large scale intervention, funded by the European Social Fund, which provides additional instruction time to selected classes of lower secondary schools in Southern Italy. Selection is addressed using institutional rules that regulate class formation: first year students are divided into groups distinguished by letters, they remain in the same group across grades at the school, and the composition of teachers assigned to groups is stable over time. Using a difference-in-differences strategy, we consider consecutive cohorts of first year students enrolled in the same group. We compare participating groups to non-participating groups within the same school, as well as to groups in non-participating schools. We find that the intervention raised scores in mathematics for students from the least advantaged backgrounds. We also find that targeting the best students with extra activities in language comes at the cost of lowering performance in mathematics. We go beyond average effects, finding that the positive effect for mathematics is driven by larger effects for the best students

The appropriateness of ignorance in the inverse kinetic Ising model

We develop efficient ways to consider and correct for the effects of hidden units for the paradigmatic case of the inverse kinetic Ising model with fully asymmetric couplings. We identify two sources of error in reconstructing the connectivity among the observed units while ignoring part of the network. One leads to a systematic bias in the inferred parameters, whereas the other involves correlations between the visible and hidden populations and has a magnitude that depends on the coupling strength. We estimate these two terms using a mean field approach and derive self-consistent equations for the couplings accounting for the systematic bias. Through application of these methods on simple networks of varying relative population size and connectivity strength, we assess how and under what conditions the hidden portion can influence inference and to what degree it can be crudely estimated. We find that for weak to moderately coupled systems, the effects of the hidden units is a simple rotation that can be easily corrected for. For strongly coupled systems, the non-systematic term becomes large and can no longer be safely ignored, further highlighting the importance of understanding the average strength of couplings for a given system of interest

Should We Increase Instruction Time in Low Achieving Schools? Evidence from Southern Italy

This paper investigates the short term effects of a large scale intervention, funded by the European Social Fund, which provides additional instruction time to selected classes of lower secondary schools in Southern Italy. Selection is addressed using institutional rules that regulate class formation: first year students are divided into groups distinguished by letters, they remain in the same group across grades at the school, and the composition of teachers assigned to groups is stable over time. Using a difference-in-differences strategy, we consider consecutive cohorts of first year students enrolled in the same group. We compare participating groups to non-participating groups within the same school, as well as to groups in non-participating schools. We find that the intervention raised scores in mathematics for students from the least advantaged backgrounds. We also find that targeting the best students with extra activities in language comes at the cost of lowering performance in mathematics. We go beyond average effects, finding that the positive effect for mathematics is driven by larger effects for the best students

Learning with unknowns: analyzing biological data in the presence of hidden variables

Despite our improved ability to probe biological systems at a higher spatio-temporal resolution, the high dimensionality of the biological systems often prevents sufficient sampling of the state space. Even with large scale datasets, such as gene microarrays or multi-neuronal recording techniques, the variables we are recording from are typically only a small subset, if wisely chosen, representing the most relevant degrees of freedom. The remaining variables, or the so called hidden variables, are most likely coupled to the observed ones, and affect their statistics and consequently our inference about the function of the system and the way it performs this function. Two important questions then arise in this context: which variables should we choose to observe and collect data from? and how much can we learn from data in the presence of hidden variables? In this paper we suggest that recent algorithmic developments rooting in the statistical physics of complex systems constitute a promising set of tools to extract relevant features from high-throughput data and a fruitful avenue of research for coming years

The Stochastic Complexity of Spin Models: Are Pairwise Models Really Simple?

Models can be simple for different reasons: because they yield a simple and computationally efficient interpretation of a generic dataset (e.g., in terms of pairwise dependencies)—as in statistical learning—or because they capture the laws of a specific phenomenon—as e.g., in physics—leading to non-trivial falsifiable predictions. In information theory, the simplicity of a model is quantified by the stochastic complexity, which measures the number of bits needed to encode its parameters. In order to understand how simple models look like, we study the stochastic complexity of spin models with interactions of arbitrary order. We show that bijections within the space of possible interactions preserve the stochastic complexity, which allows to partition the space of all models into equivalence classes. We thus found that the simplicity of a model is not determined by the order of the interactions, but rather by their mutual arrangements. Models where statistical dependencies are localized on non-overlapping groups of few variables are simple, affording predictions on independencies that are easy to falsify. On the contrary, fully connected pairwise models, which are often used in statistical learning, appear to be highly complex, because of their extended set of interactions, and they are hard to falsify