Telling Cause from Effect using MDL-based Local and Global Regression

We consider the fundamental problem of inferring the causal direction between two univariate numeric random variables $X$ and $Y$ from observational data. The two-variable case is especially difficult to solve since it is not possible to use standard conditional independence tests between the variables. To tackle this problem, we follow an information theoretic approach based on Kolmogorov complexity and use the Minimum Description Length (MDL) principle to provide a practical solution. In particular, we propose a compression scheme to encode local and global functional relations using MDL-based regression. We infer $X$ causes $Y$ in case it is shorter to describe $Y$ as a function of $X$ than the inverse direction. In addition, we introduce Slope, an efficient linear-time algorithm that through thorough empirical evaluation on both synthetic and real world data we show outperforms the state of the art by a wide margin.Comment: 10 pages, To appear in ICDM1

Causal Inference on Discrete Data using Additive Noise Models

Inferring the causal structure of a set of random variables from a finite sample of the joint distribution is an important problem in science. Recently, methods using additive noise models have been suggested to approach the case of continuous variables. In many situations, however, the variables of interest are discrete or even have only finitely many states. In this work we extend the notion of additive noise models to these cases. We prove that whenever the joint distribution \prob^{(X,Y)} admits such a model in one direction, e.g. Y=f(X)+N, N \independent X, it does not admit the reversed model X=g(Y)+\tilde N, \tilde N \independent Y as long as the model is chosen in a generic way. Based on these deliberations we propose an efficient new algorithm that is able to distinguish between cause and effect for a finite sample of discrete variables. In an extensive experimental study we show that this algorithm works both on synthetic and real data sets

Pulsar State Switching from Markov Transitions and Stochastic Resonance

Markov processes are shown to be consistent with metastable states seen in pulsar phenomena, including intensity nulling, pulse-shape mode changes, subpulse drift rates, spindown rates, and X-ray emission, based on the typically broad and monotonic distributions of state lifetimes. Markovianity implies a nonlinear magnetospheric system in which state changes occur stochastically, corresponding to transitions between local minima in an effective potential. State durations (though not transition times) are thus largely decoupled from the characteristic time scales of various magnetospheric processes. Dyadic states are common but some objects show at least four states with some transitions forbidden. Another case is the long-term intermittent pulsar B1931+24 that has binary radio-emission and torque states with wide, but non-monotonic duration distributions. It also shows a quasi-period of $38\pm5$ days in a 13-yr time sequence, suggesting stochastic resonance in a Markov system with a forcing function that could be strictly periodic or quasi-periodic. Nonlinear phenomena are associated with time-dependent activity in the acceleration region near each magnetic polar cap. The polar-cap diode is altered by feedback from the outer magnetosphere and by return currents from an equatorial disk that may also cause the neutron star to episodically charge and discharge. Orbital perturbations in the disk provide a natural periodicity for the forcing function in the stochastic resonance interpretation of B1931+24. Disk dynamics may introduce additional time scales in observed phenomena. Future work can test the Markov interpretation, identify which pulsar types have a propensity for state changes, and clarify the role of selection effects.Comment: 25 pages, 6 figures, submitted to the Astrophysical Journa

Determination of Nonlinear Genetic Architecture using Compressed Sensing

We introduce a statistical method that can reconstruct nonlinear genetic models (i.e., including epistasis, or gene-gene interactions) from phenotype-genotype (GWAS) data. The computational and data resource requirements are similar to those necessary for reconstruction of linear genetic models (or identification of gene-trait associations), assuming a condition of generalized sparsity, which limits the total number of gene-gene interactions. An example of a sparse nonlinear model is one in which a typical locus interacts with several or even many others, but only a small subset of all possible interactions exist. It seems plausible that most genetic architectures fall in this category. Our method uses a generalization of compressed sensing (L1-penalized regression) applied to nonlinear functions of the sensing matrix. We give theoretical arguments suggesting that the method is nearly optimal in performance, and demonstrate its effectiveness on broad classes of nonlinear genetic models using both real and simulated human genomes.Comment: 20 pages, 8 figures. arXiv admin note: text overlap with arXiv:1408.342

Identifiability of Causal Graphs using Functional Models

This work addresses the following question: Under what assumptions on the data generating process can one infer the causal graph from the joint distribution? The approach taken by conditional independence-based causal discovery methods is based on two assumptions: the Markov condition and faithfulness. It has been shown that under these assumptions the causal graph can be identified up to Markov equivalence (some arrows remain undirected) using methods like the PC algorithm. In this work we propose an alternative by defining Identifiable Functional Model Classes (IFMOCs). As our main theorem we prove that if the data generating process belongs to an IFMOC, one can identify the complete causal graph. To the best of our knowledge this is the first identifiability result of this kind that is not limited to linear functional relationships. We discuss how the IFMOC assumption and the Markov and faithfulness assumptions relate to each other and explain why we believe that the IFMOC assumption can be tested more easily on given data. We further provide a practical algorithm that recovers the causal graph from finitely many data; experiments on simulated data support the theoretical findings

Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines

Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines, a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. Synaptic sampling machines perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate & fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based synaptic sampling machines outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware