A mathematical theory of semantic development in deep neural networks

An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: what are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep learning dynamics to give rise to these regularities

A circumpolar perspective on fluvial sediment flux to the Arctic ocean

Quantification of sediment fluxes from rivers is fundamental to understanding land‐ocean linkages in the Arctic. Numerous publications have focused on this subject over the past century, yet assessments of temporal trends are scarce and consensus on contemporary fluxes is lacking. Published estimates vary widely, but often provide little accessory information needed to interpret the differences. We present a pan‐arctic synthesis of sediment flux from 19 arctic rivers, primarily focusing on contributions from the eight largest ones. For this synthesis, historical records and recent unpublished data were compiled from Russian, Canadian, and United States sources. Evaluation of these data revealed no long‐term trends in sediment flux, but did show stepwise changes in the historical records of two of the rivers. In some cases, old values that do not reflect contemporary fluxes are still being reported, while in other cases, typographical errors have been propagated into the recent literature. Most of the discrepancy among published estimates, however, can be explained by differences in years of records examined and gauging stations used. Variations in sediment flux from year to year in arctic rivers are large, so estimates based on relatively few years can differ substantially. To determine best contemporary estimates of sediment flux for the eight largest arctic rivers, we used a combination of newly available data, historical records, and literature values. These estimates contribute to our understanding of carbon, nutrient, and contaminant transport to the Arctic Ocean and provide a baseline for detecting future anthropogenic or natural change in the Arctic

Network sensitivity to geographical configuration

Gravitational wave astronomy will require the coordinated analysis of data from the global network of gravitational wave observatories. Questions of how to optimally configure the global network arise in this context. We have elsewhere proposed a formalism which is employed here to compare different configurations of the network, using both the coincident network analysis method and the coherent network analysis method. We have constructed a network model to compute a figure-of-merit based on the detection rate for a population of standard-candle binary inspirals. We find that this measure of network quality is very sensitive to the geographic location of component detectors under a coincident network analysis, but comparatively insensitive under a coherent network analysis.Comment: 7 pages, 4 figures, accepted for proceedings of the 4th Edoardo Amaldi conference, incorporated referees' suggestions and corrected diagra

River Discharge

In 2014, combined discharge from the eight largest Arctic rivers (2,487 km3) was 10% greater than average discharge for the period 1980-1989. Values for 2013 (2,282 km3) and 2012 (2,240 km3) were 1% greater than and 1% less than the 1980-1989 average, respectively. For the first seven months of 2015, the combined discharge for the six largest Eurasian Arctic rivers shows that peak discharge was 10% greater and five days earlier than the 1980-1989 average for those months

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions. We provide an analytical description of these phenomena by finding new exact solutions to the nonlinear dynamics of deep learning. Our theoretical analysis also reveals the surprising finding that as the depth of a network approaches infinity, learning speed can nevertheless remain finite: for a special class of initial conditions on the weights, very deep networks incur only a finite, depth independent, delay in learning speed relative to shallow networks. We show that, under certain conditions on the training data, unsupervised pretraining can find this special class of initial conditions, while scaled random Gaussian initializations cannot. We further exhibit a new class of random orthogonal initial conditions on weights that, like unsupervised pre-training, enjoys depth independent learning times. We further show that these initial conditions also lead to faithful propagation of gradients even in deep nonlinear networks, as long as they operate in a special regime known as the edge of chaos.Comment: Submission to ICLR2014. Revised based on reviewer feedbac

TEACHING CONSUMER ECONOMICS USING TASTE PANELS

Consumer/Household Economics,

The Mid-Infrared and Optical Decay of SN 2011fe

We measure the decay rate of the mid-IR luminosity from type Ia supernova 2011fe between six months and one year after explosion using Spitzer/IRAC observations. The fading in the 3.6 micron channel is 1.48+/-0.02 mag/100d, which is similar to that seen in blue optical bands. The supernova brightness fades at 0.78+/-0.02 mag/100d in the 4.5 micron channel which is close to that observed in the near-IR. We argue that the difference is a result of doubly ionized iron-peak elements dominating the bluer IRAC band while singly ionized species are controlling the longer wavelength channel. To test this, we use Large Binocular Telescope spectra taken during the same phases to show that doubly ionized emission lines do fade more slowly than their singly ionized cousins. We also find that [Co III] emission fades at more than twice the radioactive decay rate due to the combination of decreasing excitation in the nebula, recombination and cobalt decaying to iron. The nebular emission velocities of [Fe III] and [Co III] lines show a smaller blue-shift than emission from singly ionized atoms. The Si II velocity gradient near maximum light combined with our nebular velocity measurements suggest SN 2011fe was a typical member of the `low velocity gradient' class of type Ia. Analyzing IRAC photometry from other supernovae we find that mid-IR color of type Ia events is correlated with the early light curve width and can be used as an indicator of the radioactive nickel yield.Comment: 11 pages, 6 figures, 3 table

Numerical wave optics and the lensing of gravitational waves by globular clusters

We consider the possible effects of gravitational lensing by globular clusters on gravitational waves from asymmetric neutron stars in our galaxy. In the lensing of gravitational waves, the long wavelength, compared with the usual case of optical lensing, can lead to the geometrical optics approximation being invalid, in which case a wave optical solution is necessary. In general, wave optical solutions can only be obtained numerically. We describe a computational method that is particularly well suited to numerical wave optics. This method enables us to compare the properties of several lens models for globular clusters without ever calling upon the geometrical optics approximation, though that approximation would sometimes have been valid. Finally, we estimate the probability that lensing by a globular cluster will significantly affect the detection, by ground-based laser interferometer detectors such as LIGO, of gravitational waves from an asymmetric neutron star in our galaxy, finding that the probability is insignificantly small.Comment: To appear in: Proceedings of the Eleventh Marcel Grossmann Meetin