Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model

Understanding the reasons for the success of deep neural networks trained using stochastic gradient-based methods is a key open problem for the nascent theory of deep learning. The types of data where these networks are most successful, such as images or sequences of speech, are characterized by intricate correlations. Yet, most theoretical work on neural networks does not explicitly model training data or assumes that elements of each data sample are drawn independently from some factorized probability distribution. These approaches are, thus, by construction blind to the correlation structure of real-world datasets and their impact on learning in neural networks. Here, we introduce a generative model for structured datasets that we call the hidden manifold model. The idea is to construct high-dimensional inputs that lie on a lower-dimensional manifold, with labels that depend only on their position within this manifold, akin to a single-layer decoder or generator in a generative adversarial network. We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a "Gaussian equivalence property"(GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times. This approach permits us to analyze in detail how a neural network learns functions of increasing complexity during training, how its performance depends on its size, and how it is impacted by parameters such as the learning rate or the dimension of the hidden manifold

Role of the potential landscape on the single-file diffusion through channels.

Transport of colloid particles through narrow channels is ubiquitous in cell biology as well as becoming increasingly important for microfluidic applications or targeted drug delivery. Membrane channels in cells are useful models for artificial designs because of their high efficiency, selectivity, and robustness to external fluctuations. Here, we model the passive channels that let cargo simply diffuse through them, affected by a potential profile along the way. Passive transporters achieve high levels of efficiency and specificity from binding interactions with the cargo inside the channel. This however leads to a paradox: why should channels which are so narrow that they are blocked by their cargo evolve to have binding regions for their cargo if that will effectively block them? Using Brownian dynamics simulations, we show that different potentials, notably symmetric, increase the flux through narrow passive channels - and investigate how shape and depth of potentials influence the flux. We find that there exist optimal depths for certain potential shapes and that it is most efficient to apply a small force over an extended region of the channel. On the other hand, having several spatially discrete binding pockets will not alter the flux significantly. We also explore the role of many-particle effects arising from pairwise particle interactions with their neighbours and demonstrate that the relative changes in flux can be accounted for by the kinetics of the absorption reaction at the end of the channel.Simulations were funded by the Cavendish Laboratory teaching committee and per- formed using the Darwin Supercomputer of the Univer- sity of Cambridge High Performance Computing Service (http://www.hpc.cam.ac.uk/), provided by Dell Inc. us- ing Strategic Research Infrastructure Funding from the Higher Education Funding Council for England.This is the accepted manuscript. The final version is available from AIP at http://scitation.aip.org/content/aip/journal/jcp/141/22/10.1063/1.490317

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches

Perspectives on adaptive dynamical systems

Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems like the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges, and give perspectives on future research directions, looking to inspire interdisciplinary approaches.Comment: 46 pages, 9 figure

Highly efficient doping of carbon nanotube films with chloroauric acid by dip-coating

Funding Information: We thank Stepan Romanov for in situ measurements of SWCNT film resistance during dip-coating process. This work is supported by the Russian Science Foundation (RSF) under grant number 17-19-01787 (dip-coating doping results) and the Ministry of Science and Higher Education of the Russian Federation (project no. FZSR-2020-0007 in the framework of the state assignment no. 075-03-2020-097/1). The authors thank the Council on grants of the President of the Russian Federation grant number НШ-1330.2022.1.3. Publisher Copyright: © 2022 The AuthorsSingle-walled carbon nanotube (SWCNT) based transparent and conductive films (TCFs) are one of the most prospective materials for novel flexible and stretchable electronic devices. Development of reproducible and scalable doping procedure is the key step towards the widespread implementation of SWCNT TCFs. Here, we thoroughly investigate a dip-coating technique for SWCNT doping as a promising approach for the practical manufacturing of SWCNT films with high performance. We examine the effect of dip-coating parameters on optical and electrical properties of the films using HAuCl4 solution in isopropyl alcohol (IPA) and in situ investigate doping effects. This method appeared to easily fine-tune the optoelectronic parameters of SWCNT films and achieve a record sheet resistance value of 36 Ohm/sq. at the 90% transmittance in the middle of visible spectral range by increasing a work function value from 4.8 (for pristine SWCNTs) to 6.0 eV. The proposed approach allows efficient, uniform, and reproducible fabrication of highly conductive and transparent SWCNT films and opens an avenue for precise tailoring of SWCNT Fermi level for optoelectronic devices.Peer reviewe

Joint effect of ethylene and toluene on carbon nanotube growth

Funding Information: The authors thank Anton Bubis and MIPT Shared Facilities Center for the help with scanning electron microscopy. E.M.Kh. and D.V.K. acknowledge Russian Science Foundation grant No. 20-73-10256 (synthesis of SWCNTs and optical measurements). E.M.Kh. and T.K. acknowledge Academy of Finland project No. 320167 (PREIN Flagship - Aalto University). Q.Zh. and E.I.K. acknowledge Academy of Finland project No. 316572 (CNTstress). A.G.N. and A.E.G. acknowledge the Russian Science Foundation (project No. 17-19-01787 - doping of carbon nanotubes). E.I.K. acknowledge the Academy of Finland for Mobility Grant (application number 334466). A.P.Ts. acknowledges the EDUFI Fellowship (No. TM-19-11079) from the Finnish National Agency for Education and the Magnus Ehrnrooth Foundation (the Finnish Society of Sciences and Letters) for personal financial support. This work made use of the Aalto University Otanano, RAMI and Bioeconomy infrastructures. Funding Information: The authors thank Anton Bubis and MIPT Shared Facilities Center for the help with scanning electron microscopy. E.M.Kh. and D.V.K. acknowledge Russian Science Foundation grant No. 20-73-10256 (synthesis of SWCNTs and optical measurements). E.M.Kh. and T.K. acknowledge Academy of Finland project No. 320167 (PREIN Flagship - Aalto University ). Q.Zh. and E.I.K. acknowledge Academy of Finland project No. 316572 (CNTstress). A.G.N. and A.E.G. acknowledge the Russian Science Foundation (project No. 17-19-01787 - doping of carbon nanotubes). E.I.K. acknowledge the Academy of Finland for Mobility Grant (application number 334466). A.P.Ts. acknowledges the EDUFI Fellowship (No. TM-19-11079) from the Finnish National Agency for Education and the Magnus Ehrnrooth Foundation (the Finnish Society of Sciences and Letters ) for personal financial support. This work made use of the Aalto University Otanano, RAMI and Bioeconomy infrastructures. Publisher Copyright: © 2021 Elsevier LtdThis work contributes to the understanding of single-walled carbon nanotube synthesis by an aerosol CVD method using ethylene and toluene as a hybrid carbon source. We evaluated an extensive set of synthesis conditions revealing the role of ferrocene, toluene, and ethylene. We found the fundamental role of ethylene promoting nanotube nucleation and catalyst activation degree at all the concentrations studied, and enhancing nanotube growth at low ethylene content. We observed the interplay effect of toluene and ethylene concentrations on the nanotube growth rate, accompanied by the detrimental effect of toluene on catalyst activation degree. Nevertheless, toluene apparently promotes nanotube crystallinity, increasing the film conductivity while used as an individual carbon source. Adjusting the ethylene and toluene concentrations, we produced carbon nanotube-based transparent and conductive films with an equivalent sheet resistance (at 90% transmittance at 550 nm wavelength) value of 57 Ω/□ at the synthesis yield of 0.24 cm2 L−1, which is at least two times higher than the results reported earlier.Peer reviewe

Single-step extraction of small-diameter single-walled carbon nanotubes in the presence of riboflavin

Funding Information: A.G.N and A.E.G thank the Council on grants of the President of the Russian Federation grant number НШ-1330.2022.1.3 and Russian Science Foundation (Project no. 22-13-00436). The theoretical calculations (L.Y.A.) were supported by Russian Science Foundation (Project identifier 21-79-10411). E.D.O. is grateful to RSF (project no. 21-72-20050) for support of photoluminescence investigations. Publisher Copyright: © 2022 Kalachikova et al.; licensee Beilstein-Institut. License and terms: see end of document.We propose a novel approach to disperse and extract small-diameter single-walled carbon nanotubes (SWCNTs) using an aqueous solution of riboflavin and Sephacryl gel. The extraction of small-diameter semiconducting SWCNTs was observed, regardless of the initial diameter distribution of the SWCNTs. Dispersion of SWCNTs occurs due to the adsorption of π-conjugated isoalloxazine moieties on the surface of small-diameter nanotubes and interactions between hydroxy groups of ribityl chains with water. During the SWCNT extraction, specific adsorption of riboflavin to SWCNTs leads to the minimization of interactions between the SWCNTs and gel media. Our experimental findings are supported by ab initio calculations demonstrating the impact of the riboflavin wrapping pattern around the SWCNTs on their interaction with the allyl dextran gel.Peer reviewe

Synthesis of 18? annulenic fluorofullerenes from tertiary carbanions: size matters!

A range of tertiary carbanions XCH(CO2Et)2 of differing sizes have been reacted with C60F18 to assess the steric effect of X on the position of nucleophilic substitution. For X= CO2Et, NO2, P(O)(OMe)2, SO2CH2Ph, the all trans annulenes (trannulenes) were obtained as a result of extended SN2'(i.e. SN2'') substitution; in the case of the phosphorus compound, with reduced amounts of base (DBU) dephosphonylation of one or more P(O)(OMe)2 groups by hydrogen occurred. Trannulene formation did not occur for X= F, CN due to the smaller size of the nucleophile, and in the latter case substitution was shown to take place by an SN2' mechanism, resulting in the addend being adjacent to a fluorine addend. Trannulenes (X= CO2Et, Br, Cl) exhibited reversible one-electron reductions at potentials (–0.02 to –0.09 V) significantly more positive than for [60]fullerene. Trannulene (X= NO2) exhibited an irreversible one-electron reduction (0.08 V); the irreversibility may be associated with fluorine loss. Conformational isomerism at temperatures below 298 K was observed for all trannulene derivatives as a result of eclipsing addend–addend interactions. Minimum energy conformations with a rotational energy barrier of 12–15 kcal mol–1 were observed when these interactions are calculated using molecular mechanics

Direct injection of SWCNTs into liquid after supercritical nitrogen treatment

We developed a novel robust technique to produce high-quality dispersions of debundled SWCNTs in aqueous solutions. Direct injection of SWCNTs treated with supercritical nitrogen into the aqueous surfactant solution facilitates the dispersion process without the need of extensive ultrasonication. According to photoluminescence and absorbance measurements, the mild ultrasonic treatment of such dispersions resulted in a higher yield of individual SWCNTs, compared to pristine tubes and tubes collected after the supercritical treatment in the form of powder.Peer reviewe