Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials

A recent goal in the theory of deep learning is to identify how neural networks can escape the "lazy training," or Neural Tangent Kernel (NTK) regime, where the network is coupled with its first order Taylor expansion at initialization. While the NTK is minimax optimal for learning dense polynomials (Ghorbani et al, 2021), it cannot learn features, and hence has poor sample complexity for learning many classes of functions including sparse polynomials. Recent works have thus aimed to identify settings where gradient based algorithms provably generalize better than the NTK. One such example is the "QuadNTK" approach of Bai and Lee (2020), which analyzes the second-order term in the Taylor expansion. Bai and Lee (2020) show that the second-order term can learn sparse polynomials efficiently; however, it sacrifices the ability to learn general dense polynomials. In this paper, we analyze how gradient descent on a two-layer neural network can escape the NTK regime by utilizing a spectral characterization of the NTK (Montanari and Zhong, 2020) and building on the QuadNTK approach. We first expand upon the spectral analysis to identify "good" directions in parameter space in which we can move without harming generalization. Next, we show that a wide two-layer neural network can jointly use the NTK and QuadNTK to fit target functions consisting of a dense low-degree term and a sparse high-degree term -- something neither the NTK nor the QuadNTK can do on their own. Finally, we construct a regularizer which encourages our parameter vector to move in the "good" directions, and show that gradient descent on the regularized loss will converge to a global minimizer, which also has low test error. This yields an end to end convergence and generalization guarantee with provable sample complexity improvement over both the NTK and QuadNTK on their own.Comment: v2: NeurIPS 2022 camera ready versio

A Narrow Linewidth Singly Resonant ZGP OPO for Multiple Lidar Applications

A singly resonant, injection seeded Zinc Germanium Phosphide (ZGP) optical parametric oscillator (OPO), capable to tune over 4.3-10.1 microns, is demonstrated. This ZGP OPO uses a bow-tie cavity with a partially reflective mirror for injection seeding at the signal wavelength. The injection seed source can be either a continuous wave 3.39 m laser or a tunable near-infrared OPO laser, which provides wide wavelength tuning capability. The injection seeded ZGP OPO narrows the idler wavelength linewidth to less than 1nm, limited by the measurement resolution of the monochromator. This device has potential to be used as a transmitter for multiple purpose lidar applications

Highly Efficient Q-switched Ho:YLF Laser Pumped by Tm:Fiber Laser

A highly efficient Q-switched Ho:YLF laser pumped by a Tm:fiber laser has been designed and demonstrated. When the pump power is 30 W, the pulse energy is 30mJ at the repetition rate of 100Hz

Efficient Operation of Conductively Cooled Ho:Tm:LuLiF Laser Oscillator/Amplifier

A conductively-cooled Ho:Tm:LuLiF laser oscillator generates 1.6J normal mode pulses at 10Hz with optical to optical efficiency of 20%. When the laser head module is used as the amplifier, the double-pass small-signal amplification excesses 25

Neuron-Specific HuR-Deficient Mice Spontaneously Develop Motor Neuron Disease

Human Ag R (HuR) is an RNA binding protein in the ELAVL protein family. To study the neuron-specific function of HuR, we generated inducible, neuron-specific HuR-deficient mice of both sexes. After tamoxifen-induced deletion of HuR, these mice developed a phenotype consisting of poor balance, decreased movement, and decreased strength. They performed significantly worse on the rotarod test compared with littermate control mice, indicating coordination deficiency. Using the grip-strength test, it was also determined that the forelimbs of neuron-specific HuR-deficient mice were much weaker than littermate control mice. Immunostaining of the brain and cervical spinal cord showed that HuR-deficient neurons had increased levels of cleaved caspase-3, a hallmark of cell apoptosis. Caspase-3 cleavage was especially strong in pyramidal neurons and α motor neurons of HuR-deficient mice. Genome-wide microarray and real-time PCR analysis further indicated that HuR deficiency in neurons resulted in altered expression of genes in the brain involved in cell growth, including trichoplein keratin filament-binding protein, Cdkn2c, G-protein signaling modulator 2, immediate early response 2, superoxide dismutase 1, and Bcl2. The additional enriched Gene Ontology terms in the brain tissues of neuron-specific HuR-deficient mice were largely related to inflammation, including IFN-induced genes and complement components. Importantly, some of these HuR-regulated genes were also significantly altered in the brain and spinal cord of patients with amyotrophic lateral sclerosis. Additionally, neuronal HuR deficiency resulted in the redistribution of TDP43 to cytosolic granules, which has been linked to motor neuron disease. Taken together, we propose that this neuron-specific HuR-deficient mouse strain can potentially be used as a motor neuron disease model

Towards Understanding Hierarchical Learning: Benefits of Neural Representations

Deep neural networks can empirically perform efficient hierarchical learning, in which the layers learn useful representations of the data. However, how they make use of the intermediate representations are not explained by recent theories that relate them to "shallow learners" such as kernels. In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks and can be advantageous over raw inputs. We consider a fixed, randomly initialized neural network as a representation function fed into another trainable network. When the trainable network is the quadratic Taylor model of a wide two-layer network, we show that neural representation can achieve improved sample complexities compared with the raw input: For learning a low-rank degree-$p$ polynomial ($p \geq 4$) in $d$ dimension, neural representation requires only $\tilde{O}(d^{\lceil p/2 \rceil})$ samples, while the best-known sample complexity upper bound for the raw input is $\tilde{O}(d^{p-1})$. We contrast our result with a lower bound showing that neural representations do not improve over the raw input (in the infinite width limit), when the trainable network is instead a neural tangent kernel. Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.Comment: 41 pages, published in NeurIPS 202

High Repetition Rate Pulsed 2-Micron Laser Transmitter for Coherent CO2 DIAL Measurement

A high repetition rate, highly efficient, Q-switched 2-micron laser system as the transmitter of a coherent differential absorption lidar for CO2 measurement has been developed at NASA Langley Research Center. Such a laser transmitter is a master-slave laser system. The master laser operates in a single frequency, either on-line or off-line of a selected CO2 absorption line. The slave laser is a Q-switched ring-cavity Ho:YLF laser which is pumped by a Tm:fiber laser. The repetition rate can be adjusted from a few hundred Hz to 10 kHz. The injection seeding success rate is from 99.4% to 99.95%. For 1 kHz operation, the output pulse energy is 5.5mJ with the pulse length of approximately 50 ns. The optical-to-optical efficiency is 39% when the pump power is 14.5W. The measured standard deviation of the laser frequency jitter is about 3 MHz

Measure representation and multifractal analysis of complete genomes

This paper introduces the notion of measure representation of DNA sequences. Spectral analysis and multifractal analysis are then performed on the measure representations of a large number of complete genomes. The main aim of this paper is to discuss the multifractal property of the measure representation and the classification of bacteria. From the measure representations and the values of the $D_{q}$ spectra and related $C_{q}$ curves, it is concluded that these complete genomes are not random sequences. In fact, spectral analyses performed indicate that these measure representations considered as time series, exhibit strong long-range correlation. For substrings with length K=8, the $D_{q}$ spectra of all organisms studied are multifractal-like and sufficiently smooth for the $C_{q}$ curves to be meaningful. The $C_{q}$ curves of all bacteria resemble a classical phase transition at a critical point. But the 'analogous' phase transitions of chromosomes of non-bacteria organisms are different. Apart from Chromosome 1 of {\it C. elegans}, they exhibit the shape of double-peaked specific heat function.Comment: 12 pages with 9 figures and 1 tabl

Sintered silver finite element modelling and reliability based design optimisation in power electronic module

This paper discusses the design for reliability of a sintered silver structure in a power electronic module based on the computational approach that composed of high fidelity analysis, reduced order modelling, numerical risk analysis, and optimisation. The methodology was demonstrated on sintered silver interconnect sandwiched between silicon carbide chip and copper substrate in a power electronic module. In particular, sintered silver reliability due to thermal fatigue material degradation is one of the main concerns. Thermo-mechanical behaviour of the power module sintered silver joint structure is simulated by finite element analysis for cyclic temperature loading profile in order to capture the strain distribution. The discussion was on methods for approximate reduced order modelling based on interpolation techniques using Kriging and radial basis functions. The reduced order modelling approach uses prediction data for the thermo-mechanical behaviour. The fatigue lifetime of the sintered silver interconnect and the warpage of the interconnect layer was particular interest in this study. The reduced order models were used for the analysis of the effect of design uncertainties on the reliability of the sintered silver layer. To assess the effect of uncertain design data, a method for estimating the variation of reliability related metrics namely Latin Hypercube sampling was utilised. The product capability indices are evaluated from the distributions fitted to the histogram resulting from Latin Hypercube sampling technique. A reliability based design optimisation was demonstrated using Particle Swarm Optimisation algorithm for constraint optimisation task consists of optimising two different characteristic performance metrics such as the thermo-mechanical plastic strain accumulation per cycle on the sintered layer and the thermally induced warpage