Alternating Synthetic and Real Gradients for Neural Language Modeling

Training recurrent neural networks (RNNs) with backpropagation through time (BPTT) has known drawbacks such as being difficult to capture longterm dependencies in sequences. Successful alternatives to BPTT have not yet been discovered. Recently, BP with synthetic gradients by a decoupled neural interface module has been proposed to replace BPTT for training RNNs. On the other hand, it has been shown that the representations learned with synthetic and real gradients are different though they are functionally identical. In this project, we explore ways of combining synthetic and real gradients with application to neural language modeling tasks. Empirically, we demonstrate the effectiveness of alternating training with synthetic and real gradients after periodic warm restarts on language modeling tasks

A mesoscale finite element simulation of intermittent plastic flow of micropillar compression under hybrid loading mode

The plastic deformation of the micropillar proceeds as a series of strain bursts, showing an intermittent plastic flow. In this work, we present a stochastic finite element method in crystal plasticity to describe the intermittent characteristic of crystal deformation under the hybrid loading mode (HLM). The microscopic boundary conditions(MBCs) using the HLM are studied and they are demonstrated to be different in various deformation periods such as loading stage, burst slip and holding stage, which occur alternatively as the plastic flow proceeds. In order to determine the MBCs, we use the Monte Carlo (MC) stochastic model to predict the amplitude of the burst displacement and then incorporate such model into our established continuum framework accounting for the characteristics of the strain burst. By implementing this continuum model into the finite element analysis, we predict the plastic flow of single crystal nickel micropillars that deform under uniaxial compression along the [2 6 9] crystalline direction. The simulation results indicate clearly visible strain bursts in the course of plastic deformation, producing a stair-case like stress-strain behavior that agrees well with experimental observations. The computational results reveal that the intermittent flow in the micrometer-scale is intensified due to the increasing amplitude of the strain burst, as well as the occurrence of successive strain bursts rather than the discrete strain bursts, with decreasing of the specimen size. In addition, the micropillar displacement in the context of burst activity predicted from our simulations is similar to the experimental observations. We demonstrate that our simulation method could provide further insights into the intermittent plastic flow characteristics such as burst time duration, micropillar velocity; plus, it is feasible to apply this method to investigate the plastic flow behaviors under complex loading conditions

Characteristics of oligonucleotide frequencies across genomes: Conservation versus variation, strand symmetry, and evolutionary implications

One of the objectives of evolutionary genomics is to reveal the genetic information contained in the primordial genome (called the primary genetic information in this paper, with the primordial genome defined here as the most primitive nucleic acid genome for earth’s life) by searching for primitive traits or relics remained in modern genomes. As the shorter a sequence is, the less probable it would be modified during genome evolution. For that reason, some characteristics of very short nucleotide sequences would have considerable chances to persist during billions of years of evolution. Consequently, conservation of certain genomic features of mononucleotides, dinucleotides, and higher-order oligonucleotides across various genomes may exist; some, if not all, of these features would be relics of the primary genetic information. Based on this assumption, we analyzed the pattern of frequencies of mononucleotides, dinucleotides, and higher-order oligonucleotides of the whole-genome sequences from 458 species (including archaea, bacteria, and eukaryotes). Also, we studied the phenomenon of strand symmetry in these genomes. The results show that the conservation of frequencies of some dinucleotides and higher-order oligonucleotides across genomes does exist, and that strand symmetry is a ubiquitous and explicit phenomenon that may contribute to frequency conservation. We propose a new hypothesis for the origin of strand symmetry and frequency conservation as well as for the constitution of early genomes. We conclude that the phenomena of strand symmetry and the pattern of frequency conservation would be original features of the primary genetic information

A two-step approach to model precipitation extremes in California based on max-stable and marginal point processes

In modeling spatial extremes, the dependence structure is classically inferred by assuming that block maxima derive from max-stable processes. Weather stations provide daily records rather than just block maxima. The point process approach for univariate extreme value analysis, which uses more historical data and is preferred by some practitioners, does not adapt easily to the spatial setting. We propose a two-step approach with a composite likelihood that utilizes site-wise daily records in addition to block maxima. The procedure separates the estimation of marginal parameters and dependence parameters into two steps. The first step estimates the marginal parameters with an independence likelihood from the point process approach using daily records. Given the marginal parameter estimates, the second step estimates the dependence parameters with a pairwise likelihood using block maxima. In a simulation study, the two-step approach was found to be more efficient than the pairwise likelihood approach using only block maxima. The method was applied to study the effect of El Ni\~{n}o-Southern Oscillation on extreme precipitation in California with maximum daily winter precipitation from 35 sites over 55 years. Using site-specific generalized extreme value models, the two-step approach led to more sites detected with the El Ni\~{n}o effect, narrower confidence intervals for return levels and tighter confidence regions for risk measures of jointly defined events.Comment: Published at http://dx.doi.org/10.1214/14-AOAS804 in the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org

Do External Interventions Work? The Case of Trade Reform Conditions in IMF Supported Programs

Trade reform conditions are common in IMF supported programs. Of the 99 countries that had IMF programs during 1993-2003, 77 had conditions on trade reforms in their programs. Since the WTO has not been found especially effective in promoting trade openness for most developing countries, it is of great interest to see if the IMF has been more effective as it combines carrots and sticks not available to the WTO. Yet, the effectiveness of trade conditions in IMF programs has not been systematically studied. Using a unique dataset, this paper provides such an assessment. It finds that trade conditions are associated with an increase in trade openness on average, but the effect comes mostly from countries that, by some measure, have a high degree of "willingness to reform."

Collateral Damage: Exchange Controls and International Trade

While new conventional wisdom warns that developing countries should be aware of the risks of premature capital account liberalization, the costs of not removing exchange controls have received much less attention. This paper investigates the negative effects of exchange controls on trade. To minimize evasion of controls, countries often intensify inspections at the border and increase documentation requirements. Thus, the cost of conducting trade rises. The paper finds that a one standard-deviation increase in the controls on trade payment has the same negative effect on trade as an increase in tariff by about 14 percentage points. A one standard-deviation increase in the controls on FX transactions reduces trade by the same amount as a rise in tariff by 11 percentage points. Therefore, the collateral damage in terms of foregone trade is sizable.

Pair production of 125 GeV Higgs boson in the SM extension with color-octet scalars at the LHC

Although the Higgs boson mass and single production rate have been determined more or less precisely, its other properties may deviate significantly from its predictions in the standard model (SM) due to the uncertainty of Higgs data. In this work we study the Higgs pair production at the LHC in the Manohar-Wise model, which extends the SM by one family of color-octet and isospin-doublet scalars. We first scanned over the parameter space of the Manohar-Wise model considering exprimental constraints and performed fits in the model to the latest Higgs data by using the ATLAS and CMS data separately. Then we calculated the Higgs pair production rate and investigated the potential of its discovery at the LHC14. We conclude that: (i) Under current constrains including Higgs data after Run I of the LHC, the cross section of Higgs pair production in the Manohar-Wise model can be enhanced up to even $10^3$ times prediction in the SM. (ii) Moreover, the sizable enhancement comes from the contributions of the CP-odd color-octet scalar $S^A_I$. For lighter scalar $S^A_I$ and larger values of $|\lambda_I|$, the cross section of Higgs pair production can be much larger. (iii) After running again of LHC at 14 TeV, most of the parameter spaces in the Manohar-Wise model can be test. For an integrated luminosity of 100 fb$^{-1}$ at the LHC14, when the normalized ratio $R=10$, the process of Higgs pair production can be detected.Comment: 13 pages, 4 figure