Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning

In distributed training of deep neural networks, parallel mini-batch SGD is widely used to speed up the training process by using multiple workers. It uses multiple workers to sample local stochastic gradient in parallel, aggregates all gradients in a single server to obtain the average, and update each worker's local model using a SGD update with the averaged gradient. Ideally, parallel mini-batch SGD can achieve a linear speed-up of the training time (with respect to the number of workers) compared with SGD over a single worker. However, such linear scalability in practice is significantly limited by the growing demand for gradient communication as more workers are involved. Model averaging, which periodically averages individual models trained over parallel workers, is another common practice used for distributed training of deep neural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010). Compared with parallel mini-batch SGD, the communication overhead of model averaging is significantly reduced. Impressively, tremendous experimental works have verified that model averaging can still achieve a good speed-up of the training time as long as the averaging interval is carefully controlled. However, it remains a mystery in theory why such a simple heuristic works so well. This paper provides a thorough and rigorous theoretical study on why model averaging can work as well as parallel mini-batch SGD with significantly less communication overhead.Comment: No change has been made on the technical proof since V1. V2 changes the title to emphasize its value in deep learning; polishes the writing; and adds numerical simulations. This version further corrects a few typos in V2 posted a few days ago. A short version of this paper is accepted to AAAI 201

WW-representations of two-matrix models with infinite set of variables

The Hermitian, complex and fermionic two-matrix models with infinite set of variables are constructed. We show that these two-matrix models can be realized by the $W$-representations. In terms of the $W$-representations, we derive the compact expressions of correlators for these two-matrix models.Comment: 12 page

Enhancement of Hydroxyapatite Dissolution through Krypton Ion Irradiation

International audienceHydroxyapatite (HA) synthesized by a wet chemical route was subjected to heavy ion irradiation, using 4 MeV Krypton ions (Kr 17+) with ion fluence ranging from 1×10 13 to 1×10 15 ions/cm 2. Glancing incidence X-ray diffraction (GIXRD) results confirmed the phase purity of irradiated HA with a moderate contraction in lattice parameters, and further indicated irradiation-induced structural disorder, evident by a broadening of diffraction peaks. High-resolution transmission electron microscopy (HRTEM) observations indicated that the applied Kr irradiation induced significant damage in the hydroxyapatite lattice. Specifically, cavities were observed with their diameter and density varying with irradiation fluences while a radiation-induced crystalline-to-amorphous transition with increasing ion dose was identified. Raman and X-ray photoelectron spectroscopy (XPS) analysis further indicated the presence of irradiation-induced defects. Compositional analysis of pristine and irradiated materials following immersion in Tris (pH 7.4, 37℃) buffer showed that dissolution in vitro was enhanced by irradiation, reaching a peak for 0.1dpa. We examined the effects of irradiation on the early stages of the mouse osteoblast-like cells (MC3T3-E) response. A cell counting kit-8 assay (CCK-8 test) was carried out to investigate the cytotoxicity of samples, and viable cells can be observed on the irradiated materials.L'hydroxyapatite (HA) synthétisée par voie chimique a été soumise à une irradiation ionique lourde, en utilisant des ions Krypton 4 MeV (Kr 17+) avec une fluence ionique allant de 1 × 10 13 à 1 × 10 15 ions / cm 2. Incidence du regard X- Les résultats de la diffraction des rayons (GIXRD) ont confirmé la pureté de phase de l'AH irradié avec une contraction modérée des paramètres du réseau et ont en outre indiqué un trouble structurel induit par l'irradiation, évident par un élargissement des pics de diffraction. Des observations en microscopie électronique à transmission à haute résolution (HRTEM) ont indiqué que l'irradiation au Kr appliquée a induit des dommages importants dans le réseau d'hydroxyapatite. Plus précisément, des cavités ont été observées avec leur diamètre et leur densité variant avec les fluences d'irradiation tandis qu'une transition cristalline-amorphe induite par le rayonnement avec une dose ionique croissante a été identifiée. L'analyse par spectroscopie photoélectronique Raman et X (XPS) a en outre indiqué la présence de défauts induits par l'irradiation. L'analyse de la composition des matériaux vierges et irradiés après immersion dans du tampon Tris (pH 7,4, 37 ℃) a montré que la dissolution in vitro était améliorée par irradiation, atteignant un pic de 0,1 dpa. Nous avons examiné les effets de l'irradiation sur les premiers stades de la réponse des cellules de type ostéoblaste de souris (MC3T3-E). Un test de comptage de cellules kit 8 (test CCK-8) a été réalisé pour étudier la cytotoxicité des échantillons, et des cellules viables peuvent être observées sur les matériaux irradiés

The expression profile of microRNAs in a model of 7,12-dimethyl-benz[a]anthrance-induced oral carcinogenesis in Syrian hamster

<p>Abstract</p> <p>Background</p> <p>Non-coding RNA molecules, such as microRNAs, may play an important role in carcinogenesis. Recent studies have indicated that microRNAs are involved in initiation and progression of various malignancies. However, little work has been done to compare the microRNA expression patterns in oral cancer. In this study, we constructed an animal model of oral squamous cell carcinoma to investigate expression profiles of microRNAs in oral carcinogenesis.</p> <p>Methods</p> <p>The animal model of oral squamous cell carcinoma was conducted by tri-weekly (Monday, Wednesday, and Friday) painting with 5% DMBA in acetone. Six Syrian hamsters, including three from the treated group and three from the control group, were used as a training group for microRNA microarray analysis. All microarray data were analyzed by Significance Analysis of Microarrays (SAM) and CLUSTER 3.0 software, and this result was further confirmed by qRT-PCR assay.</p> <p>Results</p> <p>Seventeen microRNAs were differentially expressed in oral squamous cell carcinoma. Five microRNAs (hsa-miR-21, hsa-miR-200b, hsa-miR-221, hsa-miR-338, and mmu-miR-762) were significantly upregulated and twelve microRNAs (hsa-miR-16, hsa-miR-26a, hsa-miR-29a, hsa-miR-124a, hsa-miR-125b, mmu-miR-126-5p, hsa-miR-143, hsa-miR-145, hsa-miR-148b, hsa-miR-155, hsa-miR-199a, and hsa-miR-203) were down-regulated in cancer tissues. The expression levels of hsa-miR-21 and hsa-miR-16 seen with Stem-loop qRT-PCR were also seen in microarray analysis in all samples.</p> <p>Conclusion</p> <p>Our findings identified specific microRNA expression in oral squamous cell carcinoma and suggested that microRNAs have a role in oral carcinogenesis.</p

Latest Cosmological Constraints on Cardassian expansion models including the updated Gamma-ray bursts

In this paper, we constrain the Cardassian expansion models from the latest observations including the updated Gamma-ray bursts (GRBs), which calibrated cosmology-independently from the Union2 compilation of type Ia supernovae (SNe Ia). By combining the GRB data to the joint observations with the Union2 SNe Ia set, along with the Cosmic Microwave Background radiation observation from the seven-year Wilkinson Microwave Anisotropy Probe result, the baryonic acoustic oscillation observation from the spectroscopic Sloan Digital Sky Survey Data Release galaxy sample, we find significant constraints on model parameters of the original Cardassian model $\Omega_{{\rm M0}}=0.282_{-0.014}^{+0.015}$, $n= 0.03_{-0.05}^{+0.05}$; and $n= -0.16_{-3.26}^{+0.25}$, $\beta=0.76_{-0.58}^{+0.34}$ of the modified polytropic Cardassian model, which are consistent with the $\Lambda$CDM model in 1-$\sigma$ confidence region. From the reconstruction of the deceleration parameter $q(z)$ in Cardassian models, we obtain the transition redshift $z_{\rm T}=0.73\pm{0.04}$ for the original Cardassian model, and $z_{\rm\rm T}=0.68\pm{0.04}$ for the modified polytropic Cardassian model.Comment: 11 pages, 5 figures, 1 table; accepted for publication in Res. Astron. Astrophy

Single-shot, full characterization of the spatial wavefunction of light fields via Stokes tomography

Since the diffraction behavior of a light field is fully determined by its spatial wavefunction, i.e., its spatial complex amplitude (SCA), full characterization of spatial wavefunction, plays a vital role in modern optics from both the fundamental and applied aspects. In this work, we present a novel complex-amplitude profiler based on spatial Stokes tomography with the capability to fully determine the SCA of a light field in a single shot with high precision and resolution. The SCA slice observed at any propagation plane provides complete information about the light field, thus allowing us to further retrieve the complete beam structure in 3 dimensions space, as well as the exact modal constitution in terms of spatial degrees of freedom. The principle demonstrated here provides an important advancement for the full characterization of light beams with a broad spectrum of potential applications in various areas of optics, especially for the growing field of structured light

[μ-1,1′-(Butane-1,4-di­yl)di-1H-benz­imidazole-κ2 N 3:N 3′]bis­{[N,N′-bis(car­boxy­meth­yl)ethyl­enediamine-N,N′-di­acetato-κ5 O,O′,O′′,N,N′]mercury(II)} methanol disolvate

The binuclear title complex, [Hg2(C10H14N2O8)2(C18H18N4)]·2CH3OH, lies on an inversion center with the unique HgII ion coordinated in a disorted octa­hedral environment with one Hg—N bond significantly shorter than the other two. In the crystal structure, inter­molecular O—H⋯O hydrogen bonds link complex and solvent mol­ecules into a three-dimensional network

Fate of Antibiotic Resistant Pseudomonas putida and Broad Host Range Plasmid in Natural Soil Microcosms

Plasmid conjugation is one of the dominant mechanisms of horizontal gene transfer, playing a noticeable role in the rapid spread of antibiotic resistance genes (ARGs). Broad host range plasmids are known to transfer to diverse bacteria in extracted soil bacterial communities when evaluated by filter mating incubation. However, the persistence and dissemination of broad range plasmid in natural soil has not been well studied. In this study, Pseudomonas putida with a conjugative antibiotic resistance plasmid RP4 was inoculated into a soil microcosm, the fate and persistence of P. putida and RP4 were monitored by quantitative PCR. The concentrations of P. putida and RP4 both rapidly decreased within 15-day incubation. P. putida then decayed at a significantly lower rate during subsequent incubation, however, no further decay of RP4 was observed, resulting in an elevated RP4/P. putida ratio (up to 10) after 75-day incubation, which implied potential transfer of RP4 to soil microbiota. We further sorted RP4 recipient bacteria from the soil microcosms by fluorescence-activated cell sorting. Spread of RP4 increased during 75-day microcosm operation and was estimated at around 10-4 transconjugants per recipient at the end of incubation. Analysis of 16S rRNA gene sequences of transconjugants showed that host bacteria of RP4 were affiliated to more than 15 phyla, with increased diversity and shift in the composition of host bacteria. Proteobacteria was the most dominant phylum in the transconjugant pools. Transient transfer of RP4 to some host bacteria was observed. These results emphasize the prolonged persistence of P. putida and RP4 in natural soil microcosms, and highlight the potential risks of increased spread potential of plasmid and broader range of host bacteria in disseminating ARGs in soil