Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation

TensorFlow has been the most widely adopted Machine/Deep Learning framework. However, little exists in the literature that provides a thorough understanding of the capabilities which TensorFlow offers for the distributed training of large ML/DL models that need computation and communication at scale. Most commonly used distributed training approaches for TF can be categorized as follows: 1) Google Remote Procedure Call (gRPC), 2) gRPC+X: X=(InfiniBand Verbs, Message Passing Interface, and GPUDirect RDMA), and 3) No-gRPC: Baidu Allreduce with MPI, Horovod with MPI, and Horovod with NVIDIA NCCL. In this paper, we provide an in-depth performance characterization and analysis of these distributed training approaches on various GPU clusters including the Piz Daint system (6 on Top500). We perform experiments to gain novel insights along the following vectors: 1) Application-level scalability of DNN training, 2) Effect of Batch Size on scaling efficiency, 3) Impact of the MPI library used for no-gRPC approaches, and 4) Type and size of DNN architectures. Based on these experiments, we present two key insights: 1) Overall, No-gRPC designs achieve better performance compared to gRPC-based approaches for most configurations, and 2) The performance of No-gRPC is heavily influenced by the gradient aggregation using Allreduce. Finally, we propose a truly CUDA-Aware MPI Allreduce design that exploits CUDA kernels and pointer caching to perform large reductions efficiently. Our proposed designs offer 5-17X better performance than NCCL2 for small and medium messages, and reduces latency by 29% for large messages. The proposed optimizations help Horovod-MPI to achieve approximately 90% scaling efficiency for ResNet-50 training on 64 GPUs. Further, Horovod-MPI achieves 1.8X and 3.2X higher throughput than the native gRPC method for ResNet-50 and MobileNet, respectively, on the Piz Daint cluster.Comment: 10 pages, 9 figures, submitted to IEEE IPDPS 2019 for peer-revie

Comparison of Visual Datasets for Machine Learning

One of the greatest technological improvements in recent years is the rapid progress using machine learning for processing visual data. Among all factors that contribute to this development, datasets with labels play crucial roles. Several datasets are widely reused for investigating and analyzing different solutions in machine learning. Many systems, such as autonomous vehicles, rely on components using machine learning for recognizing objects. This paper compares different visual datasets and frameworks for machine learning. The comparison is both qualitative and quantitative and investigates object detection labels with respect to size, location, and contextual information. This paper also presents a new approach creating datasets using real-time, geo-tagged visual data, greatly improving the contextual information of the data. The data could be automatically labeled by cross-referencing information from other sources (such as weather)

Origin, Transport, and Vertical Distribution of Atmospheric Polluntants over the Northern Sourth China Sea During the 7-SEAS-Dongsha Experiment

During the spring of 2010, comprehensive in situ measurements were made for the first time on a small atoll (Dongsha Island) in the northern South China Sea (SCS), a key region of the 7-SEAS (the Seven South East Asian Studies) program. This paper focuses on characterizing the source origins, transport processes, and vertical distributions of the Asian continental outflows over the region, using measurements including mass concentration, optical properties, hygroscopicity, and vertical distribution of the aerosol particles, as well as the trace gas composition. Cluster analysis of backward trajectories classified 52% of the air masses arriving at ground level of Dongsha Island as having a continental origin, mainly from northern China to the northern SCS, passing the coastal area and being confined in the marine boundary layer (0-0.5 km). Compared to aerosols of oceanic origin, the fine mode continental aerosols have a higher concentration, extinction coefficient, and single-scattering albedo at 550 nm (i.e., 19 vs. 14 microg per cubic meter in PM(sub 2.5); 77 vs. 59 M per meter in beta(sub e); and 0.94 vs. 0.90 in omega, respectively). These aerosols have a higher hygroscopicity (f at 85% RH = 2.1) than those in the upwind inland regions, suggesting that the aerosols transported to the northern SCS were modified by the marine environment. In addition to the near-surface aerosol transport, a significant upper-layer (3-4 km) transport of biomass-burning aerosols was observed. Our results suggest that emissions from both China and Southeast Asia could have a significant impact on the aerosol loading and other aerosol properties over the SCS. Furthermore, the complex vertical distribution of aerosols-coinciding-with-clouds has implications for remote-sensing observations and aerosol-cloud-radiation interactions

Dengue-1 Envelope Protein Domain III along with PELC and CpG Oligodeoxynucleotides Synergistically Enhances Immune Responses

The major weaknesses of subunit vaccines are their low immunogenicity and poor efficacy. Adjuvants can help to overcome some of these inherent defects with subunit vaccines. Here, we evaluated the efficacy of the newly developed water-in-oil-in-water multiphase emulsion system, termed PELC, in potentiating the protective capacity of dengue-1 envelope protein domain III. Unlike aluminum phosphate, dengue-1 envelope protein domain III formulated with PELC plus CpG oligodeoxynucleotides induced neutralizing antibodies against dengue-1 virus and increased the splenocyte secretion of IFN-γ after in vitro re-stimulation. The induced antibodies contained both the IgG1 and IgG2a subclasses. A rapid anamnestic neutralizing antibody response against a live dengue virus challenge was elicited at week 26 after the first immunization. These results demonstrate that PELC plus CpG oligodeoxynucleotides broaden the dengue-1 envelope protein domain III-specific immune responses. PELC plus CpG oligodeoxynucleotides is a promising adjuvant for recombinant protein based vaccination against dengue virus

Widening of Socioeconomic Inequalities in U.S. Death Rates, 1993–2001

Background: Socioeconomic inequalities in death rates from all causes combined widened from 1960 until 1990 in the U.S., largely because cardiovascular death rates decreased more slowly in lower than in higher socioeconomic groups. However, no studies have examined trends in inequalities using recent US national data. Methodology/Principal Findings: We calculated annual age-standardized death rates from 1993–2001 for 25–64 year old non-Hispanic whites and blacks by level of education for all causes and for the seven most common causes of death using death certificate information from 43 states and Washington, D.C. Regression analysis was used to estimate annual percent change. The inequalities in all cause death rates between Americans with less than high school education and college graduates increased rapidly from 1993 to 2001 due to both significant decreases in mortality from all causes, heart disease, cancer, stroke, and other conditions in the most educated and lack of change or increases among the least educated. For white women, the all cause death rate increased significantly by 3.2 percent per year in the least educated and by 0.7 percent per year in high school graduates. The rate ratio (RR) comparing the least versus most educated increased from 2.9 (95 % CI, 2.8–3.1) in 1993 to 4.4 (4.1–4.6) in 2001 among white men, from 2.1 (1.8–2.5) to 3.4 (2.9–3–9) in black men, and from 2.6 (2.4–2.7) to 3.8 (3.6–4.0) in white women. Conclusion: Socioeconomic inequalities in mortality are increasing rapidly due to continued progress by educated whit

The Greenland Telescope: Antenna Retrofit Status and Future Plans

Since the ALMA North America Prototype Antenna was awarded to the Smithsonian Astrophysical Observatory (SAO), SAO and the Academia Sinica Institute of Astronomy & Astrophysics (ASIAA) are working jointly to relocate the antenna to Greenland. This paper shows the status of the antenna retrofit and the work carried out after the recommissioning and subsequent disassembly of the antenna at the VLA has taken place. The next coming months will see the start of the antenna reassembly at Thule Air Base. These activities are expected to last until the fall of 2017 when commissioning should take place. In parallel, design, fabrication and testing of the last components are taking place in Taiwan

Entrapment neuropathy results in different microRNA expression patterns from denervation injury in rats

<p>Abstract</p> <p>Background</p> <p>To compare the microRNA (miRNA) expression profiles in neurons and innervated muscles after sciatic nerve entrapment using a non-constrictive silastic tube, subsequent surgical decompression, and denervation injury.</p> <p>Methods</p> <p>The experimental L4-L6 spinal segments, dorsal root ganglia (DRGs), and soleus muscles from each experimental group (sham control, denervation, entrapment, and decompression) were analyzed using an Agilent rat miRNA array to detect dysregulated miRNAs. In addition, muscle-specific miRNAs (miR-1, -133a, and -206) and selectively upregulated miRNAs were subsequently quantified using real-time reverse transcription-polymerase chain reaction (real-time RT-PCR).</p> <p>Results</p> <p>In the soleus muscles, 37 of the 47 miRNAs (13.4% of the 350 unique miRNAs tested) that were significantly downregulated after 6 months of entrapment neuropathy were also among the 40 miRNAs (11.4% of the 350 unique miRNAs tested) that were downregulated after 3 months of decompression. No miRNA was upregulated in both groups. In contrast, only 3 miRNAs were upregulated and 3 miRNAs were downregulated in the denervated muscle after 6 months. In the DRGs, 6 miRNAs in the entrapment group (miR-9, miR-320, miR-324-3p, miR-672, miR-466b, and miR-144) and 3 miRNAs in the decompression group (miR-9, miR-320, and miR-324-3p) were significantly downregulated. No miRNA was upregulated in both groups. We detected 1 downregulated miRNA (miR-144) and 1 upregulated miRNA (miR-21) after sciatic nerve denervation. We were able to separate the muscle or DRG samples into denervation or entrapment neuropathy by performing unsupervised hierarchal clustering analysis. Regarding the muscle-specific miRNAs, real-time RT-PCR analysis revealed an ~50% decrease in miR-1 and miR-133a expression levels at 3 and 6 months after entrapment, whereas miR-1 and miR-133a levels were unchanged and were decreased after decompression at 1 and 3 months. In contrast, there were no statistical differences in the expression of miR-206 during nerve entrapment and after decompression. The expression of muscle-specific miRNAs in entrapment neuropathy is different from our previous observations in sciatic nerve denervation injury.</p> <p>Conclusions</p> <p>This study revealed the different involvement of miRNAs in neurons and innervated muscles after entrapment neuropathy and denervation injury, and implied that epigenetic regulation is different in these two conditions.</p

A Randomised Placebo-Controlled Trial of a Traditional Chinese Herbal Formula in the Treatment of Primary Dysmenorrhoea

BACKGROUND: Most traditional Chinese herbal formulas consist of at least four herbs. Four-Agents-Decoction (Si Wu Tang) is a documented eight hundred year old formula containing four herbs and has been widely used to relieve menstrual discomfort in Taiwan. However, no specific effect had been systematically evaluated. We applied Western methodology to assess its effectiveness and safety for primary dysmenorrhoea and to evaluate the compliance and feasibility for a future trial. METHODOLOGY/PRINCIPAL FINDINGS: A randomised, double-blind, placebo-controlled, pilot clinical trial was conducted in an ad hoc clinic setting at a teaching hospital in Taipei, Taiwan. Seventy-eight primary dysmenorrheic young women were enrolled after 326 women with self-reported menstrual discomfort in the Taipei metropolitan area of Taiwan were screened by a questionnaire and subsequently diagnosed by two gynaecologists concurrently with pelvic ultrasonography. A dosage of 15 odorless capsules daily for five days starting from the onset of bleeding or pain was administered. Participants were followed with two to four cycles for an initial washout interval, one to two baseline cycles, three to four treatment cycles, and three follow-up cycles. Study outcome was pain intensity measured by using unmarked horizontal visual analog pain scale in an online daily diary submitted directly by the participants for 5 days starting from the onset of bleeding or pain of each menstrual cycle. Overall-pain was the average pain intensity among days in pain and peak-pain was the maximal single-day pain intensity. At the end of treatment, both the overall-pain and peak-pain decreased in the Four-Agents-Decoction (Si Wu Tang) group and increased in the placebo group; however, the differences between the two groups were not statistically significant. The trends persisted to follow-up phase. Statistically significant differences in both peak-pain and overall-pain appeared in the first follow-up cycle, at which the reduced peak-pain in the Four-Agents-Decoction (Si Wu Tang) group did not differ significantly by treatment length. However, the reduced peak-pain did differ profoundly among women treated for four menstrual cycles (2.69 (2.06) cm, mean (standard deviation), for the 20 women with Four-Agents-Decoction and 4.68 (3.16) for the 22 women with placebo, p = .020.) There was no difference in adverse symptoms between the Four-Agents-Decoction (Si Wu Tang) and placebo groups. CONCLUSION/SIGNIFICANCE: Four-Agents-Decoction (Si Wu Tang) therapy in this pilot post-market clinical trial, while meeting the standards of conventional medicine, showed no statistically significant difference in reducing menstrual pain intensity of primary dysmenorrhoea at the end of treatment. Its use, with our dosage regimen and treatment length, was not associated with adverse reactions. The finding of statistically significant pain-reducing effect in the first follow-up cycle was unexpected and warrants further study. A larger similar trial among primary dysmenorrheic young women with longer treatment phase and multiple batched study products can determine the definitive efficacy of this historically documented formula. TRIAL REGISTRATION: Controlled-Trials.com ISRCTN23374750