A high-resolution dynamical view on momentum methods for over-parameterized neural networks

Due to the simplicity and efficiency of the first-order gradient method, it has been widely used in training neural networks. Although the optimization problem of the neural network is non-convex, recent research has proved that the first-order method is capable of attaining a global minimum for training over-parameterized neural networks, where the number of parameters is significantly larger than that of training instances. Momentum methods, including heavy ball method (HB) and Nesterov's accelerated method (NAG), are the workhorse first-order gradient methods owning to their accelerated convergence. In practice, NAG often exhibits better performance than HB. However, current research fails to distinguish their convergence difference in training neural networks. Motivated by this, we provide convergence analysis of HB and NAG in training an over-parameterized two-layer neural network with ReLU activation, through the lens of high-resolution dynamical systems and neural tangent kernel (NTK) theory. Compared to existing works, our analysis not only establishes tighter upper bounds of the convergence rate for both HB and NAG, but also characterizes the effect of the gradient correction term, which leads to the acceleration of NAG over HB. Finally, we validate our theoretical result on three benchmark datasets.Comment: 19 page

A DNS Tunnel Sliding Window Differential Detection Method Based on Normal Distribution Reasonable Range Filtering

A covert attack method often used by APT organizations is the DNS tunnel, which is used to pass information by constructing C2 networks. And they often use the method of frequently changing domain names and server IP addresses to evade monitoring, which makes it extremely difficult to detect them. However, they carry DNS tunnel information traffic in normal DNS communication, which inevitably brings anomalies in some statistical characteristics of DNS traffic, so that it would provide security personnel with the opportunity to find them. Based on the above considerations, this paper studies the statistical discovery methodology of typical DNS tunnel high-frequency query behavior. Firstly, we analyze the distribution of the DNS domain name length and times and finds that the DNS domain name length and times follow the normal distribution law. Secondly, based on this distribution law, we propose a method for detecting and discovering high-frequency DNS query behaviors of non-single domain names based on the statistical rules of domain name length and frequency and we also give three theorems as theoretical support. Thirdly, we design a sliding window difference scheme based on the above method. Experimental results show that our method has a higher detection rate. At the same time, since our method does not need to construct a data set, it has better practicability in detecting unknown DNS tunnels. This also shows that our detection method based on mathematical models can effectively avoid the dilemma for machine learning methods that must have useful training data sets, and has strong practical significance

Contralateral upper tract urothelial carcinoma after nephroureterectomy: the predictive role of DNA methylation

Abstract Background Aberrant methylation of genes is one of the most common epigenetic modifications involved in the development of urothelial carcinoma. However, it is unknown the predictive role of methylation to contralateral new upper tract urothelial carcinoma (UTUC) after radical nephroureterectomy (RNU). We retrospectively investigated the predictive role of DNA methylation and other clinicopathological factors in the contralateral upper tract urothelial carcinoma (UTUC) recurrence after radical nephroureterectomy (RNU) in a large single-center cohort of patients. Methods In a retrospective design, methylation of 10 genes was analyzed on tumor specimens belonging to 664 consecutive patients treated by RNU for primary UTUC. Median follow-up was 48 mo (range: 3–144 mo). Gene methylation was accessed by methylation-sensitive polymerase chain reaction, and we calculated the methylation index (MI), a reflection of the extent of methylation. The log-rank test and Cox regression were used to identify the predictor of contralateral UTUC recurrence. Results Thirty (4.5%) patients developed a subsequent contralateral UTUC after a median follow-up time of 27.5 (range: 2–139) months. Promoter methylation for at least one gene promoter locus was present in 88.9% of UTUC. Fewer methylation and lower MI (P = 0.001) were seen in the tumors with contralateral UTUC recurrence than the tumors without contralateral recurrence. High MI (P = 0.007) was significantly correlated with poor cancer-specific survival. Multivariate analysis indicated that unmethylated RASSF1A (P = 0.039), lack of bladder recurrence prior to contralateral UTUC (P = 0.009), history of renal transplantation (P < 0.001), and preoperative renal insufficiency (P = 0.002) are independent risk factors for contralateral UTUC recurrence after RNU. Conclusions Our data suggest a potential role of DNA methylation in predicting contralateral UTUC recurrence after RNU. Such information could help identify patients at high risk of new contralateral UTUC recurrence after RNU who need close surveillance during follow up.http://deepblue.lib.umich.edu/bitstream/2027.42/110306/1/13046_2015_Article_120.pd

Vertical heterogeneity of hydrocarbon-degrading bacteria in a core sediment sample from the Central Indian Ridge

Hydrocarbons are ubiquitous in marine environments and might fuel hydrocarbon-metabolizing microbes in the ocean. Numerous studies have documented microbial hydrocarbon degradation in water columns and deep-sea surface sediment. However, the degradation potential and biogeochemical cycling of hydrocarbons in subsurface sediments remain largely unknown. In this study, we used two different hydrocarbons, n-hexadecane (HEX) and methylcyclohexane (MCH), to investigate the distribution and diversity of hydrocarbon-consuming bacteria in a core sediment sample from the Central Indian Ridge (CIR), which is adjacent to mid-ridge hydrothermal vents in the Indian Ocean. We observed different vertical profiles of HEX- and MCH-degrading bacteria in the core sediments. Specifically, HEX-degrading bacteria were universally distributed, while MCH-degrading bacteria were found only in the intermediate layers of the core sediments. Changing factors including dissolved oxygen might affect the natural distribution of different hydrocarbon consumers. We found that a novel species of the genus C1-B045 might play a pivotal role in metabolizing MCH in the CIR deep biosphere. Through amino acid identity comparison with published sequences, we determined that C1-B045 harbors two novel classes of cyclohexanone monooxygenases involved in MCH metabolism. This study sheds light on the structure and function of hydrocarbon-consuming microbes in deep biospheres

Novel TLR7/8 agonists promote activation of HIV-1 latent reservoirs and human T and NK cells

Antiretroviral therapy can successfully suppress HIV-1 replication to undetectable levels but fails to eliminate latent and persistent HIV-1 reservoirs. Recent studies have focused on the immunomodulatory agents such as Toll-like receptor 7 and 8 (TLR7 and TLR8) capable of activating, thereby rendering the reservoir susceptible to antiretroviral inhibition and immune recognition and elimination. In this context, this study focused on generating a diverse repertoire of TLR7/8 agonists to identify more potent candidates for activating latent HIV-1 and immune cells’ response. Through combinational strategies of computer-aided design and biological characterization, 159 pyrido [3,2-d] pyrimidine and pyridine-2-amine-based derivatives were synthesized. Of which, two TLR7/8 dual and one TLR8-specific agonists with exceptionally high potency in activating HIV-1 latent reservoirs in cell lines and PBMCs of patients with persistent and durable virologic controls were identified. Particularly, these agonists appeared to enhance NK and T cells activity, which were correlated with the degree of surface activation markers. The outcome of this study highlights the remarkable potential of TLR7/8 agonists in simultaneously activating HIV-1 from the latently infected cells and augmenting immune effector cells