Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data

The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks. Therefore, implicit bias in non-smooth neural networks trained by gradient descent remains an open question. In this paper, we aim to answer this question by studying the implicit bias of gradient descent for training two-layer fully connected (leaky) ReLU neural networks. We showed that when the training data are nearly-orthogonal, for leaky ReLU activation function, gradient descent will find a network with a stable rank that converges to $1$, whereas for ReLU activation function, gradient descent will find a neural network with a stable rank that is upper bounded by a constant. Additionally, we show that gradient descent will find a neural network such that all the training data points have the same normalized margin asymptotically. Experiments on both synthetic and real data backup our theoretical findings.Comment: 55 pages, 7 figures. In NeurIPS 202

Pathway Bridge Based Multiobjective Optimization Approach for Lurking Pathway Prediction

Ovarian carcinoma immunoreactive antigen-like protein 2 (OCIAD2) is a protein with unknown function. Frequently methylated or downregulated, OCIAD2 has been observed in kinds of tumors, and TGFβ signaling has been proved to induce the expression of OCIAD2. However, current pathway analysis tools do not cover the genes without reported interactions like OCIAD2 and also miss some significant genes with relatively lower expression. To investigate potential biological milieu of OCIAD2, especially in cancer microenvironment, a nova approach pbMOO was created to find the potential pathways from TGFβ to OCIAD2 by searching on the pathway bridge, which consisted of cancer enriched looping patterns from the complicated entire protein interactions network. The pbMOO approach was further applied to study the modulator of ligand TGFβ1, receptor TGFβR1, intermediate transfer proteins, transcription factor, and signature OCIAD2. Verified by literature and public database, the pathway TGFβ1- TGFβR1- SMAD2/3- SMAD4/AR-OCIAD2 was detected, which concealed the androgen receptor (AR) which was the possible transcription factor of OCIAD2 in TGFβ signal, and it well explained the mechanism of TGFβ induced OCIAD2 expression in cancer microenvironment, therefore providing an important clue for the future functional analysis of OCIAD2 in tumor pathogenesis

On the equivalences and differences of evolutionary algorithms

Evolutionary algorithms (EAs) are fast and robust computation methods for global optimization, and have been widely used in many real-world applications. We first conceptually discuss the equivalences of various popular EAs including genetic algorithm (GA), biogeography-based optimization (BBO), differential evolution (DE), evolution strategy (ES) and particle swarm optimization (PSO). We find that the basic versions of BBO, DE, ES and PSO are equal to the GA with global uniform recombination (GA/GUR) under certain conditions. Then we discuss their differences based on biological motivations and implementation details, and point out that their distinctions enhance the diversity of EA research and applications. To further study the characteristics of various EAs, we compare the basic versions and advanced versions of GA, BBO, DE, ES and PSO to explore their optimization ability on a set of real-world continuous optimization problems. Empirical results show that among the basic versions of the algorithms, BBO performs best on the benchmarks that we studied. Among the advanced versions of the algorithms, DE and ES perform best on the benchmarks that we studied. However, our main conclusion is that the conceptual equivalence of the algorithms is supported by the fact that algorithmic modifications result in very different performance levels

Benefits of Installing Restrictive Orifice Plates on the Suction of Reciprocating Pumps: 1D Pulsation and CFD Studies

Case StudiesIt is well understood that static pressure at the inlet of reciprocating pumps, quantified typically by Net Positive Suction Head Available (NPSHA), must be sufficient to avoid cavitation in the pump suction manifold and chamber. In an effort to conserve NPSHA, pump designers generally rely on rules of thumb that resist the addition of pressure drop elements such as restrictive orifice plates, choke tubes and line-size reductions to the inlet piping of all pumps, including reciprocating pumps. Another design consideration of reciprocating pumps is the generation of pressure pulsations due to pump piston and valve motion. Uncontrolled pulsations can result in cavitation and vibration-related fatigue failures. In many cases, pressure drop elements are required to control pressure pulsations. Can there be a balance between the pulsation control benefits of pressure drop elements and the need to meet NPSHA? This paper is of interest to designers and engineers working with reciprocating pump installations. It aims at challenging industry resistance to using pressure drop elements in the suction piping of reciprocating pumps by, first, outlining the virtues achieved in terms of pulsation and vibration control, and second, presenting results from numerical simulations (one-dimensional pulsation and detailed CFD modelling). Recent field data from a quintuplex pump installation were used to validate the 1-D pulsation model. The results show that well-designed orifice plates, and other pressure drop elements, are beneficial for reducing pulsations and cavitation risks; and can be used in the suction piping of reciprocating pumps