Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks

The convergence of GD and SGD when training mildly parameterized neural networks starting from random initialization is studied. For a broad range of models and loss functions, including the most commonly used square loss and cross entropy loss, we prove an ``early stage convergence'' result. We show that the loss is decreased by a significant amount in the early stage of the training, and this decrease is fast. Furthurmore, for exponential type loss functions, and under some assumptions on the training data, we show global convergence of GD. Instead of relying on extreme over-parameterization, our study is based on a microscopic analysis of the activation patterns for the neurons, which helps us derive more powerful lower bounds for the gradient. The results on activation patterns, which we call ``neuron partition'', help build intuitions for understanding the behavior of neural networks' training dynamics, and may be of independent interest

Understanding Multi-phase Optimization Dynamics and Rich Nonlinear Behaviors of ReLU Networks

The training process of ReLU neural networks often exhibits complicated nonlinear phenomena. The nonlinearity of models and non-convexity of loss pose significant challenges for theoretical analysis. Therefore, most previous theoretical works on the optimization dynamics of neural networks focus either on local analysis (like the end of training) or approximate linear models (like Neural Tangent Kernel). In this work, we conduct a complete theoretical characterization of the training process of a two-layer ReLU network trained by Gradient Flow on a linearly separable data. In this specific setting, our analysis captures the whole optimization process starting from random initialization to final convergence. Despite the relatively simple model and data that we studied, we reveal four different phases from the whole training process showing a general simplifying-to-complicating learning trend. Specific nonlinear behaviors can also be precisely identified and captured theoretically, such as initial condensation, saddle-to-plateau dynamics, plateau escape, changes of activation patterns, learning with increasing complexity, etc.Comment: 88 page

Judicial Determination of Confidentiality in Trade Secrets

Trade secrets as one of the core competitiveness of enterprises, related to the survival of enterprises. In the commercial competition, trade secrets is undoubtedly the right person’s wealth code, how the right person to take effective measures to protect trade secrets from being stolen is a matter of concern. In recent years, the practice of trade secret disputes have increased year by year, in the protection of trade secrets dispute cases, the right and the defendant on the identification of trade secrets, especially on the identification of confidentiality has always been the focus of controversy. Even though more and more enterprises on trade secret protection awareness is increasing. But from the many relevant judicial cases, can be seen: trade secrets in practice for trade secrets protection measures still have great loopholes, specifically manifested as: the right to protect the meaning of trade secrets is not clear, the protection of the object is not specific, the duty of confidentiality can not be confidential subject to know or limitations; confidentiality measures and the value of trade secrets are not adaptive; confidentiality measures can not be recognized and so on. This paper combines the recent judicial practice cases, to explore in practice to achieve the “corresponding confidentiality measures” of the core conditions, and in this way for the enterprise to establish trade secret protection system to provide reference

The KLF4–p62 axis prevents vascular endothelial cell injury via the mTOR/S6K pathway and autophagy in diabetic kidney disease

Introduction: Diabetic kidney disease (DKD) is a complication of systemic diabetic microangiopathy, which has a high risk of developing into end-stage renal disease and death. This study explored the mechanism underlying autophagy in DKD vascular endothelial cell injury. Material and methods: DKD and vascular endothelial cell injury models were established using Sprague Dawley rats and human umbilical vein endothelial cells (HUVECs). HUVECs overexpressing Kruppel-like factor 4 (KLF4) were constructed by transient transfection of plasmids. Biochemical determination of urinary protein and blood urea nitrogen (BUN), superoxide dismutase (SOD), and creatinine (Scr) levels was performed. Renal pathology was observed by periodic acid–Schiff (PAS) staining. Cell Counting Kit-8 (CCK8), terminal deoxynucleotidyl transferase dUTP nick end labelling (TUNEL), and immunocytochemistry (ICC) were used to analyse the growth and apoptosis of HUVECs. Microtubule-associated protein light chain 3 (LC3) expression was observed by immunofluorescence (IF). The reactive oxygen species (ROS) levels were measured using flow cytometry. Monocyte chemoattractant protein-1 (MCP-1), KLF4, and tumour necrosis factor alpha (TNF-α) levels were detected using enzyme-linked immunosorbent assay (ELISA). The expression of KLF4, p62 protein, and LC3 was analysed using reverse transcription quantitative real-time polymerase chain reaction (RT-qPCR). S6 kinase (S6K), p70 ribosomal S6 kinase (p-S6K), Beclin1, ATG5, LC3, p62, Caspase-3, mammalian target of rapamycine (mTOR), and phsophorylated mTOR (p-mTOR) expressions were detected by western blotting. Results: PAS-positive substances (polysaccharide and glycogen) and S6K protein levels increased, and LC3 protein expression decreased in DKD rats. The levels of urinary protein, BUN, and Scr increased, and KLF4 decreased in DKD rats. High glucose (HG) levels decreased the proliferation and increased the apoptosis rate of HUVECs. The expression of ROS, TNF-α, MCP-1, and p62 increased, while the expression of SOD, KLF4, Beclin1, ATG5, and LC3 decreased in HG-induced HUVECs. KLF4 overexpression significantly increased Beclin1, ATG5, and LC3 protein expression and decreased p62 protein expression compared to the oe-NC group in HG-induced HUVECs. KLF4 overexpression inhibits the expression of Caspase-3, p-mTOR, and p-S6K in HG-induced HUVECs. Conclusions: KLF4–p62 axis improved vascular endothelial cell injury by regulating inflammation and the mTOR/S6K pathway in DKD

Infrared permittivity of the biaxial van der Waals semiconductor α\alpha-MoO3_3 from near- and far-field correlative studies

The biaxial van der Waals semiconductor $\alpha$-phase molybdenum trioxide ($\alpha$-MoO$_3$) has recently received significant attention due to its ability to support highly anisotropic phonon polaritons (PhPs) -infrared (IR) light coupled to lattice vibrations in polar materials-, offering an unprecedented platform for controlling the flow of energy at the nanoscale. However, to fully exploit the extraordinary IR response of this material, an accurate dielectric function is required. Here, we report the accurate IR dielectric function of $\alpha$-MoO$_3$ by modelling far-field, polarized IR reflectance spectra acquired on a single thick flake of this material. Unique to our work, the far-field model is refined by contrasting the experimental dispersion and damping of PhPs, revealed by polariton interferometry using scattering-type scanning near-field optical microscopy (s-SNOM) on thin flakes of $\alpha$-MoO$_3$, with analytical and transfer-matrix calculations, as well as full-wave simulations. Through these correlative efforts, exceptional quantitative agreement is attained to both far- and near-field properties for multiple flakes, thus providing strong verification of the accuracy of our model, while offering a novel approach to extracting dielectric functions of nanomaterials, usually too small or inhomogeneous for establishing accurate models only from standard far-field methods. In addition, by employing density functional theory (DFT), we provide insights into the various vibrational states dictating our dielectric function model and the intriguing optical properties of $\alpha$-MoO$_3$

The Ultrasmall Biocompatible CuS@BSA Nanoparticle and Its Photothermal Effects

Nanomaterials with localized surface plasmon resonance (LSPR) have exquisite optical properties, which allow a wide range of applications. Non-stoichiometric copper sulfides with active LSPR have drawn great attention, because its LSPR peak falls in the NIR region that is suitable for deep bioimaging and photothermal therapy (PTT). Despite numerous biomedical applications, the biocompatibility and toxicity of copper sulfides have not been studied systematically. In this contribution, we synthesized the ultrasmall biocompatible copper sulfide nanoparticle encapsulated within bovine serum albumin (BSA), CuS@BSA. The physical features of CuS@BSA were characterized. The MTT and flow cytometry assays were performed. The in vitro PTT was also investigated. The results indicated that such CuS@BSA nanoparticle had an average TEM size of 8 nm, and an average DLS size of 15 nm. A lower concentration of CuS@BSA was not toxic to HeLa cells, but the critical apoptotic events occurred in HeLa cells after co-incubation with 45 μg/mL CuS@BSA for 48 h. The photothermal effect of the CuS@BSA in aqueous medium were concentration-dependent and time-dependent, which were also verified by flow cytometry and microscopy, while the CuS@BSA were co-cultured with HeLa cells and treated with laser. This work designed an ultrasmall potential biocompatible nanoparticle, CuS@BSA, for cancer photothermal therapy, and provided the toxic information to safely guide its biomedical applications