Adaptive Optimizers with Sparse Group Lasso for Neural Networks in CTR Prediction

We develop a novel framework that adds the regularizers of the sparse group lasso to a family of adaptive optimizers in deep learning, such as Momentum, Adagrad, Adam, AMSGrad, AdaHessian, and create a new class of optimizers, which are named Group Momentum, Group Adagrad, Group Adam, Group AMSGrad and Group AdaHessian, etc., accordingly. We establish theoretically proven convergence guarantees in the stochastic convex settings, based on primal-dual methods. We evaluate the regularized effect of our new optimizers on three large-scale real-world ad click datasets with state-of-the-art deep learning models. The experimental results reveal that compared with the original optimizers with the post-processing procedure which uses the magnitude pruning method, the performance of the models can be significantly improved on the same sparsity level. Furthermore, in comparison to the cases without magnitude pruning, our methods can achieve extremely high sparsity with significantly better or highly competitive performance. The code is available at https://github.com/intelligent-machine-learning/dlrover/blob/master/tfplus.Comment: 24 pages. Published as a conference paper at ECML PKDD 2021. This version includes Appendix which was not included in the published version because of page limi

Clinical comparison of percutaneous transforaminal endoscopic discectomy and unilateral biportal endoscopic discectomy for single-level lumbar disc herniation

PurposeTo compare the clinical outcomes of percutaneous transforaminal endoscopic discectomy (PTED) and unilateral biportal endoscopic discectomy (UBE) for the treatment of single-level lumbar disc herniation (LDH).Materials and methodsFrom January 2020 to November 2021, 62 patients with single-level LDH were retrospectively reviewed. All patients underwent spinal surgeries at the Affiliated Hospital of Chengde Medical University and Beijing Tongren Hospital, Capital Medical University. Among them, 30 patients were treated with UBE, and 32 were treated with PTED. The patients were followed up for at least one year. Patient demographics and perioperative outcomes were reviewed before and after surgery. The Oswestry Disability Index (ODI), visual analog scale (VAS) for back pain and leg pain, and modified MacNab criteria were used to evaluate the clinical outcomes. x-ray examinations were performed one year after surgery to assess the stability of the lumbar spine.ResultsThe mean ages in the UBE and PTED groups were 46.7 years and 48.0 years, respectively. Compared to the UBE group, the PTED group had better VAS scores for back pain at 1 and 7 days after surgery (3.06 ± 0.80 vs. 4.03 ± 0.81, P < 0.05; 2.81 ± 0.60 vs. 3.70 ± 0.79, P < 0.05). The UBE and PTED groups demonstrated significant improvements in the VAS score for leg pain and ODI score, and no significant differences were found between the groups at any time after the first month (P > 0.05). Although the good-to-excellent rate of the modified MacNab criteria in the UBE group was similar to that in the PTED group (86.7% vs. 87.5%, P > 0.05), PTED was advantageous in terms of the operation time, estimated blood loss, incision length, and length of postoperative hospital stay.ConclusionsBoth UBE and PTED have favorable outcomes in patients with single-level LDH. However, PTED is superior to UBE in terms of short-term postoperative back pain relief and perioperative quality of life

Association between serum 25‐hydroxyvitamin D level and inflammatory markers in hemodialysis‐treated patients

Abstract Objective To investigate the relationship between serum 25‐hydroxyvitamin D (25(OH)D) level with novel inflammatory markers in hemodialysis‐treated patients. Methods A total of 167 maintenance hemodialysis‐treated patients were enrolled in this cross‐sectional study. The patients were divided into vitamin D deficiency (a serum 25(OH)D level <20 ng/mL) and nondeficiency (a serum 25(OH)D level ≥20 ng/mL) groups. The neutrophil to lymphocyte ratio (NLR), platelet to lymphocyte ratio (PLR), and monocyte to lymphocyte ratio (MLR) were calculated by the complete blood cell count. The relationship between 25(OH)D level with other parameters was assessed by bivariate correlation analysis and linear regression analysis. Results There were significant differences between the two groups in terms of age, diabetes, levels of albumin, creatinine, high‐density lipoprotein cholesterol (HDL‐C) and low‐density lipoprotein cholesterol (LDL‐C) as well as NLR and MLR (p = .004, p = .031, p < .001, p = .043, p = .008, p = .006, p = .002, and p < .001, respectively). There exist negative correlations between serum 25(OH)D level with age, diabetes, alkaline phosphatase level, NLR, PLR, and MLR (p = .002, p = .002, p = .037, p = .001, p = .041, and p < .001, respectively) and positive correlations between serum 25(OH)D level with albumin level, creatinine level, phosphorus level, HDL‐C, and LDL‐C (p < .001, p < .001, p = .013, p = .02, p = .002, respectively). Multiple analysis results showed that sex, diabetes, albumin level and NLR were independently associated with serum 25(OH)D level (p = .021, p = .015, p = .033, and p = .041, respectively). High values of NLR and MLR were associated with patients with serum 25(OH)D deficiency. There were negative interplays between serum 25(OH) D level with NLR, PLR, and MLR and also an independent association between serum 25(OH) D level with NLR. Conclusion Collectively, serum 25(OH)D level has a negative correlation with inflammatory markers

Synthesis and Assembly of Laccase-Polymer Giant Amphiphiles by Self-Catalyzed CuAAC Click Chemistry

Covalent coupling of hydrophobic polymers to the exterior of hydrophilic proteins would mediate unique macroscopic assembly of bioconjugates to generate amphiphilic superstructures as novel nanoreactors or biocompatible drug delivery systems. The main objective of this study was to develop a novel strategy for the synthesis of protein-polymer giant amphiphiles by the combination of copper-mediated living radical polymerization and azide–alkyne cycloaddition reaction (CuAAC). Azide-functionalized succinimidyl ester was first synthesized for the facile introduction of azide groups to proteins such as albumin from bovine serum (BSA) and laccase from <i>Trametes versicolor</i>. Alkyne-terminal polymers with varied hydrophobicity were synthesized by using commercial copper wire as the activators from a trimethylsilyl protected alkyne-functionalized initiator in DMSO under ambient temperature. The conjugation of alkyne-functionalized polymers to the azide-functionalized laccase could be conducted even without additional copper catalyst, which indicated a successful self-catalyzed CuAAC reaction. The synthesized amphiphiles were found to aggregate into spherical nanoparticles in water and showed strong relevance to the hydrophobicity of coupled polymers. The giant amphiphiles showed decreased enzyme activity yet better stability during storage after chemical modification and self-assembly. These findings will deepen our understanding on protein folding, macroscopic self-assembly, and support potential applications in bionanoreactor, enzyme immobilization, and water purification

Power Synchronization Compensation Strategy Based on Second-Order Compensation Links for Voltage-Controlled Inverters in Microgrids

A second-order compensation link is adopted to control voltage-controlled inverters (VCIs) in microgrid systems to enhance the performance of the power synchronization process of the inverter. The second-order compensation link is classified as both a real pole compensator (RPC) and a complex pole compensator (CPC) according to the pole position. Given a model for the VCI power output, the design process for the second-order compensation link, which is equipped with an RPC and a CPC, is detailed. Moreover, the frequency-domain compensation effects of the RPC and CPC are analyzed using the root locus and Bode diagrams of the system before and after compensation. Finally, the compensation effects of the two types of second-order compensators are compared with the commonly used high-pass filter using MATLAB/Simulink, which verifies the RPC and CPC strategies. Simulation results show that the two types of compensators designed in this study can effectively increase the system cutting frequency and improve the phase margin in the frequency domain while accelerating the power synchronization process, simultaneously making it smoother and reducing overshoot in the time domain. The RPC has better gain robustness, whereas the CPC has better time constant robustness. By implementing an RPC or a CPC, the dynamic time of the power synchronization compensation strategy is reduced within 0.5 s, and the overshoot is reduced within 10&#x0025; in the experiments with two inverters

Secondary-Frequency and Voltage-Regulation Control of Multi-Parallel Inverter Microgrid System

As an important form of distributed renewable energy utilization and consumption, the multi-parallel inverter microgrid system works in both an isolated and grid-connected operation mode. Secondary-frequency and voltage-regulation control are very important in solving problems that appears in these systems, such as the distributed secondary-frequency regulation real-time scheme, voltage and reactive power balancing, and the secondary-frequency regulation control under the disturbances and unbalanced conditions of a microgrid system. This paper introduces key technologies related to these issues, such as the consensus algorithm and event-triggered technique, the dynamic and adaptive virtual impedance technique, and the robust and self-anti-disturbance control technique. Research and design methods such as small-signal state-space analysis, the Lyapunov function design method, the impedance analysis method, &mu;-synthesis design, and the LMI matrix design method are adopted to solve the issues in secondary-frequency regulation and voltage regulation. As the number of inverters increases, the structure of the microgrid becomes more and more complex. Suggestions and prospects for future research are provided to realize control with low-communication technology and a distributed scheme. Finally, for the case study, the droop-control model and primary frequency/voltage deviation of a multi-parallel inverter microgrid system is analyzed, and a state-space model of a multi-parallel inverter microgrid system with a droop-control loop is established. Then, the quantitative relationship between the primary frequency/voltage deviation and the active and reactive power output in the system is discussed. The methods and problems of centralized and decentralized secondary-frequency regulation methods, secondary-frequency regulation methods based on a consensus algorithm and an event-triggered mechanism, reactive power and voltage equalization, power distribution, and small-signal stability of the multiple parallel inverter microgrid system regarding the virtual impedance loop are analyzed

Frequency Diversity Array Radar and Jammer Intelligent Frequency Domain Power Countermeasures Based on Multi-Agent Reinforcement Learning

With the development of electronic warfare technology, the intelligent jammer dramatically reduces the performance of traditional radar anti-jamming methods. A key issue is how to actively adapt radar to complex electromagnetic environments and design anti-jamming strategies to deal with intelligent jammers. The space of the electromagnetic environment is dynamically changing, and the transmitting power of the jammer and frequency diversity array (FDA) radar in each frequency band is continuously adjustable. Both can learn the optimal strategy by interacting with the electromagnetic environment. Considering that the competition between the FDA radar and the jammer is a confrontation process of two agents, we find the optimal power allocation strategy for both sides by using the multi-agent deep deterministic policy gradient (MADDPG) algorithm based on multi-agent reinforcement learning (MARL). Finally, the simulation results show that the power allocation strategy of the FDA radar and the jammer can converge and effectively improve the performance of the FDA radar and the jammer in the intelligent countermeasure environment

Secondary-Frequency and Voltage-Regulation Control of Multi-Parallel Inverter Microgrid System

As an important form of distributed renewable energy utilization and consumption, the multi-parallel inverter microgrid system works in both an isolated and grid-connected operation mode. Secondary-frequency and voltage-regulation control are very important in solving problems that appears in these systems, such as the distributed secondary-frequency regulation real-time scheme, voltage and reactive power balancing, and the secondary-frequency regulation control under the disturbances and unbalanced conditions of a microgrid system. This paper introduces key technologies related to these issues, such as the consensus algorithm and event-triggered technique, the dynamic and adaptive virtual impedance technique, and the robust and self-anti-disturbance control technique. Research and design methods such as small-signal state-space analysis, the Lyapunov function design method, the impedance analysis method, μ-synthesis design, and the LMI matrix design method are adopted to solve the issues in secondary-frequency regulation and voltage regulation. As the number of inverters increases, the structure of the microgrid becomes more and more complex. Suggestions and prospects for future research are provided to realize control with low-communication technology and a distributed scheme. Finally, for the case study, the droop-control model and primary frequency/voltage deviation of a multi-parallel inverter microgrid system is analyzed, and a state-space model of a multi-parallel inverter microgrid system with a droop-control loop is established. Then, the quantitative relationship between the primary frequency/voltage deviation and the active and reactive power output in the system is discussed. The methods and problems of centralized and decentralized secondary-frequency regulation methods, secondary-frequency regulation methods based on a consensus algorithm and an event-triggered mechanism, reactive power and voltage equalization, power distribution, and small-signal stability of the multiple parallel inverter microgrid system regarding the virtual impedance loop are analyzed

Fabrication and Evaluation of Alginate/Bacterial Cellulose Nanocrystals–Chitosan–Gelatin Composite Scaffolds

It is common knowledge that pure alginate hydrogel is more likely to have weak mechanical strength, a lack of cell recognition sites, extensive swelling and uncontrolled degradation, and thus be unable to satisfy the demands of the ideal scaffold. To address these problems, we attempted to fabricate alginate/bacterial cellulose nanocrystals-chitosan-gelatin (Alg/BCNs-CS-GT) composite scaffolds using the combined method involving the incorporation of BCNs in the alginate matrix, internal gelation through the hydroxyapatite-d-glucono-δ-lactone (HAP-GDL) complex, and layer-by-layer (LBL) electrostatic assembly of polyelectrolytes. Meanwhile, the effect of various contents of BCNs on the scaffold morphology, porosity, mechanical properties, and swelling and degradation behavior was investigated. The experimental results showed that the fabricated Alg/BCNs-CS-GT composite scaffolds exhibited regular 3D morphologies and well-developed pore structures. With the increase in BCNs content, the pore size of Alg/BCNs-CS-GT composite scaffolds was gradually reduced from 200 μm to 70 μm. Furthermore, BCNs were fully embedded in the alginate matrix through the intermolecular hydrogen bond with alginate. Moreover, the addition of BCNs could effectively control the swelling and biodegradation of the Alg/BCNs-CS-GT composite scaffolds. Furthermore, the in vitro cytotoxicity studies indicated that the porous fiber network of BCNs could fully mimic the extracellular matrix structure, which promoted the adhesion and spreading of MG63 cells and MC3T3-E1 cells on the Alg/BCNs-CS-GT composite scaffolds. In addition, these cells could grow in the 3D-porous structure of composite scaffolds, which exhibited good proliferative viability. Based on the effect of BCNs on the cytocompatibility of composite scaffolds, the optimum BCNs content for the Alg/BCNs-CS-GT composite scaffolds was 0.2% (w/v). On the basis of good merits, such as regular 3D morphology, well-developed pore structure, controlled swelling and biodegradation behavior, and good cytocompatibility, the Alg/BCNs-CS-GT composite scaffolds may exhibit great potential as the ideal scaffold in the bone tissue engineering field

Learning to generate maps from trajectories

Accurate and updated road network data is vital in many urban applications, such as car-sharing, and logistics. The traditional approach to identifying the road network, ie, field survey, requires a significant amount of time and effort. With the wide usage of GPS embedded devices, a huge amount of trajectory data has been generated by different types of mobile objects, which provides a new opportunity to extract the underlying road network. However, the existing trajectory-based map recovery approaches require many empirical parameters and do not utilize the prior knowledge in existing maps, which over-simplifies or over-complicates the reconstructed road network. To this end, we propose a deep learning-based map generation framework, ie, DeepMG, which learns the structure of the existing road network to overcome the noisy GPS positions. More specifically, DeepMG extracts features from trajectories in both spatial view and transition view and uses a convolutional deep neural network T2RNet to infer road centerlines. After that, a trajectory-based post-processing algorithm is proposed to refine the topological connectivity of the recovered map. Extensive experiments on two real-world trajectory datasets confirm that DeepMG significantly outperforms the state-of-the-art methods.Ministry of Education (MOE)Nanyang Technological UniversityAccepted versionThe research of Cheng Long was supported by the NTU Start-Up Grant and Singapore MOE Tier 1 Grant RG20/19 (S)