FeDXL: Provable Federated Learning for Deep X-Risk Optimization

In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of $\mathbb E_{z\sim S_1} f(\mathbb E_{z'\sim S_2} \ell(w; z, z'))$, where two sets of data $S_1, S_2$ are distributed over multiple machines, $\ell(\cdot)$ is a pairwise loss that only depends on the prediction outputs of the input data pairs $(z, z')$, and $f(\cdot)$ is possibly a non-linear non-convex function. This problem has important applications in machine learning, e.g., AUROC maximization with a pairwise loss, and partial AUROC maximization with a compositional loss. The challenges for designing an FL algorithm lie in the non-decomposability of the objective over multiple machines and the interdependency between different machines. To address the challenges, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples. Under this framework, we develop two provable FL algorithms (FeDXL) for handling linear and nonlinear $f$, respectively, based on federated averaging and merging. We develop a novel theoretical analysis to combat the latency of the passive parts and the interdependency between the local model parameters and the involved data for computing local gradient estimators. We establish both iteration and communication complexities and show that using the historical samples and models for computing the passive parts do not degrade the complexities. We conduct empirical studies of FeDXL for deep AUROC and partial AUROC maximization, and demonstrate their performance compared with several baselines

Fast Objective & Duality Gap Convergence for Nonconvex-Strongly-Concave Min-Max Problems

This paper focuses on stochastic methods for solving smooth non-convex strongly-concave min-max problems, which have received increasing attention due to their potential applications in deep learning (e.g., deep AUC maximization, distributionally robust optimization). However, most of the existing algorithms are slow in practice, and their analysis revolves around the convergence to a nearly stationary point. We consider leveraging the Polyak-\L ojasiewicz (PL) condition to design faster stochastic algorithms with stronger convergence guarantee. Although PL condition has been utilized for designing many stochastic minimization algorithms, their applications for non-convex min-max optimization remain rare. In this paper, we propose and analyze a generic framework of proximal epoch-based method with many well-known stochastic updates embeddable. Fast convergence is established in terms of both {\bf the primal objective gap and the duality gap}. Compared with existing studies, (i) our analysis is based on a novel Lyapunov function consisting of the primal objective gap and the duality gap of a regularized function, and (ii) the results are more comprehensive with improved rates that have better dependence on the condition number under different assumptions. We also conduct deep and non-deep learning experiments to verify the effectiveness of our methods

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

In this paper, we consider non-convex multi-block bilevel optimization (MBBO) problems, which involve $m\gg 1$ lower level problems and have important applications in machine learning. Designing a stochastic gradient and controlling its variance is more intricate due to the hierarchical sampling of blocks and data and the unique challenge of estimating hyper-gradient. We aim to achieve three nice properties for our algorithm: (a) matching the state-of-the-art complexity of standard BO problems with a single block; (b) achieving parallel speedup by sampling $I$ blocks and sampling $B$ samples for each sampled block per-iteration; (c) avoiding the computation of the inverse of a high-dimensional Hessian matrix estimator. However, it is non-trivial to achieve all of these by observing that existing works only achieve one or two of these properties. To address the involved challenges for achieving (a, b, c), we propose two stochastic algorithms by using advanced blockwise variance-reduction techniques for tracking the Hessian matrices (for low-dimensional problems) or the Hessian-vector products (for high-dimensional problems), and prove an iteration complexity of $O(\frac{m\epsilon^{-3}\mathbb{I}(I<m)}{I\sqrt{I}} + \frac{m\epsilon^{-3}}{I\sqrt{B}})$ for finding an $\epsilon$-stationary point under appropriate conditions. We also conduct experiments to verify the effectiveness of the proposed algorithms comparing with existing MBBO algorithms

Theoretical predictions for

The $\alpha$-decay half-lives of synthesized superheavy nuclei (SHN) from seaborgium to oganesson are calculated by employing the generalized liquid-drop model (GLDM), the Royer formula and the universal decay law (UDL) with experimental $\alpha$-decay energies $Q_{\alpha}$. For the GLDM, we consider the shell correction. The agreement between the experimental data and the calculations indicates that all the methods we used are successful to reproduce $\alpha$-decay half-lives of known SHN. The decay-modes of known nuclei on the 294Og decay-chain are also consistent with the experiments. For the unknown nuclei, the $\alpha$-decay half-lives have been predicted by inputting $Q_{\alpha}$ values extracted from the newest Weizsäcker-Skyrme-4 (WS4) model. In the GLDM with shell correction, we adopt the constant $\alpha$-preformation factor $P_{\alpha}$ as well as $P_{\alpha}$ extracted by Cluster Formation Model (CFM). To calculate CFM $P_{\alpha}$ values, we use FRDM binding energies and WS4 mass excess values. The relationship of $P_{\alpha}$ and $Q_{\alpha}$ shows that 294, 296, 314, 316, 320Og isotopes are relatively stable. The competition between $\alpha$-decay and spontaneous fission is discussed in detail for 283-339Og isotopes. The decay-chains of 290-300Og have also been presented. Since the $\alpha$-decay half-lives of 283-303Og isotopes are obviously lower than their spontaneous fission half-lives by more than 6 orders, these isotopes would mainly have $\alpha$-decay. The 306-334Og isotopes may undergo spontaneous fission. The nuclei 304, 305Og would have both $\alpha$-decay and spontaneous fission. By the shell-effect included GLDM with CFM $P_{\alpha}$, we predict 295Og undergoes $\alpha$-decay and $T_{\alpha}^{1/2} = 0.37$ ms. The 296Og is also $\alpha$-decay and has $T_{\alpha}^{1/2} = 0.40$ ms

Protein kinase Ds promote tumor angiogenesis through mast cell recruitment and expression of angiogenic factors in prostate cancer microenvironment

Abstract Background Mast cells are being increasingly recognized as critical components in the tumor microenvironment. Protein Kinase D (PKD) is essential for the progression of prostate cancer, but its role in prostate cancer microenvironment remains poorly understood. Methods The expression of PKD, mast cells and microvessel density were examined by IHC. The clinical significance was determined by statistical analyses. The biological function of PKD and the underlying mechanisms were investigated using in vitro and in vivo models. Results PKD2/3 contributed to MCs recruitment and tumor angiogenesis in the prostate cancer microenvironment. Clinical data showed that increased activation of PKD at Ser744/748 in prostate cancer was correlated with mast cell infiltration and microvascular density. PKD2/3 silencing of prostate cancer cells markedly decreased MCs migration and tube formation of HUVEC cells. Moreover, PKD2/3 depletion not only reduced SCF, CCL5 and CCL11 expression in prostate cancer cells but also inhibited angiogenic factors in MCs. Conversely, exogenous SCF, CCL5 and CCL11 reversed the effect on MCs migration inhibited by PKD2/3 silencing. Mechanistically, PKD2/3 interacted with Erk1/2 and activated Erk1/2 or NF-κB signaling pathway, leading to AP-1 or NF-κB binding to the promoter of scf, ccl5 and ccl11. Finally, PKD-specific inhibitor significantly reduced tumor volume and tumor growth in mice bearing RM-1 prostate cancer cells, which was attributed to attenuation of mast cell recruitment and tumor angiogenesis. Conclusions These results demonstrate a novel PKDs function that contributes to tumor angiogenesis and progression through mast cells recruitment in prostate cancer microenvironment

Large-scale generation of IL-12 secreting macrophages from human pluripotent stem cells for cancer therapy

Genetically engineered macrophages (GEMs) have emerged as an appealing strategy to treat cancers, but they are largely impeded by the cell availability and technical challenges in gene transfer. Here, we develop an efficient approach to generate large-scale macrophages from human induced pluripotent stem cells (hiPSCs). Starting with 1 T150 dish of 106 hiPSCs, more than 109 mature macrophages (iMacs) could be generated within 1 month. The generated iMacs exhibit typical macrophage properties such as phagocytosis and polarization. We then generate hiPSCs integrated with an IL-12 expression cassette in the AAVS1 locus to produce iMacs secreting IL-12, a strong proimmunity cytokine. hiPSC-derived iMacs_IL-12 prevent cytotoxic T cell exhaustion and activate T cells to kill different cancer cells. Furthermore, iMacs_IL-12 display strong antitumor effects in a T cell-dependent manner in subcutaneously or systemically xenografted mice of human lung cancer. Therefore, we provide an off-the-shelf strategy to produce large-scale GEMs for cancer therapy