DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

Differential private optimization for nonconvex smooth objective is considered. In the previous work, the best known utility bound is $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ in terms of the squared full gradient norm, which is achieved by Differential Private Gradient Descent (DP-GD) as an instance, where $n$ is the sample size, $d$ is the problem dimensionality and $\varepsilon_\mathrm{DP}$ is the differential privacy parameter. To improve the best known utility bound, we propose a new differential private optimization framework called \emph{DIFF2 (DIFFerential private optimization via gradient DIFFerences)} that constructs a differential private global gradient estimator with possibly quite small variance based on communicated \emph{gradient differences} rather than gradients themselves. It is shown that DIFF2 with a gradient descent subroutine achieves the utility of $\widetilde O(d^{2/3}/(n\varepsilon_\mathrm{DP})^{4/3})$, which can be significantly better than the previous one in terms of the dependence on the sample size $n$. To the best of our knowledge, this is the first fundamental result to improve the standard utility $\widetilde O(\sqrt{d}/(n\varepsilon_\mathrm{DP}))$ for nonconvex objectives. Additionally, a more computational and communication efficient subroutine is combined with DIFF2 and its theoretical analysis is also given. Numerical experiments are conducted to validate the superiority of DIFF2 framework.Comment: 26 page

Versatile Single-Loop Method for Gradient Estimator: First and Second Order Optimality, and its Application to Federated Learning

While variance reduction methods have shown great success in solving large scale optimization problems, many of them suffer from accumulated errors and, therefore, should periodically require the full gradient computation. In this paper, we present a single-loop algorithm named SLEDGE (Single-Loop mEthoD for Gradient Estimator) for finite-sum nonconvex optimization, which does not require periodic refresh of the gradient estimator but achieves nearly optimal gradient complexity. Unlike existing methods, SLEDGE has the advantage of versatility; (i) second-order optimality, (ii) exponential convergence in the PL region, and (iii) smaller complexity under less heterogeneity of data. We build an efficient federated learning algorithm by exploiting these favorable properties. We show the first and second-order optimality of the output and also provide analysis under PL conditions. When the local budget is sufficiently large and clients are less (Hessian-)~heterogeneous, the algorithm requires fewer communication rounds then existing methods such as FedAvg, SCAFFOLD, and Mime. The superiority of our method is verified in numerical experiments

TAT-dextran-mediated mitochondrial transfer enhances recovery from models of reperfusion injury in cultured cardiomyocytes

Acute myocardial infarction is a leading cause of death among single organ diseases. Despite successful reperfusion therapy, ischaemia reperfusion injury (IRI) can induce oxidative stress (OS), cardiomyocyte apoptosis, autophagy and release of inflammatory cytokines, resulting in increased infarct size. In IRI, mitochondrial dysfunction is a key factor, which involves the production of reactive oxygen species, activation of inflammatory signalling cascades or innate immune responses, and apoptosis. Therefore, intercellular mitochondrial transfer could be considered as a promising treatment strategy for ischaemic heart disease. However, low transfer efficiency is a challenge in clinical settings. We previously reported uptake of isolated exogenous mitochondria into cultured cells through co-incubation, mediated by macropinocytosis. Here, we report the use of transactivator of transcription dextran complexes (TAT-dextran) to enhance cellular uptake of exogenous mitochondria and improve the protective effect of mitochondrial replenishment in neonatal rat cardiomyocytes (NRCMs) against OS. TAT-dextran-modified mitochondria (TAT-Mito) showed a significantly higher level of cellular uptake. Mitochondrial transfer into NRCMs resulted in anti-apoptotic capability and prevented the suppression of oxidative phosphorylation in mitochondria after OS. Furthermore, TAT-Mito significantly reduced the apoptotic rates of cardiomyocytes after OS, compared to simple mitochondrial transfer. These results indicate the potential of mitochondrial replenishment therapy in OS-induced myocardial IRI