EControl: Fast Distributed Optimization with Compression and Error Control

Modern distributed training relies heavily on communication compression to reduce the communication overhead. In this work, we study algorithms employing a popular class of contractive compressors in order to reduce communication overhead. However, the naive implementation often leads to unstable convergence or even exponential divergence due to the compression bias. Error Compensation (EC) is an extremely popular mechanism to mitigate the aforementioned issues during the training of models enhanced by contractive compression operators. Compared to the effectiveness of EC in the data homogeneous regime, the understanding of the practicality and theoretical foundations of EC in the data heterogeneous regime is limited. Existing convergence analyses typically rely on strong assumptions such as bounded gradients, bounded data heterogeneity, or large batch accesses, which are often infeasible in modern machine learning applications. We resolve the majority of current issues by proposing EControl, a novel mechanism that can regulate error compensation by controlling the strength of the feedback signal. We prove fast convergence for EControl in standard strongly convex, general convex, and nonconvex settings without any additional assumptions on the problem or data heterogeneity. We conduct extensive numerical evaluations to illustrate the efficacy of our method and support our theoretical findings

AsGrad: A Sharp Unified Analysis of Asynchronous-SGD Algorithms

We analyze asynchronous-type algorithms for distributed SGD in the heterogeneous setting, where each worker has its own computation and communication speeds, as well as data distribution. In these algorithms, workers compute possibly stale and stochastic gradients associated with their local data at some iteration back in history and then return those gradients to the server without synchronizing with other workers. We present a unified convergence theory for non-convex smooth functions in the heterogeneous regime. The proposed analysis provides convergence for pure asynchronous SGD and its various modifications. Moreover, our theory explains what affects the convergence rate and what can be done to improve the performance of asynchronous algorithms. In particular, we introduce a novel asynchronous method based on worker shuffling. As a by-product of our analysis, we also demonstrate convergence guarantees for gradient-type algorithms such as SGD with random reshuffling and shuffle-once mini-batch SGD. The derived rates match the best-known results for those algorithms, highlighting the tightness of our approach. Finally, our numerical evaluations support theoretical findings and show the good practical performance of our method

Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized. For instance, the shared global parameters can be used to learn good data representations, whereas the personalized layers are fine-tuned for a specific client. Moreover, we present a simple algorithm for the partially personalized formulation that offers significant benefits to all clients. In particular, it breaks the curse of data heterogeneity in several settings, such as training with local steps, asynchronous training, and Byzantine-robust training

Clip21: Error Feedback for Gradient Clipping

Motivated by the increasing popularity and importance of large-scale training under differential privacy (DP) constraints, we study distributed gradient methods with gradient clipping, i.e., clipping applied to the gradients computed from local information at the nodes. While gradient clipping is an essential tool for injecting formal DP guarantees into gradient-based methods [1], it also induces bias which causes serious convergence issues specific to the distributed setting. Inspired by recent progress in the error-feedback literature which is focused on taming the bias/error introduced by communication compression operators such as Top-$k$ [2], and mathematical similarities between the clipping operator and contractive compression operators, we design Clip21 -- the first provably effective and practically useful error feedback mechanism for distributed methods with gradient clipping. We prove that our method converges at the same $\mathcal{O}\left(\frac{1}{K}\right)$ rate as distributed gradient descent in the smooth nonconvex regime, which improves the previous best $\mathcal{O}\left(\frac{1}{\sqrt{K}}\right)$ rate which was obtained under significantly stronger assumptions. Our method converges significantly faster in practice than competing methods

Bone Stress-Strain State Evaluation Using CT Based FEM

Nowadays, the use of a digital prototype in numerical modeling is one of the main approaches to calculating the elements of an inhomogeneous structure under the influence of external forces. The article considers a finite element analysis method based on computed tomography data. The calculations used a three-dimensional isoparametric finite element of a continuous medium developed by the authors with a linear approximation, based on weighted integration of the local stiffness matrix. The purpose of this study is to describe a general algorithm for constructing a numerical model that allows static calculation of objects with a porous structure according to its computed tomography data. Numerical modeling was carried out using kinematic boundary conditions. To evaluate the results obtained, computational and postprocessor grids were introduced. The qualitative assessment of the modeling data was based on the normalized error. Three-point bending of bone specimens of the pig forelimbs was considered as a model problem. The numerical simulation results were compared with the data obtained from a physical experiment. The relative error ranged from 3 to 15%, and the crack location, determined by the physical experiment, corresponded to the area where the ultimate strength values were exceeded, determined by numerical modeling. The results obtained reflect not only the effectiveness of the proposed approach, but also the agreement with experimental data. This method turned out to be relatively non-resource-intensive and time-efficient

New Therapy for Spinal Cord Injury: Autologous Genetically-Enriched Leucoconcentrate Integrated with Epidural Electrical Stimulation

The contemporary strategy for spinal cord injury (SCI) therapy aims to combine multiple approaches to control pathogenic mechanisms of neurodegeneration and stimulate neuroregeneration. In this study, a novel regenerative approach using an autologous leucoconcentrate enriched with transgenes encoding vascular endothelial growth factor (VEGF), glial cell line-derived neurotrophic factor (GDNF), and neural cell adhesion molecule (NCAM) combined with supra- and sub-lesional epidural electrical stimulation (EES) was tested on mini-pigs similar in morpho-physiological scale to humans. The complex analysis of the spinal cord recovery after a moderate contusion injury in treated mini-pigs compared to control animals revealed: better performance in behavioural and joint kinematics, restoration of electromyography characteristics, and improvement in selected immunohistology features related to cell survivability, synaptic protein expression, and glial reorganization above and below the injury. These results for the first time demonstrate the positive effect of intravenous infusion of autologous genetically-enriched leucoconcentrate producing recombinant molecules stimulating neuroregeneration combined with neuromodulation by translesional multisite EES on the restoration of the post-traumatic spinal cord in mini-pigs and suggest the high translational potential of this novel regenerative therapy for SCI patients

FedNL: Making Newton-Type Methods Applicable to Federated Learning

Inspired by recent work of Islamov et al (2021), we propose a family of Federated Newton Learn (FedNL) methods, which we believe is a marked step in the direction of making second-order methods applicable to FL. In contrast to the aforementioned work, FedNL employs a different Hessian learning technique which i) enhances privacy as it does not rely on the training data to be revealed to the coordinating server, ii) makes it applicable beyond generalized linear models, and iii) provably works with general contractive compression operators for compressing the local Hessians, such as Top-$K$ or Rank-$R$, which are vastly superior in practice. Notably, we do not need to rely on error feedback for our methods to work with contractive compressors. Moreover, we develop FedNL-PP, FedNL-CR and FedNL-LS, which are variants of FedNL that support partial participation, and globalization via cubic regularization and line search, respectively, and FedNL-BC, which is a variant that can further benefit from bidirectional compression of gradients and models, i.e., smart uplink gradient and smart downlink model compression. We prove local convergence rates that are independent of the condition number, the number of training data points, and compression variance. Our communication efficient Hessian learning technique provably learns the Hessian at the optimum. Finally, we perform a variety of numerical experiments that show that our FedNL methods have state-of-the-art communication complexity when compared to key baselines.Comment: 65 pages, 7 algorithms, 14 figures --- Accepted to ICML 202

Triple-Gene Therapy for Stroke: A Proof-of-Concept in Vivo Study in Rats

Natural brain repair after stroke is extremely limited, and current therapeutic options are even more scarce with no clinical break-through in sight. Despite restricted regeneration in the central nervous system, we have previously proved that human umbilical cord blood mono-nuclear cells (UCB-MC) transduced with adenoviral vectors carrying genes encoding vascular endothelial growth factor (VEGF), glial cell-derived neurotrophic factor (GDNF), and neural cell adhesion molecule (NCAM) successfully rescued neurons in amyotrophic lateral sclerosis and spinal cord injury. This proof-of-principle project was aimed at evaluating the beneficial effects of the same triple-gene approach in stroke. Rats subjected to distal occlusion of the middle cerebral artery were treated intrathecally with a combination of these genes either directly or using our cell-based (UCB-MC) approach. Various techniques and markers were employed to evaluate brain injury and subsequent recovery after treatment. Brain repair was most prominent when therapeutic genes were delivered via adenoviral vector- or UCB-MC-mediated approach. Remodeling of brain cortex in the stroke area was confirmed by reduction of infarct volume and attenuated neural cell death, depletion of astrocytes and microglial cells, and increase in the number of oligodendroglial cells and synaptic proteins expression. These results imply that intrathecal injection of genetically engineered UCB-MC over-expressing therapeutic molecules (VEGF, GDNF, and NCAM) following cerebral blood vessel occlusion might represent a novel avenue for future research into treating stroke