178 research outputs found
First-order distributed optimization methods for machine learning with linear speed-up
This thesis considers the problem of average consensus, distributed centralized and decentralized Stochastic Gradient Descent (SGD) and their communication requirements. Namely, (i) an algorithm for achieving consensus among a collection of agents is studied and its convergence to the average is shown, in the presence of link failures and delays. The new results improve upon the prior works by relaxing some of the restrictive assumptions on communication, such as bounded link failures and intercommunication intervals, as well as allowing for message delays. Next, (ii) a Robust Asynchronous Stochastic Gradient Push (RASGP) algorithm is proposed to minimize the separable objective F(z) = _{i=1}^n f_i(z) in a harsh network setting characterized by asynchronous updates, message losses and delays, and directed communication. RASGP is shown to asymptotically perform as well as the best bounds on a centralized gradient descent that takes steps in the direction of the sum of the noisy gradients of all local functions f_i(z). Next, (iii) a new communication strategy for Local SGD is proposed, a centralized optimization algorithm where workers make local updates and then calculate their average values only once in a while. It is shown that linear speed-up in the number of workers N is possible, using only O(N) communication (averaging) rounds, independent of the total number of iterations T. Empirical evidence suggests this bound is close to being tight as it is further shown that âN or N^{3/4} communications fail to achieve linear speed-up. Finally, (iv) under mild assumptions, the main of which is twice differentiability on any neighborhood of the optimal solution, one-shot averaging, which only uses a single round of communication, is shown to have optimal convergence rate asymptotically
Low complexity convergence rate bounds for the synchronous gossip subclass of push-sum algorithms
We develop easily accessible quantities for bounding the almost sure
exponential convergence rate of push-sum algorithms. We analyze the scenario of
i.i.d. synchronous gossip, every agent communicating towards its single target
at every step. Multiple bounding expressions are developed depending on the
generality of the setup, all functions of the spectrum of the network. While
the most general bound awaits further improvement, with more symmetries, close
bounds can be established, as demonstrated by numerical simulations.Comment: 15 pages, 8 figure
PersA-FL: Personalized Asynchronous Federated Learning
We study the personalized federated learning problem under asynchronous
updates. In this problem, each client seeks to obtain a personalized model that
simultaneously outperforms local and global models. We consider two
optimization-based frameworks for personalization: (i) Model-Agnostic
Meta-Learning (MAML) and (ii) Moreau Envelope (ME). MAML involves learning a
joint model adapted for each client through fine-tuning, whereas ME requires a
bi-level optimization problem with implicit gradients to enforce
personalization via regularized losses. We focus on improving the scalability
of personalized federated learning by removing the synchronous communication
assumption. Moreover, we extend the studied function class by removing
boundedness assumptions on the gradient norm. Our main technical contribution
is a unified proof for asynchronous federated learning with bounded staleness
that we apply to MAML and ME personalization frameworks. For the smooth and
non-convex functions class, we show the convergence of our method to a
first-order stationary point. We illustrate the performance of our method and
its tolerance to staleness through experiments for classification tasks over
heterogeneous datasets
Middleware services for distributed virtual environments
PhD ThesisDistributed Virtual Environments (DVEs) are virtual environments which allow
dispersed users to interact with each other and the virtual world through the
underlying network.
Scalability is a major challenge in building a successful DVE, which is directly
affected by the volume of message exchange. Different techniques have been
deployed to reduce the volume of message exchange in order to support large
numbers of simultaneous participants in a DVE. Interest management is a
popular technique for filtering unnecessary message exchange between users.
The rationale behind interest management is to resolve the "interests" of users
and decide whether messages should be exchanged between them. There are
three basic interest management approaches: region-based, aura-based and
hybrid approaches. However, if the time taken for an interest management
approach to determine interests is greater than the duration of the interaction, it
is not possible to guarantee interactions will occur correctly or at all. This is
termed the Missed Interaction Problem, which all existing interest management
approaches are susceptible to.
This thesis provides a new aura-based interest management approach, termed
Predictive Interest management (PIM), to alleviate the missed interaction
problem. PIM uses an enlarged aura to detect potential aura-intersections and
iii
initiate message exchange. It utilises variable message exchange frequencies,
proportional to the intersection degree of the objects' expanded auras, to restrict
bandwidth usage. This thesis provides an experimental system, the PIM system,
which couples predictive interest management with the de-centralised server
communication model. It utilises the Common Object Request Broker
Architecture (CORBA) middleware standard to provide an interoperable
middleware for DVEs. Experimental results are provided to demonstrate that
PIM provides a scalable interest management approach which alleviates the
missed interaction problem
Middleware services for distributed virtual environments
PhD ThesisDistributed Virtual Environments (DVEs) are virtual environments which allow
dispersed users to interact with each other and the virtual world through the
underlying network.
Scalability is a major challenge in building a successful DVE, which is directly
affected by the volume of message exchange. Different techniques have been
deployed to reduce the volume of message exchange in order to support large
numbers of simultaneous participants in a DVE. Interest management is a
popular technique for filtering unnecessary message exchange between users.
The rationale behind interest management is to resolve the "interests" of users
and decide whether messages should be exchanged between them. There are
three basic interest management approaches: region-based, aura-based and
hybrid approaches. However, if the time taken for an interest management
approach to determine interests is greater than the duration of the interaction, it
is not possible to guarantee interactions will occur correctly or at all. This is
termed the Missed Interaction Problem, which all existing interest management
approaches are susceptible to.
This thesis provides a new aura-based interest management approach, termed
Predictive Interest management (PIM), to alleviate the missed interaction
problem. PIM uses an enlarged aura to detect potential aura-intersections and
iii
initiate message exchange. It utilises variable message exchange frequencies,
proportional to the intersection degree of the objects' expanded auras, to restrict
bandwidth usage. This thesis provides an experimental system, the PIM system,
which couples predictive interest management with the de-centralised server
communication model. It utilises the Common Object Request Broker
Architecture (CORBA) middleware standard to provide an interoperable
middleware for DVEs. Experimental results are provided to demonstrate that
PIM provides a scalable interest management approach which alleviates the
missed interaction problem
Designing and Development of a Data Logging and Monitoring Tool
Since the mid 90's computer communication has become more and more common in cars and other auto mobiles. CAN based networks with sensors transmitting small data packets are utilized in the automotive industry to operate and supervise vehicles' functionality. To ease communication several higher layer protocols for CAN based networks have been developed. In some applications it is necessary to exchange information between networks using different protocols, and by connecting the two networks to a gateway, the information is translated and forwarded and intercommunication is enabled. This master thesis is conducted at Torqeedo GmbH, Munich. Theme of the thesis was âDesigning and Development of a Data Logging and Monitoring Toolâ. Term âdata loggingâ refers to the gathering or collection of specific data over a period of time. Monitoring means evaluate the data we are logging. Tools for data logging and monitoring are used in variant application these days. In medical, in-vehicle data logging and environment monitoring. This data could be voltage, current temperature, Time stump, heartbeat of the patient, vehicle fuel level etc. To capture and log data various communication channels used. Such channel varies from simple data cable to satellite link. There are variant protocols used for different communication channels. For our DBHS logging and monitoring tool we are using CANopen protocol. Main goal of this thesis is to develop a tool which can make debugging easy and log connection box data so we can use logged data later on for offline data analysis and simulation purposes
- âŠ