Robust and Communication-Efficient Collaborative Learning

We consider a decentralized learning problem, where a set of computing nodes aim at solving a non-convex optimization problem collaboratively. It is well-known that decentralized optimization schemes face two major system bottlenecks: stragglers' delay and communication overhead. In this paper, we tackle these bottlenecks by proposing a novel decentralized and gradient-based optimization algorithm named as QuanTimed-DSGD. Our algorithm stands on two main ideas: (i) we impose a deadline on the local gradient computations of each node at each iteration of the algorithm, and (ii) the nodes exchange quantized versions of their local models. The first idea robustifies to straggling nodes and the second alleviates communication efficiency. The key technical contribution of our work is to prove that with non-vanishing noises for quantization and stochastic gradients, the proposed method exactly converges to the global optimal for convex loss functions, and finds a first-order stationary point in non-convex scenarios. Our numerical evaluations of the QuanTimed-DSGD on training benchmark datasets, MNIST and CIFAR-10, demonstrate speedups of up to 3x in run-time, compared to state-of-the-art decentralized optimization methods

Distributed Robotic Vision for Calibration, Localisation, and Mapping

This dissertation explores distributed algorithms for calibration, localisation, and mapping in the context of a multi-robot network equipped with cameras and onboard processing, comparing against centralised alternatives where all data is transmitted to a singular external node on which processing occurs. With the rise of large-scale camera networks, and as low-cost on-board processing becomes increasingly feasible in robotics networks, distributed algorithms are becoming important for robustness and scalability. Standard solutions to multi-camera computer vision require the data from all nodes to be processed at a central node which represents a significant single point of failure and incurs infeasible communication costs. Distributed solutions solve these issues by spreading the work over the entire network, operating only on local calculations and direct communication with nearby neighbours. This research considers a framework for a distributed robotic vision platform for calibration, localisation, mapping tasks where three main stages are identified: an initialisation stage where calibration and localisation are performed in a distributed manner, a local tracking stage where visual odometry is performed without inter-robot communication, and a global mapping stage where global alignment and optimisation strategies are applied. In consideration of this framework, this research investigates how algorithms can be developed to produce fundamentally distributed solutions, designed to minimise computational complexity whilst maintaining excellent performance, and designed to operate effectively in the long term. Therefore, three primary objectives are sought aligning with these three stages

An Accelerated Decentralized Stochastic Proximal Algorithm for Finite Sums

Modern large-scale finite-sum optimization relies on two key aspects: distribution and stochastic updates. For smooth and strongly convex problems, existing decentralized algorithms are slower than modern accelerated variance-reduced stochastic algorithms when run on a single machine, and are therefore not efficient. Centralized algorithms are fast, but their scaling is limited by global aggregation steps that result in communication bottlenecks. In this work, we propose an efficient \textbf{A}ccelerated \textbf{D}ecentralized stochastic algorithm for \textbf{F}inite \textbf{S}ums named ADFS, which uses local stochastic proximal updates and randomized pairwise communications between nodes. On $n$ machines, ADFS learns from $nm$ samples in the same time it takes optimal algorithms to learn from $m$ samples on one machine. This scaling holds until a critical network size is reached, which depends on communication delays, on the number of samples $m$, and on the network topology. We provide a theoretical analysis based on a novel augmented graph approach combined with a precise evaluation of synchronization times and an extension of the accelerated proximal coordinate gradient algorithm to arbitrary sampling. We illustrate the improvement of ADFS over state-of-the-art decentralized approaches with experiments.Comment: Code available in source files. arXiv admin note: substantial text overlap with arXiv:1901.0986

Asynchrony and Acceleration in Gossip Algorithms

This paper considers the minimization of a sum of smooth and strongly convex functions dispatched over the nodes of a communication network. Previous works on the subject either focus on synchronous algorithms, which can be heavily slowed down by a few slow nodes (the straggler problem), or consider a model of asynchronous operation (Boyd et al., 2006) in which adjacent nodes communicate at the instants of Poisson point processes. We have two main contributions. 1) We propose CACDM (a Continuously Accelerated Coordinate Dual Method), and for the Poisson model of asynchronous operation, we prove CACDM to converge to optimality at an accelerated convergence rate in the sense of Nesterov et Stich, 2017. In contrast, previously proposed asynchronous algorithms have not been proven to achieve such accelerated rate. While CACDM is based on discrete updates, the proof of its convergence crucially depends on a continuous time analysis. 2) We introduce a new communication scheme based on Loss-Networks, that is programmable in a fully asynchronous and decentralized way, unlike the Poisson model of asynchronous operation that does not capture essential aspects of asynchrony such as non-instantaneous communications and computations. Under this Loss-Network model of asynchrony, we establish for CDM (a Coordinate Dual Method) a rate of convergence in terms of the eigengap of the Laplacian of the graph weighted by local effective delays. We believe this eigengap to be a fundamental bottleneck for convergence rates of asynchronous optimization. Finally, we verify empirically that CACDM enjoys an accelerated convergence rate in the Loss-Network model of asynchrony

Advanced Control and Optimization for Future Grid with Energy Storage Devices

In the future grid environment, more sustainable resources will be increasing steadily. Their inherent unpredictable and intermittent characteristics will inevitably cause adverse impacts on the system static, dynamic and economic performance simultaneously. In this context, energy storage (ES) devices have been receiving growing attention because of their significant falling prices. Therefore, how to utilize these ES to help alleviate the problem of renewable energy (RE) sources integration has become more and more attractive. In my thesis, I will try to resolve some of the related problems from several perspectives. First of all, a comprehensive Future Australian transmission network simulation platform is constructed in the software DIgSILENT. Then in-depth research has been done on the aspect of frequency controller design. Based on mathematical reasoning, an advanced robust H∞ Load Frequency Controller (LFC) is developed, which can be used to assist the power system to maintain a stable frequency when accommodating more renewables. Afterwards, I develop a power system sensitivity analysis based-Enhanced Optimal Distributed Consensus Algorithm (EODCA). In the following study, a Modified Consensus Alternating Direction Method of Multipliers (MC-ADMM) is proposed, with this approach it can be verified that the convergence speed is notably accelerated even for complex large dimensional systems. Overall, in the Master thesis, I successfully provide several novel and practical solutions, algorithms and methodologies in regards to tackling both the frequency, voltage and the power flow issues in a future grid with the assistance of energy storage devices. The scientific control and optimal dispatch of these facilities could provide us with a promising approach to mitigate the potential threats that the intermittent renewables posed on the power system in the following decades

A Hidden Resource: Household-led Rural Water Supply in Ethiopia

Self supply as a strategy for WASH is defined as "improvement to water supplies delivered largely or wholly through user investment usually at household level." The two research studies reported on in this paper examined self supply in rural Ethiopia, gaining insights on the performance of existing family wells, factors that affect the decision of families to build their own wells and the way they use them, and elements of the enabling environment that can be targeted to promote self supply

The EurAsEC Transport Corridors

The geographic and geo-economic location of EurAsEC countries gives them significant strategic potential for freight transit. EurAsEC has motorway and railway corridors running east-west and north-south, and a number of new corridors are being constructed. However, to handle such huge volumes of cargo, the region’s existing transport infrastructure must be modernised. Sea vs land: 2:1. Transportation of transit cargo by sea (transoceanic service) has some strong advantages, such as low delivery cost, established relationships with customers and high standards of service. This leads us to conclude that sea transit will prevail in the near future. Land transit routes offer only one competitive advantage – speed of delivery, which is two to three times faster compared with the sea routes linking East Asia with Eastern Europe. This advantage must be exploited. A considerable proportion of “time-sensitive” transit (some 16 million tonnes annually, according to the most conservative estimate) can be redirected to ITCs operated by EurAsEC. There are a number of physical and non-physical barriers to the realisation of the EurAsEC’s transit potential. Physical barriers include the poor state of motorways and railways and their related infrastructure, i.e. obsolete rolling-stock, which prevents any increase in transportation speeds and volumes; existing roads do not meet international standards; border crossing points and logistics centres have a low throughput capacity. Non-physical barriers include cumbersome permit systems, unreasonable delays in crossing borders, various charges and additional taxes imposed by regulatory and local authorities, scheduled and spot-check inspections of cargo weight, etc. The non-physical barriers are the most significant obstacles to the development of cargo transit in the region and cause serious delays in cargo delivery. Time lost does not only result in loss of money and customer trust, but also the loss of the main (in fact the only) competitive advantage land transit has over sea transit. Given their geographic position and national economic interests, Russia, Kazakhstan and their neighbours have a direct interest in the Eurasian integration process that extends beyond the boundaries of the post-Soviet space and involves the region’s most important countries. Projects implemented in certain economic sectors provide a reliable basis for regional economic integration. What begins in those key sectors eventually spreads to the institutional level. In this context, therefore, transportation must be among these priority sectors.Eurasian Economic Community, transport infrastructure, transport corridors, economic integration, post-Soviet space