Sequential Gradient Coding For Straggler Mitigation

In distributed computing, slower nodes (stragglers) usually become a bottleneck. Gradient Coding (GC), introduced by Tandon et al., is an efficient technique that uses principles of error-correcting codes to distribute gradient computation in the presence of stragglers. In this paper, we consider the distributed computation of a sequence of gradients $\{g(1),g(2),\ldots,g(J)\}$, where processing of each gradient $g(t)$ starts in round-$t$ and finishes by round-$(t+T)$. Here $T\geq 0$ denotes a delay parameter. For the GC scheme, coding is only across computing nodes and this results in a solution where $T=0$. On the other hand, having $T>0$ allows for designing schemes which exploit the temporal dimension as well. In this work, we propose two schemes that demonstrate improved performance compared to GC. Our first scheme combines GC with selective repetition of previously unfinished tasks and achieves improved straggler mitigation. In our second scheme, which constitutes our main contribution, we apply GC to a subset of the tasks and repetition for the remainder of the tasks. We then multiplex these two classes of tasks across workers and rounds in an adaptive manner, based on past straggler patterns. Using theoretical analysis, we demonstrate that our second scheme achieves significant reduction in the computational load. In our experiments, we study a practical setting of concurrently training multiple neural networks over an AWS Lambda cluster involving 256 worker nodes, where our framework naturally applies. We demonstrate that the latter scheme can yield a 16\% improvement in runtime over the baseline GC scheme, in the presence of naturally occurring, non-simulated stragglers

Some new results on majority-logic codes for correction of random errors

The main advantages of random error-correcting majority-logic codes and majority-logic decoding in general are well known and two-fold. Firstly, they offer a partial solution to a classical coding theory problem, that of decoder complexity. Secondly, a majority-logic decoder inherently corrects many more random error patterns than the minimum distance of the code implies is possible. The solution to the decoder complexity is only a partial one because there are circumstances under which a majority-logic decoder is too complex and expensive to implement. [Continues.

Sparse Modeling for Image and Vision Processing

In recent years, a large amount of multi-disciplinary research has been conducted on sparse models and their applications. In statistics and machine learning, the sparsity principle is used to perform model selection---that is, automatically selecting a simple model among a large collection of them. In signal processing, sparse coding consists of representing data with linear combinations of a few dictionary elements. Subsequently, the corresponding tools have been widely adopted by several scientific communities such as neuroscience, bioinformatics, or computer vision. The goal of this monograph is to offer a self-contained view of sparse modeling for visual recognition and image processing. More specifically, we focus on applications where the dictionary is learned and adapted to data, yielding a compact representation that has been successful in various contexts.Comment: 205 pages, to appear in Foundations and Trends in Computer Graphics and Visio

Codes on Graphs and More

Modern communication systems strive to achieve reliable and efficient information transmission and storage with affordable complexity. Hence, efficient low-complexity channel codes providing low probabilities for erroneous receptions are needed. Interpreting codes as graphs and graphs as codes opens new perspectives for constructing such channel codes. Low-density parity-check (LDPC) codes are one of the most recent examples of codes defined on graphs, providing a better bit error probability than other block codes, given the same decoding complexity. After an introduction to coding theory, different graphical representations for channel codes are reviewed. Based on ideas from graph theory, new algorithms are introduced to iteratively search for LDPC block codes with large girth and to determine their minimum distance. In particular, new LDPC block codes of different rates and with girth up to 24 are presented. Woven convolutional codes are introduced as a generalization of graph-based codes and an asymptotic bound on their free distance, namely, the Costello lower bound, is proven. Moreover, promising examples of woven convolutional codes are given, including a rate 5/20 code with overall constraint length 67 and free distance 120. The remaining part of this dissertation focuses on basic properties of convolutional codes. First, a recurrent equation to determine a closed form expression of the exact decoding bit error probability for convolutional codes is presented. The obtained closed form expression is evaluated for various realizations of encoders, including rate 1/2 and 2/3 encoders, of as many as 16 states. Moreover, MacWilliams-type identities are revisited and a recursion for sequences of spectra of truncated as well as tailbitten convolutional codes and their duals is derived. Finally, the dissertation is concluded with exhaustive searches for convolutional codes of various rates with either optimum free distance or optimum distance profile, extending previously published results

Algebraic Codes For Error Correction In Digital Communication Systems

Access to the full-text thesis is no longer available at the author's request, due to 3rd party copyright restrictions. Access removed on 29.11.2016 by CS (TIS).Metadata merged with duplicate record (http://hdl.handle.net/10026.1/899) on 20.12.2016 by CS (TIS).C. Shannon presented theoretical conditions under which communication was possible error-free in the presence of noise. Subsequently the notion of using error correcting codes to mitigate the effects of noise in digital transmission was introduced by R. Hamming. Algebraic codes, codes described using powerful tools from algebra took to the fore early on in the search for good error correcting codes. Many classes of algebraic codes now exist and are known to have the best properties of any known classes of codes. An error correcting code can be described by three of its most important properties length, dimension and minimum distance. Given codes with the same length and dimension, one with the largest minimum distance will provide better error correction. As a result the research focuses on finding improved codes with better minimum distances than any known codes. Algebraic geometry codes are obtained from curves. They are a culmination of years of research into algebraic codes and generalise most known algebraic codes. Additionally they have exceptional distance properties as their lengths become arbitrarily large. Algebraic geometry codes are studied in great detail with special attention given to their construction and decoding. The practical performance of these codes is evaluated and compared with previously known codes in different communication channels. Furthermore many new codes that have better minimum distance to the best known codes with the same length and dimension are presented from a generalised construction of algebraic geometry codes. Goppa codes are also an important class of algebraic codes. A construction of binary extended Goppa codes is generalised to codes with nonbinary alphabets and as a result many new codes are found. This construction is shown as an efficient way to extend another well known class of algebraic codes, BCH codes. A generic method of shortening codes whilst increasing the minimum distance is generalised. An analysis of this method reveals a close relationship with methods of extending codes. Some new codes from Goppa codes are found by exploiting this relationship. Finally an extension method for BCH codes is presented and this method is shown be as good as a well known method of extension in certain cases

Hardware-Conscious Wireless Communication System Design

The work at hand is a selection of topics in efficient wireless communication system design, with topics logically divided into two groups.One group can be described as hardware designs conscious of their possibilities and limitations. In other words, it is about hardware that chooses its configuration and properties depending on the performance that needs to be delivered and the influence of external factors, with the goal of keeping the energy consumption as low as possible. Design parameters that trade off power with complexity are identified for analog, mixed signal and digital circuits, and implications of these tradeoffs are analyzed in detail. An analog front end and an LDPC channel decoder that adapt their parameters to the environment (e.g. fluctuating power level due to fading) are proposed, and it is analyzed how much power/energy these environment-adaptive structures save compared to non-adaptive designs made for the worst-case scenario. Additionally, the impact of ADC bit resolution on the energy efficiency of a massive MIMO system is examined in detail, with the goal of finding bit resolutions that maximize the energy efficiency under various system setups.In another group of themes, one can recognize systems where the system architect was conscious of fundamental limitations stemming from hardware.Put in another way, in these designs there is no attempt of tweaking or tuning the hardware. On the contrary, system design is performed so as to work around an existing and unchangeable hardware limitation. As a workaround for the problematic centralized topology, a massive MIMO base station based on the daisy chain topology is proposed and a method for signal processing tailored to the daisy chain setup is designed. In another example, a large group of cooperating relays is split into several smaller groups, each cooperatively performing relaying independently of the others. As cooperation consumes resources (such as bandwidth), splitting the system into smaller, independent cooperative parts helps save resources and is again an example of a workaround for an inherent limitation.From the analyses performed in this thesis, promising observations about hardware consciousness can be made. Adapting the structure of a hardware block to the environment can bring massive savings in energy, and simple workarounds prove to perform almost as good as the inherently limited designs, but with the limitation being successfully bypassed. As a general observation, it can be concluded that hardware consciousness pays off

Non-Orthogonal Signal and System Design for Wireless Communications

The thesis presents research in non-orthogonal multi-carrier signals, in which: (i) a new signal format termed truncated orthogonal frequency division multiplexing (TOFDM) is proposed to improve data rates in wireless communication systems, such as those used in mobile/cellular systems and wireless local area networks (LANs), and (ii) a new design and experimental implementation of a real-time spectrally efficient frequency division multiplexing (SEFDM) system are reported. This research proposes a modified version of the orthogonal frequency division multiplexing (OFDM) format, obtained by truncating OFDM symbols in the time-domain. In TOFDM, subcarriers are no longer orthogonally packed in the frequency-domain as time samples are only partially transmitted, leading to improved spectral efficiency. In this work, (i) analytical expressions are derived for the newly proposed TOFDM signal, followed by (ii) interference analysis, (iii) systems design for uncoded and coded schemes, (iv) experimental implementation and (v) performance evaluation of the new proposed signal and system, with comparisons to conventional OFDM systems. Results indicate that signals can be recovered with truncated symbol transmission. Based on the TOFDM principle, a new receiving technique, termed partial symbol recovery (PSR), is designed and implemented in software de ned radio (SDR), that allows efficient operation of two users for overlapping data, in wireless communication systems operating with collisions. The PSR technique is based on recovery of collision-free partial OFDM symbols, followed by the reconstruction of complete symbols to recover progressively the frames of two users suffering collisions. The system is evaluated in a testbed of 12-nodes using SDR platforms. The thesis also proposes channel estimation and equalization technique for non-orthogonal signals in 5G scenarios, using an orthogonal demodulator and zero padding. Finally, the implementation of complete SEFDM systems in real-time is investigated and described in detail

Achievable Rate and Modulation for Bandlimited Channels with Oversampling and 1-Bit Quantization at the Receiver

Sustainably realizing applications of the future with high performance demands requires that energy efficiency becomes a central design criterion for the entire system. For example, the power consumption of the analog-to-digital converter (ADC) can become a major factor when transmitting at large bandwidths and carrier frequencies, e.g., for ultra-short range high data rate communication. The consumed energy per conversion step increases with the sampling rate such that high resolution ADCs become unfeasible in the sub-THz regime at the very high sampling rates required. This makes signaling schemes adapted to 1-bit quantizers a promising alternative. We therefore quantify the performance of bandlimited 1-bit quantized wireless communication channels using techniques like oversampling and faster-than-Nyquist (FTN) signaling to compensate for the loss of achievable rate. As a limiting case, we provide bounds on the mutual information rate of the hard bandlimited 1-bit quantized continuous-time – i.e., infinitely oversampled – additive white Gaussian noise channel in the mid-to-high signal-to-noise ratio (SNR) regime. We derive analytic expressions using runlength encoded input signals. For real signals the maximum value of the lower bound on the spectral efficiency in the high-SNR limit was found to be approximately 1.63 bit/s/Hz. Since in practical scenarios the oversampling ratio remains finite, we derive bounds on the achievable rate of the bandlimited oversampled discrete-time channel. These bounds match the results of the continuous-time channel remarkably well. We observe spectral efficiencies up to 1.53 bit/s/Hz in the high-SNR limit given hard bandlimitation. When excess bandwidth is tolerable, spectral efficiencies above 2 bit/s/Hz per domain are achievable w.r.t. the 95 %-power containment bandwidth. Applying the obtained bounds to a bandlimited oversampled 1-bit quantized multiple-input multiple-output channel, we show the benefits when using appropriate power allocation schemes. As a constant envelope modulation scheme, continuous phase modulation is considered in order to relieve linearity requirements on the power amplifier. Noise-free performance limits are investigated for phase shift keying (PSK) and continuous phase frequency shift keying (CPFSK) using higher-order modulation alphabets and intermediate frequencies. Adapted waveforms are designed that can be described as FTN-CPFSK. With the same spectral efficiency in the high-SNR limit as PSK and CPFSK, these waveforms provide a significantly improved bit error rate (BER) performance. The gain in SNR required for achieving a certain BER can be up to 20 dB.Die nachhaltige Realisierung von zukünftigen Übertragungssystemen mit hohen Leistungsanforderungen erfordert, dass die Energieeffizienz zu einem zentralen Designkriterium für das gesamte System wird. Zum Beispiel kann die Leistungsaufnahme des Analog-Digital-Wandlers (ADC) zu einem wichtigen Faktor bei der Übertragung mit großen Bandbreiten und Trägerfrequenzen werden, z. B. für die Kommunikation mit hohen Datenraten über sehr kurze Entfernungen. Die verbrauchte Energie des ADCs steigt mit der Abtastrate, so dass hochauflösende ADCs im Sub-THz-Bereich bei den erforderlichen sehr hohen Abtastraten schwer einsetzbar sind. Dies macht Signalisierungsschemata, die an 1-Bit-Quantisierer angepasst sind, zu einer vielversprechenden Alternative. Wir quantifizieren daher die Leistungsfähigkeit von bandbegrenzten 1-Bit-quantisierten drahtlosen Kommunikationssystemen, wobei Techniken wie Oversampling und Faster-than-Nyquist (FTN) Signalisierung eingesetzt werden, um den durch Quantisierung verursachten Verlust der erreichbaren Rate auszugleichen. Wir geben Grenzen für die Transinformationsrate des Extremfalls eines strikt bandbegrenzten 1-Bit quantisierten zeitkontinuierlichen – d.h. unendlich überabgetasteten – Kanals mit additivem weißen Gauß’schen Rauschen bei mittlerem bis hohem Signal-Rausch-Verhältnis (SNR) an. Wir leiten analytische Ausdrücke basierend auf lauflängencodierten Eingangssignalen ab. Für reelle Signale ist der maximale Wert der unteren Grenze der spektralen Effizienz im Hoch-SNR-Bereich etwa 1,63 Bit/s/Hz. Da die Überabtastrate in praktischen Szenarien endlich bleibt, geben wir Grenzen für die erreichbare Rate eines bandbegrenzten, überabgetasteten zeitdiskreten Kanals an. Diese Grenzen stimmen mit den Ergebnissen des zeitkontinuierlichen Kanals bemerkenswert gut überein. Im Hoch-SNR-Bereich sind spektrale Effizienzen bis zu 1,53 Bit/s/Hz bei strikter Bandbegrenzung möglich. Wenn Energieanteile außerhalb des Frequenzbandes tolerierbar sind, können spektrale Effizienzen über 2 Bit/s/Hz pro Domäne – bezogen auf die Bandbreite, die 95 % der Energie enthält – erreichbar sein. Durch die Anwendung der erhaltenen Grenzen auf einen bandbegrenzten überabgetasteten 1-Bit quantisierten Multiple-Input Multiple-Output-Kanal zeigen wir Vorteile durch die Verwendung geeigneter Leistungsverteilungsschemata. Als Modulationsverfahren mit konstanter Hüllkurve betrachten wir kontinuierliche Phasenmodulation, um die Anforderungen an die Linearität des Leistungsverstärkers zu verringern. Beschränkungen für die erreichbare Datenrate bei rauschfreier Übertragung auf Zwischenfrequenzen mit Modulationsalphabeten höherer Ordnung werden für Phase-shift keying (PSK) and Continuous-phase frequency-shift keying (CPFSK) untersucht. Weiterhin werden angepasste Signalformen entworfen, die als FTN-CPFSK beschrieben werden können. Mit der gleichen spektralen Effizienz im Hoch-SNR-Bereich wie PSK und CPFSK bieten diese Signalformen eine deutlich verbesserte Bitfehlerrate (BER). Die Verringerung des erforderlichen SNRs zur Erreichung einer bestimmten BER kann bis zu 20 dB betragen