Distributed learning and inference in deep models

In recent years, the size of deep learning problems has been increased significantly, both in terms of the number of available training samples as well as the number of parameters and complexity of the model. In this thesis, we considered the challenges encountered in training and inference of large deep models, especially on nodes with limited computational power and capacity. We studied two classes of related problems; 1) distributed training of deep models, and 2) compression and restructuring of deep models for efficient distributed and parallel execution to reduce inference times. Especially, we considered the communication bottleneck in distributed training and inference of deep models. Data compression is a viable tool to mitigate the communication bottleneck in distributed deep learning. However, the existing methods suffer from a few drawbacks, such as the increased variance of stochastic gradients (SG), slower convergence rate, or added bias to SG. In my Ph.D. research, we have addressed these challenges from three different perspectives: 1) Information Theory and the CEO Problem, 2) Indirect SG compression via Matrix Factorization, and 3) Quantized Compressive Sampling. We showed, both theoretically and via simulations, that our proposed methods can achieve smaller MSE than other unbiased compression methods with fewer communication bit-rates, resulting in superior convergence rates. Next, we considered federated learning over wireless multiple access channels (MAC). Efficient communication requires the communication algorithm to satisfy the constraints imposed by the nodes in the network and the communication medium. To satisfy these constraints and take advantage of the over-the-air computation inherent in MAC, we proposed a framework based on random linear coding and developed efficient power management and channel usage techniques to manage the trade-offs between power consumption and communication bit-rate. In the second part of this thesis, we considered the distributed parallel implementation of an already-trained deep model on multiple workers. Since latency due to the synchronization and data transfer among workers adversely affects the performance of the parallel implementation, it is desirable to have minimum interdependency among parallel sub-models on the workers. To achieve this goal, we developed and analyzed RePurpose, an efficient algorithm to rearrange the neurons in the neural network and partition them (without changing the general topology of the neural network) such that the interdependency among sub-models is minimized under the computations and communications constraints of the workers.Ph.D

Distributed Reception in the Presence of Gaussian Interference

abstract: An analysis is presented of a network of distributed receivers encumbered by strong in-band interference. The structure of information present across such receivers and how they might collaborate to recover a signal of interest is studied. Unstructured (random coding) and structured (lattice coding) strategies are studied towards this purpose for a certain adaptable system model. Asymptotic performances of these strategies and algorithms to compute them are developed. A jointly-compressed lattice code with proper configuration performs best of all strategies investigated.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201

Neural Distributed Compressor Discovers Binning

We consider lossy compression of an information source when the decoder has lossless access to a correlated one. This setup, also known as the Wyner-Ziv problem, is a special case of distributed source coding. To this day, practical approaches for the Wyner-Ziv problem have neither been fully developed nor heavily investigated. We propose a data-driven method based on machine learning that leverages the universal function approximation capability of artificial neural networks. We find that our neural network-based compression scheme, based on variational vector quantization, recovers some principles of the optimum theoretical solution of the Wyner-Ziv setup, such as binning in the source space as well as optimal combination of the quantization index and side information, for exemplary sources. These behaviors emerge although no structure exploiting knowledge of the source distributions was imposed. Binning is a widely used tool in information theoretic proofs and methods, and to our knowledge, this is the first time it has been explicitly observed to emerge from data-driven learning.Comment: draft of a journal version of our previous ISIT 2023 paper (available at: arXiv:2305.04380). arXiv admin note: substantial text overlap with arXiv:2305.0438

Fusing Dependent Decisions for Hypothesis Testing with Heterogeneous Sensors

In this paper, we consider a binary decentralized detection problem where the local sensor observations are quantized before their transmission to the fusion center. Sensor observations, and hence their quantized versions, may be heterogeneous as well as statistically dependent. A composite binary hypothesis testing problem is formulated, and a copula-based generalized likelihood ratio test (GLRT) based fusion rule is derived given that the local sensors are uniform multi-level quantizers. An alternative computationally efficient fusion rule is also designed which involves injecting a deliberate random disturbance to the local sensor decisions before fusion. Although the introduction of external noise causes a reduction in the received signal to noise ratio, it is shown that the proposed approach can result in a detection performance comparable to the GLRT detector without external noise, especially when the number of quantization levels is larg

Ultra Wideband Communications: from Analog to Digital

Ultrabreitband-Signale (Ultra Wideband [UWB]) können einen signifikanten Nutzen im Bereich drahtloser Kommunikationssysteme haben. Es sind jedoch noch einige Probleme offen, die durch Systemdesigner und Wissenschaftler gelöst werden müssen. Ein Funknetzsystem mit einer derart großen Bandbreite ist normalerweise auch durch eine große Anzahl an Mehrwegekomponenten mit jeweils verschiedenen Pfadamplituden gekennzeichnet. Daher ist es schwierig, die zeitlich verteilte Energie effektiv zu erfassen. Außerdem ist in vielen Fällen der naheliegende Ansatz, ein kohärenter Empfänger im Sinne eines signalangepassten Filters oder eines Korrelators, nicht unbedingt die beste Wahl. In der vorliegenden Arbeit wird dabei auf die bestehende Problematik und weitere Lösungsmöglichkeiten eingegangen. Im ersten Abschnitt geht es um „Impulse Radio UWB”-Systeme mit niedriger Datenrate. Bei diesen Systemen kommt ein inkohärenter Empfänger zum Einsatz. Inkohärente Signaldetektion stellt insofern einen vielversprechenden Ansatz dar, als das damit aufwandsgünstige und robuste Implementierungen möglich sind. Dies trifft vor allem in Anwendungsfällen wie den von drahtlosen Sensornetzen zu, wo preiswerte Geräte mit langer Batterielaufzeit nötigsind. Dies verringert den für die Kanalschätzung und die Synchronisation nötigen Aufwand, was jedoch auf Kosten der Leistungseffizienz geht und eine erhöhte Störempfindlichkeit gegenüber Interferenz (z.B. Interferenz durch mehrere Nutzer oder schmalbandige Interferenz) zur Folge hat. Um die Bitfehlerrate der oben genannten Verfahren zu bestimmen, wurde zunächst ein inkohärenter Combining-Verlust spezifiziert, welcher auftritt im Gegensatz zu kohärenter Detektion mit Maximum Ratio Multipath Combining. Dieser Verlust hängt von dem Produkt aus der Länge des Integrationsfensters und der Signalbandbreite ab. Um den Verlust durch inkohärentes Combining zu reduzieren und somit die Leistungseffizienz des Empfängers zu steigern, werden verbesserte Combining-Methoden für Mehrwegeempfang vorgeschlagen. Ein analoger Empfänger, bei dem der Hauptteil des Mehrwege-Combinings durch einen „Integrate and Dump”-Filter implementiert ist, wird für UWB-Systeme mit Zeit-Hopping gezeigt. Dabei wurde die Einsatzmöglichkeit von dünn besetzten Codes in solchen System diskutiert und bewertet. Des Weiteren wird eine Regel für die Code-Auswahl vorgestellt, welche die Stabilität des Systems gegen Mehrnutzer-Störungen sicherstellt und gleichzeitig den Verlust durch inkohärentes Combining verringert. Danach liegt der Fokus auf digitalen Lösungen bei inkohärenter Demodulation. Im Vergleich zum Analogempfänger besitzt ein Digitalempfänger einen Analog-Digital-Wandler im Zeitbereich gefolgt von einem digitalen Optimalfilter. Der digitale Optimalfilter dekodiert den Mehrfachzugriffscode kohärent und beschränkt das inkohärente Combining auf die empfangenen Mehrwegekomponenten im Digitalbereich. Es kommt ein schneller Analog-Digital-Wandler mit geringer Auflösung zum Einsatz, um einen vertretbaren Energieverbrauch zu gewährleisten. Diese Digitaltechnik macht den Einsatz langer Analogverzögerungen bei differentieller Demodulation unnötig und ermöglicht viele Arten der digitalen Signalverarbeitung. Im Vergleich zur Analogtechnik reduziert sie nicht nur den inkohärenten Combining-Verlust, sonder zeigt auch eine stärkere Resistenz gegenüber Störungen. Dabei werden die Auswirkungen der Auflösung und der Abtastrate der Analog-Digital-Umsetzung analysiert. Die Resultate zeigen, dass die verminderte Effizienz solcher Analog-Digital-Wandler gering ausfällt. Weiterhin zeigt sich, dass im Falle starker Mehrnutzerinterferenz sogar eine Verbesserung der Ergebnisse zu beobachten ist. Die vorgeschlagenen Design-Regeln spezifizieren die Anwendung der Analog-Digital-Wandler und die Auswahl der Systemparameter in Abhängigkeit der verwendeten Mehrfachzugriffscodes und der Modulationsart. Wir zeigen, wie unter Anwendung erweiterter Modulationsverfahren die Leistungseffizienz verbessert werden kann und schlagen ein Verfahren zur Unterdrückung schmalbandiger Störer vor, welches auf Soft Limiting aufbaut. Durch die Untersuchungen und Ergebnissen zeigt sich, dass inkohärente Empfänger in UWB-Kommunikationssystemen mit niedriger Datenrate ein großes Potential aufweisen. Außerdem wird die Auswahl der benutzbaren Bandbreite untersucht, um einen Kompromiss zwischen inkohärentem Combining-Verlust und Stabilität gegenüber langsamen Schwund zu erreichen. Dadurch wurde ein neues Konzept für UWB-Systeme erarbeitet: wahlweise kohärente oder inkohärente Empfänger, welche als UWB-Systeme Frequenz-Hopping nutzen. Der wesentliche Vorteil hiervon liegt darin, dass die Bandbreite im Basisband sich deutlich verringert. Mithin ermöglicht dies einfach zu realisierende digitale Signalverarbeitungstechnik mit kostengünstigen Analog-Digital-Wandlern. Dies stellt eine neue Epoche in der Forschung im Bereich drahtloser Sensorfunknetze dar. Der Schwerpunkt des zweiten Abschnitts stellt adaptiven Signalverarbeitung für hohe Datenraten mit „Direct Sequence”-UWB-Systemen in den Vordergrund. In solchen Systemen entstehen, wegen der großen Anzahl der empfangenen Mehrwegekomponenten, starke Inter- bzw. Intrasymbolinterferenzen. Außerdem kann die Funktionalität des Systems durch Mehrnutzerinterferenz und Schmalbandstörungen deutlich beeinflusst werden. Um sie zu eliminieren, wird die „Widely Linear”-Rangreduzierung benutzt. Dabei verbessert die Rangreduzierungsmethode das Konvergenzverhalten, besonders wenn der gegebene Vektor eine sehr große Anzahl an Abtastwerten beinhaltet (in Folge hoher einer Abtastrate). Zusätzlich kann das System durch die Anwendung der R-linearen Verarbeitung die Statistik zweiter Ordnung des nicht-zirkularen Signals vollständig ausnutzen, was sich in verbesserten Schätzergebnissen widerspiegelt. Allgemeine kann die Methode der „Widely Linear”-Rangreduzierung auch in andern Bereichen angewendet werden, z.B. in „Direct Sequence”-Codemultiplexverfahren (DS-CDMA), im MIMO-Bereich, im Global System for Mobile Communications (GSM) und beim Beamforming.The aim of this thesis is to investigate key issues encountered in the design of transmission schemes and receiving techniques for Ultra Wideband (UWB) communication systems. Based on different data rate applications, this work is divided into two parts, where energy efficient and robust physical layer solutions are proposed, respectively. Due to a huge bandwidth of UWB signals, a considerable amount of multipath arrivals with various path gains is resolvable at the receiver. For low data rate impulse radio UWB systems, suboptimal non-coherent detection is a simple way to effectively capture the multipath energy. Feasible techniques that increase the power efficiency and the interference robustness of non-coherent detection need to be investigated. For high data rate direct sequence UWB systems, a large number of multipath arrivals results in severe inter-/intra-symbol interference. Additionally, the system performance may also be deteriorated by multi-user interference and narrowband interference. It is necessary to develop advanced signal processing techniques at the receiver to suppress these interferences. Part I of this thesis deals with the co-design of signaling schemes and receiver architectures in low data rate impulse radio UWB systems based on non-coherent detection.● We analyze the bit error rate performance of non-coherent detection and characterize a non-coherent combining loss, i.e., a performance penalty with respect to coherent detection with maximum ratio multipath combining. The thorough analysis of this loss is very helpful for the design of transmission schemes and receive techniques innon-coherent UWB communication systems.● We propose to use optical orthogonal codes in a time hopping impulse radio UWB system based on an analog non-coherent receiver. The “analog” means that the major part of the multipath combining is implemented by an integrate and dump filter. The introduced semi-analytical method can help us to easily select the time hopping codes to ensure the robustness against the multi-user interference and meanwhile to alleviate the non-coherent combining loss.● The main contribution of Part I is the proposal of applying fully digital solutions in non-coherent detection. The proposed digital non-coherent receiver is based on a time domain analog-to-digital converter, which has a high speed but a very low resolution to maintain a reasonable power consumption. Compared to its analog counterpart, itnot only significantly reduces the non-coherent combining loss but also offers a higher interference robustness. In particular, the one-bit receiver can effectively suppress strong multi-user interference and is thus advantageous in separating simultaneously operating piconets.The fully digital solutions overcome the difficulty of implementing long analog delay lines and make differential UWB detection possible. They also facilitate the development of various digital signal processing techniques such as multi-user detection and non-coherent multipath combining methods as well as the use of advanced modulationschemes (e.g., M-ary Walsh modulation).● Furthermore, we present a novel impulse radio UWB system based on frequency hopping, where both coherent and non-coherent receivers can be adopted. The key advantage is that the baseband bandwidth can be considerably reduced (e.g., lower than 500 MHz), which enables low-complexity implementation of the fully digital solutions. It opens up various research activities in the application field of wireless sensor networks. Part II of this thesis proposes adaptive widely linear reduced-rank techniques to suppress interferences for high data rate direct sequence UWB systems, where second-order non-circular signals are used. The reduced-rank techniques are designed to improve the convergence performance and the interference robustness especially when the received vector contains a large number of samples (due to a high sampling rate in UWB systems). The widely linear processing takes full advantage of the second-order statistics of the non-circular signals and enhances the estimation performance. The generic widely linear reduced-rank concept also has a great potential in the applications of other systems such as Direct Sequence Code Division Multiple Access (DS-CDMA), Multiple Input Multiple Output (MIMO) system, and Global System for Mobile Communications (GSM), or in other areas such as beamforming

Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication

The paper studies distributed static parameter (vector) estimation in sensor networks with nonlinear observation models and noisy inter-sensor communication. It introduces \emph{separably estimable} observation models that generalize the observability condition in linear centralized estimation to nonlinear distributed estimation. It studies two distributed estimation algorithms in separably estimable models, the $\mathcal{NU}$ (with its linear counterpart $\mathcal{LU}$) and the $\mathcal{NLU}$. Their update rule combines a \emph{consensus} step (where each sensor updates the state by weight averaging it with its neighbors' states) and an \emph{innovation} step (where each sensor processes its local current observation.) This makes the three algorithms of the \textit{consensus + innovations} type, very different from traditional consensus. The paper proves consistency (all sensors reach consensus almost surely and converge to the true parameter value,) efficiency, and asymptotic unbiasedness. For $\mathcal{LU}$ and $\mathcal{NU}$, it proves asymptotic normality and provides convergence rate guarantees. The three algorithms are characterized by appropriately chosen decaying weight sequences. Algorithms $\mathcal{LU}$ and $\mathcal{NU}$ are analyzed in the framework of stochastic approximation theory; algorithm $\mathcal{NLU}$ exhibits mixed time-scale behavior and biased perturbations, and its analysis requires a different approach that is developed in the paper.Comment: IEEE Transactions On Information Theory, Vol. 58, No. 6, June 201

Multiterminal source coding: sum-rate loss, code designs, and applications to video sensor networks

Driven by a host of emerging applications (e.g., sensor networks and wireless video), distributed source coding (i.e., Slepian-Wolf coding, Wyner-Ziv coding and various other forms of multiterminal source coding), has recently become a very active research area. This dissertation focuses on multiterminal (MT) source coding problem, and consists of three parts. The first part studies the sum-rate loss of an important special case of quadratic Gaussian multi-terminal source coding, where all sources are positively symmetric and all target distortions are equal. We first give the minimum sum-rate for joint encoding of Gaussian sources in the symmetric case, and then show that the supremum of the sum-rate loss due to distributed encoding in this case is 1 2 log2 5 4 = 0:161 b/s when L = 2 and increases in the order of º L 2 log2 e b/s as the number of terminals L goes to infinity. The supremum sum-rate loss of 0:161 b/s in the symmetric case equals to that in general quadratic Gaussian two-terminal source coding without the symmetric assumption. It is conjectured that this equality holds for any number of terminals. In the second part, we present two practical MT coding schemes under the framework of Slepian-Wolf coded quantization (SWCQ) for both direct and indirect MT problems. The first, asymmetric SWCQ scheme relies on quantization and Wyner-Ziv coding, and it is implemented via source splitting to achieve any point on the sum-rate bound. In the second, conceptually simpler scheme, symmetric SWCQ, the two quantized sources are compressed using symmetric Slepian-Wolf coding via a channel code partitioning technique that is capable of achieving any point on the Slepian-Wolf sum-rate bound. Our practical designs employ trellis-coded quantization and turbo/LDPC codes for both asymmetric and symmetric Slepian-Wolf coding. Simulation results show a gap of only 0.139-0.194 bit per sample away from the sum-rate bound for both direct and indirect MT coding problems. The third part applies the above two MT coding schemes to two practical sources, i.e., stereo video sequences to save the sum rate over independent coding of both sequences. Experiments with both schemes on stereo video sequences using H.264, LDPC codes for Slepian-Wolf coding of the motion vectors, and scalar quantization in conjunction with LDPC codes for Wyner-Ziv coding of the residual coefficients give slightly smaller sum rate than separate H.264 coding of both sequences at the same video quality

Successive structuring of source coding algorithms for data fusion, buffering, and distribution in networks

Supervised by Gregory W. Wornell.Also issued as Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2002.Includes bibliographical references (p. 159-165).(cont.) We also explore the interactions between source coding and queue management in problems of buffering and distributing distortion-tolerant data. We formulate a general queuing model relevant to numerous communication scenarios, and develop a bound on the performance of any algorithm. We design an adaptive buffer-control algorithm for use in dynamic environments and under finite memory limitations; its performance closely approximates the bound. Our design uses multiresolution source codes that exploit the data's distortion-tolerance in minimizing end-to-end distortion. Compared to traditional approaches, the performance gains of the adaptive algorithm are significant - improving distortion, delay, and overall system robustness.by Stark Christiaan Draper