35 research outputs found
Distributed learning and inference in deep models
In recent years, the size of deep learning problems has been increased significantly, both in terms of the number of available training samples as well as the number of parameters and complexity of the model. In this thesis, we considered the challenges encountered in training and inference of large deep models, especially on nodes with limited computational power and capacity. We studied two classes of related problems; 1) distributed training of deep models, and 2) compression and restructuring of deep models for efficient distributed and parallel execution to reduce inference times. Especially, we considered the communication bottleneck in distributed training and inference of deep models. Data compression is a viable tool to mitigate the communication bottleneck in distributed deep learning. However, the existing methods suffer from a few drawbacks, such as the increased variance of stochastic gradients (SG), slower convergence rate, or added bias to SG. In my Ph.D. research, we have addressed these challenges from three different perspectives: 1) Information Theory and the CEO Problem, 2) Indirect SG compression via Matrix Factorization, and 3) Quantized Compressive Sampling. We showed, both theoretically and via simulations, that our proposed methods can achieve smaller MSE than other unbiased compression methods with fewer communication bit-rates, resulting in superior convergence rates. Next, we considered federated learning over wireless multiple access channels (MAC). Efficient communication requires the communication algorithm to satisfy the constraints imposed by the nodes in the network and the communication medium. To satisfy these constraints and take advantage of the over-the-air computation inherent in MAC, we proposed a framework based on random linear coding and developed efficient power management and channel usage techniques to manage the trade-offs between power consumption and communication bit-rate. In the second part of this thesis, we considered the distributed parallel implementation of an already-trained deep model on multiple workers. Since latency due to the synchronization and data transfer among workers adversely affects the performance of the parallel implementation, it is desirable to have minimum interdependency among parallel sub-models on the workers. To achieve this goal, we developed and analyzed RePurpose, an efficient algorithm to rearrange the neurons in the neural network and partition them (without changing the general topology of the neural network) such that the interdependency among sub-models is minimized under the computations and communications constraints of the workers.Ph.D
Distributed Reception in the Presence of Gaussian Interference
abstract: An analysis is presented of a network of distributed receivers encumbered by strong in-band interference. The structure of information present across such receivers and how they might collaborate to recover a signal of interest is studied. Unstructured (random coding) and structured (lattice coding) strategies are studied towards this purpose for a certain adaptable system model. Asymptotic performances of these strategies and algorithms to compute them are developed. A jointly-compressed lattice code with proper configuration performs best of all strategies investigated.Dissertation/ThesisDoctoral Dissertation Electrical Engineering 201
Neural Distributed Compressor Discovers Binning
We consider lossy compression of an information source when the decoder has
lossless access to a correlated one. This setup, also known as the Wyner-Ziv
problem, is a special case of distributed source coding. To this day, practical
approaches for the Wyner-Ziv problem have neither been fully developed nor
heavily investigated. We propose a data-driven method based on machine learning
that leverages the universal function approximation capability of artificial
neural networks. We find that our neural network-based compression scheme,
based on variational vector quantization, recovers some principles of the
optimum theoretical solution of the Wyner-Ziv setup, such as binning in the
source space as well as optimal combination of the quantization index and side
information, for exemplary sources. These behaviors emerge although no
structure exploiting knowledge of the source distributions was imposed. Binning
is a widely used tool in information theoretic proofs and methods, and to our
knowledge, this is the first time it has been explicitly observed to emerge
from data-driven learning.Comment: draft of a journal version of our previous ISIT 2023 paper (available
at: arXiv:2305.04380). arXiv admin note: substantial text overlap with
arXiv:2305.0438
Fusing Dependent Decisions for Hypothesis Testing with Heterogeneous Sensors
In this paper, we consider a binary decentralized detection problem where the local sensor observations are quantized before their transmission to the fusion center. Sensor observations, and hence their quantized versions, may be heterogeneous as well as statistically dependent. A composite binary hypothesis testing problem is formulated, and a copula-based generalized likelihood ratio test (GLRT) based fusion rule is derived given that the local sensors are uniform multi-level quantizers. An alternative computationally efficient fusion rule is also designed which involves injecting a deliberate random disturbance to the local sensor decisions before fusion. Although the introduction of external noise causes a reduction in the received signal to noise ratio, it is shown that the proposed approach can result in a detection performance comparable to the GLRT detector without external noise, especially when the number of quantization levels is larg
Ultra Wideband Communications: from Analog to Digital
ï»żUltrabreitband-Signale (Ultra Wideband [UWB]) können einen
signifikanten Nutzen im Bereich drahtloser Kommunikationssysteme haben. Es
sind jedoch noch einige Probleme offen, die durch Systemdesigner und
Wissenschaftler gelöst werden mĂŒssen. Ein Funknetzsystem mit einer derart
groĂen Bandbreite ist normalerweise auch durch eine groĂe Anzahl an
Mehrwegekomponenten mit jeweils verschiedenen Pfadamplituden
gekennzeichnet. Daher ist es schwierig, die zeitlich verteilte Energie
effektiv zu erfassen. AuĂerdem ist in vielen FĂ€llen der naheliegende
Ansatz, ein kohÀrenter EmpfÀnger im Sinne eines signalangepassten Filters
oder eines Korrelators, nicht unbedingt die beste Wahl. In der vorliegenden
Arbeit wird dabei auf die bestehende Problematik und weitere
Lösungsmöglichkeiten eingegangen.
Im ersten Abschnitt geht es um âImpulse Radio UWBâ-Systeme mit
niedriger Datenrate. Bei diesen Systemen kommt ein inkohÀrenter EmpfÀnger
zum Einsatz. InkohÀrente Signaldetektion stellt insofern einen
vielversprechenden Ansatz dar, als das damit aufwandsgĂŒnstige und robuste
Implementierungen möglich sind. Dies trifft vor allem in AnwendungsfÀllen
wie den von drahtlosen Sensornetzen zu, wo preiswerte GerÀte mit langer
Batterielaufzeit nötigsind. Dies verringert den fĂŒr die KanalschĂ€tzung
und die Synchronisation nötigen Aufwand, was jedoch auf Kosten der
Leistungseffizienz geht und eine erhöhte Störempfindlichkeit gegenĂŒber
Interferenz (z.B. Interferenz durch mehrere Nutzer oder schmalbandige
Interferenz) zur Folge hat.
Um die Bitfehlerrate der oben genannten Verfahren zu bestimmen, wurde
zunÀchst ein inkohÀrenter Combining-Verlust spezifiziert, welcher
auftritt im Gegensatz zu kohÀrenter Detektion mit Maximum Ratio Multipath
Combining. Dieser Verlust hÀngt von dem Produkt aus der LÀnge des
Integrationsfensters und der Signalbandbreite ab.
Um den Verlust durch inkohÀrentes Combining zu reduzieren und somit die
Leistungseffizienz des EmpfÀngers zu steigern, werden verbesserte
Combining-Methoden fĂŒr Mehrwegeempfang vorgeschlagen. Ein analoger
EmpfÀnger, bei dem der Hauptteil des Mehrwege-Combinings durch einen
âIntegrate and Dumpâ-Filter implementiert ist, wird fĂŒr UWB-Systeme
mit Zeit-Hopping gezeigt. Dabei wurde die Einsatzmöglichkeit von dĂŒnn
besetzten Codes in solchen System diskutiert und bewertet. Des Weiteren
wird eine Regel fĂŒr die Code-Auswahl vorgestellt, welche die StabilitĂ€t
des Systems gegen Mehrnutzer-Störungen sicherstellt und gleichzeitig den
Verlust durch inkohÀrentes Combining verringert.
Danach liegt der Fokus auf digitalen Lösungen bei inkohÀrenter
Demodulation. Im Vergleich zum AnalogempfÀnger besitzt ein
DigitalempfÀnger einen Analog-Digital-Wandler im Zeitbereich gefolgt von
einem digitalen Optimalfilter. Der digitale Optimalfilter dekodiert den
Mehrfachzugriffscode kohÀrent und beschrÀnkt das inkohÀrente Combining
auf die empfangenen Mehrwegekomponenten im Digitalbereich. Es kommt ein
schneller Analog-Digital-Wandler mit geringer Auflösung zum Einsatz, um
einen vertretbaren Energieverbrauch zu gewÀhrleisten. Diese Digitaltechnik
macht den Einsatz langer Analogverzögerungen bei differentieller
Demodulation unnötig und ermöglicht viele Arten der digitalen
Signalverarbeitung. Im Vergleich zur Analogtechnik reduziert sie nicht nur
den inkohÀrenten Combining-Verlust, sonder zeigt auch eine stÀrkere
Resistenz gegenĂŒber Störungen. Dabei werden die Auswirkungen der
Auflösung und der Abtastrate der Analog-Digital-Umsetzung analysiert. Die
Resultate zeigen, dass die verminderte Effizienz solcher
Analog-Digital-Wandler gering ausfÀllt. Weiterhin zeigt sich, dass im
Falle starker Mehrnutzerinterferenz sogar eine Verbesserung der Ergebnisse
zu beobachten ist. Die vorgeschlagenen Design-Regeln spezifizieren die
Anwendung der Analog-Digital-Wandler und die Auswahl der Systemparameter in
AbhÀngigkeit der verwendeten Mehrfachzugriffscodes und der Modulationsart.
Wir zeigen, wie unter Anwendung erweiterter Modulationsverfahren die
Leistungseffizienz verbessert werden kann und schlagen ein Verfahren zur
UnterdrĂŒckung schmalbandiger Störer vor, welches auf Soft Limiting
aufbaut. Durch die Untersuchungen und Ergebnissen zeigt sich, dass
inkohÀrente EmpfÀnger in UWB-Kommunikationssystemen mit niedriger
Datenrate ein groĂes Potential aufweisen.
AuĂerdem wird die Auswahl der benutzbaren Bandbreite untersucht, um einen
Kompromiss zwischen inkohÀrentem Combining-Verlust und StabilitÀt
gegenĂŒber langsamen Schwund zu erreichen. Dadurch wurde ein neues Konzept
fĂŒr UWB-Systeme erarbeitet: wahlweise kohĂ€rente oder inkohĂ€rente
EmpfÀnger, welche als UWB-Systeme Frequenz-Hopping nutzen. Der wesentliche
Vorteil hiervon liegt darin, dass die Bandbreite im Basisband sich deutlich
verringert. Mithin ermöglicht dies einfach zu realisierende digitale
Signalverarbeitungstechnik mit kostengĂŒnstigen Analog-Digital-Wandlern.
Dies stellt eine neue Epoche in der Forschung im Bereich drahtloser
Sensorfunknetze dar.
Der Schwerpunkt des zweiten Abschnitts stellt adaptiven Signalverarbeitung
fĂŒr hohe Datenraten mit âDirect Sequenceâ-UWB-Systemen in den
Vordergrund. In solchen Systemen entstehen, wegen der groĂen Anzahl der
empfangenen Mehrwegekomponenten, starke Inter- bzw.
Intrasymbolinterferenzen. AuĂerdem kann die FunktionalitĂ€t des Systems
durch Mehrnutzerinterferenz und Schmalbandstörungen deutlich beeinflusst
werden. Um sie zu eliminieren, wird die âWidely Linearâ-Rangreduzierung
benutzt. Dabei verbessert die Rangreduzierungsmethode das
Konvergenzverhalten, besonders wenn der gegebene Vektor eine sehr groĂe
Anzahl an Abtastwerten beinhaltet (in Folge hoher einer Abtastrate).
ZusÀtzlich kann das System durch die Anwendung der R-linearen Verarbeitung
die Statistik zweiter Ordnung des nicht-zirkularen Signals vollstÀndig
ausnutzen, was sich in verbesserten SchÀtzergebnissen widerspiegelt.
Allgemeine kann die Methode der âWidely Linearâ-Rangreduzierung auch in
andern Bereichen angewendet werden, z.B. in âDirect
Sequenceâ-Codemultiplexverfahren (DS-CDMA), im MIMO-Bereich, im Global
System for Mobile Communications (GSM) und beim Beamforming.The aim of this thesis is to investigate key issues encountered in the
design of transmission schemes and receiving techniques for Ultra Wideband
(UWB) communication systems. Based on different data rate applications,
this work is divided into two parts, where energy efficient and robust
physical layer solutions are proposed, respectively.
Due to a huge bandwidth of UWB signals, a considerable amount of multipath
arrivals with various path gains is resolvable at the receiver. For low
data rate impulse radio UWB systems, suboptimal non-coherent detection is a
simple way to effectively capture the multipath energy. Feasible techniques
that increase the power efficiency and the interference robustness of
non-coherent detection need to be investigated. For high data rate direct
sequence UWB systems, a large number of multipath arrivals results in
severe inter-/intra-symbol interference. Additionally, the system
performance may also be deteriorated by multi-user interference and
narrowband interference. It is necessary to develop advanced signal
processing techniques at the receiver to suppress these interferences.
Part I of this thesis deals with the co-design of signaling schemes and
receiver architectures in low data rate impulse radio UWB systems based on
non-coherent detection.â We analyze the bit error rate performance of
non-coherent detection and characterize a non-coherent combining loss,
i.e., a performance penalty with respect to coherent detection with maximum
ratio multipath combining. The thorough analysis of this loss is very
helpful for the design of transmission schemes and receive techniques
innon-coherent UWB communication systems.â We propose to use optical
orthogonal codes in a time hopping impulse radio UWB system based on an
analog non-coherent receiver. The âanalogâ means that the major part of
the multipath combining is implemented by an integrate and dump filter. The
introduced semi-analytical method can help us to easily select the time
hopping codes to ensure the robustness against the multi-user interference
and meanwhile to alleviate the non-coherent combining loss.â The main
contribution of Part I is the proposal of applying fully digital solutions
in non-coherent detection. The proposed digital non-coherent receiver is
based on a time domain analog-to-digital converter, which has a high speed
but a very low resolution to maintain a reasonable power consumption.
Compared to its analog counterpart, itnot only significantly reduces the
non-coherent combining loss but also offers a higher interference
robustness. In particular, the one-bit receiver can effectively suppress
strong multi-user interference and is thus advantageous in separating
simultaneously operating piconets.The fully digital solutions overcome the
difficulty of implementing long analog delay lines and make differential
UWB detection possible. They also facilitate the development of various
digital signal processing techniques such as multi-user detection and
non-coherent multipath combining methods as well as the use of advanced
modulationschemes (e.g., M-ary Walsh modulation).â Furthermore, we
present a novel impulse radio UWB system based on frequency hopping, where
both coherent and non-coherent receivers can be adopted. The key advantage
is that the baseband bandwidth can be considerably reduced (e.g., lower
than 500 MHz), which enables low-complexity implementation of the fully
digital solutions. It opens up various research activities in the
application field of wireless sensor networks.
Part II of this thesis proposes adaptive widely linear reduced-rank
techniques to suppress interferences for high data rate direct sequence UWB
systems, where second-order non-circular signals are used. The reduced-rank
techniques are designed to improve the convergence performance and the
interference robustness especially when the received vector contains a
large number of samples (due to a high sampling rate in UWB systems). The
widely linear processing takes full advantage of the second-order
statistics of the non-circular signals and enhances the estimation
performance. The generic widely linear reduced-rank concept also has a
great potential in the applications of other systems such as Direct
Sequence Code Division Multiple Access (DS-CDMA), Multiple Input Multiple
Output (MIMO) system, and Global System for Mobile Communications (GSM), or
in other areas such as beamforming
Distributed Parameter Estimation in Sensor Networks: Nonlinear Observation Models and Imperfect Communication
The paper studies distributed static parameter (vector) estimation in sensor
networks with nonlinear observation models and noisy inter-sensor
communication. It introduces \emph{separably estimable} observation models that
generalize the observability condition in linear centralized estimation to
nonlinear distributed estimation. It studies two distributed estimation
algorithms in separably estimable models, the (with its linear
counterpart ) and the . Their update rule combines
a \emph{consensus} step (where each sensor updates the state by weight
averaging it with its neighbors' states) and an \emph{innovation} step (where
each sensor processes its local current observation.) This makes the three
algorithms of the \textit{consensus + innovations} type, very different from
traditional consensus. The paper proves consistency (all sensors reach
consensus almost surely and converge to the true parameter value,) efficiency,
and asymptotic unbiasedness. For and , it proves
asymptotic normality and provides convergence rate guarantees. The three
algorithms are characterized by appropriately chosen decaying weight sequences.
Algorithms and are analyzed in the framework of
stochastic approximation theory; algorithm exhibits mixed
time-scale behavior and biased perturbations, and its analysis requires a
different approach that is developed in the paper.Comment: IEEE Transactions On Information Theory, Vol. 58, No. 6, June 201
Multiterminal source coding: sum-rate loss, code designs, and applications to video sensor networks
Driven by a host of emerging applications (e.g., sensor networks and wireless video),
distributed source coding (i.e., Slepian-Wolf coding, Wyner-Ziv coding and various other
forms of multiterminal source coding), has recently become a very active research area.
This dissertation focuses on multiterminal (MT) source coding problem, and consists
of three parts. The first part studies the sum-rate loss of an important special case
of quadratic Gaussian multi-terminal source coding, where all sources are positively symmetric
and all target distortions are equal. We first give the minimum sum-rate for joint
encoding of Gaussian sources in the symmetric case, and then show that the supremum of
the sum-rate loss due to distributed encoding in this case is 1
2 log2
5
4 = 0:161 b/s when L = 2
and increases in the order of
Âș
L
2 log2 e b/s as the number of terminals L goes to infinity.
The supremum sum-rate loss of 0:161 b/s in the symmetric case equals to that in general
quadratic Gaussian two-terminal source coding without the symmetric assumption. It is
conjectured that this equality holds for any number of terminals.
In the second part, we present two practical MT coding schemes under the framework
of Slepian-Wolf coded quantization (SWCQ) for both direct and indirect MT problems.
The first, asymmetric SWCQ scheme relies on quantization and Wyner-Ziv coding, and it
is implemented via source splitting to achieve any point on the sum-rate bound. In the second,
conceptually simpler scheme, symmetric SWCQ, the two quantized sources are compressed
using symmetric Slepian-Wolf coding via a channel code partitioning technique that is capable of achieving any point on the Slepian-Wolf sum-rate bound. Our practical
designs employ trellis-coded quantization and turbo/LDPC codes for both asymmetric and
symmetric Slepian-Wolf coding. Simulation results show a gap of only 0.139-0.194 bit per
sample away from the sum-rate bound for both direct and indirect MT coding problems.
The third part applies the above two MT coding schemes to two practical sources, i.e.,
stereo video sequences to save the sum rate over independent coding of both sequences.
Experiments with both schemes on stereo video sequences using H.264, LDPC codes for
Slepian-Wolf coding of the motion vectors, and scalar quantization in conjunction with
LDPC codes for Wyner-Ziv coding of the residual coefficients give slightly smaller sum
rate than separate H.264 coding of both sequences at the same video quality
Successive structuring of source coding algorithms for data fusion, buffering, and distribution in networks
Supervised by Gregory W. Wornell.Also issued as Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2002.Includes bibliographical references (p. 159-165).(cont.) We also explore the interactions between source coding and queue management in problems of buffering and distributing distortion-tolerant data. We formulate a general queuing model relevant to numerous communication scenarios, and develop a bound on the performance of any algorithm. We design an adaptive buffer-control algorithm for use in dynamic environments and under finite memory limitations; its performance closely approximates the bound. Our design uses multiresolution source codes that exploit the data's distortion-tolerance in minimizing end-to-end distortion. Compared to traditional approaches, the performance gains of the adaptive algorithm are significant - improving distortion, delay, and overall system robustness.by Stark Christiaan Draper