Adaptive Robust Distributed Learning in Diffusion Sensor Networks

In this paper, the problem of adaptive distributed learning in diffusion networks is considered. The algorithms are developed within the convex set theoretic framework. More specifically, they are based on computationally simple geometric projections onto closed convex sets. The paper suggests a novel combine-project-adapt protocol for cooperation among the nodes of the network; such a protocol fits naturally with the philosophy that underlies the projection-based rationale. Moreover, the possibility that some of the nodes may fail is also considered and it is addressed by employing robust statistics loss functions. Such loss functions can easily be accommodated in the adopted algorithmic framework; all that is required from a loss function is convexity. Under some mild assumptions, the proposed algorithms enjoy monotonicity, asymptotic optimality, asymptotic consensus, strong convergence and linear complexity with respect to the number of unknown parameters. Finally, experiments in the context of the system-identification task verify the validity of the proposed algorithmic schemes, which are compared to other recent algorithms that have been developed for adaptive distributed learning

Robust On-line Matrix Completion on Graphs

We study online robust matrix completion on graphs. At each iteration a vector with some entries missing is revealed and our goal is to reconstruct it by identifying the underlying low-dimensional subspace from which the vectors are drawn. We assume there is an underlying graph structure to the data, that is, the components of each vector correspond to nodes of a certain (known) graph, and their values are related accordingly. We give algorithms that exploit the graph to reconstruct the incomplete data, even in the presence of outlier noise. The theoretical properties of the algorithms are studied and numerical experiments using both synthetic and real world datasets verify the improved performance of the proposed technique compared to other state of the art algorithms

Trading off communications bandwidth with accuracy in adaptive diffusion networks

In this paper, a novel algorithm for bandwidth reduction in adaptive distributed learning is introduced. We deal with diffusion networks, in which the nodes cooperate with each other, by exchanging information, in order to estimate an unknown parameter vector of interest. We seek for solutions in the framework of set theoretic estimation. Moreover, in order to reduce the required bandwidth by the transmitted information, which is dictated by the dimension of the unknown vector, we choose to project and work in a lower dimension Krylov subspace. This provides the benefit of trading off dimensionality with accuracy. Full convergence properties are presented, and experiments, within the system identification task, demonstrate the robustness of the algorithmic technique

A Sparsity-Aware Adaptive Algorithm for Distributed Learning

In this paper, a sparsity-aware adaptive algorithm for distributed learning in diffusion networks is developed. The algorithm follows the set-theoretic estimation rationale. At each time instance and at each node of the network, a closed convex set, known as property set, is constructed based on the received measurements; this defines the region in which the solution is searched for. In this paper, the property sets take the form of hyperslabs. The goal is to find a point that belongs to the intersection of these hyperslabs. To this end, sparsity encouraging variable metric projections onto the hyperslabs have been adopted. Moreover, sparsity is also imposed by employing variable metric projections onto weighted $\ell_1$ balls. A combine adapt cooperation strategy is adopted. Under some mild assumptions, the scheme enjoys monotonicity, asymptotic optimality and strong convergence to a point that lies in the consensus subspace. Finally, numerical examples verify the validity of the proposed scheme, compared to other algorithms, which have been developed in the context of sparse adaptive learning

Trading off Complexity With Communication Costs in Distributed Adaptive Learning via Krylov Subspaces for Dimensionality Reduction

In this paper, the problemof dimensionality reduction in adaptive distributed learning is studied. We consider a network obeying the ad-hoc topology, in which the nodes sense an amount of data and cooperate with each other, by exchanging information, in order to estimate an unknown, common, parameter vector. The algorithm, to be presented here, follows the set-theoretic estimation rationale; i.e., at each time instant and at each node of the network, a closed convex set is constructed based on the received measurements, and this defines the region in which the solution is searched for. In this paper, these closed convex sets, known as property sets, take the form of hyperslabs. Moreover, in order to reduce the number of transmitted coefficients, which is dictated by the dimension of the unknown vector, we seek for possible solutions in a subspace of lower dimension; the technique will be developed around the Krylov subspace rationale. Our goal is to find a point that belongs to the intersection of this infinite number of hyperslabs and the respective Krylov subspaces. This is achieved via a sequence of projections onto the property sets and the Krylov subspaces. The case of highly correlated inputs that degrades the performance of the algorithm is also considered. This is overcome via a transformation whichwhitens the input. The proposed schemes are brought in a decentralized form by adopting the combine-adapt cooperation strategy among the nodes. Full convergence analysis is carried out and numerical tests verify the validity of the proposed schemes in different scenarios in the context of the adaptive distributed system identification task

Sparsity-promoting adaptive algorithm for distributed learning in diffusion networks

In this paper, a sparsity-promoting adaptive algorithm for distributed learning in diffusion networks is developed. The algorithm follows the set-theoretic estimation rationale, i.e., at each time instant and at each node, a closed convex set, namely a hyperslab, is constructed around the current measurement point. This defines the region in which the solution lies. The algorithm seeks a solution in the intersection of these hyperslabs by a sequence of projections. Sparsity is encouraged in two complimentary ways: a) by employing extra projections onto a weighted ℓ1 ball, that complies with our desire to constrain the respective weighted ℓ1 norm and b) by adopting variable metric projections onto the hyperslabs, which implicitly quantify data mismatch. A combine-adapt cooperation strategy is adopted. Under some mild assumptions, the scheme enjoys a number of elegant convergence properties. Finally, numerical examples verify the validity of the proposed scheme, compared to other algorithms, which have been developed in the context of sparse adaptive learning.compared to other algorithms, which have been developed in the context of sparse adaptive learning

Εύρωστοι προσαρμοστικοί αλγόριθμοι μηχανικής εκμάθησης για κατανεμημένη επεξεργασία σήματος

Distributed networks comprising a large number of nodes, e.g., Wireless Sensor Networks, Personal Computers (PC's), laptops, smart phones, etc., which cooperate with each other in order to reach a common goal, constitute a promising technology for several applications. Typical examples include: distributed environmental monitoring, acoustic source localization, power spectrum estimation, etc. Sophisticated cooperation mechanisms can significantly benefit the learning process, through which the nodes achieve their common objective. In this dissertation, the problem of adaptive learning in distributed networks is studied, focusing on the task of distributed estimation. A set of nodes sense information related to certain parameters and the estimation of these parameters constitutes the goal. Towards this direction, nodes exploit locally sensed measurements as well as information springing from interactions with other nodes of the network. Throughout this dissertation, the cooperation among the nodes follows the diffusion optimization rationale and the developed algorithms belong to the APSM algorithmic family. First, robust APSM--based techniques are proposed. The goal is to ``harmonize" the spatial information, received from the neighborhood, with the locally sensed one. This ``harmonization" is achieved by projecting the information of the neighborhood onto a convex set, constructed via the locally sensed measurements. Next, the scenario, in which a subset of the nodeset is malfunctioning and produces measurements heavily corrupted with noise, is considered. This problem is attacked by employing the Huber cost function, which is resilient to the presence of outliers. In the sequel, we study the issue of sparsity--aware adaptive distributed learning. The nodes of the network seek for an unknown sparse vector, which consists of a small number of non--zero coefficients. Weighted $\ell_1$--norm constraints are embedded, together with sparsity--promoting variable metric projections. Finally, we propose algorithms, which lead to a reduction of the communication demands, by forcing the estimates to lie within lower dimensional Krylov subspaces. The derived schemes serve a good trade-off between complexity/bandwidth demands and achieved performance.Κατανεμημένα δίκτυα τα οποία απαρτίζονται από έναν μεγάλο αριθμό κόμβων, π.χ. Δίκτυα Ασύρματων Αισθητήρων, προσωπικοί υπολογιστές, φορητοί υπολογιστές, έξυπνα τηλέφωνα, κλπ., οι οποίοι συνεργάζονται με σκοπό την επίτευξη ενός κοινού στόχου, αποτελούν μια υποσχόμενη τεχνολογία η οποία βρίσκει εφαρμογή σε πολλά μοντέρνα προβλήματα. Τυπικά παραδείγματα τέτοιων εφαρμογών είναι τα εξής: κατανεμημένη επίβλεψη περιβάλλοντος, εύρεση ακουστικής πηγής, εκτίμηση φάσματος, κλπ. Συνεργατικοί μηχανισμοί δύνανται να βελτιώσουν σημαντικά την διαδικασία εκμάθησης, μέσω της οποίας οι κόμβοι επιτυγχάνουν τον κοινό στόχο τους.Η παρούσα διατριβή μελετά το πρόβλημα της προσαρμοστικής μάθησης σε κατανεμημένα δίκτυα, εστιάζοντας στο πρόβλημα της κατανεμημένης εκτίμησης παραμέτρων. Ένα σύνολο από κόμβους λαμβάνουν πληροφορία, η οποία σχετίζεται με συγκεκριμένες παραμέτρους, και η εκτίμηση των εν λόγω παραμέτρων αποτελεί τον στόχο μας. Προς αυτήν την κατεύθυνση, οι κόμβοι λαμβάνουν υπόψη τόσο τις τοπικές μετρήσεις, όσο και την πληροφορία η οποία λαμβάνεται από την συνεργασία με τους υπόλοιπους κόμβους του δικτύου. Στα πλαίσια της παρούσας διατριβής, η συνεργασία μεταξύ των κόμβων ακολουθεί την φιλοσοφία της κατανεμημένης βελτιστοποίησης μέσω διάχυσης και οι προτεινόμενοι αλγόριθμοι ανήκουν στην οικογένεια APSM. Αρχικά, εύρωστοι αλγόριθμοι με βάση τον APSM προτείνονται. Ο στόχος είναι η «εναρμόνιση» της πληροφορίας, η οποία λαμβάνεται από την γειτονιά, με την τοπική πληροφορία. Η εν λόγω «εναρμόνιση» επιτυγχάνεται μέσω προβολής της πληροφορίας της γειτονιάς πάνω σε ένα κυρτό σύνολο, το οποίο κατασκευάζεται με βάση τοπικές μετρήσεις. Στην συνέχεια, αντιμετωπίζεται σενάριο κατά το οποίο ένα υποσύνολο των κόμβων του δικτύου δυσλειτουργεί και παράγει μετρήσεις, οι οποίες έχουν υποβαθμιστεί σημαντικά από τον θόρυβο. Για την επίλυση του εν λόγω προβλήματος γίνεται χρήση της συνάρτησης κόστους Huber, η οποία είναι εύρωστη στην ύπαρξη ακραίων τιμών θορύβου. Επιπλέον, μελετήθηκε το πρόβλημα της προσαρμοστικής εκτίμησης αραιών διανυσμάτων στα πλαίσια της κατανεμημένης μάθησης. Οι κόμβοι του δικτύου αναζητούν άγνωστο, αραιό διάνυσμα, το οποίο αποτελείται από μικρό αριθμό μη μηδενικών συντελεστών. Περιορισμοί σταθμισμένης l1 νόρμας καθώς και προβολές μεταβλητής μετρικής, οι οποίες ευνoούν αραιές λύσεις χρησιμοποιούνται. Τέλος, προτείνονται αλγόριθμοι, οι οποίοι οδηγούν σε μείωση της πληροφορίας που αποστέλλεται στο δίκτυο, περιορίζοντας τις εκτιμήσεις να βρίσκονται πάνω σε έναν Krylov υπόχωρο. Οι προτεινόμενοι αλγόριθμοι έχουν υψηλή απόδοση ενώ ταυτόχρονα οι απαιτούμενοι πόροι εύρους ζώνης και η πολυπλοκότητα παραμένουν σε λογικά επίπεδα