PALS/PRISM Software Design Description (SDD): Ver. 0.51

This Software Design Description (SDD) provides detailed information on the architecture and coding for the PRISM C++ library (version 0.51). The PRISM C++ library supports consistent information sharing and in- teractions between distributed components of networked embedded systems, e.g. avionics. It is designed to reduce the complexity of the networked sys- tem by employing synchronous semantics provided by the architectural pat- tern called a Physically-Asynchronous Logically-Synchronous (PALS) system.unpublishednot peer reviewe

Time Synchronization Using Distributed Observer Algorithm With Sliding Mode Control For Wireless Sensor Network

This research describes a novel distributed observer algorithm which uses augmented sliding mode control element to compensate for clock skew for wireless sensor network (WSN), in the world of Internet of Things (IoT). The algorithm is known as Time Synchronization using Distributed Observer algorithm with Sliding mode control element (TSDOS).The main purpose of proposing TSDOS is to estimate a common global clock time by which all nodes within the WSN can use it for communication purpose. Without a common global clock time, it is impossible for the nodes to exchange information since the time reference is different. By using only the local information, TDOS will be able to estimate the skew rate, the offset and the relative skew rate of the perceived virtual clock of the neighboring nodes. TSDOS is designed to be able to adapt to dynamic condition with faster convergence speed and reduced synchronization error. TSDOS has the characteristics of being totally distributed, asynchronous, scalable across different network topological structures and adaptable to ad-hoc nodes deployment and link failures.In this dissertation, TSDOS has been implemented in several experimental simulations subject to different network topology and ad-hoc nodes deployment in MATLAB. The purpose is to observe the performance of TSDOS in terms of synchronization speed and clock error of each individual node in the WSN through the simulation results. Last but not least, a comparison is made between TSDOS and another fully distributed consensus based protocol, Average Time Sync (ATS), and TSDOS proves itself to achieve faster convergence speed and reduced synchronization error induced in the network through MATLAB simulations

Macroservers: An Execution Model for DRAM Processor-In-Memory Arrays

The emergence of semiconductor fabrication technology allowing a tight coupling between high-density DRAM and CMOS logic on the same chip has led to the important new class of Processor-In-Memory (PIM) architectures. Newer developments provide powerful parallel processing capabilities on the chip, exploiting the facility to load wide words in single memory accesses and supporting complex address manipulations in the memory. Furthermore, large arrays of PIMs can be arranged into a massively parallel architecture. In this report, we describe an object-based programming model based on the notion of a macroserver. Macroservers encapsulate a set of variables and methods; threads, spawned by the activation of methods, operate asynchronously on the variables' state space. Data distributions provide a mechanism for mapping large data structures across the memory region of a macroserver, while work distributions allow explicit control of bindings between threads and data. Both data and work distributuions are first-class objects of the model, supporting the dynamic management of data and threads in memory. This offers the flexibility required for fully exploiting the processing power and memory bandwidth of a PIM array, in particular for irregular and adaptive applications. Thread synchronization is based on atomic methods, condition variables, and futures. A special type of lightweight macroserver allows the formulation of flexible scheduling strategies for the access to resources, using a monitor-like mechanism

Synchronization Landscapes in Small-World-Connected Computer Networks

Motivated by a synchronization problem in distributed computing we studied a simple growth model on regular and small-world networks, embedded in one and two-dimensions. We find that the synchronization landscape (corresponding to the progress of the individual processors) exhibits Kardar-Parisi-Zhang-like kinetic roughening on regular networks with short-range communication links. Although the processors, on average, progress at a nonzero rate, their spread (the width of the synchronization landscape) diverges with the number of nodes (desynchronized state) hindering efficient data management. When random communication links are added on top of the one and two-dimensional regular networks (resulting in a small-world network), large fluctuations in the synchronization landscape are suppressed and the width approaches a finite value in the large system-size limit (synchronized state). In the resulting synchronization scheme, the processors make close-to-uniform progress with a nonzero rate without global intervention. We obtain our results by ``simulating the simulations", based on the exact algorithmic rules, supported by coarse-grained arguments.Comment: 20 pages, 22 figure

Parallel Discrete Event Simulation with Erlang

Discrete Event Simulation (DES) is a widely used technique in which the state of the simulator is updated by events happening at discrete points in time (hence the name). DES is used to model and analyze many kinds of systems, including computer architectures, communication networks, street traffic, and others. Parallel and Distributed Simulation (PADS) aims at improving the efficiency of DES by partitioning the simulation model across multiple processing elements, in order to enabling larger and/or more detailed studies to be carried out. The interest on PADS is increasing since the widespread availability of multicore processors and affordable high performance computing clusters. However, designing parallel simulation models requires considerable expertise, the result being that PADS techniques are not as widespread as they could be. In this paper we describe ErlangTW, a parallel simulation middleware based on the Time Warp synchronization protocol. ErlangTW is entirely written in Erlang, a concurrent, functional programming language specifically targeted at building distributed systems. We argue that writing parallel simulation models in Erlang is considerably easier than using conventional programming languages. Moreover, ErlangTW allows simulation models to be executed either on single-core, multicore and distributed computing architectures. We describe the design and prototype implementation of ErlangTW, and report some preliminary performance results on multicore and distributed architectures using the well known PHOLD benchmark.Comment: Proceedings of ACM SIGPLAN Workshop on Functional High-Performance Computing (FHPC 2012) in conjunction with ICFP 2012. ISBN: 978-1-4503-1577-

A continuous-time analysis of distributed stochastic gradient

We analyze the effect of synchronization on distributed stochastic gradient algorithms. By exploiting an analogy with dynamical models of biological quorum sensing -- where synchronization between agents is induced through communication with a common signal -- we quantify how synchronization can significantly reduce the magnitude of the noise felt by the individual distributed agents and by their spatial mean. This noise reduction is in turn associated with a reduction in the smoothing of the loss function imposed by the stochastic gradient approximation. Through simulations on model non-convex objectives, we demonstrate that coupling can stabilize higher noise levels and improve convergence. We provide a convergence analysis for strongly convex functions by deriving a bound on the expected deviation of the spatial mean of the agents from the global minimizer for an algorithm based on quorum sensing, the same algorithm with momentum, and the Elastic Averaging SGD (EASGD) algorithm. We discuss extensions to new algorithms which allow each agent to broadcast its current measure of success and shape the collective computation accordingly. We supplement our theoretical analysis with numerical experiments on convolutional neural networks trained on the CIFAR-10 dataset, where we note a surprising regularizing property of EASGD even when applied to the non-distributed case. This observation suggests alternative second-order in-time algorithms for non-distributed optimization that are competitive with momentum methods.Comment: 9/14/19 : Final version, accepted for publication in Neural Computation. 4/7/19 : Significant edits: addition of simulations, deep network results, and revisions throughout. 12/28/18: Initial submissio