Efficient Parallel Reinforcement Learning Framework using the Reactor Model

Parallel Reinforcement Learning (RL) frameworks are essential for mapping RL workloads to multiple computational resources, allowing for faster generation of samples, estimation of values, and policy improvement. These computational paradigms require a seamless integration of training, serving, and simulation workloads. Existing frameworks, such as Ray, are not managing this orchestration efficiently, especially in RL tasks that demand intensive input/output and synchronization between actors on a single node. In this study, we have proposed a solution implementing the reactor model, which enforces a set of actors to have a fixed communication pattern. This allows the scheduler to eliminate work needed for synchronization, such as acquiring and releasing locks for each actor or sending and processing coordination-related messages. Our framework, Lingua Franca (LF), a coordination language based on the reactor model, also supports true parallelism in Python and provides a unified interface that allows users to automatically generate dataflow graphs for RL tasks. In comparison to Ray on a single-node multi-core compute platform, LF achieves 1.21x and 11.62x higher simulation throughput in OpenAI Gym and Atari environments, reduces the average training time of synchronized parallel Q-learning by 31.2%, and accelerates multi-agent RL inference by 5.12x.Comment: 10 pages, 11 figure

Highly parallel computation

Highly parallel computing architectures are the only means to achieve the computation rates demanded by advanced scientific problems. A decade of research has demonstrated the feasibility of such machines and current research focuses on which architectures designated as multiple instruction multiple datastream (MIMD) and single instruction multiple datastream (SIMD) have produced the best results to date; neither shows a decisive advantage for most near-homogeneous scientific problems. For scientific problems with many dissimilar parts, more speculative architectures such as neural networks or data flow may be needed

Dynamic resource allocation in a hierarchical multiprocessor system: A preliminary study

An integrated system approach to dynamic resource allocation is proposed. Some of the problems in dynamic resource allocation and the relationship of these problems to system structures are examined. A general dynamic resource allocation scheme is presented. A hierarchial system architecture which dynamically maps between processor structure and programs at multiple levels of instantiations is described. Simulation experiments were conducted to study dynamic resource allocation on the proposed system. Preliminary evaluation based on simple dynamic resource allocation algorithms indicates that with the proposed system approach, the complexity of dynamic resource management could be significantly reduced while achieving reasonable effective dynamic resource allocation

Parallel LISP

Projects in the past few years have looked into the problem of automatic parallelization of the Lisp programming language. Since it appears to be feasible to adapt Lisp to run on a general parallel computer, an implementation will be developed. This implementation will be as general as possible in order to locate the tradeoffs between implementing Lisp on a general parallel computer versus having an efficient interpreter. This implementation can be used to study the execution characteristics of Lisp in a parallel environment. It can also be used to derive information about architectural features which affect the performance of Lisp on parallel machines. This implementation will use a multitasking system and interprocess communication to simulate an MIMD machine. The implementation will include the formation, queuing, distribution, and execution of dataflow frames. Realistic Lisp application programs will be used with the implementation to examine the feasibility and efficiency of parallel Lisp. Measurements derivable from the simulator include number of processor cycles, processor utilization, memory requirements, and speedup. These tests will provide two main results. First, they will indicate possibilities for further gains by illustrating the bottlenecks in such a scheme. Second, they will help determine if it is indeed feasible to run Lisp on a parallel machine or if instead the overhead is too high for the application to be profitable. Most likely, some parallelism will be profitable. The simulation will provide information on the extent to which parallelism can be utilized

Integrated platform to assess seismic resilience at the community level

Due to the increasing frequency of disastrous events, the challenge of creating large-scale simulation models has become of major significance. Indeed, several simulation strategies and methodologies have been recently developed to explore the response of communities to natural disasters. Such models can support decision-makers during emergency operations allowing to create a global view of the emergency identifying consequences. An integrated platform that implements a community hybrid model with real-time simulation capabilities is presented in this paper. The platform's goal is to assess seismic resilience and vulnerability of critical infrastructures (e.g., built environment, power grid, socio-technical network) at the urban level, taking into account their interdependencies. Finally, different seismic scenarios have been applied to a large-scale virtual city model. The platform proved to be effective to analyze the emergency and could be used to implement countermeasures that improve community response and overall resilience

A lightweight, flow-based toolkit for parallel and distributed bioinformatics pipelines

<p>Abstract</p> <p>Background</p> <p>Bioinformatic analyses typically proceed as chains of data-processing tasks. A pipeline, or 'workflow', is a well-defined protocol, with a specific structure defined by the topology of data-flow interdependencies, and a particular functionality arising from the data transformations applied at each step. In computer science, the dataflow programming (DFP) paradigm defines software systems constructed in this manner, as networks of message-passing components. Thus, bioinformatic workflows can be naturally mapped onto DFP concepts.</p> <p>Results</p> <p>To enable the flexible creation and execution of bioinformatics dataflows, we have written a modular framework for parallel pipelines in Python ('PaPy'). A PaPy workflow is created from re-usable components connected by data-pipes into a directed acyclic graph, which together define nested higher-order map functions. The successive functional transformations of input data are evaluated on flexibly pooled compute resources, either local or remote. Input items are processed in batches of adjustable size, all flowing one to tune the trade-off between parallelism and lazy-evaluation (memory consumption). An add-on module ('NuBio') facilitates the creation of bioinformatics workflows by providing domain specific data-containers (<it>e.g</it>., for biomolecular sequences, alignments, structures) and functionality (<it>e.g</it>., to parse/write standard file formats).</p> <p>Conclusions</p> <p>PaPy offers a modular framework for the creation and deployment of parallel and distributed data-processing workflows. Pipelines derive their functionality from user-written, data-coupled components, so PaPy also can be viewed as a lightweight toolkit for extensible, flow-based bioinformatics data-processing. The simplicity and flexibility of distributed PaPy pipelines may help users bridge the gap between traditional desktop/workstation and grid computing. PaPy is freely distributed as open-source Python code at <url>http://muralab.org/PaPy</url>, and includes extensive documentation and annotated usage examples.</p

The exploitation of parallelism on shared memory multiprocessors

PhD ThesisWith the arrival of many general purpose shared memory multiple processor (multiprocessor) computers into the commercial arena during the mid-1980's, a rift has opened between the raw processing power offered by the emerging hardware and the relative inability of its operating software to effectively deliver this power to potential users. This rift stems from the fact that, currently, no computational model with the capability to elegantly express parallel activity is mature enough to be universally accepted, and used as the basis for programming languages to exploit the parallelism that multiprocessors offer. To add to this, there is a lack of software tools to assist programmers in the processes of designing and debugging parallel programs. Although much research has been done in the field of programming languages, no undisputed candidate for the most appropriate language for programming shared memory multiprocessors has yet been found. This thesis examines why this state of affairs has arisen and proposes programming language constructs, together with a programming methodology and environment, to close the ever widening hardware to software gap. The novel programming constructs described in this thesis are intended for use in imperative languages even though they make use of the synchronisation inherent in the dataflow model by using the semantics of single assignment when operating on shared data, so giving rise to the term shared values. As there are several distinct parallel programming paradigms, matching flavours of shared value are developed to permit the concise expression of these paradigms.The Science and Engineering Research Council