Parallelism in declarative languages

Imperative programming languages were initially built for uniprocessor systems that evolved out of the Von Neumann machine model. This model of storage oriented computation blocks parallelism and increases the cost of parallel program development and porting. Declarative languages based on mathematical models of computation, seem more suitable for the development of parallel programs. In the first part of this thesis we examine different language families under the declarative paradigm: functional, logic, and constraint languages. Functional languages are based on the abstract model of functions and (lamda)-calculus. They were initially developed for symbolic computation, but today they are commonly used in numerical analysis and many other application areas. Pure lisp is a widely known member of this class. Logic languages are based on first order predicate calculus. Although they were initially developed for theorem proving, fifth generation operating systems are written in them. Most logic languages are descendants or distant relatives of Prolog. Constraint languages are related to logic languages. In a constraint language you define a program object by placing constraints on its structure and its behavior. They were initially used in graphics applications, but today researchers work on using them in parallel computation. Here we will compare and contrast the language classes above, locate advantages and deficiencies, and explain different choices made by language implementors. In the second part of thesis we describe a front end for the CONSUL, a prototype constraint language for programming multiprocessors. The most important features of the front end are compact representation of constraints, type definitions, functional use of relations, and the ability to split programs into multiple files

A Formal Definition of SOL

This paper gives a formal definition of SOL, a general-purpose algorithmic language useful for describing and simulating complex systems. SOL is described using meta-linguistic formulas as used in the definition of ALGOL 60. The principal differences between SOL and problem-oriented languages such as ALGOL or FORTRAN is that SOL includes capabilities for expressing parallel computation, convenient notations for embedding random quantities within arithmetic expressions and automatic means for gathering statistics about the elements involved. SOL differs from other simulation languages such as SIMSCRIPT primarily in simplicity of use and in readability since it is capable of describing models without including computer-oriented characteristics

Models for Concurrency

Revised version of DAIMI PB-429 This is, we believe, the final version of a chapter for the Handbook of Logic and the Foundations of Computer Science, vol. IV, Oxford University Press.It surveys a range of models for parallel computation to include interleaving models like transition systems, synchronisation trees and languages (often called Hoare traces in this context), and models like Petri nets, asynchronous transition systems, event structures, pomsets and Mazurkiewicz traces where concurrency is represented more explicitly by a form of causal independence.The presentation is unified by casting the models in a category-theoretic framework. One aim is to use category theory to provide abstract characterisations of constructions like parallel composition valid throughout a range of different models and to provide formal means for translating between different models. A knowledge of basic category theory is assumed, up to an acquaintance with the notion of adjunction

LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation

Large-scale language-agnostic sentence embedding models such as LaBSE (Feng et al., 2022) obtain state-of-the-art performance for parallel sentence alignment. However, these large-scale models can suffer from inference speed and computation overhead. This study systematically explores learning language-agnostic sentence embeddings with lightweight models. We demonstrate that a thin-deep encoder can construct robust low-dimensional sentence embeddings for 109 languages. With our proposed distillation methods, we achieve further improvements by incorporating knowledge from a teacher model. Empirical results on Tatoeba, United Nations, and BUCC show the effectiveness of our lightweight models. We release our lightweight language-agnostic sentence embedding models LEALLA on TensorFlow Hub.Comment: EACL 2023 main conference; LEALLA models: https://tfhub.dev/google/collections/LEALL

Tree-size bounded alternation

AbstractThe size of an accepting computation tree of an alternating Turing machine (ATM) is introduced as a complexity measure. We present a number of applications of tree-size to the study of more traditional complexity classes. Tree-size on ATMs is shown to closely correspond to time on nondeterministic TMs and on nondeterministic auxiliary pushdown automata. One application of the later is a useful new characterization of the class of languages log-space-reducible to context-free languages. Surprising relationships with parallel-time complexity are also demonstrated. ATM computations using at most space S(n) and tree-size Z(n) (simultaneously) can be simulated in alternating space S(n) and time S(n) · log Z(n) (simultaneously). Several well-known simulations, e.g., Savitch's theorem, are special cases of this result. It also leads to improved parallel complexity bounds for many problems in terms of both time and number of “processors.” As one example we show that context-free language recognition in time O(log2 n) is possible on several parallel models. Further, this bound is achievable with only a polynomial number of processors, in contrast to all previously known sub-linear time CFL recognizers

Parallel Execution of ATL Transformation Rules

International audienceIndustrial environments that make use of Model-Driven Engineering (MDE) are starting to see the appearance of very large models, made by millions of elements. Such models are produced automatically (e.g., by reverse engineering complex systems) or manually by a large number of users (e.g., from social networks). The success of MDE in these application scenarios strongly depends on the scalability of model manipulation tools. While parallelization is one of the traditional ways of making computation systems scalable, developing parallel model transformations in a general-purpose language is a complex and error-prone task. In this paper we show that rule-based languages like ATL have strong parallelization properties. Transformations can be developed without taking into account concurrency concerns, and a transformation engine can automatically parallelize execution. We describe the implementation of a parallel transformation engine for the current version of the ATL language and experimentally evaluate the consequent gain in scalability

A Region-based Approach for Sparse Parallel Computing

This paper introduces a technique for parallel sparse computation by extending the array-language concept of regions---regular programmer-specified index sets used for specifying array computations. We introduce the notion of sparse regions which can represent an arbitrary set of indices. Sparse regions inherit the benefits of regular regions, including conciseness, a direct encapsulation of parallelism, and support for language performance models that highlight parallel overheads. We show that region-based array languages can benefit from the use of sparse regions, both in terms of the semantic richness available to the programmer and the execution times of the resulting program. We also demonstrate that regions result in efficient implementations as compared to array-based approachs, due to their role in amortizing sparse overheads and enabling optimizations