Stream Fusion, to Completeness

Stream processing is mainstream (again): Widely-used stream libraries are now available for virtually all modern OO and functional languages, from Java to C# to Scala to OCaml to Haskell. Yet expressivity and performance are still lacking. For instance, the popular, well-optimized Java 8 streams do not support the zip operator and are still an order of magnitude slower than hand-written loops. We present the first approach that represents the full generality of stream processing and eliminates overheads, via the use of staging. It is based on an unusually rich semantic model of stream interaction. We support any combination of zipping, nesting (or flat-mapping), sub-ranging, filtering, mapping-of finite or infinite streams. Our model captures idiosyncrasies that a programmer uses in optimizing stream pipelines, such as rate differences and the choice of a "for" vs. "while" loops. Our approach delivers hand-written-like code, but automatically. It explicitly avoids the reliance on black-box optimizers and sufficiently-smart compilers, offering highest, guaranteed and portable performance. Our approach relies on high-level concepts that are then readily mapped into an implementation. Accordingly, we have two distinct implementations: an OCaml stream library, staged via MetaOCaml, and a Scala library for the JVM, staged via LMS. In both cases, we derive libraries richer and simultaneously many tens of times faster than past work. We greatly exceed in performance the standard stream libraries available in Java, Scala and OCaml, including the well-optimized Java 8 streams

Computing infrastructure issues in distributed communications systems : a survey of operating system transport system architectures

The performance of distributed applications (such as file transfer, remote login, tele-conferencing, full-motion video, and scientific visualization) is influenced by several factors that interact in complex ways. In particular, application performance is significantly affected both by communication infrastructure factors and computing infrastructure factors. Several communication infrastructure factors include channel speed, bit-error rate, and congestion at intermediate switching nodes. Computing infrastructure factors include (among other things) both protocol processing activities (such as connection management, flow control, error detection, and retransmission) and general operating system factors (such as memory latency, CPU speed, interrupt and context switching overhead, process architecture, and message buffering). Due to a several orders of magnitude increase in network channel speed and an increase in application diversity, performance bottlenecks are shifting from the network factors to the transport system factors.This paper defines an abstraction called an "Operating System Transport System Architecture" (OSTSA) that is used to classify the major components and services in the computing infrastructure. End-to-end network protocols such as TCP, TP4, VMTP, XTP, and Delta-t typically run on general-purpose computers, where they utilize various operating system resources such as processors, virtual memory, and network controllers. The OSTSA provides services that integrate these resources to support distributed applications running on local and wide area networks.A taxonomy is presented to evaluate OSTSAs in terms of their support for protocol processing activities. We use this taxonomy to compare and contrast five general-purpose commercial and experimental operating systems including System V UNIX, BSD UNIX, the x-kernel, Choices, and Xinu

PERSIMMON: a visual dataflow language for machine learning

Persimmon is a visual programming interface that leverages scikit-learn to provide a drag and drop interface for developing Machine Learning and Data Mining pipelines. It is based on the dataflow programming principles, giving the user a functional visual language with a type safety system that checks connections at write time, non-strict evaluation, task parallelization, and execution visualization. It has been evaluated by participants on a three-task form, overall receiving good reviews, being praised by the use of colors to indicate types, consistent design, easy to navigate and shallow learning curve

Interleaving with Coroutines: A Practical Approach for Robust Index Joins

Index join performance is determined by the efficiency of the lookup operation on the involved index. Although database indexes are highly optimized to leverage processor caches, main memory accesses inevitably increase lookup runtime when the index outsizes the last-level cache; hence, index join performance drops. Still, robust index join performance becomes possible with instruction stream interleaving: given a group of lookups, we can hide cache misses in one lookup with instructions from other lookups by switching among their respective instruction streams upon a cache miss. In this paper, we propose interleaving with coroutines for any type of index join. We showcase our proposal on SAP HANA by implementing binary search and CSB+-tree traversal for an instance of index join related to dictionary compression. Coroutine implementations not only perform similarly to prior interleaving techniques, but also resemble the original code closely, while supporting both interleaved and non-interleaved execution. Thus, we claim that coroutines make interleaving practical for use in real DBMS codebases

Overforged : Design and Development of an Indie Simulation Game

Overforged is a simulation game set in a fictional medieval era. The game is heavily inspiredbythetitleOvercookedbydeveloperGhostTownGamesbutinthisgame,instead of cooking, the player must forge. The player plays the role of two blacksmith siblings, with customizable appearances, whoare blackmailed by the king to travel the world. Thisdocumentwillmonitorthedesignprocessofthevariouselementsandmechanics that make up the game, as well as their development. Throughout this process, the main problems encountered will be mentioned, as well as the strategies followed to overcome them. Duringalldevelopment,animationsofferedbyMixamowereused,manuallycreating only those that the tool does not offer. The development process was carried out in the Unity Engine, which is based on the C# programming language. Agamemadeoflocalcooperation mechanics, several levels that can be selected on a world map, and several characters was developed.Overforged é um jogo de simulação situado numa época medieval fictícia. O jogo é inspirado fortemente no título Overcooked da desenvolvedora Ghost Town Games, porém neste jogo, ao invés de cozinhar o jogador deve forjar. O jogador desempenha o papel de dois irmãos ferreiros, de aparência customizável, que são chantageados pelo rei a viajar o mundo. Neste documento será acompanhado o processo de design dos diversos elementos e mecânicas que compõem o jogo, bem como o desenvolvimento das mesmas. Ao longo deste processo, serão referidos os principais problemas encontrados bem como as estratégias acompanhadas para os ultrapassar. Durante todo o desenvolvimento foram utilizadas animações oferecidas pela Mixamo, criando manualmente apenas as que a ferramenta não oferece. O processo de desenvolvimento foi realizado no Unity Engine, que assenta sobre a linguagem de programação C#. Desenvolveu-se um jogo composto com mecânicas de cooperação local, vários níveis que podem ser selecionados num mapa mundo e diversos personagens

Deconstructing yield operator to enhance streams processing

Este trabalho foi financiado pelo Concurso Anual para Projetos de Investigação, Desenvolvimento, Inovação e Criação Artística (IDI&CA) 2020 do Instituto Politécnico de Lisboa. Código de referência IPL/2020/WebFluid/ISELCustomizing streams pipelines with new user-defined operations is a well-known pattern regarding streams processing. However, programming languages face two challenges when considering streams extensibility: 1) provide a compact and readable way to express new operations, and 2) keep streams’ laziness behavior. From here, we may find a consensus around the adoption of the generator operator, i.e. yield, as a means to fulfil both requirements, since most state-of-the-art programming languages provide this feature. Yet, what is the performance overhead of interleaving a yield-based operation in streams processing? In this work we present a benchmark based on realistic use cases of two different web APIs, namely: Last.fm and world weather on line, where custom yield-based operations may degrade the streams performance in twofold. We also propose a purely functional and minimalistic design, named tinyield, that can be easily adopted in any programming language and provides a concise way of chaining extension operations fluently, with low overhead in the eval uated benchmarks. The tinyield proposal was deployed in three different libraries, namely for Java (jayield), JavaScript (tinyield4ts) and .Net (tinyield4net).info:eu-repo/semantics/publishedVersio

Generating, Optimizing, and Scheduling a Compiler Level Representation of Stream Parallelism

Stream parallelism is often cited as a powerful programming model for expressing parallel computation for multi-core and heterogeneous computers. It allows programmers to concisely describe the concurrency and communication requirements found in a program and it allows compilers and runtime systems to easily generate efficient code targeting parallel hardware. This type of stream parallelism is often restricted to use the Synchronous Dataflow (SDF) model and implemented using static compilation and scheduling techniques. While powerful, SDF and the associated static methods have real limitations when applied to general purpose programing on general purpose hardware. To increase generality, we can define stream parallelism as a graph of processes communicating with one another over unidirectional data channels. Although dynamic scheduling techniques have been developed for this more general model, the powerful compiler transformations that are available under the SDF model no longer apply. This is made worse by the fact that general purpose models are typically implemented as software frameworks on top of high-level general purpose languages. The Stream and Kernel Intermediate Representation (SKIR) is a compiler level representation of stream parallelism for general purpose languages. A SKIR compiler is able to recognize and take advantage of SDF style parallelism while allowing more general programs. This thesis presents the SKIR program representation and describes how it can be used as compiler target for several different high level languages. We show a dynamic scheduling mechanism for SKIR programs based on the concepts of coroutines and task stealing. We also propose code optimizations to reduce runtime overhead associated with dynamic scheduling. Such techniques are not possible in a high-level software framework and provide performance that meets or exceeds the performance of existing systems while providing greater generality and portability than static methods

Functional Programming for Modular Bayesian Inference

We present an architectural design of a library for Bayesian modelling and inference in modern functional programming languages. The novel aspect of our approach are modular implementations of existing state-of- the-art inference algorithms. Our design relies on three inherently functional features: higher-order functions, inductive data-types, and support for either type-classes or an expressive module system. We provide a perfor- mant Haskell implementation of this architecture, demonstrating that high-level and modular probabilistic programming can be added as a library in sufficiently expressive languages. We review the core abstractions in this architecture: inference representations, inference transformations, and inference representation transformers. We then implement concrete instances of these abstractions, counterparts to particle filters and Metropolis-Hastings samplers, which form the basic building blocks of our library. By composing these building blocks we obtain state-of-the-art inference algorithms: Resample-Move Sequential Monte Carlo, Particle Marginal Metropolis-Hastings, and Sequential Monte Carlo Squared. We evaluate our implementation against existing probabilistic programming systems and find it is already com- petitively performant, although we conjecture that existing functional programming optimisation techniques could reduce the overhead associated with the abstractions we use. We show that our modular design enables deterministic testing of inherently stochastic Monte Carlo algorithms. Finally, we demonstrate using OCaml that an expressive module system can also implement our design