funcX: A Federated Function Serving Fabric for Science

Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences. These new approaches must enable computation to be mobile, so that, for example, it can occur near data, be triggered by events (e.g., arrival of new data), be offloaded to specialized accelerators, or run remotely where resources are available. They also require new design approaches in which monolithic applications can be decomposed into smaller components, that may in turn be executed separately and on the most suitable resources. To address these needs we present funcX---a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. funcX's endpoint software can transform existing clouds, clusters, and supercomputers into function serving systems, while funcX's cloud-hosted service provides transparent, secure, and reliable function execution across a federated ecosystem of endpoints. We motivate the need for funcX with several scientific case studies, present our prototype design and implementation, show optimizations that deliver throughput in excess of 1 million functions per second, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than more than 130000 concurrent workers.Comment: Accepted to ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC 2020). arXiv admin note: substantial text overlap with arXiv:1908.0490

SdrLift: A Domain-Specific Intermediate Hardware Synthesis Framework for Prototyping Software-Defined Radios

Modern design of Software-Defined Radio (SDR) applications is based on Field Programmable Gate Arrays (FPGA) due to their ability to be configured into solution architectures that are well suited to domain-specific problems while achieving the best trade-off between performance, power, area, and flexibility. FPGAs are well known for rich computational resources, which traditionally include logic, register, and routing resources. The increased technological advances have seen FPGAs incorporating more complex components that comprise sophisticated memory blocks, Digital Signal Processing (DSP) blocks, and high-speed interfacing to Gigabit Ethernet (GbE) and Peripheral Component Interconnect Express (PCIe) bus. Gateware for programming FPGAs is described at a lowlevel of design abstraction using Register Transfer Language (RTL), typically using either VHSIC-HDL (VHDL) or Verilog code. In practice, the low-level description languages have a very steep learning curve, provide low productivity for hardware designers and lack readily available open-source library support for fundamental designs, and consequently limit the design to only hardware experts. These limitations have led to the adoption of High-Level Synthesis (HLS) tools that raise design abstraction using syntax, semantics, and software development notations that are well-known to most software developers. However, while HLS has made programming of FPGAs more accessible and can increase the productivity of design, they are still not widely adopted in the design community due to the low-level skills that are still required to produce efficient designs. Additionally, the resultant RTL code from HLS tools is often difficult to decipher, modify and optimize due to the functionality and micro-architecture that are coupled together in a single High-Level Language (HLL). In order to alleviate these problems, Domain-Specific Languages (DSL) have been introduced to capture algorithms at a high level of abstraction with more expressive power and providing domain-specific optimizations that factor in new transformations and the trade-off between resource utilization and system performance. The problem of existing DSLs is that they are designed around imperative languages with an instruction sequence that does not match the hardware structure and intrinsics, leading to hardware designs with system properties that are unconformable to the high-level specifications and constraints. The aim of this thesis is, therefore, to design and implement an intermediatelevel framework namely SdrLift for use in high-level rapid prototyping of SDR applications that are based on an FPGA. The SdrLift input is a HLL developed using functional language constructs and design patterns that specify the structural behavior of the application design. The functionality of the SdrLift language is two-fold, first, it can be used directly by a designer to develop the SDR applications, secondly, it can be used as the Intermediate Representation (IR) step that is generated by a higher-level language or a DSL. The SdrLift compiler uses the dataflow graph as an IR to structurally represent the accelerator micro-architecture in which the components correspond to the fine-level and coarse-level Hardware blocks (HW Block) which are either auto-synthesized or integrated from existing reusable Intellectual Property (IP) core libraries. Another IR is in the form of a dataflow model and it is used for composition and global interconnection of the HW Blocks while making efficient interfacing decisions in an attempt to satisfy speed and resource usage objectives. Moreover, the dataflow model provides rules and properties that will be used to provide a theoretical framework that formally analyzes the characteristics of SDR applications (i.e. the throughput, sample rate, latency, and buffer size among other factors). Using both the directed graph flow (DFG) and the dataflow model in the SdrLift compiler provides two benefits: an abstraction of the microarchitecture from the high-level algorithm specifications and also decoupling of the microarchitecture from the low-level RTL implementation. Following the IR creation and model analyses is the VHDL code generation which employs the low-level optimizations that ensure optimal hardware design results. The code generation process per forms analysis to ensure the resultant hardware system conforms to the high-level design specifications and constraints. SdrLift is evaluated by developing representative SDR case studies, in which the VHDL code for eight different SDR applications is generated. The experimental results show that SdrLift achieves the desired performance and flexibility, while also conserving the hardware resources utilized

3D Sensor Placement and Embedded Processing for People Detection in an Industrial Environment

Papers I, II and III are extracted from the dissertation and uploaded as separate documents to meet post-publication requirements for self-arciving of IEEE conference papers.At a time when autonomy is being introduced in more and more areas, computer vision plays a very important role. In an industrial environment, the ability to create a real-time virtual version of a volume of interest provides a broad range of possibilities, including safety-related systems such as vision based anti-collision and personnel tracking. In an offshore environment, where such systems are not common, the task is challenging due to rough weather and environmental conditions, but the result of introducing such safety systems could potentially be lifesaving, as personnel work close to heavy, huge, and often poorly instrumented moving machinery and equipment. This thesis presents research on important topics related to enabling computer vision systems in industrial and offshore environments, including a review of the most important technologies and methods. A prototype 3D sensor package is developed, consisting of different sensors and a powerful embedded computer. This, together with a novel, highly scalable point cloud compression and sensor fusion scheme allows to create a real-time 3D map of an industrial area. The question of where to place the sensor packages in an environment where occlusions are present is also investigated. The result is algorithms for automatic sensor placement optimisation, where the goal is to place sensors in such a way that maximises the volume of interest that is covered, with as few occluded zones as possible. The method also includes redundancy constraints where important sub-volumes can be defined to be viewed by more than one sensor. Lastly, a people detection scheme using a merged point cloud from six different sensor packages as input is developed. Using a combination of point cloud clustering, flattening and convolutional neural networks, the system successfully detects multiple people in an outdoor industrial environment, providing real-time 3D positions. The sensor packages and methods are tested and verified at the Industrial Robotics Lab at the University of Agder, and the people detection method is also tested in a relevant outdoor, industrial testing facility. The experiments and results are presented in the papers attached to this thesis.publishedVersio

Phylogenetic and Transcriptomic Analyses of Vision in Two Cave Adapted Crustaceans, Asellus aquaticus (Isopoda: Asellidae) and Niphargus hrabei (Amphipoda: Niphargidae).

The unique characteristics of aquatic caves and of their predominantly crustacean biodiversity nominate them as ideal study subjects for evolutionary biology. The present dissertation capitalizes on a perfect natural experiment, the Molnar Janos thermal cave system in Budapest, Hungary. This intricate freshwater cave system and the immediately adjacent Malom Lake present the ideal opportunity to address questions of colonization, adaptation, and evolution. Despite marked environmental differences between the cave and surface waters, both localities are inhabited by natural populations of two emerging model cave species, the isopod Asellus aquaticus and the amphipod Niphargus hrabei. In the present dissertation, I first conduct an extensive literature review to examine and discuss the role that molecular methodologies have played in the study of cave biology. Additionally, I discuss the potential of “speleogenomic” methodologies to address long-standing questions in cave and evolutionary biology in fields such as biodiversity, phylogeography, and evolution. I then investigate the phylogeographic patterns and divergence-time estimates between surface and cave populations of the aforementioned species to elucidate mechanisms and processes driving the colonization of subterranean environments. These populations’ phylogenies then serve as robust frameworks on which to evaluate the transcriptional basis behind the divergence of traits involved in troglomorphy, namely vision. RNA sequencing approaches are used to identify and evaluate differences in the transcription of photoreception genes and pathways to in subterranean vs. surface populations. To achieve so, in a scalable manner suitable for modern sequencing technologies, here I produce a bioinformatics pipeline that allows for an accurate and efficient identification of genes present in a transcriptome that are involved in photoreception and visual pathways. I then use this bioinformatics pipeline to depict, in a phylogenetically informed context, the transcriptional basis behind photoreception and vision in A. aquaticus and N. hrabei, and the role these traits play in cave adaptation, and in the evolution of troglomorphy in the subphylum Crustacea. With the findings herein, the present dissertation aims to provide a framework for the discovery of evolutionarily significant molecular mechanisms that permit the survival and evolution of life in caves and other extreme environments

Reconstructing Palladio's villas : an analysis of Palladio's villa design and construction process

Thesis (Ph.D.)--Massachusetts Institute of Technology, Dept. of Architecture, 2000.Includes bibliographical references (p. 383-385).The thesis is a presentation of a method of reconstruction using a computational device to represent and evaluate two of Palladio's un-built villas in three-dimensions. The first of The Four Books of Architecture contains text and images explaining Palladio's design and construction systems in the form of text and graphic rules. These design guidelines or rules were written for the masons and craftsmen of the 16th century, offering one and two-dimensional data on each of Palladio's villas, palaces and churches. The text only offers general treatment of the villas; it missing construction data and rules needed to execute a full reconstruction of an un-built building. Many have attempted to reconstruct Palladio's work in drawings, wooden models and computation. This thesis presents a new method of reconstruction through the definition of construction rules in addition to shape and proportional rules defined by previous scholars. This reconstruction of the Villa Trissino in Meledo and the Villa Mocenigo on the Brenta River in the form of physical models, cad drawings and computer renderings from fragmented information offered in the Four Books. The end product will serve as a method for reconstruction in the form of a three-dimensional analysis of Palladio's design and construction rules and a demonstration of the new rules, through the two reconstructions. The work began with a pilot study focused on modeling Palladio's villas in three-dimensions with little detail. The next step was to reconstruct one villa in detail following the rules, which called for a complete rewriting of the rules from the Four Books of Architecture. These rewritten rules are applied to a simple floor plan and elevation drawing in order to reconstruct Palladio's original sketch in a CAD environment. The reconstructed sketches were used to create a three-dimensional CAD file by construction rules. Afterward, three-dimensional prints, two dimensional drawings and renderings were created from the model for evaluation. The final results of each study contain textural as well as visual information on the reconstruction of two un-built villas. The conclusions demonstrate how the results can be transformed into a full three-dimensional shape grammar composed of shape, proportion and construction rules.by Lawrence Sass.Ph.D

Data Mining Techniques to Understand Textual Data

More than ever, information delivery online and storage heavily rely on text. Billions of texts are produced every day in the form of documents, news, logs, search queries, ad keywords, tags, tweets, messenger conversations, social network posts, etc. Text understanding is a fundamental and essential task involving broad research topics, and contributes to many applications in the areas text summarization, search engine, recommendation systems, online advertising, conversational bot and so on. However, understanding text for computers is never a trivial task, especially for noisy and ambiguous text such as logs, search queries. This dissertation mainly focuses on textual understanding tasks derived from the two domains, i.e., disaster management and IT service management that mainly utilizing textual data as an information carrier. Improving situation awareness in disaster management and alleviating human efforts involved in IT service management dictates more intelligent and efficient solutions to understand the textual data acting as the main information carrier in the two domains. From the perspective of data mining, four directions are identified: (1) Intelligently generate a storyline summarizing the evolution of a hurricane from relevant online corpus; (2) Automatically recommending resolutions according to the textual symptom description in a ticket; (3) Gradually adapting the resolution recommendation system for time correlated features derived from text; (4) Efficiently learning distributed representation for short and lousy ticket symptom descriptions and resolutions. Provided with different types of textual data, data mining techniques proposed in those four research directions successfully address our tasks to understand and extract valuable knowledge from those textual data. My dissertation will address the research topics outlined above. Concretely, I will focus on designing and developing data mining methodologies to better understand textual information, including (1) a storyline generation method for efficient summarization of natural hurricanes based on crawled online corpus; (2) a recommendation framework for automated ticket resolution in IT service management; (3) an adaptive recommendation system on time-varying temporal correlated features derived from text; (4) a deep neural ranking model not only successfully recommending resolutions but also efficiently outputting distributed representation for ticket descriptions and resolutions

Local Features, Structure-from-motion and View Synthesis in Spherical Video

This thesis addresses the problem of synthesising new views from spherical video or image sequences. We propose an interest point detector and feature descriptor that allows us to robustly match local features between pairs of spherical images and use this as part of a structure-from-motion pipeline that allows us to estimate camera pose from a spherical video sequence. With pose estimates to hand, we propose methods for view stabilisation and novel viewpoint synthesis. In Chapter 3 we describe our contribution in the area of feature detection and description in spherical images. First, we present a novel representation for spherical images which uses a discrete geodesic grid composed of hexagonal pixels. Second, we extend the BRISK binary descriptor to the sphere, proposing methods for multiscale corner detection, sub-pixel position and sub-octave scale refinement and descriptor construction in the tangent space to the sphere. In Chapter 4 we describe our contributions in the area of spherical structure-from-motion. We revisit problems from multiview geometry in the context of spherical images. We propose methods suited to spherical camera geometry for the spherical-n-point problem and calibrated spherical reconstruction. We introduce a new probabilistic interpretation of spherical structure-from-motion which uses the von Mises-Fisher distribution in spherical feature point positions. This model provides an alternate objective function that we use in bundle adjustment. In Chapter 5 we describe our contributions in the area of view synthesis from spherical images. We exploit the camera pose estimates made by our pipeline and use these in two view synthesis applications. The first is view stabilisation where we remove the effect of viewing direction changes, often present in first person video. Second, we propose a method for synthesising novel viewpoints

The GEO Handbook on Biodiversity Observation Networks

biodiversity; conservation; ecosystem

CIRA annual report FY 2016/2017

Reporting period April 1, 2016-March 31, 2017