3,842 research outputs found
funcX: A Federated Function Serving Fabric for Science
Exploding data volumes and velocities, new computational methods and
platforms, and ubiquitous connectivity demand new approaches to computation in
the sciences. These new approaches must enable computation to be mobile, so
that, for example, it can occur near data, be triggered by events (e.g.,
arrival of new data), be offloaded to specialized accelerators, or run remotely
where resources are available. They also require new design approaches in which
monolithic applications can be decomposed into smaller components, that may in
turn be executed separately and on the most suitable resources. To address
these needs we present funcX---a distributed function as a service (FaaS)
platform that enables flexible, scalable, and high performance remote function
execution. funcX's endpoint software can transform existing clouds, clusters,
and supercomputers into function serving systems, while funcX's cloud-hosted
service provides transparent, secure, and reliable function execution across a
federated ecosystem of endpoints. We motivate the need for funcX with several
scientific case studies, present our prototype design and implementation, show
optimizations that deliver throughput in excess of 1 million functions per
second, and demonstrate, via experiments on two supercomputers, that funcX can
scale to more than more than 130000 concurrent workers.Comment: Accepted to ACM Symposium on High-Performance Parallel and
Distributed Computing (HPDC 2020). arXiv admin note: substantial text overlap
with arXiv:1908.0490
Decomposed versus integrated control of a one-stage production system
This paper considers the case of a one-stage production system with several products and operating under tight production capacity constraints. The production schedule is cyclical, and there are long and sequence dependent setup times. The production system is regarded to consist of two components, namely a production unit and an inventory unit. The performance, with respect to inventory costs, timing and production quantity determination, of two types of control of the production system are compared, namely so-called decomposed and integrated control. For the generation of production orders, decomposed control uses only information from the inventory unit, while integrated control combines the information from both units. The main conclusion, based on simulation experiments, is that the inventory costs are just slightly lower in case of integrated control. Integration outperforms decomposition with respect to timing and quantity determination. However, since the differences between both approaches are small, the less sophisticated approach of decomposition is preferable when choices between both types of control have to be made.
Efficient state-space inference of periodic latent force models
Latent force models (LFM) are principled approaches to incorporating solutions to differen-tial equations within non-parametric inference methods. Unfortunately, the developmentand application of LFMs can be inhibited by their computational cost, especially whenclosed-form solutions for the LFM are unavailable, as is the case in many real world prob-lems where these latent forces exhibit periodic behaviour. Given this, we develop a newsparse representation of LFMs which considerably improves their computational efficiency,as well as broadening their applicability, in a principled way, to domains with periodic ornear periodic latent forces. Our approach uses a linear basis model to approximate onegenerative model for each periodic force. We assume that the latent forces are generatedfrom Gaussian process priors and develop a linear basis model which fully expresses thesepriors. We apply our approach to model the thermal dynamics of domestic buildings andshow that it is effective at predicting day-ahead temperatures within the homes. We alsoapply our approach within queueing theory in which quasi-periodic arrival rates are mod-elled as latent forces. In both cases, we demonstrate that our approach can be implemented efficiently using state-space methods which encode the linear dynamic systems via LFMs.Further, we show that state estimates obtained using periodic latent force models can re-duce the root mean squared error to 17% of that from non-periodic models and 27% of thenearest rival approach which is the resonator model (S ̈arkk ̈a et al., 2012; Hartikainen et al.,2012.
Muppet: MapReduce-Style Processing of Fast Data
MapReduce has emerged as a popular method to process big data. In the past
few years, however, not just big data, but fast data has also exploded in
volume and availability. Examples of such data include sensor data streams, the
Twitter Firehose, and Facebook updates. Numerous applications must process fast
data. Can we provide a MapReduce-style framework so that developers can quickly
write such applications and execute them over a cluster of machines, to achieve
low latency and high scalability? In this paper we report on our investigation
of this question, as carried out at Kosmix and WalmartLabs. We describe
MapUpdate, a framework like MapReduce, but specifically developed for fast
data. We describe Muppet, our implementation of MapUpdate. Throughout the
description we highlight the key challenges, argue why MapReduce is not well
suited to address them, and briefly describe our current solutions. Finally, we
describe our experience and lessons learned with Muppet, which has been used
extensively at Kosmix and WalmartLabs to power a broad range of applications in
social media and e-commerce.Comment: VLDB201
Efficient State-Space Inference of Periodic Latent Force Models
Latent force models (LFM) are principled approaches to incorporating
solutions to differential equations within non-parametric inference methods.
Unfortunately, the development and application of LFMs can be inhibited by
their computational cost, especially when closed-form solutions for the LFM are
unavailable, as is the case in many real world problems where these latent
forces exhibit periodic behaviour. Given this, we develop a new sparse
representation of LFMs which considerably improves their computational
efficiency, as well as broadening their applicability, in a principled way, to
domains with periodic or near periodic latent forces. Our approach uses a
linear basis model to approximate one generative model for each periodic force.
We assume that the latent forces are generated from Gaussian process priors and
develop a linear basis model which fully expresses these priors. We apply our
approach to model the thermal dynamics of domestic buildings and show that it
is effective at predicting day-ahead temperatures within the homes. We also
apply our approach within queueing theory in which quasi-periodic arrival rates
are modelled as latent forces. In both cases, we demonstrate that our approach
can be implemented efficiently using state-space methods which encode the
linear dynamic systems via LFMs. Further, we show that state estimates obtained
using periodic latent force models can reduce the root mean squared error to
17% of that from non-periodic models and 27% of the nearest rival approach
which is the resonator model.Comment: 61 pages, 13 figures, accepted for publication in JMLR. Updates from
earlier version occur throughout article in response to JMLR review
Explicit computations for some Markov modulated counting processes
In this paper we present elementary computations for some Markov modulated
counting processes, also called counting processes with regime switching.
Regime switching has become an increasingly popular concept in many branches of
science. In finance, for instance, one could identify the background process
with the `state of the economy', to which asset prices react, or as an
identification of the varying default rate of an obligor. The key feature of
the counting processes in this paper is that their intensity processes are
functions of a finite state Markov chain. This kind of processes can be used to
model default events of some companies.
Many quantities of interest in this paper, like conditional characteristic
functions, can all be derived from conditional probabilities, which can, in
principle, be analytically computed. We will also study limit results for
models with rapid switching, which occur when inflating the intensity matrix of
the Markov chain by a factor tending to infinity. The paper is largely
expository in nature, with a didactic flavor
Real-time estimation of lane-based queue lengths at isolated signalized junctions
In this study, we develop a real-time estimation approach for lane-based queue lengths. Our aim is to determine the numbers of queued vehicles in each lane, based on detector information at isolated signalized junctions. The challenges involved in this task are to identify whether there is a residual queue at the start time of each cycle and to determine the proportions of lane-to-lane traffic volumes in each lane. Discriminant models are developed based on time occupancy rates and impulse memories, as calculated by the detector and signal information from a set of upstream and downstream detectors. To determine the proportions of total traffic volume in each lane, the downstream arrivals for each cycle are estimated by using the Kalman filter, which is based on upstream arrivals and downstream discharges collected during the previous cycle. Both the computer simulations and the case study of real-world traffic show that the proposed method is robust and accurate for the estimation of lane-based queue lengths in real time under a wide range of traffic conditions. Calibrated discriminant models play a significant role in determining whether there are residual queued vehicles in each lane at the start time of each cycle. In addition, downstream arrivals estimated by the Kalman filter enhance the accuracy of the estimates by minimizing any error terms caused by lane-changing behavior.postprin
DATASET2050 D5.1 - Mobility assessment
This document provides documentation on the mobility assessment metrics and methods for use within DATASET2050. On the one hand it describes what the key performance areas, attributes, indicators and metrics such as seamlessness, cost, duration, punctuality, comfort, resilience, etc. incorporated into the model are. On the other, it gives details about mobility metric computation, modelling methodology, visualisations used etc
- …