Building Near-Real-Time Processing Pipelines with the Spark-MPI Platform

Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three Vs (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities. Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.Comment: New York Scientific Data Summit, August 6-9, 201

The Digital Anatomist Information System and Its Use in the Generation and Delivery of Web-Based Anatomy Atlases

Advances in network and imaging technology, coupled with the availability of 3-D datasets such as the Visible Human, provide a unique opportunity for developing information systems in anatomy that can deliver relevant knowledge directly to the clinician, researcher or educator. A software framework is described for developing such a system within a distributed architecture that includes spatial and symbolic anatomy information resources, Web and custom servers, and authoring and end-user client programs. The authoring tools have been used to create 3-D atlases of the brain, knee and thorax that are used both locally and throughout the world. For the one and a half year period from June 1995–January 1997, the on-line atlases were accessed by over 33,000 sites from 94 countries, with an average of over 4000 ‘‘hits’’ per day, and 25,000 hits per day during peak exam periods. The atlases have been linked to by over 500 sites, and have received at least six unsolicited awards by outside rating institutions. The flexibility of the software framework has allowed the information system to evolve with advances in technology and representation methods. Possible new features include knowledge-based image retrieval and tutoring, dynamic generation of 3-D scenes, and eventually, real-time virtual reality navigation through the body. Such features, when coupled with other on-line biomedical information resources, should lead to interesting new ways for managing and accessing structural information in medicine

Grid simulation services for the medical community

The first part of this paper presents a selection of medical simulation applications, including image reconstruction, near real-time registration for neuro-surgery, enhanced dose distribution calculation for radio-therapy, inhaled drug delivery prediction, plastic surgery planning and cardio-vascular system simulation. The latter two topics are discussed in some detail. In the second part, we show how such services can be made available to the clinical practitioner using Grid technology. We discuss the developments and experience made during the EU project GEMSS, which provides reliable, efficient, secure and lawful medical Grid services

Analysis domain model for shared virtual environments

The field of shared virtual environments, which also encompasses online games and social 3D environments, has a system landscape consisting of multiple solutions that share great functional overlap. However, there is little system interoperability between the different solutions. A shared virtual environment has an associated problem domain that is highly complex raising difficult challenges to the development process, starting with the architectural design of the underlying system. This paper has two main contributions. The first contribution is a broad domain analysis of shared virtual environments, which enables developers to have a better understanding of the whole rather than the part(s). The second contribution is a reference domain model for discussing and describing solutions - the Analysis Domain Model

Spectroscopic Analysis in the Virtual Observatory Environment with SPLAT-VO

SPLAT-VO is a powerful graphical tool for displaying, comparing, modifying and analyzing astronomical spectra, as well as searching and retrieving spectra from services around the world using Virtual Observatory (VO) protocols and services. The development of SPLAT-VO started in 1999, as part of the Starlink StarJava initiative, sometime before that of the VO, so initial support for the VO was necessarily added once VO standards and services became available. Further developments were supported by the Joint Astronomy Centre, Hawaii until 2009. Since end of 2011 development of SPLAT-VO has been continued by the German Astrophysical Virtual Observatory, and the Astronomical Institute of the Academy of Sciences of the Czech Republic. From this time several new features have been added, including support for the latest VO protocols, along with new visualization and spectra storing capabilities. This paper presents the history of SPLAT-VO, it's capabilities, recent additions and future plans, as well as a discussion on the motivations and lessons learned up to now.Comment: 15 pages, 6 figures, accepted for publication in Astronomy & Computin

An Advanced, Three-Dimensional Plotting Library for Astronomy

We present a new, three-dimensional (3D) plotting library with advanced features, and support for standard and enhanced display devices. The library - S2PLOT - is written in C and can be used by C, C++ and FORTRAN programs on GNU/Linux and Apple/OSX systems. S2PLOT draws objects in a 3D (x,y,z) Cartesian space and the user interactively controls how this space is rendered at run time. With a PGPLOT inspired interface, S2PLOT provides astronomers with elegant techniques for displaying and exploring 3D data sets directly from their program code, and the potential to use stereoscopic and dome display devices. The S2PLOT architecture supports dynamic geometry and can be used to plot time-evolving data sets, such as might be produced by simulation codes. In this paper, we introduce S2PLOT to the astronomical community, describe its potential applications, and present some example uses of the library.Comment: 12 pages, 10 eps figures (higher resolution versions available from http://astronomy.swin.edu.au/s2plot/paperfigures). The S2PLOT library is available for download from http://astronomy.swin.edu.au/s2plo

Adaptive end-to-end optimization of mobile video streaming using QoS negotiation

Video streaming over wireless links is a non-trivial problem due to the large and frequent changes in the quality of the underlying radio channel combined with latency constraints. We believe that every layer in a mobile system must be prepared to adapt its behavior to its environment. Thus layers must be capable of operating in multiple modes; each mode will show a different quality and resource usage. Selecting the right mode of operation requires exchange of information between interacting layers. For example, selecting the best channel coding requires information about the quality of the channel (capacity, bit-error-rate) as well as the requirements (latency, reliability) of the compressed video stream generated by the source encoder. In this paper we study the application of our generic QoS negotiation scheme to a specific configuration for mobile video transmission. We describe the results of experiments studying the overall effectiveness, stability, and dynamics of adaptation of our distributed optimization approach