Interactive Visual Analysis of Networked Systems: Workflows for Two Industrial Domains

We report on a first study of interactive visual analysis of networked systems. Working with ABB Corporate Research and Ericsson Research, we have created workflows which demonstrate the potential of visualization in the domains of industrial automation and telecommunications. By a workflow in this context, we mean a sequence of visualizations and the actions for generating them. Visualizations can be any images that represent properties of the data sets analyzed, and actions typically either change the selection of data visualized or change the visualization by choice of technique or change of parameters

The Infrared Imaging Spectrograph (IRIS) for TMT: Data Reduction System

IRIS (InfraRed Imaging Spectrograph) is the diffraction-limited first light instrument for the Thirty Meter Telescope (TMT) that consists of a near-infrared (0.84 to 2.4 $\mu$m) imager and integral field spectrograph (IFS). The IFS makes use of a lenslet array and slicer for spatial sampling, which will be able to operate in 100's of different modes, including a combination of four plate scales from 4 milliarcseconds (mas) to 50 mas with a large range of filters and gratings. The imager will have a field of view of 34$\times$34 arcsec$^{2}$ with a plate scale of 4 mas with many selectable filters. We present the preliminary design of the data reduction system (DRS) for IRIS that need to address all of these observing modes. Reduction of IRIS data will have unique challenges since it will provide real-time reduction and analysis of the imaging and spectroscopic data during observational sequences, as well as advanced post-processing algorithms. The DRS will support three basic modes of operation of IRIS; reducing data from the imager, the lenslet IFS, and slicer IFS. The DRS will be written in Python, making use of open-source astronomical packages available. In addition to real-time data reduction, the DRS will utilize real-time visualization tools, providing astronomers with up-to-date evaluation of the target acquisition and data quality. The quicklook suite will include visualization tools for 1D, 2D, and 3D raw and reduced images. We discuss the overall requirements of the DRS and visualization tools, as well as necessary calibration data to achieve optimal data quality in order to exploit science cases across all cosmic distance scales.Comment: 13 pages, 2 figures, 6 tables, Proceeding 9913-165 of the SPIE Astronomical Telescopes + Instrumentation 201

PaPaS: A Portable, Lightweight, and Generic Framework for Parallel Parameter Studies

The current landscape of scientific research is widely based on modeling and simulation, typically with complexity in the simulation's flow of execution and parameterization properties. Execution flows are not necessarily straightforward since they may need multiple processing tasks and iterations. Furthermore, parameter and performance studies are common approaches used to characterize a simulation, often requiring traversal of a large parameter space. High-performance computers offer practical resources at the expense of users handling the setup, submission, and management of jobs. This work presents the design of PaPaS, a portable, lightweight, and generic workflow framework for conducting parallel parameter and performance studies. Workflows are defined using parameter files based on keyword-value pairs syntax, thus removing from the user the overhead of creating complex scripts to manage the workflow. A parameter set consists of any combination of environment variables, files, partial file contents, and command line arguments. PaPaS is being developed in Python 3 with support for distributed parallelization using SSH, batch systems, and C++ MPI. The PaPaS framework will run as user processes, and can be used in single/multi-node and multi-tenant computing systems. An example simulation using the BehaviorSpace tool from NetLogo and a matrix multiply using OpenMP are presented as parameter and performance studies, respectively. The results demonstrate that the PaPaS framework offers a simple method for defining and managing parameter studies, while increasing resource utilization.Comment: 8 pages, 6 figures, PEARC '18: Practice and Experience in Advanced Research Computing, July 22--26, 2018, Pittsburgh, PA, US

Progressive Analytics: A Computation Paradigm for Exploratory Data Analysis

Exploring data requires a fast feedback loop from the analyst to the system, with a latency below about 10 seconds because of human cognitive limitations. When data becomes large or analysis becomes complex, sequential computations can no longer be completed in a few seconds and data exploration is severely hampered. This article describes a novel computation paradigm called Progressive Computation for Data Analysis or more concisely Progressive Analytics, that brings at the programming language level a low-latency guarantee by performing computations in a progressive fashion. Moving this progressive computation at the language level relieves the programmer of exploratory data analysis systems from implementing the whole analytics pipeline in a progressive way from scratch, streamlining the implementation of scalable exploratory data analysis systems. This article describes the new paradigm through a prototype implementation called ProgressiVis, and explains the requirements it implies through examples.Comment: 10 page

Neuroimaging study designs, computational analyses and data provenance using the LONI pipeline.

Modern computational neuroscience employs diverse software tools and multidisciplinary expertise to analyze heterogeneous brain data. The classical problems of gathering meaningful data, fitting specific models, and discovering appropriate analysis and visualization tools give way to a new class of computational challenges--management of large and incongruous data, integration and interoperability of computational resources, and data provenance. We designed, implemented and validated a new paradigm for addressing these challenges in the neuroimaging field. Our solution is based on the LONI Pipeline environment [3], [4], a graphical workflow environment for constructing and executing complex data processing protocols. We developed study-design, database and visual language programming functionalities within the LONI Pipeline that enable the construction of complete, elaborate and robust graphical workflows for analyzing neuroimaging and other data. These workflows facilitate open sharing and communication of data and metadata, concrete processing protocols, result validation, and study replication among different investigators and research groups. The LONI Pipeline features include distributed grid-enabled infrastructure, virtualized execution environment, efficient integration, data provenance, validation and distribution of new computational tools, automated data format conversion, and an intuitive graphical user interface. We demonstrate the new LONI Pipeline features using large scale neuroimaging studies based on data from the International Consortium for Brain Mapping [5] and the Alzheimer's Disease Neuroimaging Initiative [6]. User guides, forums, instructions and downloads of the LONI Pipeline environment are available at http://pipeline.loni.ucla.edu