Fine-grained visualization pipelines and lazy functional languages

The pipeline model in visualization has evolved from a conceptual model of data processing into a widely used architecture for implementing visualization systems. In the process, a number of capabilities have been introduced, including streaming of data in chunks, distributed pipelines, and demand-driven processing. Visualization systems have invariably built on stateful programming technologies, and these capabilities have had to be implemented explicitly within the lower layers of a complex hierarchy of services. The good news for developers is that applications built on top of this hierarchy can access these capabilities without concern for how they are implemented. The bad news is that by freezing capabilities into low-level services expressive power and flexibility is lost. In this paper we express visualization systems in a programming language that more naturally supports this kind of processing model. Lazy functional languages support fine-grained demand-driven processing, a natural form of streaming, and pipeline-like function composition for assembling applications. The technology thus appears well suited to visualization applications. Using surface extraction algorithms as illustrative examples, and the lazy functional language Haskell, we argue the benefits of clear and concise expression combined with fine-grained, demand-driven computation. Just as visualization provides insight into data, functional abstraction provides new insight into visualization

Distributed modular RT-systems for detector DAQ, trigger and control applications

A modular approach to development of distributed modular system architecture for detector control, data acquisition and trigger data processing is proposed. A multilevel parallel-pipeline model of data acquisition, processing and control is proposed and discussed. Multiprocessor architecture with SCI-based interconnections is proposed as good high-performance system for parallel-pipeline data processing. A network (Ethernet -100) can be used for loading, monitoring and diagnostic purposes independent of basic interconnections. The modular cPCI-based structures with high speed modular interconnections are proposed for DAQ and control applications. For distributed control RT-systems, to construct the effective (cost-performance) systems the same platform of an Intel compatible processor board should be used. The basic computer multiprocessor nodes consist of high-power PC MB (Industrial Computer Systems), which are interconnected by SCI modules and link to embedded microprocessor-based sub-systems for control applications. The required number of multiprocessor nodes should be interconnected by SCI for parallel-pipeline data processing in real time (according to the multilevel model) and link to RT-systems for embedded control. (19 refs)

Bootstrapping Deep Neural Networks from Approximate Image Processing Pipelines

Complex image processing and computer vision systems often consist of a processing pipeline of functional modules. We intend to replace parts or all of a target pipeline with deep neural networks to achieve benefits such as increased accuracy or reduced computational requirement. To acquire a large amount of labeled data necessary to train the deep neural network, we propose a workflow that leverages the target pipeline to create a significantly larger labeled training set automatically, without prior domain knowledge of the target pipeline. We show experimentally that despite the noise introduced by automated labeling and only using a very small initially labeled data set, the trained deep neural networks can achieve similar or even better performance than the components they replace, while in some cases also reducing computational requirements.Comment: 6 pages, 5 figure

The VolumePro Volume Rendering Cluster: A Vital Component of Parallel End-to-End Solution

As data sets, both acquired from scanners and those generated from complex simulations, grow in size and complexity, researchers continue to push the boundaries of the amount of data that can be viewed, processed and analyzed interactively. It soon becomes clear that today's algorithms and systems must be scaled through the use of parallelism in order to create viable and cost effective solutions. But for parallelism to be most effective, it must be conceived and designed for efficiency through-out every individual component of the processing and computational pipeline. In fact, a parallel architecture must be preserved throughout the whole process in order to avoid single pipeline bottlenecks

PageRank Pipeline Benchmark: Proposal for a Holistic System Benchmark for Big-Data Platforms

The rise of big data systems has created a need for benchmarks to measure and compare the capabilities of these systems. Big data benchmarks present unique scalability challenges. The supercomputing community has wrestled with these challenges for decades and developed methodologies for creating rigorous scalable benchmarks (e.g., HPC Challenge). The proposed PageRank pipeline benchmark employs supercomputing benchmarking methodologies to create a scalable benchmark that is reflective of many real-world big data processing systems. The PageRank pipeline benchmark builds on existing prior scalable benchmarks (Graph500, Sort, and PageRank) to create a holistic benchmark with multiple integrated kernels that can be run together or independently. Each kernel is well defined mathematically and can be implemented in any programming environment. The linear algebraic nature of PageRank makes it well suited to being implemented using the GraphBLAS standard. The computations are simple enough that performance predictions can be made based on simple computing hardware models. The surrounding kernels provide the context for each kernel that allows rigorous definition of both the input and the output for each kernel. Furthermore, since the proposed PageRank pipeline benchmark is scalable in both problem size and hardware, it can be used to measure and quantitatively compare a wide range of present day and future systems. Serial implementations in C++, Python, Python with Pandas, Matlab, Octave, and Julia have been implemented and their single threaded performance has been measured.Comment: 9 pages, 7 figures, to appear in IPDPS 2016 Graph Algorithms Building Blocks (GABB) worksho