A Computational Database System for Generating Unstructured Hexahedral Meshes with Billions of Elements

For a large class of physical simulations with relatively simple geometries, unstructured octree-based hexahedral meshes provide a good compromise between adaptivity and simplicity. However, generating unstructured hexahedral meshes with over 1 billion elements remains a challenging task. We propose a database approach to solve this problem. Instead of merely storing generated meshes into conventional databases, we have developed a new kind of software system called Computational Database System (CDS) to generate meshes directly on databases. Our basic idea is to extend existing database techniques to organize and index mesh data, and use database-aware algorithms to manipulate database structures and generate meshes. This paper presents the design, implementation, and evaluation of a prototype CDS named Weaver, which has been used successfully by the CMU Quake project to generate queryable high-resolution finite element meshes for earthquake simulations with up to 1.22B elements and 1.37B nodes

Performance Issues in High Performance Fortran Implementations of Sensor-Based Applications

Applications that get their inputs from sensors are an important and often overlooked application domain for High Performance Fortran (HPF). Such sensor-based applications typically perform regular operations on dense arrays, and often have latency and through put requirements that can only be achieved with parallel machines. This article describes a study of sensor-based applications, including the fast Fourier transform, synthetic aperture radar imaging, narrowband tracking radar processing, multibaseline stereo imaging, and medical magnetic resonance imaging. The applications are written in a dialect of HPF developed at Carnegie Mellon, and are compiled by the Fx compiler for the Intel Paragon. The main results of the study are that (1) it is possible to realize good performance for realistic sensor-based applications written in HPF and (2) the performance of the applications is determined by the performance of three core operations: independent loops (i.e., loops with no dependences between iterations), reductions, and index permutations. The article discusses the implications for HPF implementations and introduces some simple tests that implementers and users can use to measure the efficiency of the loops, reductions, and index permutations generated by an HPF compiler

Computer systems: a programmer's perspective

Computer systems: A Programmer’s Perspective explains the underlying elements common among all computer systems and how they affect general application performance. Written from the programmer’s perspective, this book strives to teach readers how understanding basic elements of computer systems and executing real practice can lead them to create better programs. Spanning across computer science themes such as hardware architecture, the operating system, and systems software, the Third Edition serves as a comprehensive introduction to programming. This book strives to create programmers who understand all elements of computer systems and will be able to engage in any application of the field--from fixing faulty software, to writing more capable programs, to avoiding common flaws. It lays the groundwork for readers to delve into more intensive topics such as computer architecture, embedded systems, and cybersecurity. This book focuses on systems that execute an x86-64 machine code, and recommends that programmers have access to a Linux system for this course. Programmers should have basic familiarity with C or C++

Extracting Hexahedral Mesh Structures from Balanced Linear Octrees

Generating large 3D unstructured meshes with over 1 billion elements has been a challenging task. Fortunately, for a large class of applications with relatively simple geometries, unstructured octree-based hexahedral meshes provide a good compromise between adaptivity and simplicity. This paper presents our recent work on how to extract hexahedral mesh structures from a class of database structures known as balanced linear octrees. The proposed technique is not memory bound and is capable of extracting mesh structures with billions of elements and nodes, provided there is enough disk space to store the mesh. In practice, our new algorithm runs about 11 times faster than a conventional database search-based algorithm and uses only 10% of the storage space

Balance Refinement of Massive Linear Octree Datasets

Many applications that use octrees require that the octree decomposition be smooth throughout the domain with no sharp change in size between spatially adjacent octants, thus impose a so-called 2-to-1 constraint on the octree datasets. The process of enforcing the 2-to-1 constraint on an existing octree dataset is called balance refinement. Although it is relatively easy to conduct balance refinement on memory-resident octree datasets, it represents a major challenge when massive linear octree datasets are involved. Different from other massive data problems, the balance refinement problem is characterized not only by the sheer volume of data, but also by the intricacy of the 2-to-1 constraint. Our solution consists of two major algorithms: balance by parts and prioritized ripple propagation. The key idea is to bulk load most of the data into memory only once and enforce the 2-to-1 constraint locally using sophisticated data structure built on the fly. The software package we developed has successfully balanced world-record linear octree datasets that are used by real-world supercomputing applications

Evaluation of a Resource Selection Mechanism for Complex Network Services

Providing complex (resource-intensive) network services is challenging because the resources they need and the resources that are available can vary significantly from request to request. To address this issue, we have proposed a flexible mechanism, called active frames, that provides a basis for selecting a set of available distributed computing resources, and then mapping tasks onto those resources. As a proof of concept, we have used active frames to build a remote visualization service, called Dv, that allows users to visualize the contents of scientific datasets stored at remote locations. We evaluate the performance of active frames, in the context of Dv. In particular, we address the following two questions: (1) what performance penalty do we pay for the flexibility of the active frames mechanism? (2) can the throughput of a service based on active frames be predicted with reasonable accuracy from micro-benchmarks? The results of the evaluation suggest that the overhead imposed by active frames is reasonable (roughly 5%), and that simple models based on micro-benchmarks can conservatively predict measured throughput with reasonable accuracy (at most 20%

Computer systems a programmer's perspective

978 p. ; 24 cm

Introducing Computer Systems from a Programmer’s Perspective

The course “Introduction to Computer Systems” at Carnegie Mellon University presents the underlying principles by which programs are executed on a computer. It provides broad coverage of processor operation, compilers, operating systems, and networking. Whereas most systems courses present material from the perspective of one who designs or implements part of the system, our course presents the view visible to application programmers. Students learn that, by understanding aspects of the underlying system, they can make their programs faster and more reliable. This approach provides immediate benefits for all computer science and engineering students and also prepares them for more advanced systems courses. We have taught our course for five semesters with enthusiastic responses by the students, the instructors, and the instructors of subsequent systems courses

The Etree Library: A System for Manipulating Large Octrees on Disk

This report describes a library, called the etree library, that allows C programmers to manipulate large octrees stored on disk. Octrees are stored as a sequence of fixed sized octant records sorted by a locational code order that is equivalent to a preorder traversal of the tree and a Z-order traversal through the domain. The sorted records are indexed by a conventional file-resident B-tree index and queried using fixed-length locational code keys. A schema can be defined to make an etree portable across different platforms. The etree library provides functions for creating, modifying, and searching octrees, including efficient mechanisms for appending octants and iterating over octants in Z-order. The library is the foundation for a larger research effort aimed at enabling scientists and engineers to solve large physical simulations on their desktop systems by recasting the simulation process to work directly on large etrees stored on disk