The Data Big Bang and the Expanding Digital Universe: High-Dimensional, Complex and Massive Data Sets in an Inflationary Epoch

Recent and forthcoming advances in instrumentation, and giant new surveys, are creating astronomical data sets that are not amenable to the methods of analysis familiar to astronomers. Traditional methods are often inadequate not merely because of the size in bytes of the data sets, but also because of the complexity of modern data sets. Mathematical limitations of familiar algorithms and techniques in dealing with such data sets create a critical need for new paradigms for the representation, analysis and scientific visualization (as opposed to illustrative visualization) of heterogeneous, multiresolution data across application domains. Some of the problems presented by the new data sets have been addressed by other disciplines such as applied mathematics, statistics and machine learning and have been utilized by other sciences such as space-based geosciences. Unfortunately, valuable results pertaining to these problems are mostly to be found only in publications outside of astronomy. Here we offer brief overviews of a number of concepts, techniques and developments, some "old" and some new. These are generally unknown to most of the astronomical community, but are vital to the analysis and visualization of complex datasets and images. In order for astronomers to take advantage of the richness and complexity of the new era of data, and to be able to identify, adopt, and apply new solutions, the astronomical community needs a certain degree of awareness and understanding of the new concepts. One of the goals of this paper is to help bridge the gap between applied mathematics, artificial intelligence and computer science on the one side and astronomy on the other.Comment: 24 pages, 8 Figures, 1 Table. Accepted for publication: "Advances in Astronomy, special issue "Robotic Astronomy

Grid Integration of Robotic Telescopes

Robotic telescopes and grid technology have made significant progress in recent years. Both innovations offer important advantages over conventional technologies, particularly in combination with one another. Here, we introduce robotic telescopes used by the Astrophysical Institute Potsdam as ideal instruments for building a robotic telescope network. We also discuss the grid architecture and protocols facilitating the network integration that is being developed by the German AstroGrid-D project. Finally, we present three user interfaces employed for this purpose.Comment: 4 pages, 5 Figures, refereed proceedings of "Hot-wiring the Transient Universe", June 2007 (Tucson); version 2 including latex geometry package as recommended by arXiv and minor changes as requested by AN except removal of two figure

Interactive Visualization of the Largest Radioastronomy Cubes

3D visualization is an important data analysis and knowledge discovery tool, however, interactive visualization of large 3D astronomical datasets poses a challenge for many existing data visualization packages. We present a solution to interactively visualize larger-than-memory 3D astronomical data cubes by utilizing a heterogeneous cluster of CPUs and GPUs. The system partitions the data volume into smaller sub-volumes that are distributed over the rendering workstations. A GPU-based ray casting volume rendering is performed to generate images for each sub-volume, which are composited to generate the whole volume output, and returned to the user. Datasets including the HI Parkes All Sky Survey (HIPASS - 12 GB) southern sky and the Galactic All Sky Survey (GASS - 26 GB) data cubes were used to demonstrate our framework's performance. The framework can render the GASS data cube with a maximum render time < 0.3 second with 1024 x 1024 pixels output resolution using 3 rendering workstations and 8 GPUs. Our framework will scale to visualize larger datasets, even of Terabyte order, if proper hardware infrastructure is available.Comment: 15 pages, 12 figures, Accepted New Astronomy July 201

Processing Color in Astronomical Imagery

Every year, hundreds of images from telescopes on the ground and in space are released to the public, making their way into popular culture through everything from computer screens to postage stamps. These images span the entire electromagnetic spectrum from radio waves to infrared light to X-rays and gamma rays, a majority of which is undetectable to the human eye without technology. Once these data are collected, one or more specialists must process the data to create an image. Therefore, the creation of astronomical imagery involves a series of choices. How do these choices affect the comprehension of the science behind the images? What is the best way to represent data to a non-expert? Should these choices be based on aesthetics, scientific veracity, or is it possible to satisfy both? This paper reviews just one choice out of the many made by astronomical image processors: color. The choice of color is one of the most fundamental when creating an image taken with modern telescopes. We briefly explore the concept of the image as translation, particularly in the case of astronomical images from invisible portions of the electromagnetic spectrum. After placing modern astronomical imagery and photography in general in the context of its historical beginnings, we review the standards (or lack thereof) in making the basic choice of color. We discuss the possible implications for selecting one color palette over another in the context of the appropriateness of using these images as science communication products with a specific focus on how the non-expert perceives these images and how that affects their trust in science. Finally, we share new data sets that begin to look at these issues in scholarly research and discuss the need for a more robust examination of this and other related topics in the future to better understand the implications for science communications.Comment: 10 pages, 6 figures, published in Studies in Media and Communicatio

Data Mining and Machine Learning in Astronomy

We review the current state of data mining and machine learning in astronomy. 'Data Mining' can have a somewhat mixed connotation from the point of view of a researcher in this field. If used correctly, it can be a powerful approach, holding the potential to fully exploit the exponentially increasing amount of available data, promising great scientific advance. However, if misused, it can be little more than the black-box application of complex computing algorithms that may give little physical insight, and provide questionable results. Here, we give an overview of the entire data mining process, from data collection through to the interpretation of results. We cover common machine learning algorithms, such as artificial neural networks and support vector machines, applications from a broad range of astronomy, emphasizing those where data mining techniques directly resulted in improved science, and important current and future directions, including probability density functions, parallel algorithms, petascale computing, and the time domain. We conclude that, so long as one carefully selects an appropriate algorithm, and is guided by the astronomical problem at hand, data mining can be very much the powerful tool, and not the questionable black box.Comment: Published in IJMPD. 61 pages, uses ws-ijmpd.cls. Several extra figures, some minor additions to the tex