Introducing distributed dynamic data-intensive (D3) science: Understanding applications and infrastructure

A common feature across many science and engineering applications is the amount and diversity of data and computation that must be integrated to yield insights. Data sets are growing larger and becoming distributed; and their location, availability and properties are often time-dependent. Collectively, these characteristics give rise to dynamic distributed data-intensive applications. While "static" data applications have received significant attention, the characteristics, requirements, and software systems for the analysis of large volumes of dynamic, distributed data, and data-intensive applications have received relatively less attention. This paper surveys several representative dynamic distributed data-intensive application scenarios, provides a common conceptual framework to understand them, and examines the infrastructure used in support of applications.Comment: 38 pages, 2 figure

Impact of Radio Frequency Interference and Real-Time Spectral Kurtosis Mitigation

We catalog the ubiquity of Radio Frequency Interference (RFI) plaguing every modern radio telescope and investigate several ways to mitigate it in order to create better science-ready data products for astronomers. There are a myriad of possible RFI sources, including satellite uplinks and downlinks, cellular communications, air traffic radar, and natural sources such as lightning. Real-time RFI mitigation strategies must take these RFI characteristics into account, as the interfering signals can look significantly different at very high time and frequency resolutions. We examine Spectral Kurtosis (SK) as a real-time statistical RFI detection method, and compare its flagging efficacy against simulated RFI witha wide range of signal characteristics. We found to be weak against signals with a 50% effective duty cycle, as well as low signal-to-noise ratio sidelobe spillover from strong and frequency-wide RFI. Coarsening the SK time resolution improved flagging, as did using multi-scale SK, which averages adjacent time-frequency pixels with small rolling windows to circumvent the weakness to 50% duty cycle signals. Multiscale SK raised flagging above 90% for almost all cases, and as long as the amount of channels included in the multi-scale window wasn’t wider than the RFI signal, there was no significant increase in false positive rate. Simulated realistic incoherent astronomical signals were not detected by SK at all, as expected. To simulate real-time SK RFI detection in a real data set, raw, unaveraged data was taken with the Robert C. Byrd Green Bank Telescope (GBT). The observation targets included one pulsar, two neutral hydrogen (HI) galaxies, the Milky Way HI emission, and a hydroxyl megamaser. These targets are all easily observable on short timescales but are also nearby several sources of RFI. Flagged data was replaced with representative Gaussian noise using the statistics of adjacent time-frequency pixels. We run different variations of SK detection on copies of the raw datasets and compared to the original, to see how well the RFI was removed and if the science data product was affected in any way. The spectral line targets are all completely ignored by SK , while the pulsar results decreased in quality due to the noise replacement averaging over the time variable structure, unless care was taken to flag data on timescales shorter than the pulse length. In these cases, single pulse signal-to-noise ratio was marginally improved

When Is an Enterprise Service Bus (Esb) the Right Choice for an Integrated Technology Solution?

The Enterprise Service Bus (ESB) is an important systems integration technology often closely associated with Service Oriented Architecture (SOA). Some maintain that an ESB should not be used apart from SOA. Others see the ESB simply as the next generation of middleware, incorporating the best of its predecessors, Enterprise Application Integration (EAI) and Message Oriented Middleware (MOM), and a candidate for any integration requirement. Is the ESB a one-size-fits-all solution to be trusted for any integration requirement, or must its use be carefully considered with proper due diligence based on application complexity and/or the presence or absence of a defined SOA? This thesis probes these questions in an analysis of a world-wide survey of 230 industry SOA and middleware professionals conducted via the LinkedIn Professional Network during a six week period in November and December of 2010. In addition, the thesis applies a review of the survey results and current SOA and ESB literature to an architectural decision being made within the Systems Engineering and Application Development (SEAD) Practicum in the Master of Science program in Computer Information Systems at Regis University in Denver, which provides support for the University\u27s Academic Research Network (ARN). An ESB has been proposed as a new architectural component for the ARN infrastructure and this paper reviews the merit of this proposal. This thesis employs an interpretivist epistemology, understanding that there may be more than one acceptable answer to the question, When is an Enterprise Service Bus an appropriate component of an integrated technology solution

High Energy Astrophysics Research and Programmatic Support

This report reviews activities performed by members of the USRA contract team during the six months of the reporting period and projected activities during the coming six months. Activities take place at the Goddard Space Flight Center, within the Laboratory for High Energy Astrophysics. Developments concern instrumentation, observation, data analysis, and theoretical work in Astrophysics. Missions supported include: Advanced Satellite for Cosmology and Astrophysics (ASCA), X-ray Timing Experiment (XTE), X-ray Spectrometer (XRS), Astro-E, High Energy Astrophysics Science Archive Research Center (HEASARC), and others

Galaxy Zoo: Mergers – Dynamical models of interacting galaxies

The dynamical history of most merging galaxies is not well understood. Correlations between galaxy interaction and star formation have been found in previous studies, but require the context of the physical history of merging systems for full insight into the processes that lead to enhanced star formation. We present the results of simulations that reconstruct the orbit trajectories and disturbed morphologies of pairs of interacting galaxies. With the use of a restricted three-body simulation code and the help of citizen scientists, we sample 105 points in parameter space for each system. We demonstrate a successful recreation of the morphologies of 62 pairs of interacting galaxies through the review of more than 3 million simulations. We examine the level of convergence and uniqueness of the dynamical properties of each system. These simulations represent the largest collection of models of interacting galaxies to date, providing a valuable resource for the investigation of mergers. This paper presents the simulation parameters generated by the project. They are now publicly available in electronic format at http://data.galaxyzoo.org/mergers.html. Though our best-fitting model parameters are not an exact match to previously published models, our method for determining uncertainty measurements will aid future comparisons between models. The dynamical clocks from our models agree with previous results of the time since the onset of star formation from starburst models in interacting systems and suggest that tidally induced star formation is triggered very soon after closest approach

Investigating strong gravitational lensing with infrared space missions : AKARA, Spitzer and Herschel

This thesis addresses the use of strong gravitational lensing to facilitate deep midinfrared observations of star-forming galaxies and to study highly dust obscured sub-millimetre galaxies (SMG). 15 um extragalactic number counts were taken from ultra deep AKARI mapping of the gravitational lensing cluster Abell 2218, which is the deepest image taken by any facility at this wavelength. Via strong gravitational lensing these data probe beyond the AKARI blank field confusion limit. By de-magnifying the extracted source catalogue and performing careful photometric de-blending, using multi-wavelength positional priors, galaxy number counts down to ~0.01 mJy were achieved. These counts are ~3x deeper than previous results and resolve 70-100% of the cosmic infrared background at 15/.Lm,giving a new stronger lower limit of 2.0±0.3 nWm-2 Hz-1. These data are sampling the normal star forming population that dominates the peak epoch of star formation. Stacking analysis of the AKARI 15 um source positions at Herschel/SPIRE wavelengths, show that a deep 15 um galaxy population resolves around 40% of the 250 um cosmic infrared background, but is less representative of galaxy populations sampled at longer wavelengths, where the background is dominated by sub-mm galaxies. This thesis also focuses on strong galaxy-galaxy lensing events and the decoupling of the lens and source photometry in order to estimate redshifts and constrain physical characteristics. The first sample of bright sub-mm gravitational lenses, selected at SPIRE 500 um by the Herschel Astrophysical Terahertz Large Area Survey, are investigated with light profile and SED fitting, to derive physical characteristics. For two of the five lenses, it was possible to disentangle the lens and background galaxy components in highly photometrically blended Spitzer data, after observations by the Submillimeter Array revealed the lensed structure. The lensed background galaxies are highly dust obscured SMG with intrinsic infrared luminosities -2

Arquitectura, técnicas y modelos para posibilitar la Ciencia de Datos en el Archivo de la Misión Gaia

Tesis inédita de la Universidad Complutense de Madrid, Facultad de Informática, Departamento de Arquitectura de Computadores y Automática, leída el 26/05/2017.The massive amounts of data that the world produces every day pose new challenges to modern societies in terms of how to leverage their inherent value. Social networks, instant messaging, video, smart devices and scientific missions are just mere examples of the vast number of sources generating data every second. As the world becomes more and more digitalized, new needs arise for organizing, archiving, sharing, analyzing, visualizing and protecting the ever-increasing data sets, so that we can truly develop into a data-driven economy that reduces inefficiencies and increases sustainability, creating new business opportunities on the way. Traditional approaches for harnessing data are not suitable any more as they lack the means for scaling to the larger volumes in a timely and cost efficient manner. This has somehow changed with the advent of Internet companies like Google and Facebook, which have devised new ways of tackling this issue. However, the variety and complexity of the value chains in the private sector as well as the increasing demands and constraints in which the public one operates, needs an ongoing research that can yield newer strategies for dealing with data, facilitate the integration of providers and consumers of information, and guarantee a smooth and prompt transition when adopting these cutting-edge technological advances. This thesis aims at providing novel architectures and techniques that will help perform this transition towards Big Data in massive scientific archives. It highlights the common pitfalls that must be faced when embracing it and how to overcome them, especially when the data sets, their transformation pipelines and the tools used for the analysis are already present in the organizations. Furthermore, a new perspective for facilitating a smoother transition is laid out. It involves the usage of higher-level and use case specific frameworks and models, which will naturally bridge the gap between the technological and scientific domains. This alternative will effectively widen the possibilities of scientific archives and therefore will contribute to the reduction of the time to science. The research will be applied to the European Space Agency cornerstone mission Gaia, whose final data archive will represent a tremendous discovery potential. It will create the largest and most precise three dimensional chart of our galaxy (the Milky Way), providing unprecedented position, parallax and proper motion measurements for about one billion stars. The successful exploitation of this data archive will depend to a large degree on the ability to offer the proper architecture, i.e. infrastructure and middleware, upon which scientists will be able to do exploration and modeling with this huge data set. In consequence, the approach taken needs to enable data fusion with other scientific archives, as this will produce the synergies leading to an increment in scientific outcome, both in volume and in quality. The set of novel techniques and frameworks presented in this work addresses these issues by contextualizing them with the data products that will be generated in the Gaia mission. All these considerations have led to the foundations of the architecture that will be leveraged by the Science Enabling Applications Work Package. Last but not least, the effectiveness of the proposed solution will be demonstrated through the implementation of some ambitious statistical problems that will require significant computational capabilities, and which will use Gaia-like simulated data (the first Gaia data release has recently taken place on September 14th, 2016). These ambitious problems will be referred to as the Grand Challenge, a somewhat grandiloquent name that consists in inferring a set of parameters from a probabilistic point of view for the Initial Mass Function (IMF) and Star Formation Rate (SFR) of a given set of stars (with a huge sample size), from noisy estimates of their masses and ages respectively. This will be achieved by using Hierarchical Bayesian Modeling (HBM). In principle, the HBM can incorporate stellar evolution models to infer the IMF and SFR directly, but in this first step presented in this thesis, we will start with a somewhat less ambitious goal: inferring the PDMF and PDAD. Moreover, the performance and scalability analyses carried out will also prove the suitability of the models for the large amounts of data that will be available in the Gaia data archive.Las grandes cantidades de datos que se producen en el mundo diariamente plantean nuevos retos a la sociedad en términos de cómo extraer su valor inherente. Las redes sociales, mensajería instantánea, los dispositivos inteligentes y las misiones científicas son meros ejemplos del gran número de fuentes generando datos en cada momento. Al mismo tiempo que el mundo se digitaliza cada vez más, aparecen nuevas necesidades para organizar, archivar, compartir, analizar, visualizar y proteger la creciente cantidad de datos, para que podamos desarrollar economías basadas en datos e información que sean capaces de reducir las ineficiencias e incrementar la sostenibilidad, creando nuevas oportunidades de negocio por el camino. La forma en la que se han manejado los datos tradicionalmente no es la adecuada hoy en día, ya que carece de los medios para escalar a los volúmenes más grandes de datos de una forma oportuna y eficiente. Esto ha cambiado de alguna manera con la llegada de compañías que operan en Internet como Google o Facebook, ya que han concebido nuevas aproximaciones para abordar el problema. Sin embargo, la variedad y complejidad de las cadenas de valor en el sector privado y las crecientes demandas y limitaciones en las que el sector público opera, necesitan una investigación continua en la materia que pueda proporcionar nuevas estrategias para procesar las enormes cantidades de datos, facilitar la integración de productores y consumidores de información, y garantizar una transición rápida y fluida a la hora de adoptar estos avances tecnológicos innovadores. Esta tesis tiene como objetivo proporcionar nuevas arquitecturas y técnicas que ayudarán a realizar esta transición hacia Big Data en archivos científicos masivos. La investigación destaca los escollos principales a encarar cuando se adoptan estas nuevas tecnologías y cómo afrontarlos, principalmente cuando los datos y las herramientas de transformación utilizadas en el análisis existen en la organización. Además, se exponen nuevas medidas para facilitar una transición más fluida. Éstas incluyen la utilización de software de alto nivel y específico al caso de uso en cuestión, que haga de puente entre el dominio científico y tecnológico. Esta alternativa ampliará de una forma efectiva las posibilidades de los archivos científicos y por tanto contribuirá a la reducción del tiempo necesario para generar resultados científicos a partir de los datos recogidos en las misiones de astronomía espacial y planetaria. La investigación se aplicará a la misión de la Agencia Espacial Europea (ESA) Gaia, cuyo archivo final de datos presentará un gran potencial para el descubrimiento y hallazgo desde el punto de vista científico. La misión creará el catálogo en tres dimensiones más grande y preciso de nuestra galaxia (la Vía Láctea), proporcionando medidas sin precedente acerca del posicionamiento, paralaje y movimiento propio de alrededor de mil millones de estrellas. Las oportunidades para la explotación exitosa de este archivo de datos dependerán en gran medida de la capacidad de ofrecer la arquitectura adecuada, es decir infraestructura y servicios, sobre la cual los científicos puedan realizar la exploración y modelado con esta inmensa cantidad de datos. Por tanto, la estrategia a realizar debe ser capaz de combinar los datos con otros archivos científicos, ya que esto producirá sinergias que contribuirán a un incremento en la ciencia producida, tanto en volumen como en calidad de la misma. El conjunto de técnicas e infraestructuras innovadoras presentadas en este trabajo aborda estos problemas, contextualizándolos con los productos de datos que se generarán en la misión Gaia. Todas estas consideraciones han conducido a los fundamentos de la arquitectura que se utilizará en el paquete de trabajo de aplicaciones que posibilitarán la ciencia en el archivo de la misión Gaia (Science Enabling Applications). Por último, la eficacia de la solución propuesta se demostrará a través de la implementación de dos problemas estadísticos que requerirán cantidades significativas de cómputo, y que usarán datos simulados en el mismo formato en el que se producirán en el archivo de la misión Gaia (la primera versión de datos recogidos por la misión está disponible desde el día 14 de Septiembre de 2016). Estos ambiciosos problemas representan el Gran Reto (Grand Challenge), un nombre grandilocuente que consiste en inferir una serie de parámetros desde un punto de vista probabilístico para la función de masa inicial (Initial Mass Function) y la tasa de formación estelar (Star Formation Rate) dado un conjunto de estrellas (con una muestra grande), desde estimaciones con ruido de sus masas y edades respectivamente. Esto se abordará utilizando modelos jerárquicos bayesianos (Hierarchical Bayesian Modeling). Enprincipio,losmodelospropuestos pueden incorporar otros modelos de evolución estelar para inferir directamente la función de masa inicial y la tasa de formación estelar, pero en este primer paso presentado en esta tesis, empezaremos con un objetivo algo menos ambicioso: la inferencia de la función de masa y distribución de edades actual (Present-Day Mass Function y Present-Day Age Distribution respectivamente). Además, se llevará a cabo el análisis de rendimiento y escalabilidad para probar la idoneidad de la implementación de dichos modelos dadas las enormes cantidades de datos que estarán disponibles en el archivo de la misión Gaia...Depto. de Arquitectura de Computadores y AutomáticaFac. de InformáticaTRUEunpu

Doctor of Philosophy

dissertationStochastic methods, dense free-form mapping, atlas construction, and total variation are examples of advanced image processing techniques which are robust but computationally demanding. These algorithms often require a large amount of computational power as well as massive memory bandwidth. These requirements used to be ful lled only by supercomputers. The development of heterogeneous parallel subsystems and computation-specialized devices such as Graphic Processing Units (GPUs) has brought the requisite power to commodity hardware, opening up opportunities for scientists to experiment and evaluate the in uence of these techniques on their research and practical applications. However, harnessing the processing power from modern hardware is challenging. The di fferences between multicore parallel processing systems and conventional models are signi ficant, often requiring algorithms and data structures to be redesigned signi ficantly for efficiency. It also demands in-depth knowledge about modern hardware architectures to optimize these implementations, sometimes on a per-architecture basis. The goal of this dissertation is to introduce a solution for this problem based on a 3D image processing framework, using high performance APIs at the core level to utilize parallel processing power of the GPUs. The design of the framework facilitates an efficient application development process, which does not require scientists to have extensive knowledge about GPU systems, and encourages them to harness this power to solve their computationally challenging problems. To present the development of this framework, four main problems are described, and the solutions are discussed and evaluated: (1) essential components of a general 3D image processing library: data structures and algorithms, as well as how to implement these building blocks on the GPU architecture for optimal performance; (2) an implementation of unbiased atlas construction algorithms|an illustration of how to solve a highly complex and computationally expensive algorithm using this framework; (3) an extension of the framework to account for geometry descriptors to solve registration challenges with large scale shape changes and high intensity-contrast di fferences; and (4) an out-of-core streaming model, which enables developers to implement multi-image processing techniques on commodity hardware

A scalable packetised radio astronomy imager

Includes bibliographical referencesModern radio astronomy telescopes the world over require digital back-ends. The complexity of these systems depends on many site-specific factors, including the number of antennas, beams and frequency channels and the bandwidth to be processed. With the increasing popularity for ever larger interferometric arrays, the processing requirements for these back-ends have increased significantly. While the techniques for building these back-ends are well understood, every installation typically still takes many years to develop as the instruments use highly specialised, custom hardware in order to cope with the demanding engineering requirements. Modern technology has enabled reprogrammable FPGA-based processing boards, together with packet-based switching techniques, to perform all the digital signal processing requirements of a modern radio telescope array. The various instruments used by radio telescopes are functionally very different, but the component operations remain remarkably similar and many share core functionalities. Generic processing platforms are thus able to share signal processing libraries and can acquire different personalities to perform different functions simply by reprogramming them and rerouting the data appropriately. Furthermore, Ethernet-based packet-switched networks are highly flexible and scalable, enabling the same instrument design to be scaled to larger installations simply by adding additional processing nodes and larger network switches. The ability of a packetised network to transfer data to arbitrary processing nodes, along with these nodes' reconfigurability, allows for unrestrained partitioning of designs and resource allocation. This thesis describes the design and construction of the first working radio astronomy imaging instrument hosted on Ethernet-interconnected re- programmable FPGA hardware. I attempt to establish an optimal packetised architecture for the most popular instruments with particular attention to the core array functions of correlation and beamforming. Emphasis is placed on requirements for South Africa's MeerKAT array. A demonstration system is constructed and deployed on the KAT-7 array, MeerKAT's prototype. This research promises reduced instrument development time, lower costs, improved reliability and closer collaboration between telescope design teams