Real-Time Data Processing With Lambda Architecture

Data has evolved immensely in recent years, in type, volume and velocity. There are several frameworks to handle the big data applications. The project focuses on the Lambda Architecture proposed by Marz and its application to obtain real-time data processing. The architecture is a solution that unites the benefits of the batch and stream processing techniques. Data can be historically processed with high precision and involved algorithms without loss of short-term information, alerts and insights. Lambda Architecture has an ability to serve a wide range of use cases and workloads that withstands hardware and human mistakes. The layered architecture enhances loose coupling and flexibility in the system. This a huge benefit that allows understanding the trade-offs and application of various tools and technologies across the layers. There has been an advancement in the approach of building the LA due to improvements in the underlying tools. The project demonstrates a simplified architecture for the LA that is maintainable

Processing Seismic Data with Open Source Software

Open source seismic processing softwares provides a low cost alternative to commercial softwares and, with an appropriately directed development, an ability to adapt to the changing research needs. The best known example of such kind is Seismic Unix, a free reflection processing system developed at the Colorado School of Mines. It has been broadly used in research and teaching seismology and also in smaller scale seismic processing industry. Through this paper the seismic Unix was tested for the first time at Iraqi academic system by processing a real seismic data acquired at northern Iraq at Basra province. The test done by reliance a simple processing flow and the result show us high resolution seismic section and less noise ratio. According to the early mentioned results the seismic Unix recommended as a teaching tool for Iraqi educational system

Data processing techniques used with MST radars: A review

The data processing methods used in high power radar probing of the middle atmosphere are examined. The radar acts as a spatial filter on the small scale refractivity fluctuations in the medium. The characteristics of the received signals are related to the statistical properties of these fluctuations. A functional outline of the components of a radar system is given. Most computation intensive tasks are carried out by the processor. The processor computes a statistical function of the received signals, simultaneously for a large number of ranges. The slow fading of atmospheric signals is used to reduce the data input rate to the processor by coherent integration. The inherent range resolution of the radar experiments can be improved significant with the use of pseudonoise phase codes to modulate the transmitted pulses and a corresponding decoding operation on the received signals. Commutability of the decoding and coherent integration operations is used to obtain a significant reduction in computations. The limitations of the processors are outlined. At the next level of data reduction, the measured function is parameterized by a few spectral moments that can be related to physical processes in the medium. The problems encountered in estimating the spectral moments in the presence of strong ground clutter, external interference, and noise are discussed. The graphical and statistical analysis of the inferred parameters are outlined. The requirements for special purpose processors for MST radars are discussed

Processing MAGSAT data for comparison with geoid anomalies

A digital data library of MAGSAT data consisting of 1,615,636 measurements from the quiet data set, is geographically sorted, and allows rapid analysis and processing of all the quiet magnetic data about any selected location. Because this library of MAGSAT data is compatible with existing gravity and geoid data library processing and display system software, correlations between MAGSAT, surface gravity, GEOS-3 radar altimeter geoid and bathymetric data sets can be conveniently detected and analyzed. Polynomial trends from each half-orbit were removed as an effective way of estimating and removing ring current effects following estimation of the core field contribution. It was found that a third order polynomial is the lowest polynomial order that appears to provide the best consistency of residual anomalies between coincident orbits

Data-Driven Reporting and Processing of Digital Archives with Brunnhilde

[Excerpt] Archivists are now several decades in to appraising, arranging, describing, preserving, and providing access to digital archives and have developed and adopted a number of tools to aid in specific tasks along the way. This article discusses Brunnhilde, a new tool developed to address one of the first steps in working with born-digital materials: characterizing the overall contents of directories or disks to enable smart evidence-based decision-making in the appraisal, arrangement, and description processes

Ultrafast processing of pixel detector data with machine learning frameworks

Modern photon science performed at high repetition rate free-electron laser (FEL) facilities and beyond relies on 2D pixel detectors operating at increasing frequencies (towards 100 kHz at LCLS-II) and producing rapidly increasing amounts of data (towards TB/s). This data must be rapidly stored for offline analysis and summarized in real time. While at LCLS all raw data has been stored, at LCLS-II this would lead to a prohibitive cost; instead, enabling real time processing of pixel detector raw data allows reducing the size and cost of online processing, offline processing and storage by orders of magnitude while preserving full photon information, by taking advantage of the compressibility of sparse data typical for LCLS-II applications. We investigated if recent developments in machine learning are useful in data processing for high speed pixel detectors and found that typical deep learning models and autoencoder architectures failed to yield useful noise reduction while preserving full photon information, presumably because of the very different statistics and feature sets between computer vision and radiation imaging. However, we redesigned in Tensorflow mathematically equivalent versions of the state-of-the-art, "classical" algorithms used at LCLS. The novel Tensorflow models resulted in elegant, compact and hardware agnostic code, gaining 1 to 2 orders of magnitude faster processing on an inexpensive consumer GPU, reducing by 3 orders of magnitude the projected cost of online analysis at LCLS-II. Computer vision a decade ago was dominated by hand-crafted filters; their structure inspired the deep learning revolution resulting in modern deep convolutional networks; similarly, our novel Tensorflow filters provide inspiration for designing future deep learning architectures for ultrafast and efficient processing and classification of pixel detector images at FEL facilities.Comment: 9 pages, 9 figure

Parallel Astronomical Data Processing with Python: Recipes for multicore machines

High performance computing has been used in various fields of astrophysical research. But most of it is implemented on massively parallel systems (supercomputers) or graphical processing unit clusters. With the advent of multicore processors in the last decade, many serial software codes have been re-implemented in parallel mode to utilize the full potential of these processors. In this paper, we propose parallel processing recipes for multicore machines for astronomical data processing. The target audience are astronomers who are using Python as their preferred scripting language and who may be using PyRAF/IRAF for data processing. Three problems of varied complexity were benchmarked on three different types of multicore processors to demonstrate the benefits, in terms of execution time, of parallelizing data processing tasks. The native multiprocessing module available in Python makes it a relatively trivial task to implement the parallel code. We have also compared the three multiprocessing approaches - Pool/Map, Process/Queue, and Parallel Python. Our test codes are freely available and can be downloaded from our website.Comment: 15 pages, 7 figures, 1 table, "for associated test code, see http://astro.nuigalway.ie/staff/navtejs", Accepted for publication in Astronomy and Computin