Curracurrong: a stream processing system for distributed environments

Advances in technology have given rise to applications that are deployed on wireless sensor networks (WSNs), the cloud, and the Internet of things. There are many emerging applications, some of which include sensor-based monitoring, web traffic processing, and network monitoring. These applications collect large amount of data as an unbounded sequence of events and process them to generate a new sequences of events. Such applications need an adequate programming model that can process large amount of data with minimal latency; for this purpose, stream programming, among other paradigms, is ideal. However, stream programming needs to be adapted to meet the challenges inherent in running it in distributed environments. These challenges include the need for modern domain specific language (DSL), the placement of computations in the network to minimise energy costs, and timeliness in real-time applications. To overcome these challenges we developed a stream programming model that achieves easy-to-use programming interface, energy-efficient actor placement, and timeliness. This thesis presents Curracurrong, a stream data processing system for distributed environments. In Curracurrong, a query is represented as a stream graph of stream operators and communication channels. Curracurrong provides an extensible stream operator library and adapts to a wide range of applications. It uses an energy-efficient placement algorithm that optimises communication and computation. We extend the placement problem to support dynamically changing networks, and develop a dynamic program with polynomially bounded runtime to solve the placement problem. In many stream-based applications, real-time data processing is essential. We propose an approach that measures time delays in stream query processing; this model measures the total computational time from input to output of a query, i.e., end-to-end delay

Curracurrong: a stream processing system for distributed environments

Advances in technology have given rise to applications that are deployed on wireless sensor networks (WSNs), the cloud, and the Internet of things. There are many emerging applications, some of which include sensor-based monitoring, web traffic processing, and network monitoring. These applications collect large amount of data as an unbounded sequence of events and process them to generate a new sequences of events. Such applications need an adequate programming model that can process large amount of data with minimal latency; for this purpose, stream programming, among other paradigms, is ideal. However, stream programming needs to be adapted to meet the challenges inherent in running it in distributed environments. These challenges include the need for modern domain specific language (DSL), the placement of computations in the network to minimise energy costs, and timeliness in real-time applications. To overcome these challenges we developed a stream programming model that achieves easy-to-use programming interface, energy-efficient actor placement, and timeliness. This thesis presents Curracurrong, a stream data processing system for distributed environments. In Curracurrong, a query is represented as a stream graph of stream operators and communication channels. Curracurrong provides an extensible stream operator library and adapts to a wide range of applications. It uses an energy-efficient placement algorithm that optimises communication and computation. We extend the placement problem to support dynamically changing networks, and develop a dynamic program with polynomially bounded runtime to solve the placement problem. In many stream-based applications, real-time data processing is essential. We propose an approach that measures time delays in stream query processing; this model measures the total computational time from input to output of a query, i.e., end-to-end delay

Model-driven Scheduling for Distributed Stream Processing Systems

Distributed Stream Processing frameworks are being commonly used with the evolution of Internet of Things(IoT). These frameworks are designed to adapt to the dynamic input message rate by scaling in/out.Apache Storm, originally developed by Twitter is a widely used stream processing engine while others includes Flink, Spark streaming. For running the streaming applications successfully there is need to know the optimal resource requirement, as over-estimation of resources adds extra cost.So we need some strategy to come up with the optimal resource requirement for a given streaming application. In this article, we propose a model-driven approach for scheduling streaming applications that effectively utilizes a priori knowledge of the applications to provide predictable scheduling behavior. Specifically, we use application performance models to offer reliable estimates of the resource allocation required. Further, this intuition also drives resource mapping, and helps narrow the estimated and actual dataflow performance and resource utilization. Together, this model-driven scheduling approach gives a predictable application performance and resource utilization behavior for executing a given DSPS application at a target input stream rate on distributed resources.Comment: 54 page

Performance modelling and the representation of large scale distributed system functions

This thesis presents a resource based approach to model generation for performance characterization and correctness checking of large scale telecommunications networks. A notion called the timed automaton is proposed and then developed to encapsulate behaviours of networking equipment, system control policies and non-deterministic user behaviours. The states of pooled network resources and the behaviours of resource consumers are represented as continually varying geometric patterns; these patterns form part of the data operated upon by the timed automata. Such a representation technique allows for great flexibility regarding the level of abstraction that can be chosen in the modelling of telecommunications systems. None the less, the notion of system functions is proposed to serve as a constraining framework for specifying bounded behaviours and features of telecommunications systems. Operational concepts are developed for the timed automata; these concepts are based on limit preserving relations. Relations over system states represent the evolution of system properties observable at various locations within the network under study. The declarative nature of such permutative state relations provides a direct framework for generating highly expressive models suitable for carrying out optimization experiments. The usefulness of the developed procedure is demonstrated by tackling a large scale case study, in particular the problem of congestion avoidance in networks; it is shown that there can be global coupling among local behaviours within a telecommunications network. The uncovering of such a phenomenon through a function oriented simulation is a contribution to the area of network modelling. The direct and faithful way of deriving performance metrics for loss in networks from resource utilization patterns is also a new contribution to the work area

Impacts from above-ground activities in the Eagle Ford Shale play on landscapes and hydrologic flows, La Salle County, Texas

textExpanded production of hydrocarbons by means of horizontal drilling and hydraulic fracturing of shale formations has become one of the most important changes in the North American petroleum industry in decades, and the Eagle Ford (EF) Shale play in South Texas is currently one of the largest producers of oil and gas in the United States. Since 2008, more than 5000 wells have been drilled in the EF. To date, little research has focused on landscape impacts (e.g., fragmentation and soil erosion) from the construction of drilling pads, roads, pipelines, and other infrastructure. The goal of this study was to assess the spatial fragmentation from the recent EF shale boom, focusing on La Salle County, Texas. To achieve this goal, a database of wells and pipelines was overlain onto base maps of land cover, soil type, vegetation assemblages, and hydrologic units. Changes to the continuity of different ecoregions and supporting landscapes were then assessed using the Landscape Fragmentation Tool as quantified by land area and continuity of core landscape areas (those degraded by “edge effects”). Results show an increase in ecosystem fragmentation with a reduction in core areas of 8.7% (~333 km²) and an increase in landscape patches (0.2%; 6.4 km²), edges (1.8%; ~69 km²), and perforated areas (4.2%; ~162 km²) within the county. Pipeline construction dominates sources of landscape disturbance, followed by drilling and injection pads (85%, 15%, and 0.03% of disturbed area, respectively). This analysis indicates an increase in the potential for soil loss, with 51% (~58 km²) of all disturbance regimes occurring on soils with low water-transmission rates and a high runoff potential (hydrologic soil group D). Additionally, 88% (~100 km²) of all disturbances occurred on soils with a wind erodibility index of approximately 19 kt/km²/yr or higher, resulting in an estimated potential of 2 million tonnes of soil loss per year. Depending on the placement of infrastructure relative to surface drainage patterns and erodible soil, these results show that small changes in placement may significantly reduce ecological and hydrological impacts as they relate to surface runoff. Furthermore, rapid site reclamation of drilling pads and pipeline right-of-ways could substantially mitigate potential impacts.Energy and Earth Resource

Investigation of CO2 Sequestration for the Assessment of the Impact on Resource Storage with Co-production of Brine

In order to reduce Green House Gases, Carbon-dioxide (CO2) storage in deep saline aquifers is a viable commercial application for minimizing emissions. It is important to understand surface area needed to predict large scale CO2 storage while fully utilizing injection capacity. This study presents results from varying Injection pressure and well spacing to find minimal-effective well spacing required to store CO2. The study shows pressure management to manipulate hydrodynamic behavior of CO 2 in saline formations system. In conjunction, understanding the interplay of CO2 dissolution, buoyancy flow, and capillary forces in regulating the behavior of the injected CO2 plume are important. Pressure manipulated by changing injection pressure with selected brine co-production, a technique known as CO2 sequestration.;A 3-D reservoir model has been utilized to model CO2 sequestration behavior in a compositional simulator, CMG Builder. Mount Simon Sandstone (Cambrian) was selected as a \u27base case model\u27 for its recognition as an important deep saline reservoir with potential to serve as a largescale commercial CO2 storage field in the Midwestern United States.;The study shows the impact of selected injection pressure on the utilization of brine aquifer. It is recommended to store CO2 with 4000 -- 4500 psi injection pressure range for optimum storage and production conditions

A monitoring and threat detection system using stream processing as a virtual function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast realtime threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.A detecção tardia de ameaças de segurança causa um significante aumento no risco de danos irreparáveis, impossibilitando qualquer tentativa de defesa. Como consequência, a detecção rápida de ameaças em tempo real é essencial para a administração de segurança. Além disso, A tecnologia de virtualização de funções de rede (Network Function Virtualization - NFV) oferece novas oportunidades para soluções de segurança eficazes e de baixo custo. Propomos um sistema de detecção de ameaças rápido e eficiente, baseado em algoritmos de processamento de fluxo e de aprendizado de máquina. As principais contribuições deste trabalho são: i) um novo sistema de monitoramento e detecção de ameaças baseado no processamento de fluxo; ii) dois conjuntos de dados, o primeiro ´e um conjunto de dados sintético de segurança contendo tráfego suspeito e malicioso, e o segundo corresponde a uma semana de tráfego real de um operador de telecomunicações no Rio de Janeiro, Brasil; iii) um algoritmo de pré-processamento de dados composto por um algoritmo de normalização e um algoritmo para seleção rápida de características com base na correlação entre variáveis; iv) uma função de rede virtualizada em uma plataforma de código aberto para fornecer um serviço de detecção de ameaças em tempo real; v) posicionamento quase perfeito de sensores através de uma heurística proposta para posicionamento estratégico de sensores na infraestrutura de rede, com um número mínimo de sensores; e, finalmente, vi) um algoritmo guloso que aloca sob demanda uma sequencia de funções de rede virtual

Stratigraphy and sedimentology of Pliocene limestones in northern Hawke's Bay: The Opoiti, Whakapunake and Tahaenui Limestones

The Opoiti, Whakapunake and Tahaenui Limestone formations (Opoitian to Waipipian; Pliocene) crop out extensively in northern Hawke’s Bay between Wairoa and Mahia Peninsula where they are encased within mudstone dominated Wairoa Formation. The limestones are shallow cool-water carbonates that formed about tectonically active antiform structures inboard from the convergent subduction plate margin along eastern North Island. Their coarse skeletal fraction is dominated by barnacle plates with common brachiopod, pectinid and oyster remains. The carbonate factory for prolific skeletal production was likely sited in shoal water (30-60 m deep), high energy conditions atop the antiforms, with deposition from carbonate shedding down the flanks of the antiforms and even into the bounding synforms. The individual limestone units are laterally discontinuous and perceived as large lenses (up to c.3-4 km long by c.2 km wide). The enclosing thick mud-rich (M1-M4) to locally sandy (S1-S4) lithofacies, along with occasional volcaniclastic beds (V1), are here informally recorded as Wairoa Formation A, B, C or D, depending on their stratigraphic position with respect to the three limestone units. Stratigraphic logging of sections at Nuhaka (Tahaenui/Clonkeen), Mt Moumoukai and on Mahia Peninsula established 12 sedimentary lithofacies based on field texture and composition, namely limestones (L1-L3), sandstones (S1-S4), mudstones (M1-M4), and a volcaniclastite (V1). The lithofacies discriminate well on triangular plots involving carbonate content and insoluble sand, silt and clay grain sizes and typically show vertical similarities between field sites for each limestone, suggestive of similar depositional patterns and controls operating at the different localities. Opoiti Limestone occurs on Mahia Peninsula and Mt Moumoukai as a bedded, c.30 m thick, moderately dipping (25° W), siliciclastic sand-rich unit with common brachiopods and mudstone clasts (L1, L2, L3) that unconformably overlies late Miocene mudstone (M1). Petrography ranges from a sandy biomicrite to a variably sandy, poorly washed rounded biosparite or bioclastic arenite in which interskeletal space is occupied mainly by microbioclastic micrite with some isopachous sparite. Carbonate content is up to c.75%; siliciclastic grains range from very fine to fine sand size. The Opoiti Limestone is likely a transgressive event deposit (TST), fining upwards into sandstone and mudstone (HST) of Wairoa Formation B, and largely under tectonic control. Whakapunake Limestone within the field area is restricted to the west coast of Mahia Peninsula where it is c.40 m thick and comprises limestone (L3) units (10- 100 cm thick) interbedded with unique mudclast-bearing shelly sandstone (S4) units (20-100 cm thick). These couplets may be related to storm emplacements although their origin, and that of the mudclasts, remains problematic. The elongated mudclasts possibly mark the positions of original Skolithos/Ophiomorpha-like burrows that have been later modified by seismic shaking. No lower contact was observed. Upwards, the limestone grades via interbeds into shelly sandstone (S2) and mudstone (M1). Petrography ranges from packed biomicrites to poorly washed biosparites, while the sandstone interbeds are typically muddy bioclastic arenites. Cements are isopachous sparite rims and microbioclastic micrite with common siliciclasts. Carbonate contents range up to c.70% with the siliciclastic grains being of medium silt and fine sand size. The Whakapunake Limestone is likely a transgressive deposit (TST) fining upwards into shelly sandstone (TST/HST) of Wairoa Formation C, again overall tectonically driven but with possible superimposed eustatic sea level changes. Tahaenui Limestone (L1, L3) occurs in discrete outcrops (10-30 m thick) across the full area. At Nuhaka, it unconformably overlies late Miocene mudstone (M1), while the upper contact grades into shelly sandstone (S2). At Moumoukai, it unconformably onlaps the Opoiti Limestone. It is dominated by coarse barnacle plates with interparticle isopachous sparite rims and microbioclastic micrite. Petrographically the limestone ranges from rounded biosparite to packed biomicrite. Carbonate content is up to 90%, with the siliciclastic grains being of fine-medium silt and fine sand size. The Tahaenui Limestone is overall a transgressive deposit (TST) fining upwards into shelly sandstone and mudstone (HST) of Wairoa Formation D, largely under tectonic control. The Pliocene limestones are economically important as potential subsurface petroleum reservoirs, as a lime resource for agricultural use and as a hard stone source for aggregate. Recommendations of the immediate economic potential for development include local aggregate and fertiliser sources from the Tahaenui Limestone at Nuhaka and Mahia

Investigation of immiscible systems and potential applications

The droplet coalescence kinetics at 0 g and 1 g were considered for two systems which contained liquid droplets in a host liquid. One of these (Al-In) typified a system containing a liquid phase miscibility gap and the order (oil-water) a mixture of two essentially insoluble liquids. A number of coalescence mechanisms potentially prominent at low g in this system were analyzed and explanations are presented for the observed unusual stability of the emulsion. Ground base experiments were conducted on the coalescence of In droplets in and Al-In alloy during cooling through the miscibility gap at different cooling rates. These were in qualitative agreement with the computer simulation. Potential applications for systems with liquid phase miscibility gaps were explored. Possibilities included superconductors, electrical contact materials, superplastic materials, catalysts, magnetic materials, and others. The role of space processing in their production was also analyzed