Intelligent Management and Efficient Operation of Big Data

This chapter details how Big Data can be used and implemented in networking and computing infrastructures. Specifically, it addresses three main aspects: the timely extraction of relevant knowledge from heterogeneous, and very often unstructured large data sources, the enhancement on the performance of processing and networking (cloud) infrastructures that are the most important foundational pillars of Big Data applications or services, and novel ways to efficiently manage network infrastructures with high-level composed policies for supporting the transmission of large amounts of data with distinct requisites (video vs. non-video). A case study involving an intelligent management solution to route data traffic with diverse requirements in a wide area Internet Exchange Point is presented, discussed in the context of Big Data, and evaluated.Comment: In book Handbook of Research on Trends and Future Directions in Big Data and Web Intelligence, IGI Global, 201

A unified view of data-intensive flows in business intelligence systems : a survey

Data-intensive flows are central processes in today’s business intelligence (BI) systems, deploying different technologies to deliver data, from a multitude of data sources, in user-preferred and analysis-ready formats. To meet complex requirements of next generation BI systems, we often need an effective combination of the traditionally batched extract-transform-load (ETL) processes that populate a data warehouse (DW) from integrated data sources, and more real-time and operational data flows that integrate source data at runtime. Both academia and industry thus must have a clear understanding of the foundations of data-intensive flows and the challenges of moving towards next generation BI environments. In this paper we present a survey of today’s research on data-intensive flows and the related fundamental fields of database theory. The study is based on a proposed set of dimensions describing the important challenges of data-intensive flows in the next generation BI setting. As a result of this survey, we envision an architecture of a system for managing the lifecycle of data-intensive flows. The results further provide a comprehensive understanding of data-intensive flows, recognizing challenges that still are to be addressed, and how the current solutions can be applied for addressing these challenges.Peer ReviewedPostprint (author's final draft

Monitoring of gas emissions at landfill sites using autonomous gas sensors

Executive Summary This report details the work carried out during the Smart Plant project (2005-AIC-MS-43-M4). As part of this research, an autonomous platform for monitoring greenhouse gases (methane (CH4), carbon dioxide (CO2)) has been developed, prototyped and field validated. The modular design employed means that the platform can be readily adapted for a variety of applications involving these and other target gases such as hydrogen sulfide (H2S), ammonia (NH3) and carbon monoxide (CO) and the authors are in the process of completing several short demonstrator projects to illustrate the potential of the platform for some of these applications. The field validation for the greenhouse gas monitoring platform was carried out at two landfill sites in Ireland. The unit was used to monitor the concentration of CO2 and CH4 gas at perimeter borehole wells. The final prototype was deployed for over 4 months and successfully extracted samples from the assigned perimeter borehole well headspace, measured them and sent the data to a database via a global system for mobile (GSM) communications. The data were represented via an updating graph in a web interface. Sampling was carried out twice per day, giving a 60-fold increase on current monitoring procedures which provide one gas concentration measurement per month. From additional work described in this report, a number of conclusions were drawn regarding lateral landfill gas migration on a landfill site and the management of this migration to the site’s perimeter. To provide frequent, reliable monitoring of landfill gas migration to perimeter borehole wells, the unit needs to: • Be fully autonomous; • Be capable of extracting a gas sample from a borehole well independently of personnel; • Be able to relay the data in near real time to a base station; and • Have sensors with a range capable of adequately monitoring gas events accurately at all times. The authors believe that a unit capable of such monitoring has been developed and validated. This unit provides a powerful tool for effective management of landfill site gases. The effectiveness of this unit has been recognised by the site management team at the long-term deployment trial site, and the data gathered have been used to improve the day-to-day operations and gas management system on-site. The authors make the following recommendations: 1. The dynamics of the landfill gas management system cannot be captured by taking measurements once per month; thus, a minimum sampling rate of once per day is advised. 2. The sampling protocol should be changed: (i) Borehole well samples should not be taken from the top of the well but should be extracted at a depth within the headspace (0.5–1.0 m). The measurement depth will be dependent on the water table and headspace depth within the borehole well. (ii) The sampling time should be increased to 3 min to obtain a steady-state measurement from the headspace and to take a representative sample; and (iii) For continuous monitoring on-site, the extracted sample should be recycled back into the borehole well. However, for compliance monitoring, the sample should not be returned to the borehole well. 3. Devices should be placed at all borehole wells so the balance on the site can be maintained through the gas management system and extraction issues can be quickly recognised and addressed before there are events of high gas migration to the perimeter. 4. A pilot study should be carried out by the EPA using 10 of these autonomous devices over three to five sites to show the need and value for this type of sampling on Irish landfill sites

An agent-driven semantical identifier using radial basis neural networks and reinforcement learning

Due to the huge availability of documents in digital form, and the deception possibility raise bound to the essence of digital documents and the way they are spread, the authorship attribution problem has constantly increased its relevance. Nowadays, authorship attribution,for both information retrieval and analysis, has gained great importance in the context of security, trust and copyright preservation. This work proposes an innovative multi-agent driven machine learning technique that has been developed for authorship attribution. By means of a preprocessing for word-grouping and time-period related analysis of the common lexicon, we determine a bias reference level for the recurrence frequency of the words within analysed texts, and then train a Radial Basis Neural Networks (RBPNN)-based classifier to identify the correct author. The main advantage of the proposed approach lies in the generality of the semantic analysis, which can be applied to different contexts and lexical domains, without requiring any modification. Moreover, the proposed system is able to incorporate an external input, meant to tune the classifier, and then self-adjust by means of continuous learning reinforcement.Comment: Published on: Proceedings of the XV Workshop "Dagli Oggetti agli Agenti" (WOA 2014), Catania, Italy, Sepember. 25-26, 201