A systematic review of SQL-on-Hadoop by using compact data formats

Article also submitted for publication in Baltic J. Modern Computing (BJMC) on October 5, 2016.There are huge volumes of raw data generated every day. The question is how to store these data in order to provide faster data access. The research direction in Big Data projects using Hadoop Technology, MapReduce kind of framework and compact data formats shows that two data formats (Avro and Parquet) support schema evolution and compression in order to utilize less storage space. In this paper, a systematic review of SQL-on-Hadoop by using Avro and Parquet has been performed over the past six years (2010–2015) using publications of conference proceedings and journals of IEEEXplore, ACM Digital Library, ScienceDirect. With the help of search strategy followed, 94 research papers have been identified out of which 17 have been analyzed as relevant papers. At the end, the conclusion has been made that direct comparison by compactness and fastness between Avro and Parquet do not exist in data science

Data communication between Distributed Control System, Serial Device and Android App

Nowadays, the world is searching for new technologies or new methods to make the controlling process more automated or easier to access. This search has widely increased to reach such technology starting from HART technology and passing through Asset Management system (AMS). So developing such technology will open the field for another technologies to be introduced in the industry. One of those technologies is Android. Many of the machines in the plants have been purchased with its own control system , this system consider a remote system comparing with one in the control room which in most of the time the control system is Distributed Control System (DCS). This remote system is connected the control system through alarms and trips only. So Data communication between the remote system (Serial Device) and DCS was a must to reach any readings from the remote system. This data communication can be achievable through the Modbus. To reach more and more automated plants, the android technology has to take place in such industry which you can receive the real-time reading of any machine in the plant on your android platform (Mobile phone or Tablet). And the methodology of such technology can be achieved by many steps, starting with creating a database of readings of the DCS, this database has to be an online database to be able to retrieve data from it as long as you want, and the final part is to design an android application that responsible to retrieve the data from the online database. Such technology makes you to reach the readings of the plant from anywhere at any time. This technology is a step to create and fully automated plants, and you can supervise the plant from anywhere as long you an internet connectio

A stream processing architecture for heterogeneous data sources in the Internet of Things

The number of Internet of Things (IoT) and smart devices capable of producing, consuming and exchanging information is constantly increasing. It is estimated there will be around 30 billion of them in 2020. In most cases, the structures of the information produced by such devices are completely different, thus providing heterogeneous information. This is becoming a challenge for researchers working on IoT, who need to perform homogenisation and pre-processing tasks before using the IoT data. This paper aims to provide an architecture for processing and analysing data from heterogeneous sources with different structures in IoT scopes, allowing researchers to focus on data analysis, without having to worry about the structure of the data sources. This architecture combines the real-time stream processing paradigm for information processing and transforming, together with the complex event processing for information analysis. This provides us with capability of processing, transforming and analysing large amounts of information in real time. The results obtained from the evaluation of a real-world case study about water supply network management show that the architecture can be applied to an IoT water management scenario to analyse the information in real time. Additionally, the stress tests successfully conducted for this architecture highlight that a large incoming rate of input events could be processed without latency, resulting in efficient performance of the proposed architecture. This novel software architecture is adequate for automatically detecting situations of interest in the IoT through the processing, transformation and analysis of large amounts of heterogeneous information in real time.El número de dispositivos del Internet de las Cosas (IoT) y dispositivos inteligentes capaces de producir, consumir e intercambiar información está aumentando constantemente. Se estima que habrá alrededor de 30 mil millones de ellos en 2020. En la mayoría de los casos, las estructuras de la información producida por dichos dispositivos son completamente diferentes, proporcionando así información heterogénea. Esto se está convirtiendo en un desafío para los investigadores que trabajan en IoT, quienes necesitan realizar tareas de homogeneización y preprocesamiento antes de utilizar los datos de IoT. Este documento tiene como objetivo proporcionar una arquitectura para procesar y analizar datos de fuentes heterogéneas con diferentes estructuras en ámbitos de IoT, permitiendo a los investigadores centrarse en el análisis de datos, sin tener que preocuparse por la estructura de las fuentes de datos. Esta arquitectura combina el paradigma de procesamiento de flujo en tiempo real para el procesamiento y transformación de información, junto con el procesamiento de eventos complejos para el análisis de información. Esto nos proporciona la capacidad de procesar, transformar y analizar grandes cantidades de información en tiempo real. Los resultados obtenidos de la evaluación de un estudio de caso del mundo real sobre la gestión de la red de suministro de agua muestran que la arquitectura puede ser aplicada a un escenario de gestión de agua de IoT para analizar la información en tiempo real. Además, las pruebas de estrés realizadas con éxito para esta arquitectura destacan que una gran tasa de entrada de eventos de entrada podría ser procesada sin latencia, lo que resulta en un rendimiento eficiente de la arquitectura propuesta. Esta novedosa arquitectura de software es adecuada para detectar automáticamente situaciones de interés en el IoT a través del procesamiento, transformación y análisis de grandes cantidades de información heterogénea en tiempo real.This work was supported in part by the Spanish Ministry of Science and Innovation and the European Union FEDER Funds (No. TIN2015-65845-C3-3-R and No. RTI2018-093608-B-C33) and in part by the pre-doctoral program of the University of Cádiz (2017-020/PU/EPIF-FPI-CT/CP). In addition, we would like to thank GEN Grupo Energético for sharing their data for testing purposes

Data communication between Distributed Control System, Serial Device and Android App

Nowadays, the world is searching for new technologies or new methods to make the controlling process more automated or easier to access. This search has widely increased to reach such technology starting from HART technology and passing through Asset Management system (AMS). So developing such technology will open the field for another technologies to be introduced in the industry. One of those technologies is Android. Many of the machines in the plants have been purchased with its own control system , this system consider a remote system comparing with one in the control room which in most of the time the control system is Distributed Control System (DCS). This remote system is connected the control system through alarms and trips only. So Data communication between the remote system (Serial Device) and DCS was a must to reach any readings from the remote system. This data communication can be achievable through the Modbus. To reach more and more automated plants, the android technology has to take place in such industry which you can receive the real-time reading of any machine in the plant on your android platform (Mobile phone or Tablet). And the methodology of such technology can be achieved by many steps, starting with creating a database of readings of the DCS, this database has to be an online database to be able to retrieve data from it as long as you want, and the final part is to design an android application that responsible to retrieve the data from the online database. Such technology makes you to reach the readings of the plant from anywhere at any time. This technology is a step to create and fully automated plants, and you can supervise the plant from anywhere as long you an internet connectio

Smartphones: A Platform For Disaster Management

Bal, H.E. [Promotor]Kielmann, T. [Copromotor

Segment Logic

O'Hearn, Reynolds and Yang introduced local Hoare reasoning about mutable data structures using separation logic. They reason about the local parts of the memory accessed by programs, and thus construct their smallest complete specifications. Gardner et al. generalised their work, using context logic to reason about structured data at the same level of abstraction as the data itself. In particular, we developed a formal specification of the Document Object Model (DOM), a W3C XML update library. Whilst we kept to the spirit of local reasoning, we were not able to retain small specifications for all of the commands of DOM: for example, our specification of the appendChild command was not small. We show how to obtain such small specifications by developing a more fine-grained context structure, allowing us to work with arbitrary segments of a data structure. We introduce segment logic, a logic for reasoning about such segmented data structures, staring at first with a simple tree structure, but then showing how to generalise our approach to arbitrary structured data. Using our generalised segment logic we construct a reasoning framework for abstract program modules, showing how to reason about such modules at the client level. In particular we look at modules for trees, lists, heaps and the more complex data model of DOM. An important part of any abstraction technique is an understanding of how to link the abstraction back to concrete implementations. Building on our previous abstraction and refinement work for local reasoning, we show how to soundly implement the segment models used in our abstract reasoning. In particular we show how to implement our fine-grained list and tree modules so that their abstract specifications are satisfied by the concrete implementations. We also show how our reasoning from the abstract level can be translated to reasoning at the concrete level. Finally, we turn our attention to concurrency and show how having genuine small axioms for our commands allows for a simple treatment of abstract level concurrency constructs