Rumble: Data Independence for Large Messy Data Sets

This paper introduces Rumble, an engine that executes JSONiq queries on large, heterogeneous and nested collections of JSON objects, leveraging the parallel capabilities of Spark so as to provide a high degree of data independence. The design is based on two key insights: (i) how to map JSONiq expressions to Spark transformations on RDDs and (ii) how to map JSONiq FLWOR clauses to Spark SQL on DataFrames. We have developed a working implementation of these mappings showing that JSONiq can efficiently run on Spark to query billions of objects into, at least, the TB range. The JSONiq code is concise in comparison to Spark's host languages while seamlessly supporting the nested, heterogeneous data sets that Spark SQL does not. The ability to process this kind of input, commonly found, is paramount for data cleaning and curation. The experimental analysis indicates that there is no excessive performance loss, occasionally even a gain, over Spark SQL for structured data, and a performance gain over PySpark. This demonstrates that a language such as JSONiq is a simple and viable approach to large-scale querying of denormalized, heterogeneous, arborescent data sets, in the same way as SQL can be leveraged for structured data sets. The results also illustrate that Codd's concept of data independence makes as much sense for heterogeneous, nested data sets as it does on highly structured tables.Comment: Preprint, 9 page

Using tracked mobile sensors to make maps of environmental effects

We present a study the results of a study of environmental carbon monoxide pollution that has uses a set of tracked, mobile pollution sensors. The motivating concept is that we will be able to map pollution and other properties of the real world a fine scale if we can deploy a large set of sensors with members of the general public who would carry them as they go about their normal everyday activities. To prove the viability of this concept we have to demonstrate that data gathered in an ad-hoc manner is reliable enough in order to allow us to build interesting geo-temporal maps. We present a trial using a small number of global positioning system-tracked CO sensors. From analysis of raw GPS logs we find some well-known spatial and temporal properties of CO. Further, by processing the GPS logs we can find fine-grained variations in pollution readings such as when crossing roads. We then discuss the space of possibilities that may be enabled by tracking sensors around the urban environment – both in getting at personal experience of properties of the environment and in making summative maps to predict future conditions. Although we present a study of CO, the techniques will be applicable to other environmental properties such as radio signal strength, noise, weather and so on

VATDIS Web Mapping - A Report on the Application of Open Standards and Open Architecture in Geospatial Interoperability for Emergency Management

This report provides an overview of the web mapping activities carried out in the VATDIS action in 2007. These web mapping activities aimed at integrating the work done for the Orchestra, Floodsite and Preview IP into one single demonstrator. In the main section of this report an outline is given of the reasons for integrating the projects and the contents of this integration. The report should be seen as a summary of the technical choices made in order to accomplish this integration. The annexes on this report help the more technical reader to understand the design of the demonstrator and to duplicate it for its own purposes.JRC.G.7-Traceability and vulnerability assessmen

ERP implementation for an administrative agency as a corporative frontend and an e-commerce smartphone app

This document contains all the descriptions, arguments and demonstrations of the researches, analysis, reasoning, designs and tasks performed to achieve the requirement to technologically evolve an managing agency in a way that, through a solution that requires a reduced investment, makes possible to arrange a business management tool with e-commerce and also a mobile application that allows access and consultation of mentioned tool. The first part of the document describes the scenario in order to contextualize the project and introduces ERP (Enterprise Resources Planning). In the second part, a deep research of ERP market products is carried out, identifying the strengths and weaknesses of each one of the products in order to finish with the choice of the most suitable product for the scenario proposed in the project. A third part of the document describes the installation process of the selected product carried out based on the use of Dockers, as well as the configurations and customizations that they make on the selected ERP. A description of the installation and configuration of additional modules is also made, necessary to achieve the agreed scope of the project. In a fourth part of the thesis, the process of creating an iOS and Android App that connects to the selected ERP database is described. The process begins with the design of the App. Once designed, it is explained the process of study and documentation of technologies to choose the technology stack that allows making an application robust and contemporary without use of licensing. After choosing the technologies to use there are explained the dependencies and needs to install runtime enviornments prior to the start of coding. Later, it describes how the code of the App has been raised and developed. The compilation and verification mechanisms are indicated in continuation. And finally, it is showed the result of the development of the App once distributed. Finally, a chapter for the conclusions analyzes the difficulties encountered during the project and the achievements, analyzing what has been learned during the development of this project

On-the-fly Map Generator for OpenStreetMap Data Using WebGL

This project describes an approach to create an On-the-fly Map Generator for Openstreetmap Data Using WebGL. The most common methods to generate online maps generate PNG overlay tile images from a wide range of data sources, like GeoJSON, GeoTIFF, PostGIS, CSV, and SQLite, etc., based on the coordinates and zoom-level. This project aims to send vector data for the map to the browser and hence render maps on-the-fly using WebGL. We push all of the vector computation to the GPU. This means that less data needs to be sent to the browser. We have compared existing approaches to our method of generating maps and have been able to show that our method provides a faster and better solution

Story Parsing and Adventure Generation with Python and Postgres

Dungeons and Dragons is a tabletop roleplaying game that allows players to assume the roles of adventurers in medieval fantasy setting while one player is tasked as playing the role of the Dungeon Master (DM). This player facilitates the story and all other characters not played by the other players. Adventure Day is a toolset for Dungeons and Dragons 5th Edition that assists Dungeon Master in formatting their Story as well as gathering useful details for the challenges presented within that adventure. Adventure day aims to accomplish this by associating relevant monster data from postgres database while using the text input of a desired adventure parsed through a Python application to filter for highlighted terms and take action based on a defined dictionary within the same text file. Adventure day will be capable of building encounters with monsters specified by monster environment and/or abilities that they possess, provide inspiration for non-player characters that the DM will portray via personality traits, and finally help to scale monster difficulty depending on the power level of the player’s characters. Adventure day will work in tandem with a helper site that formats Markdown text into a format that mirrors that of published material