Efficient processing of large-scale spatio-temporal data

Millionen Geräte, wie z.B. Mobiltelefone, Autos und Umweltsensoren senden ihre Positionen zusammen mit einem Zeitstempel und weiteren Nutzdaten an einen Server zu verschiedenen Analysezwecken. Die Positionsinformationen und übertragenen Ereignisinformationen werden als Punkte oder Polygone dargestellt. Eine weitere Art räumlicher Daten sind Rasterdaten, die zum Beispiel von Kameras und Sensoren produziert werden. Diese großen räumlich-zeitlichen Datenmengen können nur auf skalierbaren Plattformen wie Hadoop und Apache Spark verarbeitet werden, die jedoch z.B. die Nachbarschaftsinformation nicht ausnutzen können - was die Ausführung bestimmter Anfragen praktisch unmöglich macht. Die wiederholten Ausführungen der Analyseprogramme während ihrer Entwicklung und durch verschiedene Nutzer resultieren in langen Ausführungszeiten und hohen Kosten für gemietete Ressourcen, die durch die Wiederverwendung von Zwischenergebnissen reduziert werden können. Diese Arbeit beschäftigt sich mit den beiden oben beschriebenen Herausforderungen. Wir präsentieren zunächst das STARK Framework für die Verarbeitung räumlich-zeitlicher Vektor- und Rasterdaten in Apache Spark. Wir identifizieren verschiedene Algorithmen für Operatoren und analysieren, wie diese von den Eigenschaften der zugrundeliegenden Plattform profitieren können. Weiterhin wird untersucht, wie Indexe in der verteilten und parallelen Umgebung realisiert werden können. Außerdem vergleichen wir Partitionierungsmethoden, die unterschiedlich gut mit ungleichmäßiger Datenverteilung und der Größe der Datenmenge umgehen können und präsentieren einen Ansatz um die auf Operatorebene zu verarbeitende Datenmenge frühzeitig zu reduzieren. Um die Ausführungszeit von Programmen zu verkürzen, stellen wir einen Ansatz zur transparenten Materialisierung von Zwischenergebnissen vor. Dieser Ansatz benutzt ein Entscheidungsmodell, welches auf den tatsächlichen Operatorkosten basiert. In der Evaluierung vergleichen wir die verschiedenen Implementierungs- sowie Konfigurationsmöglichkeiten in STARK und identifizieren Szenarien wann Partitionierung und Indexierung eingesetzt werden sollten. Außerdem vergleichen wir STARK mit verwandten Systemen. Im zweiten Teil der Evaluierung zeigen wir, dass die transparente Wiederverwendung der materialisierten Zwischenergebnisse die Ausführungszeit der Programme signifikant verringern kann.Millions of location-aware devices, such as mobile phones, cars, and environmental sensors constantly report their positions often in combination with a timestamp to a server for different kinds of analyses. While the location information of the devices and reported events is represented as points and polygons, raster data is another type of spatial data, which is for example produced by cameras and sensors. This Big spatio-temporal Data needs to be processed on scalable platforms, such as Hadoop and Apache Spark, which, however, are unaware of, e.g., spatial neighborhood, what makes them practically impossible to use for this kind of data. The repeated executions of the programs during development and by different users result in long execution times and potentially high costs in rented clusters, which can be reduced by reusing commonly computed intermediate results. Within this thesis, we tackle the two challenges described above. First, we present the STARK framework for processing spatio-temporal vector and raster data on the Apache Spark stack. For operators, we identify several possible algorithms and study how they can benefit from the underlying platform's properties. We further investigate how indexes can be realized in the distributed and parallel architecture of Big Data processing engines and compare methods for data partitioning, which perform differently well with respect to data skew and data set size. Furthermore, an approach to reduce the amount of data to process at operator level is presented. In order to reduce the execution times, we introduce an approach to transparently recycle intermediate results of dataflow programs, based on operator costs. To compute the costs, we instrument the programs with profiling code to gather the execution time and result size of the operators. In the evaluation, we first compare the various implementation and configuration possibilities in STARK and identify scenarios when and how partitioning and indexing should be applied. We further compare STARK to related systems and show that we can achieve significantly better execution times, not only when exploiting existing partitioning information. In the second part of the evaluation, we show that with the transparent cost-based materialization and recycling of intermediate results, the execution times of programs can be reduced significantly

Cross-Platform Presentation of Interactive Volumetric Imagery

Volume data is useful across many disciplines, not just medicine. Thus, it is very important that researchers have a simple and lightweight method of sharing and reproducing such volumetric data. In this paper, we explore some of the challenges associated with volume rendering, both from a classical sense and from the context of Web3D technologies. We describe and evaluate the pro- posed X3D Volume Rendering Component and its associated styles for their suitability in the visualization of several types of image data. Additionally, we examine the ability for a minimal X3D node set to capture provenance and semantic information from outside ontologies in metadata and integrate it with the scene graph

Multidimensional Marine Data: The next frontier for Hydrographic Offices

Hydrographic offices today exist in a world of accelerating technological change that is influencing human behavior, creating new needs and ways for exploiting data to understand our world. Hydrographic offices have traditionally been the producers of nautical information for safety of navigation. By the end of the 20th century, with the appearance of the IHO S-57 Standard, their main challenge was to evolve into a central database production system. Now, the main challenge and opportunity is to evolve from there into a true geospatial agency, developing a hydrospatial information system capable of providing products and services for multidimensional analysis and decision-making through apps and web browsers at the "speed of trust." The technology to do this is there; it is a matter of vision and desire to move toward the next frontier.Los Servicios Hidrográficos existen hoy en día en un mundo de cambios tecnológicos acelerados que están influyendo en el comportamiento humano, creando nuevas necesidades y formas de explotar los datos para entender nuestro mundo. Los Servicios Hidrográficos han sido tradicionalmente los productores de información náutica para la seguridad de la navegación. A finales del siglo XX, con la aparición de la Norma S-57 de la OHI, su principal desafío fue evolucionar hacia un sistema central de producción de bases de datos. Ahora, el desafío y la oportunidad principales son evolucionar hacia una verdadera agencia geoespacial, desarrollando un sistema de información hidroespacial capaz de proporcionar productos y servicios para un análisis y una toma de decisiones multidimensionales mediante aplicaciones y navegadores web a la "velocidad de la confianza". La tecnología para hacerlo está ahí; es una cuestión de visión y de deseo de avanzar hacia la próxima frontera.Les services hydrographiques évoluent aujourd’hui dans un monde où les changements technologiques s’accélèrent, influençant le comportement humain, créant de nouveaux besoins et de nouvelles manières d’exploiter les données afin de comprendre notre monde. Les services hydrographiques sont les traditionnels producteurs des informations nautiques aux fins de la sécurité de la navigation. A la fin du 20ème siècle, avec l’apparition de la norme S-57 de l’OHI, leur principal défi était d’évoluer pour devenir un système central de production de base de données. A présent, leur principal défi et opportunité est d’évoluer de ce rôle pour devenir une véritable agence géospatiale, en développant un système d’informations hydrospatiales capable de fournir des produits et services aux fins d’analyse multidimensionnelle et de prise de décision via des applications et des navigateurs web à la « vitesse de la confiance ». La technologie pour relever ce défi existe, c’est donc une question de vision et de volonté d’avancer vers la prochaine frontière

Video guidance, landing, and imaging systems

The adaptive potential of video guidance technology for earth orbital and interplanetary missions was explored. The application of video acquisition, pointing, tracking, and navigation technology was considered to three primary missions: planetary landing, earth resources satellite, and spacecraft rendezvous and docking. It was found that an imaging system can be mechanized to provide a spacecraft or satellite with a considerable amount of adaptability with respect to its environment. It also provides a level of autonomy essential to many future missions and enhances their data gathering ability. The feasibility of an autonomous video guidance system capable of observing a planetary surface during terminal descent and selecting the most acceptable landing site was successfully demonstrated in the laboratory. The techniques developed for acquisition, pointing, and tracking show promise for recognizing and tracking coastlines, rivers, and other constituents of interest. Routines were written and checked for rendezvous, docking, and station-keeping functions

Mechanical Engineering

The book substantially offers the latest progresses about the important topics of the "Mechanical Engineering" to readers. It includes twenty-eight excellent studies prepared using state-of-art methodologies by professional researchers from different countries. The sections in the book comprise of the following titles: power transmission system, manufacturing processes and system analysis, thermo-fluid systems, simulations and computer applications, and new approaches in mechanical engineering education and organization systems

Predictive maintenance of electrical grid assets: internship at EDP Distribuição - Energia S.A

Internship Report presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Knowledge Management and Business IntelligenceThis report will describe the activities developed during an internship at EDP Distribuição, focusing on a Predictive Maintenance analytics project directed at high voltage electrical grid assets including Overhead Lines, Power Transformers and Circuit Breakers. The project’s main goal is to support EDP’s asset management processes by improving maintenance and investing planning. The project’s main deliverables are the Probability of Failure metric that forecast asset failures 15 days ahead of time, estimated through supervised machine learning models; the Health Index metric that indicates asset’s current state and condition, implemented though the Ofgem methodology; and two asset management dashboards. The project was implemented by an external service provider, a consultant company, and during the internship it was possible to integrate the team, and participate in the development activities

Heterogeneous sensor database framework for the sensor observation service: integrating remote and in-situ sensor observations at the database backend

Dissertation submitted in partial fulfillment of the requirements for the Degree of Master of Science in Geospatial Technologies.Environmental monitoring and management systems in most cases deal with models and spatial analytics that involve the integration of in-situ and remote sensor observations. In-situ sensor observations and those gathered by remote sensors are usually provided by different databases and services in real-time dynamic service systems like the Geo-Web Services. Thus, data have to be pulled from different databases and transferred over the web before they are fused and processed on the service middleware. This process is very massive and unnecessary communication and work load on the service, especially when retrieving massive raster coverage data. Thus in this research, we propose a database model for heterogeneous sensortypes that enables geo-scientific processing and spatial analytics involving remote and in-situ sensor observations at the database level of the Sensor Observation Service, SOS. This approach would be used to reduce communication and massive workload on the Geospatial Web Service, as well make query request from the user end a lot more flexible. Hence the challenging task is to develop a heterogeneous sensor database model that enables geoprocessing and spatial analytics at the database level and how this could be integrated with the geo-web services to reduce communication and workload on the service and as well make query request from the client end more flexible through the use of SQL statements

The 1995 Science Information Management and Data Compression Workshop

This document is the proceedings from the 'Science Information Management and Data Compression Workshop,' which was held on October 26-27, 1995, at the NASA Goddard Space Flight Center, Greenbelt, Maryland. The Workshop explored promising computational approaches for handling the collection, ingestion, archival, and retrieval of large quantities of data in future Earth and space science missions. It consisted of fourteen presentations covering a range of information management and data compression approaches that are being or have been integrated into actual or prototypical Earth or space science data information systems, or that hold promise for such an application. The Workshop was organized by James C. Tilton and Robert F. Cromp of the NASA Goddard Space Flight Center

Tao--an architecturally balanced reconfigurable hardware processor

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 1997.Includes bibliographical references (p. 107-109).This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.by Andrew S. Huang.M.Eng

Efficient Point Clustering for Visualization

The visualization of large spatial point data sets constitutes a problem with respect to runtime and quality. A visualization of raw data often leads to occlusion and clutter and thus a loss of information. Furthermore, particularly mobile devices have problems in displaying millions of data items. Often, thinning via sampling is not the optimal choice because users want to see distributional patterns, cardinalities and outliers. In particular for visual analytics, an aggregation of this type of data is very valuable for providing an interactive user experience. This thesis defines the problem of visual point clustering that leads to proportional circle maps. It furthermore introduces a set of quality measures that assess different aspects of resulting circle representations. The Circle Merging Quadtree constitutes a novel and efficient method to produce visual point clusterings via aggregation. It is able to outperform comparable methods in terms of runtime and also by evaluating it with the aforementioned quality measures. Moreover, the introduction of a preprocessing step leads to further substantial performance improvements and a guaranteed stability of the Circle Merging Quadtree. This thesis furthermore addresses the incorporation of miscellaneous attributes into the aggregation. It discusses means to provide statistical values for numerical and textual attributes that are suitable for side-views such as plots and data tables. The incorporation of multiple data sets or data sets that contain class attributes poses another problem for aggregation and visualization. This thesis provides methods for extending the Circle Merging Quadtree to output pie chart maps or maps that contain circle packings. For the latter variant, this thesis provides results of a user study that investigates the methods and the introduced quality criteria. In the context of providing methods for interactive data visualization, this thesis finally presents the VAT System, where VAT stands for visualization, analysis and transformation. This system constitutes an exploratory geographical information system that implements principles of visual analytics for working with spatio-temporal data. This thesis details on the user interface concept for facilitating exploratory analysis and provides the results of two user studies that assess the approach