Compact and indexed representation for LiDAR point clouds

[Abstract]: LiDAR devices are capable of acquiring clouds of 3D points reflecting any object around them, and adding additional attributes to each point such as color, position, time, etc. LiDAR datasets are usually large, and compressed data formats (e.g. LAZ) have been proposed over the years. These formats are capable of transparently decompressing portions of the data, but they are not focused on solving general queries over the data. In contrast to that traditional approach, a new recent research line focuses on designing data structures that combine compression and indexation, allowing directly querying the compressed data. Compression is used to fit the data structure in main memory all the time, thus getting rid of disk accesses, and indexation is used to query the compressed data as fast as querying the uncompressed data. In this paper, we present the first data structure capable of losslessly compressing point clouds that have attributes and jointly indexing all three dimensions of space and attribute values. Our method is able to run range queries and attribute queries up to 100 times faster than previous methods.Secretara Xeral de Universidades; [ED431G 2019/01]Ministerio de Ciencia e Innovacion; [PID2020-114635RB-I00]Ministerio de Ciencia e Innovacion; [PDC2021-120917C21]Ministerio de Ciencia e Innovación; [PDC2021-121239-C31]Ministerio de Ciencia e Innovación; [PID2019-105221RB-C41]Xunta de Galicia; [ED431C 2021/53]Xunta de Galicia; [IG240.2020.1.185

Extending General Compact Querieable Representations to GIS Applications

The raster model is commonly used for the representation of images in many domains, and is especially useful in Geographic Information Systems (GIS) to store information about continuous variables of the space (elevation, temperature, etc.). Current representations of raster data are usually designed for external memory or, when stored in main memory, lack efficient query capabilities. In this paper we propose compact representations to efficiently store and query raster datasets in main memory. We present different representations for binary raster data, general raster data and time-evolving raster data. We experimentally compare our proposals with traditional storage mechanisms such as linear quadtrees or compressed GeoTIFF files. Results show that our structures are up to 10 times smaller than classical linear quadtrees, and even comparable in space to non-querieable representations of raster data, while efficiently answering a number of typical queries.Comment: This research has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Actions H2020-MSCA-RISE-2015 BIRDS GA No. 690941

Compact data structures for large and complex datasets

Programa Oficial de Doutoramento en Computación . 5009V01[Abstract] In this thesis, we study the problem of processing large and complex collections of data, presenting new data structures and algorithms that allow us to efficiently store and analyze them. We focus on three main domains: processing of multidimensional data, representation of spatial information, and analysis of scientific data. The common nexus is the use of compact data structures, which combine in a unique data structure a compressed representation of the data and the structures to access such data. The target is to be able to manage data directly in compressed form, and in this way, to keep data always compressed, even in main memory. With this, we obtain two benefits: we can manage larger datasets in main memory and we take advantage of a better usage of the memory hierarchy. In the first part, we propose a compact data structure for multidimensional databases where the domains of each dimension are hierarchical. It allows efficient queries of aggregate information at different levels of each dimension. A typical application environment for our solution would be an OLAP system. Second, we focus on the representation of spatial information, specifically on raster data, which are commonly used in geographic information systems (GIS) to represent spatial attributes (such as the altitude of a terrain, the average temperature, etc.). The new method enables several typical spatial queries with better response times than the state of the art, at the same time that saves space in both main memory and disk. Besides, we also present a framework to run a spatial join between raster and vector datasets, that uses the compact data structure previously presented in this part of the thesis. Finally, we present a solution for the computation of empirical moments from a set of trajectories of a continuous time stochastic process observed in a given period of time. The empirical autocovariance function is an example of such operations. In this thesis, we propose a method that compresses sequences of floating numbers representing Brownian motion trajectories, although it can be used in other similar areas. In addition, we also introduce a new algorithm for the calculation of the autocovariance that uses a single trajectory at a time, instead of loading the whole dataset, reducing the memory consumption during the calculation process.[Resumo] Nesta tese estudamos o problema de procesar grandes coleccións de datos, presentando novas estruturas de datos compactas e algoritmos que nos permiten almacenalas e analizalas de forma eficiente. Centrámonos en tres dominios principais: procesamento de datos multidimensionais, representación de información espacial e análise de datos científicos. O nexo común é o uso de estruturas de datos compactas, que combinan nunha única estrutura de datos unha representación comprimida dos datos e as estruturas para acceder a tales datos. O obxectivo é poder manipular os datos directamente en forma comprimida, e desta maneira, manter os datos sempre comprimidos, incluso na memoria principal. Con esto obtemos dous beneficios: podemos xestionar conxuntos de datos máis grandes na memoria principal e aproveitar un mellor uso da xerarquía da memoria. Na primera parte propoñemos unha estructura de datos compacta para bases de datos multidimensionais onde os dominios de cada dimensión están xerarquizados. Permítenos consultar eficientemente a información agregada (sumar valor máximo, etc) a diferentes niveis de cada dimensión. Un entorno de aplicación típico para a nosa solución sería un sistema OLAP. En segundo lugar, centrámonos na representación de información espacial, especificamente en datos ráster, que se utilizan comunmente en sistemas de información xeográfica (SIX) para representar atributos espaciais (como a altitude dun terreo, a temperatura media, etc.). O novo método permite realizar eficientemente varias consultas espaciais típicas con tempos de resposta mellores que o estado da arte, ao mesmo tempo que reduce o espazo utilizado tanto na memoria principal como no disco. Ademais, tamén presentamos un marco de traballo para realizar un join espacial entre conxuntos de datos vectoriais e ráster, que usa a estructura de datos compacta previamente presentada nesta parte da tese. Por último, presentamos unha solución para o cálculo de momentos empíricos a partir dun conxunto de traxectorias dun proceso estocástico de tempo continuo observadas nun período de tempo dado. A función de autocovarianza empírica é un exemplo de tales operacións. Nesta tese propoñemos un método que comprime secuencias de números flotantes que representan traxectorias de movemento Browniano, aínda que pode ser empregado noutras áreas similares. Ademais, tamén introducimos un novo algoritmo para o cálculo da autocovarianza que emprega unha única traxectoria á vez, en lugar de cargar todo o conxunto de datos, reducindo o consumo de memoria durante o proceso de cálculo.[Resumen] En esta tesis estudiamos el problema de procesar grandes colecciones de datos, presentando nuevas estructuras de datos compactas y algoritmos que nos permiten almacenarlas y analizarlas de forma eficiente. Nos centramos principalmente en tres dominios: procesamiento de datos multidimensionales, representación de información espacial y análisis de datos científicos. El nexo común es el uso de estructuras de datos compactas, que combinan en una única estructura de datos una representación comprimida de los datos y las estructuras para acceder a dichos datos. El objetivo es poder manipular los datos directamente en forma comprimida, y de esta manera, mantener los datos siempre comprimidos, incluso en la memoria principal. Con esto obtenemos dos beneficios: podemos gestionar conjuntos de datos más grandes en la memoria principal y aprovechar un mejor uso de la jerarquía de la memoria. En la primera parte proponemos una estructura de datos compacta para bases de datos multidimensionales donde los dominios de cada dimensión están jerarquizados. Nos permite consultar eficientemente la información agregada (suma, valor máximo, etc.) a diferentes niveles de cada dimensión. Un entorno de aplicación típico para nuestra solución sería un sistema OLAP. En segundo lugar, nos centramos en la representación de la información espacial, específicamente en datos ráster, que se utilizan comúnmente en sistemas de información geográfica (SIG) para representar atributos espaciales (como la altitud de un terreno, la temperatura media, etc.). El nuevo método permite realizar eficientemente varias consultas espaciales típicas con tiempos de respuesta mejores que el estado del arte, al mismo tiempo que reduce el espacio utilizado tanto en la memoria principal como en el disco. Además, también presentamos un marco de trabajo para realizar un join espacial entre conjuntos de datos vectoriales y ráster, que usa la estructura de datos compacta previamente presentada en esta parte de la tesis. Por último, presentamos una solución para el cálculo de momentos empíricos a partir de un conjunto de trayectorias de un proceso estocástico de tiempo continuo observadas en un período de tiempo dado. La función de autocovariancia empírica es un ejemplo de tales operaciones. En esta tesis proponemos un método que comprime secuencias de números flotantes que representan trayectorias de movimiento Browniano, aunque puede ser utilizado en otras áreas similares. En esta parte, también introducimos un nuevo algoritmo para el cálculo de la autocovariancia que utiliza una única trayectoria a la vez, en lugar de cargar todo el conjunto de datos, reduciendo el consumo de memoria durante el proceso de cálculoXunta de Galicia; ED431G/01Ministerio de Economía y Competitividad ;TIN2016-78011-C4-1-RMinisterio de Economía y Competitividad; TIN2016-77158-C4-3-RMinisterio de Economía y Competitividad; TIN2013-46801-C4-3-RCentro para el desarrollo Tecnológico e Industrial; IDI-20141259Centro para el desarrollo Tecnológico e Industrial; ITC-20151247Xunta de Galicia; GRC2013/05

Multiscale visualization approaches for Volunteered Geographic Information and Location-based Social Media

Today, “zoomable” maps are a state-of-the-art way to explore the world, available to anyone with Internet access. However, the process of creating this visualization has been rather loosely investigated and documented. Nevertheless, with an increasing amount of available data, interactive maps have become a more integral approach to visualizing and exploring big datasets and user-generated data. OpenStreetMap and online platforms such as Twitter and Flickr offer application programming interfaces (APIs) with geographic information. They are well-known examples of this visualization challenge and are often used as examples. In addition, an increasing number of public administrations collect open data and publish their data sets, which makes the task of visualization even more relevant. This dissertation deals with the visualization of user-generated geodata as a multiscale map. The basics of today’s multiscale maps—their history, technologies, and possibilities—are explored and abstracted. This work introduces two new multiscale-focused visualization approaches for point data from volunteered geographic information (VGI) and location-based social media (LBSM). One contribution of this effort is a visualization methodology for spatially referenced information in the form of point geometries, using nominally scaled data from social media such as Twitter or Flickr. Typical for this data is a high number of social media posts in different categories—a post on social media corresponds to a point in a specific category. Due to the sheer quantity and similar characteristics, the posts appear generic rather than unique. This type of dataset can be explored using the new method of micro diagrams to visualize the dataset on multiple scales and resolutions. The data is aggregated into small grid cells, and the numerical proportion is shown with small diagrams, which can visually merge into heterogenous areas through colors depicting a specific category. The diagram sizes allow the user to estimate the overall number of aggregated points in a grid cell. A different visualization approach is proposed for more unique points, considered points of interest (POI), based on the selection method. The goal is to identify more locally relevant points from the data set, considered more important compared to other points in the neighborhood, which are then compared by numerical attribute. The method, derived from topographic isolation and called discrete isolation, is the distance from one point to the next with a higher attribute value. By using this measure, the most essential points can be easily selected by choosing a minimum distance and producing a homogenous spatial of the selected points within the chosen dataset. The two newly developed approaches are applied to multiscale mapping by constructing example workflows that produce multiscale maps. The publicly available multiscale mapping workflows OpenMapTiles and OpenStreetMap Carto, using OpenStreetMap data, are systematically explored and analyzed. The result is a general workflow for multiscale map production and a short overview of the toolchain software. In particular, the generalization approaches in the example projects are discussed and these are classified into cartographic theories on the basis of literature. The workflow is demonstrated by building a raster tile service for the micro diagrams and a vector tile service for the discrete isolation, able to be used with just a web browser. In conclusion, these new approaches for point data using VGI and LBSM allow better qualitative visualization of geodata. While analyzing vast global datasets is challenging, exploring and analyzing hidden data patterns is fruitful. Creating this degree of visualization and producing maps on multiple scales is a complicated task. The workflows and tools provided in this thesis will make map production on a worldwide scale easier.:1 Introduction 1 1.1 Motivation .................................................................................................. 3 1.2 Visualization of crowdsourced geodata on multiple scales ............ 5 1.2.1 Research objective 1: Visualization of point collections ......... 6 1.2.2 Research objective 2: Visualization of points of interest ......... 7 1.2.3 Research objective 3: Production of multiscale maps ............. 7 1.3 Reader’s guide ......................................................................................... 9 1.3.1 Structure ........................................................................................... 9 1.3.2 Related Publications ....................................................................... 9 1.3.3 Formatting and layout ................................................................. 10 1.3.4 Online examples ........................................................................... 10 2 Foundations of crowdsourced mapping on multiple scales 11 2.1 Types and properties of crowdsourced data .................................. 11 2.2 Currents trends in cartography ......................................................... 11 2.3 Definitions .............................................................................................. 12 2.3.1 VGI .................................................................................................. 12 2.3.2 LBSM .............................................................................................. 13 2.3.3 Space, place, and location......................................................... 13 2.4 Visualization approaches for crowdsourced geodata ................... 14 2.4.1 Review of publications and visualization approaches ........... 14 2.4.2 Conclusions from the review ...................................................... 15 2.4.3 Challenges mapping crowdsourced data ................................ 17 2.5 Technologies for serving multiscale maps ...................................... 17 2.5.1 Research about multiscale maps .............................................. 17 2.5.2 Web Mercator projection ............................................................ 18 2.5.3 Tiles and zoom levels .................................................................. 19 2.5.4 Raster tiles ..................................................................................... 21 2.5.5 Vector tiles .................................................................................... 23 2.5.6 Tiling as a principle ..................................................................... 25 3 Point collection visualization with categorized attributes 26 3.1 Target users and possible tasks ....................................................... 26 3.2 Example data ......................................................................................... 27 3.3 Visualization approaches .................................................................... 28 3.3.1 Common techniques .................................................................... 28 3.3.2 The micro diagram approach .................................................... 30 3.4 The micro diagram and its parameters ............................................ 33 3.4.1 Aggregating points into a regular structure ............................ 33 3.4.2 Visualizing the number of data points ...................................... 35 3.4.3 Grid and micro diagrams ............................................................ 36 3.4.4 Visualizing numerical proportions with diagrams .................. 37 3.4.5 Influence of color and color brightness ................................... 38 3.4.6 Interaction options with micro diagrams .................................. 39 3.5 Application and user-based evaluation ............................................ 39 3.5.1 Micro diagrams in a multiscale environment ........................... 39 3.5.2 The micro diagram user study ................................................... 41 3.5.3 Point collection visualization discussion .................................. 47 4 Selection of POIs for visualization 50 4.1 Approaches for point selection .......................................................... 50 4.2 Methods for point selection ................................................................ 51 4.2.1 Label grid approach .................................................................... 52 4.2.2 Functional importance approach .............................................. 53 4.2.3 Discrete isolation approach ....................................................... 54 4.3 Functional evaluation of selection methods .................................... 56 4.3.1 Runtime comparison .................................................................... 56 4.3.2 Use cases for discrete isolation ................................................ 57 4.4 Discussion of the selection approaches .......................................... 61 4.4.1 A critical view of the use cases ................................................. 61 4.4.2 Comparing the approaches ........................................................ 62 4.4.3 Conclusion ..................................................................................... 64 5 Creating multiscale maps 65 5.1 Examples of multiscale map production .......................................... 65 5.1.1 OpenStreetMap Infrastructure ................................................... 66 5.1.2 OpenStreetMap Carto ................................................................. 67 5.1.3 OpenMapTiles ............................................................................... 73 5.2 Methods of multiscale map production ............................................ 80 5.2.1 OpenStreetMap tools ................................................................... 80 5.2.2 Geoprocessing .............................................................................. 80 5.2.3 Database ........................................................................................ 80 5.2.4 Creating tiles ................................................................................. 82 5.2.5 Caching .......................................................................................... 82 5.2.6 Styling tiles .................................................................................... 82 5.2.7 Viewing tiles ................................................................................... 83 5.2.8 The stackless approach to tile creation ................................... 83 5.3 Example workflows for creating multiscale maps ........................... 84 5.3.1 Raster tiles: OGC services and micro diagrams .................... 84 5.3.2 Vector tiles: Slippy map and vector tiles ................................. 87 5.4 Discussion of approaches and workflows ....................................... 90 5.4.1 Map production as a rendering pipeline .................................. 90 5.4.2 Comparison of OpenStreetMap Carto and OpenMapTiles .. 92 5.4.3 Discussion of the implementations ........................................... 93 5.4.4 Generalization in map production workflows .......................... 95 5.4.5 Conclusions ................................................................................. 101 6 Discussion 103 6.1 Development for web mapping ........................................................ 103 6.1.1 The role of standards in map production .............................. 103 6.1.2 Technological development ..................................................... 103 6.2 New data, new mapping techniques? ............................................. 104 7 Conclusion 106 7.1 Visualization of point collections ..................................................... 106 7.2 Visualization of points of interest ................................................... 107 7.3 Production of multiscale maps ........................................................ 107 7.4 Synthesis of the research questions .............................................. 108 7.5 Contributions ....................................................................................... 109 7.6 Limitations ............................................................................................ 110 7.7 Outlook ................................................................................................. 111 8 References 113 9 Appendix 130 9.1 Zoom levels and Scale ...................................................................... 130 9.3 Full information about selected UGC papers ................................ 131 9.4 Timeline of mapping technologies .................................................. 133 9.5 Timeline of map providers ................................................................ 133 9.6 Code snippets from own map production workflows .................. 134 9.6.1 Vector tiles workflow ................................................................. 134 9.6.2 Raster tiles workflow.................................................................. 137Heute sind zoombare Karten Alltag für jeden Internetznutzer. Die Erstellung interaktiv zoombarer Karten ist allerdings wenig erforscht, was einen deutlichen Gegensatz zu ihrer aktuellen Bedeutung und Nutzungshäufigkeit darstellt. Die Forschung in diesem Bereich ist also umso notwendiger. Steigende Datenmengen und größere Regionen, die von Karten abgedeckt werden sollen, unterstreichen den Forschungsbedarf umso mehr. Beispiele für stetig wachsende Datenmengen sind Geodatenquellen wie OpenStreetMap aber auch freie amtliche Geodatensätze (OpenData), aber auch die zunehmende Zahl georeferenzierter Inhalte auf Internetplatformen wie Twitter oder Flickr zu nennen. Das Thema dieser Arbeit ist die Visualisierung eben dieser nutzergenerierten Geodaten mittels zoombarer Karten. Dafür wird die Entwicklung der zugrundeliegenden Technologien über die letzten zwei Jahr-zehnte und die damit verbundene Möglichkeiten vorgestellt. Weitere Beiträge sind zwei neue Visualisierungsmethoden, die sich besonders für die Darstellung von Punktdaten aus raumbezogenen nutzergenerierten Daten und georeferenzierte Daten aus Sozialen Netzwerken eignen. Ein Beitrag dieser Arbeit ist eine neue Visualisierungsmethode für raumbezogene Informationen in Form von Punktgeometrien mit nominal skalierten Daten aus Sozialen Medien, wie beispielsweise Twitter oder Flickr. Typisch für diese Daten ist eine hohe Anzahl von Beiträgen mit unterschiedlichen Kategorien. Wobei die Beiträge, bedingt durch ihre schiere Menge und ähnlicher Ei-genschaften, eher generisch als einzigartig sind. Ein Beitrag in den So-zia len Medien entspricht dabei einem Punkt mit einer bestimmten Katego-rie. Ein solcher Datensatz kann mit der neuen Methode der „micro diagrams“ in verschiedenen Maßstäben und Auflösungen visualisiert und analysiert werden. Dazu werden die Daten in kleine Gitterzellen aggregiert. Die Menge und Verteilung der über die Kategorien aggregierten Punkte wird durch kleine Diagramme dargestellt, wobei die Farben die verschiedenen Kategorien visualisieren. Durch die geringere Größe der einzelnen Diagramme verschmelzen die kleinen Diagramme visuell, je nach der Verteilung der Farben für die Kategorien. Bei genauerem Hinsehen ist die Schätzung der Menge der aggregierten Punkte über die Größe der Diagramme die Menge und die Verteilung über die Kategorien möglich. Für einzigartigere Punkte, die als Points of Interest (POI) angesehen werden, wird ein anderer Visualisierungsansatz vorgeschlagen, der auf einer Auswahlmethode basiert. Ziel ist es dabei lokal relevantere Punkte aus dem Datensatz zu identifizieren, die im Vergleich zu anderen Punkten in der Nachbarschaft des Punktes verglichen nach einem numerischen Attribut wichtiger sind. Die Methode ist von dem geographischen Prinzip der Dominanz von Bergen abgeleitet und wird „discrete isolation“ genannt. Es handelt sich dabei um die Distanz von einem Punkt zum nächsten mit einem höheren Attributwert. Durch die Verwendung dieses Maßes können lokal bedeutende Punkte leicht ausgewählt werden, indem ein minimaler Abstand gewählt und so räumlich gleichmäßig verteilte Punkte aus dem Datensatz ausgewählt werden. Die beiden neu vorgestellten Methoden werden in den Kontext der zoombaren Karten gestellt, indem exemplarische Arbeitsabläufe erstellt werden, die als Er-gebnis eine zoombare Karte liefern. Dazu werden die frei verfügbaren Beispiele zur Herstellung von weltweiten zoombaren Karten mit nutzergenerierten Geo-daten von OpenStreetMap, anhand der Kartenprojekte OpenMapTiles und O-penStreetMap Carto analysiert und in Arbeitsschritte gegliedert. Das Ergebnis ist ein wiederverwendbarer Arbeitsablauf zur Herstellung zoombarer Karten, ergänzt durch eine Auswahl von passender Software für die einzelnen Arbeits-schritte. Dabei wird insbesondere auf die Generalisierungsansätze in den Beispielprojekten eingegangen und diese anhand von Literatur in die kartographische Theorie eingeordnet. Zur Demonstration des Workflows wird je ein Raster Tiles Dienst für die „micro diagrams“ und ein Vektor Tiles Dienst für die „discrete isolation“ erstellt. Beide Dienste lassen sich mit einem aktuellen Webbrowser nutzen. Zusammenfassend ermöglichen diese neuen Visualisierungsansätze für Punkt-daten aus VGI und LBSM eine bessere qualitative Visualisierung der neuen Geodaten. Die Analyse riesiger globaler Datensätze ist immer noch eine Herausforderung, aber die Erforschung und Analyse verborgener Muster in den Daten ist lohnend. Die Erstellung solcher Visualisierungen und die Produktion von Karten in verschiedenen Maßstäben ist eine komplexe Aufgabe. Die in dieser Arbeit vorgestellten Arbeitsabläufe und Werkzeuge erleichtern die Erstellung von Karten in globalem Maßstab.:1 Introduction 1 1.1 Motivation .................................................................................................. 3 1.2 Visualization of crowdsourced geodata on multiple scales ............ 5 1.2.1 Research objective 1: Visualization of point collections ......... 6 1.2.2 Research objective 2: Visualization of points of interest ......... 7 1.2.3 Research objective 3: Production of multiscale maps ............. 7 1.3 Reader’s guide ......................................................................................... 9 1.3.1 Structure ........................................................................................... 9 1.3.2 Related Publications ....................................................................... 9 1.3.3 Formatting and layout ................................................................. 10 1.3.4 Online examples ........................................................................... 10 2 Foundations of crowdsourced mapping on multiple scales 11 2.1 Types and properties of crowdsourced data .................................. 11 2.2 Currents trends in cartography ......................................................... 11 2.3 Definitions .............................................................................................. 12 2.3.1 VGI .................................................................................................. 12 2.3.2 LBSM .............................................................................................. 13 2.3.3 Space, place, and location......................................................... 13 2.4 Visualization approaches for crowdsourced geodata ................... 14 2.4.1 Review of publications and visualization approaches ........... 14 2.4.2 Conclusions from the review ...................................................... 15 2.4.3 Challenges mapping crowdsourced data ................................ 17 2.5 Technologies for serving multiscale maps ...................................... 17 2.5.1 Research about multiscale maps .............................................. 17 2.5.2 Web Mercator projection ............................................................ 18 2.5.3 Tiles and zoom levels .................................................................. 19 2.5.4 Raster tiles ..................................................................................... 21 2.5.5 Vector tiles .................................................................................... 23 2.5.6 Tiling as a principle ..................................................................... 25 3 Point collection visualization with categorized attributes 26 3.1 Target users and possible tasks ....................................................... 26 3.2 Example data ......................................................................................... 27 3.3 Visualization approaches .................................................................... 28 3.3.1 Common techniques .................................................................... 28 3.3.2 The micro diagram approach .................................................... 30 3.4 The micro diagram and its parameters ............................................ 33 3.4.1 Aggregating points into a regular structure ............................ 33 3.4.2 Visualizing the number of data points ...................................... 35 3.4.3 Grid and micro diagrams ............................................................ 36 3.4.4 Visualizing numerical proportions with diagrams .................. 37 3.4.5 Influence of color and color brightness ................................... 38 3.4.6 Interaction options with micro diagrams .................................. 39 3.5 Application and user-based evaluation ............................................ 39 3.5.1 Micro diagrams in a multiscale environment ........................... 39 3.5.2 The micro diagram user study ................................................... 41 3.5.3 Point collection vis

Large-Scale Spatial Data Management on Modern Parallel and Distributed Platforms

Rapidly growing volume of spatial data has made it desirable to develop efficient techniques for managing large-scale spatial data. Traditional spatial data management techniques cannot meet requirements of efficiency and scalability for large-scale spatial data processing. In this dissertation, we have developed new data-parallel designs for large-scale spatial data management that can better utilize modern inexpensive commodity parallel and distributed platforms, including multi-core CPUs, many-core GPUs and computer clusters, to achieve both efficiency and scalability. After introducing background on spatial data management and modern parallel and distributed systems, we present our parallel designs for spatial indexing and spatial join query processing on both multi-core CPUs and GPUs for high efficiency as well as their integrations with Big Data systems for better scalability. Experiment results using real world datasets demonstrate the effectiveness and efficiency of the proposed techniques on managing large-scale spatial data

Explorative coastal oceanographic visual analytics : oceans of data

The widely acknowledged challenge to data analysis and understanding, resulting from the exponential increase in volumes of data generated by increasingly complex modelling and sampling systems, is a problem experienced by many researchers, including ocean scientists. The thesis explores a visualization and visual analytics solution for predictive studies of coastal shelf and estuarine modelled, hydrodynamics undertaken to understand sea level rise, as a contribution to wider climate change studies, and to underpin coastal zone planning, flood prevention and extreme event management. But these studies are complex and require numerous simulations of estuarine hydrodynamics, generating extremely large datasets of multi-field data. This type\ud of data is acknowledged as difficult to visualize and analyse, as its numerous attributes present significant computational challenges, and ideally require a wide range of approaches to provide the necessary insight. These challenges are not easily overcome with the current visualization and analysis methodologies employed by coastal shelf hydrodynamic researchers, who use several software systems to generate graphs, each taking considerable time to operate, thus it is difficult to explore different scenarios and explore the data interactively and visually. The thesis, therefore, develops novel visualization and visual analytics techniques to help researchers overcome the limitations of existing methods (for example in understanding key tidal components); analyse data in a timely manner and explore different scenarios. There were a number of challenges to this: the size of the data, resulting in lengthy computing time, also many data values becoming plotted on one pixel (overplotting). The thesis presents: (1) a new visualization framework (VINCA) using caching and hierarchical aggregation techniques to make the data more interactive, plus explorative, coordinated multiple views, to enable the scientists to explore the data. (2) A novel estuarine transect profiler and flux tool, which provides instantaneous flux calculations across an estuary. Measures of flux are of great significance in oceanographic studies, yet are notoriously difficult and time consuming to calculate with the commonly used tools. This derived data is added back into the database for further investigation and analysis. (3) New views, including a novel, dynamic, spatially aggregated Parallel Coordinate Plots (Sa-PCP), are developed to provide different perspectives of the spatial, time dependent data, also methodologies for developing high-quality (journal ready) output from the visualization tool. Finally, (4) the dissertation explored the use of hierarchical data-structures and caching techniques to enable fast analysis on a desktop computer and to overcome the overplotting challenge for this data

LIPIcs, Volume 277, GIScience 2023, Complete Volume

LIPIcs, Volume 277, GIScience 2023, Complete Volum

12th International Conference on Geographic Information Science: GIScience 2023, September 12–15, 2023, Leeds, UK

No abstract available