Consistent data aggregate retrieval for sensor network systems.

Lee Lok Hang.Thesis (M.Phil.)--Chinese University of Hong Kong, 2005.Includes bibliographical references (leaves 87-93).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.ivChapter 1 --- Introduction --- p.1Chapter 1.1 --- Sensors and Sensor Networks --- p.3Chapter 1.2 --- Sensor Network Deployment --- p.7Chapter 1.3 --- Motivations --- p.7Chapter 1.4 --- Contributions --- p.9Chapter 1.5 --- Thesis Organization --- p.10Chapter 2 --- Literature Review --- p.11Chapter 2.1 --- Data Cube --- p.11Chapter 2.2 --- Data Aggregation in Sensor Networks --- p.12Chapter 2.2.1 --- Hierarchical Data Aggregation --- p.13Chapter 2.2.2 --- Gossip-based Aggregation --- p.13Chapter 2.2.3 --- Hierarchical Gossip Aggregation --- p.13Chapter 2.3 --- GAF Algorithm --- p.14Chapter 2.4 --- Concurrency Control --- p.17Chapter 2.4.1 --- Two-phase Locking --- p.17Chapter 2.4.2 --- Timestamp Ordering --- p.18Chapter 3 --- Building Distributed Data Cubes in Sensor Network --- p.20Chapter 3.1 --- Aggregation Operators --- p.21Chapter 3.2 --- Distributed Prefix (PS) Sum Data Cube --- p.22Chapter 3.2.1 --- Prefix Sum (PS) Data Cube --- p.22Chapter 3.2.2 --- Notations --- p.24Chapter 3.2.3 --- Querying a PS Data Cube --- p.25Chapter 3.2.4 --- Building Distributed PS Data Cube --- p.27Chapter 3.2.5 --- Time Bounds --- p.32Chapter 3.2.6 --- Fast Aggregate Queries on Multiple Regions --- p.37Chapter 3.2.7 --- Simulation Results --- p.43Chapter 3.3 --- Distributed Local Prefix Sum (LPS) Data Cube --- p.50Chapter 3.3.1 --- Local Prefix Sum Data Cube --- p.52Chapter 3.3.2 --- Notations --- p.55Chapter 3.3.3 --- Querying an LPS Data Cube --- p.56Chapter 3.3.4 --- Building Distributed LPS Data Cube --- p.61Chapter 3.3.5 --- Time Bounds --- p.63Chapter 3.3.6 --- Fast Aggregate Queries on Multiple Regions --- p.67Chapter 3.3.7 --- Simulation Results --- p.68Chapter 3.3.8 --- Distributed PS Data Cube Vs Distributed LPS Data Cube --- p.74Chapter 4 --- Concurrency Control and Consistency in Sensor Networks --- p.76Chapter 4.1 --- Data Inconsistency in Sensor Networks --- p.76Chapter 4.2 --- Traditional Concurrency Control Protocols and Sensor Networks --- p.80Chapter 4.3 --- The Consistent Retrieval of Data from Distributed Data Cubes --- p.81Chapter 5 --- Conclusions --- p.85References --- p.87Appendix --- p.94A Publications --- p.9

Low-latency, query-driven analytics over voluminous multidimensional, spatiotemporal datasets

2017 Summer.Includes bibliographical references.Ubiquitous data collection from sources such as remote sensing equipment, networked observational devices, location-based services, and sales tracking has led to the accumulation of voluminous datasets; IDC projects that by 2020 we will generate 40 zettabytes of data per year, while Gartner and ABI estimate 20-35 billion new devices will be connected to the Internet in the same time frame. The storage and processing requirements of these datasets far exceed the capabilities of modern computing hardware, which has led to the development of distributed storage frameworks that can scale out by assimilating more computing resources as necessary. While challenging in its own right, storing and managing voluminous datasets is only the precursor to a broader field of study: extracting knowledge, insights, and relationships from the underlying datasets. The basic building block of this knowledge discovery process is analytic queries, encompassing both query instrumentation and evaluation. This dissertation is centered around query-driven exploratory and predictive analytics over voluminous, multidimensional datasets. Both of these types of analysis represent a higher-level abstraction over classical query models; rather than indexing every discrete value for subsequent retrieval, our framework autonomously learns the relationships and interactions between dimensions in the dataset (including time series and geospatial aspects), and makes the information readily available to users. This functionality includes statistical synopses, correlation analysis, hypothesis testing, probabilistic structures, and predictive models that not only enable the discovery of nuanced relationships between dimensions, but also allow future events and trends to be predicted. This requires specialized data structures and partitioning algorithms, along with adaptive reductions in the search space and management of the inherent trade-off between timeliness and accuracy. The algorithms presented in this dissertation were evaluated empirically on real-world geospatial time-series datasets in a production environment, and are broadly applicable across other storage frameworks

Quantitative Hyperspectral Imaging Pipeline to Recover Surface Images from CRISM Radiance Data

Hyperspectral data are important for remote applications such as mineralogy, geology, agriculture and surveillance sensing. A general pipeline converting measured hyperspectral radiance to the surface reflectance image can provide planetary scientists with clean, robust and repeatable products to work on. In this dissertation, the surface single scattering albedos (SSAs), the ratios of scattering eciency to scattering plus absorption eciences of a single particle, are selected to describe the reflectance. Moreover, the IOF, the ratio of measured spectral radiance (in the unit of watts per squared-meter and micrometer) to the solar spectral radiance (in the unit of watts per squared-meter and micrometer) at the observed time, is used to indicate the measurements. This pipeline includes two main parts: retrieving SSAs from IOF and reconstructing the SSA images from the SSA cube. The Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) on the Mars Reconnaissance Orbiter (MRO) helps scientists identify locations on Mars that may have exhibit hydrated mineral phases. This dissertation mainly focuses on developing the pipeline for CRISM data. One should notice that pipelines for other hyperspectral spectrometers can also be developed based on almost the same idea. Retrieving surface kinetic temperatures and SSA values from IOF data is challenging because the problem is under-determined and ill-posed, including modulating effects of atmospheric aerosols and gases, and surface scattering and emission properties. We introduce a general framework called STANN (Separating Temperature and Albedo using Neural Networks) to solve this kind of problem. STANN takes the hyperspectral IOF cube as inputs and outputs the retrieved temperature mapping and the corresponding SSA cube. Our STANN is derived using the Discrete Ordinates Radiative Transfer function to describe the forward model from SSA and temperature to IOF. In the STANN, we have a generator to generate more training samples based on limited library spectra and a neural network to approximate the inverse function based on enough generated training samples. This framework has been implemented for the Compact Imaging Spectrometer for Mars in a detailed manner. SSA can be computed from IOF directly by modeling the thermal and solar reflectance together, based on retrieved temperatures. Because accurate retrieved temperature directly leads to accurate SSA, we compare the accuracy of retrieved temperatures from STANN. The retrieved temperature has only 4 K error by one point validation (242 K) using the Curiosity Rover\u27s thermal radiometer data. Our STANN temperature map is compared with a temperature map generated independently from a theoretical thermal model. The theoretical thermal model describes the relationship between Lambert albedo at the wavelength 1.0 µm, thermal inertia and the surface temperature. However, because the thermal inertia has pixel size larger than 100 m/pixel, the generated temperature also has the same pixel size. Our STANN temperature is projected into the same pixel size (100 m/pixel) by the classic projection method. The two temperature maps have consistent global patterns. Retrieved from an IOF cube, a noisy hyperspectral SSA cube needs to be denoised and reconstructed onto the Mars surface. We propose a new algorithm, hypothesis-based estimation with regularization (HyBER), to reconstruct and denoise hyperspectral image data without extra statistical assumptions. The hypothesis test selects the best statistical model approximating measurements based on the data only. Gaussian and Poisson distributions are common respectively for continuous and integer random variables, due to the law of large numbers. Hyperspectral IOF data result from converting discrete photon counting data to continuous electrical signals after calibration. Thus, so far, Gaussian and Poisson are candidate distributions for our hypothesis tests. A regularized maximum log-likelihood estimation method is derived based on the selected model. A spatially dependent weighting on the regularization penalty is presented, substantially eliminating row artifacts that are due to non-uniform sampling. A new spectral weighting penalty is introduced to suppress varying detector-related noise. HyBER generates reconstructions with sharpened images and spectra in which the noise is suppressed, whereas fine-scale mineral absorptions are preserved. The performance is quantitatively analyzed for simulations with relative error 0.002%, which is better than the traditional non-statistical methods (baselines) and statistical methods with improper assumptions. When applied to the Mars Reconnaissance Orbiter\u27s Compact Reconnaissance Imaging Spectrometer for Mars data, the spatial resolution and contrast are about 2 times better as compared to map projecting data without the use of HyBER. So far, part of our results has enabled planetary scientists to identify minerals and understand the forming history of Mars craters. Some of these findings are verified by the Opportunity Rover\u27s measurements. In the future, results from this pipeline for CRISM are promising to play more and more critical roles in the planetary science

An Object SLAM Framework for Association, Mapping, and High-Level Tasks

Object SLAM is considered increasingly significant for robot high-level perception and decision-making. Existing studies fall short in terms of data association, object representation, and semantic mapping and frequently rely on additional assumptions, limiting their performance. In this paper, we present a comprehensive object SLAM framework that focuses on object-based perception and object-oriented robot tasks. First, we propose an ensemble data association approach for associating objects in complicated conditions by incorporating parametric and nonparametric statistic testing. In addition, we suggest an outlier-robust centroid and scale estimation algorithm for modeling objects based on the iForest and line alignment. Then a lightweight and object-oriented map is represented by estimated general object models. Taking into consideration the semantic invariance of objects, we convert the object map to a topological map to provide semantic descriptors to enable multi-map matching. Finally, we suggest an object-driven active exploration strategy to achieve autonomous mapping in the grasping scenario. A range of public datasets and real-world results in mapping, augmented reality, scene matching, relocalization, and robotic manipulation have been used to evaluate the proposed object SLAM framework for its efficient performance.Comment: Accepted by IEEE Transactions on Robotics(T-RO

Entropy in Image Analysis II

Image analysis is a fundamental task for any application where extracting information from images is required. The analysis requires highly sophisticated numerical and analytical methods, particularly for those applications in medicine, security, and other fields where the results of the processing consist of data of vital importance. This fact is evident from all the articles composing the Special Issue "Entropy in Image Analysis II", in which the authors used widely tested methods to verify their results. In the process of reading the present volume, the reader will appreciate the richness of their methods and applications, in particular for medical imaging and image security, and a remarkable cross-fertilization among the proposed research areas

AUTOMATED TREE-LEVEL FOREST QUANTIFICATION USING AIRBORNE LIDAR

Traditional forest management relies on a small field sample and interpretation of aerial photography that not only are costly to execute but also yield inaccurate estimates of the entire forest in question. Airborne light detection and ranging (LiDAR) is a remote sensing technology that records point clouds representing the 3D structure of a forest canopy and the terrain underneath. We present a method for segmenting individual trees from the LiDAR point clouds without making prior assumptions about tree crown shapes and sizes. We then present a method that vertically stratifies the point cloud to an overstory and multiple understory tree canopy layers. Using the stratification method, we modeled the occlusion of higher canopy layers with respect to point density. We also present a distributed computing approach that enables processing the massive data of an arbitrarily large forest. Lastly, we investigated using deep learning for coniferous/deciduous classification of point cloud segments representing individual tree crowns. We applied the developed methods to the University of Kentucky Robinson Forest, a natural, majorly deciduous, closed-canopy forest. 90% of overstory and 47% of understory trees were detected with false positive rates of 14% and 2% respectively. Vertical stratification improved the detection rate of understory trees to 67% at the cost of increasing their false positive rate to 12%. According to our occlusion model, a point density of about 170 pt/m² is needed to segment understory trees located in the third layer as accurately as overstory trees. Using our distributed processing method, we segmented about two million trees within a 7400-ha forest in 2.5 hours using 192 processing cores, showing a speedup of ~170. Our deep learning experiments showed high classification accuracies (~82% coniferous and ~90% deciduous) without the need to manually assemble the features. In conclusion, the methods developed are steps forward to remote, accurate quantification of large natural forests at the individual tree level

Forecasting in Database Systems

Time series forecasting is a fundamental prerequisite for decision-making processes and crucial in a number of domains such as production planning and energy load balancing. In the past, forecasting was often performed by statistical experts in dedicated software environments outside of current database systems. However, forecasts are increasingly required by non-expert users or have to be computed fully automatically without any human intervention. Furthermore, we can observe an ever increasing data volume and the need for accurate and timely forecasts over large multi-dimensional data sets. As most data subject to analysis is stored in database management systems, a rising trend addresses the integration of forecasting inside a DBMS. Yet, many existing approaches follow a black-box style and try to keep changes to the database system as minimal as possible. While such approaches are more general and easier to realize, they miss significant opportunities for improved performance and usability. In this thesis, we introduce a novel approach that seamlessly integrates time series forecasting into a traditional database management system. In contrast to flash-back queries that allow a view on the data in the past, we have developed a Flash-Forward Database System (F2DB) that provides a view on the data in the future. It supports a new query type - a forecast query - that enables forecasting of time series data and is automatically and transparently processed by the core engine of an existing DBMS. We discuss necessary extensions to the parser, optimizer, and executor of a traditional DBMS. We furthermore introduce various optimization techniques for three different types of forecast queries: ad-hoc queries, recurring queries, and continuous queries. First, we ease the expensive model creation step of ad-hoc forecast queries by reducing the amount of processed data with traditional sampling techniques. Second, we decrease the runtime of recurring forecast queries by materializing models in a specialized index structure. However, a large number of time series as well as high model creation and maintenance costs require a careful selection of such models. Therefore, we propose a model configuration advisor that determines a set of forecast models for a given query workload and multi-dimensional data set. Finally, we extend forecast queries with continuous aspects allowing an application to register a query once at our system. As new time series values arrive, we send notifications to the application based on predefined time and accuracy constraints. All of our optimization approaches intend to increase the efficiency of forecast queries while ensuring high forecast accuracy

Deployment of Heterogeneous Swarm Robotic Agents Using a Task-Oriented Utility-Based Algorithm

In a swarm robotic system, the desired collective behavior emerges from local decisions made by robots, themselves, according to their environment. Swarm robotics is an emerging area that has attracted many researchers over the last few years. It has been proven that a single robot with multiple capabilities cannot complete an intended job within the same time frame as that of multiple robotic agents. A swarm of robots, each one with its own capabilities, are more flexible, robust, and cost-effective than an individual robot. As a result of a comprehensive investigation of the current state of swarm robotic research, this dissertation demonstrates how current swarm deployment systems lack the ability to coordinate heterogeneous robotic agents. Moreover, this dissertation's objective shall define the starting point of potential algorithms that lead to the development of a new software environment interface. This interface will assign a set of collaborative tasks to the swarm system without being concerned about the underlying hardware of the heterogeneous robotic agents. The ultimate goal of this research is to develop a task-oriented software application that facilitates the rapid deployment of multiple robotic agents. The task solutions are created at run-time, and executed by the agents in a centralized or decentralized fashion. Tasks are fractioned into smaller sub-tasks which are, then, assigned to the optimal number of robots using a novel Robot Utility Based Task Assignment (RUTA) algorithm. The system deploys these robots using it's application program interfaces (API's) and uploads programs that are integrated with a small routine code. The embedded routine allows robots to configure solutions when the decentralized approach is adopted. In addition, the proposed application also offers customization of robotic platforms by simply defining the available sensing and actuation devices. Another objective of the system is to improve code and component reusability to reduce efforts in deploying tasks to swarm robotic agents. Usage of the proposed framework prevents the need to redesign or rewrite programs should any changes take place in the robot's platform