Navigating in Numerous Video Data: User Interface Design for an On-Camera Video Analytics Engine

Video analytics powered by artificial intelligence shows high promise in making our society smarter. Harnessing large amounts of video data, however, requires the development of processing systems demonstrating high performance and high efficiency. To this end, this work has contributed to a video analytics system powered by artificial intelligence for object detection and recognition. Rather than streaming all the video frames to the cloud, the system analyzes images on-camera and only returns those of interest to the cloud. This edge analytics research-grade software is available, but it lacks a simple web interface for general use by scientists, engineers, and other experts. To make the system versatile and user-friendly, a web interface was developed for receiving user queries (including parameters such as camera geolocations, video time span, and object class of interest) and presenting the query results. This web interface will make our video analytics system more accessible to domain experts (those in law enforcement, health care, environmental monitoring, etc.)

Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras

Overlapping cameras offer exciting opportunities to view a scene from different angles, allowing for more advanced, comprehensive and robust analysis. However, existing visual analytics systems for multi-camera streams are mostly limited to (i) per-camera processing and aggregation and (ii) workload-agnostic centralized processing architectures. In this paper, we present Argus, a distributed video analytics system with cross-camera collaboration on smart cameras. We identify multi-camera, multi-target tracking as the primary task of multi-camera video analytics and develop a novel technique that avoids redundant, processing-heavy identification tasks by leveraging object-wise spatio-temporal association in the overlapping fields of view across multiple cameras. We further develop a set of techniques to perform these operations across distributed cameras without cloud support at low latency by (i) dynamically ordering the camera and object inspection sequence and (ii) flexibly distributing the workload across smart cameras, taking into account network transmission and heterogeneous computational capacities. Evaluation of three real-world overlapping camera datasets with two Nvidia Jetson devices shows that Argus reduces the number of object identifications and end-to-end latency by up to 7.13x and 2.19x (4.86x and 1.60x compared to the state-of-the-art), while achieving comparable tracking quality.Comment: 18 pages, under revie

Approximate Query Service on Autonomous IoT Cameras

Elf is a runtime for an energy-constrained camera to continuously summarize video scenes as approximate object counts. Elf's novelty centers on planning the camera's count actions under energy constraint. (1) Elf explores the rich action space spanned by the number of sample image frames and the choice of per-frame object counters; it unifies errors from both sources into one single bounded error. (2) To decide count actions at run time, Elf employs a learning-based planner, jointly optimizing for past and future videos without delaying result materialization. Tested with more than 1,000 hours of videos and under realistic energy constraints, Elf continuously generates object counts within only 11% of the true counts on average. Alongside the counts, Elf presents narrow errors shown to be bounded and up to 3.4x smaller than competitive baselines. At a higher level, Elf makes a case for advancing the geographic frontier of video analytics

Partitioning and Offloading for IoT and Video Streaming Applications that Utilize Computing Resources at the Network Edge

The Internet of Things (IoT) is a concept in which physical objects embedded with sensors, actuators, and network connectivity can communicate and react to their surroundings. IoT applications connect physical objects for the purpose of decision making by sensing and analysing generated data from the embedded sensors in physical objects. IoT applications are growing rapidly as sensors become less expensive. Sensors generate large amounts of data that may meaningless unless the data is used to derive knowledge with in a certain period of time. Stream processing paradigm is used by IoT to provide requirements of IoT applications. In a stream processing paradigm, unlike traditional data bases, data is not stored but rather processed as it is generated. To transfer generated data from distributed data sources to a processing center such as cloud may not allow for real-time processing due to the network delay. Another high-demand application is live streaming of video. The performance of live video stream systems is inferior when there is a sudden large demand in the number of users. This thesis addresses some of the limitations of current architectures for video streaming systems and IoT applications based on the use of nearby computing resources (e.g., cloudlet, fog). First, we addressed the degrading performance in video stream systems when a flash crowd occurs. The performance of video streaming systems is affected by flash crowd and degrade the quality of service for subscribers to the content delivery system. A flash crowd happens when there is a sudden large increase in the number of users. Therefore, flash crowds increase network traffic for any particular server. The main challenge is to make sure that the video streaming system has sufficient capacity to handle the occurrence of flash crowds. Second, we address the limitation of current architectures for running mobile applications by introducing a dynamic partitioning and offloading of a mobile application. Mobile devices have limited resources including short battery life, storage capacity and processor performance. This limits the applications that can run on it. Mobile applications can be partitioned so that some of the application runs on a cloud. This works well for applications with relatively little data to be transferred and that do not have a high level of interaction with the user. Challenges with applications that have large amounts of data to be transferred and have a high level interactiveness is the high latency incurred by the network and packet loss of the wireless network. A mobile application can be partitioned so that part of it runs on a nearby computing resource e.g., fog node or cloudlet. This thesis presents a framework that introduces fine-grained offloading approach and support for runtime and dynamic partitioning of an application. Third, we present a solution for placement of stream operators over distributed fog nodes for live processing of data streams from geographically distributed data sources. This placement of stream operators takes place in such a way that it supports applications with a high volume of data that require real-time (or near real-time) analysis To this end, this thesis proposed a set of algorithms for placement of stream operators among fog nodes

System Abstractions for Scalable Application Development at the Edge

Recent years have witnessed an explosive growth of Internet of Things (IoT) devices, which collect or generate huge amounts of data. Given diverse device capabilities and application requirements, data processing takes place across a range of settings, from on-device to a nearby edge server/cloud and remote cloud. Consequently, edge-cloud coordination has been studied extensively from the perspectives of job placement, scheduling and joint optimization. Typical approaches focus on performance optimization for individual applications. This often requires domain knowledge of the applications, but also leads to application-specific solutions. Application development and deployment over diverse scenarios thus incur repetitive manual efforts. There are two overarching challenges to provide system-level support for application development at the edge. First, there is inherent heterogeneity at the device hardware level. The execution settings may range from a small cluster as an edge cloud to on-device inference on embedded devices, differing in hardware capability and programming environments. Further, application performance requirements vary significantly, making it even more difficult to map different applications to already heterogeneous hardware. Second, there are trends towards incorporating edge and cloud and multi-modal data. Together, these add further dimensions to the design space and increase the complexity significantly. In this thesis, we propose a novel framework to simplify application development and deployment over a continuum of edge to cloud. Our framework provides key connections between different dimensions of design considerations, corresponding to the application abstraction, data abstraction and resource management abstraction respectively. First, our framework masks hardware heterogeneity with abstract resource types through containerization, and abstracts away the application processing pipelines into generic flow graphs. Further, our framework further supports a notion of degradable computing for application scenarios at the edge that are driven by multimodal sensory input. Next, as video analytics is the killer app of edge computing, we include a generic data management service between video query systems and a video store to organize video data at the edge. We propose a video data unit abstraction based on a notion of distance between objects in the video, quantifying the semantic similarity among video data. Last, considering concurrent application execution, our framework supports multi-application offloading with device-centric control, with a userspace scheduler service that wraps over the operating system scheduler