Computationally Efficient Target Classification in Multispectral Image Data with Deep Neural Networks

Detecting and classifying targets in video streams from surveillance cameras is a cumbersome, error-prone and expensive task. Often, the incurred costs are prohibitive for real-time monitoring. This leads to data being stored locally or transmitted to a central storage site for post-incident examination. The required communication links and archiving of the video data are still expensive and this setup excludes preemptive actions to respond to imminent threats. An effective way to overcome these limitations is to build a smart camera that transmits alerts when relevant video sequences are detected. Deep neural networks (DNNs) have come to outperform humans in visual classifications tasks. The concept of DNNs and Convolutional Networks (ConvNets) can easily be extended to make use of higher-dimensional input data such as multispectral data. We explore this opportunity in terms of achievable accuracy and required computational effort. To analyze the precision of DNNs for scene labeling in an urban surveillance scenario we have created a dataset with 8 classes obtained in a field experiment. We combine an RGB camera with a 25-channel VIS-NIR snapshot sensor to assess the potential of multispectral image data for target classification. We evaluate several new DNNs, showing that the spectral information fused together with the RGB frames can be used to improve the accuracy of the system or to achieve similar accuracy with a 3x smaller computation effort. We achieve a very high per-pixel accuracy of 99.1%. Even for scarcely occurring, but particularly interesting classes, such as cars, 75% of the pixels are labeled correctly with errors occurring only around the border of the objects. This high accuracy was obtained with a training set of only 30 labeled images, paving the way for fast adaptation to various application scenarios.Comment: Presented at SPIE Security + Defence 2016 Proc. SPIE 9997, Target and Background Signatures I

Smart environment monitoring through micro unmanned aerial vehicles

In recent years, the improvements of small-scale Unmanned Aerial Vehicles (UAVs) in terms of flight time, automatic control, and remote transmission are promoting the development of a wide range of practical applications. In aerial video surveillance, the monitoring of broad areas still has many challenges due to the achievement of different tasks in real-time, including mosaicking, change detection, and object detection. In this thesis work, a small-scale UAV based vision system to maintain regular surveillance over target areas is proposed. The system works in two modes. The first mode allows to monitor an area of interest by performing several flights. During the first flight, it creates an incremental geo-referenced mosaic of an area of interest and classifies all the known elements (e.g., persons) found on the ground by an improved Faster R-CNN architecture previously trained. In subsequent reconnaissance flights, the system searches for any changes (e.g., disappearance of persons) that may occur in the mosaic by a histogram equalization and RGB-Local Binary Pattern (RGB-LBP) based algorithm. If present, the mosaic is updated. The second mode, allows to perform a real-time classification by using, again, our improved Faster R-CNN model, useful for time-critical operations. Thanks to different design features, the system works in real-time and performs mosaicking and change detection tasks at low-altitude, thus allowing the classification even of small objects. The proposed system was tested by using the whole set of challenging video sequences contained in the UAV Mosaicking and Change Detection (UMCD) dataset and other public datasets. The evaluation of the system by well-known performance metrics has shown remarkable results in terms of mosaic creation and updating, as well as in terms of change detection and object detection

MusA: Using Indoor Positioning and Navigation to Enhance Cultural Experiences in a museum

In recent years there has been a growing interest into the use of multimedia mobile guides in museum environments. Mobile devices have the capabilities to detect the user context and to provide pieces of information suitable to help visitors discovering and following the logical and emotional connections that develop during the visit. In this scenario, location based services (LBS) currently represent an asset, and the choice of the technology to determine users' position, combined with the definition of methods that can effectively convey information, become key issues in the design process. In this work, we present MusA (Museum Assistant), a general framework for the development of multimedia interactive guides for mobile devices. Its main feature is a vision-based indoor positioning system that allows the provision of several LBS, from way-finding to the contextualized communication of cultural contents, aimed at providing a meaningful exploration of exhibits according to visitors' personal interest and curiosity. Starting from the thorough description of the system architecture, the article presents the implementation of two mobile guides, developed to respectively address adults and children, and discusses the evaluation of the user experience and the visitors' appreciation of these application

The crowd as a cameraman : on-stage display of crowdsourced mobile video at large-scale events

Recording videos with smartphones at large-scale events such as concerts and festivals is very common nowadays. These videos register the atmosphere of the event as it is experienced by the crowd and offer a perspective that is hard to capture by the professional cameras installed throughout the venue. In this article, we present a framework to collect videos from smartphones in the public and blend these into a mosaic that can be readily mixed with professional camera footage and shown on displays during the event. The video upload is prioritized by matching requests of the event director with video metadata, while taking into account the available wireless network capacity. The proposed framework's main novelty is its scalability, supporting the real-time transmission, processing and display of videos recorded by hundreds of simultaneous users in ultra-dense Wi-Fi environments, as well as its proven integration in commercial production environments. The framework has been extensively validated in a controlled lab setting with up to 1 000 clients as well as in a field trial where 1 183 videos were collected from 135 participants recruited from an audience of 8 050 people. 90 % of those videos were uploaded within 6.8 minutes

S6: a Smart, Social and SDN-based Surveillance System for Smart-cities

Abstract In the last few years, Software Defined Networks (SDN) and Network Functions Virtualization (NFV) have been introduced in the Internet as a new way to design, deploy and manage networking services. Working together, they are able to consolidate and deliver the networking components using standard IT virtualization technologies not only on high-volume servers, but also in end user premises, Telco operator edge and access nodes thus allowing the emergence of new services. In this context, this paper presents a smart video surveillance platform designed to exploit the facilities offered by full SDN-NFV networks. This platform is based on free and open source software running on Provider Equipment (PE), so allowing function deployment simplification and management cost reduction

Is Wide-Area Persistent Surveillance by State and Local Governments Constitutional?

This dissertation addresses the following question: “Can wide-area persistent surveillance (WAPS) developed by the United States military and employed abroad as a tool in the Global War on Terror be employed domestically as a law enforcement tool without violating the US Constitution’s Fourth Amendment?” The most likely and controversial application of WAPS by state and local governments is for law enforcement. Aircraft will loiter over a city persistently taking high-definition photographs to capture locations of unidentified persons with the intent to identify persons and areas of interest for criminal investigations. Based on the Flyover Cases, aerial surveillance has few constitutional limitations which WAPS can be consistent. The key challenge in determining the constitutionality of WAPS depends on the Court’s interpretation of the Fourth Amendment concerning emerging technologies. Legal scholars have suggested various forms of the Mosaic Theory, which was introduced in two concurring opinions in Jones v. United States . The Supreme Court has been reticent to engage new technology’s constitutionality. WAPS is among the less intrusive tools when compared to other emerging technologies like digital information or facial recognition. This research argues why the Courts should view Personal Identifying Information (PII) as the line of reasonable expectations of privacy for WAPS and other emerging technologies. Aerial surveillance by nature, collects passive information, new data is not being created by photographing the happenings in public spaces from an aerial platform. In Carpenter v. United States, the Court ruled that warrantless surveillance of cell site location information (CSLI) for more than seven days was an unreasonable search. However, the court repeatedly referred to CSLI as “unique,” whereas “conventional surveillance and tools, such as security cameras,” are not. WAPS should not be limited by the Constitution for the operational duration, time of day/night, camera resolution, location of collection, altitude, or any other variable at the collection stage of the operations. The analysis and exploitation of WAPS data encounters constitutional limits necessary to protect individuals’ PII absent probable cause standards

The Exploitation of Data from Remote and Human Sensors for Environment Monitoring in the SMAT Project

In this paper, we outline the functionalities of a system that integrates and controls a fleet of Unmanned Aircraft Vehicles (UAVs). UAVs have a set of payload sensors employed for territorial surveillance, whose outputs are stored in the system and analysed by the data exploitation functions at different levels. In particular, we detail the second level data exploitation function whose aim is to improve the sensors data interpretation in the post-mission activities. It is concerned with the mosaicking of the aerial images and the cartography enrichment by human sensors—the social media users. We also describe the software architecture for the development of a mash-up (the integration of information and functionalities coming from the Web) and the possibility of using human sensors in the monitoring of the territory, a field in which, traditionally, the involved sensors were only the hardware ones.JRC.H.6-Digital Earth and Reference Dat

A System for the Generation of Synthetic Wide Area Aerial Surveillance Imagery

The development, benchmarking and validation of aerial Persistent Surveillance (PS) algorithms requires access to specialist Wide Area Aerial Surveillance (WAAS) datasets. Such datasets are difficult to obtain and are often extremely large both in spatial resolution and temporal duration. This paper outlines an approach to the simulation of complex urban environments and demonstrates the viability of using this approach for the generation of simulated sensor data, corresponding to the use of wide area imaging systems for surveillance and reconnaissance applications. This provides a cost-effective method to generate datasets for vehicle tracking algorithms and anomaly detection methods. The system fuses the Simulation of Urban Mobility (SUMO) traffic simulator with a MATLAB controller and an image generator to create scenes containing uninterrupted door-to-door journeys across large areas of the urban environment. This ‘pattern-of-life’ approach provides three-dimensional visual information with natural movement and traffic flows. This can then be used to provide simulated sensor measurements (e.g. visual band and infrared video imagery) and automatic access to ground-truth data for the evaluation of multi-target tracking systems

Human Detection for Flood Rescue: Application of YOLOv5 Algorithm and DeepSort Object Tracking

This thesis proposes a method of human detection using high-resolution surveillance cameras to monitor sections of the Chattahoochee River that require frequent search and rescue efforts due to flooding. The areas of interest are located in the city of Columbus, Georgia. The goals of this study are to evaluate the effectiveness of the YOLO (You Only Look Once) algorithm for human detection on the river as well as to propose future improvements to the city’s existing alert methods in the event of a flood.M.S