Semi-supervised wildfire smoke detection based on smoke-aware consistency

The semi-transparency property of smoke integrates it highly with the background contextual information in the image, which results in great visual differences in different areas. In addition, the limited annotation of smoke images from real forest scenarios brings more challenges for model training. In this paper, we design a semi-supervised learning strategy, named smokeaware consistency (SAC), to maintain pixel and context perceptual consistency in different backgrounds. Furthermore, we propose a smoke detection strategy with triple classification assistance for smoke and smoke-like object discrimination. Finally, we simplified the LFNet fire-smoke detection network to LFNet-v2, due to the proposed SAC and triple classification assistance that can perform the functions of some specific module. The extensive experiments validate that the proposed method significantly outperforms state-of-the-art object detection algorithms on wildfire smoke datasets and achieves satisfactory performance under challenging weather conditions.Peer ReviewedPostprint (published version

A Data-Flow Middleware Platform for Real-Time Video Analysis

2015 - 2016In this thesis we introduce a new software platform for the development of real-time video analysis applications, that has been designed to simplify the realization and the deployment of intelligent video-surveillance systems. The platform has been developed following the Plugin Design Pattern: there is an applicationindependent middleware, providing general purpose services, and a collection of dynamically loaded modules (plugins) carrying out domain-specific tasks. Each plugin defines a set of node types, that can be instantiated to form a processing network, according to the data-flow paradigm: the control of the execution flow is not wired in the application-specific code but is demanded to the middleware, which activates each node as soon as its inputs are available and a processor is ready. A first benefit of this architecture is its impact on the software development process: the plugins are loosely coupled components that are easier to develop and test, and easier to reuse in a different project. A second benefit, due to the shift of the execution control to the middleware, is the performance improvement, since the middleware can automatically parallelize the processing using the available processors or cores, as well as using the same information or data for different thread of execution. In order to validate the proposed software architecture, in terms of both performance and services provided by the middleware, we have undertaken the porting to the new middleware of two novel intelligent surveillance applications, by implementing all the nodes required by the algorithms. The first application is an intelligent video surveillance system based on people tracking algorithm. The application uses a single, fixed camera; on the video stream produced by the camera, background subtraction is performed (with a dynamically updated background) to detect foreground objects. These objects are tracked, and their trajectories are used to detect events of interest, like entering a forbidden area, transiting on a one-way passage in the wrong direction, abandoning objects and so on. The second application integrated is a fire detection algorithm, which combines information based on color, shape and movement in order to detect the flame. Two main novelties have been introduced: first, complementary information, respectively based on color, shape variation and motion analysis, are combined by a multi expert system. The main advantage deriving from this approach lies in the fact that the overall performance of the system significantly increases with a relatively small effort made by designer. Second, a novel descriptor based on a bag-of-words approach has been proposed for representing motion. The proposed method has been tested on a very large dataset of fire videos acquired both in real environments and from the web. The obtained results confirm a consistent reduction in the number of false positives, without paying in terms of accuracy or renouncing the possibility to run the system on embedded platforms. [edited by author]In questa tesi introduciamo una nuova piattaforma software per lo sviluppo di applicazioni di video analisi, progettato per semplificare lo sviluppo e la messa in opera di un sistema di video analisi intelligente. La piattaforma è stata sviluppata seguendo il Design Pattern Plugin: c’è un middleware indipendente dalla piattaforma che mette a disposizione servizi per vari scopi, ed una collezione di moduli caricati dinamicamente (plugin) per la risoluzione di specifici task. Ogni plugin definisce un set di tipi di nodi, che possono essere istanziati per formare una rete di elaborazione, in accordo al paradigma data-flow: Il controllo del flusso di esecuzione non è cablato nel codice specifico dell'applicazione ma viene richiesto al middleware che attiva ogni nodo non appena i suoi ingressi sono disponibili e un processore è pronto. Un primo vantaggio di questa architettura è il suo impatto sul processo di sviluppo del software: i plugin sono componenti poco accoppiati che sono più facili da sviluppare e testare e più facilmente riutilizzabili in un altro progetto. Un secondo beneficio, dovuto allo spostamento del controllo di esecuzione al middleware, è il miglioramento delle prestazioni, dal momento che il middleware può automaticamente parallelizzare l'elaborazione utilizzando i processori o i core disponibili, nonché utilizzando le stesse informazioni o dati per differenti thread di esecuzione . Al fine di convalidare l'architettura software proposta, sia in termini di prestazioni che di servizi forniti dal middleware, è stato effettuato il porting all’interno del middleware di due applicazioni di sorveglianza intelligenti, implementando tutti i nodi richiesti dagli algoritmi. La prima applicazione è un sistema di videosorveglianza intelligente basato su un algoritmo di tracking delle persone. L'applicazione utilizza una singola telecamera fissa; sul flusso video prodotto dalla telecamera viene eseguita una sottrazione del background (con un aggiornamento dinamicamente del backgroung) per rilevare oggetti di foreground. Questi oggetti vengono tracciati e le loro traiettorie vengono utilizzate per rilevare eventi di interesse, come accesso in una zona proibita, oggetti abbandonati e così via. La seconda applicazione integrata è un algoritmo di rilevazione del fuoco che combina informazioni basate su colore, forma e movimento per rilevare le fiamme. Sono state introdotte due novità principali: in primo luogo, informazioni complementari, rispettivamente basate sul colore, sulla variazione di forma e sull'analisi del movimento, sono combinate tra loro da un sistema multi-esperto. Il vantaggio principale derivante da questo approccio risiede nel fatto che le prestazioni complessive del sistema aumentano significativamente con uno sforzo relativamente piccolo. In secondo luogo, un innovativo descrittore basato su un approccio "bag-of-words" per rappresentare il movimento. Il metodo proposto è stato testato su un grande dataset di video acquisiti sia in ambienti reali che dal web. I risultati ottenuti confermano una consistente riduzione del numero di falsi positivi, senza pagare in termini di precisione o rinunciare alla possibilità di eseguire il sistema su piattaforme embedded. [a cura dell'autore]XXIX n.s

Efficient deep CNN-based fire detection and localization in video surveillance applications

Convolutional neural networks (CNNs) have yielded state-of-the-art performance in image classification and other computer vision tasks. Their application in fire detection systems will substantially improve detection accuracy, which will eventually minimize fire disasters and reduce the ecological and social ramifications. However, the major concern with CNN-based fire detection systems is their implementation in real-world surveillance networks, due to their high memory and computational requirements for inference. In this paper, we propose an original, energy-friendly, and computationally efficient CNN architecture, inspired by the SqueezeNet architecture for fire detection, localization, and semantic understanding of the scene of the fire. It uses smaller convolutional kernels and contains no dense, fully connected layers, which helps keep the computational requirements to a minimum. Despite its low computational needs, the experimental results demonstrate that our proposed solution achieves accuracies that are comparable to other, more complex models, mainly due to its increased depth. Moreover, this paper shows how a tradeoff can be reached between fire detection accuracy and efficiency, by considering the specific characteristics of the problem of interest and the variety of fire data

Efficient Deep CNN-Based Fire Detection and Localisation in Video Surveillance Applications

Convolutional neural networks (CNN) have yielded state-of-the-art performance in image classification and other computer vision tasks. Their application in fire detection systems will substantially improve detection accuracy, which will eventually minimize fire disasters and reduce the ecological and social ramifications. However, the major concern with CNN-based fire detection systems is their implementation in real-world surveillance networks, due to their high memory and computational requirements for inference. In this work, we propose an energy-friendly and computationally efficient CNN architecture, inspired by the SqueezeNet architecture for fire detection, localization, and semantic understanding of the scene of the fire. It uses smaller convolutional kernels and contains no dense, fully connected layers, which helps keep the computational requirements to a minimum. Despite its low computational needs, the experimental results demonstrate that our proposed solution achieves accuracies that are comparable to other, more complex models, mainly due to its increased depth. Moreover, the paper shows how a trade-off can be reached between fire detection accuracy and efficiency, by considering the specific characteristics of the problem of interest and the variety of fire data

Facilitating Internet of Things on the Edge

The evolution of electronics and wireless technologies has entered a new era, the Internet of Things (IoT). Presently, IoT technologies influence the global market, bringing benefits in many areas, including healthcare, manufacturing, transportation, and entertainment. Modern IoT devices serve as a thin client with data processing performed in a remote computing node, such as a cloud server or a mobile edge compute unit. These computing units own significant resources that allow prompt data processing. The user experience for such an approach relies drastically on the availability and quality of the internet connection. In this case, if the internet connection is unavailable, the resulting operations of IoT applications can be completely disrupted. It is worth noting that emerging IoT applications are even more throughput demanding and latency-sensitive which makes communication networks a practical bottleneck for the service provisioning. This thesis aims to eliminate the limitations of wireless access, via the improvement of connectivity and throughput between the devices on the edge, as well as their network identification, which is fundamentally important for IoT service management. The introduction begins with a discussion on the emerging IoT applications and their demands. Subsequent chapters introduce scenarios of interest, describe the proposed solutions and provide selected performance evaluation results. Specifically, we start with research on the use of degraded memory chips for network identification of IoT devices as an alternative to conventional methods, such as IMEI; these methods are not vulnerable to tampering and cloning. Further, we introduce our contributions for improving connectivity and throughput among IoT devices on the edge in a case where the mobile network infrastructure is limited or totally unavailable. Finally, we conclude the introduction with a summary of the results achieved

Cyber-Physical Threat Intelligence for Critical Infrastructures Security

Modern critical infrastructures can be considered as large scale Cyber Physical Systems (CPS). Therefore, when designing, implementing, and operating systems for Critical Infrastructure Protection (CIP), the boundaries between physical security and cybersecurity are blurred. Emerging systems for Critical Infrastructures Security and Protection must therefore consider integrated approaches that emphasize the interplay between cybersecurity and physical security techniques. Hence, there is a need for a new type of integrated security intelligence i.e., Cyber-Physical Threat Intelligence (CPTI). This book presents novel solutions for integrated Cyber-Physical Threat Intelligence for infrastructures in various sectors, such as Industrial Sites and Plants, Air Transport, Gas, Healthcare, and Finance. The solutions rely on novel methods and technologies, such as integrated modelling for cyber-physical systems, novel reliance indicators, and data driven approaches including BigData analytics and Artificial Intelligence (AI). Some of the presented approaches are sector agnostic i.e., applicable to different sectors with a fair customization effort. Nevertheless, the book presents also peculiar challenges of specific sectors and how they can be addressed. The presented solutions consider the European policy context for Security, Cyber security, and Critical Infrastructure protection, as laid out by the European Commission (EC) to support its Member States to protect and ensure the resilience of their critical infrastructures. Most of the co-authors and contributors are from European Research and Technology Organizations, as well as from European Critical Infrastructure Operators. Hence, the presented solutions respect the European approach to CIP, as reflected in the pillars of the European policy framework. The latter includes for example the Directive on security of network and information systems (NIS Directive), the Directive on protecting European Critical Infrastructures, the General Data Protection Regulation (GDPR), and the Cybersecurity Act Regulation. The sector specific solutions that are described in the book have been developed and validated in the scope of several European Commission (EC) co-funded projects on Critical Infrastructure Protection (CIP), which focus on the listed sectors. Overall, the book illustrates a rich set of systems, technologies, and applications that critical infrastructure operators could consult to shape their future strategies. It also provides a catalogue of CPTI case studies in different sectors, which could be useful for security consultants and practitioners as well

Sofie: Smart Operating System For Internet Of Everything

The proliferation of Internet of Things and the success of rich cloud services have pushed the horizon of a new computing paradigm, Edge computing, which calls for processing the data at the edge of the network. Applications such as cloud offloading, smart home, and smart city are idea area for Edge computing to achieve better performance than cloud computing. Edge computing has the potential to address the concerns of response time requirement, battery life constraint, bandwidth cost saving, as well as data safety and privacy. However, there are still some challenges for applying Edge computing in our daily life. The missing of the specialized operating system for Edge computing is holding back the flourish of Edge computing applications. Service management, device management, component selection as well as data privacy and security is also not well supported yet in the current computing structure. To address the challenges for Edge computing systems and applications in these aspects, we have planned a series of empirical and theoretical research. We propose SOFIE: Smart Operating System For Internet Of Everything. SOFIE is the operating system specialized for Edge computing running on the Edge gateway. SOFIE could establish and maintain a reliable connection between cloud and Edge device to handle the data transportation between gateway and Edge devices; to provide service management and data management for Edge applications; to protect data privacy and security for Edge users; to guarantee the wellness of the Edge devices. Moreover, SOFIE also provide a naming mechanism to connect Edge device more efficiently. To solve the component selection problem in Edge computing paradigm, SOFIE also include our previous work, SURF, as a model to optimize the performance of the system. Finally, we deployed the design of SOFIE on an IoT/M2M system and support semantics with access control

Cyber-Physical Threat Intelligence for Critical Infrastructures Security

Modern critical infrastructures can be considered as large scale Cyber Physical Systems (CPS). Therefore, when designing, implementing, and operating systems for Critical Infrastructure Protection (CIP), the boundaries between physical security and cybersecurity are blurred. Emerging systems for Critical Infrastructures Security and Protection must therefore consider integrated approaches that emphasize the interplay between cybersecurity and physical security techniques. Hence, there is a need for a new type of integrated security intelligence i.e., Cyber-Physical Threat Intelligence (CPTI). This book presents novel solutions for integrated Cyber-Physical Threat Intelligence for infrastructures in various sectors, such as Industrial Sites and Plants, Air Transport, Gas, Healthcare, and Finance. The solutions rely on novel methods and technologies, such as integrated modelling for cyber-physical systems, novel reliance indicators, and data driven approaches including BigData analytics and Artificial Intelligence (AI). Some of the presented approaches are sector agnostic i.e., applicable to different sectors with a fair customization effort. Nevertheless, the book presents also peculiar challenges of specific sectors and how they can be addressed. The presented solutions consider the European policy context for Security, Cyber security, and Critical Infrastructure protection, as laid out by the European Commission (EC) to support its Member States to protect and ensure the resilience of their critical infrastructures. Most of the co-authors and contributors are from European Research and Technology Organizations, as well as from European Critical Infrastructure Operators. Hence, the presented solutions respect the European approach to CIP, as reflected in the pillars of the European policy framework. The latter includes for example the Directive on security of network and information systems (NIS Directive), the Directive on protecting European Critical Infrastructures, the General Data Protection Regulation (GDPR), and the Cybersecurity Act Regulation. The sector specific solutions that are described in the book have been developed and validated in the scope of several European Commission (EC) co-funded projects on Critical Infrastructure Protection (CIP), which focus on the listed sectors. Overall, the book illustrates a rich set of systems, technologies, and applications that critical infrastructure operators could consult to shape their future strategies. It also provides a catalogue of CPTI case studies in different sectors, which could be useful for security consultants and practitioners as well

Ambient Intelligence for Next-Generation AR

Next-generation augmented reality (AR) promises a high degree of context-awareness - a detailed knowledge of the environmental, user, social and system conditions in which an AR experience takes place. This will facilitate both the closer integration of the real and virtual worlds, and the provision of context-specific content or adaptations. However, environmental awareness in particular is challenging to achieve using AR devices alone; not only are these mobile devices' view of an environment spatially and temporally limited, but the data obtained by onboard sensors is frequently inaccurate and incomplete. This, combined with the fact that many aspects of core AR functionality and user experiences are impacted by properties of the real environment, motivates the use of ambient IoT devices, wireless sensors and actuators placed in the surrounding environment, for the measurement and optimization of environment properties. In this book chapter we categorize and examine the wide variety of ways in which these IoT sensors and actuators can support or enhance AR experiences, including quantitative insights and proof-of-concept systems that will inform the development of future solutions. We outline the challenges and opportunities associated with several important research directions which must be addressed to realize the full potential of next-generation AR.Comment: This is a preprint of a book chapter which will appear in the Springer Handbook of the Metavers

Performance Evaluation of the Object Detection Algorithms on Embedded Devices

Edge computing has seen a dramatic rise in demand, driven by the necessity for real-time, low-latency applications across various domains from autonomous vehicles to surveillance systems. Among these, real-time object detection stands as a crucial technology. However, the inherent constraints of edge devices, including limited computational power, present significant challenges. This thesis provides a comprehensive evaluation of several Convolutional Neural Networks based object detection models when deployed on resource-constrained edge devices, specifically Raspberry Pi and Google’s Coral TPU. The models examined include EfficientDet, YOLO, and variants of the MobileNet family combined with SSD for object detection tasks. We developed a novel benchmarking framework that allowed the evaluation of these models under different configurations, enabling an accurate assessment of their performance characteristics. The benchmarking framework and the metrics used for evaluation can provide a foundation for future work, focusing on the design and deployment of efficient real-time object detection models on edge devices. The performance of these models was scrutinized based on an exhaustive set of metrics including processing speed (frames per second), model accuracy (F1 score), energy consumption, CPU utilization, memory footprint, and device temperature. A novel benchmarking framework was developed to evaluate these models under diverse configurations, providing a precise assessment of their respective performance characteristics. This benchmarking framework, along with the evaluation metrics, sets the foundation for future research concentrating on the design and deployment of efficient real-time object detection models on edge devices. The findings of this study underscore the fact that no single model is a universal solution for all edge applications; instead, the choice of model is heavily dependent on the specific requirements and constraints of the given application. By offering a detailed overview of the performance traits of each model, we aim to guide practitioners in making informed decisions when deploying object detection models in edge computing environments. This work sets the stage for future exploration in the development of more efficient and effective models for real-time object detection on edge device