Deep learning-based anomalous object detection system powered by microcontroller for PTZ cameras

Automatic video surveillance systems are usually designed to detect anomalous objects being present in a scene or behaving dangerously. In order to perform adequately, they must incorporate models able to achieve accurate pattern recognition in an image, and deep learning neural networks excel at this task. However, exhaustive scan of the full image results in multiple image blocks or windows to analyze, which could make the time performance of the system very poor when implemented on low cost devices. This paper presents a system which attempts to detect abnormal moving objects within an area covered by a PTZ camera while it is panning. The decision about the block of the image to analyze is based on a mixture distribution composed of two components: a uniform probability distribution, which represents a blind random selection, and a mixture of Gaussian probability distributions. Gaussian distributions represent windows in the image where anomalous objects were detected previously and contribute to generate the next window to analyze close to those windows of interest. The system is implemented on a Raspberry Pi microcontroller-based board, which enables the design and implementation of a low-cost monitoring system that is able to perform image processing.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Real-time camera operation and tracking for the streaming of teaching activities

The primary driving force of this work comes from the Lab’s urgent needs to offer students the opportunity to attend a remote event from home or anywhere in the world in real-time. The main objective of this work is to build a real-time tracker to follow the movements of the lecturer. After that we will build a framework to handle a PTZ (Pan Tilt and Zoom) camera based on the lecturer movements. That is, if the lecturer goes to the left, the camera will turn to the left. To tackle this project we will follow a project developed by Gebrehiwot, A. which involved building a real-time tracker. The problem of this tracker is that was implemented on Ubuntu and running with a very complex CNN which required the use a good GPU on our computer. As Gebrehiwot, A. rightly points out at the end of his report, not everyone has an Ubuntu partition or a GPU on their computers so we started moving the real time tracker to Windows. To achieve this objective we used Anaconda Windows which made our work much easier. After that we implemented a lightweight backbone of the tracker allowing us to run it on computers with a fewer processing power. Once that all this process was done, we put into practice the mentioned framework for handling the movement of the PTZ camera. This framework uses the implemented lightweight tracker to follow the lecturer moves and depending on these movements the camera will pan and tilt automatically. We tested this framework on streaming platforms like YouTube proving that can greatly improve the quality of online classes. Finally we draw conclusions from the work done and propose future work to improve the framework

Real-Time, Multiple Pan/Tilt/Zoom Computer Vision Tracking and 3D Positioning System for Unmanned Aerial System Metrology

The study of structural characteristics of Unmanned Aerial Systems (UASs) continues to be an important field of research for developing state of the art nano/micro systems. Development of a metrology system using computer vision (CV) tracking and 3D point extraction would provide an avenue for making these theoretical developments. This work provides a portable, scalable system capable of real-time tracking, zooming, and 3D position estimation of a UAS using multiple cameras. Current state-of-the-art photogrammetry systems use retro-reflective markers or single point lasers to obtain object poses and/or positions over time. Using a CV pan/tilt/zoom (PTZ) system has the potential to circumvent their limitations. The system developed in this paper exploits parallel-processing and the GPU for CV-tracking, using optical flow and known camera motion, in order to capture a moving object using two PTU cameras. The parallel-processing technique developed in this work is versatile, allowing the ability to test other CV methods with a PTZ system using known camera motion. Utilizing known camera poses, the object\u27s 3D position is estimated and focal lengths are estimated for filling the image to a desired amount. This system is tested against truth data obtained using an industrial system

Development of Automated Incident Detection System Using Existing ATMS CCTV

Indiana Department of Transportation (INDOT) has over 300 digital cameras along highways in populated areas in Indiana. These cameras are used to monitor traffic conditions around the clock, all year round. Currently, the videos from these cameras are observed by human operators. The main objective of this research is to develop an automatic real-time system to monitor traffic conditions using the INDOT CCTV video feeds by a collaborative research team of the Transportation Active Safety Institute (TASI) at Indiana University-Purdue University Indianapolis (IUPUI) and the Traffic Management Center (TMC) of INDOT. In this project, the research team developed the system architecture based on a detailed system requirement analysis. The first prototype of major system components of the system has been implemented. Specifically, the team has successfully accomplished the following: An AI based deep learning algorithm provided in YOLO3 is selected for vehicle detection which generates the best results for daytime videos. The tracking information of moving vehicles is used to derive the locations of roads and lanes. A database is designed as the center place to gather and distribute the information generated from all camera videos. The database provides all information for the traffic incident detection. A web-based Graphical User Interface (GUI) was developed. The automatic traffic incident detection will be implemented after the traffic flow information being derived accurately. The research team is currently in the process of integrating the prototypes of all components of the system together to establish a complete system prototype

Optical Flow Background Estimation for Real-time Pan/tilt Camera Object Tracking

As Computer Vision (CV) techniques develop, pan/tilt camera systems are able to enhance data capture capabilities over static camera systems. In order for these systems to be effective for metrology purposes, they will need to respond to the test article in real-time with a minimum of additional uncertainty. A methodology is presented here for obtaining high-resolution, high frame-rate images, of objects traveling at speeds ⩾1.2 m/s at 1 m from the camera by tracking the moving texture of an object. Strong corners are determined and used as flow points using implementations on a graphic processing unit (GPU), resulting in significant speed-up over central processing units (CPU). Based on directed pan/tilt motion, a pixel-to-pixel relationship is used to estimate whether optical flow points fit background motion, dynamic motion or noise. To smooth variation, a two-dimensional position and velocity vector is used with a Kalman filter to predict the next required position of the camera so the object stays centered in the image. High resolution images can be stored by a parallel process resulting in a high frame rate procession of images for post-processing. The results provide real-time tracking on a portable system using a pan/tilt unit for generic moving targets where no training is required and camera motion is observed from high accuracy encoders opposed to image correlation

Eagle: End-to-end Deep Reinforcement Learning based Autonomous Control of PTZ Cameras

Existing approaches for autonomous control of pan-tilt-zoom (PTZ) cameras use multiple stages where object detection and localization are performed separately from the control of the PTZ mechanisms. These approaches require manual labels and suffer from performance bottlenecks due to error propagation across the multi-stage flow of information. The large size of object detection neural networks also makes prior solutions infeasible for real-time deployment in resource-constrained devices. We present an end-to-end deep reinforcement learning (RL) solution called Eagle to train a neural network policy that directly takes images as input to control the PTZ camera. Training reinforcement learning is cumbersome in the real world due to labeling effort, runtime environment stochasticity, and fragile experimental setups. We introduce a photo-realistic simulation framework for training and evaluation of PTZ camera control policies. Eagle achieves superior camera control performance by maintaining the object of interest close to the center of captured images at high resolution and has up to 17% more tracking duration than the state-of-the-art. Eagle policies are lightweight (90x fewer parameters than Yolo5s) and can run on embedded camera platforms such as Raspberry PI (33 FPS) and Jetson Nano (38 FPS), facilitating real-time PTZ tracking for resource-constrained environments. With domain randomization, Eagle policies trained in our simulator can be transferred directly to real-world scenarios.Comment: 20 pages, IoTD

MadEye: Boosting Live Video Analytics Accuracy with Adaptive Camera Configurations

Camera orientations (i.e., rotation and zoom) govern the content that a camera captures in a given scene, which in turn heavily influences the accuracy of live video analytics pipelines. However, existing analytics approaches leave this crucial adaptation knob untouched, instead opting to only alter the way that captured images from fixed orientations are encoded, streamed, and analyzed. We present MadEye, a camera-server system that automatically and continually adapts orientations to maximize accuracy for the workload and resource constraints at hand. To realize this using commodity pan-tilt-zoom (PTZ) cameras, MadEye embeds (1) a search algorithm that rapidly explores the massive space of orientations to identify a fruitful subset at each time, and (2) a novel knowledge distillation strategy to efficiently (with only camera resources) select the ones that maximize workload accuracy. Experiments on diverse workloads show that MadEye boosts accuracy by 2.9-25.7% for the same resource usage, or achieves the same accuracy with 2-3.7x lower resource costs.Comment: 19 pages, 16 figure

Panoramic Background Modeling for PTZ Cameras with Competitive Learning Neural Networks

The construction of a model of the background of a scene still remains as a challenging task in video surveillance systems, in particular for moving cameras. This work presents a novel approach for constructing a panoramic background model based on competitive learning neural networks and a subsequent piecewise linear interpolation by Delaunay triangulation. The approach can handle arbitrary camera directions and zooms for a Pan-Tilt-Zoom (PTZ) camera-based surveillance system. After testing the proposed approach on several indoor sequences, the results demonstrate that the proposed method is effective and suitable to use for real-time video surveillance applications.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Development of artificial neural network-based object detection algorithms for low-cost hardware devices

Finally, the fourth work was published in the “WCCI” conference in 2020 and consisted of an individuals' position estimation algorithm based on a novel neural network model for environments with forbidden regions, named “Forbidden Regions Growing Neural Gas”.The human brain is the most complex, powerful and versatile learning machine ever known. Consequently, many scientists of various disciplines are fascinated by its structures and information processing methods. Due to the quality and quantity of the information extracted from the sense of sight, image is one of the main information channels used by humans. However, the massive amount of video footage generated nowadays makes it difficult to process those data fast enough manually. Thus, computer vision systems represent a fundamental tool in the extraction of information from digital images, as well as a major challenge for scientists and engineers. This thesis' primary objective is automatic foreground object detection and classification through digital image analysis, using artificial neural network-based techniques, specifically designed and optimised to be deployed in low-cost hardware devices. This objective will be complemented by developing individuals' movement estimation methods by using unsupervised learning and artificial neural network-based models. The cited objectives have been addressed through a research work illustrated in four publications supporting this thesis. The first one was published in the “ICAE” journal in 2018 and consists of a neural network-based movement detection system for Pan-Tilt-Zoom (PTZ) cameras deployed in a Raspberry Pi board. The second one was published in the “WCCI” conference in 2018 and consists of a deep learning-based automatic video surveillance system for PTZ cameras deployed in low-cost hardware. The third one was published in the “ICAE” journal in 2020 and consists of an anomalous foreground object detection and classification system for panoramic cameras, based on deep learning and supported by low-cost hardware

Deep learning-based anomalous object detection system for panoramic cameras managed by a Jetson TX2 board

Social conflicts appearing in the media are increas ing public awareness about security issues, resulting in a higher demand of more exhaustive environment monitoring methods. Automatic video surveillance systems are a powerful assistance to public and private security agents. Since the arrival of deep learn ing, object detection and classification systems have experienced a large improvement in both accuracy and versatility. However, deep learning-based object detection and classification systems often require expensive GPU-based hardware to work properly. This paper presents a novel deep learning-based foreground anomalous object detection system for video streams supplied by panoramic cameras, specially designed to build power efficient video surveillance systems. The system optimises the process of searching for anomalous objects through a new potential detection generator managed by three different multivariant homoscedastic distributions. Experimental results obtained after its deployment in a Jetson TX2 board attest the good performance of the system, postulating it as a solvent approach to power saving video surveillance systems.This work is partially supported by the Ministry of Economy and Competitiveness of Spain under grants TIN2016-75097- P and PPIT.UMA.B1.2017. It is also partially supported by the Ministry of Science, Innovation and Universities of Spain under grant RTI2018-094645-B-I00, project name Automated detection with low-cost hardware of unusual activities in video sequences. It is also partially supported by the Autonomous Government of Andalusia (Spain) under project UMA18- FEDERJA-084, project name Detection of anomalous behavior agents by deep learning in low-cost video surveillance intel ligent systems. All of them include funds from the European Regional Development Fund (ERDF). The authors thankfully acknowledge the computer resources, technical expertise and assistance provided by the SCBI (Supercomputing and Bioin formatics) center of the University of Malaga. They also ´ Authorized licensed use limited to: Universidad de Malaga. Downloaded on February 06,2024 at 07:21:43 UTC from IEEE Xplore. Restrictions apply. gratefully acknowledge the support of NVIDIA Corporation with the donation of two Titan X GPUs used for this research. The authors acknowledge the funding from the Universidad de Malaga