A Vision-based Real-time Conductor Gesture Tracking System

[[abstract]]In recent years, interaction between humans and computers is becoming more important. “Virtual Orchestra” is an Human Computer Interface (HCI) software which attempts to authentically reproduce a live orchestra using synthesized and sampled instruments sounds. Compared with the traditional HCIs, using vision-based gesture can provide a touch-free interface which is less bounding than mechanical instruments. In this research, we design a vision-based system that can track the hand motions of a conductor from webcam and extract musical beats from motions. The algorithm used is based on a robust nonparametric technique for climbing density gradients to find the mode of probability distributions. For each frame, the mean shift algorithm converges to the mode of the distribution. Then, the CAMSHIFT algorithm is used to track the moving objects in a video scene. After acquiring the target center point continuously, we can form the trajectory of moving target (such as baton, conductor’s hand…etc). By computing an approximation of k-curvature for the trajectory, and the angle between these two motion vectors, we can compute the point of the change of direction. In this thesis, a system was developed for interpreting a conductor’s gestures and translating theses gestures into musical beats that can be explained as the major part of the music. This system does not require the use of active sensing, special baton, or other constraints on the physical motion of the conductor.

Joint localization of pursuit quadcopters and target using monocular cues

Pursuit robots (autonomous robots tasked with tracking and pursuing a moving target) require accurate tracking of the target's position over time. One possibly effective pursuit platform is a quadcopter equipped with basic sensors and a monocular camera. However, combined noise of the quadcopter's sensors causes large disturbances of target's 3D position estimate. To solve this problem, in this paper, we propose a novel method for joint localization of a quadcopter pursuer with a monocular camera and an arbitrary target. Our method localizes both the pursuer and target with respect to a common reference frame. The joint localization method fuses the quadcopter's kinematics and the target's dynamics in a joint state space model. We show that predicting and correcting pursuer and target trajectories simultaneously produces better results than standard approaches to estimating relative target trajectories in a 3D coordinate system. Our method also comprises a computationally efficient visual tracking method capable of redetecting a temporarily lost target. The efficiency of the proposed method is demonstrated by a series of experiments with a real quadcopter pursuing a human. The results show that the visual tracker can deal effectively with target occlusions and that joint localization outperforms standard localization methods

Comparison between gaze and moving objects in videos for smooth pursuit eye movement evaluation

When viewing moving objects in videos the movement of the eyes is called smooth pursuit. For evaluating the relationship of eye tracking data to the moving objects, the objects in the videos need to be detected and tracked. In the first part of this thesis, a method for detecting and tracking of moving objects in videos is developed. The method mainly consists of a modified version of the Gaussian mixture model, The Tracking feature point method, a modified version of the Mean shift algorithm, Matlabs function bwlabel and a set of new developed methods. The performance of the method is highest when the background is static and the objects differ in colour from the background. The false detection rate increases, when the video environment becomes more dynamic and complex. In the second part of this thesis the distance between the point of gaze and the moving objects centre point is calculated. The eyes may not always follow the centre position of an object, but rather some other part of the object. Therefore, the method gives more satisfactory result when the objects are small.Utvärdering av smooth pursuit-rörelser. En jämförelse mellan ögonrörelser och rörliga objekt i videosekvenser Populärvetenskaplig sammanfattning av examensarbetet: Andrea Åkerström Ett forskningsområde som har vuxit mycket de senaste åren är ”eye tracking”: en teknik för att undersöka ögonrörelser. Tekniken har visat sig intressant för studier inom exempelvis visuella system, i psykologi och i interaktioner mellan datorer och människor. Ett eye tracking system mäter ögonens rörelser så att de punkterna ögat tittar på kan bli estimerade. Tidigare har de flesta studier inom eye tracking baserats på bilder, men på senare tid har även intresset för att studera filmsekvenser vuxit. Den typ av rörelse som ögat utför när det följer ett rörligt objekt kallas för smooth pursuitrörelse. En av svårigheterna med att utvärdera relationen mellan eye tracking-data och rörliga objekten i filmer är att objekten, antingen manuellt mäts ut eller att ett intelligent system utvecklas för en automatisk utvärdering. Det som gör processen att detektera och följa rörliga objekt i filmer komplex är att olika videosekvenser kan ha många olika typer av svåra videoscenarion som metoden måste klara av. Till exempel kan bakgrunden i en video vara dynamisk, det kan finnas störningar som regn eller snö, eller kan problemet vara att kameran skakar eller rör sig. Syftet med detta arbete består av två delar. Den först delen, som också har varit den största, har varit att utveckla en metod som kan detektera och följa rörliga objekt i olika typer av videosekvenser, baserad på metoder från tidigare forskning. Den andra delen har varit att försöka utveckla en automatisk utvärdering av ögonrörelsen smooth persuit, genom att använda de detekterade och följda objekten i videosekvenserna tillsammans med redan existerande ögondata. För att utveckla den metod har olika metoder från tidigare forskning kombinerat. Alla metoder som har utvecklas i detta område har olika för och nackdelar och fungerade bättre eller sämre för olika typer av videoscenarion. Målet för metoden i detta arbete har varit att hitta en kombination av olika metoder som, genom att kompensera varandras för- och nackdelar, kan ge en så bra detektering som möjligt för olika typer av filmsekvenser. Min metod är till största del uppbyggd av tre metoder: En modifierad version av Guasssian Mixture Model, Tracking Feature Point och en modifierad version av Mean Shift Algorithmen. Guassian Mixture Model-metoden används för att detekterar pixlar i filmen som tillhör objekt som är i rörelse. Metoden tar fram dynamiska modeller av bakgrunden i filmen och detekterar pixlar som skiljer sig från backgrundsmodellerna. Detta är en väl använd metod som kan hantera komplexa bakgrunder med periodiskt brus, men den ger samtidigt ofta upphov till felaktiga detektioner och den kan inte hantera kamerarörelser. För att hantera kamerarörelser används Tracking Feature Point-metoden och på så sätt kompenseras denna brist hos Guassian Mixture Modell-metoden. Tracking Feature Point tar fram ”feature points” ut videobilder och med hjälp av dem kan metoden estimera kameraförflyttningar. Denna metod räknar dock endast ut de förflyttningar som kameran gör, men den tar inte hänsyn till om kameran roterar. Mean Shift Algoritm är en metod som används för att räkna ut det rörliga objektets nya position i en efterföljande bild. För mitt arbete har endast delar av denna metod används till att bestämma vilka detektioner av objekt i de olika bilderna som representerar samma objekt. Genom att ta fram modeller för objekten i varje bild, vilka sedan jämförs, kan metoden bestämma vilka objekt som kan klassas som samma objekt. Den metod som har utvecklat i detta arbete gav bäst resultat när bakgrunden var statisk och objektets färg skiljde sig från bakgrunden. När bakgrunden blir mer dynamisk och komplex ökade mängden falska detektioner och för vissa videosekvenser misslyckas metoden att detektera hela objekten. Den andra delen av detta arbetes syfte var att använda resultatet från metoden för att utvärdera eye tracking-data. Den automatiska utvärderingen av ögonrörelsen smooth pursuit ger ett mått på hur bra ögat kan följa objekt som rör sig. För att utföra detta mäts avståndet mellan den punkt som ögat tittar på och det detekterade objektets centrum. Den automatiskt utvärderingen av smooth pursuit-rörelsen gav bäst resultat när objekten var små. För större objekt följer ögat inte nödvändigtvis objektets mittenpunkt utan istället någon annan del av objektet och metoden kan därför i dessa fall ge ett missvisande resultat. Detta arbete har inte resulterat i en färdig metod utan det finns många områden för förbättringar. Exempelvis skulle en estimering av kamerans rotationer förbättra resultaten. Utvärderingen av hur väl ögat följer rörliga objekt kan även utvecklas mer, genom att konturerna av objekten beräknades. På detta sätt skulle även avståndet mellan punkterna ögat tittar på och objektets area kunnat bestämmas. Både eye tracking och att detektera och följa rörliga objekt i filmer är idag aktiva forskningsområden och det finns alltså fortfarande mycket att utveckla i dessa områden. Syfte med detta arbete har varit att försöka utveckla en mer generell metod som kan fungera för olika typer av filmsekvenser

Target Centroid Position Estimation of Phase-Path Volume Kalman Filtering

For the problem of easily losing track target when obstacles appear in intelligent robot target tracking, this paper proposes a target tracking algorithm integrating reduced dimension optimal Kalman filtering algorithm based on phase-path volume integral with Camshift algorithm. After analyzing the defects of Camshift algorithm, compare the performance with the SIFT algorithm and Mean Shift algorithm, and Kalman filtering algorithm is used for fusion optimization aiming at the defects. Then aiming at the increasing amount of calculation in integrated algorithm, reduce dimension with the phase-path volume integral instead of the Gaussian integral in Kalman algorithm and reduce the number of sampling points in the filtering process without influencing the operational precision of the original algorithm. Finally set the target centroid position from the Camshift algorithm iteration as the observation value of the improved Kalman filtering algorithm to fix predictive value; thus to make optimal estimation of target centroid position and keep the target tracking so that the robot can understand the environmental scene and react in time correctly according to the changes. The experiments show that the improved algorithm proposed in this paper shows good performance in target tracking with obstructions and reduces the computational complexity of the algorithm through the dimension reduction

Object detection and tracking in video image

In recent days, capturing images with high quality and good size is so easy because of rapid improvement in quality of capturing device with less costly but superior technology. Videos are a collection of sequential images with a constant time interval. So video can provide more information about our object when scenarios are changing with respect to time. Therefore, manually handling videos are quite impossible. So we need an automated devise to process these videos. In this thesis one such attempt has been made to track objects in videos. Many algorithms and technology have been developed to automate monitoring the object in a video file. Object detection and tracking is a one of the challenging task in computer vision. Mainly there are three basic steps in video analysis: Detection of objects of interest from moving objects, Tracking of that interested objects in consecutive frames, and Analysis of object tracks to understand their behavior. Simple object detection compares a static background frame at the pixel level with the current frame of video. The existing method in this domain first tries to detect the interest object in video frames. One of the main difficulties in object tracking among many others is to choose suitable features and models for recognizing and tracking the interested object from a video. Some common choice to choose suitable feature to categories, visual objects are intensity, shape, color and feature points. In this thesis, we studied about mean shift tracking based on the color pdf, optical flow tracking based on the intensity and motion; SIFT tracking based on scale invariant local feature points. Preliminary results from experiments have shown that the adopted method is able to track targets with translation, rotation, partial occlusion and deformation

Estimation for Motion in Tracking and Detection Objects with Kalman Filter

The Kalman filter has long been regarded as the optimal solution to many applications in computer vision for example the tracking objects, prediction and correction tasks. Its use in the analysis of visual motion has been documented frequently, we can use in computer vision and open cv in different applications in reality for example robotics, military image and video, medical applications, security in public and privacy society, etc. In this paper, we investigate the implementation of a Matlab code for a Kalman Filter using three algorithm for tracking and detection objects in video sequences (block-matching (Motion Estimation) and Camshift Meanshift (localization, detection and tracking object)). The Kalman filter is presented in three steps: prediction, estimation (correction) and update. The first step is a prediction for the parameters of the tracking and detection objects. The second step is a correction and estimation of the prediction parameters. The important application in Kalman filter is the localization and tracking mono-objects and multi-objects are given in results. This works presents the extension of an integrated modeling and simulation tool for the tracking and detection objects in computer vision described at different models of algorithms in implementation systems

Vision-Aided Navigation for GPS-Denied Environments Using Landmark Feature Identification

In recent years, unmanned autonomous vehicles have been used in diverse applications because of their multifaceted capabilities. In most cases, the navigation systems for these vehicles are dependent on Global Positioning System (GPS) technology. Many applications of interest, however, entail operations in environments in which GPS is intermittent or completely denied. These applications include operations in complex urban or indoor environments as well as missions in adversarial environments where GPS might be denied using jamming technology. This thesis investigate the development of vision-aided navigation algorithms that utilize processed images from a monocular camera as an alternative to GPS. The vision-aided navigation approach explored in this thesis entails defining a set of inertial landmarks, the locations of which are known within the environment, and employing image processing algorithms to detect these landmarks in image frames collected from an onboard monocular camera. These vision-based landmark measurements effectively serve as surrogate GPS measurements that can be incorporated into a navigation filter. Several image processing algorithms were considered for landmark detection and this thesis focuses in particular on two approaches: the continuous adaptive mean shift (CAMSHIFT) algorithm and the adaptable compressive (ADCOM) tracking algorithm. These algorithms are discussed in detail and applied for the detection and tracking of landmarks in monocular camera images. Navigation filters are then designed that employ sensor fusion of accelerometer and rate gyro data from an inertial measurement unit (IMU) with vision-based measurements of the centroids of one or more landmarks in the scene. These filters are tested in simulated navigation scenarios subject to varying levels of sensor and measurement noise and varying number of landmarks. Finally, conclusions and recommendations are provided regarding the implementation of this vision-aided navigation approach for autonomous vehicle navigation systems

Dynamic Data Assimilation

Data assimilation is a process of fusing data with a model for the singular purpose of estimating unknown variables. It can be used, for example, to predict the evolution of the atmosphere at a given point and time. This book examines data assimilation methods including Kalman filtering, artificial intelligence, neural networks, machine learning, and cognitive computing