Online Action Detection

In online action detection, the goal is to detect the start of an action in a video stream as soon as it happens. For instance, if a child is chasing a ball, an autonomous car should recognize what is going on and respond immediately. This is a very challenging problem for four reasons. First, only partial actions are observed. Second, there is a large variability in negative data. Third, the start of the action is unknown, so it is unclear over what time window the information should be integrated. Finally, in real world data, large within-class variability exists. This problem has been addressed before, but only to some extent. Our contributions to online action detection are threefold. First, we introduce a realistic dataset composed of 27 episodes from 6 popular TV series. The dataset spans over 16 hours of footage annotated with 30 action classes, totaling 6,231 action instances. Second, we analyze and compare various baseline methods, showing this is a challenging problem for which none of the methods provides a good solution. Third, we analyze the change in performance when there is a variation in viewpoint, occlusion, truncation, etc. We introduce an evaluation protocol for fair comparison. The dataset, the baselines and the models will all be made publicly available to encourage (much needed) further research on online action detection on realistic data.Comment: Project page: http://homes.esat.kuleuven.be/~rdegeest/OnlineActionDetection.htm

Finished Genome of the Fungal Wheat Pathogen Mycosphaerella graminicola Reveals Dispensome Structure, Chromosome Plasticity, and Stealth Pathogenesis.

The plant-pathogenic fungus Mycosphaerella graminicola (asexual stage: Septoria tritici) causes septoria tritici blotch, a disease that greatly reduces the yield and quality of wheat. This disease is economically important in most wheat-growing areas worldwide and threatens global food production. Control of the disease has been hampered by a limited understanding of the genetic and biochemical bases of pathogenicity, including mechanisms of infection and of resistance in the host. Unlike most other plant pathogens, M. graminicola has a long latent period during which it evades host defenses. Although this type of stealth pathogenicity occurs commonly in Mycosphaerella and other Dothideomycetes, the largest class of plant-pathogenic fungi, its genetic basis is not known. To address this problem, the genome of M. graminicolawas sequenced completely. The finished genome contains 21 chromosomes, eight of which could be lost with no visible effect on the fungus and thus are dispensable. This eight-chromosome dispensome is dynamic in field and progeny isolates, is different from the core genome in gene and repeat content, and appears to have originated by ancient horizontal transfer from an unknown donor. Synteny plots of the M. graminicola chromosomes versus those of the only other sequenced Dothideomycete, Stagonospora nodorum, revealed conservation of gene content but not order or orientation, suggesting a high rate of intra-chromosomal rearrangement in one or both species. This observed “mesosynteny” is very different from synteny seen between other organisms. A surprising feature of the M. graminicolagenome compared to other sequenced plant pathogens was that it contained very few genes for enzymes that break down plant cell walls, which was more similar to endophytes than to pathogens. The stealth pathogenesis of M. graminicola probably involves degradation of proteins rather than carbohydrates to evade host defenses during the biotrophic stage of infection and may have evolved from endophytic ancestors

Video Interpretation: from Classification to Online Detection

The amount of video data has grown exponentially over the last years. It is not feasible anymore to analyze all this data by hand. Video interpretation methods can automatically find concepts in videos to summarize them or extract relevant parts. In this thesis, we present methods for action recognition, object recognition and online action detection. For action recognition, we design two methods based on 2D dense interest points: one looks for 3D dense interest points, while the other uses trajectories that start on dense interest points. These methods are a suitable alternative to standard dense methods when a lower sampling density is required. For object recognition, we show that it is beneficial to use motion features instead of simply applying still-image recognition methods on key frames. The characteristic motion of objects helps their recognition. In particular, we demonstrate the effectiveness of dense trajectories, commonly used for action recognition, on datasets with animal classes and means of transportation. In online action detection, the input is a video stream. After every frame, a decision needs to be made on what action is currently happening. The system needs to make a decision without having seen the whole action. It becomes more important to recognize the early stages of the actions. We collect a dataset and demonstrate that current standard methods are insufficient to solve this problem. An LSTM seems very suitable for online action detection: it processes the input on a per-frame basis and it can model both long-term and short-term patterns. In practice, however, the detection accuracy is low. We experiment with a series of techniques that could help the LSTM to learn long-term dependencies between actions. A two-stream feedback network, where one stream focuses on interpreting the input and the other on discovering the temporal patterns, works better than a standard LSTM on both artificial and real-life data.De Geest R., ''Video interpretation: from classification to online detection'', Proefschrift voorgedragen tot het behalen van het doctoraat in de ingenieurswetenschappen, KU Leuven, January 2018, Leuven, Belgium.nrpages: 144status: publishe

Spatio-temporal object recognition

Object recognition in video is in most cases solved by extracting keyframes from the video and then applying still image recognition methods on these keyframes only. This procedure largely ignores the temporal dimension. Nevertheless, the way an object moves may hold valuable information on its class. Therefore, in this work, we analyze the effectiveness of different motion descriptors, originally developed for action recognition, in the context of action-invariant object recognition. We conclude that a higher classification accuracy can be obtained when motion descriptors (specifically, HOG and MBH around trajectories) are used in combination with standard static descriptors extracted from keyframes. Since currently no suitable dataset for this problem exists, we introduce two new datasets and make them publicly available

Spatio-temporal object recognition

© Springer International Publishing Switzerland 2015. Object recognition in video is in most cases solved by extracting keyframes from the video and then applying still image recognition methods on these keyframes only. This procedure largely ignores the temporal dimension. Nevertheless, the way an object moves may hold valuable information on its class. Therefore, in this work, we analyze the effectiveness of different motion descriptors, originally developed for action recognition, in the context of action-invariant object recognition. We conclude that a higher classification accuracy can be obtained when motion descriptors (specifically, HOG and MBH around trajectories) are used in combination with standard static descriptors extracted from keyframes. Since currently no suitable dataset for this problem exists, we introduce two new datasets and make them publicly available.De Geest R., Deboeverie F., Philips W., Tuytelaars T., ''Spatio-temporal object recognition'', Advanced concepts for intelligent vision systems - Acivs 2015, 12 pp., October 26-29, 2015, Catania, Italy.status: publishe

Curvature-based human body parts segmentation in physiotherapy

Analysing human sports activity in computer vision requires reliable segmentation of the human body into meaningful parts, such as arms, torso and legs. Therefore, we present a novel strategy for human body segmentation. Firstly, greyscale images of human bodies are divided into smooth intensity patches with an adaptive region growing algorithm based on low-degree polynomial fitting. Then, the key idea in this paper is that human body parts are approximated by nearly cylindrical surfaces, of which the axes of minimum curvature accurately reconstruct the human body skeleton. Next, human body segmentation is qualitatively evaluated with a line segment distance between reconstructed human body skeletons and ground truth skeletons. When compared with human body parts segmentations based on mean shift, normalized cuts and watersheds, the proposed method achieves more accurate segmentations and better reconstructions of human body skeletons

Curvature-based human body parts segmentation in physiotherapy

Copyright © 2015 SCITEPRESS - Science and Technology Publications All rights reserved. Analysing human sports activity in computer vision requires reliable segmentation of the human body into meaningful parts, such as arms, torso and legs. Therefore, we present a novel strategy for human body segmentation. Firstly, greyscale images of human bodies are divided into smooth intensity patches with an adaptive region growing algorithm based on low-degree polynomial fitting. Then, the key idea in this paper is that human body parts are approximated by nearly cylindrical surfaces, of which the axes of minimum curvature accurately reconstruct the human body skeleton. Next, human body segmentation is qualitatively evaluated with a line segment distance between reconstructed human body skeletons and ground truth skeletons. When compared with human body parts segmentations based on mean shift, normalized cuts and watersheds, the proposed method achieves more accurate segmentations and better reconstructions of human body skeletons.Deboeverie F., De Geest R., Tuytelaars T., Veelaert P., Philips W., ''Curvature-based human body parts segmentation in physiotherapy'', Proceedings 10th international conference on computer vision theory and applications - VISAPP 2015, pp. 630-637, March 11-14, 2015, Berlin, Germany.status: publishe

Online action detection

© Springer International Publishing AG 2016. In online action detection, the goal is to detect the start of an action in a video stream as soon as it happens. For instance, if a child is chasing a ball, an autonomous car should recognize what is going on and respond immediately. This is a very challenging problem for four reasons. First, only partial actions are observed. Second, there is a large variability in negative data. Third, the start of the action is unknown, so it is unclear over what time window the information should be integrated. Finally, in real world data, large within-class variability exists. This problem has been addressed before, but only to some extent. Our contributions to online action detection are threefold. First, we introduce a realistic dataset composed of 27 episodes from 6 popular TV series. The dataset spans over 16 h of footage annotated with 30 action classes, totaling 6,231 action instances. Second, we analyze and compare various baseline methods, showing this is a challenging problem for which none of the methods provides a good solution. Third, we analyze the change in performance when there is a variation in viewpoint, occlusion, truncation, etc. We introduce an evaluation protocol for fair comparison. The dataset, the baselines and the models will all be made publicly available to encourage (much needed) further research on online action detection on realistic data.De Geest R., Gavves E., Ghodrati A., Li Z., Snoek C., Tuytelaars T., ''Online action detection'', Lecture notes in computer science, vol. 9909, pp. 269-284, 2016 (14th European conference on computer vision - ECCV 2016, October 11-14, 2016, Amsterdam, The Netherlands).status: publishe