BEHAVE - Behavioral analysis of visual events for assisted living scenarios

International audienceThis paper proposes BEHAVE, a person-centered pipeline for probabilistic event recognition. The proposed pipeline firstly detects the set of people in a video frame, then it searches for correspondences between people in the current and previous frames (i.e., people tracking). Finally, event recognition is carried for each person using proba-bilistic logic models (PLMs, ProbLog2 language). PLMs represent interactions among people, home appliances and semantic regions. They also enable one to assess the probability of an event given noisy observations of the real world. BEHAVE was evaluated on the task of online (non-clipped videos) and open-set event recognition (e.g., target events plus none class) on video recordings of seniors carrying out daily tasks. Results have shown that BEHAVE improves event recognition accuracy by handling missed and partially satisfied logic models. Future work will investigate how to extend PLMs to represent temporal relations among events

Ontology Evolution for Personalized and Adaptive Activity Recognition

Ontology-based knowledge driven Activity Recognition (AR) models play a vital role in realm of Internet of Things (IoTs). However, these models suffer the shortcomings of static nature, inability of self-evolution and lack of adaptivity. Also, AR models cannot be made comprehensive enough to cater all the activities and smart home inhabitants may not be restricted to only those activities contained in AR model. So, AR models may not rightly recognize or infer new activities. In this paper, a framework has been proposed for dynamically capturing the new knowledge from activity patterns to evolve behavioural changes in AR model (i.e. ontology based model). This ontology based framework adapts by learning the specialized and extended activities from existing user-performed activity patterns. Moreover, it can identify new activity patterns previously unknown in AR model, adapt the new properties in existing activity models and enrich ontology model by capturing change representation to enrich ontology model. The proposed framework has been evaluated comprehensively over the metrics of accuracy, statistical heuristics and Kappa Coefficient. A well-known dataset named DAMSH has been used for having an empirical insight to the effectiveness of proposed framework that shows a significant level of accuracy for AR models This paper is a postprint of a paper submitted to and accepted for publication in IET Wireless Sensor Systems and is subject to Institution of Engineering and Technology Copyright. The copy of record is available at the IET Digital Librar

Knowledge and Reasoning for Image Understanding

abstract: Image Understanding is a long-established discipline in computer vision, which encompasses a body of advanced image processing techniques, that are used to locate (“where”), characterize and recognize (“what”) objects, regions, and their attributes in the image. However, the notion of “understanding” (and the goal of artificial intelligent machines) goes beyond factual recall of the recognized components and includes reasoning and thinking beyond what can be seen (or perceived). Understanding is often evaluated by asking questions of increasing difficulty. Thus, the expected functionalities of an intelligent Image Understanding system can be expressed in terms of the functionalities that are required to answer questions about an image. Answering questions about images require primarily three components: Image Understanding, question (natural language) understanding, and reasoning based on knowledge. Any question, asking beyond what can be directly seen, requires modeling of commonsense (or background/ontological/factual) knowledge and reasoning. Knowledge and reasoning have seen scarce use in image understanding applications. In this thesis, we demonstrate the utilities of incorporating background knowledge and using explicit reasoning in image understanding applications. We first present a comprehensive survey of the previous work that utilized background knowledge and reasoning in understanding images. This survey outlines the limited use of commonsense knowledge in high-level applications. We then present a set of vision and reasoning-based methods to solve several applications and show that these approaches benefit in terms of accuracy and interpretability from the explicit use of knowledge and reasoning. We propose novel knowledge representations of image, knowledge acquisition methods, and a new implementation of an efficient probabilistic logical reasoning engine that can utilize publicly available commonsense knowledge to solve applications such as visual question answering, image puzzles. Additionally, we identify the need for new datasets that explicitly require external commonsense knowledge to solve. We propose the new task of Image Riddles, which requires a combination of vision, and reasoning based on ontological knowledge; and we collect a sufficiently large dataset to serve as an ideal testbed for vision and reasoning research. Lastly, we propose end-to-end deep architectures that can combine vision, knowledge and reasoning modules together and achieve large performance boosts over state-of-the-art methods.Dissertation/ThesisDoctoral Dissertation Computer Science 201