Evaluation of a Real-Time Voice Order Recognition System from Multiple Audio Channels in a Home

International audienceThe SWEET-HOME project aims at providing audio-based interaction technology that lets the user have full control over their home environment, at detecting distress situations and at easing the social inclusion of the elderly and frail population. This paper presents an overview of the project focusing on the implemented techniques for speech and sound recognition as context-aware decision making with uncertainty. A user experiment in a smart home demonstrates the interest of this audio-based technology

Human Activity Recognition using a Semantic Ontology-Based Framework

In the last years, the extensive use of smart objects embedded in the physical world, in order to monitor and record physical or environmental conditions, has increased rapidly. In this scenario, heterogeneous devices are connected together into a network. Data generated from such system are usually stored in a database, which often shows a lack of semantic information and relationship among devices. Moreover, this set can be incomplete, unreliable, incorrect and noisy. So, it turns out to be important both the integration of information and the interoperability of applications. For this reason, ontologies are becoming widely used to describe the domain and achieve efficient interoperability of information system. An example of the described situation could be represented by Ambient Assisted Living context, which intends to enable older or disabled people to remain living independently longer in their own house. In this contest, human activity recognition plays a main role because it could be considered as starting point to facilitate assistance and care for elderly. Due to the nature of human behavior, it is necessary to manage the time and spatial restrictions. So, we propose a framework that implements a novel methodology based on the integration of an ontology for representing contextual knowledge and a Complex Event Processing engine for supporting timed reasoning. Moreover, it is an infrastructure where knowledge, organized in conceptual spaces (based on its meaning) can be semantically queried, discovered, and shared across applications. In our framework, benefits deriving from the implementation of a domain ontology are exploited into different levels of abstrac- tion. Thereafter, reasoning techniques represent a preprocessing method to prepare data for the final temporal analysis. The results, presented in this paper, have been obtained applying the methodology into AALISABETH, an Ambient Assisted Living project aimed to monitor the lifestyle of old people, not suffering from major chronic diseases or severe disabilities

Using Markov Logic Network for On-line Activity Recognition from Non-Visual Home Automation Sensors

International audienceThis paper presents a Markov Logic Networks (MLN) approach for the on-line recognition of human activities in a smart home. The method recognises activity from non visual and non wearable sensors data. The classification model benefit from a logic formal representation and uses probabilistic inference to deal with uncertainty. The evaluation was carried out on a real smart home where 21 participants performed several daily activities recorded by microphones and several classical home automation sensors. The MLN approach reaches an accuracy of 85.3% while the basleine support vector machine and naive Bayes approches leads to 59.6% and 66.1% respectively. Results show not only the great abilities of MLN as a classifier but also its potential to be integrable into a formal knowledge representation framework

A data-driven situation-aware framework for predictive analysis in smart environments

In the era of Internet of Things (IoT), it is vital for smart environments to be able to efficiently provide effective predictions of user’s situations and take actions in a proactive manner to achieve the highest performance. However, there are two main challenges. First, the sensor environment is equipped with a heterogeneous set of data sources including hardware and software sensors, and oftentimes complex humans as sensors, too. These sensors generate a huge amount of raw data. In order to extract knowledge and do predictive analysis, it is necessary that the raw sensor data be cleaned, understood, analyzed, and interpreted. Second challenge refers to predictive modeling. Traditional predictive models predict situations that are likely to happen in the near future by keeping and analyzing the history of past user’s situations. Traditional predictive analysis approaches have become less effective because of the massive amount of data that both affects data processing efficiency and complicates the data semantics. In this study, we propose a data-driven, situation-aware framework for predictive analysis in smart environments that addresses the above challenges

Recognition of activities of daily living

Activities of daily living (ADL) are things we normally do in daily living, including any daily activity such as feeding ourselves, bathing, dressing, grooming, work, homemaking, and leisure. The ability or inability to perform ADLs can be used as a very practical measure of human capability in many types of disorder and disability. Oftentimes in a health care facility, with the help of observations by nurses and self-reporting by residents, professional staff manually collect ADL data and enter data into the system. Technologies in smart homes can provide some solutions to detecting and monitoring a resident’s ADL. Typically multiple sensors can be deployed, such as surveillance cameras in the smart home environment, and contacted sensors affixed to the resident’s body. Note that the traditional technologies incur costly and laborious sensor deployment, and cause uncomfortable feeling of contacted sensors with increased inconvenience. This work presents a novel system facilitated via mobile devices to collect and analyze mobile data pertaining to the human users’ ADL. By employing only one smart phone, this system, named ADL recognition system, significantly reduces set-up costs and saves manpower. It encapsulates rather sophisticated technologies under the hood, such as an agent-based information management platform integrating both the mobile end and the cloud, observer patterns and a time-series based motion analysis mechanism over sensory data. As a single-point deployment system, ADL recognition system provides further benefits that enable the replay of users’ daily ADL routines, in addition to the timely assessment of their life habits

Interaction Analysis in Smart Work Environments through Fuzzy Temporal Logic

Interaction analysis is defined as the generation of situation descriptions from machine perception. World models created through machine perception are used by a reasoning engine based on fuzzy metric temporal logic and situation graph trees, with optional parameter learning and clustering as preprocessing, to deduce knowledge about the observed scene. The system is evaluated in a case study on automatic behavior report generation for staff training purposes in crisis response control rooms

Intelligent home automation security system based on novel logical sensing and behaviour prediction

The thesis, Intelligent Home Automation Security System Based on Novel Logical Sensing and Behavior Prediction, was designed to enhance authentication, authorization and security in smart home devices and services. The work proposes a three prong defensive strategy each of which are analyzed and evaluated separately to drastically improve security. The Device Fingerprinting techniques proposed, not only improves the existing approaches but also identifies the physical device accessing the home cybernetic and mechatronic systems using device specific and browser specific parameters. The Logical Sensing process analyses home inhabitant actions from a logical stand point and develops sophisticated and novel sensing techniques to identify intrusion attempts to a home’s physical and cyber space. Novel Behavior prediction methodology utilizes Bayesian networks to learn normal user behavior which is later compared to distinguish and identify suspicious user behaviors in the home in a timely manner. The logical sensing, behavior prediction and device fingerprinting techniques proposed were successfully tested, evaluated and verified in an actual home cyber physical system. The algorithms and techniques proposed in the thesis can be easily modified and adapted into many practical applications in Industrial Internet of Things, Industry 4.0 and cyber-physical systems.Thesis (PhD)--University of Pretoria, 2017.Electrical, Electronic and Computer EngineeringPhDUnrestricte

State of the art of audio- and video based solutions for AAL

Working Group 3. Audio- and Video-based AAL ApplicationsIt is a matter of fact that Europe is facing more and more crucial challenges regarding health and social care due to the demographic change and the current economic context. The recent COVID-19 pandemic has stressed this situation even further, thus highlighting the need for taking action. Active and Assisted Living (AAL) technologies come as a viable approach to help facing these challenges, thanks to the high potential they have in enabling remote care and support. Broadly speaking, AAL can be referred to as the use of innovative and advanced Information and Communication Technologies to create supportive, inclusive and empowering applications and environments that enable older, impaired or frail people to live independently and stay active longer in society. AAL capitalizes on the growing pervasiveness and effectiveness of sensing and computing facilities to supply the persons in need with smart assistance, by responding to their necessities of autonomy, independence, comfort, security and safety. The application scenarios addressed by AAL are complex, due to the inherent heterogeneity of the end-user population, their living arrangements, and their physical conditions or impairment. Despite aiming at diverse goals, AAL systems should share some common characteristics. They are designed to provide support in daily life in an invisible, unobtrusive and user-friendly manner. Moreover, they are conceived to be intelligent, to be able to learn and adapt to the requirements and requests of the assisted people, and to synchronise with their specific needs. Nevertheless, to ensure the uptake of AAL in society, potential users must be willing to use AAL applications and to integrate them in their daily environments and lives. In this respect, video- and audio-based AAL applications have several advantages, in terms of unobtrusiveness and information richness. Indeed, cameras and microphones are far less obtrusive with respect to the hindrance other wearable sensors may cause to one’s activities. In addition, a single camera placed in a room can record most of the activities performed in the room, thus replacing many other non-visual sensors. Currently, video-based applications are effective in recognising and monitoring the activities, the movements, and the overall conditions of the assisted individuals as well as to assess their vital parameters (e.g., heart rate, respiratory rate). Similarly, audio sensors have the potential to become one of the most important modalities for interaction with AAL systems, as they can have a large range of sensing, do not require physical presence at a particular location and are physically intangible. Moreover, relevant information about individuals’ activities and health status can derive from processing audio signals (e.g., speech recordings). Nevertheless, as the other side of the coin, cameras and microphones are often perceived as the most intrusive technologies from the viewpoint of the privacy of the monitored individuals. This is due to the richness of the information these technologies convey and the intimate setting where they may be deployed. Solutions able to ensure privacy preservation by context and by design, as well as to ensure high legal and ethical standards are in high demand. After the review of the current state of play and the discussion in GoodBrother, we may claim that the first solutions in this direction are starting to appear in the literature. A multidisciplinary 4 debate among experts and stakeholders is paving the way towards AAL ensuring ergonomics, usability, acceptance and privacy preservation. The DIANA, PAAL, and VisuAAL projects are examples of this fresh approach. This report provides the reader with a review of the most recent advances in audio- and video-based monitoring technologies for AAL. It has been drafted as a collective effort of WG3 to supply an introduction to AAL, its evolution over time and its main functional and technological underpinnings. In this respect, the report contributes to the field with the outline of a new generation of ethical-aware AAL technologies and a proposal for a novel comprehensive taxonomy of AAL systems and applications. Moreover, the report allows non-technical readers to gather an overview of the main components of an AAL system and how these function and interact with the end-users. The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted. The report ends with an overview of the challenges, the hindrances and the opportunities posed by the uptake in real world settings of AAL technologies. In this respect, the report illustrates the current procedural and technological approaches to cope with acceptability, usability and trust in the AAL technology, by surveying strategies and approaches to co-design, to privacy preservation in video and audio data, to transparency and explainability in data processing, and to data transmission and communication. User acceptance and ethical considerations are also debated. Finally, the potentials coming from the silver economy are overviewed.publishedVersio

On Leveraging Statistical and Relational Information for the Representation and Recognition of Complex Human Activities

Machine activity recognition aims to automatically predict human activities from a series of sensor signals. It is a key aspect to several emerging applications, especially in the pervasive computing field. However, this problem faces several challenges due to the complex, relational and ambiguous nature of human activities. These challenges still defy the majority of traditional pattern recognition approaches, whether they are knowledge-based or data-driven. Concretely, the current approaches to activity recognition in sensor environments fall short to represent, reason or learn under uncertainty, complex relational structure, rich temporal context and abundant common-sense knowledge. Motivated by these shortcomings, our work focuses on the combination of both data-driven and knowledge-based paradigms in order to address this problem. In particular, we propose two logic-based statistical relational activity recognition frameworks which we describe in two different parts. The first part presents a Markov logic-based framework addressing the recognition of complex human activities under realistic settings. Markov logic is a highly flexible statistical relational formalism combining the power of first-order logic with Markov networks by attaching real-valued weights to formulas in first-order logic. Thus, it unites both symbolic and probabilistic reasoning and allows to model the complex relational structure as well as the inherent uncertainty underlying human activities and sensor data. We focus on addressing the challenge of recognizing interleaved and concurrent activities while preserving the intuitiveness and flexibility of the modelling task. Using three different models we evaluate and prove the viability of using Markov logic networks for that problem statement. We also demonstrate the crucial impact of domain knowledge on the recognition outcome. Implementing an exhaustive model including heterogeneous information sources comes, however, at considerable knowledge engineering efforts. Hence, employing a standard, widely used formalism can alleviate that by enhancing the portability, the re-usability and the extension of the model. In the second part of this document, we apply a hybrid approach that goes one step further than Markov logic network towards a formal, yet intuitive conceptualization of the domain of discourse. Concretely, we propose an activity recognition framework based on log-linear description logic, a probabilistic variant of description logics. Log-linear description logic leverages the principles of Markov logic while allowing for a formal conceptualization of the domain of discourse, backed up with powerful reasoning and consistency check tools. Based on principles from the activity theory, we focus on addressing the challenge of representing and recognizing human activities at three levels of granularity: operations, actions and activities. Complying with real-life scenarios, we assess and discuss the viability of the proposed framework. In particular, we show the positive impact of augmenting the proposed multi-level activity ontology with weights compared to using its conventional weight-free variant

Contrôle intelligent de la domotique à partir d'informations temporelles multi sources imprécises et incertaines

La Maison Intelligente est une résidence équipée de technologie informatique qui assiste ses habitant dans les situations diverses de la vie domestique en essayant de gérer de manière optimale leur confort et leur sécurité par action sur la maison. La détection des situations anormales est un des points essentiels d'un système de surveillance à domicile. Ces situations peuvent être détectées en analysant les primitives générées par les étages de traitement audio et par les capteurs de l'appartement. Par exemple, la détection de cris et de bruits sourds (chute d'un objet lourd) dans un intervalle de temps réduit permet d'inférer l'occurrence d'une chute. Le but des travaux de cette thèse est la réalisation d'un contrôleur intelligent relié à tous les périphériques de la maison capable de réagir aux demandes de l'habitant (par commande vocale) et de reconnaître des situations à risque ou détresse. Pour accomplir cet objectif, il est nécessaire de représenter formellement et raisonner sur des informations, le plus souvent temporelles, à des niveaux d'abstraction différents. Le principale défi est le traitement de l'incertitude, l'imprécision, et incomplétude, qui caractérisent les informations dans ce domaine d'application. Par ailleurs, les décisions prises par le contrôleur doivent tenir compte du contexte dans lequel une ordre est donné, ce qui nous place dans l'informatique sensible au contexte. Le contexte est composé des informations de haut niveau tels que la localisation, l'activité en cours de réalisation, la période de la journée. Les recherches présentées dans ce manuscrit peuvent être divisés principalement en trois axes: la réalisation des méthodes d'inférence pour acquérir les informations du contexte(notamment, la localisation de l'habitant y l'activité en cours) à partir des informations incertains, la représentation des connaissances sur l'environnement et les situations à risque, et finalement la prise de décision à partir des informations contextuelles. La dernière partie du manuscrit expose les résultats de la validation des méthodes proposées par des évaluations amenées à la plateforme expérimental Domus.A smart home is a residence featuring ambient intelligence technologies in order to help its dwellers in different situations of common life by trying to manage their comfort and security through the execution of actions over the effectors of the house. Detection of abnormal situations is paramount in the development of surveillance systems. These situations can be detected by the analysis of the traces resulting from audio processing and the data provided by the network of sensors installed in the smart home. For instance, detection of cries along with thuds(fall of a heavy object) in a short time interval can help to infer that the resident has fallen. The goal of the research presented in this thesis is the implementation of an intelligence controller connected with the devices in the house that is able to react to user's commands(through vocal interfaces) and recognize dangerous situations. In order to fulfill this goal, it is necessary to create formal representation and to develop reasoning mechanism over informations that are often temporal and having different levels of abstraction. The main challenge is the processing the uncertainty, imprecision, and incompleteness that characterise this domain of application. Moreover, the decisions taken by the intelligent controller must consider the context in which a user command is given, so this work is made in the area of Context Aware Computing. Context includes high level information such as the location of the dweller, the activity she is making, and the time of the day. The research works presented in this thesis can be divided mainly in three parts: the implementation of inference methods to obtain context information(namely, location and activity) from uncertain information, knowledge representation about the environment and dangerous situations, and finally the development of decision making models that use the inferred context information. The last part of this thesis shows the results from the validation of the proposed methods through experiments performed in an experimental platform, the Domus apartment.SAVOIE-SCD - Bib.électronique (730659901) / SudocGRENOBLE1/INP-Bib.électronique (384210012) / SudocGRENOBLE2/3-Bib.électronique (384219901) / SudocSudocFranceF