Sensor Fusion and Environmental Modelling for Multimodal Sentient Computing

Sentient computing uses networks of sensors to capture and maintain an internal representation (“world model”) of an indoor environment, thereby allowing applications to have greater awareness of users and their requirements. This chapter shows how computer vision information obtained by several cameras can be used to enhance the capabilities of a sentient computing system which previously relied on ultrasound to track people and devices. Integration is achieved at the system level through the metaphor of shared perceptions in the sense that the different modalities are guided by and provide updates for a shared internal model. This world model incorporates aspects of both the static (e.g. positions of office walls and doors) and dynamic (e.g. location and appearance of devices and people) environments. It serves both as an ontology of prior information and as a source of context which is shared between applications. Fusion and inference are performed by Bayesian networks which model the probabilistic dependencies and reliabilities of different sources of information over time. It is shown that the fusion process significantly enhances the capabilities and robustness of the system, thus enabling it to maintain a richer and more accurate world model. Further details are available in [2, 3]. 2 The Sentient Office A sentient office uses sensor and resource status data to maintain a model of the world which is shared between users and applications. Sensors and telemetry are used to keep the model accurate and up to date, while applications see the world via the model. The SPIRIT [1] system uses mobile ultrasonic sensor devices known as “Bats”. The achieved spatial granularity is better than 3cm for> 95 % of Bat observations (assuming only small motion) and Bats may be polled using radio base stations and a variable quality of service to give update frequencies of up to 25Hz while remaining scalable to hundreds of tagged people and devices in a large office. Th