A paradigm of an interaction context-aware pervasive multimodal multimedia computing system

Communication is a very important aspect of human life; it is communication that helps human beings to connect with each other as individuals and as independent groups. Communication is the fulcrum that drives all human developments in all fields. In informatics, one of the main purposes of the existence of computer is information dissemination – to be able to send and receive information. Humans are quite successful in conveying ideas to one another, and reacting appropriately. This is due to the fact that we share the richness of the language, have a common understanding of how things work and an implicit understanding of everyday situations. When humans communicate with humans, they comprehend the information that is apparent to the current situation, or context, hence increasing the conversational bandwidth. This ability to convey ideas, however, does not transfer when humans interact with computers. On its own, computers do not understand our language, do not understand how the world works and cannot sense information about the current situation. In a typical computing set-up where we have an impoverished typical mechanism for providing computer with information using mouse, keyboard and screen, the end result is we explicitly provide information to computers, producing an effect that is contrary to the promise of transparency and calm technology in Weiser’s vision of ubiquitous computing (Weiser 1991; Weiser and Brown 1996). To reverse this trend, it is imperative that we researchers find ways that will enable computers to have access to context. It is through context-awareness that we can increase the richness of communication in human-computer interaction, through which we can reap the most likely benefit of more useful computational services. Context is a subjective idea as demonstrated by the state-of-the art in which each researcher has his own understanding of the term, which continues to evolve nonetheless. The acquisition of contextual information is essential but it is the end user, however, that will have the final say as to whether the envisioned context is correctly captured/acquired or not. Current literature informs us that some contextual information is already predefined by some researchers from the very beginning – this is correct if the application domain is fixed but is incorrect if we infer that a typical user does different computing tasks on different occasions. With the aim of coming up with more conclusive and inclusive design, we conjecture that what contextual information should be left to the judgment of the end user who is the one that has the knowledge determine which information is important to him and which is not. This leads us to the concept of incremental acquisition of context where context parameters are added, modified or deleted one context parameter at a time. In conjunction with our idea of inclusive context, we broaden the notion of context that it has become context of interaction. Interaction context is the term that is used to refer to the collective context of the user (i.e. user context), of his working environment (i.e. environmental context) and of his computing system (i.e. system context). Logically and mathematically, each of these interaction context elements – user context, environment context and system context – is composed of various parameters that describe the state of the user, of his workplace and his computing resources as he undertakes an activity in accomplishing his computing task, and each of these parameters may evolve over time. For example, user location is a user context parameter and its value will evolve as the user moves from one place to another. The same can be said about noise level as an environment context parameter; its value evolves over time. The same can be said with available bandwidth that continuously evolves which we consider as a system context parameter. To realize the incremental definition of incremental context, we have developed a tool called the virtual machine for incremental interaction context. This tool can be used to add, modify and delete a context parameter on one hand and determine the sensor-based context (i.e. context that is based on parameters whose values are obtained from raw data supplied by sensors) on the other. In order to obtain the full benefit of the richness of interaction context with regards to communication in human-machine interaction, the modality of interaction should not be limited to the traditional use of mouse-keyboard-screen alone. Multimodality allows for a much wider range of modes and forms of communication, selected and adapted to suit the given user’s context of interaction, by which the end user can transmit data to the computer and computer can respond or yield results to the user’s queries. In multimodal communication, the weaknesses of one mode of interaction, with regards to its suitability to a given situation, is compensated by replacing it with another mode of communication that is more suitable to the situation. For example, when the environment becomes disturbingly noisy, using voice may not be the ideal mode to input data; instead, the user may opt for transmitting text or visual information. Multimodality also promotes inclusive informatics as those with a permanent or temporary disability are given the opportunity to use and benefit from information technology advancement. For example, the work on presentation of mathematical expressions to visually-impaired users (Awdé 2009) would not have been made possible without multimodality. With mobile computing within our midst coupled with wireless communication that allows access to information and services, pervasive and adaptive multimodality is more than ever apt to enrich communication in human-computer interaction and in providing the most suitable modes for data input and output in relation to the evolving interaction context. A look back at the state of the art informs us that a great amount of effort was expended in finding the definition of context, in the acquisition of context, in the dissemination of context and the exploitation of context within a system that has a fixed domain of application (e.g. healthcare, education, etc.). Also, another close look tells us that much research efforts on ubiquitous computing were devoted to various application domains (e.g. identifying the user whereabouts, identifying services and tools, etc.) but there is rarely, if ever, an effort made to make multimodality pervasive and accessible to various user situations. In this regard, we come up with a research work that will provide for the missing link. Our work – the paradigm of an interaction context-sensitive pervasive multimodal multimedia computing system is an architectural design that exhibits adaptability to a much larger context called interaction context. It is intelligent and pervasive, meaning it is functional even when the end user is stationary or on the go. It is conceived with two purposes in mind. First, given an instance of interaction context, one which evolves over time, our system determines the optimal modalities that suit such interaction context. By optimal, we mean a selection decision on appropriate multimodality based on the given interaction context, available media devices that support the modalities and user preferences. We designed a mechanism (i.e. a paradigm) that will do this task and simulated its functionality with success. This mechanism employs machine learning (Mitchell 1997; Alpaydin 2004; Hina, Tadj et al. 2006) and uses case-based reasoning with supervised learning (Kolodner 1993; Lajmi, Ghedira et al. 2007). An input to this decision-making component is an instance of interaction context and its output is the optimal modality and its associated media devices that are for activation. This mechanism is continuously monitoring the user’s context of interaction and on behalf of the user continuously adapts accordingly. This adaptation is through dynamic reconfiguration of the pervasive multimodal system’s architecture. Second, given an instance of interaction context and the user’s task and preferences, we designed a mechanism that allows the automatic selection of user’s applications, the preferred suppliers to these applications and the preferred quality of service (QoS) dimensions’ configurations of these suppliers. This mechanism does its task in consultation with computing resources, sensing the available suppliers and possible configuration restrictions within the given computing set-up. Apart from the above-mentioned mechanisms, we also formulated scenarios as to how a computing system must provide the user interface given that we have already identified the optimal modalities that suit the user’s context of interaction. We present possible configurations of unimodal and bimodal interfaces based on the given interaction context as well as user preferences. Our work is different from previous work in that while other systems capture, disseminate and consume context to suit the preferred domain of application, ours captures the interaction context and reconfigures its architecture dynamically in generic fashion in order that the user could continue working on his task anytime, anywhere he wishes regardless of the application domain the user wishes to undertake. In effect, the system that we have designed along with all of its mechanisms, being generic in design, can be adapted or integrated with ease or with very little modification into various computing systems of various domains of applications. Simulations and mathematical formulations were provided to support our ideas and concepts related to the design of the paradigm. An actual program in Java was developed to support our concept of a virtual machine for incremental interaction context

User-centred design of flexible hypermedia for a mobile guide: Reflections on the hyperaudio experience

A user-centred design approach involves end-users from the very beginning. Considering users at the early stages compels designers to think in terms of utility and usability and helps develop the system on what is actually needed. This paper discusses the case of HyperAudio, a context-sensitive adaptive and mobile guide to museums developed in the late 90s. User requirements were collected via a survey to understand visitors’ profiles and visit styles in Natural Science museums. The knowledge acquired supported the specification of system requirements, helping defining user model, data structure and adaptive behaviour of the system. User requirements guided the design decisions on what could be implemented by using simple adaptable triggers and what instead needed more sophisticated adaptive techniques, a fundamental choice when all the computation must be done on a PDA. Graphical and interactive environments for developing and testing complex adaptive systems are discussed as a further step towards an iterative design that considers the user interaction a central point. The paper discusses how such an environment allows designers and developers to experiment with different system’s behaviours and to widely test it under realistic conditions by simulation of the actual context evolving over time. The understanding gained in HyperAudio is then considered in the perspective of the developments that followed that first experience: our findings seem still valid despite the passed time

Proceedings of the 4th Workshop on Interacting with Smart Objects 2015

These are the Proceedings of the 4th IUI Workshop on Interacting with Smart Objects. Objects that we use in our everyday life are expanding their restricted interaction capabilities and provide functionalities that go far beyond their original functionality. They feature computing capabilities and are thus able to capture information, process and store it and interact with their environments, turning them into smart objects

Human Action Recognition and Monitoring in Ambient Assisted Living Environments

Population ageing is set to become one of the most significant challenges of the 21st century, with implications for almost all sectors of society. Especially in developed countries, governments should immediately implement policies and solutions to facilitate the needs of an increasingly older population. Ambient Intelligence (AmI) and in particular the area of Ambient Assisted Living (AAL) offer a feasible response, allowing the creation of human-centric smart environments that are sensitive and responsive to the needs and behaviours of the user. In such a scenario, understand what a human being is doing, if and how he/she is interacting with specific objects, or whether abnormal situations are occurring is critical. This thesis is focused on two related research areas of AAL: the development of innovative vision-based techniques for human action recognition and the remote monitoring of users behaviour in smart environments. The former topic is addressed through different approaches based on data extracted from RGB-D sensors. A first algorithm exploiting skeleton joints orientations is proposed. This approach is extended through a multi-modal strategy that includes the RGB channel to define a number of temporal images, capable of describing the time evolution of actions. Finally, the concept of template co-updating concerning action recognition is introduced. Indeed, exploiting different data categories (e.g., skeleton and RGB information) improve the effectiveness of template updating through co-updating techniques. The action recognition algorithms have been evaluated on CAD-60 and CAD-120, achieving results comparable with the state-of-the-art. Moreover, due to the lack of datasets including skeleton joints orientations, a new benchmark named Office Activity Dataset has been internally acquired and released. Regarding the second topic addressed, the goal is to provide a detailed implementation strategy concerning a generic Internet of Things monitoring platform that could be used for checking users' behaviour in AmI/AAL contexts

08231 Abstracts Collection -- Virtual Realities

From 1st to 6th June 2008, the Dagstuhl Seminar 08231 ``Virtual Realities\u27\u27 was held in the International Conference and Research Center (IBFI), Schloss Dagstuhl. Virtual Reality (VR) is a multidisciplinary area of research aimed at interactive human-computer mediated simulations of artificial environments. Typical applications include simulation, training, scientific visualization, and entertainment. An important aspect of VR-based systems is the stimulation of the human senses -- typically sight, sound, and touch -- such that a user feels a sense of presence (or immersion) in the virtual environment. Different applications require different levels of presence, with corresponding levels of realism, sensory immersion, and spatiotemporal interactive fidelity. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. Links to extended abstracts or full papers are provided, if available

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications

CHORUS Deliverable 2.2: Second report - identification of multi-disciplinary key issues for gap analysis toward EU multimedia search engines roadmap

After addressing the state-of-the-art during the first year of Chorus and establishing the existing landscape in multimedia search engines, we have identified and analyzed gaps within European research effort during our second year. In this period we focused on three directions, notably technological issues, user-centred issues and use-cases and socio- economic and legal aspects. These were assessed by two central studies: firstly, a concerted vision of functional breakdown of generic multimedia search engine, and secondly, a representative use-cases descriptions with the related discussion on requirement for technological challenges. Both studies have been carried out in cooperation and consultation with the community at large through EC concertation meetings (multimedia search engines cluster), several meetings with our Think-Tank, presentations in international conferences, and surveys addressed to EU projects coordinators as well as National initiatives coordinators. Based on the obtained feedback we identified two types of gaps, namely core technological gaps that involve research challenges, and “enablers”, which are not necessarily technical research challenges, but have impact on innovation progress. New socio-economic trends are presented as well as emerging legal challenges