Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum \u27stand-alone\u27; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Multimodal interaction with mobile devices : fusing a broad spectrum of modality combinations

This dissertation presents a multimodal architecture for use in mobile scenarios such as shopping and navigation. It also analyses a wide range of feasible modality input combinations for these contexts. For this purpose, two interlinked demonstrators were designed for stand-alone use on mobile devices. Of particular importance was the design and implementation of a modality fusion module capable of combining input from a range of communication modes like speech, handwriting, and gesture. The implementation is able to account for confidence value biases arising within and between modalities and also provides a method for resolving semantically overlapped input. Tangible interaction with real-world objects and symmetric multimodality are two further themes addressed in this work. The work concludes with the results from two usability field studies that provide insight on user preference and modality intuition for different modality combinations, as well as user acceptance for anthropomorphized objects.Diese Dissertation präsentiert eine multimodale Architektur zum Gebrauch in mobilen Umständen wie z. B. Einkaufen und Navigation. Außerdem wird ein großes Gebiet von möglichen modalen Eingabekombinationen zu diesen Umständen analysiert. Um das in praktischer Weise zu demonstrieren, wurden zwei teilweise gekoppelte Vorführungsprogramme zum 'stand-alone'; Gebrauch auf mobilen Geräten entworfen. Von spezieller Wichtigkeit war der Entwurf und die Ausführung eines Modalitäts-fusion Modul, das die Kombination einer Reihe von Kommunikationsarten wie Sprache, Handschrift und Gesten ermöglicht. Die Ausführung erlaubt die Veränderung von Zuverlässigkeitswerten innerhalb einzelner Modalitäten und außerdem ermöglicht eine Methode um die semantisch überlappten Eingaben auszuwerten. Wirklichkeitsnaher Dialog mit aktuellen Objekten und symmetrische Multimodalität sind zwei weitere Themen die in dieser Arbeit behandelt werden. Die Arbeit schließt mit Resultaten von zwei Feldstudien, die weitere Einsicht erlauben über die bevorzugte Art verschiedener Modalitätskombinationen, sowie auch über die Akzeptanz von anthropomorphisierten Objekten

Computer-Supported Collaborative Production

This paper proposes the concept of collaborative production as a focus of concern within the general area of collaborative work. We position the concept with respect to McGrath's framework for small group dynamics and the more familiar collaboration processes of awareness, coordination, and communication (McGrath 1991). After reviewing research issues and computer-based support for these interacting aspects of collaboration, we turn to a discussion of implications for how to design improved support for collaborative production. We illustrate both the challenges of collaborative production and our design implications with a collaborative map-updating scenario drawn from the work domain of geographical information systems

Design and application of a contact barcode reader, for use on low-visibility printed conductive patterns

This thesis was submitted for the degree of Master of Philosophy and awarded by Brunel University.This thesis presents the design and development of a hand-held electronic reader, designed to decode conductive patterns printed on a paper substrate. Data read from the patterns, by the reader, is used to trigger events in the digital domain. The reader and associated conductive patterns are devices for linking paper documents with the digital world. The patterns are formed by masking conductive-coated paper with a non-conductive, printed lacquer. The reader is a low cost and ergonomic device, capable of transmitting the embedded data from the conductive paper to the computer. The first reader designed and developed was tethered to a computer by data cable, using the USB communication protocol. The second design was developed further, with transmission of data achieved by replacing the cable with short-range Bluetooth wireless technology. Both devices were designed and developed using embedded systems and low cost electronic components. Additional work was undertaken to optimise the device's mechanical structure, ergonomics and integration of hardware. Alongside the development of the reader, test and development work was carried out to optimise the printed media, in materials and design. User trials demonstrated that the complete printed and reading system was functional, with varied rates of success among participants. Further work is required to improve the conductivity of the coated paper, and the accuracy of the decoding algorithm.This work resulted from a research project funded by EU FP6

Actas do 12º Encontro Português de Computação Gráfica

Actas do 12º Encontro Portugês de Computação Gráfica, Porto, 8-10 de Outubro de 2003O Encontro Português de Computação Gráfica teve lugar nesse ano 2003, naquela que foi a sua 12ª edição, no ISEP – Instituto Superior de Engenharia do Porto, entre os 8 a 10 de Outubro. O 12º Encontro Português de Computação Gráfica (12EPCG) veio no seguimento de encontros anteriores realizados anualmente e reuniu investigadores, docentes e profissionais nacionais e estrangeiros, que realizam trabalho ou utilizam a Computação Gráfica, Realidade Virtual e Multimédia, assim como todas as suas áreas afins, no sentido de permitir a divulgação de projectos realizados ou em curso e fomentar a troca de experiências e a discussão de questões relacionadas com a Computação Gráfica em Portugal, entre as comunidades académica,industrial e a de utilizadores finais. Este é o livro de actas do 12EPCG.Fundação Ilídio PinhoFC

Context-Aware Software

With the advent of PDAs (Personal Digital Assistants), smart phones, and other forms of mobile and ubiquitous computers, our computing resources are increasingly moving off of our desktops and into our everyday lives. However, the software and user interfaces for these devices are generally very similar to that of their desktop counterparts, despite the radically different and dynamic environments that they face. We propose that to better assist their users, such devices should be able to sense, react to, and utilise, the user's current environment or context. That is, they should become context-aware. In this thesis we investigate context-awareness at three levels: user interfaces, applications, and supporting architectures/frameworks. To promote the use of context-awareness, and to aid its deployment in software, we have developed two supporting frameworks. The first is an application-oriented framework called stick-e notes. Based on an electronic version of the common Post-It Note, stick-e notes enable the attachment of any electronic resource (e.g. a text file, movie, Java program, etc.) to any type of context (e.g. location, temperature, time, etc.). The second framework we devised seeks to provide a more universal support for the capture, manipulation, and representation of context information. We call it the Context Information Service (CIS). It fills a similar role in context-aware software development as GUI libraries do in user interface development. Our applications research explored how context-awareness can be exploited in real environments with real users. In particular, we developed a suite of PDA-based context-aware tools for fieldworkers. These were used extensively by a group of ecologists in Africa to record observations of giraffe and rhinos in a remote Kenyan game reserve. These tools also provided the foundations for our HCI work, in which we developed the concept of the Minimal Attention User Interface (MAUI). The aim of the MAUI is to reduce the attention required by the user in operating a device by carefully selecting input/output modes that are harmonious to their tasks and environment. To evaluate our ideas and applications a field study was conducted in which over forty volunteers used our system for data collection activities over the course of a summer season at the Kenyan game reserve. The PDA-based tools were unanimously preferred to the paper-based alternatives, and the context-aware features were cited as particular reasons for preferring them. In summary, this thesis presents two frameworks to support context-aware software, a set of applications demonstrating how context-awareness can be utilised in the ''real world'', and a set of HCI guidelines and principles that help in creating user interfaces that fit to their context of use

Context-aware software

With the advent of PDAs (Personal Digital Assistants), smart phones, and other forms of mobile and ubiquitous computers, our computing resources are increasingly moving off of our desktops and into our everyday lives. However, the software and user interfaces for these devices are generally very similar to that of their desktop counterparts, despite the radically different and dynamic environments that they face. We propose that to better assist their users, such devices should be able to sense, react to, and utilise, the user's current environment or context. That is, they should become context-aware. In this thesis we investigate context-awareness at three levels: user interfaces, applications, and supporting architectures/frameworks. To promote the use of context-awareness, and to aid its deployment in software, we have developed two supporting frameworks. The first is an application-oriented framework called stick-e notes. Based on an electronic version of the common Post-It Note, stick-e notes enable the attachment of any electronic resource (e.g. a text file, movie, Java program, etc.) to any type of context (e.g. location, temperature, time, etc.). The second framework we devised seeks to provide a more universal support for the capture, manipulation, and representation of context information. We call it the Context Information Service (CIS). It fills a similar role in context-aware software development as GUI libraries do in user interface development. Our applications research explored how context-awareness can be exploited in real environments with real users. In particular, we developed a suite of PDA-based context-aware tools for fieldworkers. These were used extensively by a group of ecologists in Africa to record observations of giraffe and rhinos in a remote Kenyan game reserve. These tools also provided the foundations for our HCI work, in which we developed the concept of the Minimal Attention User Interface (MAUI). The aim of the MAUI is to reduce the attention required by the user in operating a device by carefully selecting input/output modes that are harmonious to their tasks and environment. To evaluate our ideas and applications a field study was conducted in which over forty volunteers used our system for data collection activities over the course of a summer season at the Kenyan game reserve. The PDA-based tools were unanimously preferred to the paper-based alternatives, and the context-aware features were cited as particular reasons for preferring them. In summary, this thesis presents two frameworks to support context-aware software, a set of applications demonstrating how context-awareness can be utilised in the ''real world'', and a set of HCI guidelines and principles that help in creating user interfaces that fit to their context of use

Universal "Chip Based Smart Card" Snooping Device

The objective of this project is basically to design, build and test a Universal "Chip Based Smart Card" Snooping Device. This device would function to notify its user of the actual communication between the smart card and the smart card reader. The existence of this device not only serves as an educational tool but also saves a lot of time and money that are spent on debugging by smart card manufacturing companies. In addition to that, technological advances in the smart card world can also be sped up with the help of this device for research and testing. The hardware design of the project has been divided into 3 main sections, the snooping module, the communication module and the processing module. The snooping module taps the data, the communication module formats, encodes and transfers the data to the processing module, and lastly the processing module translates the data into useful information and display's it. The project has been divided into two major milestones where the first one was to set up the snooping device prototype with the processing module being a Computer. The second milestone on the other hand is the final design itself which would be to replace the Computer with a PALM in order to make it portable and affordable. This report gives a complete and detailed illustration on the hardware and software design process for the Universal "Chip Based Smart Card" Snooping Device. All aspects from the design decision, the underlying transmission protocols and also programming logics have been dissected and elaborately explained in the report. The design presented is a complete working device that fulfills all the objectives that have been set. The hardware device is certainly good enough to be marketed as it works flawlessly and achieves the most important objective of the project, which is to obtain the data transfer between the card and the smart card reader