Mobile Phone Enabled Museum Guidance with Adaptive Classification

Although audio guides are widely established in many museums, they suffer from several drawbacks compared to state-of-the-art multimedia technologies: First, they provide only audible information to museum visitors, while other forms of media presentation, such as reading text or video could be beneficial for museum guidance tasks. Second, they are not very intuitive. Reference numbers have to be manually keyed in by the visitor before information about the exhibit is provided. These numbers are either displayed on visible tags that are located near the exhibited objects, or are printed in brochures that have to be carried. Third, offering mobile guidance equipment to visitors leads to acquisition and maintenance costs that have to be covered by the museum. With our project PhoneGuide we aim at solving these problems by enabling the application of conventional camera-equipped mobile phones for museum guidance purposes. The advantages are obvious: First, today’s off-the-shelf mobile phones offer a rich pallet of multimedia functionalities ---ranging from audio (over speaker or head-set) and video (graphics, images, movies) to simple tactile feedback (vibration). Second, integrated cameras, improvements in processor performance and more memory space enable supporting advanced computer vision algorithms. Instead of keying in reference numbers, objects can be recognized automatically by taking non-persistent photographs of them. This is more intuitive and saves museum curators from distributing and maintaining a large number of physical (visible or invisible) tags. Together with a few sensor-equipped reference tags only, computer vision based object recognition allows for the classification of single objects; whereas overlapping signal ranges of object-distinct active tags (such as RFID) would prevent the identification of individuals that are grouped closely together. Third, since we assume that museum visitors will be able to use their own devices, the acquisition and maintenance cost for museum-owned devices decreases

A design space for social object labels in museums

Taking a problematic user experience with ubiquitous annotation as its point of departure, this thesis defines and explores the design space for Social Object Labels (SOLs), small interactive displays aiming to support users' in-situ engagement with digital annotations of physical objects and places by providing up-to-date information before, during and after interaction. While the concept of ubiquitous annotation has potential applications in a wide range of domains, the research focuses in particular on SOLs in a museum context, where they can support the institution's educational goals by engaging visitors in the interpretation of exhibits and providing a platform for public discourse to complement official interpretations provided on traditional object labels. The thesis defines and structures the design space for SOLs, investigates how they can support social interpretation in museums and develops empirically validated design recommendations. Reflecting the developmental character of the research, it employs Design Research as a methodological framework, which involves the iterative development and evaluation of design artefacts together with users and other stakeholders. The research identifies the particular characteristics of SOLs and structures their design space into ten high-level aspects, synthesised from taxonomies and heuristics for similar display concepts and complemented with aspects emerging from the iterative design and evaluation of prototypes. It presents findings from a survey exploring visitors' mental models, preferences and expectations of commenting in museums and translates them into requirements for SOLs. It reports on scenario-based design activities, expert interviews with museum professionals, formative user studies and co-design sessions, and two empirical evaluations of SOL prototypes in a gallery environment. Pulling together findings from these research activities it then formulates design recommendations for SOLs and supports them with related evidence and implementation examples. The main contributions are (i) to delineate and structure the design space for SOLs, which helps to ground SOLs in the literature and understand them as a distinct display concept with its own characteristics; (ii) to explore, for the first time, a visitor perspective on commenting in museums, which can inform research, development and policies on user-generated content in museums and the wider cultural heritage sector; (iii) to develop empirically validated design recommendations, which can inform future research and development into SOLs and related display concept. The thesis concludes by summarising findings in relation to its stated research questions, restating its contributions from ubiquitous computing, domain and methodology perspectives, and discussing open issues and future work

Physical Selection in Ubiquitous Computing

Jokapaikan tietotekniikassa (ubiquitous computing) tietotekniset laitteet sulautuvat fyysiseen ympäristöön siten että niiden käyttäjät voivat olla yhtä aikaa vuorovaikutuksessa näiden laitteiden kanssa ja toimia fyysisessä ympäristössään. Laitteet ovat yhteydessä toisiinsa, ne ovat eri kokoisia ja niillä on erilaisia syöttö- ja tulostusmahdollisuuksia tarkoituksestaan riippuen. Nämä jokapaikan tietotekniikan ominaisuudet luovat tarpeen vuorovaikutustavoille, jotka eroavat huomattavasti tavanomaisten työpöytätietokoneiden vuorovaikutustavoista. Fyysinen valinta (physical selection) on jokapaikan tietotekniikan vuorovaikutustehtävä, jota käytetään kertomaan käyttäjän kannettavalle päätelaitteelle minkä fyysisen esineen kanssa käyttäjä haluaa olla vuorovaikutuksessa. Fyysinen valinta perustuu tunnisteisiin (tag), jotka yksilöivät fyysiset esineet tai sisältävät fyysisen hyperlinkin digitaalisessa muodossa olevaan tietoon, joka liittyy esineeseen, johon kyseinen tunniste on liitetty. Käyttäjä valitsee fyysisen hyperlinkin koskettamalla, osoittamalla tai skannaamalla tunnistetta sopivalla lukulaitteella varustetulla päätelaitteellaan. Fyysinen valinta voidaan toteuttaa erilaisilla teknologioilla, kuten sähköisesti luettavilla tunnisteilla ja niiden lukijoilla, infrapunalähettimillä sekä optisesti luettavilla tunnisteilla ja matkapuhelinten kameroilla. Tässä väitöskirjassa analysoidaan fyysistä valintaa vuorovaikutustehtävänä ja toteutusteknisestä näkökulmasta sekä esitellään eri valintatavat ­ kosketus, osoitus ja skannaus. Koskeusta ja osoitusta on tutkittu toteuttamalla prototyyppi ja tutkimalla sen avulla valintatapoja kokeellisesti. Tämän väitöskirjan tuloksiin kuuluu fyysisen valinnan analysointi jokapaikan tietotekniikan kontekstissa, ehdotuksia fyysisten hyperlinkkien visualisoinnista sekä fyysisessä ympäristössä että päätelaitteessa, ja käyttäjävaatimuksia fyysiselle valinnalle osana jokapaikan tietotekniikan arkkitehtuuria.In ubiquitous computing, the computing devices are embedded into the physical environment so that the users can interact with the devices at the same time as they interact with the physical environment. The various devices are connected to each other, and have various sizes and input and output capabilities depending on their purpose. These features of ubiquitous computing create a need for interaction methods that are radically different from the desktop computer interactions. Physical selection is an interaction task for ubiquitous computing and it is used to tell the user s mobile terminal which physical object the user wants to interact with. It is based on tags that identify physical objects or store a physical hyperlink to digital information related to the object the tag is attached to. The user selects the physical hyperlink by touching, pointing or scanning the tag with the mobile terminal that is equipped with an appropriate reader. Physical selection has been implemented with various technologies, such as radio-frequency tags and readers, infrared transceivers, and optically readable tags and mobile phone cameras. In this dissertation, physical selection is analysed as a user interaction task, and from the implementation viewpoint. Different selection methods ­ touching, pointing and scanning ­ are presented. Touching and pointing have been studied by implementing a prototype and conducting user experiments with it. The contributions of this dissertation include an analysis of physical selection in the ubiquitous computing context, suggestions for visualising the physical hyperlinks in both the physical environment and in the mobile terminal, and user requirements for physical selection as a part of an ambient intelligence architecture

Adaptive Image Classification on Mobile Phones

The advent of high-performance mobile phones has opened up the opportunity to develop new context-aware applications for everyday life. In particular, applications for context-aware information retrieval in conjunction with image-based object recognition have become a focal area of recent research. In this thesis we introduce an adaptive mobile museum guidance system that allows visitors in a museum to identify exhibits by taking a picture with their mobile phone. Besides approaches to object recognition, we present different adaptation techniques that improve classification performance. After providing a comprehensive background of context-aware mobile information systems in general, we present an on-device object recognition algorithm and show how its classification performance can be improved by capturing multiple images of a single exhibit. To accomplish this, we combine the classification results of the individual pictures and consider the perspective relations among the retrieved database images. In order to identify multiple exhibits in pictures we present an approach that uses the spatial relationships among the objects in images. They make it possible to infer and validate the locations of undetected objects relative to the detected ones and additionally improve classification performance. To cope with environmental influences, we introduce an adaptation technique that establishes ad-hoc wireless networks among the visitors’ mobile devices to exchange classification data. This ensures constant classification rates under varying illumination levels and changing object placement. Finally, in addition to localization using RF-technology, we present an adaptation technique that uses user-generated spatio-temporal pathway data for person movement prediction. Based on the history of previously visited exhibits, the algorithm determines possible future locations and incorporates these predictions into the object classification process. This increases classification performance and offers benefits comparable to traditional localization approaches but without the need for additional hardware. Through multiple field studies and laboratory experiments we demonstrate the benefits of each approach and show how they influence the overall classification rate.Die Einführung von Mobiltelefonen mit eingebauten Sensoren wie Kameras, GPS oder Beschleunigungssensoren, sowie Kommunikationstechniken wie Bluetooth oder WLAN ermöglicht die Entwicklung neuer kontextsensitiver Anwendungen für das tägliche Leben. Insbesondere Applikationen im Bereich kontextsensitiver Informationsbeschaffung in Verbindung mit bildbasierter Objekterkennung sind in den Fokus der aktuellen Forschung geraten. Der Beitrag dieser Arbeit ist die Entwicklung eines bildbasierten, mobilen Museumsführersystems, welches unterschiedliche Adaptionstechniken verwendet, um die Objekterkennung zu verbessern. Es wird gezeigt, wie Ojekterkennungsalgorithmen auf Mobiltelefonen realisiert werden können und wie die Erkennungsrate verbessert wird, indem man zum Beispiel ad-hoc Netzwerke einsetzt oder Bewegungsvorhersagen von Personen berücksichtigt

Physical Selection in Ubiquitous Computing

Jokapaikan tietotekniikassa (ubiquitous computing) tietotekniset laitteet sulautuvat fyysiseen ympäristöön siten että niiden käyttäjät voivat olla yhtä aikaa vuorovaikutuksessa näiden laitteiden kanssa ja toimia fyysisessä ympäristössään. Laitteet ovat yhteydessä toisiinsa, ne ovat eri kokoisia ja niillä on erilaisia syöttö- ja tulostusmahdollisuuksia tarkoituksestaan riippuen. Nämä jokapaikan tietotekniikan ominaisuudet luovat tarpeen vuorovaikutustavoille, jotka eroavat huomattavasti tavanomaisten työpöytätietokoneiden vuorovaikutustavoista. Fyysinen valinta (physical selection) on jokapaikan tietotekniikan vuorovaikutustehtävä, jota käytetään kertomaan käyttäjän kannettavalle päätelaitteelle minkä fyysisen esineen kanssa käyttäjä haluaa olla vuorovaikutuksessa. Fyysinen valinta perustuu tunnisteisiin (tag), jotka yksilöivät fyysiset esineet tai sisältävät fyysisen hyperlinkin digitaalisessa muodossa olevaan tietoon, joka liittyy esineeseen, johon kyseinen tunniste on liitetty. Käyttäjä valitsee fyysisen hyperlinkin koskettamalla, osoittamalla tai skannaamalla tunnistetta sopivalla lukulaitteella varustetulla päätelaitteellaan. Fyysinen valinta voidaan toteuttaa erilaisilla teknologioilla, kuten sähköisesti luettavilla tunnisteilla ja niiden lukijoilla, infrapunalähettimillä sekä optisesti luettavilla tunnisteilla ja matkapuhelinten kameroilla. Tässä väitöskirjassa analysoidaan fyysistä valintaa vuorovaikutustehtävänä ja toteutusteknisestä näkökulmasta sekä esitellään eri valintatavat ­ kosketus, osoitus ja skannaus. Koskeusta ja osoitusta on tutkittu toteuttamalla prototyyppi ja tutkimalla sen avulla valintatapoja kokeellisesti. Tämän väitöskirjan tuloksiin kuuluu fyysisen valinnan analysointi jokapaikan tietotekniikan kontekstissa, ehdotuksia fyysisten hyperlinkkien visualisoinnista sekä fyysisessä ympäristössä että päätelaitteessa, ja käyttäjävaatimuksia fyysiselle valinnalle osana jokapaikan tietotekniikan arkkitehtuuria.In ubiquitous computing, the computing devices are embedded into the physical environment so that the users can interact with the devices at the same time as they interact with the physical environment. The various devices are connected to each other, and have various sizes and input and output capabilities depending on their purpose. These features of ubiquitous computing create a need for interaction methods that are radically different from the desktop computer interactions. Physical selection is an interaction task for ubiquitous computing and it is used to tell the user s mobile terminal which physical object the user wants to interact with. It is based on tags that identify physical objects or store a physical hyperlink to digital information related to the object the tag is attached to. The user selects the physical hyperlink by touching, pointing or scanning the tag with the mobile terminal that is equipped with an appropriate reader. Physical selection has been implemented with various technologies, such as radio-frequency tags and readers, infrared transceivers, and optically readable tags and mobile phone cameras. In this dissertation, physical selection is analysed as a user interaction task, and from the implementation viewpoint. Different selection methods ­ touching, pointing and scanning ­ are presented. Touching and pointing have been studied by implementing a prototype and conducting user experiments with it. The contributions of this dissertation include an analysis of physical selection in the ubiquitous computing context, suggestions for visualising the physical hyperlinks in both the physical environment and in the mobile terminal, and user requirements for physical selection as a part of an ambient intelligence architecture

Integrating Usability Models into Pervasive Application Development

This thesis describes novel processes in two important areas of human-computer interaction (HCI) and demonstrates ways to combine these in appropriate ways. First, prototyping plays an essential role in the development of complex applications. This is especially true if a user-centred design process is followed. We describe and compare a set of existing toolkits and frameworks that support the development of prototypes in the area of pervasive computing. Based on these observations, we introduce the EIToolkit that allows the quick generation of mobile and pervasive applications, and approaches many issues found in previous works. Its application and use is demonstrated in several projects that base on the architecture and an implementation of the toolkit. Second, we present novel results and extensions in user modelling, specifically for predicting time to completion of tasks. We extended established concepts such as the Keystroke-Level Model to novel types of interaction with mobile devices, e.g. using optical markers and gestures. The design, creation, as well as a validation of this model are presented in some detail in order to show its use and usefulness for making usability predictions. The third part is concerned with the combination of both concepts, i.e. how to integrate user models into the design process of pervasive applications. We first examine current ways of developing and show generic approaches to this problem. This leads to a concrete implementation of such a solution. An innovative integrated development environment is provided that allows for quickly developing mobile applications, supports the automatic generation of user models, and helps in applying these models early in the design process. This can considerably ease the process of model creation and can replace some types of costly user studies.Diese Dissertation beschreibt neuartige Verfahren in zwei wichtigen Bereichen der Mensch-Maschine-Kommunikation und erläutert Wege, diese geeignet zu verknüpfen. Zum einen spielt die Entwicklung von Prototypen insbesondere bei der Verwendung von benutzerzentrierten Entwicklungsverfahren eine besondere Rolle. Es werden daher auf der einen Seite eine ganze Reihe vorhandener Arbeiten vorgestellt und verglichen, die die Entwicklung prototypischer Anwendungen speziell im Bereich des Pervasive Computing unterstützen. Ein eigener Satz an Werkzeugen und Komponenten wird präsentiert, der viele der herausgearbeiteten Nachteile und Probleme solcher existierender Projekte aufgreift und entsprechende Lösungen anbietet. Mehrere Beispiele und eigene Arbeiten werden beschrieben, die auf dieser Architektur basieren und entwickelt wurden. Auf der anderen Seite werden neue Forschungsergebnisse präsentiert, die Erweiterungen von Methoden in der Benutzermodellierung speziell im Bereich der Abschätzung von Interaktionszeiten beinhalten. Mit diesen in der Dissertation entwickelten Erweiterungen können etablierte Konzepte wie das Keystroke-Level Model auf aktuelle und neuartige Interaktionsmöglichkeiten mit mobilen Geräten angewandt werden. Der Entwurf, das Erstellen sowie eine Validierung der Ergebnisse dieser Erweiterungen werden detailliert dargestellt. Ein dritter Teil beschäftigt sich mit Möglichkeiten die beiden beschriebenen Konzepte, zum einen Prototypenentwicklung im Pervasive Computing und zum anderen Benutzermodellierung, geeignet zu kombinieren. Vorhandene Ansätze werden untersucht und generische Integrationsmöglichkeiten beschrieben. Dies führt zu konkreten Implementierungen solcher Lösungen zur Integration in vorhandene Umgebungen, als auch in Form einer eigenen Applikation spezialisiert auf die Entwicklung von Programmen für mobile Geräte. Sie erlaubt das schnelle Erstellen von Prototypen, unterstützt das automatische Erstellen spezialisierter Benutzermodelle und ermöglicht den Einsatz dieser Modelle früh im Entwicklungsprozess. Dies erleichtert die Anwendung solcher Modelle und kann Aufwand und Kosten für entsprechende Benutzerstudien einsparen