Tangible user interfaces : past, present and future directions

In the last two decades, Tangible User Interfaces (TUIs) have emerged as a new interface type that interlinks the digital and physical worlds. Drawing upon users' knowledge and skills of interaction with the real non-digital world, TUIs show a potential to enhance the way in which people interact with and leverage digital information. However, TUI research is still in its infancy and extensive research is required in or- der to fully understand the implications of tangible user interfaces, to develop technologies that further bridge the digital and the physical, and to guide TUI design with empirical knowledge. This paper examines the existing body of work on Tangible User In- terfaces. We start by sketching the history of tangible user interfaces, examining the intellectual origins of this field. We then present TUIs in a broader context, survey application domains, and review frame- works and taxonomies. We also discuss conceptual foundations of TUIs including perspectives from cognitive sciences, phycology, and philoso- phy. Methods and technologies for designing, building, and evaluating TUIs are also addressed. Finally, we discuss the strengths and limita- tions of TUIs and chart directions for future research

Passive Electric Field Sensing for Ubiquitous and Environmental Perception

Electric Field Sensing plays an important role in the research branches of Environmental Perception as well as in Ubiquitous Computing. Environmental Perception aims to collect data of the surroundings, while Ubiquitous Computing has the objective of making computing available at any time. This includes the integration of sensors to perceive environmental influences in an unobtrusive way. Electric Field Sensing, also referenced as Capacitive Sensing, is an often used sensing modality in these research fields, for example, to detect the presence of persons or to locate touches and interactions on user interfaces. Electric Field Sensing has a number of advantages over other technologies, such as the fact that Capacitive Sensing does not require direct line-of-sight contact with the object being sensed and that the sensing system can be compact in design. These advantages facilitate high integrability and allow the collection of data as required in Environmental Perception, as well as the invisible incorporation into a user's environment, needed in Ubiquitous Computing. However, disadvantages are often attributed to Capacitive Sensing principles, such as a low sensing range of only a few centimeters and the generation of electric fields, which wastes energy and has several more problems concerning the implementation. As shown in this thesis, this only affects a subset of this sensing technology, namely the subcategory of active capacitive measurements. Therefore, this thesis focuses on the mainly open area of Passive Electric Field Sensing in the context of Ubiquitous Computing and Environmental Perception, as active Capacitive Sensing is an open research field which already gains a lot of attention. The thesis is divided into three main research questions. First, I address the question of whether and how Passive Electric Field Sensing can be made available in a cost-effective and simple manner. To this end, I present various techniques for reducing installation costs and simplifying the handling of these sensor systems. After the question of low-cost applicability, I examine for which applications passive electric field sensor technology is suitable at all. Therefore I present several fields of application where Passive Electric Field Sensing data can be collected. Taking into account the possible fields of application, this work is finally dedicated to the optimization of Passive Electric Field Sensing in these cases of application. For this purpose, different, already known signal processing methods are investigated for their application for Passive Electric Field sensor data. Furthermore, besides these software optimizations, hardware optimizations for the improved use of the technology are presented

3D-LIVE : live interactions through 3D visual environments

This paper explores Future Internet (FI) 3D-Media technologies and Internet of Things (IoT) in real and virtual environments in order to sense and experiment Real-Time interaction within live situations. The combination of FI testbeds and Living Labs (LL) would enable both researchers and users to explore capacities to enter the 3D Tele-Immersive (TI) application market and to establish new requirements for FI technology and infrastructure. It is expected that combining both FI technology pull and TI market pull would promote and accelerate the creation and adoption, by user communities such as sport practitioners, of innovative TI Services within sport events

Blending the Material and Digital World for Hybrid Interfaces

The development of digital technologies in the 21st century is progressing continuously and new device classes such as tablets, smartphones or smartwatches are finding their way into our everyday lives. However, this development also poses problems, as these prevailing touch and gestural interfaces often lack tangibility, take little account of haptic qualities and therefore require full attention from their users. Compared to traditional tools and analog interfaces, the human skills to experience and manipulate material in its natural environment and context remain unexploited. To combine the best of both, a key question is how it is possible to blend the material world and digital world to design and realize novel hybrid interfaces in a meaningful way. Research on Tangible User Interfaces (TUIs) investigates the coupling between physical objects and virtual data. In contrast, hybrid interfaces, which specifically aim to digitally enrich analog artifacts of everyday work, have not yet been sufficiently researched and systematically discussed. Therefore, this doctoral thesis rethinks how user interfaces can provide useful digital functionality while maintaining their physical properties and familiar patterns of use in the real world. However, the development of such hybrid interfaces raises overarching research questions about the design: Which kind of physical interfaces are worth exploring? What type of digital enhancement will improve existing interfaces? How can hybrid interfaces retain their physical properties while enabling new digital functions? What are suitable methods to explore different design? And how to support technology-enthusiast users in prototyping? For a systematic investigation, the thesis builds on a design-oriented, exploratory and iterative development process using digital fabrication methods and novel materials. As a main contribution, four specific research projects are presented that apply and discuss different visual and interactive augmentation principles along real-world applications. The applications range from digitally-enhanced paper, interactive cords over visual watch strap extensions to novel prototyping tools for smart garments. While almost all of them integrate visual feedback and haptic input, none of them are built on rigid, rectangular pixel screens or use standard input modalities, as they all aim to reveal new design approaches. The dissertation shows how valuable it can be to rethink familiar, analog applications while thoughtfully extending them digitally. Finally, this thesis’ extensive work of engineering versatile research platforms is accompanied by overarching conceptual work, user evaluations and technical experiments, as well as literature reviews.Die Durchdringung digitaler Technologien im 21. Jahrhundert schreitet stetig voran und neue Geräteklassen wie Tablets, Smartphones oder Smartwatches erobern unseren Alltag. Diese Entwicklung birgt aber auch Probleme, denn die vorherrschenden berührungsempfindlichen Oberflächen berücksichtigen kaum haptische Qualitäten und erfordern daher die volle Aufmerksamkeit ihrer Nutzer:innen. Im Vergleich zu traditionellen Werkzeugen und analogen Schnittstellen bleiben die menschlichen Fähigkeiten ungenutzt, die Umwelt mit allen Sinnen zu begreifen und wahrzunehmen. Um das Beste aus beiden Welten zu vereinen, stellt sich daher die Frage, wie neuartige hybride Schnittstellen sinnvoll gestaltet und realisiert werden können, um die materielle und die digitale Welt zu verschmelzen. In der Forschung zu Tangible User Interfaces (TUIs) wird die Verbindung zwischen physischen Objekten und virtuellen Daten untersucht. Noch nicht ausreichend erforscht wurden hingegen hybride Schnittstellen, die speziell darauf abzielen, physische Gegenstände des Alltags digital zu erweitern und anhand geeigneter Designparameter und Entwurfsräume systematisch zu untersuchen. In dieser Dissertation wird daher untersucht, wie Materialität und Digitalität nahtlos ineinander übergehen können. Es soll erforscht werden, wie künftige Benutzungsschnittstellen nützliche digitale Funktionen bereitstellen können, ohne ihre physischen Eigenschaften und vertrauten Nutzungsmuster in der realen Welt zu verlieren. Die Entwicklung solcher hybriden Ansätze wirft jedoch übergreifende Forschungsfragen zum Design auf: Welche Arten von physischen Schnittstellen sind es wert, betrachtet zu werden? Welche Art von digitaler Erweiterung verbessert das Bestehende? Wie können hybride Konzepte ihre physischen Eigenschaften beibehalten und gleichzeitig neue digitale Funktionen ermöglichen? Was sind geeignete Methoden, um verschiedene Designs zu erforschen? Wie kann man Technologiebegeisterte bei der Erstellung von Prototypen unterstützen? Für eine systematische Untersuchung stützt sich die Arbeit auf einen designorientierten, explorativen und iterativen Entwicklungsprozess unter Verwendung digitaler Fabrikationsmethoden und neuartiger Materialien. Im Hauptteil werden vier Forschungsprojekte vorgestellt, die verschiedene visuelle und interaktive Prinzipien entlang realer Anwendungen diskutieren. Die Szenarien reichen von digital angereichertem Papier, interaktiven Kordeln über visuelle Erweiterungen von Uhrarmbändern bis hin zu neuartigen Prototyping-Tools für intelligente Kleidungsstücke. Um neue Designansätze aufzuzeigen, integrieren nahezu alle visuelles Feedback und haptische Eingaben, um Alternativen zu Standard-Eingabemodalitäten auf starren Pixelbildschirmen zu schaffen. Die Dissertation hat gezeigt, wie wertvoll es sein kann, bekannte, analoge Anwendungen zu überdenken und sie dabei gleichzeitig mit Bedacht digital zu erweitern. Dabei umfasst die vorliegende Arbeit sowohl realisierte technische Forschungsplattformen als auch übergreifende konzeptionelle Arbeiten, Nutzerstudien und technische Experimente sowie die Analyse existierender Forschungsarbeiten

Capacitive Sensing and Communication for Ubiquitous Interaction and Environmental Perception

During the last decade, the functionalities of electronic devices within a living environment constantly increased. Besides the personal computer, now tablet PCs, smart household appliances, and smartwatches enriched the technology landscape. The trend towards an ever-growing number of computing systems has resulted in many highly heterogeneous human-machine interfaces. Users are forced to adapt to technology instead of having the technology adapt to them. Gathering context information about the user is a key factor for improving the interaction experience. Emerging wearable devices show the benefits of sophisticated sensors which make interaction more efficient, natural, and enjoyable. However, many technologies still lack of these desirable properties, motivating me to work towards new ways of sensing a user's actions and thus enriching the context. In my dissertation I follow a human-centric approach which ranges from sensing hand movements to recognizing whole-body interactions with objects. This goal can be approached with a vast variety of novel and existing sensing approaches. I focused on perceiving the environment with quasi-electrostatic fields by making use of capacitive coupling between devices and objects. Following this approach, it is possible to implement interfaces that are able to recognize gestures, body movements and manipulations of the environment at typical distances up to 50cm. These sensors usually have a limited resolution and can be sensitive to other conductive objects or electrical devices that affect electric fields. The technique allows for designing very energy-efficient and high-speed sensors that can be deployed unobtrusively underneath any kind of non-conductive surface. Compared to other sensing techniques, exploiting capacitive coupling also has a low impact on a user's perceived privacy. In this work, I also aim at enhancing the interaction experience with new perceptional capabilities based on capacitive coupling. I follow a bottom-up methodology and begin by presenting two low-level approaches for environmental perception. In order to perceive a user in detail, I present a rapid prototyping toolkit for capacitive proximity sensing. The prototyping toolkit shows significant advancements in terms of temporal and spatial resolution. Due to some limitations, namely the inability to determine the identity and fine-grained manipulations of objects, I contribute a generic method for communications based on capacitive coupling. The method allows for designing highly interactive systems that can exchange information through air and the human body. I furthermore show how human body parts can be recognized from capacitive proximity sensors. The method is able to extract multiple object parameters and track body parts in real-time. I conclude my thesis with contributions in the domain of context-aware devices and explicit gesture-recognition systems

Ami-deu : un cadre sémantique pour des applications adaptables dans des environnements intelligents

Cette thèse vise à étendre l’utilisation de l'Internet des objets (IdO) en facilitant le développement d’applications par des personnes non experts en développement logiciel. La thèse propose une nouvelle approche pour augmenter la sémantique des applications d’IdO et l’implication des experts du domaine dans le développement d’applications sensibles au contexte. Notre approche permet de gérer le contexte changeant de l’environnement et de générer des applications qui s’exécutent dans plusieurs environnements intelligents pour fournir des actions requises dans divers contextes. Notre approche est mise en œuvre dans un cadriciel (AmI-DEU) qui inclut les composants pour le développement d’applications IdO. AmI-DEU intègre les services d’environnement, favorise l’interaction de l’utilisateur et fournit les moyens de représenter le domaine d’application, le profil de l’utilisateur et les intentions de l’utilisateur. Le cadriciel permet la définition d’applications IoT avec une intention d’activité autodécrite qui contient les connaissances requises pour réaliser l’activité. Ensuite, le cadriciel génère Intention as a Context (IaaC), qui comprend une intention d’activité autodécrite avec des connaissances colligées à évaluer pour une meilleure adaptation dans des environnements intelligents. La sémantique de l’AmI-DEU est basée sur celle du ContextAA (Context-Aware Agents) – une plateforme pour fournir une connaissance du contexte dans plusieurs environnements. Le cadriciel effectue une compilation des connaissances par des règles et l'appariement sémantique pour produire des applications IdO autonomes capables de s’exécuter en ContextAA. AmI- DEU inclut également un outil de développement visuel pour le développement et le déploiement rapide d'applications sur ContextAA. L'interface graphique d’AmI-DEU adopte la métaphore du flux avec des aides visuelles pour simplifier le développement d'applications en permettant des définitions de règles étape par étape. Dans le cadre de l’expérimentation, AmI-DEU comprend un banc d’essai pour le développement d’applications IdO. Les résultats expérimentaux montrent une optimisation sémantique potentielle des ressources pour les applications IoT dynamiques dans les maisons intelligentes et les villes intelligentes. Notre approche favorise l'adoption de la technologie pour améliorer le bienêtre et la qualité de vie des personnes. Cette thèse se termine par des orientations de recherche que le cadriciel AmI-DEU dévoile pour réaliser des environnements intelligents omniprésents fournissant des adaptations appropriées pour soutenir les intentions des personnes.Abstract: This thesis aims at expanding the use of the Internet of Things (IoT) by facilitating the development of applications by people who are not experts in software development. The thesis proposes a new approach to augment IoT applications’ semantics and domain expert involvement in context-aware application development. Our approach enables us to manage the changing environment context and generate applications that run in multiple smart environments to provide required actions in diverse settings. Our approach is implemented in a framework (AmI-DEU) that includes the components for IoT application development. AmI- DEU integrates environment services, promotes end-user interaction, and provides the means to represent the application domain, end-user profile, and end-user intentions. The framework enables the definition of IoT applications with a self-described activity intention that contains the required knowledge to achieve the activity. Then, the framework generates Intention as a Context (IaaC), which includes a self-described activity intention with compiled knowledge to be assessed for augmented adaptations in smart environments. AmI-DEU framework semantics adopts ContextAA (Context-Aware Agents) – a platform to provide context-awareness in multiple environments. The framework performs a knowledge compilation by rules and semantic matching to produce autonomic IoT applications to run in ContextAA. AmI-DEU also includes a visual tool for quick application development and deployment to ContextAA. The AmI-DEU GUI adopts the flow metaphor with visual aids to simplify developing applications by allowing step-by-step rule definitions. As part of the experimentation, AmI-DEU includes a testbed for IoT application development. Experimental results show a potential semantic optimization for dynamic IoT applications in smart homes and smart cities. Our approach promotes technology adoption to improve people’s well-being and quality of life. This thesis concludes with research directions that the AmI-DEU framework uncovers to achieve pervasive smart environments providing suitable adaptations to support people’s intentions

Tangible auditory interfaces : combining auditory displays and tangible interfaces

Bovermann T. Tangible auditory interfaces : combining auditory displays and tangible interfaces. Bielefeld (Germany): Bielefeld University; 2009.Tangible Auditory Interfaces (TAIs) investigates into the capabilities of the interconnection of Tangible User Interfaces and Auditory Displays. TAIs utilise artificial physical objects as well as soundscapes to represent digital information. The interconnection of the two fields establishes a tight coupling between information and operation that is based on the human's familiarity with the incorporated interrelations. This work gives a formal introduction to TAIs and shows their key features at hand of seven proof of concept applications

Selection Constructive based Hyper-heuristic for Dynamic Scheduling

A função de escalonamento desempenha um papel importante nos sistemas de produção. Os sistemas de escalonamento têm como objetivo gerar um plano de escalonamento que permite gerir de uma forma eficiente um conjunto de tarefas que necessitam de ser executadas no mesmo período de tempo pelos mesmos recursos. Contudo, adaptação dinâmica e otimização é uma necessidade crítica em sistemas de escalonamento, uma vez que as organizações de produção têm uma natureza dinâmica. Nestas organizações ocorrem distúrbios nas condições requisitos de trabalho regularmente e de forma inesperada. Alguns exemplos destes distúrbios são: surgimento de uma nova tarefa, cancelamento de uma tarefa, alteração na data de entrega, entre outros. Estes eventos dinâmicos devem ser tidos em conta, uma vez que podem influenciar o plano criado, tornando-o ineficiente. Portanto, ambientes de produção necessitam de resposta imediata para estes eventos, usando um método de reescalonamento em tempo real, para minimizar o efeito destes eventos dinâmicos no sistema de produção. Deste modo, os sistemas de escalonamento devem de uma forma automática e inteligente, ser capazes de adaptar o plano de escalonamento que a organização está a seguir aos eventos inesperados em tempo real. Esta dissertação aborda o problema de incorporar novas tarefas num plano de escalonamento já existente. Deste modo, é proposta uma abordagem de otimização – Hiper-heurística baseada em Seleção Construtiva para Escalonamento Dinâmico- para lidar com eventos dinâmicos que podem ocorrer num ambiente de produção, a fim de manter o plano de escalonamento, o mais robusto possível. Esta abordagem é inspirada em computação evolutiva e hiper-heurísticas. Do estudo computacional realizado foi possível concluir que o uso da hiper-heurística de seleção construtiva pode ser vantajoso na resolução de problemas de otimização de adaptação dinâmica.Scheduling plays an important role in manufacturing systems. It produces a scheduling plan, in order to share resources to produce several different products in the same time period. However, dynamic adaptation and optimization is a critical need in real-world manufacturing scheduling systems, since contemporary manufacturing organizations have a dynamic nature, where disturbances on working conditions and requirements occur on a continuous basis. Disturbances often arise unexpectedly, and can be for example: urgent job arrival, job cancelation, due date change, delay in the arrival, among others. These dynamic events must be taken into account, since they may have a major impact on the scheduling plan, they can disorder the plan making it ineffective. Therefore, manufacturing environments require immediate response to these dynamic events, using a real-time rescheduling method, in order to minimize the effect of such unexpected events in the performance of the production’ system. As result, scheduling systems should have the ability of automatically and intelligently maintain real-time adaptation and optimization to efficiently update the scheduling plan to the unexpected events. This way, the organization keeps clients satisfied and achieves its objectives (costs minimized and profits maximized). This dissertation addresses the problem of incorporating new tasks in a scheduling plan already generated by the scheduling system. Therefore, it proposes an optimization approach - Selection Constructive based Hyper-heuristic for Dynamic Scheduling - to deal with dynamic events that can occur over time in a manufacturing environment, with the main goal of maintaining the current scheduling plan feasible and most robust as possible. The development of this dynamic adaptation approach is inspired on evolutionary computation and hyper-heuristics. The viability of the proposed approach is tested by performing a set of experiments and analysing the results achieved. From the obtained results it is possible to conclude that the use of a selection constructive hyper-heuristic could be advantageous on solving dynamic adaptation optimization problems