GAINE - A Portable Framework for the Development of Edutainment Applications Based on Multitouch and Tangible Interaction

In the last few years, Multitouch and Tangible User Interfaces have emerged as a powerful tool to integrate interactive surfaces and responsive spaces that embody digital information. Besides providing a natural interaction with digital contents, they allow the interaction of multiple users at the same time, thus promoting collaborative activities and information sharing. In particular, these characteristics have opened new exploration possibilities in the edutainment context, as witnessed by the many applications successfully developed in different areas, from children’s collaborative learning to interactive storytelling, cultural heritage and medical therapy support. However, due to the availability of different multitouch and tangible interaction technologies and of different target computing platforms, the development and deployment of such applications can be challenging. To this end, in this paper we present GAINE (tanGible Augmented INteraction for Edutainment), a software framework that enables rapid prototyping and development of tangible augmented applications for edutainment purposes. GAINE has two main features. First, it offers developers high-level context specific constructs that significantly reduces the implementation burden. Second, the framework is portable on different operating systems and offers independence from the underlying hardware and tracking technology. In this paper, we also discuss several case studies to show the effectiveness of GAINE in simplifying the development of entertainment and edutainment applications based on multitouch and tangible interaction

Subjective User Experience and Performance with Active Tangibles on a Tabletop Interfaces

We developed active tangibles (Sensators) that can be used in combination with multitouch tabletops and that can provide multisensory (visual, auditory, and vibrotactile) feedback. For spatial alignment and rotation tasks we measured subjective user experience and objective performance with these Sensators. We found that active feedback increased accuracy in both tasks, for all feedback modalities. Active visual feedback yielded the highest overall subjective user experience and preference scores. Our contribution is that active feedback improves subjectively perceived performance and reduces perceived mental workload. Additionally, our findings indicate that users prefer to be guided by visual signs over auditory and vibrotactile signs

deForm: An interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch

We introduce a novel input device, deForm, that supports 2.5D touch gestures, tangible tools, and arbitrary objects through real-time structured light scanning of a malleable surface of interaction. DeForm captures high-resolution surface deformations and 2D grey-scale textures of a gel surface through a three-phase structured light 3D scanner. This technique can be combined with IR projection to allow for invisible capture, providing the opportunity for co-located visual feedback on the deformable surface. We describe methods for tracking fingers, whole hand gestures, and arbitrary tangible tools. We outline a method for physically encoding fiducial marker information in the height map of tangible tools. In addition, we describe a novel method for distinguishing between human touch and tangible tools, through capacitive sensing on top of the input surface. Finally we motivate our device through a number of sample applications

New generation of interactive platforms based on novel printed smart materials

Programa doutoral em Engenharia Eletrónica e de Computadores (área de Instrumentação e Microssistemas Eletrónicos)The last decade was marked by the computer-paradigm changing with other digital devices suddenly becoming available to the general public, such as tablets and smartphones. A shift in perspective from computer to materials as the centerpiece of digital interaction is leading to a diversification of interaction contexts, objects and applications, recurring to intuitive commands and dynamic content that can proportionate more interesting and satisfying experiences. In parallel, polymer-based sensors and actuators, and their integration in different substrates or devices is an area of increasing scientific and technological interest, which current state of the art starts to permit the use of smart sensors and actuators embodied within the objects seamlessly. Electronics is no longer a rigid board with plenty of chips. New technological advances and perspectives now turned into printed electronics in polymers, textiles or paper. We are assisting to the actual scaling down of computational power into everyday use objects, a fusion of the computer with the material. Interactivity is being transposed to objects erstwhile inanimate. In this work, strain and deformation sensors and actuators were developed recurring to functional polymer composites with metallic and carbonaceous nanoparticles (NPs) inks, leading to capacitive, piezoresistive and piezoelectric effects, envisioning the creation of tangible user interfaces (TUIs). Based on smart polymer substrates such as polyvinylidene fluoride (PVDF) or polyethylene terephthalate (PET), among others, prototypes were prepared using piezoelectric and dielectric technologies. Piezoresistive prototypes were prepared with resistive inks and restive functional polymers. Materials were printed by screen printing, inkjet printing and doctor blade coating. Finally, a case study of the integration of the different materials and technologies developed is presented in a book-form factor.A última década foi marcada por uma alteração do paradigma de computador pelo súbito aparecimento dos tablets e smartphones para o público geral. A alteração de perspetiva do computador para os materiais como parte central de interação digital levou a uma diversificação dos contextos de interação, objetos e aplicações, recorrendo a comandos intuitivos e conteúdos dinâmicos capazes de tornarem a experiência mais interessante e satisfatória. Em simultâneo, sensores e atuadores de base polimérica, e a sua integração em diferentes substratos ou dispositivos é uma área de crescente interesse científico e tecnológico, e o atual estado da arte começa a permitir o uso de sensores e atuadores inteligentes perfeitamente integrados nos objetos. Eletrónica já não é sinónimo de placas rígidas cheias de componentes. Novas perspetivas e avanços tecnológicos transformaram-se em eletrónica impressa em polímeros, têxteis ou papel. Neste momento estamos a assistir à redução da computação a objetos do dia a dia, uma fusão do computador com a matéria. A interatividade está a ser transposta para objetos outrora inanimados. Neste trabalho foram desenvolvidos atuadores e sensores e de pressão e de deformação com recurso a compostos poliméricos funcionais com tintas com nanopartículas (NPs) metálicas ou de base carbónica, recorrendo aos efeitos capacitivo, piezoresistivo e piezoelétrico, com vista à criação de interfaces de usuário tangíveis (TUIs). Usando substratos poliméricos inteligentes tais como fluoreto de polivinilideno (PVDF) ou politereftalato de etileno (PET), entre outos, foi possível a preparação de protótipos de tecnologia piezoelétrica ou dielétrica. Os protótipos de tecnologia piezoresistiva foram feitos com tintas resistivas e polímeros funcionais resistivos. Os materiais foram impressos por serigrafia, jato de tinta, impressão por aerossol e revestimento de lâmina doctor blade. Para terminar, é apresentado um caso de estudo da integração dos diferentes materiais e tecnologias desenvolvidos sob o formato de um livro.This project was supported by FCT – Fundação para a Ciência e a Tecnologia, within the doctorate grant with reference SFRH/BD/110622/2015, by POCH – Programa Operacional Capital Humano, and by EU – European Union

Development platform for elderly-oriented tabletop games

Tese de mestrado integrado. Engenharia Informática e Computação. Universidade do Porto. Faculdade de Engenharia. 201

Modeling On and Above a Stereoscopic Multitouch Display

International audienceWe present a semi-immersive environment for conceptual design where virtual mockups are obtained from gestures we aim to get closer to the way people conceive, create and manipulate three-dimensional shapes. We developed on-and-above-the-surface interaction techniques based on asymmetric bimanual interaction for creating and editing 3D models in a stereoscopic environment. Our approach combines hand and nger tracking in the space on and above a multitouch surface. This combination brings forth an alternative design environment where users can seamlessly switch between interacting on the surface or in the space above it to leverage the bene t of both interaction spaces

AQUA-G: a universal gesture recognition framework

In this thesis, I describe a software architecture and implementation which is designed to ease the process of 1) developing gesture-enabled applications and 2) using multiple disparate interaction devices simultaneously to create gestures. Developing gesture-enabled applications from scratch can be a time-consuming process involving obtaining input from novel input devices, processing that input in order to recognize gestures, and connecting this information to the application. Previously, developers have turned to gesture recognition systems to assist them in developing these applications. However, existing systems to date are limited in flexibility and adaptability. I propose AQUA-G, a universal gesture recognition framework that utilizes a unified event architecture to communicate with a limitless variety of input devices. AQUA-G provides abstraction of gesture recognition and allows developers to write custom gestures. Its features have been driven in part by previous architectures and are partially based on a needs assessment with a sample of developers. This research contributes a scalable and reliable software system for gesture-enabled application development, which makes developing and prototyping novel interaction styles more accessible to a larger development community

Toward New Ecologies of Cyberphysical Representational Forms, Scales, and Modalities

Research on tangible user interfaces commonly focuses on tangible interfaces acting alone or in comparison with screen-based multi-touch or graphical interfaces. In contrast, hybrid approaches can be seen as the norm for established mainstream interaction paradigms. This dissertation describes interfaces that support complementary information mediations, representational forms, and scales toward an ecology of systems embodying hybrid interaction modalities. I investigate systems combining tangible and multi-touch, as well as systems combining tangible and virtual reality interaction. For each of them, I describe work focusing on design and fabrication aspects, as well as work focusing on reproducibility, engagement, legibility, and perception aspects

Casier: Structures for Composing Tangibles and Complementary Interactors for Use Across Diverse Systems

International audienceCasiers are a class of tangible interface elements that structure the physical and functional composition of tangibles and complementary interactors (e.g., buttons and sliders). Casiers allow certain subsets of interactive functionality to be accessible across diverse interactive systems (with and without graphical mediation, employing varied sensing capabilities and supporting software). We illustrate examples of casiers in use, including iterations around a custom walk-up-and-use kiosk, as well as casiers operable across com- mercial platforms of widely varying cost and capability

Software support for multitouch interaction: the end-user programming perspective

Empowering users with tools for developing multitouch interaction is a promising step toward the materialization of ubiquitous computing. This survey frames the state of the art of existing multitouch software development tools from an end-user programming perspective.This research has been partially funded by the EUFP7 project meSch (grant agreement 600851 and CREAx grant (Spanish Ministry of Economy and Competitivity TIN2014-56534-R