D-touch: A Consumer-Grade Tangible Interface Module and Musical Applications

We define a class of tangible media applications that can be implemented on consumer-grade personal computers. These applications interpret user manipulation of physical objects in a restricted space and produce unlocalized outputs. We propose a generic approach to the implementation of such interfaces using flexible fiducial markers, which identify objects to a robust and fast video-processing algorithm, so they can be recognized and tracked in real time. We describe an implementation of the technology, then report two new, flexible music performance applications that demonstrate and validate it

Detection and Identification Techniques for Markers Used in Computer Vision

This paper summarizes and compares techniques for detecting and identifying markers in the context of computer vision. Existing approaches either use correlation, digital or topological methods for marker identification. The comparison points out, that all marker processing algorithms which employ sophisticated digital codes perform more robust and reliable. Existing bit representation schemes for these codes and marker designs are compared with each other. In the overall context it is illustrated, why the marker processing algorithm is the best performer regarding marker occlusion and minimal detectable pattern size

Sonophenology: A Tangible Interface for Sonification of Geo-Spatial Phenological Data at Multiple Time-scales

Presented at the 16th International Conference on Auditory Display (ICAD2010) on June 9-15, 2010 in Washington, DC.Phenology is the study of periodic biological processes, such as when plants flower and birds arrive in the spring. In this paper we sonify phenology data and control the sonification process through a tangible interface consisting of a physical paper map and tracking of fiducial markers. The designed interface enables one or more users to concurrently specify point and range queries in both time and space and receive immediate sonic feedback. This system can be used to study and explore the effects of climate change, both as tool to be used by scientists, and as a way to educate members of the general public

Parallel Image Processing Using a Pure Topological Framework

Image processing is a fundamental operation in many real time applications, where lots of parallelism can be extracted. Segmenting the image into different connected components is the most known operations, but there are many others like extracting the region adjacency graph (RAG) of these regions, or searching for features points, being invariant to rotations, scales, brilliant changes, etc. Most of these algorithms part from the basis of Tracing-type approaches or scan/raster methods. This fact necessarily implies a data dependence between the processing of one pixel and the previous one, which prevents using a pure parallel approach. In terms of time complexity, this means that linear order O(N) (N being the number of pixels) cannot be cut down. In this paper, we describe a novel approach based on the building of a pure Topological framework, which allows to implement fully parallel algorithms. Concerning topological analysis, a first stage is computed in parallel for every pixel, thus conveying the local neighboring conditions. Then, they are extended in a second parallel stage to the necessary global relations (e.g. to join all the pixels of a connected component). This combinatorial optimization process can be seen as the compression of the whole image to just one pixel. Using this final representation, every region can be related with the rest, which yields to pure topological construction of other image operations. Besides, complex data structures can be avoided: all the processing can be done using matrixes (with the same indexation as the original image) and element-wise operations. The time complexity order of our topological approach for a m×n pixel image is near O(log(m+n)), under the assumption that a processing element exists for each pixel. Results for a multicore processor show very good scalability until the memory bandwidth bottleneck is reached, both for bigger images and for much optimized implementations. The inherent parallelism of our approach points to the direction that even better results will be obtained in other less classical computing architectures.1Ministerio de Economía y Competitividad (España) TEC2012-37868-C04-02AEI/FEDER (UE) MTM2016-81030-PVPPI of the University of Sevill

Expanding tangible tabletop interfaces beyond the display

L’augment de popularitat de les taules i superfícies interactives està impulsant la recerca i la innovació en una gran varietat d’àrees, incloent-­‐hi maquinari, programari, disseny de la interacció i noves tècniques d’interacció. Totes, amb l’objectiu de promoure noves interfícies dotades d’un llenguatge més ric, potent i natural. Entre totes aquestes modalitats, la interacció combinada a sobre i per damunt de la superfície de la taula mitjançant tangibles i gestos és actualment una àrea molt prometedora. Aquest document tracta d’expandir les taules interactives més enllà de la superfície per mitjà de l’exploració i el desenvolupament d’un sistema o dispositiu enfocat des de tres vessants diferents: maquinari, programari i disseny de la interacció. Durant l’inici d’aquest document s’estudien i es resumeixen els diferents trets característics de les superfícies interactives tangibles convencionals o 2D i es presenten els treballs previs desenvolupats per l’autor en solucions de programari que acaben resultant en aplicacions que suggereixen l’ús de la tercera dimensió a les superfícies tangibles. Seguidament, es presenta un repàs del maquinari existent en aquest tipus d’interfícies per tal de concebre un dispositiu capaç de detectar gestos i generar visuals per sobre de la superfície, per introduir els canvis realitzats a un dispositiu existent, desenvolupat i cedit per Microsoft Reseach Cambridge. Per tal d’explotar tot el potencial d’aquest nou dispositiu, es desenvolupa un nou sistema de visió per ordinador que estén el seguiment d’objectes i mans en una superfície 2D a la detecció de mans, dits i etiquetes amb sis graus de llibertat per sobre la superfície incloent-­‐hi la interacció tangible i tàctil convencional a la superfície. Finalment, es presenta una eina de programari per a generar aplicacions per al nou sistema i es presenten un seguit d’aplicacions per tal de provar tot el desenvolupament generat al llarg de la tesi que es conclou presentant un seguit de gestos tant a la superfície com per sobre d’aquesta i situant-­‐los en una nova classificació que alhora recull la interacció convencional 2D i la interacció estesa per damunt de la superfície desenvolupada.The rising popularity of interactive tabletops and surfaces is spawning research and innovation in a wide variety of areas, including hardware and software technologies, interaction design and novel interaction techniques, all of which seek to promote richer, more powerful and more natural interaction modalities. Among these modalities, combined interaction on and above the surface, both with gestures and with tangible objects, is a very promising area. This dissertation is about expanding tangible and tabletops surfaces beyond the display by exploring and developing a system from the three different perspectives: hardware, software, and interaction design. This dissertation, studies and summarizes the distinctive affordances of conventional 2D tabletop devices, with a vast literature review and some additional use cases developed by the author for supporting these findings, and subsequently explores the novel and not yet unveiled potential affordances of 3D-­‐augmented tabletops. It overviews the existing hardware solutions for conceiving such a device, and applies the needed hardware modifications to an existing prototype developed and rendered to us by Microsoft Research Cambridge. For accomplishing the interaction purposes, it is developed a vision system for 3D interaction that extends conventional 2D tabletop tracking for the tracking of hand gestures, 6DoF markers and on-­‐surface finger interaction. It finishes by conceiving a complete software framework solution, for the development and implementation of such type of applications that can benefit from these novel 3D interaction techniques, and implements and test several software prototypes as proof of concepts, using this framework. With these findings, it concludes presenting continuous tangible interaction gestures and proposing a novel classification for 3D tangible and tabletop gestures

Connecting Everyday Objects with the Metaverse: A Unified Recognition Framework

The recent Facebook rebranding to Meta has drawn renewed attention to the metaverse. Technology giants, amongst others, are increasingly embracing the vision and opportunities of a hybrid social experience that mixes physical and virtual interactions. As the metaverse gains in traction, it is expected that everyday objects may soon connect more closely with virtual elements. However, discovering this "hidden" virtual world will be a crucial first step to interacting with it in this new augmented world. In this paper, we address the problem of connecting physical objects with their virtual counterparts, especially through connections built upon visual markers. We propose a unified recognition framework that guides approaches to the metaverse access points. We illustrate the use of the framework through experimental studies under different conditions, in which an interactive and visually attractive decoration pattern, an Artcode, is used as the approach to enable the connection. This paper will be of interest to, amongst others, researchers working in Interaction Design or Augmented Reality who are seeking techniques or guidelines for augmenting physical objects in an unobtrusive, complementary manner.Comment: This paper includes 6 pages, 4 figures, and 1 table, and has been accepted to be published by the 2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), Los Alamitos, CA, US

An Abstraction Framework for Tangible Interactive Surfaces

This cumulative dissertation discusses - by the example of four subsequent publications - the various layers of a tangible interaction framework, which has been developed in conjunction with an electronic musical instrument with a tabletop tangible user interface. Based on the experiences that have been collected during the design and implementation of that particular musical application, this research mainly concentrates on the definition of a general-purpose abstraction model for the encapsulation of physical interface components that are commonly employed in the context of an interactive surface environment. Along with a detailed description of the underlying abstraction model, this dissertation also describes an actual implementation in the form of a detailed protocol syntax, which constitutes the common element of a distributed architecture for the construction of surface-based tangible user interfaces. The initial implementation of the presented abstraction model within an actual application toolkit is comprised of the TUIO protocol and the related computer-vision based object and multi-touch tracking software reacTIVision, along with its principal application within the Reactable synthesizer. The dissertation concludes with an evaluation and extension of the initial TUIO model, by presenting TUIO2 - a next generation abstraction model designed for a more comprehensive range of tangible interaction platforms and related application scenarios

From Codes to Patterns: Designing Interactive Decoration for Tableware

ABSTRACT We explore the idea of making aesthetic decorative patterns that contain multiple visual codes. We chart an iterative collaboration with ceramic designers and a restaurant to refine a recognition technology to work reliably on ceramics, produce a pattern book of designs, and prototype sets of tableware and a mobile app to enhance a dining experience. We document how the designers learned to work with and creatively exploit the technology, enriching their patterns with embellishments and backgrounds and developing strategies for embedding codes into complex designs. We discuss the potential and challenges of interacting with such patterns. We argue for a transition from designing ‘codes to patterns’ that reflects the skills of designers alongside the development of new technologies