Exploring Choreographers’ Conceptions of Motion Capture for Full Body Interaction

We present the results of a group interview of choreographers aimed at understanding their conceptions of how movement can be used to in live performance. This understanding intended to inform research into full body interaction for live performance and other more general full body interfaces. The results of the interview suggest a new way of conceiving of interaction with digital technology, neither as a representation of movement, not as an interface that responds to movement but as a means of transforming movement. This transformed movement can then serve as a starting point for a dancers responses to transformations of their own movement thus setting up an improvisational feedback loop

Establishing a Framework for the development of Multimodal Virtual Reality Interfaces with Applicability in Education and Clinical Practice

The development of Virtual Reality (VR) and Augmented Reality (AR) content with multiple sources of both input and output has led to countless contributions in a great many number of fields, among which medicine and education. Nevertheless, the actual process of integrating the existing VR/AR media and subsequently setting it to purpose is yet a highly scattered and esoteric undertaking. Moreover, seldom do the architectures that derive from such ventures comprise haptic feedback in their implementation, which in turn deprives users from relying on one of the paramount aspects of human interaction, their sense of touch. Determined to circumvent these issues, the present dissertation proposes a centralized albeit modularized framework that thus enables the conception of multimodal VR/AR applications in a novel and straightforward manner. In order to accomplish this, the aforesaid framework makes use of a stereoscopic VR Head Mounted Display (HMD) from Oculus Rift©, a hand tracking controller from Leap Motion©, a custom-made VR mount that allows for the assemblage of the two preceding peripherals and a wearable device of our own design. The latter is a glove that encompasses two core modules in its innings, one that is able to convey haptic feedback to its wearer and another that deals with the non-intrusive acquisition, processing and registering of his/her Electrocardiogram (ECG), Electromyogram (EMG) and Electrodermal Activity (EDA). The software elements of the aforementioned features were all interfaced through Unity3D©, a powerful game engine whose popularity in academic and scientific endeavors is evermore increasing. Upon completion of our system, it was time to substantiate our initial claim with thoroughly developed experiences that would attest to its worth. With this premise in mind, we devised a comprehensive repository of interfaces, amid which three merit special consideration: Brain Connectivity Leap (BCL), Ode to Passive Haptic Learning (PHL) and a Surgical Simulator

Review: Development and technical design of tangible user interfaces in wide-field areas of application

A tangible user interface or TUI connects physical objects and digital interfaces. It is more interactive and interesting for users than a classic graphic user interface. This article presents a descriptive overview of TUI's real-world applications sorted into ten main application areas-teaching of traditional subjects, medicine and psychology, programming, database development, music and arts, modeling of 3D objects, modeling in architecture, literature and storytelling, adjustable TUI solutions, and commercial TUI smart toys. The paper focuses on TUI's technical solutions and a description of technical constructions that influences the applicability of TUIs in the real world. Based on the review, the technical concept was divided into two main approaches: the sensory technical concept and technology based on a computer vision algorithm. The sensory technical concept is processed to use wireless technology, sensors, and feedback possibilities in TUI applications. The image processing approach is processed to a marker and markerless approach for object recognition, the use of cameras, and the use of computer vision platforms for TUI applications.Web of Science2113art. no. 425

Pathway to Future Symbiotic Creativity

This report presents a comprehensive view of our vision on the development path of the human-machine symbiotic art creation. We propose a classification of the creative system with a hierarchy of 5 classes, showing the pathway of creativity evolving from a mimic-human artist (Turing Artists) to a Machine artist in its own right. We begin with an overview of the limitations of the Turing Artists then focus on the top two-level systems, Machine Artists, emphasizing machine-human communication in art creation. In art creation, it is necessary for machines to understand humans' mental states, including desires, appreciation, and emotions, humans also need to understand machines' creative capabilities and limitations. The rapid development of immersive environment and further evolution into the new concept of metaverse enable symbiotic art creation through unprecedented flexibility of bi-directional communication between artists and art manifestation environments. By examining the latest sensor and XR technologies, we illustrate the novel way for art data collection to constitute the base of a new form of human-machine bidirectional communication and understanding in art creation. Based on such communication and understanding mechanisms, we propose a novel framework for building future Machine artists, which comes with the philosophy that a human-compatible AI system should be based on the "human-in-the-loop" principle rather than the traditional "end-to-end" dogma. By proposing a new form of inverse reinforcement learning model, we outline the platform design of machine artists, demonstrate its functions and showcase some examples of technologies we have developed. We also provide a systematic exposition of the ecosystem for AI-based symbiotic art form and community with an economic model built on NFT technology. Ethical issues for the development of machine artists are also discussed

ThirdLight: low-cost and high-speed 3D interaction using photosensor markers

We present a low-cost 3D tracking system for virtual reality, gesture modeling, and robot manipulation applications which require fast and precise localization of headsets, data gloves, props, or controllers. Our system removes the need for cameras or projectors for sensing, and instead uses cheap LEDs and printed masks for illumination, and low-cost photosensitive markers. The illumination device transmits a spatiotemporal pattern as a series of binary Gray-code patterns. Multiple illumination devices can be combined to localize each marker in 3D at high speed (333Hz). Our method has strengths in accuracy, speed, cost, ambient performance, large working space (1m-5m) and robustness to noise compared with conventional techniques. We compare with a state-of-the-art instrumented glove and vision-based systems to demonstrate the accuracy, scalability, and robustness of our approach. We propose a fast and accurate method for hand gesture modeling using an inverse kinematics approach with the six photosensitive markers. We additionally propose a passive markers system and demonstrate various interaction scenarios as practical applications

Expanding tangible tabletop interfaces beyond the display

L’augment de popularitat de les taules i superfícies interactives està impulsant la recerca i la innovació en una gran varietat d’àrees, incloent-­‐hi maquinari, programari, disseny de la interacció i noves tècniques d’interacció. Totes, amb l’objectiu de promoure noves interfícies dotades d’un llenguatge més ric, potent i natural. Entre totes aquestes modalitats, la interacció combinada a sobre i per damunt de la superfície de la taula mitjançant tangibles i gestos és actualment una àrea molt prometedora. Aquest document tracta d’expandir les taules interactives més enllà de la superfície per mitjà de l’exploració i el desenvolupament d’un sistema o dispositiu enfocat des de tres vessants diferents: maquinari, programari i disseny de la interacció. Durant l’inici d’aquest document s’estudien i es resumeixen els diferents trets característics de les superfícies interactives tangibles convencionals o 2D i es presenten els treballs previs desenvolupats per l’autor en solucions de programari que acaben resultant en aplicacions que suggereixen l’ús de la tercera dimensió a les superfícies tangibles. Seguidament, es presenta un repàs del maquinari existent en aquest tipus d’interfícies per tal de concebre un dispositiu capaç de detectar gestos i generar visuals per sobre de la superfície, per introduir els canvis realitzats a un dispositiu existent, desenvolupat i cedit per Microsoft Reseach Cambridge. Per tal d’explotar tot el potencial d’aquest nou dispositiu, es desenvolupa un nou sistema de visió per ordinador que estén el seguiment d’objectes i mans en una superfície 2D a la detecció de mans, dits i etiquetes amb sis graus de llibertat per sobre la superfície incloent-­‐hi la interacció tangible i tàctil convencional a la superfície. Finalment, es presenta una eina de programari per a generar aplicacions per al nou sistema i es presenten un seguit d’aplicacions per tal de provar tot el desenvolupament generat al llarg de la tesi que es conclou presentant un seguit de gestos tant a la superfície com per sobre d’aquesta i situant-­‐los en una nova classificació que alhora recull la interacció convencional 2D i la interacció estesa per damunt de la superfície desenvolupada.The rising popularity of interactive tabletops and surfaces is spawning research and innovation in a wide variety of areas, including hardware and software technologies, interaction design and novel interaction techniques, all of which seek to promote richer, more powerful and more natural interaction modalities. Among these modalities, combined interaction on and above the surface, both with gestures and with tangible objects, is a very promising area. This dissertation is about expanding tangible and tabletops surfaces beyond the display by exploring and developing a system from the three different perspectives: hardware, software, and interaction design. This dissertation, studies and summarizes the distinctive affordances of conventional 2D tabletop devices, with a vast literature review and some additional use cases developed by the author for supporting these findings, and subsequently explores the novel and not yet unveiled potential affordances of 3D-­‐augmented tabletops. It overviews the existing hardware solutions for conceiving such a device, and applies the needed hardware modifications to an existing prototype developed and rendered to us by Microsoft Research Cambridge. For accomplishing the interaction purposes, it is developed a vision system for 3D interaction that extends conventional 2D tabletop tracking for the tracking of hand gestures, 6DoF markers and on-­‐surface finger interaction. It finishes by conceiving a complete software framework solution, for the development and implementation of such type of applications that can benefit from these novel 3D interaction techniques, and implements and test several software prototypes as proof of concepts, using this framework. With these findings, it concludes presenting continuous tangible interaction gestures and proposing a novel classification for 3D tangible and tabletop gestures