Motion Modeling for Expressive Interaction

While human-human or human-object interactions involve very rich, complex and nuanced gestures, gestures as they are captured for human-computer interaction remain relatively simplistic. Our approach is to consider the study of variation of motion input as a way of understanding expression and expressivity in human-computer interaction and in order to propose computational solutions for capturing and using these expressive variations. The paper reports an attempt at drawing the lines of design guidelines for modeling systems adapting to motion variations. We propose to illustrate them through two case studies: the first model is used to estimate temporal and geometrical motion variations while the second is used to track variations of motion dynamics. These case studies are illustrated in two application

Optimising the Unexpected: Computational Design Approach in Expressive Gestural Interaction

In our work on computational design of expressive gestural interaction, we experienced various challenges for advanced optimisation methods. Here we want to highlight two of these challenges based on the design and the use of a Bayesian model called Gesture Variation Follower, with the aim to discuss such challenges with a broader community of designers and HCI practitioners during the workshop

Using models of baseline gameplay to design for physical rehabilitation

Modified digital games manage to drive motivation in repetitive exercises needed for motor rehabilitation, however designing modifications that satisfy both rehabilitation and engagement goals is challenging. We present a method wherein a statistical model of baseline gameplay identifies design configurations that emulate behaviours compatible with unmodified play. We illustrate this approach through a case study involving upper limb rehabilitation with a custom controller for a Pac-Man game. A participatory design workshop with occupational therapists defined two interaction parameters for gameplay and rehabilitation adjustments. The parameters' effect on the interaction was measured experimentally with 12 participants. We show that a low-latency model, using both user input behaviour and internal game state, identifies values for interaction parameters that reproduce baseline gameplay under degraded control. We discuss how this method can be applied to systematically balance gamification problems involving trade-offs between physical requirements and subjectively engaging experiences.Comment: 19 pages, 10 figure

The Machine Learning Algorithm as Creative Musical Tool

Machine learning is the capacity of a computational system to learn structures from datasets in order to make prediction in front of newly seen datasets. Such approach offers a significant advantage in music scenarios in which musicians can teach the system to learn an idiosyncratic style, or can break the rules to explore the system capacity in unexpected ways. In this chapter we draw on music, machine learning, and human-computer interaction to elucidate an understanding of machine learning algorithms as creative tools for music and the sonic arts. We motivate a new understanding of learning algorithms as human-computer interfaces. We show that, like other interfaces, learning algorithms can be characterised by the ways their affordances intersect with goals of human users. We also argue that the nature of interaction between users and algorithms impacts the usability and usefulness of those algorithms in profound ways. This human-centred view of machine learning motivates our concluding discussion of what it means to employ machine learning as a creative tool

Machine Learning of Musical Gestures: Principles and Review

We present an overview of machine learning (ML) techniques and their application in interactive music and new digital instrument design. We first provide the non-specialist reader an introduction to two ML tasks, classification and regression, that are particularly relevant for gestural interaction. We then present a review of the literature in current NIME research that uses ML in musical gesture analysis and gestural sound control. We describe the ways in which machine learning is useful for creating expressive musical interaction, and in turn why live music performance presents a pertinent and challenging use case for machine learning

Interactive Sound Texture Synthesis through Semi-Automatic User Annotations

We present a way to make environmental recordings controllable again by the use of continuous annotations of the high-level semantic parameter one wishes to control, e.g. wind strength or crowd excitation level. A partial annotation can be propagated to cover the entire recording via cross-modal analysis between gesture and sound by canonical time warping (CTW). The annotations serve as a descriptor for lookup in corpus-based concatenative synthesis in order to invert the sound/annotation relationship. The workflow has been evaluated by a preliminary subject test and results on canonical correlation analysis (CCA) show high consistency between annotations and a small set of audio descriptors being well correlated with them. An experiment of the propagation of annotations shows the superior performance of CTW over CCA with as little as 20 s of annotated material

INTERACCIÓN GESTUAL PARA ENTORNOS DE INMERSIÓN ARQUEOLÓGICOS: TRABAJO EN CURSO

[EN] Archaeological data are heterogeneous (i.e., data-sheets and pictures, stratigraphic data, 3D models), and innovative virtual reconstractuions helps to visualize and study those data. In this short paper, we describe our work in progress in the design of an innovative way to interact with the complexity of a virtual reconstruction, using natural gestures and advanced machine learning, in close collaboration with archeaeologists.[ES] Los datos arqueológicos son heterogéneos (por ejemplo, ficha técnica e imágenes, datos estratigráficos y modelos 3D), y las nuevas tecnologías pueden ser capaces de ayudar en la visualizacion y el estudio de dichos datos. En este documento se presenta nuestro trabajo en curso que describe el diseño de una forma innovadora de interactuar con la complejidad de una reconstrucción virtual, mediante gestos naturales y avanzadas técnicas de aprendizaje, en directa colaboración con los arqueólogos.Albertini, N.; Brogni, A.; Caramiaux, B.; Gillies, M.; Olivito, R.; Taccola, E. (2016). NATURAL GESTURE INTERACTION IN ARCHAEOLOGICAL VIRTUAL ENVIRONMENTS: WORK IN PROGRESS. En 8th International congress on archaeology, computer graphics, cultural heritage and innovation. Editorial Universitat Politècnica de València. 284-287. https://doi.org/10.4995/arqueologica8.2016.3400OCS28428

"Explorers of Unknown Planets": Practices and Politics of Artificial Intelligence in Visual Arts

International audienceAlongside recent advances in artificial intelligence (AI), a new art practice has emerged in recent years that borrows and transforms these advances in the production of artworks. The actors of this emergent practice are coming from contemporary art, media and digital arts. These artists have developed an original practice of AI within their creative field. In this article, we propose a qualitative study to explore the nature of this practice. We interviewed five internationally renowned artists about how AI is integrated into their work. Through a thematic analysis of the interviews, we first find that their practice relies on crafting algorithms and data as materials. We uncover how they explicitly use this material unpredictability rather than avoid it. Secondly, we highlight the politics of their practice that consist of resisting the culture of AI research, as well as its inherent power dynamics. We also highlight how their relationship with the technology is imbued with ethics and how they rethink their role with respect to the technology. In this paper, we aim to provide the CSCW community with a way to expand the framework in which AI can be understood not only as a tool but also as cultural and political design material

Natural Gesture Interaction In Archaeological Virtual Environments: Work In Progress

Archaeological data are heterogeneous (i.e., data-sheets and pictures, stratigraphic data, 3D models), and innovative virtual reconstructions help to visualize and study those data. In this short paper, we describe our work in progress in the design of an innovative way to interact with the complexity of a virtual reconstruction, using natural gestures and advanced machine learning, in close collaboration with archaeologists.Los datos arqueológicos son heterogéneos (por ejemplo, ficha técnica e imágenes, datos estratigráficos y modelos 3D), y las nuevas tecnologías pueden ser capaces de ayudar en la visualizacion y el estudio de dichos datos. En este documento se presenta nuestro trabajo en curso que describe el diseño de una forma innovadora de interactuar con la complejidad de una reconstrucción virtual, mediante gestos naturales y avanzadas técnicas de aprendizaje, en directa colaboración con los arqueólogos