Creating a responsive, real-time, malleable environment for dancers

The traditional choreographer/dancer paradigm uses the dancer as a tool of the choreographer. By contrast, a real-time responsive environment, consisting of rendered video and motion-generated video and sound allows the dancer to become a co-creator in both the rehearsal process and the performances. A malleable visual and sonic environment means that the dancer\u27s relation to their environment becomes a self-directed mixture of expressive and responsive elements. This results in a more authentic experience, as the dancer can use his or her movement to not just alter the environment, but use that environment to accompany the dancer rather than having the environment direct the dancer. However, the selection of technology for such a malleable dance-performance envionment is by no means trivial. From piezo and flex sensors, radio frequency transceivers and bluetooth radios, to the ultimate development of a novel sensor system (the VDancer) that can eliminate the need for wires and breakable flex sensors, this thesis details the search for a system that is tough enough to withstand hard use no by modern dancers, and inexpensive enough for a modern dance company. Part of the intent of this thesis is to detail the application of interactive technologies that was developed to allow dancers to manipulate sound and video projections on any stage. This thesis also documents similar previous work, the development of the VDancer, and a number of avenues for advancing that system and the interactions it can offer users

Dancing into the digital age: experimenting the digitization of the Pauliteiros folk dances

Digitizing intangible cultural heritage is a crucial step in preserving and protecting traditions, customs, and knowledge that have been passed down from generation to generation. By creating a comprehensive record of intangible cultural practices, digitization allows for greater access, research, and education, promoting cultural diversity and understanding. This paper discusses approaches to the digitization of intangible cultural heritage, which allow the dissemination and preservation of knowledge applicable to folk dances, particularly in the context of ethnographic museums. Possible applications with this motion data will also be explored, such as Virtual Reality, Augmented Reality or video games. This paper finally discusses and proposes a framework for digitization of folk dances based on the practical case of the 'Dança dos Paulitos' or 'Dança dos Pauliteiros' [1], a Portuguese festive dance ritual tradition from Miranda do Douro in northeastern Portugal.info:eu-repo/semantics/publishedVersio

The integrated sound, space and movement environment : The uses of analogue and digital technologies to correlate topographical and gestural movement with sound

This thesis investigates correlations between auditory parameters and parameters associated with movement in a sensitised space. The research examines those aspects of sound that form correspondences with movement, force or position of a body or bodies in a space sensitised by devices for acquiring gestural or topographical data. A wide range of digital technologies are scrutinised to establish what the most effective technologies are in order to achieve detailed and accurate information about movement in a given space, and the methods and procedures for analysis, transposition and synthesis into sound. The thesis describes pertinent work in the field from the last 20 years, the issues that have been raised in those works and issues raised by my work in the area. The thesis draws conclusions that point to further development of an integrated model of a space that is sensitised to movement, and responds in sound in such a way that it can be appreciated by performers and audiences. The artistic and research practices that are cited, are principally from the areas of danceand- technology, sound installation and alternative gestural controllers for musical applications

4DHumanOutfit: a multi-subject 4D dataset of human motion sequences in varying outfits exhibiting large displacements

This work presents 4DHumanOutfit, a new dataset of densely sampled spatio-temporal 4D human motion data of different actors, outfits and motions. The dataset is designed to contain different actors wearing different outfits while performing different motions in each outfit. In this way, the dataset can be seen as a cube of data containing 4D motion sequences along 3 axes with identity, outfit and motion. This rich dataset has numerous potential applications for the processing and creation of digital humans, e.g. augmented reality, avatar creation and virtual try on. 4DHumanOutfit is released for research purposes at https://kinovis.inria.fr/4dhumanoutfit/. In addition to image data and 4D reconstructions, the dataset includes reference solutions for each axis. We present independent baselines along each axis that demonstrate the value of these reference solutions for evaluation tasks

DeepMoCap: Deep Optical Motion Capture Using Multiple Depth Sensors and Retro-Reflectors

In this paper, a marker-based, single-person optical motion capture method (DeepMoCap) is proposed using multiple spatio-temporally aligned infrared-depth sensors and retro-reflective straps and patches (reflectors). DeepMoCap explores motion capture by automatically localizing and labeling reflectors on depth images and, subsequently, on 3D space. Introducing a non-parametric representation to encode the temporal correlation among pairs of colorized depthmaps and 3D optical flow frames, a multi-stage Fully Convolutional Network (FCN) architecture is proposed to jointly learn reflector locations and their temporal dependency among sequential frames. The extracted reflector 2D locations are spatially mapped in 3D space, resulting in robust 3D optical data extraction. The subject’s motion is efficiently captured by applying a template-based fitting technique on the extracted optical data. Two datasets have been created and made publicly available for evaluation purposes; one comprising multi-view depth and 3D optical flow annotated images (DMC2.5D), and a second, consisting of spatio-temporally aligned multi-view depth images along with skeleton, inertial and ground truth MoCap data (DMC3D). The FCN model outperforms its competitors on the DMC2.5D dataset using 2D Percentage of Correct Keypoints (PCK) metric, while the motion capture outcome is evaluated against RGB-D and inertial data fusion approaches on DMC3D, outperforming the next best method by 4.5% in total 3D PCK accuracy

Ergowear: desenvolvimento de um vestuário inteligente para monitorização postural e biofeedback

Dissertação de mestrado em Engenharia Biomédica (especialização em Eletrónica Médica)Atualmente, as Lesões Musculoesqueléticas Relacionadas com o Trabalho (LMERT) são considera das o ”problema relacionado com o trabalho mais prevalente”na União Europeia, levando a um custo estimado de cerca de 240 biliões de euros. Em casos mais severos, estes distúrbios podem causar danos vitalícios à saúde do trabalhador, reduzindo a sua qualidade de vida. De facto, LMERTs são con sideradas a principal causa da reforma precoce dos trabalhadores. Foi reportado que os segmentos da parte superior do corpo são mais suceptíveis ao desenvolvimento de LMERTs. Para mitigar a prevalência de LMERTs, ergonomistas maioritariamente aplicam métodos de avaliação observacionais, que são alta mente dependentes da experiência do analista, e apresentam baixa objetividade e repetibilidade. Desta maneira, esforços têm sido feitos para desenvolver ferramentas de avaliação ergonómica baseadas na instrumentação, para compensar essas limitações. Além disso, com a ascensão do conceito da indústria 5.0, o trabalhador humano volta a ser o foco principal na indústria, juntamente com o robô colaborativo. No entanto, para alcançar uma relação verdadeiramente colaborativa e simbiótica entre o trabalhador e o robô, este último precisa de reconhecer as intenções do trabalhador. Para superar este obstáculo, sis temas de captura de movimento podem ser integrados nesta estrutura, fornecendo dados de movimento ao robô colaborativo. Esta dissertação visa a melhoria de um sistema de captura de movimento autónomo, da parte supe rior do corpo, de abordagem inercial que servirá, não apenas para monitorizar a postura do trabalhador, mas também avaliar a ergonomia do usuário e fornecer consciencialização postural ao usuário, por meio de motores de biofeedback. Além disso, o sistema foi já idealizado tendo em mente a sua integração numa estrutura colaborativa humano-robô. Para atingir estes objetivos, foi aplicada uma metodologia de design centrado no utilizador, começando pela análise do Estado da Arte, a avaliação das limitações do sistema anterior, a definição dos requisitos do sistema, o desenvolvimento da peça de vestuário, arquite tura do hardware e arquitetura do software do sistema. Por fim, o sistema foi validado para verificar se estava em conformidade com os requisitos especificados. O sistema é composto por 9 Unidades de Medição Inercial (UMI), posicionados na parte inferior e superior das costas, cabeça, braços, antebraços e mãos. Também foi integrado um sistema de atuação, para biofeedback postural, composto por 6 motores vibrotáteis, localizados na região lombar e próximo do pescoço, cotovelos e pulsos. O sistema é alimentado por uma powerbank e todos os dados adquiridos são enviados para uma estação de processamento, via WiFi (User Datagram Protocol (UDP)), garantindo autonomia. O sistema tem integrado um filtro de fusão Complementar Extendido e uma sequência de calibração Sensor-para-Segmento estática, de maneira a aumentar a precisão da estimativa dos ângulos das articulações. Além disso, o sistema é capaz de amostrar os dados angulares a 240 Hz, enquanto que o sistema anterior era capaz de amostrar no máximo a 100 Hz, melhorando a resolução da aquisição dos dados. O sistema foi validado em termos de hardware e usabilidade. Os testes de hardware abordaram a caracterização da autonomia, frequência de amostragem, robustez mecânica e desempenho da comuni cação sem fio do sistema, em diversos contextos, e também para verificar se estes estão em conformidade com os requisitos técnicos previamente definidos, que foi o caso. Adicionalmente, as especificações da nova versão do sistema foram comparadas com a anterior, onde se observou uma melhoria direta signifi cativa, como por exemplo, maior frequência de amostragem, menor perda de pacote, menor consumo de corrente, entre outras, e com sistemas comerciais de referência (XSens Link). Testes de usabilidade foram realizados com 9 participantes que realizaram vários movimentos uniarticulares e complexos. Após os testes, os usuários responderam a um questionário baseado na Escala de Usabilidade do Sistema (EUS). O sistema foi bem aceite pelos os usuários, em termos de estética e conforto, em geral, comprovando um elevado nível de vestibilidade.Nowadays, Work-Related Musculoskeletal Disorders (WRMSDs) are considered the ”most prevalent work-related problem” in the European Union (EU), leading to an estimated cost of about 240 billion EUR. In more severe cases, these disorders can cause life-long impairments to the workers’ health, reducing their quality of life. In fact, WRMSDs are the main cause for the workers’ early retirement. It was reported that the upper body segments of the worker are more susceptible to the development of WRMSDs. To mitigate the prevalence of WRMSD, ergonomists mostly apply observational assessment methods, which are highly dependant on the analyst’s expertise, have low objectivity and repeatability. Therefore, efforts have been made to develop instrumented-based ergonomic assessment tools, to compensate for these limitations. Moreover, with the rise of the 5.0 industry concept, the human worker is once again the main focus in the industry, along with the Collaborative Robot (cobot). However, to achieve a truly collaborative relation between the worker and the cobot, the latter needs to know the worker’s intentions. To surpass this obstacle, Motion Capture (MoCap) systems can be integrated in this framework, providing motion data to the cobot. This dissertation aims at the improvement of a stand-alone, upper-body, inertial, MoCap system, that will serve to not only monitor the worker’s posture, but also to assess the user’s ergonomics and provide posture awareness to the user, through biofeedback motors. Furthermore, it was also designed to integrate a human-robot collaborative framework. To achieve this, a user-centred design methodology was applied, starting with analyzing the State of Art (SOA), assessing the limitations of the previous system, defining the system’s requirements, developing the garment, hardware architecture and software architecture of the system. Lastly, the system was validated to ascertain if it is in conformity with the specified requirements. The developed system is composed of 9 Inertial Measurement Units (IMUs), placed on the lower and upper back, head, upper arms, forearms and hands. An actuation system was also integrated, for postural biofeedback, and it is comprised of 6 vibrotactile motors, located in the lower back, and in close proximity to the neck, elbows and wrists. The system is powered by a powerbank and all of the acquired data is sent to a main station, via WiFi (UDP), granting a standalone characteristic. The system integrates an Extended Complementary Filter (ECF) and a static Sensor-to-Segment (STS) calibration sequence to increase the joint angle estimation accuracy. Furthermore, the system is able to sample the angular data at 240 Hz, while the previous system was able to sample it at a maximum 100 Hz, improving the resolution of the data acquisition. The system was validated in terms of hardware and usability. The hardware tests addressed the char acterization of the system’s autonomy, sampling frequency, mechanical robustness and wireless commu nication performance in different contexts, and ascertain if they comply with the technical requirements, which was the case. Moreover, the specifications of the new version were compared with the previous one, where a significant direct improvement was observed, such as, higher sampling frequency, lower packet loss, lower current consumption, among others, and with a commercial system of reference (XSens Link). Usability tests were carried out with 9 participants who performed several uni-joint and complex motions. After testing, users answered a questionnaire based on the System Usability Scale (SUS). The system was very well accepted by the participants, regarding aesthetics and overall comfort, proving to have a high level of wearability

Aesthetic potential of human-computer interaction in performing arts

Human-computer interaction (HCI) is a multidisciplinary area that studies the communication between users and computers. In this thesis, we want to examine if and how HCI when incorporated into staged performances can generate new possibilities for artistic expression on stage. We define and study four areas of technology-enhanced performance that were strongly influenced by HCI techniques: multimedia expression, body representation, body augmentation and interactive environments. We trace relevant artistic practices that contributed to the exploration of these topics and then present new forms of creative expression that emerged after the incorporation of HCI techniques. We present and discuss novel practices like: performer and the media as one responsive entity, real-time control of virtual characters, on-body projections, body augmentation through humanmachine systems and interactive stage design. The thesis concludes by showing some concrete examples of these novel practices implemented in performance pieces. We present and discuss technologyaugmented dance pieces developed during this master’s degree. We also present a software tool for aesthetic visualisation of movement data and discuss its application in video creation, staged performances and interactive installations

A systematic review of smart clothing in sports: Possible applications to extreme sports

Background. Traditional monitoring of athletes during sports has long been hampered by bulky, complicated and tethered systems. In the past decade, this has changed due to the miniaturization of sensors and improvement of systems to store and transmit data. These systems have been integrated into textiles to create ‘smart clothing’ which has been so ubiquitous that a review of the recent literature is crucial for understanding its full potential and potential use in extreme sports. Methods. An electronic data base search was performed from 2003 to April 2019 for full length articles including “Smart” AND “Clothing” OR “Clothing” AND “Sport(s)” written in English with human subjects. Articles were evaluated according to the Newcastle-Ottawa Scale. Results. Twenty-four studies resulted in 18 systems comprised of 22 types of clothing with various capabilities, including: monitoring heart rate, electromyography, respiratory rate, steps, GPS, energy expenditure, posture, body temperature and identifying the activity. Conclusions. Many types of smart clothing from socks and gloves, to pants, shirts and bras are increasingly utilized to monitor sports activity worldwide and gather previously unavailable, yet highly valuable data. This provides a unique opportunity to study athletes during training and competition, potentially providing more effective training and better safety protocols

Low-Cost Sensors and Biological Signals

Many sensors are currently available at prices lower than USD 100 and cover a wide range of biological signals: motion, muscle activity, heart rate, etc. Such low-cost sensors have metrological features allowing them to be used in everyday life and clinical applications, where gold-standard material is both too expensive and time-consuming to be used. The selected papers present current applications of low-cost sensors in domains such as physiotherapy, rehabilitation, and affective technologies. The results cover various aspects of low-cost sensor technology from hardware design to software optimization

Markerless motion capture system via kinematic analysis of angular lower limb

The introduction of markerless sensor technology in motion capture system offers a comparable alternative to the conventional systems by employing infrared-depth sensors and retaining the ability to acquire two (2D) and three-dimensional (3D) data on human movement. However, its accuracy is often questioned compared to the established technologies such as passive marker systems. Therefore, this study sets an alternative method to evaluate Kinect Xbox 360 markerless system accuracy based on two positioning coordinates of two pairs of sensors. Through this approach, the length of lower limb segments was measured in 2D and 3D on each motion frame while performing squat movement and compared with the actual segment length. Interestingly, all segment lengths in the 3D showed excellent accuracy with the actual length of the segment. The angle of knee joints was also evaluated to identify the types of squat movements. The same evaluation is also used for the accuracy of a passive marker system while capturing the turning kick motion. In addition, the velocity of the knee joint was also studied at each phase of movement to determine the speed and angular of the knee required to enable the subject's foot to reach the target. For validation purposes, simulations of all recorded motions were implemented to evaluate the squat and the phases of movement in a turning kick from a visual angle. Successfully, the study was able to compare the accuracy and precision of the system constructed using lower limb data relative to the passive marker system using actual lower limb data. The markerless gave a remarkable difference value between the highest and lowest percentage coefficients of variation with 3.90%, while the passive marker system gave 5.72%. It is suggested that the multi-camera markerless motion capture system used in this study be used only for applications that do not require a significant level of accuracy such as animations, gaming and recreational sports analyses