BioSync: An Informed Participatory Interface for Audience Dynamics and Audiovisual Content Co-creation using Mobile PPG and EEG

www.yuanyifan.co

Rhythmism: a VJ performance system with maracas based devices

ABSTRACT We propose a nonconventional VJ performance system rhythmism which uses an original maraca based device that has 2 different functions, material maraca and effect maraca. Rhythmism uses the structure characteristic and the operating characteristics of maracas and adopts the interface to ensure the freedom of the user's physical movement and to realize the powerful attraction of the performance itself and aims to attain the multi-functionality and the arbitrary controllability. In this paper, we introduce the advantages and the interaction techniques of the maracas based device when used for VJ performances

A Human-Computer Duet System for Music Performance

Virtual musicians have become a remarkable phenomenon in the contemporary multimedia arts. However, most of the virtual musicians nowadays have not been endowed with abilities to create their own behaviors, or to perform music with human musicians. In this paper, we firstly create a virtual violinist, who can collaborate with a human pianist to perform chamber music automatically without any intervention. The system incorporates the techniques from various fields, including real-time music tracking, pose estimation, and body movement generation. In our system, the virtual musician's behavior is generated based on the given music audio alone, and such a system results in a low-cost, efficient and scalable way to produce human and virtual musicians' co-performance. The proposed system has been validated in public concerts. Objective quality assessment approaches and possible ways to systematically improve the system are also discussed

ElectroCutscenes: Realistic Haptic Feedback in Cutscenes of Virtual Reality Games Using Electric Muscle Stimulation

Cutscenes in Virtual Reality (VR) games enhance story telling by delivering output in the form of visual, auditory, or haptic feedback (e.g., using vibrating handheld controllers). Since they lack interaction in the form of user input, cutscenes would significantly benefit from improved feedback. We introduce the concept and implementation of ElectroCutscenes, a concept in which Electric Muscle Stimulation (EMS) is leveraged to elicit physical user movements to correspond to those of personal avatars in cutscenes of VR games while the user stays passive. Through a user study (N=22) in which users passively received kinesthetic feedback resulting in involuntarily movements, we show that ElectroCutscenes significantly increases perceived presence and realism compared to controller-based vibrotactile and no haptic feedback. Furthermore, we found preliminary evidence that combining visual and EMS feedback can evoke movements that are not actuated by either of them alone. We discuss how to enhance realism and presence of cutscenes in VR games even when EMS can partially rather than completely actuate the desired body movements

デザイン プロセス ニ オケル スケッチ ト ブツリ コンピューティング

マイコンシステムの高性能化とオープンソース・ソフトウェアの潮流から最近注目されている「フィジカル・コンピューティング」について、デザインプロセスおよびメディアアートとの関係に注目して紹介する。具体的なシステムとして4種類のプラットフォームを解説し、大学院レベルのメディアデザイン教育への応用について検討した。合わせて、この分野で近年注目されている国際ワークショップ"Sketching in Hardware 2008"の参加報告を行った。This is a report of physical computing to which people recently pay attention in design process field and media-arts field. I introduce and discuss (1) four popular systems as platforms, (2) consideration for application in graduate course in media design, and (3) report of the international workshop "Sketching 2008"

オンガク ゲイジュツ ヒョウゲン ノ タメ ノ シンインターフェース

コンピュータ音楽、インタラクティブ・メディアアート、メディア・インスタレーションなどに関して、「音楽/芸術表現のための新しいインターフェース」という視点からの報告を行った。具体的には、(1) これまでに開発研究してきたインターフェースの実例概要、(2) NIME (New Interfaces for Musical Expression) という若い国際会議の活動経緯と事例紹介、(3) 2003年5月に研究発表と作品公演で参加した Montreal でのNIMEO3の参加報告、(4) 2004年6月にSUACで開催することになった国際会議MMEO4とメディアアートフェスティバルMAF2004についての状況、などを紹介した。This paper reports some activities in Media Art fields with the keywords "New Interfaces for Musical Expression". I report (1) Some results of my research and creation with new sensing systems, (2) About the new International Conference : NIME (New Interfaces for Musical Expression), (3) Report of the NIME03 held at McGill University in Montreal Canada (May 2003) which I had my presentation and performance of my work, (4) NIME2004 and MAF2004 (Media Art Festival) which will be held at SUAC in June 2004. SUAC will be better known as the pioneer of this field with these activities

Compositional approaches within new media paradigms

"Compositional Approaches to New Media Paradigms" is the discursive accompaniment to the original composition BoMoH, (a new media chamber opera. A variety of new media concepts and practices are discussed in relation to their use as a contemporary compositional methodology for computer musicians and digital content producers. This paper aligns relevant discourse with a variety of concepts as they influence and affect the compositional process. This paper does not propose a new working method; rather it draws attention to a contemporary interdisciplinary practice that facilitates new possibilities for engagement and aesthetics in digital art/music. Finally, in demonstrating a selection of the design principals, from a variety of new media theories of interest, in compositional structure and concept, it is my hope to provide composers and computer musicians with a tested resource that will function as a helpful set of working guidelines for producing new media enabled art, sonic or otherwise

Location-based Responsive Space for Multimedia Environment Using Ultra-wideband Technology

The rapid advances in Real-Time Location Systems (RTLSs), such as Ultra-Wideband (UWB) technology, provide opportunities to realize new types of multimedia environments. Taking advantage of the UWB technology, this research investigates how location information enhances multimedia environments by utilizing the relationship between the physical environment and its habitants. In this thesis, we present an approach, called UWB-based Responsive Multimedia Space (URMS), that combines near real-time location data with multimedia devices. This approach has three main parts: (1) the concept of URMS incorporating multimedia devices for the realization of a series of location-based events, (2) the requirements of the UWB system and multimedia devices for the development of URMS, and (3) the methods for the grid-based event selection, and continuous and smooth adjustment of location-based interaction based on distance. Furthermore, a touch user interface is proposed to define interactively multimedia events. In order to validate the proposed approach, a prototype system is developed and tested in two case studies