Leman, Marc

Moens, Bart

van Noorden, Leon

English

Ghent University Academic Bibliography

 1 D-Jogger: a multimodal music interface for music selection based on user step frequency  Bart Moens IPEM,  Ghent University Blandijnberg 2, Ghent, Belgium bmmoens.moens@ugent.be   Leon Van Noorden IPEM,  Ghent University Blandijnberg 2, Ghent, Belgium leonvannoorden@mac.com   Marc Leman IPEM,  Ghent University Blandijnberg 2, Ghent, Belgium marc.leman@ugent.be   ABSTRACT Modern mobile media players have sensors embedded in their hardware, allowing new ways to interact with the device. We build upon D-Jogger, an application that uses an accelerometer and gyroscope to analyze body movement in order to dynamically select music and adapt its tempo to the users‟ pace. Choosing the correct music during training sessions can have a positive effect on the performance, possibly due to the entrainment effect. However, the user control remains an issue: during a run session for example it is impossible to navigate in menus. For this, we propose a method of gesture recognition on a touchscreen for simple user input and feedback, making D-Jogger a multimodal media player. Author Keywords Multimodal interface, BPM aware playlist, pace detection, gesture recognition, entrainment ACM Classification Keywords H5.2. Input devices and strategies.  INTRODUCTION AND RELATED WORK Thanks to recent developments in mobile music players, many people walk or run on music. During such a workout session it can be cumbersome to manually pick the best music; therefore the use of predefined playlists is common. However, this can lead to situations where the music is not adjusted to the users‟ actions, for example a slow song when running at a high tempo. In our previous work, we introduced D-Jogger [1]; a music interface that matches the tempo of the music (beats per minute, BPM) with the walking tempo of the user (steps per minute, SPM), switching songs when appropriate. This method however has a disadvantage over the use of traditional playlists because user preferrence is not taken into account. We propose the use of gesture recognition on the touchscreen for user feedback on the musical choice. The idea of a system that adapts its music to the movement of the user is not novel in itself. Yamaha was the first to introduce BODiBeat1 (2007), followed by Philips Activa2  (2010). Elliott et al. introduced SynchStep3, an iOS application for recent mobile Apple devices. SynchStep is based on the prototype PersonalSoundtrack, presented in [2]. These devices use a traditional menu-driven interface and are focused on selecting music to optimize workout performance, depending on the users‟ pace or heart rate.  D-JOGGER D-Jogger distinguishes itself from the before mentioned devices because we use an alignment algorithm that time stretches the music depending on the users‟ pace. This allows for optimal alignment between steps and music without having to change songs when small gait tempo variations occur. A proof of concept was presented by Hockman et al. in [3]. The prototype of D-Jogger was build using Max/MSP, a graphical programming environment for real time audio processing that uses objects as basic building blocks. Objects developed for Max/MSP by third parties are called externals. D-Jogger consists of several externals, connectable in different ways to create a highly flexible framework for the rapid development of applications involving movement analysis and dynamic playlist generation. Figure 1 provides an overview of the system. The alignment algorithm acts as the „moderator‟ of the music: it decides whether the tempo of the current song should be adapted and whether a new song should be chosen.  We present the Dynamic Song and Tempo (DSaT) algorithm, consisting out of 2 phases:  Song Selection: DSaT starts by choosing a song with a BPM close to the SPM value, taking optional user rating into account.  If no suitable song is found, the SPM value is doubled and the search restarted. This feature is used with very low SPM values (SPM < 90), because very few songs with a low BPM are available [4].                                                             1 http://www.yamaha.com/bodibeat/ 2 http://www.usa.philips.com 3 http://www.synchstep.com                    Dynamic Tempo: the required tempo adjustment for the song is calculated by dividing the SPM value by the BPM value. When this adjustment falls outside predetermined boundaries for more than 5 seconds, a new song is selected. The prototype has been used in several experiments, using a treadmill and an accelerometer placed at the users‟ ankle for reliable pace detection. There are some interesting conclusions about the users‟ entrainment to the beat. Entrainment is the synchronization of a system with a variable frequency to an external frequency [5]. The study showed that when SPM and BPM are sufficiently close to each other, users tend to align their steps to the perceived pulse of the music. Surveys also indicated that users reported feeling more motivated when in sync with the music [1]. The concept of D-Jogger can thus be used as a tool for assisting the user in music selection during workouts. MOBILE D-JOGGER A mobile version of D-Jogger is being implemented on a portable device. The main goal of this version is to perform experiments to indicate the users‟ entrainment in a less constrained environment. However, programming D-Jogger on mobile devices poses several challenges. Because the user has several options to place the device during a workout, for example the upper arm, the belt or in a pocket, pace detection is not as straightforward as in the prototype. Step and pace detection algorithms have to adapt to their location on the body to produce reliable results. A comparative study of pace detection algorithms and sensor locations is being performed. Results will include the best performing algorithm for each typical location. Preliminary data shows promising results for both upper arm and pocket sensor locations. MULTIMODAL INTERFACE Controlling a device during a workout session is a challenging task for the user. In most cases it is necessary to temporarily stop the session in order to navigate through a menu-based interface. D-Jogger minimizes the need to manually control the device, but a basic feedback loop is necessary to cope with user preferences. Basic controls include a positive and negative rating of the current song. A negative rating implies that D-Jogger automatically switches to the next song and avoids the song in the future, while a positive rating results in the song being played more often. The mobile version is implemented on a device featuring a touchscreen. This allows us to develop a user-friendly, gesture based rating system. We propose the following gestures:  Swipe gesture means a negative rating. This common gesture represents throwing away a song.    Double tap for a positive rating The proposed system makes it easy to give feedback to the system during a workout, without interrupting the session. We plan a study in the near future to evaluate the design. CONCLUSION Building upon the prototype of D-Jogger, a mobile version was proposed featuring a multimodal interface. Music selection is based upon walking or running tempo of the user, while intuitive gestures on a touchscreen are used for user feedback on song selection. This combination leads to easy media player control during workout sessions. REFERENCES [1] B. Moens, L. Van Noorden and M. Leman, “D-Jogger: syncing music with walking”, In Proceedings of the 2010 Sound and Music Computing Conference [2] G.T. Elliott, B. Tomlinson, “PersonalSoundtrack: context-aware playlists that adapt to user pace”, CHI '06 extended abstracts on Human factors in computing systems, 2006.  [3] J. Hockman, M.M. Wanderley and I. Fujinaga, “Phase vocoder manipulation by runner‟s pace”, In Proceedings of the 2009 Conference on New Instruments for Musical Expression, pp. 90-93, 2009.  [4] F. Gouyon, “Dance music classification: A tempo-based approach”, In Proceedings of the International Conference on Music Information Retrieval, Vol. 2004, pp. 501-504, 2004. [5] M. Clayton, R. Sager and U. Will, “In Time with the Music: The Concept of Entrainment and its Significance for Ethnomusicology”, European Meetings in Ethnomusicology, Vol.11, pp. 3–142, 2005.  Figure 1. Overview of D-Jogger components D-Jogger Sensor and DSP BPM-aware Playlist Accelerometer Interpolation Low pass filter Autocorrelation Smoothing Alignment algorithm Phase vocoder Audio output Audio file BPM analysis Annotated playlist item: BPM value User rating   Playlist SPM Song selection Audio file BPM Tempo adjustments Audio Tempo-matched audio 

D-Jogger: a multimodal music interface for music selection based on user step frequency

https://biblio.ugent.be/publication/1070515/file/1070517.pdf

D-Jogger: a multimodal music interface for music selection based on user step frequency

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography