This paper describes experiments with a virtual fitness trainer capable of mutually coordinated interaction. The virtual human co-exercises along with the user, leading as well as following in tempo, to motivate the user and to influence the speed with which the user performs the exercises. In a series of three experiments (20 participants in total) we attempted to influence the users' performance by manipulating the (timing of the) exercise behavior of the virtual trainer. The results show that it is possible to do this implicitly, using only micro adjustments to its bodily behavior. As such, the system is a rst step in the direction of mutually coordinated bodily interaction for virtual humans

Dehling, Eike

Nijholt, Anton

Reidsma, Dennis

Welbergen, Herwin van

Zwiers, Job

English

van Welbergen, H.

Zwiers, Jakob

Nijholt, Antinus

University of Twente Research Information

Leading and following with a virtual trainerDennis ReidsmaHuman Media InteractionPO Box 2177500AE EnschedeNetherlandsd.reidsma@utwente.nlEike DehlingHuman Media InteractionPO Box 2177500AE EnschedeNetherlandsHerwin van WelbergenHuman Media InteractionPO Box 2177500AE EnschedeNetherlandsh.vanwelbergen@utwente.nlJob ZwiersHuman Media InteractionPO Box 2177500AE EnschedeNetherlandsj.zwiers@utwente.nlAnton NijholtHuman Media InteractionPO Box 2177500AE EnschedeNetherlandsa.nijholt@utwente.nlABSTRACTThis paper describes experiments with a virtual fitness trainercapable of mutually coordinated interaction. The virtualhuman co-exercises along with the user, leading as well asfollowing in tempo, to motivate the user and to influencethe speed with which the user performs the exercises. Ina series of three experiments (20 participants in total) weattempted to influence the users’ performance by manipu-lating the (timing of the) exercise behavior of the virtualtrainer. The results show that it is possible to do this im-plicitly, using only micro adjustments to its bodily behavior.As such, the system is a first step in the direction of mutuallycoordinated bodily interaction for virtual humans.Categories and Subject DescriptorsH5.m. [Information interfaces and presentation (e.g.,HCI)]: MiscellaneousKeywordsVirtual Human, Embodied Conversational Agent, Continu-ous Interaction, Virtual Trainer, Coordinated Interaction1. INTRODUCTIONThis paper concerns our experiments with a virtual humanthat is capable of coordinated interaction. The virtual hu-man, a fitness trainer, supervises a user who is doing fitnessexercises. The virtual human co-exercises along with theuser, leading as well as following in tempo, to motivate theuser and to influence the speed with which the user performsthe exercises. This paper starts with a discussion of back-ground, motivation and related work, and a short descrip-tion the architecture of the system. After that, the paperdescribes a series of three experiments (20 participants intotal) in which we attempted to influence the users’ perfor-mance by manipulating the exercise behavior of the virtualtrainer.2. BACKGROUND AND MOTIVATIONInteraction between humans, be it talking, negotiating, danc-ing, rowing a boat or reading poetry to someone, is a jointactivity [3]. A major requirement for successfully engagingin joint activities is effective coordination [7]. This coordi-nation may be, e.g., on the semantic level, on the level of thecontent and behavior selection in the interaction, or even onlow level aspects such as the temporal dynamics of the inter-action (see, e.g., [8] and [15] for a discussion of the variouslevels on which this coordination takes place).Until recently, coordination between humans and comput-ers was a far more simple matter than the complex socialprocesses governing coordination between humans. Com-puter applications were built for information transfer andprocessing. Information goes in; processing occurs; informa-tion comes out: that was basically the definition of interac-tion with a computer. Nowadays, through realization of thevision of Ambient Intelligence, our daily environment is in-strumented with sensors, interpreters, displays, and compu-tational devices. The environment may observe and supportus, but may also pro-actively join in the interactions occur-ring in it, for example through virtual humans and socialrobots. In such an environment, the boundaries betweenhuman-human interaction and human-machine interactionblur. We get interaction between multiple social actors,some of which may be humans and some of which may becomputers.In this context, there are two ways in which coordinatedinteraction becomes relevant to human computer interac-tion. Firstly, ambient intelligence environments are built todetect and interpret as much as possible about the activi-ties of their inhabitants. Interaction with a computer movesfrom mouse-and-keyboard interaction towards whole bodyinteraction, in which all behavior from humans is potentialinput to the system [5]. In these environments, mutual co-ordination between human inhabitants is simply one moreaspect of behavior to be detected and interpreted [12]. Sec-ondly, when the computer itself becomes a social actor [9],it should also be able to employ mechanisms of coordinatedinteraction similar to what humans do. Interaction with acomputer in the form of a social robot or virtual human be-comes a joint activity, too, and requires the same types ofcoordination on the same levels.This paper describes experiments with a virtual fitness trainerthat should behave as much as possible like a human trainer.A good human trainer does not always use words to say“please move faster”, or “move in exactly this way”. Some-times they lead their pupil by moving together with them.By co-performing the exercise they show the proper form ofthe movements. By slowly speeding up (or slowing down)in such a way as to lead the pupil along, they can makethe pupil move faster (or slower) without any spoken in-structions. Such implicit manipulation of a pupil’s behaviorcan be much more effective, more enjoyable, and less face-threatening, than using verbal comments, criticism, and in-structions. However, such leading-and-following behavior isnot trivial to implement in a virtual human. It requires ap-propriate perception of the user’s behavior and coordinatedgeneration of the virtual human’s behavior [10]. Our goal,reported on in this paper, was to experiment with influenc-ing the behavior of the user implicitly, through implementingnonverbal mechanisms for coordinated bodily interaction.3. RELATED WORKDavis and Bobick combined computer vision technology withrecordings of feedback by a real coach [4]. The focus of theirsystem was on the recognition technology and on the choiceof appropriate feedback utterances. Babu et al. developeda virtual trainer that can demonstrate exercises, describeand show the user’s mistakes, and praise correct executionof exercises [1]. For perception, they used computer visionto track markers in 3D space. Their aim was to give highquality feedback on exercises, using the system as a virtualphysiotherapist. Chua et al. performed experiments to seehow effectively users learn Tai-Chi movements when theycan see one or more virtual Tai-Chi trainers from differ-ent angles [2]. Their trainer did not give feedback on theperformance of the user. IJsselsteijn et al. compared train-ing situations with and without a virtual coach that givesfeedback, varying the immersiveness of the experience, andevaluated motivational factors, showing that a more immer-sive application can enhance user performance [6]. Ruttkayet al. collected a corpus of recordings of people perform-ing fitness exercises, and analysed the corpus for the ap-propriate feedback that should be given for each recording[14]. When giving corrective feedback, these earlier virtualtrainers tended to be focused on verbal explanations and‘demonstrative’ performance of the exercise. The novelty ofour work is that we look at how a virtual trainer can im-plicitly influence the tempo with which a user performs theexercises using only coordinated bodily interaction. In this,we build upon earlier work with a virtual orchestra conduc-tor that guided human musicians through the performanceof a musical piece [11], the main difference being that in thecurrent work we performed structured experiments to inves-tigate the relationship between the users’ performance andthe bodily behavior of the trainer.Figure 1: One of the exercise animations used in thevirtual trainer4. ARCHITECTUREOur virtual trainer was implemented in Elckerlyc, a BMLcompliant behavior realizer for generating multimodal ver-bal and nonverbal behavior for virtual humans [16]. Werecorded animations for a few (very basic) fitness exercises,such as jumping jacks, squats and side steps, that the trainershould perform together with the user. The performance ofthe user was measured in two ways: using an HxM ZephyrHeartrate MonitorTMto measure the intensity with whichthe user was performing the exercise, and a Nintendo WiiRemoteTMto measure the rhythm (tempo) of the perfor-mance. The latter worked by taking the average rhythmof the most recent peaks in the acceleration of the Wii Re-mote. This method served well because the exercises haverhythmic phases (jump, land, extend arms, etc) that pro-duce clearly detectable peaks.By default, the trainer and the user perform the exercise atthe same tempo. By monitoring the heart rate the systemdetermines whether the user is working hard enough. If theheart rate is too low, or too high, the trainer attempts tolead the user to a new tempo by modifying his own perfor-mance. To achieve the required modifications, we use El-ckerlyc’s flexible plan representation that allows on-the-flymicro adjustments to the content and timing of the virtualhuman’s behavior [13].5. EVALUATIONWe evaluated the system in several user experiments aimedto find out whether we could influence the speed of the user,and whether this could be done implicitly, i.e., without writ-ten or spoken instructions. In this section, we summarize thesetup and the results.We experimented with three different scenarios. In the firstscenario, the users were told they had to follow the tempo ofthe trainer, and the trainer tried to increase the tempo. Dur-ing the second exercise, the trainer increased or decreasedthe tempo of its performance to attempt to keep the heartrate of the user inside a target range. During the third sce-nario, the trainer again tried to increase the tempo, but theparticipants in this scenario were not told that they hadto follow the tempo of the trainer, or that it would changeduring the exercise.Twenty subjects participated in the study, about half ofthem male and half of them female. Most did not haveexperience with fitness exercises specifically, but did haveexperience with other sports or with dancing. During theexercise, we recorded the user’s heart rate, the tempo withFigure 2: The trainer increases the tempo; the userfollows the tempo, as they were told.Figure 3: Increasing the tempo every 5 seconds by5% did not give the user enough time to settle com-fortably in the new tempo, and caused wild fluctu-ations in the user’s tempo.which the user performed the exercise, and the tempo of thetrainer. In the third scenario, we also made video recordingsof some of the sessions.Figure 2 shows the tempo changes of one user for the firstscenario, in which the trainer attempted to increase thetempo. After a short warm up period, the trainer raisedthe tempo by 10% every 10 seconds. After an initial phasein which the user needs to find their base rhythm, they areclearly able to follow the trainer’s tempo.1 As Figure 3shows, raising the tempo by 5% every 5 seconds had a dras-tically different effect.Figure 4 shows results for a participant in the second sce-nario, in which the trainer attempted to keep the heart rateof the user inside a target range. After the short warm upperiod, the trainer would increase or decrease its own tempoby 10% every 10 seconds, if the heart rate of the user war-ranted it. This experiment achieved the hoped-for resultsfor only two of the six participants who participated in thissession; Figure 4 shows one of these two. For the other fourparticipants, the trainer was not even able to get the heartrate of the user inside the target range in the first place.The third scenario, tested with subjects who did not par-ticipate in the other two scenarios, was aimed at findingout whether the influence of the trainer on the users’ tempowould also be present when the users were not told before-hand that the tempo of the trainer would change, or thatthey were to follow that tempo. The exercise in this scenariostarted with the trainer saying “jump” for 8 counts. This(implicitly) gave the user a rhythm, linked to the motion.After 8 repetitions, the trainer stopped saying “jump”, butcontinued the motions in the same tempo for another 8 rep-1Visual inspection of the video recordings made during theexperiment showed that, although some of the “error peaks”in the graph were caused by the user temporarily losing thetempo, some of them were in fact caused by the tempo mea-surements erroneously measuring double the actual tempo.Figure 4: For two of the six participants in the sec-ond experiment, the trainer was able to keep theuser’s heart rate inside the target range. The graphsshow trainer’s tempo, measured user’s tempo, anduser’s heartrate, for one of these two users. Goalheartrate was 100. Heartrate rises and falls out ofrange, but this is corrected implicitly by speed ad-justments to the trainers movements.Figure 5: In the third scenario, this user followsthe trainer’s tempo very well, even without receiv-ing instructions either beforehand (from the exper-imenter) or during the exercise (from the virtualtrainer).etitions. Subsequently, the trainer started to (attempt to)influence the user’s tempo. Figure 5 shows how the trainerwas able to speed up the performance of the user implicitly,without giving verbal instructions, solely by making microadjustments to its own bodily behavior.In summary, we can say that we have achieved our goal ofimplementing real coordinated interaction between a virtualhuman and a user. Building upon these results we can startto create more human like virtual trainers that use moresubtle types of feedback.6. DISCUSSION AND CONCLUSIONMutually coordinated interaction between humans and vir-tual humans is a challenging topic that requires new devel-opments in user perception as well as behavior generationfor virtual humans. At the same time, such coordinated in-teraction is an important prerequisite for building virtualhumans that are really capable of entering into joint activ-ities with human users. The fitness exercises and sportstraining scenarios described in this paper are not the onlyapplication in which such interaction plays an importantrole. People sit down together, after shaking hands, whenthey have a meeting. When the meeting is over, likely asnot they will indicate they are about to get up and go away,and then proceed to do so in a coordinated fashion. When ina conversation, the breathing of two persons may synchro-nize; their body sway may display similar rhythms, and inmany other ways they will exhibit embodied entrainment.We discussed in this paper our experiments with a virtualtrainer that was able to influence the users’ performance ofa fitness exercise implicitly, using only micro adjustments toits bodily behavior. As such, this system is a first step inthe direction of mutually coordinated bodily interaction forvirtual humans.7. ACKNOWLEDGMENTSThis research has been supported by the GATE project,funded by the Dutch Organization for Scientific Research(NWO). The authors would like to thank the participantsin the experiments for their participation as well as for theirhelpful comments afterwards.References[1] S. Babu, C. Zanbaka, J. Jackson, T. Chung, B. Lok,M. C. Shin, and L. F. Hodges. Virtual human physio-therapist framework for personalized training and reha-bilitation. In Proc. Graphics Interface 2005, May 2005.[2] P. T. Chua, R. Crivella, B. Daly, N. Hu, R. Schaaf,D. Ventura, T. Camill, J. Hodgins, and R. Pausch.Training for physical tasks in virtual environments: Taichi. In VR ’03: Proceedings of the IEEE Virtual Reality,page 87. IEEE Computer Society, 2003.[3] H. H. Clark. Using Language. Cambridge UniversityPress, 1996.[4] J. W. Davis and A. F. Bobick. Virtual PAT: A virtualpersonal aerobics trainer. In Proceedings of Workshopon Perceptual User Interfaces (PUI’98), pages 13–18,New York, 1998. IEEE.[5] D. England, editor. Whole Body Interaction. Human-Computer Interaction Series. Springer-Verlag, London,May 2011.[6] W. IJsselsteijn, Y. de Kort, J. H. D. M. Westerink,M. de Jager, and R. Bonants. Fun and sports: Enhanc-ing the home fitness experience. In M. Rauterberg, ed-itor, Third International Conference on EntertainmentComputing, volume 3166 of Lecture Notes in ComputerScience, pages 46–56. Springer, 2004.[7] G. Klein, P. J. Feltovich, J. M. Bradshaw, and D. D.Woods. Common Ground and Coordination in JointActivity, chapter 6, pages 139–178. Wiley Series in Sys-tems Engineering and Management. John Wiley andsons, Hoboken, New Jersey, 2004.[8] S. Kopp. Social resonance and embodied coordinationin face-to-face conversation with artificial interlocutors.Speech Communication, 52(6):587–597, 2010.[9] C. Nass, J. Steuer, and E. R. Tauber. Computers aresocial actors. In Proceedings of the SIGCHI conferenceon Human factors in computing systems: celebratinginterdependence, CHI ’94, pages 72–78, New York, NY,USA, 1994. ACM.[10] A. Nijholt, D. Reidsma, H. van Welbergen, H. J. A.op den Akker, and Z. M. Ruttkay. Mutually coordinatedanticipatory multimodal interaction. In Nonverbal Fea-tures of Human-Human and Human-Machine Interac-tion, volume 5042 of LNCS, pages 70–89, Berlin, 2008.Springer Verlag.[11] D. Reidsma, A. Nijholt, and P. Bos. Temporal interac-tion between an artificial orchestra conductor and hu-man musicians. Computers in Entertainment, 6(4):1–22, 2008.[12] D. Reidsma, A. Nijholt, W. Tschacher, and F. Ram-seyer. Measuring multimodal synchrony for human-computer interaction. In D. England, J. Sheridan, andB. Crane, editors, Proceedings of the CHI Workshop onWhole Body Interaction, 2010.[13] D. Reidsma, H. v. Welbergen, and J. Zwiers. Multi-modal plan representation for adaptable BML schedul-ing. In Proc. of IVA 2011, 2011.[14] Z. M. Ruttkay and H. van Welbergen. Elbows higher!Performing, observing and correcting exercises by avirtual trainer. In H. Prendinger, J. C. Lester, andM. Ishizuka, editors, Proceedings of the 8th Interna-tional Conference on Intelligent Virtual Agents, volume5208 of Lecture Notes in Artificial Intelligence, pages409–416, Berlin, Heidelberg, 2008. Springer-Verlag.[15] K. R. Thórisson. Natural turn-taking needs no man-ual: Computational theory and model, from perceptionto action. In B. Granström, D. House, and I. Karls-son, editors, Multimodality in Language and SpeechSystems, pages 173–207. Kluwer Academic Publishers,Dordrecht, The Netherlands, 2002.[16] H. van Welbergen, D. Reidsma, Z. M. Ruttkay, andJ. Zwiers. Elckerlyc: A BML realizer for continuous,multimodal interaction with a virtual human. Journalon Multimodal User Interfaces, 3(4):271–284, 2010.

Leading and following with a virtual trainer

Computers are social actors.

Elbows higher! Performing, observing and correcting exercises by a virtual trainer. In

Elckerlyc: A BML realizer for continuous, multimodal interaction with a virtual human.

Fun and sports: Enhancing the home  experience.

Measuring multimodal synchrony for humancomputer interaction. In

Multimodal plan representation for adaptable BML scheduling.

Mutually coordinated anticipatory multimodal interaction.

Social resonance and embodied coordination in face-to-face conversation with arti interlocutors.

Temporal interaction between an arti orchestra conductor and human musicians.

Th orisson. Natural turn-taking needs no manual: Computational theory and model, from perception to action.

Training for physical tasks in virtual environments: Tai chi.

Using Language.

Virtual human physiotherapist framework for personalized training and rehabilitation.

Virtual PAT: A virtual personal aerobics trainer.

Whole Body Interaction. HumanComputer Interaction Series.

https://ris.utwente.nl/ws/files/5356166/WBI_HMI-P3.pdf

Leading and following with a virtual trainer

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information