In recent years, thanks to the development of 3DCG animation editing tools (e.g. MikuMikuDance), a lot of 3D character dance animation movies are created by amateur users. However it is very difficult to create choreography from scratch without any technical knowledge. Shiratori et al. [2006] produced the dance automatic generation system considering rhythm and intensity of dance motions. However each segment is selected randomly from database, so the generated dance motion has no linguistic or emotional meanings. Takano et al. [2010] produced a human motion generation system considering motion labels. However they use simple motion labels like “running” or “jump”, so they cannot generate motions that express emotions. In reality, professional dancers make choreography based on music features or lyrics in music, and express emotion or how they feel in music. In our work, we aim at generating more emotional dance motion easily. Therefore, we use linguistic information in lyrics, and generate dance motion.



In this paper, we propose the system to generate the sign dance motion from continuous sign language motion based on lyrics of music. This system could help the deaf to listen to music as visualized music application

Asahina, Wakana

Iwamoto, Naoya

Morishima, Shigeo

Shum, Hubert P. H.

Northumbria Research Link

Northumbria Research Link
Citation:  Asahina,  Wakana,  Iwamoto,  Naoya,  Shum,  Hubert  P.  H.  and  Morishima,  Shigeo  (2016) 
Automatic Dance Generation System Considering Sign Language Information. In: SIGGRAPH 2016 - 
43rd International Conference and Exhibition on Computer Graphics and Interactive Techniques, 24th 
- 28th July 2016, Anaheim, California. 
URL: http://s2016.siggraph.org/programs <http://s2016.siggraph.org/programs>
This version was downloaded from Northumbria Research Link: http://nrl.northumbria.ac.uk/28248/
Northumbria University has developed Northumbria Research Link (NRL) to enable users to access 
the University’s research output. Copyright © and moral rights for items on NRL are retained by the 
individual author(s) and/or other copyright owners.  Single copies of full items can be reproduced, 
displayed or performed, and given to third parties in any format or medium for personal research or 
study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, 
title and full bibliographic details are given, as well as a hyperlink and/or URL to the original metadata 
page. The content must not be changed in any way. Full items must not be sold commercially in any  
format or medium without formal permission of the copyright holder.  The full policy is available online: 
http://nrl.northumbria.ac.uk/pol  i cies.html  
This  document  may differ  from the  final,  published version of  the research  and has been made 
available online in accordance with publisher policies. To read and/or cite from the published version 
of the research, please visit the publisher’s website (a subscription may be required.)
                        
Automatic Dance Generation System 
Considering Sign Language Information
Wakana ASAHINA1† Naoya IWAMOTO1, ‡ Hubert. P. H. Shum2, § Shigeo MORISHIMA3,
Waseda University1 Northumbria University2 Waseda Research Institute for Science and Engineering3
1. Introduction
In recent years, thanks to the development of 3DCG animation 
editing tools (e.g. MikuMikuDance), a lot of 3D character dance 
animation movies are created by amateur users. However it is 
very difficult to create choreography from scratch without any 
technical knowledge. Shiratori et al. [2006] produced the dance 
automatic generation system considering rhythm and intensity of 
dance motions. However each segment is selected randomly from 
database, so the generated dance motion has no linguistic or emo-
tional meanings. Takano et al. [2010] produced a human motion 
generation system considering motion labels. However they use 
simple motion labels like “running” or “jump”, so they cannot 
generate motions that express emotions. In reality, professional 
dancers make choreography based on music features or lyrics in 
music, and express emotion or how they feel in music. In our
work, we aim at generating more emotional dance motion easily.
Therefore, we use linguistic information in lyrics, and generate 
dance motion. 
Concepts: • Computing methodologies~Motion processing
Keywords: sign dance, motion analysis, linguistic information
†e-mail: 2236-wakana@fuji.waseda.jp
‡e-mail: iwamoto@toki.waseda.jp
§e-mail: hubert.shum@northumbria.ac.uk (corresponding author)
e-mail: shigeo@waseda.jp
Permission to make digital or hard copies of part or all of this work for 
personal or classroom use is granted without fee provided that copies are 
not made or distributed for profit or commercial advantage and that copies 
bear this notice and the full citation on the first page. Copyrights for third-
party components of this work must be honored. For all other uses, contact 
the Owner/Author.
Copyright is held by the owner/author(s).
SIGGRAPH '16 Posters, July 24-28, 2016, Anaheim, CA,
ACM 978-1-4503-4371-8/16/07.
http://dx.doi.org/10.1145/2945078.2945101
In this paper, we propose the system to generate the sign dance
motion from continuous sign language motion based on lyrics of 
music. This system could help the deaf to listen to music as visu-
alized music application.
2. Sign dance motion synthesis
Fig.1 shows the outline of our system. In our system, the input is a 
sign language motion and by using our Sign Dance Model, we can 
generate a new sign dance motion as output automatically. First, 
we divide the input sign language motion into 4 count segments 
each ( ). Then, based on Sign Dance Model, we estimate the 
sign dance motion features ( ). Next, we choose the candidate 
sign dance motion segments from the sign dance database, which 
is the most similar with estimated dance motion features ( ). 
Finally, we synthesize hand joint motion from input sign language
motion and other joint motion from estimated sign dance candi-
date motion ( ).
Construction of learning motion data:
To build a Sign Dance Model, lots of learning motion data is re-
quired. Therefore, by using marker-based motion capture system,
we captured the movement of a professional sign dancer. Then,
we got sign language motion and sign dance motion data of 10
pieces of music each. For those ten pieces of music, we used j-pop 
music with lyrics which is around 2 minutes each. Then, we cut 
those each 20 motions into 4 count segments. Finally, the data-
base includes 360 segments each (only sign language: 360 seg-
ments, sign dance segments: 360 ) .
Build the Sign Dance Model:
We use multiple regression analysis to estimate a sign dance mo-
tion. In particular, we use sign language motion as an explanation 
variable, and use sign dance motion as an objective variable. Here,
we use the quaternion of 6 joints (24 dimensional vectors) that is
related to hand, arm and roots as motion features. We decided to 
use these joints based on the assumption that there is a correlation 
between the orientation of the pelvis and the hand joints of sign 
language motion and those of sign dance motion.
Figure 1 :The overview of our work
Choice of the candidate segments
Here, we describe how to choose the appropriate candidate seg-
ments from learning sign dance motion data.
First, we defined the similarity variable  as:= ( , , ) …(1)
where , denotes estimated sign dance motion features, and, denotes sign dance motion features of motion segments in 
sign dance database (segment: k, frame: i). Next, we narrow the 
candidates down to top 5 candidates by thresholding the beats per 
minute (BPM) of each dance segment according to the music 
BPM. Moreover, to connect each segment naturally, we use a
posture similarity variable as:= ( ) ( ) …(2)
where and denote frame number of segment k and seg-
ment k-1. denotes direction vector of each joint rink. Finally, 
we select the first candidate segments that minimize the degree of= + 1/ from 5 candidates. Here, we decided to 
use as 1.5, and as 1.0.
3. Results and Conclusions
To evaluate our system, we compared our results dance motion 
with the input sign language motion and the original sign dance 
motion. First, we found that in our results, dance motion segments 
that emphasize the input sign language movement tend to be se-
lected (Fig.2). In addition, in our results, a lot of motion segments 
that are similar with the original sign dance motions were selected
(Fig.3). This means our sign dance estimated model works well.
However, there are some foot gliding motion and physically in-
correct motion, which happens when the character turns around.
To solve this problem, we plan to analyze the stepping pattern and 
apply inverse kinematics on the supporting feet in the future.
This paper presented an automatic sign dance generation system. 
By this system, we can generate a dance motion that includes 
linguistic information in lyrics. The generated sign dance motion 
can express the emotion in music and dance more richly by using 
sign language and dance motion that emphasize sign language 
motion. As a future direction, we want to consider using music 
features like beat or timbre of a song. In addition, by conducting a
user study, we plan to evaluate the precision of our system. More-
over, by building the large sign language database that is corre-
sponding to each word, we wish the system to take lyrics only as 
the input, instead of the sign language motion. We believe that our 
work can help people with hearing difficulties to enjoy music.
ACKNOWLEDGMENTS:
This work was supported by CREST, JST.
This work was also supported by EPSRC (Ref: EP/M002632/1).
References
TAKAAKI, S., ATSUSHI, N., AND KATSUSHI, I. 2006. Dance-to-Music 
Character Animation. In Proceedings of EUROGRAPHICS2006, Vol. 
25, No. 3, pp.449-458, 2006
TAKANO, W., KATSU, Y., AND YOSHIHIRO, N. 2010. Retrieval and 
Generation of Human Motions Based on Associative Model between 
Motion Symbols and Motion Labels. In Proceedings of Journal of the 
Robotics Society of Japan2010, Vol. 28, No. 6, pp.723-734, 2010.
Input Sign language Output Sign dance
Example1
Input Sign language Output Sign dance
Example2
Input Sign language Output Sign dance
Example3
Figure 2 : Comparison with input sign motion
We got the output dance which emphasizes input sign motion.
Original Sign dance Output Sign dance
Example1
Original Sign dance Output Sign dance
Example2
Original Sign dance Output Sign dance
Example3
Figure 3 : Comparison with original dance
We got the similar output dance motion with original dance motion.


Automatic Dance Generation System Considering Sign Language Information

Wakana Asahina

Naoya Iwamoto

Hubert. P. H. Shum

Shigeo Morishima

Crossref

Automatic dance generation system considering sign language information

Citation: Asahina, Wakana, Iwamoto, Naoya, Shum, Hubert P. H. and Morishima, Shigeo (2016)  Automatic  Dance Generation System Considering Sign Language Information.  In: SIGGRAPH 2016 - 43rd International Conference and Exhibition on Computer Graphics and Interactive Techniques, 24th - 28th July 2016, Anaheim, California. URL: http://s2016.siggraph.org/programs <http://s2016.siggraph.org/programs>This  version  was  downloaded  from  Northumbria  Research  Link: http://nrl.northumbria.ac.uk/28248/Northumbria University has developed Northumbria Research Link (NRL) to enable users to access the University’s research output. Copyright © and moral rights for items on NRL are retained by the individual author(s) and/or other copyright owners.  Single copies of full items can be reproduced,  displayed or  performed,  and given to  third parties in  any format  or medium for personal research or study, educational, or not-for-profit purposes without prior permission or charge, provided the authors, title and full bibliographic details are given, as well  as a hyperlink and/or URL to the original metadata page. The content must  not  be changed in any way. Full items must not be sold commercially in any format or medium without  formal  permission  of  the  copyright  holder.   The  full  policy  is  available  online: http://nrl.northumbria.ac.uk/policies.htmlThis document may differ from the final, published version of the research and has been made available online in accordance with publisher policies. To read and/or cite from the published version of the research, please visit the publisher’s website (a subscription may be required.)Automatic Dance Generation System Considering Sign Language InformationWakana ASAHINA1†? ? Naoya IWAMOTO1, ‡? ? Hubert. P. H. Shum2, §? ? Shigeo MORISHIMA3, ?Waseda University1 ? Northumbria University2 ? Waseda Research Institute for Science and Engineering31. IntroductionIn recent years, thanks to the development of 3DCG animation editing tools (e.g. MikuMikuDance), a lot of 3D character dance animation movies are created by amateur users. However it is very difficult to create choreography from scratch without any technical knowledge. Shiratori et al. [2006] produced the dance automatic generation system considering rhythm and intensity of dance motions. However each segment is selected randomly from database, so the generated dance motion has no linguistic or emo-tional meanings. Takano et al. [2010] produced a human motion generation system considering motion labels. However they use simple motion labels like “running” or “jump”, so they cannot generate motions that express emotions. In reality, professional dancers make choreography based on music features or lyrics in music, and express emotion or how they feel in music. In ourwork, we aim at generating more emotional dance motion easily.Therefore, we use linguistic information in lyrics, and generate dance motion. Concepts: • Computing methodologies~Motion processingKeywords: sign dance, motion analysis, linguistic information†e-mail: 2236-wakana@fuji.waseda.jp‡e-mail: iwamoto@toki.waseda.jp§e-mail: hubert.shum@northumbria.ac.uk (corresponding author)?e-mail: shigeo@waseda.jpPermission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.Copyright is held by the owner/author(s).SIGGRAPH '16 Posters, July 24-28, 2016, Anaheim, CA,ACM 978-1-4503-4371-8/16/07.http://dx.doi.org/10.1145/2945078.2945101In this paper, we propose the system to generate the sign dancemotion from continuous sign language motion based on lyrics of music. This system could help the deaf to listen to music as visu-alized music application.2. Sign dance motion synthesisFig.1 shows the outline of our system. In our system, the input is a sign language motion and by using our Sign Dance Model, we can generate a new sign dance motion as output automatically. First, we divide the input sign language motion into 4 count segments each (?). Then, based on Sign Dance Model, we estimate the sign dance motion features (?). Next, we choose the candidate sign dance motion segments from the sign dance database, which is the most similar with estimated dance motion features (?). Finally, we synthesize hand joint motion from input sign languagemotion and other joint motion from estimated sign dance candi-date motion (?).Construction of learning motion data:To build a Sign Dance Model, lots of learning motion data is re-quired. Therefore, by using marker-based motion capture system,we captured the movement of a professional sign dancer. Then,we got sign language motion and sign dance motion data of 10pieces of music each. For those ten pieces of music, we used j-pop music with lyrics which is around 2 minutes each. Then, we cut those each 20 motions into 4 count segments. Finally, the data-base includes 360 segments each (only sign language: 360 seg-ments, sign dance segments: 360 ) .?Build the Sign Dance Model:We use multiple regression analysis to estimate a sign dance mo-tion. In particular, we use sign language motion as an explanation variable, and use sign dance motion as an objective variable. Here,we use the quaternion of 6 joints (24 dimensional vectors) that isrelated to hand, arm and roots as motion features. We decided to use these joints based on the assumption that there is a correlation between the orientation of the pelvis and the hand joints of sign language motion and those of sign dance motion.Figure 1 :The overview of our workChoice of the candidate segmentsHere, we describe how to choose the appropriate candidate seg-ments from learning sign dance motion data.First, we defined the similarity variable ߙ as:ߙ = σ σ ඥሺܨ݁௜,௞ െ ܨ݀௜,௞ሻଶே೑௜ேೞ௞ …(1)where ܨ݁௜,௞ denotes estimated sign dance motion features, andܨ݀௜,௞ denotes sign dance motion features of motion segments in sign dance database (segment: k, frame: i). Next, we narrow the candidates down to top 5 candidates by thresholding the beats per minute (BPM) of each dance segment according to the music BPM. Moreover, to connect each segment naturally, we use aposture similarity variable ߚ as:ߚ = σ ݒ௟ሺ݅௞ሻ ή ݒ௟ሺ݆௞ିଵሻே௟ …(2)where ݅௞ and ݆௞ିଵ denote frame number of segment k and seg-ment k-1. ݒ௟ denotes direction vector of each joint rink. Finally, we select the first candidate segments that minimize the degree ofɀ = ݓଵ כ ߙ + ݓଶ כ 1/ߚ from 5 candidates. Here, we decided to use ݓଵ as 1.5, and ݓଶ as 1.0.3. Results and ConclusionsTo evaluate our system, we compared our results dance motion with the input sign language motion and the original sign dance motion. First, we found that in our results, dance motion segments that emphasize the input sign language movement tend to be se-lected (Fig.2). In addition, in our results, a lot of motion segments that are similar with the original sign dance motions were selected(Fig.3). This means our sign dance estimated model works well.However, there are some foot gliding motion and physically in-correct motion, which happens when the character turns around.To solve this problem, we plan to analyze the stepping pattern and apply inverse kinematics on the supporting feet in the future.This paper presented an automatic sign dance generation system. By this system, we can generate a dance motion that includes linguistic information in lyrics. The generated sign dance motion can express the emotion in music and dance more richly by using sign language and dance motion that emphasize sign language motion. As a future direction, we want to consider using music features like beat or timbre of a song. In addition, by conducting auser study, we plan to evaluate the precision of our system. More-over, by building the large sign language database that is corre-sponding to each word, we wish the system to take lyrics only as the input, instead of the sign language motion. We believe that our work can help people with hearing difficulties to enjoy music.ACKNOWLEDGMENTS:This work was supported by CREST, JST.This work was also supported by EPSRC (Ref: EP/M002632/1).ReferencesTAKAAKI, S., ATSUSHI, N., AND KATSUSHI, I. 2006. Dance-to-Music Character Animation. In Proceedings of EUROGRAPHICS2006, Vol. 25, No. 3, pp.449-458, 2006?TAKANO, W., KATSU, Y., AND YOSHIHIRO, N. 2010. Retrieval and Generation of Human Motions Based on Associative Model between Motion Symbols and Motion Labels. In Proceedings of Journal of the Robotics Society of Japan2010, Vol. 28, No. 6, pp.723-734, 2010.Input Sign language Output Sign danceExample1Input Sign language Output Sign danceExample2Input Sign language Output Sign danceExample3Figure 2 : Comparison with input sign motionWe got the output dance which emphasizes input sign motion.Original Sign dance Output Sign danceExample1Original Sign dance Output Sign danceExample2Original Sign dance Output Sign danceExample3Figure 3 : Comparison with original danceWe got the similar output dance motion with original dance motion.

http://nrl.northumbria.ac.uk/28248/1/siggraph2016%5B1%5D.pdf

Automatic Dance Generation System Considering Sign Language Information

Abstract

Similar works

Full text

Available Versions

Northumbria Research Link

Northumbria Research Link

Crossref

Northumbria Research Link