Maeda, Yoichiro

Nakamura, Takayuki

Nonoshita, Hiroaki

Takahashi, Yasutake

University of Fukui Repository

Behavioral Development of Ball Kicking Motion of a Two-wheeled
Inverted Pendulum Mobile Robot
Yasutake Takahashi, Hiroaki Nonoshita, Takayuki Nakamura, and Yoichiro Maeda
Abstract—In this paper, we introduce a method for gen-
erating a dynamic motion such that a two-wheeled inverted
pendulum robot kicks a ball far away utilizing its own body
dynamics while it keeps standing. Such a dynamic motion can
be acquired through trial and error based on a reinforcement
learning scheme. We utilize a simple policy gradient method
to acquire a kicking motion which is designed by defining the
desired parameters such as body angle, wheel angular velocity
and so on. To show the validity of our approach, we perform
computer simulation experiments of behavior acquisition for the
two-wheeled inverted pendulum robot. Based our approach, we
succeeded in acquiring the kicking motion of the two-wheeled
inverted pendulum robot. A very interesting finding is: each
of the acquired motions deviates from the desired trajectory,
which is given by the human designer while keeping evaluation
value of the acquired motion high.
I. INTRODUCTION
Many kinds of mobile robots working in a human living
environment have been developed. Most of the existing mo-
bile robots have three-wheeled or four-wheeled locomotion
mechanism and low center of gravity for achieving static
stability. Furthermore, such mobile robots tend to have a
larger support surface and lower height than human’s ones.
As a result, the motion of such mobile robot becomes slow
and any motion with a rapid acceleration and deceleration is
impossible for such mobile robot. It is considered that such
three-wheeled or four-wheeled mobile robot is not suitable
as human partners robot.
Recently, many kinds of humanoid robots have been de-
veloped so far as human partners under the human-symbiotic
environment, because it is reasonable for the mobile robot to
have a similar shape and size of a human. However, a two-
legged humanoid robot[1] still has a lot of difficulties. For
example, it has problems of robustness against unpredictable
disturbances, heavy weight, and dynamic motions. A pneu-
matic actuated biped robot[2][3] has a lightweight body and
a potential to generate dynamic motions but still has big
problems in terms of the stability for usage under the human
living environment.
On the other hand, a two-wheeled inverted pendulum
robot[4][5] has many advantages over the statically stable
wheeled robots and any other biped robots. It requires a
smaller amount of space to stand and stay upright than the
Yasutake Takahashi, Hiroaki Nonoshita, and Yoichiro Maeda are
with the Department of Human and Artificial Intelligent Systems,
Graduate School of Engineering, University of Fukui, 3-9-1, Bunkyo,
Fukui, Fukui, 910-8507, Japan (phone: +81 776 27 8540; email:
{yasutake,hnonosita,maeda}@ir.his.u-fukui.ac.jp).
Takayuki Nakamura is with Faculty of Systems Engineering, Wakayama
University, Sakaetani 930, Wakayama 640-8510, Japan (phone: +81 73 457
8071; email: ntakayuk@sys.wakayama-u.ac.jp).
other wheeled robots and smaller number of actuators than
the conventional biped robots. Furthermore, it can easily
generate more dynamic motions only by its body balancing
control. The two-wheeled inverted pendulum robot has a
possibility to be standard platform as a human partner robot.
Most of researches regarding to the two-wheeled inverted
pendulum robot focuses on the control theory for achieving
stable navigation. There are few researches to realize more
dynamic motions such as throwing, jumping, pushing, kick-
ing, and so on.
For example, the two-wheeled inverted pendulum robot
needs to break static stability while swinging the body back
and forth, when it kicks a ball as fast as possible. This
means that the robot has to utilize its own body dynamics as
much as possible in order to realize such dynamic motions
in the same way that a human does. Of course, it should
achieve global stability so that it avoids falling down during
the dynamic motion. The dynamic motion requires not only
the conventional control theory to stabilize body balance but
also a good trajectory of body posture and traveling. It is not
trivial to find such good trajectory to generate the dynamic
motion by hand.
In this paper, we introduce a method for generating a
dynamic motion such that a two-wheeled inverted pendulum
robot kicks a ball far away utilizing its own body dynamics
while it keeps standing. Our method assumes that a kicking
motion consists of two primitive motions. In this work, we
utilize a simple policy gradient method[6] to acquire the two
primitive motions each of which is designed by defining
the desired parameters such as body angle, wheel angular
velocity and so on. The kicking motions are evaluated with
the ball velocity after the robot kicks and its stableness. Our
policy gradient method just updates parameters of the kicking
motions so that the evaluations become as higher as possible.
To show the validity of our approach, we perform com-
puter simulation experiments of behavior acquisition for the
two-wheeled inverted pendulum robot. As a result of these
learning process, we confirmed that a stepwise learning
curve with two learning phases emerges. It means that the
robot changes a learning strategy between the phases during
the learning process. Based our approach, we succeeded in
acquiring the kicking motion of the two-wheeled inverted
pendulum robot. A very interesting finding is: each of the
acquired motions deviates from the desired trajectory which
is given by the human designer while keeping evaluation
value of the acquired motion high. Based on the experiment
results, we discuss the learning profile in detail.
WCCI 2010 IEEE World Congress on Computational Intelligence 
July, 18-23, 2010 - CCIB, Barcelona, Spain FUZZ-IEEE
978-1-4244-8126-2/10/$26.00 c©2010 IEEE 830
II. TWO-WHEELED INVERTED PENDULUM ROBOT
Fig. 1. Two-wheeled inverted pendulum robot with a ball
Fig. 1 shows a two-wheeled inverted pendulum robot
which we designed and built and a ball. The robot body
size is 20cm length, 30.5cm width, and 35cm height. The
weight is 8kg including the body, wheels, batteries, motors,
gears, encoders, an accelero-gyrometer unit, and control unit.
The wheel radius is 8.5 cm and its weight is 500g each.
The ball kicked by the robot is a soccer ball, roughly 22cm
in diameter, and about 450g in weight. In order to perform
comprehensive experiments for learning a kicking motion
with the robot, we develop a computer simulation of the
robot and the ball using the ODE(open dynamics engine)
library.
III. DESIGN FOR LEARNING KICKING MOTION
Fig. 2. Model of the two-wheeled inverted pendulum robot
The kicking motion is designed with two control layers, a
low-level posture controller and a posture generator. The pos-
ture controller follows a conventional torque control theory.
Torque for the wheels T is calculated as follows:
T = −k1(θ − θd)− k2θ˙ − k3ϕ˙− k4
∑
(ϕ˙− ϕ˙d), (1)
where θ, θd, ϕ˙, ϕ˙d are body angle, desired body angle,
wheel angular velocity, and desired wheel angular velocity,
respectively. k1, k2, k3, k4 are gains for the body angle, body
angular velocity, wheel angular velocity, and accumulated
error of wheel angular velocity, respectively.
In this work, we design the kicking motion in combination
with two primitive motions (Fig.3). Each primitive motion
defines the desired body angle, desired wheel angular ve-
locity, gain parameters, and a period of time. The posture
Fig. 3. Two primitive motions of Kicking motion
generator sends the control parameters to the posture con-
troller within the period defined by the primitive motion,
one by one. The first primitive motion tries to lean forward
to kick the ball and the second tries to make the robot kick
the ball actually. Before and after the kicking motion, the
robot controls itself to stay upright avoiding falling-down,
autonomously.
IV. POLICY GRADIENT METHOD
In order to generate kicking motion of our robot, our
method utilizes a simple policy gradient method introduced
by Kohl and Stone[6]. The posture generator learns the
control parameters based on the policy gradient method. A
parameter set of the current motion isΘ. The learning system
prepares T similar policies R1, R2, ..., RT by adding small
disturbances εj , 0, or −εj to the current motion parameter
Θj(Θj ∈ Θ):
Θij = Θj + rεj where r ∈ (−1, 0, 1) (2)
It evaluates the policies Ri = {Θij} one by one after the robot
tries to generate the kicking motions based on the policies.
First of all, evaluation averages for the disturbances are
estimated as follows:
• Avg+ε,j is average of evaluation of policies which
parameter Θij is Θj + εj
• Avg0,j is average of evaluation of policies which pa-
rameter Θij is Θj
• Avg−ε,j is average of evaluation of policies which
parameter Θij is Θj − εj
Then, Aj , that is, the gradient of evaluation to the policy
parameters Θj is estimated approximately as follows: Aj
is regarded as 0 if the Avg0,j is greater than Avg+ε,j and
Avg−ε,j . It is regarded as Avg+ε,j −Avg−ε,j , else:
Aj =

0 Avg+0,j > Avg+ε,j
&Avg+0,j > Avg−ε,j
Avg+ε,j −Avg−ε,j else
(3)
Finally, it updates the policy parameters Θ according to
the A as follows:
Θj ← Θj + Aj|Aj | ∗ η, (4)
where η is a certain learning step size. The learning system
repeats this procedure and updates the motion parameters
to reach the local maximum of evaluation. Fig.4 shows a
flowchart of the learning procedure.
831
Fig. 4. Flowchart of learning procedure
V. LEARNING KICKING MOTION
A. Experiment Setup
As mentioned in section III, each primitive motion is
defined by the desired body angle θd, the desired wheel
angular velocity ϕ˙d, the gain parameters k1, k2, k3, and
k4, and a period of time t. The learning parameters are
those primitive motion parameters for each, therefore, 14
parameters in all. We determine the evaluation criterion of
the kicking motion while learning process in consideration
of the following points:
• Traveling distance of the robot body should be small.
• Velocity of the kicked ball should be large.
• The robot should avoid falling down by kicking the ball.
The actual evaluation is defined as follows:
E =

w1
1 + lb
+ w2vb − wf In case the robot fell down
w1
1 + lb
+ w2vb else,
(5)
where lb, vb, w1, w2, and w3 are the traveling distance of
the robot body, the velocity of the kicked ball, weights for
the traveling distance, the ball velocity, and cost of falling
down, respectively. In this work, w2 and wf are fixed to 1000
and 100, respectively. w1 is set to 1, 10, 100, and 1000 for
analysis.
The parameters were initially set as shown in TABLE I.
The learning step size parameters are shown in TABLE II.
TABLE I
INITIAL PARAMETERS OF PRIMITIVE MOTION 1 AND 2
learning parameter primitive motion 1 primitive motion 2
T [sec] 1.0 1.0
θd [rad] -1.0 1.0
φd [rad/sec] 0 0
k1 15.0 15.0
k2 -0.3 -0.3
k3 0.01 0.01
k4 0.0 0.0
The number of deviated policies for estimating gradient of
TABLE II
STEP SIZE OF LEARNING PARAMETERS
learning parameter step size
T [sec] 0.001
θd [rad] 0.01
φd [rad/sec] 0
k1 0.01
k2 0.001
k3 0.0001
k4 0.00001
the evaluation T is set to 50 in the experiments.
B. Learning Curves
Fig.5 shows the transition profile of evaluation, running
distance, and ball velocity while learning kicking motion. All
learning curves with varied values of w1 show similar trends.
The evaluations gradually increase from the beginning of the
learning to around 250 updates of learning parameters. They
increase rapidly from about 250 to 300 updates of parameters
and still increases slowly after the 300th update.
Fig.6 shows the transition profile of posture parameters,
that is, desired body angles and periods of time for primitive
motion 1 and 2. The periods of time are almost fixed and
have constant trends from the beginning to the end of the
learning. The desired body angles changes rapidly from the
beginning to around the 15000 trials, 300 updates of learning
parameters. After that, they increase or degrease very slowly.
Fig.7 shows the transition profile of gain parameters for
primitive motion 1 and 2 during the learning. Gains for
body angular velocity k2 and wheel angular velocity k3
of primitive motion 2 produces interesting learning curves.
They decrease before around 300 updates of parameters and
increase after that. This means, the robot explores posture
parameters of the primitive motions while it relaxes the
feedback gains for angular velocities. Once it finds good pos-
ture parameters, then, it recover the feedbacks again for the
stability. The other curves increase or decrease monotonically
and converge after the 300th update of parameters.
Fig.8 shows trajectories of the body angle while the robot
is kicking the ball before the learning, after the learning with
350 and 1000 updates of parameters.
832
 200
 400
 600
 800
 1000
 1200
 1400
 1600
 1800
 2000
 2200
 0  100  200  300  400  500  600  700  800  900  1000
a
ve
ra
ge
 e
va
lu
at
ed
 v
al
ue
number of learning
 w1 = 1
 w1 = 10
 w1 = 100
 w1 = 1000
(a)Motion evaluation
 0.1
 0.2
 0.3
 0.4
 0.5
 0.6
 0.7
 0.8
 0.9
 1
 0  100  200  300  400  500  600  700  800  900  1000
a
ve
ra
ge
 m
ov
em
en
t d
ist
an
ce
 [m
]
number of learning
 w1 = 1
 w1 = 10
 w1 = 100
 w1 = 1000
(b)Running distance
 0.2
 0.4
 0.6
 0.8
 1
 1.2
 1.4
 1.6
 0  100  200  300  400  500  600  700  800  900  1000
a
ve
ra
ge
 v
al
oc
ity
 [m
/se
c]
number of learning
 w1 = 1
 w1 = 10
 w1 = 100
 w1 = 1000
(c)Ball velocity
Fig. 5. Evaluation profiles
It also shows the desired trajectories of the kicking motion.
Before the learning, the motion is small and the posture
controller follows the desired trajectory. After the learning,
the actual trajectory does not follow the desired trajectory
well but it does not matter because the objective of the
motion is kicking a ball as fast as possible without falling
down. The gains for wheel angular velocity deviation k4 of
primitive motion 1 always becomes negative in comparison
with Fig.7. The negative gain leads the robot body to lean
forward to the ball more than the desired body angle. As a
result of adding positive feedback of posture controller, the
acquired motion deviates from a stepwise trajectory designed
by hand.
-0.08
-0.06
-0.04
-0.02
 0
 0.02
 0.04
 0.06
 0.08
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
de
sir
ed
 b
od
y 
an
gl
e 
[ra
d]
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(d)Desired body angles for primitive motion 1 and 2
 0.6
 0.8
 1
 1.2
 1.4
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
e
la
ps
e 
tim
e 
[se
c]
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(e)Periods of time for primitive motion 1 and 2
Fig. 6. Profile of posture parameters
VI. CONCLUSION
In this paper, we introduced a method for generating a
dynamic motion such that a two-wheeled inverted pendulum
robot kicks a ball far away utilizing its own body dynamics
while it keeps standing. We used the simple policy gradient
method to acquire the primitive motions of the kicking
motion. Based our approach, we succeeded in acquiring the
kicking motion of the two-wheeled inverted pendulum robot.
The policy gradient method just updates parameters of the
kicking motion so that the evaluation becomes as higher as
possible. As a result of these learning process, we confirmed
that a stepwise learning curve with two learning phases
emerged. From the profile of the learning parameters, we
confirmed that the robot decreased feedback gains for angular
velocities and updated posture parameters rapidly in the
earlier learning phase while it increased the feedback gains
once it found good posture parameters in latter learning
phase. It means that the robot changes a learning strategy
between the phases during the learning process. A very
interesting finding is: as a result of adding positive feedback
of posture controller, the trajectory of the acquired motions
deviated from the desired one given by the human designer
while keeping evaluation value of the acquired motion high.
In the near future, we will perform real robot experiments.
REFERENCES
[1] K. Hirai, M. Hirose, Y. Haikawa, and T. Takenaka, “The development of
honda humanoid robot,” in Proceedings of 1998 {IEEE} International
833
Conference on Robotics and Automation, vol. 2, 1998, pp. 1321 – 1326.
[2] M. Wisse and J. van Frankenhuyzen, “Design and construction of
mike and -D autonomous biped based on passive dynamic walking,”
in Adaptive Motion of Animals and Machines, H. Kimura, K. Tsuchiya,
A. Ishiguro, and H. Witte, Eds. Springer Tokyo, 2006, ch. 4, pp.
143–154.
[3] K. Narioka and K. Hosoda, “Designing synergistic walking of a whole-
body humanoid driven by pneumatic artificial muscles: An empirical
study,” Advanced Robotics, vol. 22, no. 10, pp. 1107–1123, 2008.
[4] S. Ha, Y. Yuta, “Trajectory tracking control for navigation of self-
contained mobileinverse pendulum,” in Proceedings of the IEEE/RSJ/GI
International Conference on Intelligent Robots and Systems ’94, vol. 3,
1994, pp. 1875–1882.
[5] B. Browning, J. Searock, P. E. Rybski, and M. Veloso, “Turning segways
into soccer robots,” Industrial Robot: An International Journal, vol. 32,
no. 2, pp. 149 – 156, 2005.
[6] N. Kohl and P. Stone, “Policy gradient reinforcement learning for fast
quadrupedal locomotion,” in In Proceedings of the IEEE International
Conference on Robotics and Automation, May 2004, pp. 2619–2624.
 13
 13.5
 14
 14.5
 15
 15.5
 16
 16.5
 17
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
ga
in
 o
f b
od
y 
an
gl
e
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(f)Gain for body angular error
 0.1
 0.15
 0.2
 0.25
 0.3
 0.35
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
ga
in
 o
f b
od
y 
an
gu
la
r v
el
oc
ity
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(g)Gain for body angular velocity
-0.01
-0.005
 0
 0.005
 0.01
 0.015
 0.02
 0.025
 0.03
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
ga
in
 o
f m
ot
or
 a
ng
ul
ar
 v
el
oc
ity
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(h)Gain for wheel angular velocity
-0.003
-0.002
-0.001
 0
 0.001
 0.002
 0.003
 0  5000  10000  15000  20000  25000  30000  35000  40000  45000  50000
ga
in
 o
f m
ot
or
 a
ng
ul
ar
 v
el
oc
ity
 d
ev
ia
tio
n
number of trial
 motion1: w1 = 1
 motion1: w1 = 10
 motion1: w1 = 100
 motion1: w1 = 1000
 motion2: w1 = 1
 motion2: w1 = 10
 motion2: w1 = 100
 motion2: w1 = 1000
(i)Gain for wheel angular velocity deviation
Fig. 7. Profiles of gain parameters for primitive motions
834
-0.4
-0.3
-0.2
-0.1
 0
 0.1
 0.2
 0.3
 0.4
1.0 2.0 3.0
bo
dy
 a
ng
le
 [ra
d]
time [sec]
 body agnel  disired body angle
(a) initial kicking motion
-0.4
-0.3
-0.2
-0.1
 0
 0.1
 0.2
 0.3
 0.4
1.0 2.0
bo
dy
 a
ng
le
 [ra
d]
time [sec]
 body agnel  disired body angle
(b) after 350 updates
-0.4
-0.3
-0.2
-0.1
 0
 0.1
 0.2
 0.3
 0.4
1.0 2.0 3.0
bo
dy
 a
ng
le
 [ra
d]
time [sec]
 body agnel  disired body angle
(c) after 1000 updates
Fig. 8. Trajectories of body angle while kicking the ball after updates of
learning parameters
835


Behavioral Development of Ball Kicking Motion of a Two-wheeled Inverted Pendulum Mobile Robot

Abstract

Similar works

Full text

Available Versions

University of Fukui Repository