This paper describes a methodology for gesture control of a custom developed mobile robot, using body gestures and Microsoft Kinect sensor. The Microsoft Kinect sensor’s ability is to track joint positions has been used in order to develop software application gestures recognition and their mapping into control commands. The proposed methodology has been experimentally evaluated. The results of the experimental evaluation, presented in the paper, showed that the proposed methodology is accurate and reliable and it could be used for mobile robot control in practical applications

Cekova, Katerina

Koceska, Natasa

Koceski, Saso

Katerina Cekova

Natasa Koceska

Saso Koceski

Crossref

Gesture Control of a Mobile Robot using Kinect Sensor

UGD Academic Repository

FIKT Repository

International Conference on Applied Internet and Information 
Technologies, 2016 
DOI:10.20544/AIIT2016.31
Gesture Control of a Mobile Robot using Kinect Sensor 
Katerina Cekova1, Natasa Koceska1, Saso Koceski1 
1 Faculty of Computer Science, Krste Misirkov 10-A, 
2000 Stip, Macedonia 
katerina.210171@student.ugd.edu.mk 
{natasa.koceska, saso.koceski}@ugd.edu.mk 
Abstract. This paper describes a methodology for gesture control of a custom 
developed mobile robot, using body gestures and Microsoft Kinect sensor. The 
Microsoft Kinect sensor’s ability is to track joint positions has been used in order 
to develop software application gestures recognition and their mapping into 
control commands. The proposed methodology has been experimentally 
evaluated. The results of the experimental evaluation, presented in the paper, 
showed that the proposed methodology is accurate and reliable and it could be 
used for mobile robot control in practical applications. 
Keywords: Kinect sensor, gesture recognition, mobile robot, control, human-
robot interaction. 
1. Introduction
With the development of technology, robots are gradually entering our life. The 
applications are ranging from rehabilitation [1, 2], assisted living [3], education [4], 
housework assistance [5], to warfare applications [6]. Various applications require 
specific control strategies and controllers. Development of a myriad of low-cost sensing 
devices, even nowadays, makes remote control of robotics devices a topic of interest 
among researchers. In particular, gesture control of robotic devices with different 
complexity and degrees-of-freedom is still considered as a challenging task [7].  
In this context, recently developed depth sensors, like Kinect sensor [8], have 
provided new opportunities for human-robot interaction. Kinect can recognize different 
gestures performed by a human operator, thus simplifying the interaction process. In 
this way, robots can be controlled easily and naturally. The key enabling technology is 
human body language understanding. The computer must first understand what a user is 
doing before it can respond [9, 10, 11]. 
The concept of gesture control to manipulate robots has been used in many research 
studies. Thanh et al. [12] developed a system in which human user can interact with the 
robot using body language. They used a semaphore communication method for 
controlling the iRobot to move up, down, to turn left or right. Waldherr et al. [13] 
describe a gesture interface for controlling a mobile robot equipped with a manipulator. 
They have developed an interface which uses a camera to track a person and recognize 
gestures involving arm motion. Luo et al. [14] use the Kinect sensor as motion capture 
device to directly control the arms by using the Cartesian impedance control to follow 
251
Cekova et al. 
the human motion. Jacob and Wachs [15] have used hand’s orientation and hand 
gestures to interact with a robot which uses the recognized pointing direction to define 
its goal on a map. Kim and Hong [16] have proposed a system intended to support 
natural interaction with autonomous robots in public places, such as museums and 
exhibition centers.  
In this paper we have used Kinect sensor for a real-time mobile robot control. The 
developed application allows us to control the robot with a predefined set of a body 
gestures. The operator, standing in front of the Kinect, performs a particular gesture 
which is recognized by the system. The system then sends commands to the 
microcontroller Arduino Uno, which operates with the robot. 
2. System architecture
The proposed system is composed of several components: operator, Microsoft Kinect 
sensor, mobile robot, computer, and mobile robot. The architecture of the system is 
shown in Figure 1. 
Fig. 1. System architecture 
To control the mobile robot user has to make some pre-defined gestures using its 
body. The Kinect sensor than captures the movement in real time and sends the skeleton 
joints’ data to the computer via USB for processing. Application on the computer 
processes the information from the Kinect sensor converts them into control commands 
and sends them to the robot i.e. to the Arduino microcontroller via Bluetooth. 
Depending on the received command data the mobile robot is moving. 
252
 Gesture Control of a Mobile Robot using Kinect Sensor 
2.1. Microsoft Kinect sensor 
Kinect sensor was introduced on the market in 2010 as a line of motion sensing input 
devices by Microsoft and was intended to be used with Xbox 360 console. It is a 
peripheral input device composed of several sensors. Namely, it contains a depth 
sensor, a RGB camera, and four-microphone array. The core component of Kinect is the 
range camera (originally developed by PrimeSense) which is using an infrared projector 
and camera and a special microchip to track objects in 3D. So, the sensor provides full-
body 3D motion capture, facial recognition, and voice recognition capabilities. The 
internal structure (sensor components) and the architecture of Kinect for Windows are 
presented in Figure 2. 
Fig. 2. Kinect Sensor internal structure 
Capabilities of image and depth capturing, audio recording as well as its low cost 
make Kinect very popular input device for various applications. It allows users natural 
interaction with the computer and control of applications or games solely using their 
bodies. This is enabled by identification of the position and orientation of 25 individual 
joints (including thumbs) and their motion tracking. The angular field of view of the 
Kinect sensor in the horizontal direction is 57° and in the vertical direction is 43°. It is 
also equipped with an additional motorized pivot which can tilt the sensor up for an 
additional 27° either up or down. The sensor can maintain tracking through an extended 
range of about 0.7–6 m. It can track up to 6 human body skeletons in the working area. 
2.2. Mobile robot description 
Developed mobile robot contains the following components: Bluetooth Module HC-06, 
Arduino Uno, H-Bridge L293D, DC motors, Jumper Wire and Breadboard. The control 
algorithm runs on the Arduino Uno. The communication with the computer is 
established via the HC-06 Bluetooth Module. The L293D is designed to provide 
bidirectional drive currents of up to 600-MA at voltages from 4.5V to 36V. The 
connection scheme of the robot is depicted in Figure 3 and the developed robot is 
presented in Figure 4. 
253
Cekova et al. 
Fig. 3. Connection schema of the mobile robot 
Fig. 4. Developed robot and its model 
The develop robot has two powered wheels driven by separate DC motors and one 
castor wheel used for stability. Robot position in the global reference frame is defined 
by the position of the mid-point axis (M) and the heading angle i.e. with following 
vector (x, y, θ). Considering the robot geometry (Figure 4) one may conclude that 
during the motion its left and right wheels are moving around the ICC with the same 
angular velocity as defined by Eq.1. 
L
VV
LR
V
LR
V
R
V LRLRM
M






)2/()2/(

(1) 
The instantaneous curvature radius of the trajectory is defined by the Eq.2. 
2
L
VV
VV
R
LR
LR 



(2) 
The kinematic model of the developed robot can be represented with Eq.3. 
254
 Gesture Control of a Mobile Robot using Kinect Sensor 










































M
MV
Vy
Vx
y
x



 10
0sin
0cos
.
.
. (3) 
Having in mind that VM=(VR+VL)/2 the above equation could be rewritten as: 
































L
R
V
V
LL
y
x
/1/1
sin2/1sin2/1
cos2/1cos2/1
.
.
.



(4) 
3. System evaluation
To evaluate the proposed system a dedicated application written in Visual Studio using 
C# programming language was developed. The interface of the application is shown in 
Figure 5. Joints that are recognized and tracked are marked with blue circles. 
Fig. 5. Screenshots of the developed application 
The set of gestures aimed at robot control and their meaning are presented in Table 
1. 
To stop the robot user needs to move down right and left elbow. If one elbow is up 
and other is down the robot is moving right or left depending on which elbow is up. To 
move the car forward or backward the user must place the right and left elbow in front 
or behind the hip center. The software application than detects the user’s gesture in real-
time. The application converts detected movements into control commands. Bluetooth 
255
Cekova et al. 
communication is used to sends the commands to the microcontroller on the robot. With 
detected body movement user controls the motors. Also, the microcontroller returns 
direction of movement back in the application.  
Table 1. Overview of gesture sets. 
Command Action description 
Stop elbow right and elbow left down 
Right elbow right up elbow left down 
Left elbow right down elbow left up 
Forward elbow right and left placed before hip center 
Backward elbow right and left placed behind hip center 
The system has been evaluated in the laboratory environment. The aim of the 
evaluation was to determine the percentage of successfully recognized and executed 
commands. The robot was controlled by five different users that were initially 
instructed on how the system works, and each user was given 20 test trials. Afterwards 
they were asked to perform each command three times in a random order. The results 
(in %) of successfully recognized and executed commands are presented in the Table 2 
bellow. 
Table 2. Results of the experimental evaluation. 
User 
No. 
Stop Right Left Forward Backward 
% 
Rec 
% 
Exe
c 
% 
Rec 
% 
Exe
c 
% 
Rec 
% 
Exec 
% 
Rec 
% 
Exec 
% 
Rec 
% 
Exec 
U1 85 94 100 100 100 100 95 100 90 100 
U2 90 89 100 100 100 100 85 100 95 100 
U3 90 100 100 100 100 100 100 100 100 100 
U4 95 100 100 100 100 100 100 100 100 100 
U5 95 100 100 100 100 100 100 100 100 100 
Moreover, during the tests we have measured execution times for each command 
(the time from the moment the gesture is performed by the operator till its execution), 
and we have determined that they vary between 50 and 150ms, which makes this 
interface applicable even for real-time control.   
4. Conclusion
In this paper we present a system that enables human robot interaction using low-cost 
Microsoft Kinect sensor. The ability of depth perception enables full-human body 
skeleton tracking and gesture recognition. Recognized gestures afterwards are used to 
turn the mobile robot left, right or move it forward, backward and to stop it. The 
evaluation of the proposed system suggests that the proposed human-robot interface is 
reliable (with high percentage of recognized gestures) and accurate (high percentage of 
256
 Gesture Control of a Mobile Robot using Kinect Sensor 
executed actions) and thus it could be used in practice for various applications of 
mobile robot control. The proposed system could be applied in control of various 
robotic devices (such as robotic wheelchairs) that are aimed at disabled persons having 
functional upper limbs. Moreover, the proposed system could be applied for control of 
industrial or medical processes where the user could not directly interact with the 
equipment or apparatus. From user perspective, the system could support more than one 
user because the Kinect sensor and the developed control application have the 
possibility to track multiple skeletons in the same time.  
References 
1. Koceska, Natasa, Saso Koceski, Francesco Durante, Pierluigi Beomonte, and Terenziano
Raparelli. "Control architecture of a 10 DOF lower limbs exoskeleton for gait rehabilitation."
International Journal of Advanced Robotic Systems 10 (2012): 1.
2. Koceska, Natasa, and Saso Koceski. "Review: Robot Devices for Gait Rehabilitation."
International Journal of Computer Applications 62, no. 13 (2013).
3. Koceski, Saso, and Natasa Koceska. "Evaluation of an Assistive Telepresence Robot for
Elderly Healthcare." Journal of Medical Systems 40, no. 5 (2016): 1-7.
4. Serafimov, Kire, Dimitrija Angelkov, Natasa Koceska, and Saso Koceski. "Using mobile-
phone accelerometer for gestural control of soccer robots." In Embedded Computing
(MECO), 2012 Mediterranean Conference on, pp. 140-143. IEEE, 2012. 
5. Koceski, Saso, Natasa Koceska, and Ivica Kocev. "Design and evaluation of cell phone
pointing interface for robot control." International Journal of Advanced Robotic Systems 9
(2012). 
6. Arkin, Ronald C. "Ethical robots in warfare." Technology and Society Magazine, IEEE 28, no.
1 (2009): 30-33.
7. Koceski, Saso, and Natasa Koceska. "Vision-based gesture recognition for human-computer
interaction and mobile robot's freight ramp control." In Information Technology Interfaces
(ITI), 2010 32nd International Conference on, pp. 289-294. IEEE, 2010. 
8. Zhang, Z.: Microsoft kinect sensor and its effect. MultiMedia, IEEE, 19(2), pp.4-10.(2012)
9. Xia, L., Chen, C.C. and Aggarwal, J.K.: Human detection using depth information by kinect.
In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer
Society Conference on (pp. 15-22). IEEE. (2011) 
10. Gabriele Pozzato, Stefano Michieletto, Emanuele Menegatti, Fabio Dominio, Giulio Marin,
Ludovico Minto, Simone Milani, Pietro Zanuttigh,   HUMAN-ROBOT INTERACTION
WITH DEPTH-BASED GESTURE RECOGNITION
11. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R.,  Blake, A.
(2011). Real-time human pose recognition in parts from single depth images. In Proceedings
of IEEE conference on computer vision and pattern recognition (CVPR 2011) (pp. 1297–
1304).
12. Nguyen-Duc-Thanh, N., Stonier, D., Lee, S. and Kim, D.H.: A new approach for human-
robot interaction using human body language. In Convergence and Hybrid Information
Technology (pp. 762-769). Springer Berlin Heidelberg.(2011)
13. Waldherr, S., Romero, R. and Thrun, S.: A gesture based interface for human-robot
interaction. Autonomous Robots, 9(2), pp.151-173.(2000)
14. Luo, R.C., Shih, B.H. and Lin, T.W.: October. Real time human motion imitation of
anthropomorphic dual arm robot based on Cartesian impedance control. In Robotic and
Sensors Environments (ROSE), 2013 IEEE International Symposium on (pp. 25-30).
IEEE.(2013)4.
257
Cekova et al. 
15. M. G. Jacob and J. P. Wachs. Context-based hand gesture reognition for the operating room.
Pattern Recognition Letters, 36:196–203, 2014.
16. J.Oh, T. Kim, and H. Hong. Using binary decision tree and multiclass svm for human gesture
recognition. In International Conference on Information Science and Applications (ICISA),
pages 1–4, 2013.
258


http://eprints.ugd.edu.mk/16979/1/Proceedings%20AIIT2016%20-%201.pdf

Gesture Control of a Mobile Robot using Kinect Sensor

Abstract

Similar works

Full text

Available Versions

Crossref

UGD Academic Repository

FIKT Repository