This paper presents an implementa- tion of three Fuzzy Logic controllers working in parallel onboard a UAV, two for a pan-tilt camera platform and the third for control the yaw of the helicopter. This implementation uses a Lucas-Kanade tracker algo- rithm with a pyramidal optical  ow implementation, which gives infor- mation to follow statics and moving objects, besides the UAV vibrations and movements. The platform con- troller is helped by the heading con- troller, in order to make smooth the big movements to the platform, re- ducing the risk of lost the warp selec- tion of the object to track. Also, the heading control remove the physic limit of the platform at the yaw axis. Some laboratory and UAV tests are presented in order to show the di

er- ent behaviors and the good response of the presented controllers

Campoy Cervera, Pascual

Martínez Luna, Carol Viviana

Mondragon Bernal, Ivan Fernando

Olivares Méndez, Miguel Ángel

peer reviewedThis paper presents an implementa-
tion of three Fuzzy Logic controllers
working in parallel onboard a UAV,
two for a pan-tilt camera platform
and the third for control the yaw of
the helicopter. This implementation
uses a Lucas-Kanade tracker algo-
rithm with a pyramidal optical 
ow
implementation, which gives infor-
mation to follow statics and moving
objects, besides the UAV vibrations
and movements. The platform con-
troller is helped by the heading con-
troller, in order to make smooth the
big movements to the platform, re-
ducing the risk of lost the warp selec-
tion of the object to track. Also, the
heading control remove the physic
limit of the platform at the yaw axis.
Some laboratory and UAV tests are
presented in order to show the differ-
ent behaviors and the good response
of the presented controllers

Olivares Mendez, Miguel Angel

Campoy, Pascual

Mondragon, Ivan F.

Martinez, Carol

Open Repository and Bibliography - Luxembourg

Visual Servoing using Fuzzy Controllers on an UnmannedAerial VehicleMiguel A.Olivares-Me´ndezUniversidad Polite´nicade Madridmig olivares@hotmail.comPascual CampoyCerveraUniversidad Polite´nicade Madridpascual.campoy@upm.esIva´n Mondrago´nUniversidad Polite´nicade Madridivanmond@yahoo.comCarol Mart´ınezUniversidad Polite´nicade Madridcarofide@hotmail.comAbstractThis paper presents an implementa-tion of three Fuzzy Logic controllersworking in parallel onboard a UAV,two for a pan-tilt camera platformand the third for control the yaw ofthe helicopter. This implementationuses a Lucas-Kanade tracker algo-rithm with a pyramidal optical flowimplementation, which gives infor-mation to follow statics and movingobjects, besides the UAV vibrationsand movements. The platform con-troller is helped by the heading con-troller, in order to make smooth thebig movements to the platform, re-ducing the risk of lost the warp selec-tion of the object to track. Also, theheading control remove the physiclimit of the platform at the yaw axis.Some laboratory and UAV tests arepresented in order to show the differ-ent behaviors and the good responseof the presented controllers.Keywords: Unmanned Aerial Ve-hicles, Fuzzy Control, Visual servo-ing, pan-tilt platform.1 IntroductionThe object detection and tracking is a com-mon problem in vision systems in all of therobotics platforms. On one hand, using Un-manned Aerial Vehicle enlarge the possibleapplications by the increase of the workspacesize, covering a bigger area to inspect, de-tect and tracking, but, on the other hand, in-crease the problem of visual servoing, basedon the difficulties of the aircraft stabilizationand control when we are trying to track anobject. There are different visual algorithmsin order to face this problem working withUAV, some of then are presented in [1]. Forthis work we use the pyramidal Lucas-Kanadeoptical flow implementation. The problem in-crease when we have not got total freedomof movements by big environment perturba-tions, inaccessible environments positions orunattainable objets or targets to track, as wellas have a path planning desires.The uses of Fuzzy logic are growing in somefields at robotics, trying to made a betteradaptation to resolve the non-linear problemsof the real world. This technique give us easy,rapid and comprehensive way to create con-trollers by the definition of the linguistic valueof its variables, base of rules and the easy wayto tuning this kind of controllers, against theuses of the classical PID controllers, for whatit is necessary to calculate the model of thehelicopter and the visual platform.On the other hand the interest in UAVs isincreasing too, based on the significant de-velopments on UAVs fields on control archi-tectures and the cheap and suitable alterna-tive that it bring. Currently, some applica-tions have been developed, among which wecan find Valavanis’ works on traffic monitor-ing [7], and fire detection [4] or the worksin obstacles avoidance and 3D path planningfor UAVs by [3]. There are some tests usinga pan-tilt video-platform, like [8] based thetracking on a biomimetic Eye or the uses fora target tracking and motion estimation formoving targets by [2].For a better comprehension we divided thispaper in the next sections. In the section 2we show a description about the UAV used inthis work. Section 3 will show the Miguel Oli-vares’ Fuzzy Software definition and the con-figurations of the different fuzzy controllers.The presentation of some results obtained forthis work are shown in the chapter 4 and, tofinalize, we present the conclusions and thefuture works in section 5.2 UAV System DescriptionThe Colibri project has three totally operativeUAV platforms. One electric helicopter, andtwo gas powered helicopters. The COLIBRItestbeds [1], are equipped with an xscale-based flight computer augmented with sen-sors (GPS, IMU, Magnetometer, fused witha Kalman filter for state estimation). For thiswork, the most used helicopter has been theSR20 of “Rotomotion”, the electric one (fig-ure 1).Figure 1: COLIBRI III Electric helicopter.It includes a two axis pan-tilt video-platformpowered by two servo-motors, with an ac-tion range of 180 degrees. For the control ofthe platform we send commands to the ser-vos through the helicopter controller, basedon the visual information acquired. To per-form vision processing, it has a VIA nano-ITX1.5 GHz onboard computer with 1 Gb RAM.The principal aim of this work is to createa suitable automatic control to the differentsituations or desirable solutions that we canface with a UAV with vision support. For thisreason we make the fusion between control ofdifferent parts of the helicopter, the video-platform and the heading of the helicopter.Besides the difficulties of the environment orthe necessities of the pilot, giving the possibil-ity to track an object in the next situations:1. With a preprogrammed series of way points(path planning).2. When flight commands are sent from thepilot radio control or from the ground station.3. Staying in hovering position in a safe place.Some of those work modes are shown in thechapter 4.3 Visual servoing using Fuzzycontrollers3.1 Fuzzy Software ImplementationFor this work we use the MOFS (Miguel Oli-vares’ Fuzzy Software), developed in previ-ous works [6], [5]. This software was in-dependently designed defining one class foreach part of the fuzzy-logic environment (vari-ables, rules, membership functions and de-fuzzification modes) in order to facilitate thefuture updates and making easier work withit. There are different classes for Dependingon the system we want to create we can de-fine the number of inputs and outputs that weprefer or make some different system in serialmode or working in parallel.One of the differences between this softwareand others Fuzzy software is that it lets to rep-resent a more important sector in each fuzzyvariables, giving us the possibility to reducethe size of the rule-base, that will give improveresponse-time and reduce the computationalcost. Others differences and a more explaineddocumentation can be consulted in [6], [5].The MOFS is defined like a controller class in-side the server program running onboard thehelicopter.3.2 Fuzzy ControllersThe implementation of the task of the visualtracking control system is made with the fu-sion between the Lucas-Kanade tracker algo-rithm and three Fuzzy controllers working inparallel. Two of these controller are for thevision platform of the UAV, one for each axisand the other is for the yaw control of thehelicopter (heading). All of these controllersare implemented using the previous softwaredefinition (MOFS). We used a camera with320x240 resolution. Getting the informationof the error from -160 to 160 pixels in yaw andfrom -120 to 120 for the pitch axis.The fuzzification of the inputs and the out-puts are defined by using a triangular mem-bership function, for the platform controllers,and trapezoidal membership functions, for theheading. The controllers has two inputs, theerror between the center of the object and thecenter of the image (figures 2(a) and 2(c))and the difference between the last and theactual error (figures 2(b) and 2(d)), deriva-tive of the position or the velocity of the ob-ject to track. The platform controllers outputrepresents how many position must the servo-motor turn, from 0 to 255, in the two axis, tominimize the error, with the same definitionof the output of both (figure 3(a)). The head-ing controller have the same inputs of the yawcontroller (figure 2(a) and 2(b)) , and the out-put represented how many radians must theUAV rotate to line up the object (figure 3(b)).The three controllers are working in parallelgiving a redundant operation to the yaw axis,in order to reduce the error that we have withthe yaw-platform controller, where the limi-tations of the visual algorithm and the move-ments velocity of the servos hinders us to takea quicker response. The other utility of thisthird controller is to eliminate the turn limita-tion of the platform when the object to trackgoing to the back part of the UAV. All of theseare guided by a 49 rules base. Defining for theoutput of the platform controllers a more im-portant sector in the section near to zero, asshown in figure 3(a). This option, give us thepossibility to define a very sensible controllerwhen the error is so small, the object is verynear to the center of the image, and a veryquick respond controller when the object is sofar. For the heading controller we defined atrapezoidal part in the middle of the outputin order to help the platform controller, justwhen the object to track is with so far to thecenter of the image. With these trapezoidaldefinition we get a more stable behavior of thehelicopter, in the situations where the objectto track is near to the center, obtaining a 0value.(a) Yaw Error.(b) Derivative of the Yaw error.(c) Pitch Error.(d) Derivative of the Pitch error.Figure 2: Inputs Variables of the controllers.(a) Output of the Yaw and the Pitchcontrollers.(b) Output of the Heading controller.Figure 3: Output Variables of the controllers.4 Experiments4.1 Laboratory ExperimentsIn order to test and fit the heading controllerwe make some tests with the platform control,tracking a real object inside the laboratoryand with a communication in real time withthe simulator of the helicopter. In figure 4 isshown the error, in pixels. The response of theheading controller is shown in figure 5, wherewe can see that it just response to big errorsin the yaw angle of the image. Also, we cansee, in figure 6, how these signals affect to thehelicopter heading, changing the yaw angle inorder to collaborate with the yaw controllerof the visual platform.Figure 4: Error between the static object trackedand the center of the image, running with the UAVsimulator.Figure 5: Response of the Fuzzy control for theheading of the helicopter.Figure 6: Heading Response.4.2 Tests on UAVIn this subsection we show the real tests on-board the UAV, tracking static and movingobjects. For these tests we use the controllersof the visual platform.4.2.1 Tracking Static ObjectsHere a path-planning flight with a static ob-ject tracking is presented. In figure 7 is showna 3D reconstruction of the flight using theGPS and the IMU data. The frame-rate is15 frames per seconds, and the 2500 framesrepresent a real flight of almost 3 minutes. Infigure 8 is shown the movements of Yaw andPitch of the UAV, which give us a better mea-sure about the movements and the problemsto solve by the controller and the tracker fortracking and follow the mark.Figure 7: 3D flight reconstruction from the GPSand the IMU data from the UAV. Where, the ’X’axis represents the NORTH axis of the surface ofthe tangent of the earth, the ’Y’ axis representsthe EAST axis of the earth, the ’Z’ is the altitudeof the helicopter and the red arrows show the pitchangle of the helicopter.(a) Pitch angle movements.(b) Yaw angle movements.Figure 8: Different pitch and yaw movements ofthe UAV.In figure 10, the output of the two Fuzzy-MOFS controllers in order to compensate theerror caused by the changes of the differentmovements and angle changes of the UAVflight, where we can see all the responses ofthe controllers, depending the sizes and thetypes of the perturbations.Figure 9: Error between center of the image andcenter of the object to track.The table 4.2.1 shown the data of the flight.We had a maximum error in the flight, takingapart the ignition of the motor, of +100 pix-Table 1: Data from big attitude changes sections of the flight.Section Frames Interval Attitude angle Degrees Frames Num. Time degrees per sec. Pixels Error1 540-595 Yaw +8 55 3.6s +2.28/sec +100 (Yaw)1 590-595 Roll -5 5 0.33s -15/sec +100 (Yaw)1 570-595 Pitch -4 25 1.6s -2.5/sec +40 (Pitch)1 595-620 Yaw -22 25 1.6s -13.75/sec +50 (Yaw)1 595-660 Roll +10 65 4.3s +2.35/sec +50 (Yaw)1 620-660 Yaw +20 40 2.6s +15.38/sec -75 (Yaw)2 1460-1560 Yaw -40 100 6.6s -6.06/sec +52 (Yaw)2 1560-1720 Yaw +28 160 10.6s +2.64/sec 48 (Yaw)3 2170-2260 Yaw -35 90 6s -5.8/sec 55 (Yaw)4 2375-2450 Yaw -27 75 5s -5.4/sec 48 (Yaw)Figure 10: Output from the Fuzzy Controller.els during the initial elevation of the aircraftand a maximum error of 55 pixels during theflight. The error represents a 62.2% in theinitial elevation, where we have a fusion of allthe possible movements and angles changes ina UAV, with a bigger change of 15.38 degreesper sec. And a 34.375% of the yaw axis ofthe camera, with a maximum angle change of6.06 degrees per sec. So we can say that thecontroller has a good behavior solving thosekind of problems.4.2.2 Tracking Moving ObjectsHere we present a tracking of a van with con-tinuous movements of the helicopter for in-crease the difficulty of the test. In figure 11 wecan see the error in pixels of the two axis of theimage. Also, we can see the moments wherewe deselected the template and re-selected itfor increase the difficulty for the controller,those intervals are represented when the errorremain fixed in one value for a long time. InFigure 11: Error between center of the image andcenter of the dynamic object (a van) to track.the figures 12 and 13 we can see the responseof the to controllers, where we can see the bigmovements of the controller send to the ser-vos when the warp is re-selected. We mustnoticed that in all of those figures that showthe controller responses there are not regis-ter the data when the warp selection is de-selected, because in those movements we donot made any track. We can see that the big-ger response of the controllers are almost ±10degrees for the yaw controller and almost 25degrees for the pitch controller, being the cor-rection in a few frames, less than 10 frames.Figure 12: Response of the Fuzzy control for theYaw axis of the visual platform tracking a dynamicobject (a van).Figure 13: Response of the Fuzzy control for thePitch axis of the visual platform tracking a dy-namic object (a van).5 Conclusions and Future WorksIn this work we present an autonomous fuzzycontrol of pan and tilt video platform onboard a UAV and a UAV-heading controllerusing a Lucas-Kanade tracker for static andmoving objects. The controllers have a ex-cellent behavior tracking static objects andmoving objects, besides the perturbations ofthe environments and the own vibrations ofthe helicopter. The use of the pan and tiltvisual platform give to the helicopter a free-dom of movements, as well as, a faster re-sponse with moving objects tracking, beingthe others implementations of visual servoingon UAV without pan and tilt platform morelimited and slower than the platform servosresponse. With the heading controller we canmade the movements of the platform a littlebit soft. Also, this control implementation ap-port more states to work with the helicopterand tracking object giving us a lot of possi-ble applications, because is possible to trackobject from a hover position due to limita-tions of the environments or the helicoptermovements, like situations of stay near to anelectric structures, fields of wind turbines ordams. Another work state is with a path plan-ning, giving the possibility of create a reliabletestbed for different controllers or visual algo-rithm, as well with commands sent from thepilot controller and the ground station.The principal and immediate future work is toimprove the fuzzy system for a better behav-ior and compare with other controllers. Alsonow we are working for taking the up-levelcontrol of the helicopter with other Fuzzy-MOFS controllers that send to the UAV com-mands of yaw and pitch to increase the pos-sibility to follow dynamic objects for a longtime and in order to find more applications.It is possible to view the tests videos and moreon the web www.disam.upm.es/colibri.AcknowledgmentsThis work is the product of several researchstages at the Computer Vision Group Univer-sidad Polite´cnica de Madrid. Has been spon-sored by the Spanish Science and Technol-ogy Ministry under grants CICYT DPI2004-06624, CICYT DPI2000-1561-C02-02 and MI-CYT DPI2007-66156.References[1] P. Campoy, J. Correa, I. Mondragon,C. Martinez, M. Olivares, L. Mejias, andJ. Artieda. Computer vision onboard uavsfor civilian tasks. Journal of Intelligentand Robotic Systems, 2008.[2] V. Dobrokhodov, I. Kaminer, K. Jones,and R. Ghabcheloo. Vision-based trackingand motion estimation for moving targetsusing small uavs. American Control Con-ference, 2006, pages 6 pp.–, June 2006.[3] S. Hrabar. 3d path planning and stereo-based obstacle avoidance for rotorcraftuavs. Intelligent Robots and Systems,2008. IROS 2008. IEEE/RSJ Interna-tional Conference on, pages 807–814,Sept. 2008.[4] I. K. Nikolos, N. C. Tsourveloudis, andK. P. Valavanis. A uav vision system forairborne surveillance. In Robotics and Au-tomation, 2004. Proceedings. ICRA ’04.2004 IEEE International Conference on,pages 77–83, New Orleans, LA, USA, May2004.[5] M. Olivares, P. Campoy, J. Correa,C. Martinez, and I. Mondragon. Fuzzycontrol system navigation using priorityareas. In Proceedings of the 8th Interna-tional FLINS Conference, pages 987–996,Madrid,Spain, September 2008.[6] M. Olivares and J. Madrigal. Fuzzy logicuser adaptive navigation control systemfor mobile robots in unknown environ-ments. Intelligent Signal Processing, 2007.WISP 2007. IEEE International Sympo-sium on, pages 1–6, Oct. 2007.[7] A. Puri, K. Valavanis, and M. Kontit-sis. Statistical profile generation for traf-fic monitoring using real-time uav basedvideo data. In Control and Automation,2007.MED ’07. Mediterranean Conferenceon, MED, pages 1–6, 2007.[8] H. Zou, Z. Gong, S. Xie, and W. Ding.A pan-tilt camera control system of uavvisual tracking based on biomimetic eye.Robotics and Biomimetics, 2006. ROBIO’06. IEEE International Conference on,pages 1477–1482, Dec. 2006.

Visual servoing using fuzzy controllers on an unmanned aerial vehicle

Servicio de Coordinación de Bibliotecas de la Universidad Politécnica de Madrid

Visual Servoing using Fuzzy Controllers on an Unmanned
Aerial Vehicle
Miguel A.
Olivares-Me´ndez
Universidad Polite´nica
de Madrid
mig olivares@hotmail.com
Pascual Campoy
Cervera
Universidad Polite´nica
de Madrid
pascual.campoy@upm.es
Iva´n Mondrago´n
Universidad Polite´nica
de Madrid
ivanmond@yahoo.com
Carol Mart´ınez
Universidad Polite´nica
de Madrid
carofide@hotmail.com
Abstract
This paper presents an implementa-
tion of three Fuzzy Logic controllers
working in parallel onboard a UAV,
two for a pan-tilt camera platform
and the third for control the yaw of
the helicopter. This implementation
uses a Lucas-Kanade tracker algo-
rithm with a pyramidal optical flow
implementation, which gives infor-
mation to follow statics and moving
objects, besides the UAV vibrations
and movements. The platform con-
troller is helped by the heading con-
troller, in order to make smooth the
big movements to the platform, re-
ducing the risk of lost the warp selec-
tion of the object to track. Also, the
heading control remove the physic
limit of the platform at the yaw axis.
Some laboratory and UAV tests are
presented in order to show the differ-
ent behaviors and the good response
of the presented controllers.
Keywords: Unmanned Aerial Ve-
hicles, Fuzzy Control, Visual servo-
ing, pan-tilt platform.
1 Introduction
The object detection and tracking is a com-
mon problem in vision systems in all of the
robotics platforms. On one hand, using Un-
manned Aerial Vehicle enlarge the possible
applications by the increase of the workspace
size, covering a bigger area to inspect, de-
tect and tracking, but, on the other hand, in-
crease the problem of visual servoing, based
on the difficulties of the aircraft stabilization
and control when we are trying to track an
object. There are different visual algorithms
in order to face this problem working with
UAV, some of then are presented in [1]. For
this work we use the pyramidal Lucas-Kanade
optical flow implementation. The problem in-
crease when we have not got total freedom
of movements by big environment perturba-
tions, inaccessible environments positions or
unattainable objets or targets to track, as well
as have a path planning desires.
The uses of Fuzzy logic are growing in some
fields at robotics, trying to made a better
adaptation to resolve the non-linear problems
of the real world. This technique give us easy,
rapid and comprehensive way to create con-
trollers by the definition of the linguistic value
of its variables, base of rules and the easy way
to tuning this kind of controllers, against the
uses of the classical PID controllers, for what
it is necessary to calculate the model of the
helicopter and the visual platform.
On the other hand the interest in UAVs is
increasing too, based on the significant de-
velopments on UAVs fields on control archi-
tectures and the cheap and suitable alterna-
tive that it bring. Currently, some applica-
tions have been developed, among which we
can find Valavanis’ works on traffic monitor-
ing [7], and fire detection [4] or the works
in obstacles avoidance and 3D path planning
for UAVs by [3]. There are some tests using
a pan-tilt video-platform, like [8] based the
tracking on a biomimetic Eye or the uses for
a target tracking and motion estimation for
moving targets by [2].
For a better comprehension we divided this
paper in the next sections. In the section 2
we show a description about the UAV used in
this work. Section 3 will show the Miguel Oli-
vares’ Fuzzy Software definition and the con-
figurations of the different fuzzy controllers.
The presentation of some results obtained for
this work are shown in the chapter 4 and, to
finalize, we present the conclusions and the
future works in section 5.
2 UAV System Description
The Colibri project has three totally operative
UAV platforms. One electric helicopter, and
two gas powered helicopters. The COLIBRI
testbeds [1], are equipped with an xscale-
based flight computer augmented with sen-
sors (GPS, IMU, Magnetometer, fused with
a Kalman filter for state estimation). For this
work, the most used helicopter has been the
SR20 of “Rotomotion”, the electric one (fig-
ure 1).
Figure 1: COLIBRI III Electric helicopter.
It includes a two axis pan-tilt video-platform
powered by two servo-motors, with an ac-
tion range of 180 degrees. For the control of
the platform we send commands to the ser-
vos through the helicopter controller, based
on the visual information acquired. To per-
form vision processing, it has a VIA nano-ITX
1.5 GHz onboard computer with 1 Gb RAM.
The principal aim of this work is to create
a suitable automatic control to the different
situations or desirable solutions that we can
face with a UAV with vision support. For this
reason we make the fusion between control of
different parts of the helicopter, the video-
platform and the heading of the helicopter.
Besides the difficulties of the environment or
the necessities of the pilot, giving the possibil-
ity to track an object in the next situations:
1. With a preprogrammed series of way points
(path planning).
2. When flight commands are sent from the
pilot radio control or from the ground station.
3. Staying in hovering position in a safe place.
Some of those work modes are shown in the
chapter 4.
3 Visual servoing using Fuzzy
controllers
3.1 Fuzzy Software Implementation
For this work we use the MOFS (Miguel Oli-
vares’ Fuzzy Software), developed in previ-
ous works [6], [5]. This software was in-
dependently designed defining one class for
each part of the fuzzy-logic environment (vari-
ables, rules, membership functions and de-
fuzzification modes) in order to facilitate the
future updates and making easier work with
it. There are different classes for Depending
on the system we want to create we can de-
fine the number of inputs and outputs that we
prefer or make some different system in serial
mode or working in parallel.
One of the differences between this software
and others Fuzzy software is that it lets to rep-
resent a more important sector in each fuzzy
variables, giving us the possibility to reduce
the size of the rule-base, that will give improve
response-time and reduce the computational
cost. Others differences and a more explained
documentation can be consulted in [6], [5].
The MOFS is defined like a controller class in-
side the server program running onboard the
helicopter.
3.2 Fuzzy Controllers
The implementation of the task of the visual
tracking control system is made with the fu-
sion between the Lucas-Kanade tracker algo-
rithm and three Fuzzy controllers working in
parallel. Two of these controller are for the
vision platform of the UAV, one for each axis
and the other is for the yaw control of the
helicopter (heading). All of these controllers
are implemented using the previous software
definition (MOFS). We used a camera with
320x240 resolution. Getting the information
of the error from -160 to 160 pixels in yaw and
from -120 to 120 for the pitch axis.
The fuzzification of the inputs and the out-
puts are defined by using a triangular mem-
bership function, for the platform controllers,
and trapezoidal membership functions, for the
heading. The controllers has two inputs, the
error between the center of the object and the
center of the image (figures 2(a) and 2(c))
and the difference between the last and the
actual error (figures 2(b) and 2(d)), deriva-
tive of the position or the velocity of the ob-
ject to track. The platform controllers output
represents how many position must the servo-
motor turn, from 0 to 255, in the two axis, to
minimize the error, with the same definition
of the output of both (figure 3(a)). The head-
ing controller have the same inputs of the yaw
controller (figure 2(a) and 2(b)) , and the out-
put represented how many radians must the
UAV rotate to line up the object (figure 3(b)).
The three controllers are working in parallel
giving a redundant operation to the yaw axis,
in order to reduce the error that we have with
the yaw-platform controller, where the limi-
tations of the visual algorithm and the move-
ments velocity of the servos hinders us to take
a quicker response. The other utility of this
third controller is to eliminate the turn limita-
tion of the platform when the object to track
going to the back part of the UAV. All of these
are guided by a 49 rules base. Defining for the
output of the platform controllers a more im-
portant sector in the section near to zero, as
shown in figure 3(a). This option, give us the
possibility to define a very sensible controller
when the error is so small, the object is very
near to the center of the image, and a very
quick respond controller when the object is so
far. For the heading controller we defined a
trapezoidal part in the middle of the output
in order to help the platform controller, just
when the object to track is with so far to the
center of the image. With these trapezoidal
definition we get a more stable behavior of the
helicopter, in the situations where the object
to track is near to the center, obtaining a 0
value.
(a) Yaw Error.
(b) Derivative of the Yaw error.
(c) Pitch Error.
(d) Derivative of the Pitch error.
Figure 2: Inputs Variables of the controllers.
(a) Output of the Yaw and the Pitch
controllers.
(b) Output of the Heading controller.
Figure 3: Output Variables of the controllers.
4 Experiments
4.1 Laboratory Experiments
In order to test and fit the heading controller
we make some tests with the platform control,
tracking a real object inside the laboratory
and with a communication in real time with
the simulator of the helicopter. In figure 4 is
shown the error, in pixels. The response of the
heading controller is shown in figure 5, where
we can see that it just response to big errors
in the yaw angle of the image. Also, we can
see, in figure 6, how these signals affect to the
helicopter heading, changing the yaw angle in
order to collaborate with the yaw controller
of the visual platform.
Figure 4: Error between the static object tracked
and the center of the image, running with the UAV
simulator.
Figure 5: Response of the Fuzzy control for the
heading of the helicopter.
Figure 6: Heading Response.
4.2 Tests on UAV
In this subsection we show the real tests on-
board the UAV, tracking static and moving
objects. For these tests we use the controllers
of the visual platform.
4.2.1 Tracking Static Objects
Here a path-planning flight with a static ob-
ject tracking is presented. In figure 7 is shown
a 3D reconstruction of the flight using the
GPS and the IMU data. The frame-rate is
15 frames per seconds, and the 2500 frames
represent a real flight of almost 3 minutes. In
figure 8 is shown the movements of Yaw and
Pitch of the UAV, which give us a better mea-
sure about the movements and the problems
to solve by the controller and the tracker for
tracking and follow the mark.
Figure 7: 3D flight reconstruction from the GPS
and the IMU data from the UAV. Where, the ’X’
axis represents the NORTH axis of the surface of
the tangent of the earth, the ’Y’ axis represents
the EAST axis of the earth, the ’Z’ is the altitude
of the helicopter and the red arrows show the pitch
angle of the helicopter.
(a) Pitch angle movements.
(b) Yaw angle movements.
Figure 8: Different pitch and yaw movements of
the UAV.
In figure 10, the output of the two Fuzzy-
MOFS controllers in order to compensate the
error caused by the changes of the different
movements and angle changes of the UAV
flight, where we can see all the responses of
the controllers, depending the sizes and the
types of the perturbations.
Figure 9: Error between center of the image and
center of the object to track.
The table 4.2.1 shown the data of the flight.
We had a maximum error in the flight, taking
apart the ignition of the motor, of +100 pix-
Table 1: Data from big attitude changes sections of the flight.
Section Frames Interval Attitude angle Degrees Frames Num. Time degrees per sec. Pixels Error
1 540-595 Yaw +8 55 3.6s +2.28/sec +100 (Yaw)
1 590-595 Roll -5 5 0.33s -15/sec +100 (Yaw)
1 570-595 Pitch -4 25 1.6s -2.5/sec +40 (Pitch)
1 595-620 Yaw -22 25 1.6s -13.75/sec +50 (Yaw)
1 595-660 Roll +10 65 4.3s +2.35/sec +50 (Yaw)
1 620-660 Yaw +20 40 2.6s +15.38/sec -75 (Yaw)
2 1460-1560 Yaw -40 100 6.6s -6.06/sec +52 (Yaw)
2 1560-1720 Yaw +28 160 10.6s +2.64/sec 48 (Yaw)
3 2170-2260 Yaw -35 90 6s -5.8/sec 55 (Yaw)
4 2375-2450 Yaw -27 75 5s -5.4/sec 48 (Yaw)
Figure 10: Output from the Fuzzy Controller.
els during the initial elevation of the aircraft
and a maximum error of 55 pixels during the
flight. The error represents a 62.2% in the
initial elevation, where we have a fusion of all
the possible movements and angles changes in
a UAV, with a bigger change of 15.38 degrees
per sec. And a 34.375% of the yaw axis of
the camera, with a maximum angle change of
6.06 degrees per sec. So we can say that the
controller has a good behavior solving those
kind of problems.
4.2.2 Tracking Moving Objects
Here we present a tracking of a van with con-
tinuous movements of the helicopter for in-
crease the difficulty of the test. In figure 11 we
can see the error in pixels of the two axis of the
image. Also, we can see the moments where
we deselected the template and re-selected it
for increase the difficulty for the controller,
those intervals are represented when the error
remain fixed in one value for a long time. In
Figure 11: Error between center of the image and
center of the dynamic object (a van) to track.
the figures 12 and 13 we can see the response
of the to controllers, where we can see the big
movements of the controller send to the ser-
vos when the warp is re-selected. We must
noticed that in all of those figures that show
the controller responses there are not regis-
ter the data when the warp selection is de-
selected, because in those movements we do
not made any track. We can see that the big-
ger response of the controllers are almost ±10
degrees for the yaw controller and almost 25
degrees for the pitch controller, being the cor-
rection in a few frames, less than 10 frames.
Figure 12: Response of the Fuzzy control for the
Yaw axis of the visual platform tracking a dynamic
object (a van).
Figure 13: Response of the Fuzzy control for the
Pitch axis of the visual platform tracking a dy-
namic object (a van).
5 Conclusions and Future Works
In this work we present an autonomous fuzzy
control of pan and tilt video platform on
board a UAV and a UAV-heading controller
using a Lucas-Kanade tracker for static and
moving objects. The controllers have a ex-
cellent behavior tracking static objects and
moving objects, besides the perturbations of
the environments and the own vibrations of
the helicopter. The use of the pan and tilt
visual platform give to the helicopter a free-
dom of movements, as well as, a faster re-
sponse with moving objects tracking, being
the others implementations of visual servoing
on UAV without pan and tilt platform more
limited and slower than the platform servos
response. With the heading controller we can
made the movements of the platform a little
bit soft. Also, this control implementation ap-
port more states to work with the helicopter
and tracking object giving us a lot of possi-
ble applications, because is possible to track
object from a hover position due to limita-
tions of the environments or the helicopter
movements, like situations of stay near to an
electric structures, fields of wind turbines or
dams. Another work state is with a path plan-
ning, giving the possibility of create a reliable
testbed for different controllers or visual algo-
rithm, as well with commands sent from the
pilot controller and the ground station.
The principal and immediate future work is to
improve the fuzzy system for a better behav-
ior and compare with other controllers. Also
now we are working for taking the up-level
control of the helicopter with other Fuzzy-
MOFS controllers that send to the UAV com-
mands of yaw and pitch to increase the pos-
sibility to follow dynamic objects for a long
time and in order to find more applications.
It is possible to view the tests videos and more
on the web www.disam.upm.es/colibri.
Acknowledgments
This work is the product of several research
stages at the Computer Vision Group Univer-
sidad Polite´cnica de Madrid. Has been spon-
sored by the Spanish Science and Technol-
ogy Ministry under grants CICYT DPI2004-
06624, CICYT DPI2000-1561-C02-02 and MI-
CYT DPI2007-66156.
References
[1] P. Campoy, J. Correa, I. Mondragon,
C. Martinez, M. Olivares, L. Mejias, and
J. Artieda. Computer vision onboard uavs
for civilian tasks. Journal of Intelligent
and Robotic Systems, 2008.
[2] V. Dobrokhodov, I. Kaminer, K. Jones,
and R. Ghabcheloo. Vision-based tracking
and motion estimation for moving targets
using small uavs. American Control Con-
ference, 2006, pages 6 pp.–, June 2006.
[3] S. Hrabar. 3d path planning and stereo-
based obstacle avoidance for rotorcraft
uavs. Intelligent Robots and Systems,
2008. IROS 2008. IEEE/RSJ Interna-
tional Conference on, pages 807–814,
Sept. 2008.
[4] I. K. Nikolos, N. C. Tsourveloudis, and
K. P. Valavanis. A uav vision system for
airborne surveillance. In Robotics and Au-
tomation, 2004. Proceedings. ICRA ’04.
2004 IEEE International Conference on,
pages 77–83, New Orleans, LA, USA, May
2004.
[5] M. Olivares, P. Campoy, J. Correa,
C. Martinez, and I. Mondragon. Fuzzy
control system navigation using priority
areas. In Proceedings of the 8th Interna-
tional FLINS Conference, pages 987–996,
Madrid,Spain, September 2008.
[6] M. Olivares and J. Madrigal. Fuzzy logic
user adaptive navigation control system
for mobile robots in unknown environ-
ments. Intelligent Signal Processing, 2007.
WISP 2007. IEEE International Sympo-
sium on, pages 1–6, Oct. 2007.
[7] A. Puri, K. Valavanis, and M. Kontit-
sis. Statistical profile generation for traf-
fic monitoring using real-time uav based
video data. In Control and Automation,
2007.MED ’07. Mediterranean Conference
on, MED, pages 1–6, 2007.
[8] H. Zou, Z. Gong, S. Xie, and W. Ding.
A pan-tilt camera control system of uav
visual tracking based on biomimetic eye.
Robotics and Biomimetics, 2006. ROBIO
’06. IEEE International Conference on,
pages 1477–1482, Dec. 2006.


Visual servoing using fuzzy controllers on an unmanned aerial vehicles

This paper presents an implementa- tion of three Fuzzy Logic controllers working in parallel onboard a UAV, two for a pan-tilt camera platform and the third for control the yaw of the helicopter. This implementation uses a Lucas-Kanade tracker algo- rithm with a pyramidal optical  ow implementation, which gives infor- mation to follow statics and moving objects, besides the UAV vibrations and movements. The platform con- troller is helped by the heading con- troller, in order to make smooth the big movements to the platform, re- ducing the risk of lost the warp selec- tion of the object to track. Also, the heading control remove the physic limit of the platform at the yaw axis. Some laboratory and UAV tests are presented in order to show the di er- ent behaviors and the good response of the presented controllers

Archivo Digital UPM

http://orbilu.uni.lu/bitstream/10993/19746/1/EUROFUSE_2009_OlivaresMendez.pdf

Visual servoing using fuzzy controllers on an unmanned aerial vehicles

Abstract

Similar works

Full text

Available Versions

Open Repository and Bibliography - Luxembourg

Servicio de Coordinación de Bibliotecas de la Universidad Politécnica de Madrid

Archivo Digital UPM