Grayscale optical-flow methods have long been the focus of methods for recovering optical flow. Optical flow recovery from color-images can be implemented using direct methods, i.e. without using computationally costly iterations or search strategies. The quality of recovered optical flow can be assessed and tailored after processing, providing an effective, efficient tool for motion estimation. In this paper, a brief introduction to optical flow is presented, the optical flow constraint equation and its extension to color images is presented. New methods for solving this extended equation are given. Results of applying these methods to two synthetic image sequences are presented

Andrews, Robert J.

Lovell, Brian C.

University of Queensland eSpace

Color Optical Flow
Robert J. Andrews and Brian C. Lovell
School of Information Technology and Electrical Engineering
Intelligent Real-Time Imaging and Sensing
University of Queensland
E-mail: s341199@student.uq.edu.au, lovell@itee.uq.edu.au
Abstract
Grayscale optical-flow methods have long been the fo-
cus of methods for recovering optical flow. Optical flow re-
covery from color-images can be implemented using direct
methods, i.e. without using computationally costly itera-
tions or search strategies. The quality of recovered optical
flow can be assessed and tailored after processing, provid-
ing an effective, efficient tool for motion estimation.
In this paper, a brief introduction to optical flow is pre-
sented, the optical flow constraint equation and its exten-
sion to color images is presented. New methods for solving
this extended equation are given. Results of applying these
methods to two synthetic image sequences are presented.
1 Introduction
Optical flow is a useful tool for many tasks in com-
puter vision. It has been applied to problems of motion-
segmentation, time-to-contact and three-dimensional recon-
struction (structure from motion) among others. Tradition-
ally, most researchers in this field have focussed their efforts
on extending Horn and Shunck [8] or Lucas and Kanade’s
[9] methods, all working with grayscale intensity images.
Color image sequences have been largely ignored, de-
spite three planes of information being available instead of
one. Golland proposed and discussed two simple methods
which incorporate color information [7]. She investigated
the RGB, normalized RGB and HSV color models. Her re-
sults indicated that color methods provide a good estimate
of the flow in image regions of non-constant color.
This paper compares traditional grayscale with Gol-
land’s methods and two new color methods. It also de-
scribes the proposed the extension of grayscale methods
into color.
2 Optical flow
The optical flow of an image sequence is a set of vector
fields, relating each image to the next. Each vector field rep-
resents the apparent displacement of each pixel from image
to image. If we assume the pixels conserve their intensity,
we arrive at the “brightness conservation equation”,
I(x, y, t) = I(x + dx, y + dy, t + dt) (2.1)
where I is an image sequence, [dx, dy] is the displacement
vector for the pixel at coordinate [x, y] and t and dt are the
frame and temporal displacement of the image sequence.
The idea of brightness conservation and optical flow were
first proposed by Fennema [6].
The obvious solution to 2.1 is to use template-based
search strategies. A template of a certain size around each
pixel is created and the best match is searched for in the
next image. The best match is usually found using corre-
lation, sum of absolute difference or sum of squared dif-
ference metrics. This process is often referred to as block-
matching. Such a search strategy is computationally costly
and generally doesn’t represent sub-pixel displacements.
Most methods presented in the last twenty years have
been gradient-based. They solve the differential form of
2.1, derived by Taylor expansion. After discarding higher
order terms, this is
∂I
∂x
u +
∂I
∂y
v +
∂I
∂t
= 0 (2.2)
Here we have two unknowns in only one equation, the prob-
lem is ill-posed and extra constraints must be imposed in
order to arrive at a solution.
The two most commonly used and earliest optical flow
recovery methods in this category are briefly outlined be-
low, Horn and Shunck’s [8] and Lucas and Kanade’s [9] op-
tical flow methods. These and other traditional methods are
outlined and quantitatively compared in Barron et.al. [4][3].
2.1 Horn and Schunck
Horn and Shunck [8] were the first to impose a global
smoothness constraint, assuming the flow to be smooth
across the image. Their minimization function,
∫ ∫
(Ixu + Iyv + It)2 + λ2(‖ ∇u ‖22 + ‖ ∇v ‖22)dxdy
(2.3)
can be expressed as a pair of Gauss-Siedel iterative equa-
tions,
un+1 = un − Ix [Ixun + Iyvn + It]
α2 + I2x + I2y
(2.4)
and
vn+1 = vn − Iy [Ixun + Iyvn + It]
α2 + I2x + I2y
(2.5)
2.2 Lucas and Kanade
Lucas and Kanade [9] put forth the assumption of con-
stant flow in a local neighborhood. Their method is gener-
ally implemented with neighborhoods of size 5 × 5 pixels
centered around the pixel whose displacement is being es-
timated. Measurements nearer the centre of the neighbor-
hood are given greater weight in the weighted-least-squares
formulation.
2.3 Other methods
Later methods generally extended of these two tradi-
tional methods. More recently, researches have been focus-
ing on using concepts of robustness to modify Lucas and
Kanade’s method [2][1]. These methods choose a function
other than the squared difference of the measurement to the
line of fit (implicit in least squares calculation) to provide
an estimate of the measurement’s contribution to the best
line. Functions are chosen so that outliers are ascribed less
weight than those points which lie close to the line of best
fit. This formulation results in a method which utilises itera-
tive numerical methods, e.g. gradient descent or successive
over-relaxation.
3. Using color images
Recovering optical flow from color images seems to have
been long overlooked by researchers in the field of image
processing and computer vision. Ohta [11] mentioned the
idea, but presented no algorithms or methods. Golland pro-
posed some methods in a thesis and a related paper [7]. She
proposed using the three color planes to infer three equa-
tions, then solving these using standard least squares tech-
niques.
∂IR
∂x
u +
∂IR
∂y
v +
∂IR
∂t
= 0
∂IG
∂x
u +
∂IG
∂y
v +
∂IG
∂t
= 0 (3.1)
∂IB
∂x
u +
∂IB
∂y
v +
∂IB
∂t
= 0
The other idea proposed by Golland was the concept of
”color conservation”. By constructing a linear system to
solve from only color components, e.g. Hue and Saturation
from the HSV color model, the illumination is allowed to
change, the assumption is now that the color, rather than
brightness is conserved.
3.1. Color Models
Three color models have been implemented and tested in
this paper. These are RGB, HSV and normalized RGB.
The RGB (Red, Green, Blue) color model decomposes
colors into their respective red, green and blue components.
Normalized RGB is calculated as
N = R + G + B, Rn =
R
N
, Gn =
G
N
, Bn =
B
N(3.2)
each color being normalized by the sum of all colors at that
point. If the color value at that point is zero the normalized
color at that point is taken as zero.
The HSV (Hue, Saturation, Value) model expresses the
intensity of the image (V) independently of the color (H,
S). Optical flow based purely on V is relying on brightness
conservation. Conversely, methods which are based on H
and S rely purely on color conservation. Methods which
combine the two incorporate both assumptions.
Similar to HSV, the YUV model decomposes the color as
a brightness (Y) and a color coordinate system (U,V). The
difference between the two is the description of the color
plane. H and S describe a vector in polar form, representing
the angular and magnitudinal components respectively. Y,
U and V, however, form an orthogonal euclidean space.
An alternative to these spaces is CIE perceptually lin-
ear color space also known as UCS (Uniform Chromaticity
Scale). This color system has the advantage of euclidean
distances in color space corresponding linearly to percep-
tion of color or intensity change.
Neither YUV, nor UCS have been implemented, though
this is the next step in analysing color optical flow.
3.2. Methods
Two obvious methods for arriving at a solution to the ex-
tended brightness conservation equation 3.2 are apparent:
• Disregarding one plane so as to solve quickly and di-
rectly, using Gaussian Elimination.
• Solving the overdetermined system as is, using either
least squares or pseudo-inverse methods.
Disregarding one of the planes arbitrarily may throw away
data that is more useful to the computation of optical flow
than those kept. However, if speed of the algorithm is of the
essence, disregarding one plane reduces memory require-
ments and computational cost. Another possibility is merg-
ing two planes and using this as the second equation in the
system. Numerical stability of the solution should be con-
sidered when constructing each system. By using the simple
method of pivoting it is possible to ensure the best possible
conditioning of the solution.
The methods of least squares and pseudo-inverse calcu-
lation are discussed in nearly all linear algebra texts.
A simple neighborhood least-squares algorithm, akin to
Lucas and Kanade’s [9], though not utilising weighting, has
also been implemented. Values in a 3 × 3 × 3 neighbor-
hood around the center pixel were incorporated into a large,
overdetermined system.
Another option for the computation of optical flow from
color images is to estimate the optical flow of each plane
using traditional grayscale techniques and then fuse these
results to recover one vector field. This fusion has been
implemented here by simply selecting the estimated vector
with the smallest intrinsic error at each point.
All of the the methods mentioned above have been im-
plemented and compared in this study.
4 Error Analysis
Image reconstruction is a standard technique for assess-
ing the accuracy of optical flow methods, especially for se-
quences with unknown ground truth (see Barron and Lin
[5]). The flow field recovered from an optical flow method
is used to warp the first image into a reconstructed image,
an approximation to the second image. If the optical flow
is accurate then the reconstructed image should be the same
as the second image in the image sequence. Generally, the
RMS error of the entire reconstructed image is taken as the
image reconstruction error. However, it is advantageous to
calculate the image reconstruction error at each point in the
image. This enables a level of thresholding in addition to,
or instead of culling estimates with high intrinsic error.
The density of the flow field after thresholding at chosen
image reconstruction errors can also be used to compare dif-
ferent methods. This is the method applied for comparison
herein.
Method 64× 64 128× 128 240× 320
Color (Least Sq) 0.66 2.11 9.24
Color (G E, 3 rows) 0.05 0.24 1.58
Color (GE, pivot, 2 rows) 0.03 0.14 0.54
H & S (20 its) 0.23 0.45 2.18
Lucas & Kanade 0.73 3.52 15.19
Nagel 1.63 6.70 36.68
Uras, et. al. 2.06 8.19 37.07
NCC 3.00 9.08 40.20
Black & Anandan 0.60 2.94 19.08
Table 1. Time taken for computation
5. Results and Discussion
Table 1 compares the time taken for recovery of opti-
cal flow using Matlab, excluding low-pass filtering and
derivative calculation times. The times recorded from com-
putation on a 700Mhz Pentium III processor. This high-
lights the drastic decrease in computational cost of direct
color methods. The two row partial pivoting Gaussian Elim-
ination method is seen to perform at approximately 20Hz.
Compared to Horn and Shunck’s method [8], the best per-
former in the field of grayscale methods, this represents an
approximately fourfold increase in speed.
Figure 5.1 compares three common grayscale optical
flow methods; Horn and Shunck [8], Lucas and Kanade
[9] and Nagel [10]. This figure illustrates the density of
the computed flow field when thresholded at chosen image
reconstruction errors. It is seen that Lucas and Kanade’s
method [9] slightly outperforms Horn and Shunck’s [8]
method, which itself performs better than Nagel’s [10]
method at image reconstruction errors >≈ 1.35.
Figure 5.2 compares the performance of Lucas and
Kanade’s [9] with three color methods. The first frame of
this image sequence is shown in figure 5.3. This sequence
was translating with velocity [-1,-1] pixels per frame. The
three color methods shown here are gaussian elimination
(with pivoting) of the saturation and value planes of HSV,
Gaussian elimination of RGB color planes and neighbor-
hood least squares. Neighborhood least squares is seen to
perform the best out of the color methods, closely approxi-
mating Lucas and Kanade at higher densities. Both gaussian
elimination versions performed worse than the others.
An image sequence displaying a one degree anticlock-
wise rotation around the center of the image was used to
assess three other color optical flow methods. Pixel dis-
placement ranges between zero and 1.5 pixels per frame.
The methods compared were “Color Constancy” [7], least
squares solution to 3.2 [7] and Combined-Horn and Shunck.
Horn and Shunck’s [8] (grayscale) algorithm was used as a
yardstick for this comparison. The results are displayed in
1 1.5 2 2.5 3 3.5 4 4.5 5
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Image Reconstruction Error
D
en
si
ty
Horn and Schunck
Lucas and Kanade
Nagel
Figure 5.1. Comparison of Grayscale meth-
ods applied to translating colored clouds
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5
0.4
0.5
0.6
0.7
0.8
0.9
1
Image Reconstruction Error
D
en
si
ty
Lucas & Kanade
Color − GE 2 rows− SV
Color − GE 3 − RGB
Color − LS(neigh) − HSV
Figure 5.2. Comparison of gray and color
methods applied to translating colored
clouds
Figure 5.3. First frame of the translating RGB
clouds sequence
1 1.5 2 2.5 3 3.5 4 4.5 5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1
Image Reconstruction Error
D
en
si
ty
Horn and Shunck
Color Constancy − 2 rows − HS
Color (LS)
Combined HornShunck
Figure 5.4. Comparison of techniques applied
to a rotating image sequence
figure 5.4. Combined-Horn and Shunck applied Horn and
Shunck optical flow recovery to each plane of the RGB im-
age and fused them into one flow field utilising a winner-
takes-all strategy based on their associated error. It can
be seen that the Combined-Horn and Shunck method per-
formed similarly to Horn and Shunck [8]. The methods of
least squares [7] and direct solution of the color constancy
equation [7] did not perform as well.
Figure 5.5 gives an example of the optical flow recov-
ered by the neighborhood least squares algorithm. This cor-
responds to the rotating image sequence. Larger vectors
(magnitude greater than 5) have been removedand replaced
with zero vectors. This field has a density of 95%.
0 10 20 30 40 50 60 70
0
10
20
30
40
50
60
70
Figure 5.5. Optical flow recovered by direct
two-row optical flow and thresholding
6 Conclusion and future work
Color optical flow has been shown to be quite simple
to compute and to have a level of accuracy similar to tradi-
tional grayscale methods. The speed of these algorithms is a
significant benefit; the linear optical flow methods presented
run substantially faster than grayscale, non-linear methods.
YUV and UCS color models will be implemented and
compared.
Accuracy of the neighborhood least squares approach
can be improved in a number of ways. Using robust meth-
ods, e.g. least-median of squares [2], could provide a much
better estimate of the correct flow. Applying the weighted
least squares approach of Lucas and Kanade [9] could like-
wise improve the results.
A better data-fusion algorithm could be used to improve
the Combined-Horn and Shunck method. The three flows
being combined could be calculated using any grayscale
method.
Methods that iterate towards a solution usually perform
better with a good initial starting estimate. Color-optical
flow could be used to provide this estimate, speeding the
computation of some of the slower, well-known grayscale
methods.
These issues will be investigated in future work.
7 Acknowledgements
The authors would like to thank David McKinnon and
Nic Geard for their support in the creation of this document
References
[1] P. Anandan and Michael J. Black, The robust estima-
tion of multiple motions: Parametric and piecewise-
smooth flow fields, Computer Vision and Image Un-
derstanding 63 (1996), no. 1, 75–104.
[2] Alireza Bab-Hadiashar and David Suter, Robust optic
flow estimation using least median of squares, Inter-
national Conference on Image Processing (1996).
[3] J. Barron, D. Fleet, S. S. Beauchemin, and T. Burkitt,
Performance of optical flow techniques, RPL-TR-
9107, Queen’s University, July 1993.
[4] J. L. Barron, D. J. Fleet, and S. S. Beauchemin, Sys-
tems and experiment performance of optical flow tech-
niques, International Journal of Computer Vision 12
(1994), 43–77.
[5] John L. Barron and T. Lin, Image reconstruction error
for optical flow, (1995).
[6] C. Fennema and W. Thompson, Velocity determi-
nation in scenes containing several moving objects,
Computer Graphics and Image Processing 9 (1979),
301–315.
[7] P. Golland and A. M. Bruckstein, Motion from color,
Tech. report, Computer Science Dept, Technion, I.I.T.,
Haifa, Israel, 1997.
[8] B. Horn and B. Shunck, Determining optical flow, Ar-
tificial Intelligence (1981), no. 17, 185–203.
[9] B. Lucas and T. Kanade, An iterative image restora-
tion technique with an application to stereo vision,
Proceedings of the DARPA IU Workshop (1981),
121–130.
[10] H. H. Nagel, Displacement vectors derived from
second-order intensity variations in image sequences,
CGIP (1983), no. 21, 85–117.
[11] N. Ohta, Optical flow detection by color images, Pro-
ceedings of IEEE International Conference on Image
Processing (1989), 801–805.


Color Optical Flow

http://espace.library.uq.edu.au/view/UQ:10958/n135.pdf

Color Optical Flow

Abstract

Similar works

Full text

Available Versions

University of Queensland eSpace