A model of brightness variations due to illumination changes and non-rigid motion using spherical harmonics

Pixel brightness variations in an image sequence depend both on the objects ‘surface reflectance and on the motion of the camera and object. In the case of rigid shapes some proposed models have been very successful explaining the relation among these strongly coupled components. On the other hand, shapes which deform pose new challenges since the relation between pixel brightness variation with non-rigid motion is not yet clear. In this paper, we introduce a new model which describes brightness variations with two independent components represented as linear basis shapes. Lighting influence is represented in terms of Spherical Harmonics and non-rigid motion as a linear model which represents image coordinates displacement. We then propose an efficient procedure for the estimation of this image model in two distinct steps. First, shape normal’s and albedo are estimated using standard photometric stereo on a sequence with varying lighting and no deformable motion. Then, given the knowledge of the object’s shape normal’s and albedo, we efficiently compute the 2D coordinates bases by minimizing image pixel residuals over an image sequence with constant lighting and only non-rigid motion. Experiments on real tests show the effectiveness of our approach in a face modelling context