Facial motion analysis on monocular images for telecom applications : coupling expression and pose understanding

Facial animation has become an active research topic in telecommunications. This field aims at replacing traditional communication systems by more human oriented solutions based on virtual reality technology. This thesis relates to a complete analysis/synthesis framework for facial rigid and non-rigid motion analysis from monocular video sequences. The obtained motion data are suitable to animate the realistic head clone of the analyzed speaker by generating face animation parameters. The core of the system is the rigid-motion tracking algorithm, which is able to provide the head pose parameters. The Kalman filter being used predicts the translation and rotation parameters, which are applied on the synthetic clone of the user. This process enables us to benefit from the synthetically generated virtual image by providing visual feedback for the analysis. This work exposes in detail novel techniques to study non-rigid facial motion coupled with head pose tracking. Specific feature analysis methods have been developed to study each one of the features that we believe are the most relevant while communicating: eye, eyebrows and mouth. We have designed image-processing algorithms based on the physiognomy of the speaker and individual motion models that exploit the correlation existing among the analyzed features. The analysis techniques have been first developed for faces being analyzed from a frontal point of view and then, using the pose parameters derived from the tracking and the 3D data of the clone, they have been adapted to allow the speaker more freedom of movement in front of the camera. This adaptation is possible by redefining the 2D analysis models over the clone (3D head model), in 3D, and reinterpreting the analyzed data in accordance with the 3D location of the head. This dissertation contains experimental results, main contributions and relevant bibliographic references of the overall research.