A 3D-assisted framework to evaluate the quality of head motion replication by reenactment deepfake generators

Husseini, Sahar; Dugelay, Jean-Luc; Aili, Fabien; Nars, Emmanuel
ICASSP 2023, IEEE International Conference on Acoustics, Speech and Signal Processing, 4-10 June 2023, Rhodes Islands, Greece

In recent years we have assisted the proliferation of deepfakes. The progress concerning both creation and, to a certain extent, automatic detection is spectacular. Nevertheless, there is a lack of protocols concerning the objective evaluation of deepfakes. In this article, we focus on the quality of head motion replication by deepfake generators that use a pilot video of a particular person to animate a single source image of another person. We test several publicly available generators to reproduce particular head movements (rotation around yaw, pitch, and a combination of pitch and yaw). In order to measure how well the deepfake generators replicate head motion, a 3D head model is utilized to render video sequences with the known head pose. Then the generated head movements by deepfake are compared to an exact 3D simulation that can be used as ground-truth. Several measures, such as SSIM and average facial keypoint distance, are used to quantify results.

Rhodes Island
Sécurité numérique
Eurecom Ref:
© 2023 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

PERMALINK : https://www.eurecom.fr/publication/7317