A 3D-assisted framework to evaluate the quality of head motion replication by reenactment deepfake generators

In recent years we have assisted the proliferation of deepfakes. The progress concerning both creation and, to a certain extent, automatic detection is spectacular. Nevertheless, there is a lack of protocols concerning the objective evaluation of deepfakes. In this article, we focus on the quality of head motion replication by deepfake generators that use a pilot video of a particular person to animate a single source image of another person. We test several publicly available generators to reproduce particular head movements (rotation around yaw, pitch, and a combination of pitch and yaw). In order to measure how well the deepfake generators replicate head motion, a 3D head model is utilized to render video sequences with the known head pose. Then the generated head movements by deepfake are compared to an exact 3D simulation that can be used as ground-truth. Several measures, such as SSIM and average facial keypoint distance, are used to quantify results.