A comprehensive framework for evaluating deepfake generators: Dataset, metrics performance, and comparative analysis

Husseini, Sahar; Dugelay, Jean-Luc
ICCV 2023, IEEE International Conference on Computer Vision, DFAD2023 Workshop and Challenge on DeepFake Analysis and Detection, 2-3 October 2023, Paris, France

Assessing the realism and accuracy of deepfake generators, especially in cross-reenactment situations, is a major challenge. This challenge is primarily attributed to
the absence of ground-truth data, which restricts the application of metrics that rely on explicit ground-truth, such as SSIM and LPIPS. To overcome this challenge, this paper
introduces a novel protocol for quantitatively assessing images generated by face-reenactment techniques. To address the scarcity of suitable datasets, two video datasets
are generated: the Real Head and the synthesized Metahuman datasets. Furthermore, user studies are conducted to evaluate the efficacy of our proposed protocol. The results
demonstrate a strong correlation between subjective evaluations and quantitative metrics obtained within our protocol. Comparative analysis with existing evaluation protocols
further validates the effectiveness of our proposed approach. Notably, our protocol exhibits superior performance in analyzing identity preservation, head pose, and
facial expression replication. The source code and datasets are made publicly available at https://github.com/SaharHusseini/deepfake_evaluation.git

Digital Security
Eurecom Ref:
© 2023 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

PERMALINK : https://www.eurecom.fr/publication/7425