Integrating facial makeup detection into multimodal biometric user verification system

Multimodal biometric fusion is generally used for increasing the verification accuracy by combining two or more biometric traits. Fusion systems with predefined constant weight values for each biometry becomes much popular. Among biometrics, face modality is one of the most common traits that is used in such fusion system. However, face verification suffers from many challenging difficulties, one of which is facial makeup. Recently, it has been shown that the accuracy of face verification can be impacted by the presence of facial makeup. And as such, the verification result of a multimodal fusion system with constant weight value for each biometry can be degraded by facial cosmetics. In this work, we propose a method of integrating facial makeup detection into the fusion system to increase performance. In our investigated scenario, score level fusion of face, fingerprint and iris verification are performed, while the weight value of each trait changes dynamically according to the level of makeup classification of test facial image. So far, this is the first work taking into account the facial makeup within a multimodal biometric verification system. Experiments on 1600 different subjects reveal that our proposed method can help in increasing the overall performance of fusion system than without using the facial makeup information.