Efficient integration of thermal technology in facial image processing through interspectral synthesis

Face biometric systems are now a reality in numerous mainstream applications including access control, banking, and forensics. Notably, face recognition systems have recently advanced and achieved striking performances due to the uprise of deep learning and the abundant, almost endless, amount of available training data. However, these systems, which are mainly deployed in the visible spectrum, are subject to fail when employed in unconstrained scenarios. Among the main challenges in visible spectrum based systems, variable or low illumination conditions have been proved to be some of their major weaknesses. A promising approach to acquire crisp images in total darkness is to use thermal imagery. Thermal imaging technology has significantly evolved during the last couple of decades, mostly thanks to thermal cameras having become more affordable and user friendly. However, and given that the exploration of thermal imagery is reasonably new, only a few public databases are available to the research community. This limitation consequently prevents the impact of deep learning technologies from generating improved and reliable face recognition systems that operate in the thermal spectrum. A possible solution relates to the development of technologies that bridge the gap between the visible and thermal spectrum. In attempting to respond to this necessity, the research presented in this dissertation aims to explore interspectral synthesis as a direction for efficient and prompt integration of thermal technology in already deployed face biometric systems. As a first contribution, a new database, containing paired visible and thermal face images, which was acquired with a dual camera that allowed for the simultaneous capture of face images in both spectra, was collected and made publicly available to foster research in thermal face image processing. Motivated by the need for fast and straightforward integration into existing face recognition systems, a following contribution consisted in proposing a cross-spectrum face recognition framework based on a novel approach of thermal-to-visible face synthesis in order to estimate the visible face from the thermal input, when the visible image cannot be provided, e.g. in poorly lit environments. The proposed approach is based on deep generative models and was trained on a set of paired visible and thermal data to learn a mapping from the thermal face to its visible equivalent. After this initial work, another contribution presents the development of i Abstract an illumination invariant face recognition system that incorporates a novel, dynamic quality-weighted fusion of visible and thermal spectrum at the score level. Thanks to the proposed mechanism, uninterrupted and efficient functioning of a face recognition setup during day and night time may be ensured. Motivated by the favorable results achieved in the first part of our research work, additional contributions presented in this thesis explore the process of interspectral synthesis in the reverse direction, i.e. from visible to thermal spectrum. Visible-tothermal image synthesis was employed to address the shortage of annotated public face databases in the thermal spectrum, which limits the development of fundamental task in thermal face image processing. With the scope of this study being focused on the facial landmark detection task, fully annotated synthesized thermal face databases were obtained by transforming public annotated visible face databases into thermal spectrum. Facial landmark detectors trained on the synthesized thermal face databases led to significant improvements in landmark detection accuracy. A final contribution explored visible-to-thermal synthesis to study the impact of spoofing attacks on thermal face biometric systems. The robustness of thermal-based systems lies in the acquisition process itself as it provides proof of liveness by detecting the heat emitted by the face. A new thermal attack, at the post-sensor level, is then proposed. Thermal face images, that are obtained by visible-to-thermal face synthesis, are directly injected into the communication channel after the sensor. In order to increase the difficulty of the proposed setup, a scenario where the attacker has a priori knowledge about the spoofing countermeasure employed by the system is also considered. Such a priori knowledge is exploited in order to synthesize more threatening attacks for a given countermeasure technique. The evaluation of spoofing detection systems when facing the proposed attack highlights the vulnerability of thermal face recognition systems to the proposed indirect attack.