color:#002060;mso-ansi-language:EN-US">Artificial Intelligence (AI) based analysis of multimodal data collected using inexpensive and accessible wearable sensors is emerging as a promising opportunity to democratise access to healthcare. It would facilitate the prevention of various health problems and reduce the need for expensive clinical examinations that are difficult to access for a large portion of the population. However, concerns remain about the reliability and robustness of AI algorithms that are frequently overlooked in healthcare research. Yet, these aspects are crucial to the deployment of AI in medical applications. On one hand, most existing algorithms are trained on data that is hardly representative of the real world, and on the other, their architectures make them vulnerable to various perturbations commonly encountered in real data, once deployed. The aim of this PhD project is to develop innovative, robust and reliable methodologies for the analysis of wearable sensor data – with a particular focus on robustness to missing data. Our aim is to design novel multimodal methodologies that are evaluated on real sensor data and designed to be transposed and adapted to various medical applications – ranging from physiological signals analysis to the analysis of multimodal imaging data. We propose 5 contributions to achieve this goal.
mso-ansi-language:EN-US">(1) We introduce StressID, a new dataset specifically designed for stress identification from unimodal and multimodal data, that we made publicly available for researchers. It contains videos, audio recordings, and physiological signals collected in ambulatory settings using wearable sensors. As it is collected from 65 participants, it includes a wide range of participant’s responses. As such, it is a valuable support for building reliable and robust applications for stress identification. (2) We propose an open-source suite of baseline models for the analysis of StressID, that is representative of the current state-of-the-art in the domain, and facilitates future contributions to this domain by providing a starting point for researchers who wish to use the dataset. We investigate the next steps needed to ensure reliability and robustness of existing models, and identify robustness to missing data as an essential aspect to exploiting the benefits of real-life multimodal datasets. (3) We explore whether the rich existing literature on missing values in tabular data can be leveraged to address this limitation. We conduct a comprehensive evaluation of existing methods for dealing with missing data, and assess their reliability within healthcare applications. This enables us to identify the strengths and limitations of existing approaches, to ultimately derive a set of guidelines to properly and responsibly handle missing values in healthcare applications. (4) Based on the considerations thus identified, we propose PicMi, an end-to-end imputation-free model designed for supervised learning with missing values in tabular data, that uses a permutation-invariant architecture to handle inputs of varying dimensions; integrates missing value patterns as a condition in its objective function to ensure robustness to various missing values scenarios; and is locally interpretable. (5) We extend our approach to multimodal learning with missing modalities, and introduce HyperMM, a framework designed for handling missing modalities without using reconstruction before training – as opposed to existing solutions. We introduce a novel strategy for training a universal feature extractor using a conditional hypernetwork, and propose a permutation invariant neural network that can handle inputs of varying dimensions to process the extracted features, in a two-phase task-agnostic framework. Our method is end-to-end and can be used in various applications, and thus contributes to the development of more reliable and robust AI systems in healthcare.
