Modeling affective evaluation of multimedia contents: user models to associate subjective experience, physiological expression and contents description

Ten years after the foundation book "Affective Computing" which proposed concepts and approaches related to the measure and the use of emotion in human computer interaction scenarios, several decades after the first psychological and physiological study on emotion and hundreds years after philosophical study about aesthetical emotion, very few automated systems are nowadays able to manipulate multimedia contents (selection, design, creation) according to the felt affective states and emotions (which can be measured by different means) by an individual. Among several limitations, we consider in this thesis that one important problem is the amount of inter-individual differences both in the indirect measure of emotion and in our affective evaluation of multimedia contents. In this thesis we propose enhancements of computer possibilities to manipulate media contents on the basis of felt affective states by modeling affective and emotional associations to multimedia contents and by automating the process of interpretation of emotion from indirect measure. This enhancement is achieved by the design of two user models which can take into account user's specificities toward a better adaptation. First we introduce the Embodied Affective Relationship to Multimedia Contents (EAR) as a model aiming at formalizing association between multimedia contents and emotional experience of individuals. This model is then presented in a practical way toward systematic affective handling of multimedia contents. Then we introduce the Psycho Physiological Emotion Map (PPEM) as a parametric model of emotion interpretation from physiological signals taking into account inter and intra-individual differences. Our technique aims at psychologically interpreting physiological parameters (skin conductance and heart rate), and at producing a continuous extraction of the user's affective state during Human Computer Interaction. An experiment is presented to estimate emotion from physiological signals. Both models are built upon an engineering cognitive science approach, i.e. implementing psychological and neuroscience knowledge to design a computer system. Finally, an experimental Application Programming Interface built upon the proposed models is presented to enable novel form of affective state and emotion driven-human multimedia interaction.