Rethinking voice privacy: Evaluating and mitigating residual identity leakage in voice anonymisation

While speech technologies have brought tremendous new opportunities, they have also highlighted the sensitivity of speech data: beyond the linguistic content they convey, they constitute a biometric footprint that may reveal a speaker’s identity as well as a range of personal attributes, such as age, emotional state, or accent. In this context, speech data anonymisation represents a major challenge for protecting individuals’ privacy and meeting the requirements of the General Data Protection Regulation (GDPR). In a broad sense, speech anonymisation must account for several sources of personal information. It may therefore require the transformation of acoustic characteristics that would otherwise enable speaker recognition, while also removing or modifying personally identifying information contained in the linguistic content, such as a name or phone number. This second problem, referred to as linguistic content de-identification, lies outside the scope of this thesis. Our work focuses on voice anonymisation, as defined within the VoicePrivacy Challenge (VPC): the transformation of a speaker’s voice to protect their identity while preserving, as much as possible, the linguistic content and other information required to support downstream applications. This work is built around a simple question: do anonymisation systems truly provide the level of protection suggested by their evaluation? Existing approaches have historically focused largely on transforming the timbral voice characteristics, which are typically exploited by automatic speaker verification systems. Yet voice identity is not limited to timbre. Other dimensions, referred to as non-timbral cues, such as rhythm, intonation, prosody, speaking style, and accent, may also carry discriminative speaker information and persist after anonymisation.  We first sought to reassess the performance of anonymisation systems by developing attacks systems specifically adapted to exploit such non-timbral information. Our results show that evaluation protocols relying solely on conventional speaker verification systems may underestimate the amount of residual information available after anonymisation and, consequently, overestimate the level of protection achieved. We then focus more specifically on accent, a relatively stable non-timbral speech component. The objective is to determine to what extent the voice identity can be transformed if original accent cues remain. We show that accent information can persist after anonymisation and that its preservation depends both on the anonymisation system and on the accent group considered, revealing differences in protection across groups of speakers. We also evaluate a mitigation strategy based on textual conditioning to reduce the amount of accent-related information retained after anonymisation. Finally, we propose an anonymisation system that jointly targets timbral and non-timbral dimensions of voice. Our approach introduces an explicit modification of speech rhythm through the transformation of phoneme durations, with the aim of reducing speaker-specific information carried by temporal structure. This approach offers more comprehensive anonymisation by considering the voice identity not only through timbre, but also as a multidimensional set of voice characteristics, thereby enhancing privacy protection.