Graduate School and Research Center in Digital Sciences

Neighbourhood structure preserving cross-modal embedding for video hyperlinking

Hao, Yanbin; Ngo, Chong-Wah; Huet, Benoit

IEEE Transactions on Multimedia, June 2019

Video hyperlinking is a task aiming to enhance the accessibility of large archives, by establishing links between fragments of videos. The links model the aboutness between fragments for efficient traversal of video content. This paper addresses the problem of link construction from the perspective of cross-modal embedding. To this end, a generalized multi-modal auto-encoder is proposed. The encoder learns two embeddings from visual and speech modalities respectively, while each of the embeddings performs self-modal and cross-modal translation of modalities. Furthermore, to preserve the neighbourhood structure of fragments, which is important for video hyperlinking, the auto-encoder is devised to model data distribution of fragments in a dataset. Experiments are conducted on Blip10000 dataset using the anchor fragments provided by TRECVid Video Hyperlinking (LNK) task over the years of 2016 and 2017. This paper shares the empirical insights on a number of issues in cross-modal learning, including the preservation of neighbourhood structure in embedding, model fine-tuning and issue of missing modality, for video hyperlinking.

Doi Bibtex

Title:Neighbourhood structure preserving cross-modal embedding for video hyperlinking
Department:Data Science
Eurecom ref:5914
Copyright: © 2019 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Bibtex: @article{EURECOM+5914, doi = {}, year = {2019}, month = {06}, title = {{N}eighbourhood structure preserving cross-modal embedding for video hyperlinking}, author = {{H}ao, {Y}anbin and {N}go, {C}hong-{W}ah and {H}uet, {B}enoit}, journal = {{IEEE} {T}ransactions on {M}ultimedia, {J}une 2019}, url = {} }
See also: