Media Hyperlinking

Inspired by the ubiquitous text-based hyperlinks and the way that text hyperlinking has transformed how people navigate through textual and other information, its analogous for multimodal content, i.e. video hyperlinking, is emerging as a promising approach to making audio-visual documents more easily reachable, discoverable and consumable. The Hyperlinking paradigm consists in providing enrichments to a document or part thereof  based on its content in a transparent manner. Users are proposed additional relevant content (the target content) based on what they are currently consuming (the seed content or anchor), which contribute vastly to improving user experience.

Content Analysis for Hyperlinking

Work on hyperlinking focuses on two equally important research directions: Content analysis that is needed for understanding/modelling the documents to hyperlink; and, Link creation which the results of the analysis to effectively discover and establish meaningful links between related content.

Hyperlinking and in particular video to video linking brings new challenges in automatically processing the audio-visual content and understanding the information it conveys at different granularities, in processing associated audio and textual information, and in intelligently exploiting all these analysis results for creating meaningful video hyperlinks. It also raises important questions concerning the granularities that are most appropriate for decomposing and linking the video content. These are some of the research questions we are currently investigating.

Realistic evaluation of hyperlinking approaches is among our concerns. We are active participants and among the organizing team of the TRECVid Hyperlinking task (formerly organized within MediaEval) where international research teams propose and compare theirs latest and most advanced scientific results. 


HyperTED: exploring multimedia collection at the fragment level

HyperTED offers a unique and innovative way for exploring over 1681 TED talks at the fragment level, opening a window to a more effective and efficient dissemination of the knowledge present in those inspirational conferences. In this application, the concept of complete video talk as a first class citizen is further refined through the notion of chapters. Similar to paragraphs in a textual document, chapters delimit particular ideas that sequentially illustrate the context of the video. In practice, viewers generally fast browse a video without necessarily watch each particular chapter. HyperTED tackles this problem by proposing a set of automatically annotated media fragments called Hot Spots which intend to highlight the main concepts and topics discussed in a video talk.

A dedicated interface eases the exploration and sharing of those Hot Spots. Hot Spots are detected based on named entities, concepts and topics extracted from the talk subtitles. Furthermore, those Hot Spots are ranked and used as anchors for finding related educational resources where the user can learn more about what the video is talking about, such as courses and online tutorials.

TED talks touch many different topics but they also form a closed collection of high quality video material that would benefit from being better interlinked at the fragment level, going way beyond the current top 3 talks that TED is recommending watching at the end of a talk. HyperTED offers this functionality and recommends links between fragments of TED talks based on the Hot Spots which have been detected.

HyperTED is already not limited to TED talks and it can also ingest videos from YouTube, Dailymotion or Vimeo, as soon as subtitles are available. The system is available at


Syndicate content

Data Science