VERT : a method for automatic evaluation of video summaries

Li, Yingbo; Merialdo, Bernard
Research Report RR-10-237




Video Summarization has become an important tool for Multimedia

Information processing, but the automatic evaluation of a video

summarization system remains a challenge. A major issue is that an

ideal "best" summary does not exist, although people can easily

distinguish "good" from "bad" summaries. A similar situation arise

in machine translation and text summarization, where specific

automatic procedures, respectively BLEU and ROUGE, evaluate the

quality of a candidate by comparing its local similarities with several

human-generated references. These procedures are now routinely

used in various benchmarks. In this paper, we extend this idea to the

video domain and propose the VERT (Video Evaluation by Relevant

Threshold) algorithm to automatically evaluate the quality video

summaries. VERT mimics the theories of BLEU and ROUGE, and

counts the weighted number of overlapping selected units between

the computer-generated video summary and several human-made

references. Several variants of VERT are suggested and compared,

and the best variant is selected through experimentation.

Data Science
Eurecom Ref:
© EURECOM. Personal use of this material is permitted. The definitive version of this paper was published in Research Report RR-10-237 and is available at :