Graduate School and Research Center in Digital Sciences

GERBIL : General entity annotator benchmarking framework

Usbeck, Ricardo; Röder, Michael; Ngonga Ngomo, Axel-Cyrille; Baron, Ciro; Both, Andreas; Brümmer, Martin; Ceccarelli, Diego; Cornolti, Marco; Cherix, Didier; Eickmann, Bernd; Ferragina, Paolo; Lemke, Christiane; Moro, Andrea; Navigli, Roberto; Piccinno, Francesco; Rizzo, Giuseppe; Sack, Harald; Speck, René; Troncy, Raphaël; Waitclonis, Jörg; Wesemann, Lars

WWW 2015, 24th International World Wide Web Conference, May 18-22, 2015, Florence, Italy

The need to bridge between the unstructured data on the Document Web and the structured data on the Web of Data has led to the development of a considerable number of annotation tools. However, these tools are currently still hard to compare since the published evaluation results are calculated on diverse datasets and evaluated based on di erent measures. We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile,  ne-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the ecient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further re ned, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

Document Doi Bibtex

Title:GERBIL : General entity annotator benchmarking framework
Keywords:Semantic Entity Annotation System, Reusability, Archiv- ability, Benchmarking Framework
Department:Data Science
Eurecom ref:4520
Copyright: © ACM, 2015. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in WWW 2015, 24th International World Wide Web Conference, May 18-22, 2015, Florence, Italy
Bibtex: @inproceedings{EURECOM+4520, doi = {}, year = {2015}, title = {{GERBIL} : {G}eneral entity annotator benchmarking framework}, author = {{U}sbeck, {R}icardo and {R}{\"o}der, {M}ichael and {N}gonga {N}gomo, {A}xel-{C}yrille and {B}aron, {C}iro and {B}oth, {A}ndreas and {B}r{\"u}mmer, {M}artin and {C}eccarelli, {D}iego and {C}ornolti, {M}arco and {C}herix, {D}idier and {E}ickmann, {B}ernd and {F}erragina, {P}aolo and {L}emke, {C}hristiane and {M}oro, {A}ndrea and {N}avigli, {R}oberto and {P}iccinno, {F}rancesco and {R}izzo, {G}iuseppe and {S}ack, {H}arald and {S}peck, {R}en{\'e} and {T}roncy, {R}apha{\"e}l and {W}aitclonis, {J}{\"o}rg and {W}esemann, {L}ars}, booktitle = {{WWW} 2015, 24th {I}nternational {W}orld {W}ide {W}eb {C}onference, {M}ay 18-22, 2015, {F}lorence, {I}taly }, address = {{F}lorence, {ITALY}}, month = {05}, url = {} }
See also: