Graduate School and Research Center in Digital Sciences

A benchmark for fact checking algorithms built on knowledge bases

Huynh, Viet-Phi; Papotti, Paolo

CIKM 2019, 28th ACM International Conference on Information and Knowledge Management, November 3rd-7th, 2019, Beijing, China

Fact checking is the task of determining if a given claim holds. Several algorithms have been developed to check claims with reference information in the form of facts in a knowledge base. While individual algorithms have been experimentally evaluated in the past, we provide the first comprehensive and publicly available benchmark infrastructure for evaluating methods across a wide range of assumptions about the claims and the reference information. We show how, by changing the popularity, transparency, homogeneity, and functionality properties of the facts in an experiment, it is possible to influence significantly the performance of the fact checking algorithms. We introduce a benchmark framework to systematically enforce such properties in training and testing datasets with fine tune control over their properties. We then use our benchmark to compare fact checking algorithms with one another, as well as with methods that can solve the link prediction task in knowledge bases. Our evaluation shows the impact of the four data properties on the qualitative performance of the fact checking solutions and reveals a number of new insights concerning their applicability and performance.

Document Doi Bibtex

Title:A benchmark for fact checking algorithms built on knowledge bases
Type:Conference
Language:English
City:Beijing
Country:CHINA
Date:
Department:Data Science
Eurecom ref:5996
Copyright: © ACM, 2019. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in CIKM 2019, 28th ACM International Conference on Information and Knowledge Management, November 3rd-7th, 2019, Beijing, China http://dx.doi.org/10.1145/3357384.3358036
Bibtex: @inproceedings{EURECOM+5996, doi = {http://dx.doi.org/10.1145/3357384.3358036}, year = {2019}, title = {{A} benchmark for fact checking algorithms built on knowledge bases}, author = {{H}uynh, {V}iet-{P}hi and {P}apotti, {P}aolo}, booktitle = {{CIKM} 2019, 28th {ACM} {I}nternational {C}onference on {I}nformation and {K}nowledge {M}anagement, {N}ovember 3rd-7th, 2019, {B}eijing, {C}hina}, address = {{B}eijing, {CHINA}}, month = {11}, url = {http://www.eurecom.fr/publication/5996} }
See also: