Graduate School and Research Center in Digital Sciences

Scrutinizer: A mixed-initiative approach to large-scale, data-driven claim verification

Karagiannis, Georgios; Saeed, Mohammed; Papotti, Paolo; Trummer, Immanuel

Submitted on ArXiV, 14 March 2020

Organizations such as the International Energy Agency (IEA) spend significant amounts of time and money to manually fact check text documents summarizing data. The goal of the Scrutinizer system is to reduce verification overheads by supporting human fact checkers in translating text claims into SQL queries on an associated database. Scrutinizer coordinates teams of human fact checkers. It reduces verification time by proposing queries or query fragments to the users. Those proposals are based on claim text classifiers, that gradually improve during the verification of a large document. In addition, Scrutinizer uses tentative execution of query candidates to narrow down the set of alternatives. The verification process is controlled by a cost-based optimizer. It optimizes the interaction with users and prioritizes claim verifications. For the latter, it considers expected verification overheads as well as the expected claim utility as training samples for the classifiers. We evaluate the Scrutinizer system using simulations and a user study, based on actual claims and data and using professional fact checkers employed by IEA. Our experiments consistently demonstrate significant savings in verification time, without reducing result accuracy.

Arxiv Bibtex

Title:Scrutinizer: A mixed-initiative approach to large-scale, data-driven claim verification
Type:Conference
Language:English
City:
Date:
Department:Data Science
Eurecom ref:6216
Copyright: © EURECOM. Personal use of this material is permitted. The definitive version of this paper was published in Submitted on ArXiV, 14 March 2020 and is available at :
Bibtex: @inproceedings{EURECOM+6216, year = {2020}, title = {{S}crutinizer: {A} mixed-initiative approach to large-scale, data-driven claim verification}, author = {{K}aragiannis, {G}eorgios and {S}aeed, {M}ohammed and {P}apotti, {P}aolo and {T}rummer, {I}mmanuel}, booktitle = {{S}ubmitted on {A}r{X}i{V}, 14 {M}arch 2020}, address = {}, month = {03}, url = {http://www.eurecom.fr/publication/6216} }
See also: