Scrutinizer: A mixed-initiative approach to large-scale, data-driven claim verification

Organizations spend signicant amounts of time and money to manually fact check text documents summarizing data. The goal of the Scrutinizer system is to reduce verication overheads by supporting human fact checkers in translating text claims into SQL queries on an database. Scrutinizer coordinates teams of human fact checkers. It reduces verification time by proposing queries or query fragments to the users. Those proposals are based on claim text classiers, that gradually improve during the verication of a large document. In addition, Scrutinizer uses tentative execution of query candidates to narrow down the set of alternatives. The verication process is controlled by a cost-based optimizer. It optimizes the interaction with users and prioritizes claim verications. For the latter, it considers expected verification overheads as well as the expected claim utility as training samples for the classiers. We evaluate the Scrutinizer system using simulations and a user study with professional fact checkers, based on actual claims and data. Our experiments consistently demonstrate signicant savings in verication time, without reducing result accuracy.