This article was accepted into the Workshops on Data Science with Human in the Loop at EMNLP 2022
Identifying and integrating missing facts is a crucial task in completing the knowledge graph to ensure robustness towards downstream applications, such as answering questions. Adding new facts to a knowledge graph in a real-world system often involves a human verification effort, where human annotators check candidate facts for accuracy. This process is laborious, time consuming and inefficient as only a small number of missing facts can be identified. This paper proposes a simple but effective human-in-loop framework for fact gathering that searches for a diverse set of highly relevant candidate facts for human annotation. The empirical results presented in this paper demonstrate that the proposed solution leads to improvements both in i) the quality of candidate facts and ii) the ability to discover more facts to grow the knowledge graph without requiring additional human effort.
