Datasets that combine knowledge graphs (KG) and text (KG-T) can be used to train forward and inverse neural models that generate text from KG and vice versa. However, models trained on data sets where KG and text pairs are not equivalent may suffer from more hallucinations and poorer recall. In this paper, we verified this empirically by generating data sets with different noise levels and found that noisier data sets indeed lead to more hallucinations. We argue that the ability of forward and inverse models trained on a data set to cyclically regenerate the KG or source text is an indicator of the equivalence between the KG and the text in the data set. Using cyclic evaluation, we found that manually created WebNLG is much better than automatically created TeKGen and T-REx. Informed by these observations, we constructed a new and improved dataset called LAGRANGE using heuristics aimed at improving the equivalence between KG and text and showing the impact of each of the heuristics on cyclic evaluation. We also constructed two synthetic data sets using large language models (LLMs) and observed that these lead to models that perform significantly well on cyclic text generation, but not as well on cyclic KG generation, probably due to the lack of a consistent base. ontology.
Constructing a Paired Knowledge Graph: Text Datasets Informed by Cyclic Evaluation
