*=Equal taxpayers
Controversy is a reflection of our zeitgeist and an important aspect of any discourse. The rise of large language models (LLMs) as conversational systems has increased the public’s trust in these systems to get answers to their various questions. Consequently, it is crucial to systematically examine how these models answer questions related to ongoing debates. However, there are few such datasets that provide human-annotated labels that reflect contemporary discussions. To encourage research in this area, we propose a novel construction of a controversial question dataset, expanding the publicly released Quora question pair dataset. This data set presents challenges related to currency of knowledge, security, fairness, and bias. We evaluate different LLMs using a subset of this data set, illuminating how they handle controversial topics and the stances they take. Ultimately, this research contributes to our understanding of LLMs’ interaction with controversial issues, paving the way for improving their understanding and handling of complex social debates.
