FinTextQA – A Long Question Answering LFQA Dataset Designed Specifically for Finance

The expansion of question answering (QA) systems powered by artificial intelligence (ai) is a result of the growing demand for financial data analysis and management. In addition to improving customer service, these technologies assist in risk management and provide individualized action suggestions. Accurate and useful responses to financial data require a deep understanding of the financial domain due to the complexity of data, domain-specific terminology and concepts, market uncertainty, and decision-making processes. Due to the complex tasks involved such as information retrieval, summarization, data analysis, comprehension and reasoning, long form question answering (LFQA) scenarios have added importance in this environment.

While there are several LFQA datasets available in the public domain, such as ELI5, WikiHowQA, and WebCPM, none of them are designed for the financial sector. This gap in the market is significant, as complex, open-domain questions often require paragraph-long answers and retrievals of relevant documents. Current financial quality assurance standards, which rely heavily on number crunching and sentiment analysis, often struggle to handle the diversity and complexity of these questions.

In light of these difficulties, researchers from HSBC Lab, Hong Kong University of Science and technology (Guangzhou) and Harvard University present FinTextQA, a new dataset for testing quality control models in finance-related questions. , regulation or general policies. This data set is comprised of LFQAs taken from textbooks on the field as well as government agency websites. The 1,262 question-answer pairs and document contexts that make up FinTextQA are of excellent quality and have their source attributed. Selected from five rounds of human selection, it includes six question categories with an average text length of 19.7 thousand words. By incorporating financial rules and regulations into LFQA, this dataset challenges models with more complex content and represents innovative work in the field.

The team presented the dataset and compared state-of-the-art (SOTA) models using FinTextQA to set standards for future studies. Many existing LFQA systems rely on pre-trained and fine-tuned language models, such as GPT-3.5-turbo, LLaMA2, Baichuan2, etc. However, these models are not always prepared to answer complex financial queries or provide comprehensive answers. . They end up using the RAG framework as an answer. The RAG system can improve the performance and explanation capabilities of LLMs by preprocessing documents in multiple steps and providing them with the most relevant information.

The researchers highlight that FinTextQA has fewer QA peers despite its professional curation and high quality, in contrast to larger ai-generated data sets. Due to this restriction, models trained on it may not be able to extend to more general real-world scenarios. Acquiring high-quality data is difficult, and copyright restrictions often prevent it from being shared. Consequently, state-of-the-art approaches to data sparsity and data augmentation should be the focus of future studies. It may also be useful to investigate more sophisticated RAG capabilities and recovery methods and expand the data set to include more diverse sources.

However, the team believes that this work represents an important step forward in improving understanding and support of the financial concept by introducing the first LFQA financial data set and conducting extensive benchmarking against it. FinTextQA provides a robust and comprehensive framework for developing and testing LFQA systems in general finance. In addition to demonstrating the effectiveness of different model configurations, experimental research emphasizes the importance of improving existing approaches to make financial question answering systems more accurate and easier to understand.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our 42k+ ML SubReddit