Meet LMSYS-Chat-1M: A large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs

Large Language Models (LLM) have become an integral part of various ai applications, from virtual assistants to code generation. Users adapt their behavior when interacting with LLM, using specific queries and question formats for different purposes. Studying these patterns can provide insights into user expectations and trust in various LLMs. Additionally, understanding the range of questions, from simple facts to complex queries with a lot of context, can help improve LLMs to better serve users, prevent misuse, and improve ai safety. It can be said that:

• The high operational costs associated with running large language modeling services make it financially challenging for many organizations to collect real data from user questions.
• Companies that possess substantial data sets of user questions are hesitant to share them due to concerns about revealing their competitive advantages and a desire to maintain data privacy.
• Encouraging users to interact with open language models is a challenge because these models often do not perform as well as those developed by large companies.
• This difficulty in user engagement with open models makes it difficult to compile a substantial data set that accurately reflects actual user interactions with these models for research purposes.

To address this gap, this research paper presents a new large-scale real-world dataset called LMSYS-Chat-1M. This dataset was carefully curated from an extensive collection of real-world interactions between large language models (LLMs) and users. These interactions were collected over a five-month period by hosting a free online LLM service that provided access to 25 popular LLMs, spanning both open source and proprietary models. The service incurred significant computational resources, including several thousand A100 hours.

To maintain user engagement over time, the authors implemented a competitive element known as a “chatbot arena” and incentivized users to use the service by regularly updating rankings and leaderboards for popular LLMs. Consequently, LMSYS-Chat-1M comprises more than one million user conversations, displaying a wide range of languages ​​and topics. Users provided consent for their interactions to be used for this data set through the “Terms of Use” section on the data collection website.

This data set was collected from the Vicuña demo and Chatbot Arena website between April and August 2023. The website offers users three chat interface options: a single-model chat, a chatbot arena where chatbots fight and a chatbot arena that allows users to compare two. chatbots side by side. This platform is completely free and users are not compensated or charged any fees for its use.

In this article, the authors explore potential applications of LMSYS-Chat-1M in four different use cases. They demonstrate that LMSYS-Chat-1M can effectively tune small language models to serve as powerful content moderators, achieving performance similar to GPT-4. Furthermore, despite security measures in some served models, LMSYS-Chat-1M still contains conversations that may challenge the safeguards of major language models, offering a new benchmark for studying the robustness and security of models.

Additionally, the data set includes high-quality user language model dialogs suitable for tuning instructions. Using a subset of these dialogs, the authors show that the Llama-2 models can achieve performance levels comparable to Vicuña and Llama2 Chat on specific benchmarks. Finally, LMSYS-Chat-1M’s extensive topic and task coverage makes it a valuable resource for generating new reference questions for language models.

Review the Paper and Data set. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our SubReddit of more than 30,000 ml, Facebook community of more than 40,000 people, Discord Channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

<!– ai CONTENT END 2 –>