UC Berkeley researchers propose DocETL: a declarative system that optimizes complex document processing tasks using LLM

Large language models (LLMs) have gained significant attention in data management, with applications spanning data integration, database tuning, query optimization, and data cleansing. However, analysis of unstructured data, especially complex documents, remains a challenge in data processing. Recent declarative frameworks designed for LLM-based unstructured data processing focus more on reducing costs than improving accuracy. This creates problems for complex tasks and data, where LLM results often lack precision in user-defined operations, even with refined prompts. For example, LLMs may have difficulty identifying each occurrence of specific clauses, such as force majeure or indemnity, in lengthy legal documents, making it necessary to decompose both data and tasks.

For Police Misconduct Identification (PMI), journalists at Berkeley's Investigative Reporting Program want to analyze a large corpus of police records obtained through records requests to uncover patterns of officer misconduct and potential procedural violations. PMI poses the challenge of analyzing complex document sets, such as police records, to identify patterns of officer misconduct. This task involves processing heterogeneous documents to extract and summarize key information, compile data across multiple documents, and create detailed behavioral summaries. Current approaches handle these tasks as single-pass map operations, with one LLM call per document. However, this method often lacks accuracy due to problems such as the length of the document exceeding the limits of the LLM context, missing critical details, or including irrelevant information.

Researchers at UC Berkeley and Columbia University have proposed DocETL, an innovative system designed to optimize complex document processing processes while addressing the limitations of LLMs. This method provides a declarative interface for users to define processing pipelines and uses an agent-based framework for automatic optimization. Key features of DocETL include logical rewriting of processes tailored to LLM-based tasks, an agent-guided plan evaluation mechanism that creates and manages task-specific validation messages, and an optimization algorithm that efficiently identifies promising plans within of time constraints based on LLM. Furthermore, DocETL shows significant improvements in the quality of results in several unstructured document analysis tasks.

DocETL is evaluated on PMI tasks using a dataset of 227 documents from California police departments. The dataset presented significant challenges, including long documents with an average of 12,500 tokens, and some exceeding the context window limit of 128,000 tokens. The task involves generating detailed misconduct summaries for each officer, including names, types of misconduct, and complete summaries. The initial process in DocETL consists of a map operation to extract officers exhibiting misconduct, a teardown operation to flatten the list, and a narrowing operation to summarize the misconduct across all documents. The system evaluated multiple pipeline variants using GPT-4o-mini, demonstrating DocETL's ability to optimize complex document processing tasks. The pipelines are DocETL._YesDocETL_tand DocETL_oh.

Human evaluation is carried out on a subset of data using GPT-4o-mini as a judge on 1500 results to validate the LLM judgments, revealing high agreement (92-97%) between the LLM judge and the evaluator. human. The results show that DocETL𝑂 is 1.34 times more accurate than the baseline. DocETL_Yes and DocETL_t Pipelines behaved similarly, with DDocETL._Yes often omitting dates and places. The evaluation highlights the complexity of evaluating LLM-based processes and the importance of optimization and evaluation of specific tasks in LLM-driven document analysis. DocETL's custom validation agents are crucial to finding the relative strengths of each plan and highlighting the system's effectiveness in handling complex document processing tasks.

In conclusion, the researchers introduced DocETL, a declarative system to optimize complex document processing tasks using LLM, addressing critical limitations in existing LLM-based data processing frameworks. It uses innovative rewrite policies, an agent-based framework for rewriting and plan evaluation, and an opportunistic optimization strategy to address the specific challenges of complex document processing. Additionally, DocETL can produce results that are 1.34 to 4.6 times higher quality than manually designed baselines. As LLM technology continues to evolve and new challenges arise in document processing, DocETL's flexible architecture offers a robust platform for future research and applications in this rapidly growing field.

look at the Paper and GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter and join our Telegram channel and LinkedIn Grabove. If you like our work, you will love our information sheet.. Don't forget to join our SubReddit over 50,000ml.

(Next live webinar: October 29, 2024) Best platform to deliver optimized models: Predibase inference engine (promoted)

Sajjad Ansari is a final year student of IIT Kharagpur. As a technology enthusiast, he delves into the practical applications of ai with a focus on understanding the impact of ai technologies and their real-world implications. Its goal is to articulate complex ai concepts in a clear and accessible way.