CMU researchers propose DocPrompting: a natural language for code generation approach by retrieving code documentation

Publicly available source code libraries are always evolving and expanding. Therefore, it is difficult for code models to keep up to date with all accessible APIs by only training these models on existing code repositories. DocPrompting is a new way of generating code from natural language that uses documentation explicitly by requesting the appropriate documentation components in response to an NL intent.

DocPrompting’s flexibility means that it can be used with any programming language and is independent of the specific neural model being used. To help developers, docprompting can get documentation sections and write code based on those sections. By scanning the documentation, an LM code (such as Codex or CodeT5) can create calls to libraries and functions that you have never encountered in your training data.

How does it work

To begin, a document retriever will access the documentation pool of the code being retrieved and, using the NL intent, retrieve any applicable documentation. A code generator then feeds the documentation into a notice that produces the code. New content (such as documentation for newly released libraries) can be added to the external data store documentation group without retraining any part of the model. This allows DocPrompting to use newly added documentation and produce code that uses libraries and functions that were previously invisible or unused. The DocPrompting framework is generic and can be used with any programming language or underlying base architecture.

Study and analysis by researchers

A group of researchers have provided a set of newly chosen benchmarks for testing future retrieval-based code generation models. DocPrompting was evaluated using a shell scripting task in which researchers had to write sophisticated intent-based shell commands and a Python programming task in which they had to generate Python responses to queries from DocPrompting. NL. Researchers present a newly selected benchmark tldr before discussing the recent split of the popular benchmark CoNaLa index. The researchers provide a global documentation set D for each reference point for training the retriever, including examples and Oracle Dn documents.

According to the study authors, models that use DocPrompting regularly outperform their NL intent-only code generation counterparts. CoNaLa’s run-based evaluation sees a 2.85% increase in pass@1 (52% relative gain) when using DocPrompting over already powerful base models like CodeT5.

DocPrompting consistently outperforms state-of-the-art methods on the new NL->Bash “tldr” dataset. In the case of CodeT5 and GPT-Neo1.3B, for example, you can increase the exact match rate by up to 6.9%.

According to the researchers, one of the main reasons is that the documentation comprises both natural language descriptions and function signatures, which simplifies the mapping between intents and NL code. The researchers determined the n-gram overlap between NL intents and the code snippets that matched them (NLcode), and the researchers determined the overlap between NL intents and the top 10 documents that were retrieved ((NL+docs ) code). The amount of information shared between n-grams increases dramatically when documentation is included. In other words, documentation retrieval helps in generating code accuracy, as it helps to bridge the gap between “intent terminology” and “code terminology”.

In conclusion, DocPrompting is a straightforward method of generating code by getting the proper documentation. DocPrompting reliably improves NLcode models in several robust basic models, two tasks, and two programming languages. Using the well-known Python CoNaLa benchmark, DocPrompting boosts strong base models like CodeT5 by 2.85% on pass@1 (52% relative gain) in performance-based testing; in the new Bash tldr dataset, DocPrompting increases CodeT5 and GPT-Neo-1.3B by up to 6.9% exact match and Codex by 6.78 in charBLEU score. These findings pave the way for a hopeful future for the NLcode generation. Further improvements are possible through cooperative training of the retriever and generator, which should prevent cascading errors, and through smarter coding of the organized nature of large texts.

review the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 14k+ ML SubReddit, discord channel, and electronic newsletterwhere we share the latest AI research news, exciting AI projects, and more.