Salesforce AI Research proposes PerfCodeGen: a training-free framework that improves the performance of LLM-generated code with execution feedback

Large language models (LLMs) have become essential tools in software development, offering capabilities such as generating code snippets, automating unit testing, and debugging. However, these models often fail to produce code that is not only functionally correct but also efficient at runtime. Overlooking runtime efficiency can cause software to perform poorly, increase operating costs, and impact user experience. This problem is particularly pronounced for less experienced developers, who may rely on ai-suggested code without fully understanding its implications. Salesforce Research addresses these challenges with PerfCodeGen, a framework that aims to improve both the correctness and performance of code generated by LLM.

Salesforce ai's PerfCodeGen is a training-free framework designed to improve the runtime efficiency of LLM-generated code. It achieves this by using execution feedback in an iterative self-refinement process. Unlike approaches that require fine-tuning with extensive training data, PerfCodeGen employs a feedback loop that evaluates and refines code based on runtime metrics during test execution. The framework operates in two key phases: refining correctness and optimizing performance. Initially, it ensures that the generated code meets the functional requirements by addressing the issues identified in the unit tests. Once correctness is established, the framework focuses on runtime efficiency, optimizing the code by targeting and refining the most resource-intensive test cases. This iterative process results in solutions that are correct and efficient.

Technical information and benefits

PerfCodeGen integrates with existing LLM workflows and starts by generating multiple candidate solutions using core sampling. In the first phase, the correctness of these candidates is evaluated using unit tests. Feedback from failed tests is used to refine solutions. Once functional correctness is guaranteed, the framework moves to the second phase, analyzing runtime metrics to identify bottlenecks. This information is then used to further optimize the code, focusing on the most time-consuming test cases.

This two-phase process increases the probability of producing optimally efficient programs. PerfCodeGen's methodology reflects human optimization and debugging practices, making it effective and intuitive. Additionally, the framework's reliance on feedback rather than retraining allows it to scale across multiple LLMs and application domains. It has shown consistent improvements in runtime efficiency and correctness on models such as Phi-3-mini, Llama 3, and GPT-4.

PerfCodeGen has been tested on benchmarks such as HumanEval, MBPP and APPS, proving its effectiveness:

1. Runtime efficiency: In HumanEval, the GPT-4 optimization rate (%Opt) increased from 24.54% to 28.83% with PERFCODEGEN, and similar improvements were seen in other models.
2. Improved fix: In MBPP, the GPT-3.5 correction rate (%correct) increased from 66.38% to 73.36% with a single sample (Best@1).
3. Overcoming the ground truth: PERFCODEGEN enabled LLMs to generate solutions more efficient than ground truth in approximately 55% of HumanEval tasks and 67% of MBPP tasks.
4. Scalability: Open models such as Phi-3-mini and Mixtral achieved comparable performance to closed models such as GPT-3.5 and GPT-4.

These results highlight PERFCODEGEN's ability to balance correctness and runtime efficiency effectively, making it a valuable addition to LLM-based code generation workflows.

Conclusion:

PerfCodeGen offers a practical solution to a key limitation of current LLMs: their focus on correctness at the expense of runtime efficiency. By incorporating execution feedback into an iterative refinement process, PerfCodeGen enables the generation of correct and efficient code. This approach improves the usability of LLMs in software development, providing developers with tools to produce higher quality code without extensive retraining. The success of the framework across various benchmarks demonstrates its potential as a step forward in creating efficient, reliable, and accessible ai-powered programming solutions.

Verify he Paper and GitHub page. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter and join our Telegram channel and LinkedIn Grabove. Don't forget to join our SubReddit over 65,000 ml.

Recommend open source platform: Parlant is a framework that transforms the way ai agents make decisions in customer-facing scenarios. ^(Promoted)