Large language models (LLMs) have proven successful in various tasks in the software engineering domain, such as generating code and documentation, translating code between programming languages, writing unit tests, and detecting and fixing errors. The availability of large open source code datasets and Github-enabled models such as CodeLlama, ChatGPT and Codex to develop a statistical understanding of various languages significantly improves the coding experience. Some models, such as AlphaCode, are pre-trained on competitive programming tasks, allowing the model to optimize code at the source level for multiple languages.
Previous research on machine learning-driven code optimization tested multiple techniques, from manually designed functions to complex and sophisticated graph neural networks. However, a common limitation in these approaches is that the input program representation provided to the machine learning model remains incomplete, resulting in some loss of critical information. For example, MLGO provides insight into feature addition, but lacks a faithful reconstruction of aspects such as the call graph and control flow. To overcome this problem, LLMs were proposed to tune the optimization of LLVM, which demonstrated remarkable code reasoning capabilities.
Meta ai and Rice University researchers present a unique paradigm in LLM-driven compiler optimization with compiler feedback to optimize LLVM assembly code size. The model takes non-optimized IR LLVM as input and produces optimized IR, best optimization passes, and optimized and non-optimized IR instruction counts. The input is then satisfied with the generated optimization passes and evaluated to see if the expected instruction count is correct, if the generated IR is compileable, and if it corresponds to the compiled code. This feedback is returned to LLM and given another opportunity to optimize the code.
The method proceeds by evaluating the consistency of the model generation by the compiler and
Provide feedback to the model. For each generation of the original model, if the generated pass list is valid and the predicted instruction counts are correct and calculated, the blue score between the generated code and the code is obtained by compiling the generated pass list. The generated and compiled code is also provided in the comments.
The proposed model outperforms the original model by 0.53%, closing the gap with the autotuner by 10%. On the other hand, when sampling is enabled, it is shown that the original model achieves up to 98% of the autotuner performance when using 100 samples. When comparing the performance of the iterative feedback model with the original model, it was given the same amount of calculation per sample and it was found that with two or more samples and a temperature greater than 0.4, the original model outperformed the feedback model .
The researchers have made the following contributions in this article:
- Three compiler-generated feedback models for LLM were presented.
- I carried out evaluation of 3 sampling methods with feedback.
- The generation of iterative feedback was evaluated.
In conclusion, researchers at Meta ai and Rice University introduce the idea of compiler-generated feedback for LLMs. The model starts from unoptimized LLVM IR and predicts the best optimization passes, instruction counts, and optimized IR. Then they build feedback on the model generation and ask you to try again. All forms of feedback outperform the original model at 0 temperature by 0.11%, 0.4% and 0.53%.
Review the Paper. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on Twitter. Join our Telegram channel, Discord Channeland LinkedIn Grabove.
If you like our work, you will love our Newsletter..
Don't forget to join our 39k+ ML SubReddit
Asjad is an internal consultant at Marktechpost. He is pursuing B.tech in Mechanical Engineering at Indian Institute of technology, Kharagpur. Asjad is a machine learning and deep learning enthusiast who is always researching applications of machine learning in healthcare.
<!– ai CONTENT END 2 –>