UC Berkeley researchers propose a neural diffusion model that operates on syntax trees for program synthesis

Large language models (LLMs) have revolutionized code generation, but their autoregressive nature poses a significant challenge. These models generate code token by token, without access to the program runtime output from previously generated tokens. This lack of a feedback loop, where the model can observe the program's output and adjust accordingly, makes error correction difficult. While LLMs can be trained to suggest edits to existing code, acquiring sufficient high-quality training data for this task remains an obstacle. Researchers are striving to overcome these limitations and develop more effective methodologies for using LLM in code generation and bug fixing.

Several existing approaches have addressed the challenges of code generation and error correction. Neural program synthesis methods generate programs from input and output examples, combining neural networks with search strategies. While effective, these techniques build programs incrementally, exploring a vast space of partial programs. Neural diffusion models have shown impressive results for generative modeling of high-dimensional data such as images. Recent work has extended the diffusion to discrete and structured data, such as graphs and molecules. Direct code editing using neural models, training on real-world code patch datasets, or fine-tuning language models have also been explored. However, these methods often require large code editing data sets or lack inherent guarantees of syntactic validity.

Researchers at the University of California, Berkeley present an effective approach to Program synthesis using neural diffusion models that operate directly on syntax trees.. Using diffusion allows the model to refine programs iteratively while ensuring syntactic validity. Crucially, the approach allows the model to observe the program output at each step, effectively facilitating a debugging process. Inspired by systems like AlphaZero, the iterative nature of diffusion lends itself well to search-based program synthesis. By training a value model alongside the diffusion model, the denoising process can be guided toward programs that are likely to achieve the desired outcome, allowing for efficient exploration of the program space.

The central idea of ​​this method is to develop denoising diffusion models for syntax trees, analogous to image diffusion models. Using context-free grammar (CFG), the method defines a noise process that randomly mutates programs while ensuring syntactic validity. This involves sampling mutations by restricting the “size” of the program within a range and replacing subtrees with alternative subtrees derived from the CFG production rules. A neural network is then trained to reverse this noise generation process, learning to denoise programs conditioned on the output of the target program (e.g. a rendered image). Additionally, a value network is trained to predict edit distances between programs, enabling efficient beam-finding exploration guided by promising candidate programs.

This method significantly outperforms two basic approaches (CSGNet and REPL Flow) on inverse graph tasks in the CSG2D and TinySVG domains. CSGNet represents a modern autoregressive approach, which generates programs in an autoregressive manner until a match is found. REPL Flow is based on primitive pre-work creation programs with access to intermediate rendered results. In both domains, the beam-search diffusion policy solves problems with fewer renderer calls than baselines. Qualitative examples highlight the method's ability to solve smaller problems that other approaches miss. Beyond that, the observation model can handle stochastic hand-drawn sketches, successfully recovering programs from noisy sketch inputs.

This research work introduced a robust neural diffusion model that operates directly on syntax trees for program synthesis. The proposed approach was successfully implemented for inverse graphics tasks, with the goal of finding programs that represent a given target image. Unlike previous methods, the model can iteratively build, run, and edit programs, allowing for a crucial feedback loop to correct errors. Extensive evaluations in graphics domains demonstrated the superiority of this approach over basic methods for the synthesis of inverse graphics programs. Additionally, the ablation experiments provided insight into the impact of key design choices behind the diffusion model architecture and training process.

Review the Paper, Projectand GitHub. All credit for this research goes to the researchers of this project. Also, don't forget to follow us on twitter.com/Marktechpost”>twitter. Join our Telegram channel, Discord channeland LinkedIn Grabove.

If you like our work, you will love our Newsletter..

Don't forget to join our 43k+ ML SubReddit | Also, check out our ai Event Platform