Salesforce AI Research introduces a speculative decoding guided by rewards (RSD): a novel frame that improves the efficiency of inference in large language models (LLM) up to 4.4 × less failures

In recent years, the rapid scale of large language models (LLM) has led to extraordinary improvements in the understanding of natural language and reasoning capabilities. However, this progress comes with a significant warning: the inference process, the responses degeneration of one token at the same time, lifts a computational bottleneck. As the LLMs grow in size and complexity, the demands of latency and energy for the generation of sequential token become substantial. These challenges are particularly acute in real world implementations, where cost, speed and scalability are critical. Traditional decoding approaches, such as greedy or beam search methods, often require repeated evaluations of large models, leading to high computational overload. In addition, even with parallel decoding techniques, maintaining both efficiency and the quality of the generated outputs can be difficult to achieve. This scenario has stimulated a search for new techniques that can reduce inference costs without sacrificing precision. Therefore, researchers have been exploring hybrid approaches that combine light models with more powerful counterparts, fighting for an optimal balance between speed and performance, a balance that is essential for real -time applications, interactive systems and large -scale deployment in cloud environments.

Salesforce ai Research presents a speculative decoding guided by rewards (RSD), a novel framework for improving the efficiency of inference in large language models (LLM). In essence, RSD takes advantage of a double model strategy: a rapid and light “draft” model works together with a more robust “target” model. The draft model generates preliminary candidates quickly, while a process reward model (PRM) evaluates the quality of these outputs in real time. Unlike traditional speculative decoding, which insists on a strict impartial token coincidence between draft and objective models, RSD introduces a controlled bias. This bias is carefully designed to favor high reward, which are considered more likely to be correctly or contextually relevant, which significantly reduces unnecessary calculations. The approach is based on a mathematically derived threshold strategy that determines when the target model should intervene. When mixing dynamically exits from both models based on a reward function, RSD not only accelerates the inference process, but also improves the general quality of the responses generated. Detailed in the attached document, this progress methodology represents a significant leap forward in the approaching of the inherent inefficiencies of the generation of sequential token in LLM.

Technical details and benefits of RSD

By deepening the technical aspects, RSD operates integrating two models sequentially but collaboratively. Initially, the draft of the model produces candidate tokens or reasoning steps to a low computational cost. Then, each candidate is evaluated using a reward function, which acts as a quality door. If the reward of a candidate token exceeds a default threshold, the exit is accepted; If not, the system requires that the most computationally intensive destination model generates a refined token. This process is guided by a weighting function, typically a binary passage function, which adjusts the dependence of the draft versus the destination model. The dynamic quality control offered by the process reward model (PRM) ensures that only the most promising exits avoid the destination model, thus saving in the calculation. One of the outstanding benefits of this approach is the “biased acceleration”, where controlled bias is not a detriment, but a strategic choice to prioritize high reference results. This results in two key benefits: first, the general inference process can be up to 4.4 × faster compared to executing the objective model alone; Secondly, it often produces an average precision improvement of +3.5 on conventional parallel decoding lines. In essence, the RSD harmonizes efficiency with precision, which follows by a substantial reduction in the number of floating point operations (flop) while delivering outputs that meet or even exceed the performance of the target model. The theoretical bases and algorithmic details, such as the distribution of the mixture defined by PRSD and the adaptive acceptance criteria, provide a robust framework for practical implementation in various reasoning tasks.

Perspectives

RSD's empirical validation is convincing. The experiments detailed in the document show that, at challenging reference points such as GSM8K, Math500, Olympiadbench and GPQA, RSD constantly offers higher performance. For example, at the Math500 reference point, a data set designed to prove mathematical reasoning, RSD achieved an accuracy of 88.0 when it was configured with a 72b destination model and an 7B 7B destination, compared to 85.6 for the destination model that is executed alone. This configuration not only reduces the computational load by almost 4.4 × less failures, but also improves the accuracy of reasoning. The results underline the potential of RSD to overcome traditional methods, such as speculative decoding (SD) and even advanced techniques based on search such as beam search or the best N strategies.

Conclusion: a new paradigm for an efficient LLM inference

In conclusion, speculative decoding guided by rewards (RSD) marks a significant milestone in the search for a more efficient LLM inference. By intelligent combining a light draft model with a powerful objective model, and by introducing an acceptance criterion based on rewards, RSD effectively addresses the dual challenges of the computational cost and the quality of production. The innovative biased acceleration approach allows the system to selectively ignore the expensive calculations for high reward outputs, thus speeding up the inference process. The dynamic quality control mechanism, canceled by a process reward model, ensures that computational resources are judgantly assigned, involving the objective model only when necessary. With empirical results that show up to 4.4 × faster inference and an average precision improvement of +3.5 on traditional methods, RSD not only rays the path for more scalable LLM implementations, but also establishes a new standard in Marcos design of hybrid decoding.

Verify he Paper and Github page. All credit for this investigation goes to the researchers of this project. In addition, feel free to follow us <a target="_blank" href="https://x.com/intent/follow?screen_name=marktechpost” target=”_blank” rel=”noreferrer noopener”>twitter And don't forget to join our 75K+ ml of submen.

Recommended open source ai platform: 'Intellagent is a framework of multiple open source agents to evaluate the conversational the complex system' _(Promoted)