Superhuman Performance At Atari 100K Benchmark: The Power Of BBF – A New RL Agent Based On The Value Of Google DeepMind, Mila And The University Of Montreal

Deep reinforcement learning (RL) has emerged as a powerful machine learning algorithm to tackle complex decision-making tasks. To overcome the challenge of achieving human-level sample efficiency in deep RL training, a team of researchers from Google DeepMind, Mila, and the University of Montreal have introduced a new values-based RL agent called “faster, better.” , faster” (BBF). . In his recent article, “Bigger, Better, Faster: Human-Level Atari with Human-Level Efficiency”, the team introduces Agent BBF, demonstrating superhuman performance on the Atari 100K benchmark using a single GPU.

Address the problem of scale

The main focus of the research team was to address the scaling problem of deep RL neural networks when there are limited samples. Based on the SR-SPR agent developed by D’Oro et al. (2023), employing a perturb and reduce method, BBF perturbs 50 percent of the parameters of the convolutional layers towards a random target. By contrast, SR-SPR perturbs only 20 percent of the parameters. This modification results in improved performance of the BBF agent.

Scalable network capacity

To scale the capacity of the network, the researchers use the Impala-CNN network and increase the size of each layer four times. BBF was observed to consistently outperform SR-SPR as network width increases, while SR-SPR peaks at 1-2 times the original size.

Improvements for better performance

BBF introduces an upgrade horizon component that decreases exponentially from 10 to 3. Surprisingly, this modification produces a stronger agent than fixed value agents like Rainbow and SR-SPR. In addition, the researchers apply a strategy of decreasing the weight and increasing the discount factor during learning to alleviate the problems of statistical overfitting.

Empirical Study and Results

In their empirical study, the research team compares the performance of the BBF agent against several basic RL agents, including SR-SPR, SPR, DrQ (eps), and IRIS, on the Atari 100K benchmark. BBF outperforms all competitors in terms of performance and computational cost. Specifically, BBF achieves a 2x performance improvement over SR-SPR while using almost the same computational resources. Additionally, BBF demonstrates comparable performance to the model-based EfficientZero approach, but with more than a 4-fold reduction in execution time.

Future implications and availability

The introduction of the BBF agent represents a significant advance in achieving superhuman performance in deep RL, particularly on the Atari 100K benchmark. The research team hopes that their work will inspire future efforts to push the limits of sample efficiency in deep RL. The code and data associated with the BBF agent is publicly available on the project website. GitHub repositoryallowing researchers to explore and develop their findings.

With the introduction of the BBF agent, Google DeepMind and its collaborators have shown remarkable progress in deep reinforcement learning. By addressing the challenge of sample efficiency and leveraging advances in network scaling and performance improvements, the BBF Agent achieves superhuman performance on the Atari 100K benchmark. This work opens up new possibilities to improve the efficiency and effectiveness of RL algorithms, paving the way for further advances in the field.

review the Paper and Github. Don’t forget to join our 23k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club