A new family of helper tasks based on the successor measure to improve the representations acquired by deep reinforcement learning agents

In deep reinforcement learning, an agent uses a neural network to assign observations to a policy or a return prediction. The function of this network is to convert the observations into a sequence of progressively finer features, which the final layer then linearly combines to obtain the desired prediction. The agent’s representation of its current state is how most people view this change and the intermediate features it creates. According to this perspective, the learning agent performs two tasks: representation learning, which involves finding valuable state features, and crediting, which involves translating these features into accurate predictions.

Modern RL methods often incorporate machinery that encourages learning of good state representations, such as predicting immediate rewards, states, or future observations, encoding a similarity metric, and data augmentation. End-to-end RL has been shown to perform well on a wide variety of problems. It is often feasible and desirable to acquire sufficiently wealthy representation before making the assignment of credit; Representation learning has been a central component of RL since its inception. Using the network to forecast additional tasks related to each state is an efficient way to learn state representations.

It can be shown that a collection of properties corresponding to the parent components of the auxiliary task matrix is ​​induced by additional tasks in an idealized environment. Thus, the theoretical approximation error, the generalization and the stability of the learned representation can be examined. It may come as a surprise to learn how little is known about their behavior in a larger-scale setting. It remains to be determined how the use of more tasks or the expansion of the network capacity would affect the scaling characteristics of representation learning from auxiliary activities. This essay seeks to bridge that information gap. They use a family of additional incentives that can be sampled as a starting point for their strategy.

Researchers at McGill University, the University of Montreal, the Institut Québec AI, the University of Oxford, and Google Research specifically apply the successor measure, which broadens successor representation by substituting state equality for established inclusion. In this situation, a family of binary functions on states serves as the implicit definition for these sets. Most of his research focuses on binary operations obtained from randomly initialized networks, which have already proven useful as random cumulants. Despite the possibility that your findings will apply to other ancillary rewards as well, there are several advantages to your approach:

• It can be easily scaled by using additional random network samples as additional tasks.
• It is directly related to the binary reward features found in RL deep benchmarks.
• It is partially understandable.

Predicting the expected return of the random policy for the relevant ancillary incentives is the real additional task; in the tabular environment, this corresponds to proto-value functions. As a result, they refer to their approach as proto-value networks. They investigate how well this approach works in the arcade learning environment. When used with the approximation of linear functions, they examine the features learned by PVN and demonstrate how well they represent the temporal structure of the environment. In general, they find that PVN needs only a small part of the interactions with the environment reward function to generate state features rich enough to support linear value estimates equivalent to those of DQN in various games.

They found in ablation research that expanding value network capacity significantly improves the performance of their line agents and that larger networks can handle more jobs. They also discover, somewhat unexpectedly, that their strategy works best with what may seem like a modest number of additional tasks: the smallest networks they analyze create their best representations from 10 or fewer tasks, and the largest from 10 or fewer tasks. 50 to 100 tasks. They conclude that specific tasks can result in representations that are much richer than anticipated and that the impact of any given work on fixed-size networks has yet to be fully understood.

review the Paper. Don’t forget to join our 21k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club