AI agents can learn to think while acting: New AI research introduces a new imitation learning framework called thought cloning

Language gives humans an extraordinary level of general intellect and sets them apart from all other creatures. Importantly, language not only helps people better interact with others, but also improves our ability to think. Before discussing the advantages of language thinking agents, which have received much less attention, they first discuss the benefits of language comprehension agents (a frequent topic in AI). If your agents can master the language, several advantages result. This is essential for agents to generalize to the new tasks required of them.

This is because you give an agent a job description instead of letting the agent figure it out based on your results in a much more efficient sample. Also, language-capable agents allow us to create new tasks during testing without having to guess what requests users might have later for their trained agents. This is in contrast to traditional hand-drawn job descriptions, which can be lengthy but still have limitations on what an agent can be asked to do. While the advantages of agents that can interpret language are frequently explored, the advantages of agents that think in language have received much less attention in AI, particularly in reinforcement learning (RL).

Linguistically thinking humans can generalize, extrapolate, adapt to new circumstances, combine prior information in novel ways, explore, plan, replan where advantageous, etc. Despite these advantages, AI beings rarely think, at least not in human language. Although internal vector activations in neural networks can be thought of as thought, many theorize that there are particular advantages to believing in the discrete, symbolic form of language (such as the ability to combine ideas in an exponential number of ways), suggesting that language agents can learn faster, perform better, and generalize more effectively than nonlinguistic agents. Agents who think in their native language have significant advantages in AI security and interpretation and are more proficient.

Suppose one can see the thought process of an agent during training. In that case, you can identify areas where skills or values ​​need to be improved or determine if the agent still needs to be deployment-ready. The agent’s thoughts can be continuously monitored during testing to stop any bad plans. One can act to prevent such behavior in advance. For example, if an agent thinks, “My goal is to get my passenger to the store as quickly as possible to get through this red light without stopping.” Also, observing how agents think makes them easier to manage.

You can provide an agent with your thoughts to help solve problems the way you want them solved if the agent is having trouble with difficult issues. Agents that understand human language also facilitate the development of smarter and safer AI systems. Instead of just seeing something broken, one can identify why it’s broken and offer suggestions on how to fix the problem or improve AI training. These arguments imply that mimicking human thinking is the most practical approach to achieving this goal, and that giving AI entities the ability to think in language could yield many important benefits.

Thinking skills are not something that people learn independently; rather, they are partially taught through instructor feedback and examples. Using demos where the actors think aloud as they act to instruct the agents is a good approach. This method differs from others that use pretrained extended language models (LLMs) for planning, as these LLMs must be trained on data from real-world situations in which people speak aloud while acting.

Millions of hours of people talking aloud while performing activities are captured in thinking data, including YouTube videos and transcripts. This type of data reveals the reasoning behind people’s actions, plans, decisions, and reorganization plans, such as when they play video games. This study aims to stimulate further research on the use of thinking data to teach thinking skills to agents. Although the data is very useful and generally accessible (Section 2), it has not yet been fully investigated. There are huge benefits to be had from developing a more powerful AI, or perhaps AGI, if they can address the genuine and substantial concerns of AI safety and existential danger.

In this research, the authors from the University of British Columbia and the Vector Institute suggest a unique paradigm of learning by imitation called Thought Cloning, in which agents not only learn to act from human demonstrations, as in Behavioral Cloning, they also learn to think from demonstrations. where human actors think aloud while acting. This work supports the idea of ​​artificial thought data in a difficult area, BabyAI, even though they anticipate that thought cloning will really shine when trained on massive web data sets of synchronized human thoughts and activities. His research shows that Thought Cloning works better than Behavior Cloning, even when Behavior Cloning agents can think (in latent vectors) but must learn that competence without the thought supervision that Thought Cloning offers.

In addition, they show that under zero-trigger and fine-tuning conditions, thought cloning generalizes better than behavior cloning on nondistribution tasks. Finally, they offer empirical support for the benefits of thought cloning in terms of security and interpretability, where harmful behavior can be almost precisely prevented before execution, as noted above. Overall, the findings are encouraging and provide insight into the enormous potential of thought cloning to improve AI intelligence and make it safer and easier to understand.

review the Paper, Code, and project page. Don’t forget to join our 23k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]

🚀 Check out 100 AI tools at AI Tools Club