Has OpenAI invented an ai technology with the potential to “threaten humanity”? Judging by some of the recent headlines, one might be inclined to think so.
technology/sam-altmans-ouster-openai-was-precipitated-by-letter-board-about-ai-breakthrough-2023-11-22/”>Reuters and ai-breakthrough-before-altman-firing-stoking-excitement-and-concern”>Information First reported last week that several OpenAI staff members, in a letter to the ai startup’s board, had pointed out the “skill” and “potential danger” of an internal research project known as “Q *”. This ai project, according to the report, could solve certain mathematical problems, although only at the primary school level, but, in the researchers’ opinion, it had the potential to move towards an elusive technical breakthrough.
There is now debate over whether OpenAI’s board of directors ever received such a letter: The Verge cites one fountain suggesting that this was not the case. But setting aside the Q* framework, Q* might actually not be as monumental – or threatening – as it seems. It may not even be new.
ai researchers at of ai. In a post on X, Rick Lamers, who writes the Substack Coding with Intelligence newsletter, pointed to an MIT guest lecture that OpenAI co-founder John Schulman gave seven years ago during which he described a mathematical function called “Q*.”
Several researchers believe that the “Q” in the name “Q*” refers to “Q-learning”, an artificial intelligence technique that helps a model learn and improve at a particular task by performing (and being rewarded) actions specific “correct” ones. Meanwhile, researchers say the asterisk could be a reference to A*, an algorithm for checking the nodes that make up a graph and exploring the paths between these nodes.
Both have been around for a while.
Google DeepMind applied Q-learning to build an ai algorithm that could play Atari 2600 games at a human level… in 2014. A* has its origins in an academic paper published in 1968. And researchers at UC Irvine several years ago explored improve A* with Q-learning, which could be exactly what OpenAI is pursuing now.
Nathan Lambert, a research scientist at the Allen ai Institute, told TechCrunch that he believes Q* is related to approaches in ai “primarily (to) study high school math problems,” not to destroy humanity.
“OpenAI even shared work earlier this year to improve the mathematical reasoning of language models with a technique called reward process models,” Lambert said, “but what remains to be seen is how better mathematical skills do anything.” more to do (OpenAI’s ai-powered chatbot) ChatGPT is a better code assistant.”
Mark Riedl, a computer science professor at Georgia tech, was equally critical of the Reuters and The Information reports on Q*, and of the broader media narrative around OpenAI and its quest toward artificial general intelligence (i.e., a that can perform any task as well as a human can). Reuters, citing a source, hinted that Q* could be a step towards artificial general intelligence (AGI). But researchers, including Riedl, dispute that.
“There is no evidence to suggest that large language models (like ChatGPT) or any other technology under development at OpenAI are on the path to AGI or any of the doom scenarios,” Riedl told TechCrunch. “OpenAI itself has been, at best, a ‘fast follower’, having taken existing ideas… and found ways to extend them. While OpenAI hires top researchers, much of what they have done can be done by researchers at other organizations. It could also be done if the OpenAI researchers were in a different organization.
Riedl, like Lambert, did not guess whether Q* might imply Q-learning or A*. But if it involved either (or a combination of both), it would be consistent with current trends in ai research, he said.
“These are all ideas that other researchers in academia and industry are actively pursuing, with dozens of papers on these topics in the last six months or more,” Riedl added. “It is unlikely that OpenAI researchers would have had ideas that a substantial number of researchers who are also pursuing advances in ai have not also had.”
This isn’t to say that Q*, which reportedly featured Ilya Sutskever, OpenAI’s chief scientist, can’t move the needle forward.
Lamers claims that if Q* uses some of the techniques described in a paper published by OpenAI researchers in May, could “significantly” increase the capabilities of language models. According to the article, OpenAI might have discovered a way to control the “chains of reasoning” of language models, Lamers says, allowing them to guide the models to follow more desirable and logically sound “paths” to achieve results.
“This would make models less likely to follow ‘non-human thoughts’ and spurious patterns to reach malicious or erroneous conclusions,” Lamers said. “I think this is really a win for OpenAI in terms of alignment… Most ai researchers agree that we need better ways to train these large models, so they can consume information more efficiently.
But whatever emerges from Q*, neither it nor the relatively simple mathematical equations it solves will spell doom for humanity.
