A group of OpenAI experts are exposing what they say is a culture of recklessness and secrecy at the San Francisco artificial intelligence company, which is racing to build the most powerful ai systems ever created.
The group, which includes nine current and former OpenAI employees, has come together in recent days over shared concerns that the company has not done enough to prevent its ai systems from becoming dangerous.
Members say OpenAI, which started as a nonprofit research lab and came into the public eye with the launch of ChatGPT in 2022, is prioritizing profits and growth as it tries to build artificial general intelligence, or AGI. , the industry term for a computer. program capable of doing anything a human can do.
They also claim that OpenAI has used harsh tactics to prevent workers from raising concerns about the technology, including restrictive non-disparagement agreements that outgoing employees were asked to sign.
“OpenAI is really excited about building AGI, and they are racing recklessly to be the first to achieve it,” said Daniel Kokotajlo, a former researcher in OpenAI's governance division and one of the group's organizers.
The group published a ai/” title=”” rel=”noopener noreferrer” target=”_blank”>open letter on tuesday calling on leading ai companies, including OpenAI, to establish greater transparency and more protection for whistleblowers.
Other members include William Saunders, a research engineer who left OpenAI in February, and three other former OpenAI employees: Carroll Wainwright, Jacob Hilton and Daniel Ziegler. Several current OpenAI employees anonymously supported the letter because they feared retaliation from the company, Kokotajlo said. Also signing were a current employee and a former employee of Google DeepMind, Google's central artificial intelligence laboratory.
An OpenAI spokesperson, Lindsey Held, said in a statement: “We are proud of our track record of providing the most capable and safest ai systems and believe in our scientific approach to addressing risk. “We agree that rigorous debate is crucial given the importance of this technology, and we will continue to engage with governments, civil society and other communities around the world.”
A Google spokesperson declined to comment.
The campaign comes at a difficult time for OpenAI. It is still reeling from an attempted coup last year, when members of the company's board of directors voted to fire Sam Altman, the chief executive, over concerns about his outspokenness. Altman returned days later and the board was reorganized with new members.
The company is also facing legal battles with content creators who accuse it of stealing copyrighted works to train its models. (The New York Times sued OpenAI and its partner, Microsoft, for copyright infringement last year.) And its recent unveiling of a hyper-realistic voice assistant was marred by a public dispute with Hollywood actress Scarlett Johansson, who claimed that OpenAI had imitated his voice without permission.
But nothing has endured as long as the accusation that OpenAI has been too cavalier about security.
Last month, two senior ai researchers, Ilya Sutskever and Jan Leike, left OpenAI under a cloud. Dr. Sutskever, who had been on OpenAI's board of directors and voted to fire Mr. Altman, had raised the alarm about the potential risks of powerful ai systems. Some safety-conscious employees saw his departure as a setback.
So was the departure of Dr. Leike, who along with Dr. Sutskever had led OpenAI's “superalignment” team, which focused on managing the risks of powerful ai models. In a series of public posts announcing his departure, Dr. Leike said he believed “safety culture and processes have taken a backseat to shiny products.”
Neither Dr. Sutskever nor Dr. Leike signed the open letter written by former employees. But their departures prompted other former OpenAI employees to speak out.
“When I signed up for OpenAI, I didn't sign up for this attitude of 'Let's put things in the world and see what happens and fix them later,'” Saunders said.
Some of the former employees have ties to effective altruism, a utilitarian-inspired movement that in recent years has been concerned with preventing existential threats from ai. Critics have accused the movement of promoting apocalyptic scenarios about technology, such as the notion that an out-of-control artificial intelligence system could take over and wipe out humanity.
Kokotajlo, 31, joined OpenAI in 2022 as a governance researcher and was asked to forecast ai progress. He was not, to put it mildly, optimistic.
In his previous job at an ai security organization, he predicted that AGI could arrive in 2050. But after seeing how quickly ai was improving, he shortened his timeline. He now believes there is a 50 percent chance that AGI will arrive in 2027, in just three years.
He also believes that the probability of advanced ai destroying or catastrophically harming humanity (a grim statistic often abbreviated to “p(doom)” in ai circles) is 70 percent.
At OpenAI, Kokotajlo saw that although the company had security protocols in place, including a joint effort with Microsoft known as the “deployment security board,” which was supposed to review new models for significant risks before they were released publicly, he rarely seemed to hold anything back.
For example, he said, in 2022 Microsoft began quietly testing in India a new version of its Bing search engine that some OpenAI employees believed contained an unreleased version of GPT-4, OpenAI's next-generation large language model. Kokotajlo said he was told that Microsoft had not obtained security board approval before testing the new model, and after the board learned of the testing, via a series of reports that Bing was acting strangely toward users; it did nothing to stop Microsoft from rolling it out more widely.
A Microsoft spokesman, Frank Shaw, disputed those claims. He said India's tests had not used GPT-4 or any OpenAI models. Microsoft first released GPT-4-based technology in early 2023, he said, and it was reviewed and approved by a security board predecessor.
Over time, Kokotajlo said, he became so concerned that last year he told Altman that the company should “pivot to security” and spend more time and resources protecting against ai risks instead of moving forward to improve. their models. She said that Mr. Altman had stated that he agreed with him, but that nothing much had changed.
In April he resigned. In an email to his team, he said he was leaving because he had “lost confidence that OpenAI will behave responsibly” as its systems move closer to human-level intelligence.
“The world is not ready and we are not ready,” Kokotajlo wrote. “And I'm concerned that we're moving forward anyway and rationalizing our actions.”
OpenAI said last week that it had begun training a new flagship ai model and was forming a new safety committee to explore the risks associated with the new model and other future technologies.
Upon leaving, Kokotajlo refused to sign OpenAI's standard documentation for outgoing employees, which included a strict non-disparagement clause that prohibited them from saying negative things about the company, or risk having their vested equity taken away.
Many employees could lose millions of dollars if they refused to sign. Kokotajlo's acquired capital was worth about $1.7 million, he said, which was equivalent to the vast majority of his net worth, and he was willing to lose it all.
(A small firestorm occurred last month after Vox reported news of these agreements. In response, OpenAI stated that it had never recovered capital acquired from former employees and would not do so. Altman said he was “really embarrassed” for not having known about the settlements, and the company said he would remove non-disparagement clauses from its standard documentation and release former employees from their agreements.)
In their open letter, Kokotajlo and the other former OpenAI employees call for an end to the use of non-disparagement and confidentiality agreements at OpenAI and other ai companies.
“Broad confidentiality agreements prevent us from expressing our concerns except to the same companies that may not be addressing these issues,” they write.
They also call on ai companies to “support a culture of open criticism” and establish a reporting process for employees to anonymously raise safety-related concerns.
They have hired a pro bono attorney, Lawrence Lessig, the prominent jurist and activist. Lessig also advised Frances Haugen, a former facebook employee who became a whistleblower and accused the company of putting profits before safety.
In an interview, Lessig said that while traditional whistleblower protections generally applied to reports of illegal activity, it was important that employees at ai companies be able to discuss the risks and potential harms freely, given the importance of the technology. .
“Employees are an important line of safety defense, and if they can't speak freely without retaliation, that channel will be closed,” he said.
Held, a spokesperson for OpenAI, said the company had “ways for employees to express concerns,” including an anonymous integrity hotline.
Kokotajlo and his group are skeptical that self-regulation alone will be enough to prepare for a world with more powerful ai systems. That is why they ask legislators to also regulate the industry.
“There needs to be some kind of transparent and democratically accountable governance structure in charge of this process,” Kokotajlo said. “Instead of just a couple of different private companies competing against each other and keeping everything secret.”
