When Microsoft added a chatbot to its Bing search engine this month, people noticed that it offered all sorts of false information about the Gap, Mexican nightlife and singer Billie Eilish.
Then, when journalists and other early testers engaged in lengthy conversations with Microsoft’s AI bot, it slipped into rude and disturbingly creepy behavior.
In the days since the Bing bot’s behavior became a worldwide sensation, people have struggled to understand the weirdness of this new creation. More often than not, scientists have said that humans deserve much of the blame.
But there is still a bit of a mystery as to what the new chatbot can do and why it would do it. Its complexity makes it hard to dissect and even harder to predict, and researchers are looking at it through a philosophical lens, as well as computer hard code.
Like any other learner, an AI system can learn bad information from bad sources. And that strange behavior? It can be a chatbot’s distorted reflection of the words and intentions of the people who use it, said Terry Sejnowski, a neuroscientist, psychologist and computer scientist who helped lay the intellectual and technical foundation for modern artificial intelligence.
“This happens when you go deeper and deeper into these systems,” said Dr. Sejnowski, a professor at the Salk Institute for Biological Studies and the University of California, San Diego, who published a research work about this phenomenon this month in the scientific journal Neural Computation. “Whatever you’re looking for, whatever you want, they’ll provide it for you.”
google too He showed a new chatbot, Bard, this month, but scientists and journalists quickly realized that it was writing nonsense about the James Webb Space Telescope. OpenAI, a San Francisco startup, launched the chatbot boom in November when it introduced ChatGPT, which also doesn’t always tell the truth.
The new chatbots are powered by a technology scientists call the large language model, or LLM. These systems learn by analyzing vast amounts of digital text pulled from the Internet, including volumes of false, biased, and toxic material. The text that chatbots learn from is also a bit outdated, because they have to spend months parsing it before the public can use them.
As you sift through the sea of good and bad information on the Internet, an LLM learns to do one thing in particular: guess the next word in a sequence of words.
It works like a giant version of autocomplete technology that suggests the next word as you type an email or instant message on your smartphone. Given the sequence “Tom Cruise is a ____”, it could guess “actor”.
When you chat with a chatbot, the bot doesn’t just draw on everything it has learned from the internet. It’s based on everything you’ve told him and everything he’s told you back. It’s not just guessing the next word in your sentence. It’s guessing the next word in the long block of text that includes both your words and their words.
The longer the conversation becomes, the more influence a user has unknowingly on what the chatbot says. If you want him to get mad, get mad, said Dr. Sejnowski. If you persuade him to get creepy, he gets creepy.
The alarm reactions to the strange behavior of the Microsoft chatbot overshadowed an important point: the chatbot has no personality. It offers instant results delivered by an incredibly complex computer algorithm.
Microsoft seemed to reduce the weirder behavior when it put a limit on the length of conversations with the Bing chatbot. That was like learning from a car test driver that going too fast for too long will burn out the engine. Microsoft partner OpenAI and Google are also exploring ways to control the behavior of their bots.
But there’s a caveat to this reassurance: Because chatbots are learning from so much material and putting it together in such complex ways, researchers aren’t entirely clear on how chatbots are producing their end results. Researchers watch what bots do and learn to put limits on that behavior, often after it happens.
Microsoft and OpenAI have decided that the only way they can find out what chatbots will do in the real world is to let them loose and pull them in when they stray. They think their big public experiment is worth the risk.
Dr. Sejnowski compared the Microsoft chatbot’s behavior to the Mirror of Erised, a mystical artifact in JK Rowling’s Harry Potter novels and the many movies based on her inventive world of young wizards.
“Eised” is “wish” spelled backwards. When people discover the mirror, it seems to provide truth and understanding. But it’s not like that. It shows the deep-rooted desires of anyone who looks at it. And some people go crazy if they stare too long.
“Because both the human and the LLMs mirror each other, they will tend toward a common conceptual state over time,” said Dr. Sejnowski.
It was not surprising, he said, that journalists began to see creepy behavior on the Bing chatbot. Whether consciously or unconsciously, they were pushing the system in an uncomfortable direction. As chatbots take our words and reflect them back to us, they can reinforce and amplify our beliefs and convince us to believe what they tell us.
Dr. Sejnowski was among a small group of researchers in the late 1970s and early 1980s who began to seriously explore a type of artificial intelligence called a neural network, which powers today’s chatbots.
A neural network is a mathematical system that learns skills by analyzing digital data. This is the same technology that allows Siri and Alexa to recognize what you say.
Around 2018, researchers at companies like Google and OpenAI began building neural networks that learned from vast amounts of digital text, including books, Wikipedia articles, chat logs, and other things posted on the internet. By identifying billions of patterns in all of this text, these LLMs learned to generate text on their own, including tweets, blog posts, speeches, and computer programs. They could even hold a conversation.
These systems are a reflection of humanity. They learn their abilities by analyzing text that humans have posted on the Internet.
But that’s not the only reason chatbots generate problematic language, said Melanie Mitchell, an artificial intelligence researcher at the Santa Fe Institute, an independent laboratory in New Mexico.
When they generate text, these systems do not repeat what is on the Internet word for word. They produce new text on their own by combining billions of patterns.
Even if researchers trained these systems solely on the peer-reviewed scientific literature, they could still produce statements that were scientifically ridiculous. Even if they learned solely from the text that it was true, they could still produce falsehoods. Even if they learned from just one text that it was wholesome, they could still come up with something creepy.
“There’s nothing stopping them from doing this,” Dr. Mitchell said. “They’re just trying to produce something that sounds like human language.”
Artificial intelligence experts have long known that this technology exhibits all sorts of unexpected behaviors. But they can’t always agree on how this behavior should be interpreted or how quickly chatbots will improve.
Because these systems learn from so much more data than humans could understand, even AI experts can’t understand why they generate a particular text at any given time.
Dr. Sejnowski said that he believed that, in the long term, the new chatbots had the power to make people more efficient and give them ways to do their jobs better and faster. But this comes with a caveat for both the companies that build these chatbots and the people who use them: They can also lead us away from the truth and into dark places.
“This is terra incognita,” Dr. Sejnowski said. “Humans have never experienced this before.”
