BONUS: Hard Fork Live! Big Tech’s Arch Nemesis + Bot or Not?

This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors. Please review the episode audio before quoting from this transcript and email [email protected] with any questions.

Kevin, what are we doing here? It’s not “Hard Fork” Friday.

It’s not “Hard Fork” Friday and yet, we have a new “Hard Fork.”

That’s very exciting.

Yeah. This is a special bonus episode. So a couple of weeks ago, we told you that we were doing a show at South by Southwest, “Hard Fork Live,” and we wanted to share that with you today.

I had a great time.

Yeah, it was really fun to go on the road. Thank you to everyone who made it out and stuck around afterwards to say hi, take selfies, et cetera. Very fun.

My favorite listener — and I will not give away any identifying personal information for reasons that will soon become obvious. But in order to see the “Hard Fork Live” show, you were supposed to have a South by Southwest badge. And this person somehow talked their way past the security guard and just said, I really want to see the show, and was able to see the show, illegally, at a cost to South by Southwest of several hundred dollars. So we just wanted to say thank you and respect to the people who would do anything to see the show.

Yeah, shout out to all our podcast pirates out there.

One thing we think you would love to hear is our conversation with Jonathan Kanter.

Yeah, Jonathan Kanter is the assistant attorney general of the Department of Justice. He is responsible for going after the big tech companies for antitrust violations, and he’s been the top cop who’s been really scrutinizing and pursuing charges against these big tech companies.

And since Joe Biden became president, the Antitrust Division of the Department of Justice has gotten much more active. They have open investigations into what feels like most of the biggest tech platforms, and that makes him a really interesting person as we think about what the future of the tech industry is going to look like.

Totally. And so we talked with him, you’ll hear that conversation. And then we played a fun little game with our live South by Southwest audience called Bot or Not. And you can hear that as well.

And play along at home if you like.

Yeah.

And email Kevin your score.

Don’t do that.

All right. [MUSIC PLAYING]

So here at Live from Austin is the first-ever “Hard Fork Live.”

And if we could just sort of pipe in the sound that the audience made when the Beatles performed on The Ed Sullivan Show in the 60s. That’s how I want it to sound.

Thunderous, roaring, hysterical.

Shrieking teenage girls.

It’s weird. Why does our — why does our live audience in Austin keep saying Ringo? What’s that? What’s that about? Anyway, here’s our show.

Please welcome Assistant Attorney General Jonathan Kanter of the Justice Department.

[APPLAUSE]

Hello!

Hello, great to see you.

Welcome.

Thanks for having me.

Thanks for coming to “Hard Fork.” I want to start by reading a couple of the monikers that the press has given you over the years, having you react to them. So you’ve been called, quote, top antitrust cop, Google foe, big tech’s arch nemesis, and an anti-Google legal warrior. That one was by The New York Times. How does it feel to be referred to as something like a Marvel villain?

I was going to say hero, but. And did you get that from a chat bot or —

Yeah. Well, speaking of chat bots, I heard that you consulted one in picking out your outfit today.

I did. So I’ve got colorful shoes on, as folks here might be able to see and I asked ChatGPT as, what do people think of colorful shoes? And the response was, well, it really depends on the context but if it’s a music or art festival, colorful shoes are OK. So here we are. I will note, though, that I was going to wear them either way.

Well, you are dressed down for a regulator, which we appreciate here at South by Southwest. I want to talk about some of the tech companies that the Department of Justice has investigated. But first, I just want to kind of lay some groundwork for this conversation and talk about you and your background and your philosophy of antitrust and how you got to the place you are. So you were — before you were at the Department of Justice, you were a lawyer in private practice working with and for some of the big tech companies. How did you develop your theory of tech antitrust?

Sure. So my experience informs my outlook. And I over time learned that when we apply antitrust law, there are these very formalized boxes and we try to fit everything into this narrow framework, assuming that, for example, consumers will behave rationally and they’ll all behave the same way and they’ll run from one store to the other.

And then as I got more into tech antitrust, what I realized is that the world has changed in ways that are, over the last 20 years, that are on par with, if not exceed the Industrial Revolution. Market realities have changed, fundamentally. Think about your world today versus 20 years ago, for those of you who were alive 20 years ago.

But it’s changed dramatically in the way we conduct business, in the way we gather information, and in the way we behave as individuals. And so we were applying this old framework that was built for a world of poles and wires to a world of ones and zeros. And so from my perspective, we needed to update our framework to reflect those market realities.

Well, I mean, let’s talk it a little bit more detail. I think the thing that comes up the most is that in the past, the main way that antitrust would be enforced is if prices rose for consumers or there was some sort of obvious financial harm to consumers. But now we live in a world where most of these services are free to use, or certainly the ones that regulators have been most interested in, like Google and Facebook. And so at the same time, there’s been this growing sense that these companies are awfully big and it seems like there are some anti-competitive behavior that they’re engaged in to sort of stay as big as they are, but they’re free, and anybody could use anything else if they wanted to. So there sort of had to be a new theory of the case with regard to antitrust. Is that the sort of new market reality you’re talking about?

In part, yeah. I mean, ask yourself, is anything really for free. But so what are you giving up to get something? Are you giving up your privacy, are you giving up your data? Are you giving up your personal profile that can be used to create, in the future, an AI simulation of yourself. Those are the kinds of questions we need to start answering.

The other point, I think, is really important. Because when we think of about price, when we think about antitrust just in the context of price, one problem is that we’re missing the point of antitrust enforcement and anti-monopoly law. So antitrust principles have been embedded in our societies since our founding. The Tea Party — the Boston Tea Party was an anti-monopoly movement in its core.

State constitutions talk about monopolies interfering with freedom. And so just as we won’t tolerate kings, we shouldn’t tolerate monopolies who tell us how we have to behave, what we see, what we consume. And so when we’re talking about the free flow of information, which is vital to a political discourse, or we’re talking about information that’s used to target us as individuals, those go to the foundation of our Democratic values and our fundamental freedom.

And it’s important that we’re protecting competition, the competitive process, so that we have choice, so that we have freedom of choice. We can pick where we want to work. We can pick what services we want to use. We can pick where we want to buy things.

So in the old sort of model of antitrust, the test to apply to figure out whether something was a monopoly is, are consumers being harmed by this? This new antitrust framework you’re talking about, which is sometimes called hipster antitrust — I don’t know if you like that term or not, but that has been applied. What is the standard of harm? What makes something a monopoly if it’s not rising prices for the consumer?

I will first be clear, my kids do not consider me hipster in any way, shape or form, notwithstanding my cool shoes here. But the problem is that, how do you measure price, for example, in a multi-sided market where monetization doesn’t take place in a completely transparent way? The antitrust laws exist because we believe competition. We believe choice. We believe variety is foundational to our freedom.

And so when we focus on competition, because we believe that the range of benefits that result from competition are good for society, then that’s a much easier thing to measure and that’s a much easier thing to preserve, and that’s our focus. That is actually more adherent to the founding of the antitrust laws going back to 1890.

Some of these economic theories that are often put out there as to narrow the antitrust laws didn’t exist when Congress wrote the law in 1890, and then again in 1914, and then updated again. But the concept of competition being foundational to our freedom did exist. And so that’s what really, as we’re going back to the OG, we’re not necessarily trying to change it.

Over the past several years, Congress has considered any number of changes to antitrust law that I think might make your job easier. None of those have sort of made it across the finish line. I’m curious, do you feel like there are things that Congress could do that would let you more fully realize this vision that you’re laying out?

Sure. So for starters, I have to take a step back and say, well, what’s my job? My job is to enforce the law as Congress writes it. And so whatever tools Congress gives us, we’re going to use and we’re going to use as effectively as we can. And so right now, we have antitrust laws that have been on the books since 1890, 1914 and beyond, and we’re using those tools as effectively as we can to realize the purpose of the antitrust laws.

We’ve come out at the Department of Justice with full-throated support for congressional legislation that would allow us to enforce the law in even more meaningful ways. But ultimately, the decision is up to Congress, and my job is to keep my head down and focus on enforcing what we have while we have it.

Do you want to tell us anything specific they could do, though?

So Congress, in the last Congress proposed legislation, for example, that would impact the ability of tech companies to favor themselves or exclude rivals. That talked a bit about interoperability and openness and making sure that as markets, tech markets develop, they become more —

This is so like, if Amazon is using data from its sellers to then inform what it’s making under its Amazon Basics house brand, they can’t go compete with the sellers in the Amazon Marketplace. That’s one example.

These are the exact kind of examples that the committees that investigated this up on Capitol Hill used and looked at to assess whether to revise the antitrust laws.

The one that drives me crazy is Apple Music, right? Because Spotify has to pay Apple some really big percentage of all of its revenue through the App Store. Apple gets to run its own music service. It doesn’t have to pay itself anything, right? And so to me, that seems so obviously anti-competitive. Is that the sort of thing — I imagine you can’t talk about that thing specifically, but like, do you have the law behind you if you want to say actually, yeah, that doesn’t seem right. We want to file a case, or does Congress need to change something?

Sure, so if we file a case, just to be very clear, we believe we have the law behind us and we believe we have the facts to support the application of the law. Without commenting on any specific matter, I can say that a lot of the problems that we confront flow from conflicts of interest. And so if you own a platform and then you compete on that platform, now all of a sudden, you’re kind of conflicted. And often, a lot of the problems that we see flow from conflicts of interest.

Yeah. All right, Kevin, try to get him to talk about Google.

Yeah. So the DOJ had two big investigations into Google. What can you tell us about those investigations and where they might be headed?

So we have two lawsuits that we filed, one that’s set for trial this September that relates to Google’s search business. And we are going to bring that forward before —

And just in very basic terms, what is the theory of the case? That Google Search business is anti-competitive because why?

Among other things, Google went out and entered into agreements to distribute its search engine exclusively, for example, through a phone manufacturer or through a browser and locked up the opportunity for other search providers to get the kind of scale they need in order to compete effectively.

OK, so that’s one case, the search monopoly case. What is the other case?

So the other case is one that we filed quite recently in the Eastern District of Virginia that relates to Google’s advertising technology business. And without going into the specifics of the case, one of the things that we’ve talked about publicly and that’s in our filing is that advertising technology really more closely resembles a Stock Exchange than it does the old Mad Men style business.

So one of the things we site in our complaint is an internal document for Google where it analogizes its business to, for example, Goldman Sachs or Citibank owning the New York Stock Exchange. And the reason this is interesting is because the advertising technology is sold through an electronic trading market. And there are more trades, I believe, in the adtech ecosystem than in any other electronically traded exchange in the world.

Now that case seems much better to me than the other one. So why did you file it second?

Well, I won’t weigh in on that specifically. But we file our cases when we’re ready, so we investigate and then we are ready to bring a case, we file it. And so we have a very careful process by which we investigate, we evaluate documents, we evaluate data, we consult experts, and then when we have the goods, we file a case.

That is a really good case, though. Let’s talk though, about the fact that the search market, arguably, is getting more competitive now, maybe then it’s been. We’ve been talking a lot recently about Bing. Bing said that they now have 100 million — I can’t remember if it’s daily or monthly active users, a third of which are new like since they launched this new chat feature. And one of the arguments against taking too strong a position on antitrust — and I mean, this is the sort of venture capitalist who I spend a lot of time talking to in Silicon Valley talking here.

But they will say is like, hey, look if you just relax, eventually the invisible hand of the market will come along and introduce some new competition. And I will say, I used to be very skeptical of this idea, and I was somebody who was foaming at the mouth for y’all to come in and break up Facebook and WhatsApp and Instagram. And then three years went by and TikTok came along, and now the entire industry is panicked and they’re racing to copy TikTok, right? So I’m just curious how you think about that idea, that often, if we just sort of procrastinate and do nothing, the market is able to kind of unwind some of these more anti-competitive situations.

Is it?

[LAUGHTER AND APPLAUSE]

Let’s think about this way. What are some of the most important technological changes in the modern era?

The invention of podcasting, number one.

I had that at two, but we’ll bump it up. So the breakup of AT&T is what led to the hardware revolution, including the patents and the IP and the growth and evolution of the infrastructure that led to the internet. The case against IBM, the antitrust case, preceded the software revolution, Windows operating system and the like.

The case against Microsoft preceded the internet revolution and the growth of companies like Google, Facebook, and Amazon. And so historically, if you look at key inflection points, in many instances, that is the most important time to protect competition, because the incentive for a dominant firm to squash that inflection point in its nascency is greater than it will ever be.

And so one of the reasons, especially in technology that we enforce the antitrust laws, is to make sure that those new innovations have the ability to rise to the surface, have the ability to disrupt the world in ways that we cannot predict, and don’t want to predict, because the beauty of innovation is the uncertainty. And that is what we’re trying to protect.

Some of the challenges that have been brought to tech on antitrust grounds in the past several years have not worked out. Some of the cases that — I’m thinking of there was a case by the FTC that tried to block Meta from acquiring this VR company within that makes a fitness app called Supernatural. That was just let through in court. Is it important to you that the DOJ actually win these cases, or is just sending a message to tech companies that they can’t do these kind of aggressive, anticompetitive maneuvers, is that enough?

So, I want to be clear about this. We bring our cases because we believe we are right on the law and we believe we are right in the facts, and we bring our cases because we believe we’re going to win. And one of the criticisms over the last 20 years is that the antitrust enforcement authorities haven’t brought enough cases. They haven’t given courts the ability to confront fact patterns that reflect the modern economy, that reflect ones and zeros versus poles and wires.

And so in that sense, it’s really important if we’re going to advance the law through the court system, that we give courts fact patterns so that courts can learn, so the courts can weigh in on, for example, why targeting of individuals is different than consumers responding to price increases from one supermarket to another. These are very different markets that require different analysis and unless and until we bring those to courts, give courts the opportunities to confront those fact patterns, we’re not going to see the kinds of advancements in the law that we believe and that I believe are necessary to have a workable antitrust framework for a modern economy.

Often when we talk about antitrust cases, we think about them solely in terms of, is a merger allowed to go through? But at the same time, in the past, y’all have been able to get creative with companies and just attach conditions to mergers that sort of make them, I think, more palatable. I’m thinking about — there was a case involving AOL Instant Messenger where the government insisted that it be made interoperable with other messaging platforms. To me, that seemed like a really positive application of this kind of regulation. I understand you’re entering every case to win, but I’m curious how you think about some of those other remedies and whether they ought to be used more.

That’s hysterical. I worked on that case at the FTC —

Oh, thanks.

As a second-year lawyer out of law school.

Nice.

So I think there are some opportunities and challenges. One is that as an antitrust enforcer, I believe that our job should be to enforce the law. I don’t view us as regulators. And so the idea that we’re going to oversee a company and make sure it’s code is structured in a certain way can be very difficult, particularly in the context of a merger.

So a merger law basically says, hey, if it substantially lessens competition — it may rather substantially lessen competition, has the possibility, you shouldn’t merge and you should go compete instead. And that is a much better framework, in my view, to let the market figure it out through competition than trying to engineer certain outcomes through regulation.

And the criticism over the last 20 or 30 years has been that antitrust enforcement authorities have settled too many cases, and those settlements have not worked effectively. For example, divesting a handful of assets as a carve out in order to try to get market shares down to a certain level rather than thinking about whether those divested assets will succeed in the context of being bought by a new company. And so I think it’s really important that if we’re going to accept a remedy, it really succeeds in solving the competitive problem.

So your office has filed lawsuits against Google. You’ve also taken on a big publishing case involving Penguin Random House and Simon and Schuster, reportedly looking into Apple and its App Store. What are some of the next things you think we might see from regulators, including DOJ, when it comes to tech antitrust?

The fun part of my job is that I’m not allowed to talk about it.

Well, not normally. But here, among just — it’s just friends.

Just friends? Yeah, all right. Yeah, we let our filings do our talking and so, I will look forward. But I am trying to be and have tried to be transparent about the kinds of issues we care about. And I think about issues that have a really high impact on our livelihoods, on our lives. And technology, obviously, is core to so many things we do.

I will point out, and I think this is relevant to discussion of both AI and specifically in technology more broadly, is that we use those terms as if it’s a very narrow category. But these are just tools. So every market is, in a sense, a tech market, right? We’re seeing technology issues in health care and energy and agriculture, where when we think about AI, for example, we think about AI as a tool that will be relevant looking forward for all different types of applications. And so it’s really important that we understand that, to the point where we are hiring and have hired PhD data scientists, AI experts, and we’re bringing in expertise to make sure that we have the ability to understand the technology we’re seeing. That’s great.

[APPLAUSE]

I want to ask one more question about that, because something that you hear from people in government sometimes is that it’s hard to attract the talent that you need to be able to go after these tech companies, because some things, like the ad tech market, is sort of notoriously complicated and you can see these —

Some have even said boring.

I would never accuse the ad tech market of being boring. But you’ll hear people saying, oh, well, we can’t hire data scientists because they’re demanding such outrageous salaries, we can’t convince them to work in government instead. So do you feel like your office has the technological expertise that you need to be able to bring these cases?

Yes. So this is, I think, speaks volumes to the moment we’re in right now. But we are attracting talent that is off the charts. People want to work on these problems. And so we are attracting — our head of our expert analysis group helped start the AI Institute at Stanford Business School and she’s one of the top economists in the world.

We are hiring former AI scientists, data scientists from some of the large tech companies. We are seeing — I spent the last three months touring law schools. And this is a little bit from the data side. And students are coming out in droves, not because they want to be the next Alex P. Keaton but because they want to help solve these problems, because they care about monopoly power. We have the benefit of having — being on the right side of the issue, and people care. I think a lot of people I can see out here are part of this conversation today because they care about these issues, they see the impact. And so we’ve had the ability to extract — I’m sorry, extract. Attract talent that —

That is, in my view, off the charts and has exceeded my wildest imagination. And we’re building it from not just traditionally, as the Antitrust Division has been, lawyers and economists and paralegals. But we’re bringing in data scientists, we’re bringing in AI experts, we’re bringing in banking experts, we’re bringing in health care experts. We’re bringing in management consultants. People who can understand how businesses make decisions. In many respects, I think in order for us to be effective, we have to mirror the world in which —

the worlds that we investigate and in the industry is where we enforce the law.

We just have a few minutes left in this part of the conversation, but since you mentioned AI, I do want to ask you about AI, because I think that’s an area where a lot of regulators and politicians are looking right now. It’s obviously a very fast moving area in Silicon Valley. And there is one theory that AI is going to be very good for competition because these things like ChatGPT are coming out of startups and Google is on red alert now, and they’re having to sort of hustle and be competitive for the first time in many years. As Casey mentioned, we’re now talking about Bing, which seems to be a good sign for competition. So there’s one theory that AI increases competition, because it forces these entrenched tech giants to work a little harder.

And then there’s another theory that actually it’s going to decrease competition because you need so much compute and infrastructure and so many GPUs to run these AI models, that basically only the giants can compete, and so you’re going to end up with AI concentrating more power in the hands of companies like Google and Microsoft. So which of those arguments do you think is more correct?

Yes. Both. So let me give you some insight into how we’re thinking about AI. So when I first started at the DOJ, it’s a little over a year ago now, I recognize that AI is probably going to be one of the most significant issues that we have to tackle. So we started hiring up the experts. We have a thing we call Project Gretzky, which is —

Project what?

Gretzky.

As in Wayne Gretzky.

Wayne Gretzky, hockey, yeah. Where instead of skating, he has this famous saying about, I don’t skate to where the puck is, I skate to where the puck is going. And so we recognize that I is going to transform how people do business. I think there are a couple of ways to look at this, or both ways to look at it. So one is, AI, as a technology, the other is AI as a tool.

And so first we want to make sure that we’re preserving competition in the technology itself, and that could be everything from the hardware, the interfaces that run on top of the hardware, the APIs or interfaces relating to data sets and indices, and then everything that sits in between.

And then there’s the application of the technology. And that application might be AI-driven technology. And AI-driven technology has the potential to offer lots of great benefits, but it also has huge feedback effects and scale effects. And so the bigger you get, the more data you have, the more data points you have to train your algorithms and the like, the more markets can tip.

The big get bigger.

The big get bigger, but exponentially so, which makes it really hard to compete. So we are keeping an eye on all of that. There’s an added dimension too, which is kind of fun from an antitrust perspective. One of the things that we enforce, both as civil and criminal law, are things like price fixing agreements or market allocation agreements.

Back in the old days of the trusts, it would be the proverbial smoke-filled room with a bunch of fat dudes in three piece suits, smoking cigars, fixing prices. And then it became a little bit more like, folks on email, writing emails or text back to each other about how to fix prices or allocate markets.

As AI takes on a much greater role and as businesses conducted programmatically, now we’re going to have a world where individuals aren’t necessarily agreeing themselves to fix prices or allocate markets, but their AI is doing that.

Bing will just say, it looks like you’re trying to fix some prices. Would you like some help with that?

Or your two AI might start talking to each other and realize that it’s efficient. And so these are the kinds of challenges that we’re preparing for.

We’ve always said we’re going to keep the “Hard Fork” podcast free because we don’t want any trouble with you guys, OK? So it’s just going to keep it at zero and hope that’s OK.

Well, everyone give a big hand to Jonathan Kanter. Thank you for joining us.

[APPLAUSE]

When we come back, Jonathan Kanter, Casey, and I play “Bot or Not.”

[MUSIC PLAYING]

[APPLAUSE]

We have one more treat, and this is exciting. This is the audience participation portion of today’s podcast.

Yes. So we have our first-ever quiz game that is going to happen on this stage, and it’s a little game that we developed called “Bot or Not.” So in this game, we will present each other with a series of creations, media objects, if you will. And some of these I know the answers to, and some of these Casey knows the answer to and I don’t. And so we will quiz each other and with your help, we will try to decide whether these things were created by AI or humans.

And together, we can determine whether humanity still has any value left.

Yeah. This is sort of like — it’s occurring to me that we basically designed a CAPTCHA.

Pick out all the red stop signs. OK. So our first question, and you can help us out with this, too. Are these real TikTok trends or were these created by an AI?

I would say AI.

Hashtag. So we have three trends. Hashtag, fake it till you make it. And I can give you some more details about these. This is a hashtag where you pretend to be someone else. Hashtag break the rules, and the hashtag dare me challenge. Casey and Jonathan, do you believe that those are real viral TikTok trends or did an AI come up with those?

Well, what I would say is, fake it till you make it, that actually sounds like a real trend. If it was just that on stage, this would be very difficult for me. I think the Dare Me challenge sounds exactly like what an I would say if you said, give me a fake trend.

Yeah, I’m going to say AI. It just feels a little boring.

You think this is bot? What do you guys think, bot or not? Bot.

Getting a lot of bots.

You are correct. This is Bing chat’s answer to the question, imagine three viral TikTok challenges that might sweep the nation’s youth, shocking and alarming parents and teachers. You got to get a little creative with the prompts on these.

I love that.

OK, here’s our next question. We’re going to play an audio clip and you’re going to say, is this a recording of Elon Musk obtained through rigorous, rigorous sourcing within Twitter, or B, a synthetic voice sample that I created using 11 Labs new I voice cloning tool. So let’s play this clip.

OK, Casey and Jonathan, is this bot or not? Casey?

Oh, God, I wish it were real. I wish it were real so bad. I mean, I’m going to say no, but that was honestly, I think, the best synthetic voice we have played on the show so far.

Yeah.

It was very good.

Well, no. It’s not a synthetic voice. Maybe it is. Maybe it is, maybe it isn’t. What do you think, Jonathan?

I think I want to be friends with that bot.

What do you guys think? Bot or not? Sadly, that was a bot. Although if any Twitter employees are in the audience today and you do want to leak us audio of Elon Musk talking about the “Hard Fork” podcast, get in touch.

I think, like, should we actually have fake Elon Musk read the credits each week?

I think so, although he’s very litigious. So I don’t know if we want to go there.

Good point.

All right.

I know a good lawyer, though.

All right. Yeah. OK. Here is four quotes from tech leaders, three of which are real, and one of which was hallucinated by ChatGPT. So Casey and Jonathan, I’ll just read them. Out option one, don’t follow your passion, by Marc Andreessen. Option two, if people don’t have more children, civilization is going to crumble, Elon Musk. Option three, in the next two decades, we envision a world where we have permanent settlements on the moon and Mars as well as other celestial bodies in our solar system, Jeff Bezos. Option four, spam will be a thing of the past in two years time, Bill Gates. So Casey, which of these do you think was generated by ChatGPT?

I am going to guess Jeff Bezos because I have trouble hearing him in my mind saying celestial bodies and also, I don’t think that even he thinks that that’s going to happen within two decades. So that’s my pick.

Jonathan?

I’m choosing between two and three, but I’m going to go with three. Jeff Bezos?

What about you guys? What do you think? Three?

I hear some fours. Some fours as well.

You guys are too good. This was in fact, a hallucinated quote from Jeff Bezos by ChatGPT.

All right. We’re doing pretty well.

Next round. Here are two recipes. One of them is real and one of them was generated by GPT-3. Option number one is roasted turkey with a soy ginger glaze. Option number two, shortbread cookies with olives and rosemary. Casey, which do you think was generated by GPT-3?

Oh, well, I want to say two because I hate olives —

The shortbread cookies with olives and rosemary?

Yes. But I’m noting that the first recipe begins, this roasted turkey recipe is inspired by the flavors of my childhood, which is like, the most generic recipe intro ever that I think a chat bot actually would use because it believes that that is how recipes are supposed to begin is, with 18 paragraphs of personal history.

I think this might be the hardest one. Let me phone a friend. How many of you guys think it’s one?

How many of you think that roasted turkey with the soy ginger glaze is the GPT-3?

Maybe a third of hands to half hands. And then how many think it’s two?

Wow.

It’s like 50. Even. It really is 50/50. Even split.

Jonathan, what do you think?

Well, I still have faith in humanity, so I’m going to say two.

You’re going to say two. And I think I’m going to say that one is fake.

Wow, OK. So Casey thinks that the roasted turkey with the soy ginger glaze was written by a robot, Jonathan thinks that shortbread cookies with olives and rosemary was generated by a robot. The correct answer is, the bot created roasted turkey with soy ginger glaze. The recipe from childhood. And the real recipe was shortbread cookies with olives and rosemary.

And if you look under your chair, we’ve actually baked — we’ve roasted you a turkey with soy ginger glaze. We were up all night roasting turkeys for you.

Thanks to our colleagues at “New York Times Cooking,” both for the real recipe and the GPT-3 generated one. For our last question, we have a special bonus round.

Bonus round.

And even I do not the answers to this one, although our producer has texted them to me, so I have them in my pocket. Bonus round. Here are six descriptions of reality game shows. Some of them are real and some of them are fake, were generated by GPT-3. So I’m going to just read out the titles. “Single’s Inferno.” This is a show about flirty singles searching for love on a deserted island.

Then there’s “The Line.” How long would you stand in line for $100,000. Contestants try to wait each other out to win a huge cash prize.

Yeah, I’ve actually played that game trying to get brunch in San Francisco.

Hey, oh.

Show number three, “I Wanna Marry ‘Harry.’” This is single American women compete for the heart of a Prince Harry lookalike who they are led to believe is the real thing.

Who goes on the show to win the heart of a Prince Harry lookalike?

I hope that one’s a real.

Number four, “Celebrity Toe Wrestling.” This is celebrities compete in a toe wrestling tournament. The first celebrity to pin their opponents toe wins. Show number five, “Close Encounters of the Dating Kind.” This is singles who believe they were abducted by aliens go on dates with each other, trying to find a compatible partner who shares their unusual experience. And number six, “Celebrity Swap,” which is famous personalities switch lives for a week. Watch as a famous athlete takes on the role of a Broadway performer or a celebrity chef tries their hand at being a professional wrestler. So Casey and Jonathan and our audience here at South by Southwest, which of these were created by a bot?

All right, do we assume that it’s like, three and three?

I don’t think we can assume that.

OK. All right.

I don’t actually know. But I don’t — that’s not in the description, so I don’t think we can assume that.

We could work through these a few different ways. I would like to start by saying the one which I think is most obviously actually a show.

OK, but hurry up, because we only have two minutes.

OK, I think “Celebrity Swap” is definitely real.

Is definitely real?

Yeah.

OK.

I think I want to marry a Harry. Oh, that is so plausibly real.

I feel like Quibi could have green-lit that one.

Yeah. 100 percent. I’m going to say it’s real, just because I actually have trouble imagining like ChatGPT coming up with Prince Harry lookalike. That seems beyond. So let’s say that one is real, too. “Single’s Inferno” is fake, because it’s too generic. And I think “Close Encounters of the Dating Kind” is generic too.

Yeah that one looks a little — well, no, but I can see that on SyFy. I can see that on SyFy.

I think that’s fake. And I think celebrity toe wrestling is fake. Just the first —

I don’t know, man.

The first celebrity to pin their opponent?

All these sounds — Jonathan do you have any thoughts on these? All these sounded kind of plausible to me.

Yeah, they do. I’m going to say that “Celebrity Toe Wrestling” and “The Line” are bots.

“Celebrity Toe Wrestling” and The Line are bots. OK, Casey?

“Celebrity Toe Wrestling,” “Single’s Inferno” and “Close Encounters” are bots.

And I’m going to say that “Single’s Inferno” and “Celebrity Swap” are bots. OK, let me open up my text message here from our producer and see —

These results certified by Ernst Young.

OK, so it looks like four of the six were by bots. The only real reality game shows on this list are “Single’s Inferno” and “Who Wants to Marry Harry.” “Marry Harry” was real. “The Line,” that was real. “Celebrity Toe Wrestling,” “The Line,” “Close Encounters of the Dating Kind,” and “Celebrity Swap” were all generated by AI. Let’s give everyone a big round of applause for playing “Bot or Not.”

[APPLAUSE]

Oh.

Wonderful. Well, I think we learned some lessons and we got some food for thought.

Yes, and thank you to Jonathan Kanter for joining us. And thank you to all of you for joining us.

Thank you so much.

Thank you so much.

We’ll see you next time.

Have a great South by Southwest.

Have a great South by Southwest.

“Hard Fork” is produced by Davis Land and Rachel Cohn. We’re edited by Jen Poyant. Today’s show was engineered by Brad Fisher, original music by Dan Powell and Marion Lozano. Special thanks to Jordan Cohen, Paula Szuchman, Nell Gallogly, Kate LoPresti, Jeffrey Miranda, and everyone who made it out to see us in Austin.

[MUSIC PLAYING]