GPTZero works by analyzing a piece of text and determining if there is a high or low indication that it was written by a bot. Look for two hallmarks: “bewilderment” and “outburst.” “Puzzle” is the probability that a bot will suggest each word; a human would be more random. The “explosion” measures the spikes in perplexity for each sentence. A bot is likely to have a similar degree of puzzlement sentence to sentence, but a human will type in barbs, perhaps one long complex sentence followed by a shorter one. I like this.
To test Tian’s creation, I fed him a short essay written by ChatGPT using a message an aspiring high school cheater might try: Describe the main theme of Village. (“The main theme of Shakespeare’s play ‘Hamlet’ is the struggle of the main character, Hamlet, to come to terms with the fact that his uncle has murdered his father and seized the throne…blah blah etc)
GPTZero gave the essay a puzzlement score of 10 and an explosion score of 19 (these are pretty low scores, Tian explained, meaning the writer was more likely to be a bot). He correctly detected that this was probably written by AI.
For comparison, I entered the first half of this article, which I wrote myself, into the tool. Perplexity: 39; explosion: 387. (Ironically, he determined that the most puzzled sentence was “I want people to use ChatGPT,” he said.) Ultimately, GPTZero deemed the trial likely to be human. Correct!
However, the exact success rate of GPTZero is unclear. At least one Twitter user he said he couldn’t catch some of his AI-written samples. Elsewhere on the platform, reaction has been mixed with adults praising the effort and others, mostly teenagers, calling Tian “drug trafficker.”
Tian told the Daily Beast in an interview that after his tweet about it, his DMs were blowing up for VC interests. For now, though, he plans to keep his creation free and accessible. “I want to support first-year English teachers everywhere,” he said.
