ChatGPT has become an essential part of our daily life right now. Most of us use it on a daily basis to solve mundane tasks or get guidance on how to tackle complex problems, get advice on decisions, etc. More importantly, AI-assisted typing has become the norm for most, and we’re even starting to see the effects. I already eat companies started to replace their redactors with ChatGPT.
While GPT models have proven to be useful assistants, they have also presented challenges, such as the proliferation of fake news and technology-assisted plagiarism. Cases of AI-generated scientific abstracts misleading scientists have led to a loss of confidence in scientific knowledge. So it seems that AI-generated text detection will be crucial as we move forward. However, it is not easy, as it poses fundamental difficulties, and progress in detection methods lags behind the rapid advancement of AI itself.
Existing methods, such as perturbation-based approaches or rank/entropy-based methods, often fail when the probability of the token is not provided, as in the case of ChatGPT. Furthermore, the lack of transparency in the development of powerful linguistic models poses an additional challenge. To effectively detect GPT-generated text and match advances in LLMs, there is a pressing demand for a robust detection methodology that is explainable and capable of adapting to continuous updates and improvements.
So at this point, the need for a robust AI-generated text detection method is increasing. But we know that LLMs are advancing faster than screening methods. So how can we come up with a method that can keep up with the advancement of LLMs? time to meet DNA-GPT.
DNA-GPT addresses two scenarios: White box detection, where access to the model output token probability is available, and black box detection, when such access is not available. When considering both cases, DNA-GPT aims to provide comprehensive solutions.
DNA-GPT is based on the observation that LLMs tend to decode repetitive n-grams from earlier generations, while human-written text is less likely to be decoded. Theoretical analysis focuses on the possibility of AI-generated text in terms of true positive rate (TPR) and false positive rate (FPR), adding an orthogonal perspective to the current debate on detectability.
The assumption is that each AI model has its distinctive DNA, which can be manifested in its tendency to generate comparable n-grams or in the shape of its probability curve. Then, the detection task is defined as a binary classification task, where given a text sequence S and a specific LM language model like GPT-4, the goal is to classify whether S is generated by the LM or written by humans.
DNA-GPT is a zero-shot detection algorithm for texts generated by GPT models, which adapts to both black-box and white-box scenarios. The effectiveness of the algorithms is validated using the five most advanced LLMs on five data sets. In addition, the robustness of the algorithm is tested against revised text and non-English text attacks. In addition, the detection method provides the ability to get patterns, allowing identification of the specific language pattern used for text generation. Finally, DNA-GPT includes provisions for providing explainable evidence for screening decisions.
review the Paper and Github. Don’t forget to join our 23k+ ML SubReddit, discord channel, and electronic newsletter, where we share the latest AI research news, exciting AI projects, and more. If you have any questions about the article above or if we missed anything, feel free to email us at [email protected]
🚀 Check out 100 AI tools at AI Tools Club
Ekrem Çetinkaya received his B.Sc. in 2018, and M.Sc. in 2019 from Ozyegin University, Istanbul, Türkiye. She wrote her M.Sc. thesis on denoising images using deep convolutional networks. She received her Ph.D. He graduated in 2023 from the University of Klagenfurt, Austria, with his dissertation titled “Video Coding Improvements for HTTP Adaptive Streaming Using Machine Learning”. His research interests include deep learning, computer vision, video encoding, and multimedia networking.
