Is This The Real Life - AI Content
Is this just fantasy? Obviously that's a quote from Bohemian Rhapsody by Queen and we can all detect that, but can we reliably detect what's AI generated, and what's created by the human hand?
There are AI detection tools out there, the best known being ZeroGPT. We know that Google checks content to see if it's AI generated as part of its commitment to penalising websites that over-use AI to create content at the expense of human jobs. Google being Google, however, we don't know exactly what tool they use but as experts in building algorithms we can be sure it's pretty sophisticated.
Researchers at the Lyle School of Engineering, Southern Methodist University, have been looking at how well AI models can detect content generated by itself – essentially whether they can recognise their own output as AI generated compared with human generated output. The results found that both ChatGPT and Bard were reasonably good at detecting their own content, while a third model, Claude, was not so successful at recognising its own writing style.
This seems worryingly like sentience on the surface of it, but really it's just a case of these models knowing what “artifacts” they each use. Artifacts are signals or signs of a particular AI model that arise because of how it's trained, a bit like being able to tell what country someone is from by how they react in situations requiring an orderly queue – artifacts are a sign of the “culture” of the AI model.
This research project is one of the first that aims to test these Large Language Models' (LLM) ability to detect their own content and that generated by other AI models. The very action of this project trains the models to become better at detecting their own content. This means that very soon you'll be able to use an LLM to detect its own content and that of other models, and perhaps this will lead to LLMs creating new artifacts, or new writing styles that are harder to detect?
Humans can develop the skill of authorship detection too – most teachers and lecturers learn the individual writing styles of their students and can detect plagiarism pretty well without detection software. If you're a keen cruciverbalist (a crossword enthusiast) you'll be able to tell who has set today's crossword just from the style of the clues. If you think about your group chat conversations you can probably tell who has said what without looking at their username just from the “artifacts” each individual uses. It might be an anachronistic spelling, a favourite punctuation character or a certain je ne sais quoi about the style of writing.
Those of us whose jobs don't expose us to large volumes of content from the same hand might not be as skilled at detecting AI generated from human content, but that is what ZeroGPT and other detection software is built for. As LLMs get better at detecting their own content, and that of others, will humans be able to keep up? We think that it'll be commonplace to use AI to check for AI generated content but that some humans with a particular flair for detecting writing conventions and those tell-tale clues of personal style will still be in high demand.
