In 1950, Alan Turing asked whether machines can think. Rather than argue over definitions, he turned the question into a party game, where an interrogator trades typed messages with hidden players and tries to work out who is who. Since then, chatbots have gotten so good that they have already fooled humans in modern versions of the test.
So, four researchers from Paris Cité University asked the obvious follow-up question in a new study published in Computers in Human Behavior Reports: can AI at least catch its own kind?
How did the AI detectives do?
The team put seven chatbots, including ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3, in the interrogator’s chair. Each could ask up to 10 questions and was warned that participants might lie. The hidden players were 20 human volunteers and the same seven models. Every bot played twice, once with a short instruction to act human and once with a detailed persona of a slightly reserved 21-year-old who jogs, plays jazz trumpet, and makes the odd typo.
The bots were almost untouchable. Out of 98 tests with AI participants, the evaluators caught a machine only three times, and never once when the bot used the detailed persona. On average, AI participants were rated more likely to be human than the actual humans were.
The few bots that got caught didn’t slip up on what they said. They gave themselves away with overly neat, structured answers. The detailed persona wiped out those clues and kept replies to about 39 words, practically the same as the human average of 38, which is why detection dropped to zero.
My favorite detail? Claude initially refused to pretend to be human until the researchers explained it was part of a research study.
Why should you care?
According to the paper, bots already make up more than half of all web traffic. If you hoped AI would one day filter out scam bots and fake accounts for you, this study is a reality check. The researchers suggest we will need safeguards like cryptographic signatures and human oversight alongside AI detectors.
Turing wondered if machines could pass as human. Seventy-six years later, they’re so convincing that even other machines can’t tell.
RELATED COVERAGE: