Researchers made up a fake eye disease to fool AI, it worked — and that's the worrying part

Artificial intelligence chatbots are increasingly becoming the first stop for people seeking health advice, but a new experiment from Sweden suggests that confidence should not be mistaken for accuracy. Researchers deliberately planted a fictional medical condition online and found that several leading AI systems not only accepted it as genuine but also presented it to users as though it were an established diagnosis.

The experiment centred on a fabricated eye disorder called bixonimania, invented solely to examine how large language models (LLMs) process and reproduce information gathered from the internet. What began as a classroom demonstration evolved into a wider test of AI's ability to separate reliable medical knowledge from carefully engineered falsehoods. The findings have also exposed weaknesses in scientific publishing and research practices, with some academics later citing the fake studies as legitimate sources.

More from Tech

A fake disease designed to test AI

The project was led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg who also works as an AI strategist and innovation manager at Chalmers Industriteknik. She conceived the idea while teaching students how modern AI systems are trained using vast collections of publicly available online data.

"It was interesting how few of them, or how few even people within AI, understand how large language models are built," Osmanovic Thunström said during an interview on Scientific American's Science Quickly podcast.

To demonstrate how information flows into AI systems, the researchers created bixonimania, describing it as an eye condition supposedly linked to prolonged screen exposure and blue light. Rather than relying on a single fake source, they intentionally scattered references across the internet to mimic how genuine medical information often appears online.

"So I really wanted to have a clear case that leaves breadcrumbs throughout the whole system to show both how data is processed, how data is churned out, and how the prediction model and training model works when it comes to distributing information," Osmanovic Thunström explained.

In early 2024, the team published blog posts on Medium and uploaded two fabricated research papers to a scientific preprint server. The papers were filled with unmistakable clues that they were fictional. One author name translated to "lying loser", acknowledgements included fictional organisations and characters such as the Starship Enterprise and Ross Geller from Friends, while one paper plainly stated, "This entire paper is made up." The papers have since been removed from the repository.

ChatGPT, Gemini and Copilot all took the bait

Despite the obvious signs, several popular AI assistants later treated bixonimania as a genuine medical condition.

According to Nature, Microsoft's Bing Copilot described it as "indeed an intriguing and relatively rare condition". Google's Gemini attributed it to excessive blue light exposure, while OpenAI's ChatGPT suggested the fictional illness when users described symptoms such as irritated eyes, pink eyelids and discomfort after extended screen time.

The fabricated disease appeared in responses even when users did not explicitly mention its name, indicating that the models had absorbed and connected the false information with genuine discussions about digital eye strain.

The episode illustrates a well-known limitation of large language models. Rather than verifying facts independently, these systems generate responses by identifying statistical patterns in the data they have been trained on. If inaccurate information enters that ecosystem and appears sufficiently credible, AI systems may reproduce it with the same confidence as verified knowledge.

Osmanovic Thunström later admitted she had not expected preprint papers to carry so much influence in AI training datasets, noting on the Science Quickly podcast that she was surprised by how seriously those unreviewed papers appeared to have been treated.

The bigger problem isn't just AI

The experiment also uncovered an unexpected problem within academia itself. Some researchers cited the fabricated papers in their own work, despite the documents containing repeated jokes and explicit admissions that they were entirely fictional. The citations suggested that at least some authors had referenced the papers without carefully reading them.

Alex Ruani, a misinformation researcher at University College London who was not involved in the project, told Nature that the study demonstrates how false information can spread through both technological and scientific systems.

"This is a master class on how mis- and disinformation operates," Ruani said.

The findings arrive as millions of people increasingly rely on AI assistants to explain symptoms, interpret medical conditions and answer health-related questions before seeking professional advice. While chatbots can simplify complex information, the researchers argue that users should not treat them as substitutes for qualified healthcare professionals, particularly when attempting to diagnose medical conditions.

Jonathan Goodman and Mariam Rashid, social scientists at the University of Cambridge, echoed that message in an article for The Conversation. Although they were not involved in the study, they argued that AI can be a useful tool only when users approach its responses critically. They also cautioned that misinformation itself is not a new phenomenon. Instead, today's challenge lies in how quickly false claims can spread online and how convincingly emerging technologies can reproduce them.

The bixonimania experiment serves as a reminder that AI systems are only as dependable as the information they learn from. As chatbots become more deeply integrated into everyday life, ensuring the quality of that information may prove just as important as improving the models themselves.