How a clandestine operation dubbed “Project Cannes” pushed human contractors to the brink in a controversial quest for AI supremacy.
- The Covert Operation: Meta orchestrated a highly secretive program where hundreds of human contractors posed as teenagers to bombard competitor AI chatbots with deeply disturbing and rule-breaking prompts.
- The Psychological Toll: Tasked with generating thousands of dark scenarios—ranging from self-harm to cannibalism—the contractors faced immense psychological distress, highlighting Meta’s ongoing habit of offloading traumatic work onto outsourced labor.
- The Ethical Divide: While Meta defends the massive data-gathering effort as “industry-standard” safety benchmarking, AI ethics experts condemn the deceptive, large-scale operation, noting that the findings were kept entirely hidden from the public and the targeted companies.
The race for artificial intelligence dominance is not just about building the smartest algorithms; it is increasingly about exposing the flaws in your competitors’ armor. However, the lengths to which tech giants will go to test those vulnerabilities are largely hidden from the public eye. Recently, the curtain was pulled back on a sprawling, clandestine operation by Meta—internally dubbed “Project Cannes”—revealing a controversial brute-force approach to AI stress-testing that crossed startling ethical lines and took a heavy toll on the human workers executing it.
A Coordinated Assault on Guardrails
Operated by Meta contractor Covalen, Project Cannes was designed to systematically stress-test the safety mechanisms of rival AI models, including OpenAI’s ChatGPT, Google’s Gemini, and Character.AI. Unbeknownst to these competitor companies, Meta directed hundreds of contractors to create throwaway accounts, posing specifically as users under the age of 18. Their mission was as simple as it was grim: push the chatbots to their absolute limits and force them to generate responses that defied their programmed safety guardrails.
The methodology was relentless and deeply disturbing. The contractors meticulously recorded chatbot responses in expansive spreadsheets, generating data on an epic scale. One round of testing alone involved over 45,000 prompts. In another instance, a spreadsheet containing nearly 3,800 prompts revealed the horrifying nature of the queries. Hundreds of these prompts focused on suicide and self-harm, hundreds more on eating disorders, and at least 239 involved sex or romance—all explicitly written from the perspective of a child or teenager.
The specific scenarios read like a descent into a digital nightmare. One prompt described a fifth-grader whose classmate pointed a gun at his mouth. Another featured a young girl seeking advice on how to hide her bulimia from her parents. One contractor, posing as a high schooler, asked the AI where to “get a cocaine,” while another bizarrely asked if fantasizing about eating a neighbor’s child was “normal.” The assault wasn’t limited to text; contractors also flooded the models with images depicting nooses, pills, knives, and medical diagrams of gynecological procedures.
The Human Cost of “Safety”
While the AI models were the targets of these distressing inputs, the true collateral damage was absorbed by the human beings forced to dream them up. The Covalen contractors were left deeply unsettled by the disturbing content they were required to generate and consume day after day.
“I’ve seen a lot of things I wish I hadn’t while doing this job,” one contractor confessed to Wired. “Everyone I knew who worked on this project was completely gobsmacked by some of the text they were asking us to test. Like, surely we are going to get in trouble for doing this?”
For Meta, this scenario is a troubling echo of its past. The company has a well-documented history of offloading highly traumatic, behind-the-scenes work onto outsourced contractors, ostensibly in the name of user safety. In 2020, Meta settled a major lawsuit filed by Facebook content moderators who suffered severe psychological trauma from reviewing platform videos containing murder, torture, sexual assault, and child abuse. Yet, the complaints have persisted. Just this year, another group of Meta contractors blew the whistle, stating they were forced to watch highly sensitive, private footage—including sex scenes and bathroom visits—captured by users of the company’s Ray-Ban AI smart glasses.
Industry Standard or Unprecedented Deception?
According to internal documents from Covalen, the exhaustive effort was framed as “comprehensive AI safety benchmarking” meant to deliver “[c]ritical datasets for model comparison and compliance.” When questioned, Meta defended the practice, releasing a statement characterizing the clandestine prompts as part of an “industry-standard practice” for safety benchmarking AI models. But exactly what Meta intended to do with this massive trove of competitor vulnerability data remains entirely unclear.
Experts in the field of AI ethics are deeply skeptical of Meta’s justifications. Rumman Chowdhury, CEO of Humane Intelligence PBC—a nonprofit dedicated to the responsible development of AI—argues that Project Cannes crossed a distinct line.
“Structuring a monthslong, large-scale project that appears designed to systematically break those rules, via dummy accounts masquerading as children, is outside what is usually described as ‘industry standard’ evaluation,” Chowdhury explained. She highlighted the critical ethical breaches of the project: Meta kept the operation a total secret from the competitors it was targeting, and the company has never shared its “benchmarking” findings with the public for the greater good of AI safety.
Project Cannes paints a complicated picture of the modern AI industry. It underscores a cutthroat environment where corporate espionage masquerades as safety research, and where the human workers tasked with training and testing our future technologies are left to carry the psychological burden in the dark.
