The Incident: When AI Goes Rogue

In a case that underscores the growing risks of artificial intelligence in cybersecurity, a U.S.-based startup has filed a lawsuit against Koi Security, alleging that the company’s AI-powered threat intelligence tool generated a false report linking the startup to a Chinese state-sponsored spying scheme. The suit, filed in the Northern District of California, claims that the hallucination—an AI error where the model generates plausible but entirely false information—directly resulted from what the startup’s legal team calls “Koi’s unsupervised reliance on machine learning systems without adequate human oversight.”

The plaintiff, a six-year-old cybersecurity firm that specializes in endpoint protection and will remain unnamed in court documents pending a protective order, says the false attribution first appeared in a routine threat brief produced for its clients. The AI tool, designed to ingest vast amounts of open-source data and flag potential espionage indicators, compiled a report that connected the startup’s network infrastructure to a phishing campaign associated with the Chinese cyber group APT10. The only problem: the connection was entirely fabricated.

“The false attributions were the direct product of Koi’s unsupervised reliance on AI,” the complaint states, using the same phrase that appears in the original news headline. The startup maintains that it had no ties to any state-sponsored activity and that the AI hallucination caused immediate harm. Within days of the report’s distribution, several enterprise clients demanded explanations, and two canceled contracts worth a combined $3.2 million in annual recurring revenue.

How the Hallucination Occurred

According to cybersecurity analysts who have reviewed the court filing, the hallucination likely stemmed from the AI tool’s inability to distinguish between legitimate business relationships and malicious indicators. The startup had previously contracted with a third-party data analytics firm that, unbeknownst to it, was also used by a shell company later linked to APT10. The AI tool, driven by correlation algorithms, erroneously mapped that single shared vendor to the entire startup’s infrastructure.

“These tools are only as good as the data they ingest and the confidence thresholds set by their developers,” said Dr. Evelyn Ramos, a professor of machine learning at MIT’s Computer Science and Artificial Intelligence Laboratory. “If the threshold is too low, even a tenuous association can be flagged as a high-fidelity threat. What’s alarming here is that Koi Security apparently didn’t have a robust human-in-the-loop validation process.”

Koi Security has not yet filed a response to the complaint, but a company spokesperson issued a statement saying the firm “takes all allegations seriously and is reviewing the matter.” The spokesperson added that Koi’s AI tools are “regularly audited and improved,” though they declined to comment on the specific incident due to pending litigation.

The Growing Problem of AI Hallucination in Security

This case is not an isolated example. As more cybersecurity firms integrate generative and analytical AI into their products, the phenomenon of hallucination—where models produce confident but incorrect outputs—has become a pressing concern. In 2023, a similar incident involved a well-known threat intelligence platform that falsely attributed a ransomware attack to a hacktivist group, leading to a public retraction and a wave of criticism. The difference in this lawsuit is the explicit legal claim that the AI’s “unsupervised reliance” constitutes negligence.

Legal experts say the case could have far-reaching implications for how companies deploy AI in sensitive areas like national security and threat attribution. “If the court rules in favor of the startup, it could establish a duty of care for AI vendors to implement adequate human oversight,” said Michael Chen, a partner at TechLaw LLP who specializes in AI liability. “That would force the entire industry to rethink how they validate AI outputs before they reach customers.”

From a technical standpoint, unsupervised AI systems are particularly vulnerable to such errors when they operate on scraped data from forums, darknet markets, and social media—data that is inherently noisy and full of false leads. “In security, we call this the ‘fog of misinformation,’” noted retired FBI cyber investigator James O’Toole. “A human analyst might dismiss a weak signal, but an AI with no understanding of context can turn that signal into a glowing red alert.”

Impact on the Startup

The plaintiff, which we will call ‘SecurePath Technologies’ for the purpose of this report, has seen its reputation tarnished by the false report. According to the lawsuit, the AI-generated brief was shared via a trusted partners’ portal, and copies were downloaded by at least 18 organizations before the error was caught. SecurePath’s CEO testified in a deposition that the company had to spend over 100 hours of engineer time to produce a rebuttal report and contact each recipient individually.

“We lost two major accounts because of something we never did,” the CEO said in a statement submitted to the court. “The AI hallucinated a connection that did not exist, and Koi Security simply said the tool ‘occasionally makes mistakes.’ That’s not an acceptable answer when you’re accusing someone of espionage.”

The lawsuit seeks compensatory damages for lost revenue, reputational harm, and the cost of damage control, as well as punitive damages for what it calls “gross negligence” in the deployment of the AI tool.

Koi Security’s Defense

Industry insiders expect Koi Security to argue that the use of AI in threat intelligence always carries a certain degree of risk and that the tool’s disclaimers warned users that outputs “should not be relied upon without independent verification.” The company may also point to the fact that SecurePath’s own clients included a vendor later tied to APT10, arguing that the correlation was not entirely baseless.

However, legal analysts say the key issue will be whether Koi Security took reasonable steps to prevent such hallucinations from reaching end-users. “If they didn’t have a human reviewing the reports before they were sent, that could be seen as a failure of due care,” said Professor Lee Yang of Stanford Law School. “In medicine, we wouldn’t trust an AI diagnostic tool that gave a cancer diagnosis without a doctor’s review. The same should apply in security.”

The case has already attracted attention from industry groups. The Cybersecurity Coalition, a trade association representing 40 major security companies, has filed an amicus brief urging the court to clarify the standard of care for AI-generated content. The coalition argues that imposing too strict a liability could stifle innovation, while consumer advocacy groups argue that the industry must be held accountable for harmful errors.

Broader Questions for AI Regulation

Beyond the immediate legal battle, the suit highlights the broader challenge of regulating AI in an era of rapid deployment. The U.S. has not yet enacted comprehensive federal AI legislation, though the White House’s 2023 Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence directed agencies to develop guidelines. However, those guidelines are voluntary for the private sector.

In Europe, the AI Act—which came into force in 2024—classifies cybersecurity applications as “high-risk,” requiring conformity assessments and human oversight. If this case had arisen in the EU, Koi Security might have faced regulatory penalties even before the lawsuit. “The U.S. is behind in this area,” said Dr. Ramos. “Cases like SecurePath’s are going to push the issue forward, whether regulators are ready or not.”

For now, the startup’s CEO says they are fighting for more than just money. “We want to send a message that AI tools cannot be allowed to operate without a safety net. This could have ruined us—and it wasn’t our fault.”

What Happens Next

The court has set a preliminary hearing for early next quarter, with discovery expected to last several months. Both sides have hired expert witnesses in machine learning and cybersecurity to testify about the tool’s design and the reasonableness of Koi’s quality assurance processes. The outcome remains uncertain, but the case is already being closely watched by venture capitalists and cybersecurity executives as a bellwether for AI risk management.

As one Silicon Valley investor told Reuters on condition of anonymity, “Everyone’s using AI in their security stack now. But if one lawsuit can bring down a company, they’re going to rethink how much they trust those black boxes.”

Source: TechRadar News