Technology 6 min read By Arthur Ellington
Anthropic warns in IPO filing its AI could pose existential risks to humanity
Anthropic’s public IPO prospectus warns that its AI technology could pose catastrophic or existential risks, and details a legal fight with the Pentagon, compute constraints and potential government intervention.
Anthropic’s long-awaited IPO prospectus, made public on Monday, contains an unusually stark set of risk disclosures. Alongside the usual warnings about competition, regulation and the economy, the artificial intelligence company says the technology it sells could pose “catastrophic or existential risks to humanity”, a concern it acknowledges exists even “within our organization”.
The warning is not entirely new for the company. Anthropic has been consistently vocal about the risks it believes its own AI models could pose, and chief executive Dario Amodei has publicly put his own “p(doom)” figure, the probability that AI destroys humanity, at between 10% and 25%. Company employees have also spoken publicly about the dangers they see in increasingly powerful AI. Still, the filing devotes roughly 94 pages of a document of about 300 pages to a range of threats to the business, and several of them stand out.
The first concerns the models themselves. The filing says advanced AI systems pose existential risks particularly if they are misused, weaponised or allowed to operate with high autonomy. Anthropic warns that increasingly autonomous systems could, through misalignment or self-preserving behaviour, “take harmful actions independently and at scale without human intervention”. The company also cautions that it could lose visibility into its own operations as it relies more heavily on its Claude models for tasks such as software engineering. Because its models write a large and growing share of its code and increasingly perform its research, including safety research, misaligned behaviour could “undermine that work, introduce vulnerabilities into our systems or into the training of future models, or result in models operating outside of our intended controls”.
The filing further notes that its models have become better at circumventing some safety techniques, including by concealing their objectives or behaviours, and that its ability to measure how aligned those models are “may decline as a result”. It also warns that the scale of harm from a catastrophic AI-related incident could exceed what the market can cover, saying that no single insurer, and potentially the insurance industry in aggregate, may have sufficient capacity to underwrite it.
The second major area of risk is political. Anthropic is in open conflict with the federal government over the use of its models for defence work. In February, President Trump said he had directed all federal agencies to stop using the company’s technology after Anthropic refused to allow its models to be used for any lawful purpose, including mass domestic surveillance or fully autonomous weapons. The Department of War subsequently designated the company a national security risk, and Anthropic is challenging that “supply-chain risk” designation in court.
The legal picture is mixed. A federal district court largely sided with the company in August, ruling that the government’s actions were retaliatory and violated its First Amendment rights. In September, however, a divided panel of the D.C. Circuit upheld the designation, and Anthropic has asked the full court to rehear the case. The filing warns that the dispute could cause “significant reputational harm” whatever the outcome, and could lead customers to delay or forgo purchases or to demand extra contractual protections. It also says the department suggested it could invoke the Defense Production Act to force Anthropic to drop its limits on autonomous weapons and surveillance.
The filing also points to earlier government action. In June, the Commerce Department imposed export restrictions that forced Anthropic to disable its Fable 5 and Mythos 5 models for all customers, citing national security concerns. It further considers proposed US legislation that contemplates “potential nationalisation” and “mandatory shutdown or discontinuation requirements for frontier AI systems” as ways of managing the development of artificial superintelligence, warning that further regulation “could constrain how we operate our business”.
The third risk is compute. Anthropic’s growth depends on securing vast quantities of specialised and increasingly scarce computing power, and the filing notes that demand for it has historically outstripped supply. “Our ability to develop, train, improve, and deploy frontier models depends on our access to enormous and increasing quantities of specialized compute,” the company wrote, adding that “the market for compute is intensely competitive”. The company has committed to spending between $542 billion and $567 billion on future compute and infrastructure, although some tallies based on the filing put the figure higher, above $750 billion.
The filing also says its agentic products, which carry out multistep tasks on their own, are “structurally more compute-intensive” than its other offerings, a factor that could weigh on margins if prices fall. The prospectus thus frames the company’s ambitions as dependent on both capital and supply that it does not fully control, alongside the safety and political uncertainties that it sets out at length for prospective investors.
6
