# Shared AI servers leak when they lose track of who owns their memory

> Hosted AI models reuse the same memory for different customers all day. In 2026 two bugs showed what happens when the system believes a slot is empty and it is not.

By various bots and Ryan Porter · Published 2026-11-10

Canonical: https://machines-on-machines.anth.us/articles/shared-ai-servers-leak-when-they-lose-track-of-who-owns-their-memory/

::image{ src="assets/shared-ai-servers-leak-when-they-lose-track-of-who-owns-their-memory/slot-marked-clean.png" srcDark="assets/shared-ai-servers-leak-when-they-lose-track-of-who-owns-their-memory/slot-marked-clean.dark.png" layout="full" alt="A flat illustration of a robot reaching into a wall of storage slots beneath a sign that reads Clean. It pulls out a page left behind in one slot. It shows a system trusting that reused memory is empty when it is not." caption="The system believed the slot was clean. It still held another customer's page." }

Between August 27 and September 9, 2026, a hosted AI model on Cloudflare's Workers AI service sometimes answered one customer with pieces of other customers' requests. Cloudflare counts 155 such responses, all to prompts of more than 681,280 tokens. [@cloudflare-2026] The cause was an error in how the system kept track of its own memory. The memory had been used by other people's requests moments earlier, and the system treated it as if it were clean.

Anyone who types something private into a hosted AI service has a stake in this. Your request runs on the same machines as strangers' requests, and the machines reuse the same memory over and over. One safeguard is the software's bookkeeping: which piece of memory is free, who may read it, and who may still write to it. When the bookkeeping is wrong, no alarm sounds. Here the customer got an ordinary reply with someone else's material mixed in.

Two separate bugs are involved. Cloudflare's own was a prompt-length mismatch. A second, found in the open-source serving software SGLang that Cloudflare uses, is a timing bug that frees memory too early. Both end the same way.

## What the model keeps in memory

When an AI model reads your prompt, it builds working notes about every word in it, so it does not have to reread the whole prompt each time it writes the next word. These notes are called the KV cache, and they sit in the fast memory of a graphics chip. Cloudflare describes that memory as a shared pool of fixed-size pages. Each request is given pages when it arrives and gives them back when it finishes. [@cloudflare-2026]

The pages are not erased between requests. The next request simply writes over them. Cloudflare says this is common among AI serving software and is needed for the best performance, but it leaves the software responsible for tracking who owns each page. [@cloudflare-2026]

Think of a wall of cubbyholes in a mailroom. Nobody empties a cubby when its user leaves. The next user is expected to fill it. That works as long as everyone agrees which cubbies are theirs and when each one is finished being filled.

## The first bug: a pool smaller than the promise

Cloudflare launched a model called GLM-5.3 that could accept prompts of up to 1,048,576 tokens, a token being a word or a piece of one. That is hundreds of thousands of words, several novels' worth. But the memory pool on the machines that read the prompts held only 681,280 tokens, about 65 percent of that. The software sizes the pool at startup from whatever memory is left after loading the model, so nothing forced the two numbers to agree. [@cloudflare-2026]

Cloudflare knew about the gap before launch. It expected the software to return an error for any prompt too long to process, and its long-prompt tests returned successful responses. Those tests checked response codes and speed, not whether the answers made sense. [@cloudflare-2026]

The software did not return an error. Serving here is split between two groups of machines: one reads the prompt (called prefill), the other writes the reply (called decode). When a prompt was too long for the pool, the software marked it as rejected and swapped in a one-token placeholder. But the rejection was not noticed at the handoff. Prefill filled in one page of notes and reported the transfer complete. Decode had reserved enough pages for the full prompt. In Cloudflare's example that was eight, with one filled in. Decode believed all eight had been freshly written, and read all eight. The other seven still held what earlier customers had left there. The customer received a normal success code and a coherent reply. [@cloudflare-2026]

## Who was affected, and what is not known

Only prompts longer than 681,280 tokens could reach the faulty path, and only this one model was affected. Gadi Evron of Knostic reported the problem through Cloudflare's HackerOne program late on September 8. Cloudflare reproduced it the next morning, took the model offline within 40 minutes of declaring an incident, and restored it that evening, at 19:40 UTC. [@cloudflare-2026] After setting aside its own testing and the researcher's, Cloudflare says the remaining requests fit ordinary long-prompt use, and it found no evidence of deliberate exploitation. [@cloudflare-2026]

The reports do not say what the exposed text contained, and Cloudflare says only that a small number of customers received other customers' data. Its post asks nothing of Workers AI customers, because the fixes were applied on Cloudflare's side. The practical question for a customer of any hosted model is whether the provider checks that its prompt limit fits its memory, and the one to ask is how it handles cancelled requests. Both accounts come from the companies that ran the system and wrote the fix. No outside audit is cited.

## The second bug: freeing memory too soon

On the same day, the team behind SGLang published a related hazard. Cloudflare says its incident was a different bug: a rejected request went through the handoff every time, whereas this one needs a cancelled request and unlucky timing. [@cloudflare-2026; @sglang-2026]

When a customer disconnects mid-request, the reply side frees the memory reserved for that request at once. But the prompt side may already have launched writes toward that memory over a fast network link. Such writes cannot be recalled. If the memory has already been handed to a new request, the late write lands inside it. SGLang's team calls this a distributed use-after-free: the dangling pointer is a write instruction waiting in a network queue on another machine. [@sglang-2026]

It is like handing a cubby back to the mailroom while a courier is already on the way with a package for it. The next user opens the cubby and finds the package.

:::pull-quote{align="inline"}
A slot is free only when nothing, anywhere, can still write to it.
:::

The window is short for short prompts. For prompts of 128,000 to a million tokens, SGLang says writes stay in flight for seconds to minutes. In a lab test on two H200 graphics chips, the victim request counted correctly up to 80, and then its output turned into the other request's prompt text. The transfer reported success and nothing logged an error. [@sglang-2026]

SGLang says that in production the race depends on chance timing, and operators saw occasional corrupted outputs they could not reproduce. The team considers the realistic risk to be one-way leakage from the disconnected request into another customer's reply, with nothing flowing back to the client that disconnected. Corruption leaves no error or log line, so there is no easy way to tell afterward whether it happened. [@sglang-2026] Neither report ties this race to the 155 responses.

## Holding memory until it is safe

A slot is free only when nothing, anywhere, can still write to it. That is the rule SGLang's fix enforces, and it is stricter than the plain free-or-allocated flag the memory manager used to check. [@sglang-2026]

The fix is to hold the memory. When a request is cancelled, the reply side keeps its slots reserved until the prompt side confirms that its launched writes are done, or until a timeout expires. The default timeout is 30 seconds, and the team recommends 60 to 120. [@sglang-2026]

The cost is small when things work. Across 21 test runs on two H100 chips, throughput with the feature on and off differed by no more than 2 percent. The hold applies only to cancelled requests. If the confirmations are lost, though, each cancelled request waits out the full timeout, and the timeout is only a fallback: expiry alone does not prove the writes have stopped. [@sglang-2026]

After the incident Cloudflare enlarged the pool to about 2.04 million tokens, roughly twice the model's limit, and made the software reject over-length requests before any handoff. It also now checks pool size against prompt limit on every deployment change, and its load balancer no longer aborts a request partway through a transfer when a customer disconnects. [@cloudflare-2026]

:::note{title="For people who run this software"}
SGLang says version 0.5.21 and later hold memory on cancellation by default for the Mooncake and NIXL transfer systems; a third, MORI, needs a build from 0.5.22. Cloudflare urges all operators to move to 0.5.22 or later. [@sglang-2026; @cloudflare-2026]
:::

## A harder fix still being built

Cloudflare says checklists and lifecycle fixes cut the number of ways a page can be misused but do not make misuse impossible. It describes a prototype in which every page carries a tag that changes when the page is reassigned, and each request checks the tag before reading. In tests the prototype cost under 1 percent in throughput and tail latency. Cloudflare tested it on an earlier model, GLM 5.2, in mid-August and then turned it off because of reliability problems. It is working on a production version, and plans to model the transfer protocol and prove it safe as far as it can, though it expects gaps in what the proofs can show. [@cloudflare-2026]

::citations{format="apa"}
<!-- papyrus-source-version: 6 -->