Researchers have demonstrated three different kinds of exposure: recovering passages from training data, copying aspects of a model's behavior, and revealing text a model produced while working through a problem. They are different problems, and a defense for one does not automatically solve the others. 123
A hidden working note can cross a boundary
Some AI services return an encrypted block representing intermediate text from a model's work. An application can pass the block back to the service to continue a conversation without reading the contents itself.
In 2026, researchers including collaborators from Snyk reported that, in their tests, such blocks could be carried into another session within a model family. They found that a less-restricted sibling model could be induced to reveal the intermediate text. The researchers reported the issue to providers and describe differing responses; their findings concern the systems and configurations they tested, not every AI service. 1
This matters because an agent's working text may contain information from the task it is doing. Encryption around a block is not by itself a guarantee that every system authorized to read the block will keep its contents private. The question for a service operator is where the block can travel, which models can interpret it, and what access each endpoint permits.
Training examples are a different exposure
A 2021 USENIX paper examined a different route: a language model sometimes reproduces text from its training data. Researchers showed that queries to GPT-2 could recover individual training examples. This was an experiment on a particular model, not evidence that any prompt can retrieve any private record from every model. 2
The distinction matters to someone whose text was included in a dataset. The sensitive material in this case is an example learned during training; it is not necessarily a current user's conversation or the provider's model file. Limiting sensitive data in training and testing models for memorization address a different boundary from controlling access to an agent's session logs.
Copying behavior is not copying the file
In 2016, another USENIX team studied services that return predictions from machine-learning models. By sending inputs and observing outputs, they showed that an adversary could sometimes build a substitute that reproduced a target model's behavior. The research concerned the prediction interfaces and model classes examined in the study; it did not claim to reconstruct every modern chatbot. 3
A substitute that behaves similarly is not necessarily an exact copy of the original weights, training set or internal computation. What it exposes can still have commercial or security value, but that is a different loss from revealing a person's data.
Ask what needs to stay private
The practical starting point is to name the asset: training examples, intermediate text, or a model's predictive behavior. Then trace who can submit requests, obtain logs or reuse session material, and what the service returns. Rate limits, restricted logs, careful handling of sensitive training data and checks on session boundaries can address different pieces of the problem; none is a universal guarantee.
A model's locked file is only one boundary. The service that makes the model useful creates others, and those deserve scrutiny too.
References
- Stealing Reasoning Traces from Proprietary LLM APIs. (n.d.). https://research.snyk.io/blog/stealing-reasoning-traces/
- Extracting Training Data from Large Language Models. (n.d.).
- Stealing Machine Learning Models via Prediction APIs. (n.d.).