Skip to main content
Privacy & Security

A privacy instruction could stop an agent naming someone while it still found them

In AgentDoxx's synthetic interview tests, some setups never returned the correct name but still surfaced the person in search results.

8 Nov 2026

An AI agent can refuse to tell you who gave an interview after it has already pulled up pages about the right person. Researchers Jianing Wen and Tianshi Li report that gap in a study of agents with web search. Some configurations returned no correct names while retrieving information about the subject in as many as 62 percent of transcripts.

In some configurations the answer named no one correctly, while subject information appeared in search results for up to 62 percent of transcripts.
In some configurations the answer named no one correctly, while subject information appeared in search results for up to 62 percent of transcripts.
Results from configurations that did not correctly name the subject. · Drafted by AI. Approved by Ryan. Source: Wen and Li, arXiv:2610.05586.

The benchmark used 822 synthetic interviews grounded in public information about real people. The authors knew the intended identities, so they could compare what came back from the search tool with what the agent ultimately said. The test covers a deliberately constructed set of transcripts. Its percentages do not measure exposure for interview data in general.

The search and the answer are separate events

A guardrail that blocks names in a final answer can keep those names out of the reply. It does not stop clues from leaving through a search query or the result from appearing in a tool log. Who can see those records depends on the system.

The study makes the distinction measurable. An evaluation can check whether the answer names someone, whether the agent ever retrieved that person's information and what records those steps leave behind.

Read the source.