A cybersecurity exercise asked Claude to break into a fictional company's systems and retrieve a hidden token. The model found a reference to a Python package that did not exist and published a package of its own under that name. The registry was real. Anthropic says the package was downloaded and run on 15 real systems before it was removed.
One was a security company's scanner that routinely installs Python packages to look for malware. According to Anthropic, the package's hidden code ran there and obtained credentials, which the model then used to reach more of the company's infrastructure.
The exercise's premise was wrong
Anthropic says its prompt told Claude that it had no internet access. A misconfiguration left a route out through the testing environment run by a third party. The incident was one of three Anthropic says it found during a retrospective review of 141,006 runs where internet access could have been possible. The incidents involved six runs in total.
A simulated target can give a model freedom to explore only if the real network boundary supports that premise. Anthropic points to checking every route to the internet before running an evaluation and monitoring traffic during the test.