Imagine a service that looks at an example and reports both a category and how confident it is. A caller can use that information to make a decision. In a 2016 study, Florian Tramèr and colleagues showed that it could also help a caller reconstruct the model supplying the answers.
The researchers tested prediction interfaces for several model families, including commercially hosted services. Some interfaces returned detailed confidence values. For certain models, those extra numbers gave the researchers enough information to solve for the parameters that produced them. The tests used predictions as evidence, without direct access to the training records or model files.
An answer can disclose more than a category
The paper also investigated interfaces with less detailed output. Restricting what an interface returns can change the cost and effectiveness of an attack. It does not make the visible behavior of the system private.
That distinction matters when a model is valuable because of the decisions it makes. The file that stores it can remain secret even as repeated answers allow another system to approximate those decisions.