A shop assistant can receive thousands of pieces of fixed instruction before it sees a merchant's question. Shopify reports that it used a technique called gisting to compress those repeated instructions for Sidekick, its merchant assistant. The system learned a shorter representation the model could use in place of the longer text.
Shopify says the change reduced the fixed prompt from about 6,000 tokens to about 1,500. Tokens are the chunks of input a language model processes. In a load test at 350 requests a minute, the company reports that the time before a response began fell by about 19 percent and total response time fell by about 38 percent.
The figures describe Shopify's test environment. They illustrate a recurring cost of running an agent at scale. Even when its rules stay the same, the system may have to process them again for every request.