How to keep an agent prompt cache stable as tools change
Agent prompt caching works when instructions and tool schemas stay stable. Use a clear context contract so changing tasks do not erase latency and cost gains.
Niels Kaspers
TL;DR
Treat the stable part of an agent prompt as an interface: keep instructions and tool schemas predictable, put volatile task state after them, and investigate cache misses like a workflow regression.
If an agent gets slower and more expensive every time you add a tool, the problem may not be the tool. It may be that you keep breaking the prompt prefix the model can reuse.
Prompt caching is often framed as a billing optimisation. For persistent agents, it is really a workflow-design constraint. The stable instructions, policies, tool definitions, schemas, and ordering form a reusable interface. The live task, fresh retrieval, and one-off evidence belong after that interface.
That distinction is practical. A cache-friendly agent does not need to be frozen. It needs a clear boundary between what is stable enough to reuse and what must change for the job in front of it.
OpenAI now says this directly in its update on prompt caching for GPT-6: keep tool definitions, schemas, and ordering stable as tool use changes. Its prompt-caching guide makes the payoff concrete: shared prefixes can reduce repeated input work, latency, and cost.
The rule: treat your prompt prefix like an interface
An interface is useful because other parts of a system can rely on it. Your agent's reusable context should work the same way.
I would split the request into three layers.
1. The stable operating layer
This is the part worth preserving across similar jobs:
- system instructions and safety boundaries
- the product or workflow role
- durable output standards
- tool names, descriptions, schemas, and order
- stable reference material that the agent genuinely needs often
Do not move these around casually. A harmless-looking tool rename or schema reshuffle can turn a reusable prefix into a new prompt.
2. The task packet
This is the work for this run:
- the current objective
- the relevant files, records, or URLs
- fresh user constraints
- the requested deliverable
- temporary facts that may change tomorrow
It should come after the stable layer. That gives the agent current information without making the reusable part of every request look different.
3. The evidence and results layer
Tool outputs, search results, and retrieved documents belong here. They are often necessary, but they should not masquerade as permanent context.
This is the same boundary behind AI workflow memory: what to store, what to forget, and what to verify. Stable rules, current task state, and fresh evidence are different jobs. Mixing them into one expanding blob makes the system slower to run and harder to debug.
Why tool changes break cache reuse
Adding a tool feels local. It is not always local to the prompt.
When a client sends the available tools with every request, the tool list is part of the context. If you change a schema, reorder definitions, or include a different tool set for every subtask, the shared prefix changes too. The cache has less to reuse.
That does not mean you should expose every tool all the time. It means the selection policy needs to be deliberate.
A dependable pattern is to keep a stable core toolbox for a workflow class, then use a narrow, predictable capability set for common steps. For genuinely different jobs, create a separate workflow shape rather than constantly mutating one giant agent.
This is also why I prefer workflow packages over prompt libraries. A package captures the job, sources, rules, checks, and output shape. It gives the agent a stable operating contract instead of asking a pile of ad hoc tools to become a system by accident. How product teams replace prompt libraries with workflow packages describes that difference from the PM side.
What should change, and what should not
The useful question is not whether something changes. Everything changes eventually. The useful question is whether it changes often enough to belong outside the reusable prefix.
Keep stable:
- tool identifiers and schemas that the workflow uses repeatedly
- evaluation criteria
- response formats
- permission boundaries
- durable product vocabulary
Keep dynamic:
- a new ticket or brief
- retrieved documents
- current prices, statuses, and deployment facts
- per-run source files
- raw tool results
Do not solve a volatile-facts problem by baking old facts into the prefix. That creates cached risk: the system gets cheaper while becoming less correct. For anything that can affect a consequential action, retrieve or verify it when the task needs it.
The metric that matters is not cache hit rate alone
A high hit rate can hide a bad workflow. If the agent is efficiently reusing stale instructions or bloated context, it is still not healthy.
I would watch four signals together:
- Cache read rate: Is the stable prefix actually being reused?
- Latency by workflow step: Did a tool or instruction change create a visible regression?
- Input cost per completed job: Are you paying for useful evidence or repeated boilerplate?
- Verification and correction rate: Did the faster run make worse decisions or create more cleanup?
That last signal matters most. The goal is not to maximise caching. The goal is to make repeated work cheaper without making the output less reliable.
On this site, the useful agent patterns all retain a fresh verification step. How to audit an AI workflow before it turns into agent debt is the relevant guardrail: a system earns more autonomy when its visible checks keep pace with its speed.
A small context contract for agent teams
You do not need an elaborate platform to apply this. Write down a short contract for every repeated agent workflow:
- What is the stable instruction block?
- Which tools are normally available, in what order, with which schemas?
- What is the canonical current-state artifact?
- Which facts must be freshly retrieved?
- Which output needs a verification gate before it moves on?
That contract does two useful things. It improves cache reuse, and it makes a regression legible. When a run gets slower or worse, you can ask whether the stable interface changed, whether the task packet grew, whether retrieval became noisy, or whether verification failed.
This is close to the operator pattern I trust in OpenClaw and other long-running workflows: explicit state, scoped tools, stable conventions, and a reviewable outcome. The system should not need to rediscover its shape from a transcript every time.
The anti-pattern: a universal agent with a shuffled toolbox
The common failure mode is a universal agent that receives a different bundle of tools, notes, and policies for every request. It can look flexible in a demo. In production, it is difficult to measure and expensive to stabilise.
The alternative is not rigidity. It is composable workflows. Give each repeated job a clear stable core, a controlled task packet, and a deliberate escalation path for work that does not fit.
If a task needs a different tool family, route it to a different workflow or a scoped subagent. Do not casually splice a new capability into the middle of every prompt and hope the cache, behaviour, and safety model stay intact.
Agent prompt-cache audit
Interactive
Agent prompt-cache audit
Use this before calling a cache miss an unavoidable cost of agent work.
Completion
This is the gap between understanding the article and actually using it.
- Use this block as the practical summary, not just the article ending.
- If one item feels vague, the article probably needs sharper guidance.
- A short checklist beats a long recap when the reader needs to act.
My take
Prompt caching is not a trick for shaving a line item. It is feedback about whether your agent has a stable operating shape.
Keep the reusable contract predictable. Put live work after it. Treat fresh facts as fresh facts. And when tool changes are necessary, make them intentional enough that you can see their effect on cost, latency, and outcome quality.
That is how a persistent agent becomes cheaper without quietly becoming brittle.
FAQ
What is agent prompt caching?
Agent prompt caching reuses computation for a shared prompt prefix across related model requests. It can reduce repeated input processing, cost, and latency when the stable context remains eligible for reuse.
Should every agent receive the same full tool list?
No. Give a workflow the tools it genuinely needs, but make the common tool contract stable. If work needs a different capability family, a separate scoped workflow is often clearer than mutating one universal agent on every request.
Where should live data go in a cache-friendly agent prompt?
Put current task instructions, retrieved evidence, tool results, and other volatile facts after the stable instruction and tool layer. Re-verify facts that can drift or influence consequential actions.
Does a high cache hit rate prove an agent is well designed?
No. Track cache reuse with latency, completed-job cost, and verification or correction rates. Efficient reuse of stale or irrelevant context is still a workflow failure.
What usually causes unexpected cache misses?
Common causes include changing instruction order, moving cacheable context, renaming or reordering tools, editing schemas, and mixing variable retrieval results into what should be the stable prefix.