> ## Documentation Index
> Fetch the complete documentation index at: https://runinfra.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Prompt caching for agents
> Cached input costs about a tenth of uncached on most models. Here is what keeps a prefix reusable across an agent's turns, and what quietly breaks it.
Cached input is billed at the cached rate, roughly a tenth of the standard
input price on most models. For an agent that sends a growing conversation on
every turn, that is the difference between paying full price for your whole
context once and about a tenth of it on every later turn.
Caching is automatic. No header or parameter is required. What
you control is whether your prompt stays *reusable* between turns.
## The rule
The cache matches a **prefix**: the longest run of tokens from the start of
your prompt that is byte-identical to a previous request. Everything from the
first difference onward is recomputed and billed as uncached.
So an agent turn that appends to the end of its context reuses almost
everything. A turn that changes something near the beginning reuses almost
nothing, however small the change.
The match is counted in whole cache blocks, and the trailing partial block is
billed at the input rate. The unit size varies by model, so a very short prompt may never hit, and the discount does the most work on long prefixes. Read the cached-token count on your own responses to see how much of your prompt is matching: `usage.prompt_tokens_details.cached_tokens` on Chat Completions, with the Responses and Messages fields under [Reading your own hit rate](#reading-your-own-hit-rate).
One more shape matters on Qwen3.8 Flash Next: a conversation that appends
warms on turn 2, as in the diagram, but a shared prefix re-sent with a
changing suffix, many questions against one document, may take an extra send before `cached_tokens` shows it warm.
## What breaks a prefix
Every item below has been observed reducing a real workload's hit rate. They
are ordered by how often they turn out to be the cause.
The single most common cause. `Current time: 2026-08-21T09:14:22Z` at the top
of a system prompt changes on every request, so nothing after it can ever
match.
Move it to the **end** of the conversation, as the last user message or a
trailing system note. It is just as visible to the model there and it stops
invalidating everything behind it.
If your tool list is built from a set, a dictionary, or a directory listing,
its order can change between processes even when the tools do not. Sort tool
definitions by name before serializing, once, and keep that order for the life
of the session.
`JSON.stringify` over an object preserves insertion order, which can differ
between runs. Python's `json.dumps` accepts `sort_keys=True`. If you build
tool schemas or context blocks programmatically, serialize them
deterministically.
Whitespace counts too: a pretty-printer that changes indentation between
versions changes the bytes.
Anything unique per request belongs outside the prompt. If you need the model
to know a session id, put it at the end, not in the system block.
Summarizing older turns is good practice, but replacing the middle of a
conversation invalidates the prefix from the summary onward. Compact at a
boundary you keep stable, and prefer appending a summary to rewriting history
in place.
If retrieved context is sorted by a score that shifts slightly between calls,
the block order changes. Sort retrieved chunks by a stable key, such as
document id, rather than by score.
## Reading your own hit rate
Chat Completions responses report `usage.prompt_tokens_details.cached_tokens`,
the count of input tokens that request was billed at the cached input rate,
and the same number as `usage.runinfra.cached_input_tokens` beside the cost.
It is `0` rather than absent when nothing was billed as cached. The Responses
API reports it as `usage.input_tokens_details.cached_tokens`, and Anthropic
Messages as `usage.cache_read_input_tokens`. Print it on
every turn: on a workload that should be repetitive, the value that keeps
coming back is where your prompt stops matching itself. The field is
described in [Chat completions](/docs/api-reference/chat-completions#the-cost-of-the-request-and-its-cached-input-on-its-usage).
Your workspace's cache report is on the dashboard's
[Usage Analytics](https://runinfra.ai/inference/usage) page, read with your
signed-in dashboard session rather than an API key. The **Cache by API key**
band carries the workspace totals: cached input tokens, the confirmed hit
rate, confirmed misses, and confirmed savings. The **By API key** and
**By model** tables on the same page carry cached tokens, hit rate, and
savings per key and per model. A low rate on a workload that should be
repetitive means something above is changing.
To compare many requests, record their cached-token counts and look at the
**distribution**, not just an average. Its most useful summary is the mode:
the cached-token count that recurs most often. Because matching is counted in
whole cache blocks, this is roughly where your prompt stops matching itself;
past that point, a field in your prompt may be differing between requests.
```python Python theme={"dark"}
response = client.chat.completions.create(
model="nemotron-3-5-lightning-30b", messages=messages, max_tokens=16384
)
print(response.usage.prompt_tokens_details.cached_tokens)
```
```typescript TypeScript theme={"dark"}
const response = await client.chat.completions.create({
model: "nemotron-3-5-lightning-30b", messages, max_tokens: 16384,
});
console.log(response.usage?.prompt_tokens_details?.cached_tokens);
```
A worked example from real traffic, measured in August 2026: one workload
showed a mode of **9,984 cached tokens on a fifth of its requests**, against
prompts averaging about 70,000 tokens. Everything up to roughly token 10,000
was reusable and nothing after it was, on every affected turn. That is the
signature of a field changing early in the prompt, not of a cache problem.
A sibling workload on the same model reused 92 percent of a 465,000-token
prompt. These are measured examples, not a promised hit rate.
If the count that recurs sits far below your prompt size, look at what is
between that offset and the start of your context.
## Keeping a session warm
Check current model availability in [`GET /v1/models`](/docs/api-reference/models). The retention behavior below describes configured models and does not guarantee that a model is serving.
On every chat model, retention of a warm prefix is best effort: an idle prefix can be evicted under
capacity pressure, and no lifetime is promised.
On [`/v1/models`](/docs/api-reference/models), `cache_tiers` and `cache_retention`
tell you which kind of cache a model runs. `best_effort` with `cache_tiers`
`["gpu"]` is the GPU copy alone. `tiered_host_memory` means a prefix can outlive the fastest tier: DeepSeek V4 Pro publishes it, and retention there is still best effort.
`session_affinity_idle_ttl_seconds` is a routing window, how long a
quiet session keeps its place close to its cache, not a
retention promise.
The cached-token count on the first turn after a pause is the readout of what
survived it. If the copy was lost while your agent ran tools, waited on a
human, or slept overnight, that turn is billed as a miss; the count and the
bill agree. The matching rule at the top of this page is unchanged: a
returning prompt must be byte-identical to the prefix it wants back.
On any chat model, a stable [`x-session-id`](/docs/api-reference/session-affinity)
keeps a session's follow-up turns close to its warm cache, and pins that placement for the model's published `session_affinity_idle_ttl_seconds` idle window. Without one, a
`prompt_cache_key` in the request body routes the request by its value but
does not pin it. Without either, placement still keeps steady sessions warm;
the full placement order is on the session affinity page.
## What does not affect caching
* **Sampling parameters.** `temperature`, `top_p`, `seed`, and `max_tokens` are
not part of the prefix.
* **Streaming.** A streamed and a non-streamed request cache identically.
* **Your traffic volume.** Caching is per prefix, not a plan or an allowance.
## Isolation
On models that publish `cache_isolation: "isolated"` in
[`/v1/models`](/docs/api-reference/models), a cached prefix is scoped to your
workspace within the model's published isolation scope. Another customer
does not reuse your isolated prefix, and you do not reuse theirs. That scoping
applies on every request. Read the per-model tier scope in
[Data retention](/docs/security/data-retention#cached-working-state).
On a model that does not publish isolation, a hit is neither disclosed nor
priced. A billable response charges every input token at the standard input
rate. The response reports `cached_tokens: 0`, including when it settles at zero.
## Next steps
Keep every turn close to your warm prefix.
Read cache isolation, tiers, and retention per model.
The agent loop these prefixes come from.