Prompt Caching Isn't Free Speed: Find Your Break-Even Ratio

Turning on prompt caching without measuring your reuse rate first is a coin flip. It either cuts your bill in half or adds cost to every call, and the percentage on the vendor slide tells you nothing about which one you get.

Prompt caching is the provider-side trick where the model keeps the processed form of a long, unchanging prefix (your system prompt, tool definitions, a big document) in memory so the next request that starts with the exact same prefix skips reprocessing it. Anthropic, Amazon Bedrock, and OpenAI all ship some version of it. All of them charge you differently for writing the cache than for reading it, and that asymmetry is where the decision lives.

Why the vendor number doesn't apply to you

Two multipliers govern everything. The write multiplier w is what you pay to store a prefix, expressed as a factor of the normal input token price. The read multiplier r is what you pay when a later request hits that stored prefix. Anthropic's five minute tier, for example, charges 1.25x to write and 0.1x to read.

Derive the break-even rather than trusting a rule of thumb. For a prefix of P tokens, one write followed by N reads:

uncached:  P * (1 + N)
cached:    P * (w + N * r)

Set them equal, and the write premium cancels out into a single threshold:

N* = (w - 1) / (1 - r)

N* is the number of reads per write that a cache entry needs before it has paid for itself.

Same formula, very different gates. The one hour tier's threshold is four times higher, which in whole reads means a cache entry has to be read twice instead of once, because you paid double to write it. A break-even number quoted without a TTL attached is not a usable number.

One correction worth making loudly, because the wrong version is circulating in blog posts and at least one published package: you will often see N* = (w - r) / (1 - r), which produces 1.28 reads for the five minute tier. That expression is 1 + (w - 1) / (1 - r). It counts total requests including the write itself, not reads. Compare against Anthropic's own wording, which says caching pays off after one cache read on the five minute tier and after two on the one hour tier. If your monitoring compares a reads-per-write ratio against 1.28, it will tell you to leave caching off on workloads that are already saving you money.

Measure your real hit rate before you flip the flag

Most agent loops that see no savings after enabling caching don't have a pricing problem. They have a structural bug: one volatile field sits ahead of the cache boundary, so the prefix differs on every single call and the hit rate is exactly zero.

The usual culprits are a timestamp injected into the system prompt, a session identifier, a user id, or tool schemas that get re-serialized from a dict in a different key order each call. Caching matches on an exact token prefix. One changed character before the breakpoint invalidates everything after it.

The ordering rule is the one to memorize: the cached prefix is tools, then system, then messages, in that order, up to and including the block that carries cache_control. Everything before your breakpoint is in the cache key. Everything after it is free to change.

{
  "system": [
    {
      "type": "text",
      "text": "You are a support agent. Current time: 2026-05-22T09:14:03Z",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "tools": [ { "name": "lookup_order" }, { "name": "issue_refund" } ]
}

Every request writes a fresh cache entry and reads none. You pay the 1.25x premium forever. The fix keeps all static content ahead of the breakpoint and pushes anything that varies into the message list, after it, with no breakpoint of its own.

{
  "tools": [ { "name": "issue_refund" }, { "name": "lookup_order" } ],
  "system": [
    {
      "type": "text",
      "text": "You are a support agent.",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Current time: 2026-05-22T09:14:03Z" },
        { "type": "text", "text": "Where is order 4417?" }
      ]
    }
  ]
}

The breakpoint now sits on the last static block, so the cached prefix is tools plus system and nothing else. Putting it on the final user block instead, which is the mistake I see most often in code review, is the same bug in a new costume: the prefix would then include the timestamp and the question, and it would miss on every call.

Note the tool array is now sorted by name. If you build tool definitions programmatically, sort them and serialize with stable key ordering, or you've just recreated the bug in a less obvious place.

Both JSON snippets above are schematic. Real prefixes have to clear the model's minimum cacheable length before anything is stored at all, so a two line system prompt like this one would simply be ignored by the cache, premium included.

For reference points: coding agents with fixed tool definitions can sustain hit rates near 98 percent because the prefix barely moves turn to turn (one measured 12 turn session reported 98.7 percent, a single data point, not a guarantee). General multi-turn assistants typically land between 40 and 60 percent. Both clear the break-even gate for the five minute tier comfortably. If you're measuring near zero, the problem is upstream of the cache, not the cache.

The TTL is a sliding window, not a countdown

One mechanic changes how you read a low hit rate: every cache read refreshes the TTL, at no extra cost. The five minute window is not five minutes from the write, it is five minutes from the last hit. A conversation with a turn every ninety seconds keeps a five minute entry warm indefinitely.

The practical consequence is that "my TTL is too short" is almost never the real diagnosis. If the prefix is stable and requests keep arriving inside the window, the entry stays alive on the cheap tier. A low reads-per-write ratio points at one of two other things: a prefix that drifts between calls, or request density too low to keep anything warm, which is a traffic shape problem. Buy the one hour tier only after you have ruled both out, because it doubles the write premium and quadruples your break-even.

The script that replaces the vendor promise with a verdict

Feed it your request logs. Each entry needs the prefix token count and whether that call hit a warm cache. It computes your observed hit rate, applies the break-even formula, and prints one of four answers.

# Minimum cacheable prefix is per model, not a platform constant.
# Roughly: 512 on the smallest tier, 1024 on mid tier, 4096 on the
# cheap/fast models. Check your model's docs and pass it in.
DEFAULT_MIN_CACHEABLE = 1024


def verdict(logs, w=1.25, r=0.10, min_cacheable=DEFAULT_MIN_CACHEABLE):
    """logs: list of (prefix_tokens, hit: bool)"""
    total = len(logs)
    if total == 0:
        return "NO DATA: empty log"

    hits = sum(1 for _, h in logs if h)
    misses = total - hits
    hit_rate = hits / total
    avg_prefix = sum(p for p, _ in logs) / total
    ineligible = sum(1 for p, _ in logs if p < min_cacheable) / total

    # Averages hide bimodal traffic, so count the calls that are
    # individually too short rather than testing the mean.
    if ineligible > 0.5:
        return (f"LEAVE OFF: {ineligible:.0%} of calls carry a prefix under "
                f"{min_cacheable} tok and are never cached")

    n_star = (w - 1) / (1 - r)               # reads per write to break even
    reads_per_write = hits / max(misses, 1)

    if reads_per_write >= n_star:
        saved = (hits * avg_prefix * (1 - r)
                 - misses * avg_prefix * (w - 1))
        return (f"TURN ON: {reads_per_write:.2f} reads/write vs N*={n_star:.2f}, "
                f"net {saved:.0f} token-equivalents saved")
    if hit_rate < 0.05:
        return "LEAVE OFF: hit rate ~0. Volatile field ahead of cache boundary."
    return (f"MARGINAL: {reads_per_write:.2f} reads/write below N*={n_star:.2f}. "
            f"Prefix drifts between calls, or traffic is too sparse to keep "
            f"an entry warm.")


if __name__ == "__main__":
    sample = [(8200, i % 3 != 0) for i in range(300)]
    print(verdict(sample))

The savings line uses the average prefix across all calls, which is accurate enough when your prefix length is stable and optimistic when it is not. If hits and misses carry noticeably different prefix sizes, split avg_prefix into a hit average and a miss average before you quote the number to anyone holding a budget.