Model tiers and the prompt cache

Two mechanisms that decide what a turn costs, neither of which changes what the agent says. The first picks a cheaper model for work nobody reads as an answer. The second stops paying, on every single message, for the part of the request that never changes.

Two tiers

A call may say what class of model the work deserves instead of naming one:

level	model	for
simple	ANTHROPIC_MODEL_SIMPLE — claude-haiku-4-5	the ordinary question, and service work: compacting a long conversation, deciding what a question deserves
smart (the default for a call that names no level)	ANTHROPIC_MODEL — claude-opus-4-8	genuine multi-step work
genius	ANTHROPIC_MODEL_GENIUS — claude-opus-5	the rare question where deep reasoning is the whole task

Since AGNT2-367 the three levels are also what a team sees: its settings hold the highest level its agents may reach, within what its plan allows. Every turn a person starts is assessed on simple and the verdict is pulled down to that ceiling — a ceiling, never a starting point.

An explicitly named model beats a tier, because a model is a statement about one call and a tier is a statement about a kind of work.

A tier that cannot be resolved refuses. It does not quietly use the main model — that failure returns a perfectly correct answer, logs nothing, breaks no test, and shows up only on the invoice, months later. So the cheap tier carries a default, and an installation that blanks it gets a refusal on the first call instead of a silent bill.

The spend is charged to the model actually used, not to the one the caller asked for by name. Pricing is per model, so that is the only row where the saving is visible at all.

The prompt cache — the larger of the two

Every message to an agent used to re-send the same thing: the description of all its tools, the platform's rules, and what it can do. On a real conversation that is close to nine tenths of everything paid for, and from message to message it does not change.

The provider can keep that beginning and hand it back roughly ten times cheaper. The capability existed here for months and was switched on for exactly one caller — knowledge indexing. The agent's own turn, which happens on every message and carries the largest prompt in the product, never used it. It does now.

What it saves

Measured twice, independently, on the same four-turn conversation with real calls to claude-haiku-4-5, priced from this product's own spend rows:

	input	cache write	cache read	output	cost
before	58 367	0	0	313	$0.0599
after, cache cold	5 949	10 447	41 788	268	$0.0245 — −59.1%
after, cache warm	5 923	0	52 235	240	$0.0123 — −79.4%

Both numbers are real and they mean different things. Cold is a conversation that starts with nothing kept — it pays the write premium once. Warm is every conversation after that, while the kept prefix is still alive. Quoting only the second is the optimistic half.

The trap, and where the mark is placed

Writing to the cache costs more than ordinary input. So if the beginning of the prompt changed from turn to turn, we would pay the write premium every time and never read anything back — more expensive, not less.

The mark therefore sits exactly on the boundary between what repeats and what does not:

above the mark — kept	below the mark — sent fresh every turn
the platform's own instructions	the notes this turn recalled
how this agent's machinery works, which follows from its type and what its owner unlocked	fragments of knowledge pulled in for this question
	what is known about the person asking
	the person's local time
	the agent's persona, always last

The persona is below the mark, and that surprises people. It never changes during a conversation, so it looks like it belongs above — but a prompt has one order and the persona sits underneath the layers a turn recalls, which do change. A mark cannot skip over what is between them. Editing an agent's persona therefore does not throw the kept prefix away. What it costs to leave it out is about 230 tokens of the ~10 600 the head holds — roughly 2 %, because the toolset, which the provider keeps ahead of the system prompt, is 9 700 of them.

The prompt the model is shown is byte for byte the same string either way — the mark splits it, it does not rewrite it. On the first turn of the measurement above, the old arm counted 10 927 input tokens and the new arm counted 480 input plus 10 447 written: the same 10 927.

What it costs, said honestly

A single turn with nothing after it got about 22% more expensive: it pays the write premium and nobody ever reads the result. That is a property of the mechanism, not a defect. Conversations are the normal case and they are the case that wins.

Where these numbers come from

Worth knowing before you quote them. The three figures above were measured twice on the same four-turn conversation — once by the author of the change and once, independently, by its reviewer — and both runs live in the reports on AGNT2-376, which are linked from that ticket and are not in this repository. The 3.4× for per-request rotation is from AGNT2-366's report, the same way.

What is in the repository is where the investigation started: specs/AGNT2-369-openrouter-or-own-keys/research.md, which found the capability written and unused and forecast 56.4 %. That is a forecast, not a measurement, and it is lower than both figures above — read it for the reasoning, not for the numbers.

What it means for the key carousel

The provider's cache belongs to the workspace a key belongs to, and each key is a workspace. So rotating to a different key means starting cold.

That is affordable, and it was measured rather than assumed: one rotation is one cold start, about a quarter of a cent, and it pays for itself on the next call. Rotation does not cancel the saving; it postpones it by one call.

The order the two were built in was not an accident: cache first, rotation second.

What bounds a call

Four numbers, each bounding a different thing. Nothing here lets the provider decide how long we wait.

setting	default	bounds
ANTHROPIC_STREAM_STALL_MS	60 s	silence on a streamed response
ANTHROPIC_REQUEST_TIMEOUT_MS	300 s	one attempt of a blocking call
ANTHROPIC_RETRY_WAIT_MAX_MS	10 s	a pause between two attempts
ANTHROPIC_CALL_BUDGET_MS	300 s	the whole blocking call, pauses included

Where the line falls between «slow but alive» and «not waiting», because a bound that cuts honest slow answers is a different bug:

• A stream is judged on progress. Every event re-arms the silence budget, so a response that keeps talking is never interrupted however long it talks.
• A blocking call has no progress to observe, so its line is time — and the whole-call budget equals one full attempt, so a single honest slow answer has the entire ceiling to itself. What runs out is the room to retry after it.
• A pause is not an answer, so it is capped separately and hard.

Check it

The four token counts are recorded per call, so the whole of this page is answerable from this product's own data rather than from the provider's console.

1. Have a conversation with an agent — four turns or more. The first turn has no cache by definition, so a single message measures only the write premium.

2. Read the spend rows for that conversation:

   SELECT unit, SUM(quantity)
   FROM "BillingEvent"
   WHERE "threadId" = '<the conversation>'
   GROUP BY unit;

3. cache_write_tokens is non-zero on the first turn and drops to zero afterwards; cache_read_tokens is large and roughly constant on every turn after the first.

4. Priced at the vendor's rates, the total for turns 2–4 is a fraction of the input those turns would have cost as plain input — the row input_tokens shows how little is left.

The control, which is the step that catches a vacuous measurement: run the same conversation with the cache mark removed. Every cache_* row is zero, input_tokens is roughly six times larger, and the cost returns to what it was.