How Claude handles its sessions is none of my business. I'm going to let them do the best they can to provide good service for everyone, and if they can't/won't, I'll switch to a provider that can.
Using these massive models is already pretty danged extravagant, I'm not going to demand to be at the front of the queue at all times, too.
Cached input tokens cost 10% of uncached. So if you’re model runs for 45 minutes, generates 300k output tokens and asks you a question, it costs 10x more if you wait 5.01 minutes to answer.
Sure, you may be willing to pay 10x more (or get 10x less for your subscription). But the time limit is arbitrary and has nothing to do with other peoples’ workloads. So I think your point is a non sequitur.
No it has to do with others workloads, now you keep their cache for longer so others will get less. And no its not arbitrary, they run out of memory, if more people do this they will have the dial it down further or run out of capacity.
Are you imagining this a fixed MRU where duration scales with usage? Becasue that is not at all what Anthropic documents: https://platform.claude.com/docs/en/build-with-claude/prompt...
You would not get more than 5 minutes if you were the only user in the world. You would not get less at their peak hours.
In this specific case, Anthropic can avoid keeping the cache if it detects this kind of prompt (i.e., if max tokens < some number).
> Detect the danger window. When the main lineage goes idle and a subagent is actively running, the main prefix is at risk of expiring.
So, this is not demanding to be at the front of the queue, it's just paying someone to take the place you already had in the queue, when you want to take a leak.
If you're paying API rates, you can choose 5m or 1hr yourself (and pay different rates).
Keeping a 1hr cache warm could still be useful, sure, but outside that, I don't see much use of this today.
https://code.claude.com/docs/en/prompt-caching#on-a-claude-s...
(Thank you to EliasWatson for giving me this link just a few days ago, as I was previously confused too.)
You would have avoided that cache hit if the LLM session was kept "alive" for those few hours. Why not automate the part where you keep the large main thread alive until you're ready to analyze the results?
Hearing one byte refreshes the whole thing is huge! 5min is wayy too slow, because sometimes I want to spent more than 5 min looking at a diff before choosing where to go next.
Kind of outrageous, I hope this kind of feature gets built into claude code =D
*UPD:* actually it appears the default is authentication-dependent. API key gets 5 minutes, subscriptions - 1 hour.
For example, there might be something I intended to complete in one sitting, but took two sittings in the same day unexpectedly. Maybe it could just be a few cache delays per day or something, tagged in advance somehow.
[dead]
[dead]
[dead]
Because ram/memory is free and not in demande at all these days?
2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?
If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
[deleted]
How is this comparable to going to lunch or taking a walk?
The cache is discounted for a reason. They WANT you to use it.
1) Start charging for VRAM reservations.
2) Charge _other_ customers more.
3) Eat the cost themselves.
Anthropic (and now OpenAI too for 5.6) prompt caching is not free.
I don’t think your understanding works.
Keeping your cache warm is a good thing, caching saves compute and electricity.
Cached input is cheap for a reason, it is in everyone’s mutual interests to maximise cache hit rates.
[deleted]
Fixed that for you.
Cache duration is arbitrary. What it actually does (if used en masse) is decrease the amount of oversubscription their infra can handle..
Imo it's their fault for not having pricing that aligns incentives.
"They already charge me to park my car, why can't I leave it there for a year for the same price as 1 week?"
Lately, I’ve been thinking about how this related to fractional banking. If you were to eliminate fractional banking introduced in the US by Hamilton, you would destroy a lot of current prosperity.