The cost of a turn
A persona chat product has an unusual cost structure and it catches teams out in a specific way. The bill does not scale with users. It scales with messages multiplied by how much state each message carries — and the second factor grows for the life of every account.
Which means your most engaged users, the ones the product is for, get more expensive per message the longer they stay. Nothing else in a normal SaaS cost model behaves like that.
Where a turn’s cost comes from
Every turn assembles an input and generates an output, and both are billed.
Going in: the character definition, whatever memory you carry, the recent turns, the user’s message. Coming out: the reply. Input volume dominates in almost every persona system, because the definition and the state are large and a single reply is small.
Three consequences follow directly.
Cost is per message, not per user. A user sending two hundred messages a day costs roughly a hundred times one sending two.
Cost rises with history. A window keeps it flat by discarding. A summary keeps it roughly flat by compressing. Extracted facts grow slowly and monotonically. Raw history grows without bound. Your memory architecture is your cost curve, and that is the same choice as the one made for continuity reasons — the two decisions are one decision.
Output length is a cost lever that lives in the definition. A character instructed toward brevity is materially cheaper to run than one that writes paragraphs, and that instruction is a line of persona text rather than an infrastructure change. It is the cheapest cost optimisation in the system and it changes the product’s character, so it is a product decision wearing an engineering hat.
Illustrative arithmetic
Round hypothetical numbers, purely to show the shape — not measurements of anything.
Say a fixed definition, a summary, and a handful of recent turns come to some quantity of input per turn, and call that quantity 1 unit early in a conversation. Nothing else changes, and by turn two hundred the carried state has grown so the input is 3 units. You are now paying three times as much per message to the same user as you were at the start, for a conversation they value more.
Now the useful bit. If a user sends 50 messages a day, moving from 1 unit to 3 units is not a 3× increase in that user’s cost — it is 3× on all 50 messages, every day, permanently. The multiplier applies to a rate, not to a total, which is why per-turn state growth is the single most expensive thing you can be casual about.
Run the same arithmetic on a definition you never trimmed. Cutting unused backstory reduces a fixed quantity present in every input for every user forever. It is a small percentage on a big number and it is the least glamorous work in the system.
The turn
THE TURN — cost
· One message
→ input plus output, billed. Input
dominates in almost every persona
system.
· Growing carried state
→ PAID EVERY TURN, at a rate that rises
for the life of the account. The
multiplier lands on a rate, not a total.
· Window / summary / facts / raw history
→ four different cost curves. Your memory
architecture is your cost model.
· Trimming the definition
→ a fixed saving on every input for every
user, forever. Dullest, most reliable
win available.
· Reply length
→ a cost lever that lives in the persona
text, and changes the product.
· Re-anchoring to fight drift
→ spends input space on consistency
instead of memory. Both are paid every
turn, from the same budget.
· Actual unit costs
→ VARIES BY MODEL, and changes under you.
Instrument, don't assume.Why this shows up as product design
If you have ever wondered why the products in this category are shaped the way they are, this is why. The shape is downstream of the cost structure rather than of anybody’s preference.
Message caps exist because unbounded use is unbounded cost. Memory length differs by tier because memory is a recurring per-message cost rather than a feature you switch on. Cheaper models on lower tiers exist because the model is the largest single lever. Shorter replies are cheaper. Credits price heavy use without committing to unlimited.
None of that requires a decision to be stingy. A business whose marginal cost rises with engagement arrives at caps and tiers whether it intends to or not, and knowing that in advance is the difference between designing the limits deliberately and having them imposed on you by a surprising invoice.
Detection
Four things worth instrumenting before you need them.
Input volume per turn, bucketed by conversation depth. The curve, not the average. The average hides exactly the population you care about, because heavy users are a small fraction of accounts and a large fraction of cost.
Cost per active user per day, by cohort age. If older cohorts cost more per message than newer ones, that is state growth, visible.
The composition of a typical input. How much is definition, how much is memory, how much is recent turns. This one number informs both the cost work and the drift work, because they are competing for the same space.
The top percentile of message volume. Know what your heaviest one per cent costs before you promise anybody unlimited anything.
What this costs and what it doesn’t fix
The instrumentation is cheap and the discipline is not: keeping per-turn state small is a constraint on every subsequent feature, and every continuity improvement anybody proposes will make the number worse. That tension does not resolve. It gets managed, deliberately, or it gets discovered later.
And cost control does not make the product better. Every lever here — shorter state, cheaper models, briefer replies, tighter caps — trades away some of what the product is for. A cheaper turn is a thinner turn. The reason to understand the arithmetic is not to minimise it but to know what you are buying with each unit, which is the only way the tradeoff against a summary’s losses or a drifting character becomes a decision rather than an accident.