Detecting drift without reading transcripts

You find out about drift from support tickets. Someone says the character has not been itself for a while, and there is no dashboard that would have shown it, because none of the numbers you collect describe a character.

Offline probes tell you where a definition stops binding. They cannot tell you what is happening in the conversations you have, and those are the ones that generate complaints. Production monitoring is a different problem with one defining difficulty: there is no correct answer to compare against.

Every conversation is its own baseline

The instinct is to compare responses against a reference — a golden sample, a corpus, an average. It does not work, because conversations legitimately differ from each other, and the same definition produces genuinely different characters across long histories. A global average is the mean of many diverging characters, and a conversation deviating from it is not evidence of anything.

What does work is comparing a conversation against its own early turns. In the first stretch, the definition is most of the input and the character is expressed most strongly. That stretch is the closest thing to ground truth you will get, and it is available per conversation, for free, already stored.

So the measurement is intra-conversation drift: a set of cheap statistics computed over an early window and recomputed over a recent window, per conversation, with the delta as the signal. The absolute values are meaningless. The deltas are comparable across conversations, and their distribution across your traffic is the dashboard that does not currently exist.

Response length, sentence length, vocabulary variety, the rate of question-asking, the rate of hedging phrases, the rate of first-person statements: none of these describe character on their own, and all of them move when a character flattens toward a generic assistant. They are proxies, they are nearly free to compute, and they are directional rather than diagnostic — which is the correct level of ambition for a monitoring signal.

There is a signal that is not a proxy. Instrument the proportion of each assembled input that is character definition, and record it per turn.

It falls monotonically over a conversation’s life, it is exact, it requires no judgement, and it is the mechanism itself rather than a correlate of it. Every continuity decision moves it: a longer summary lowers it, a bigger fact store lowers it, a rebuild raises it, a definition addition raises it and raises cost with it.

Two things make it worth more than any other single number. It is predictive — you can see conversations approaching the range where drift complaints come from before they get there. And it is comparable across conversations regardless of how they differ, because it is a ratio of your own assembled state.

Pair it with depth. A scatter of definition share against turn count across your traffic shows you the shape of your product’s drift pressure in one picture, and where the tail is.

Signals you already have and are not counting

Several drift indicators are sitting in existing data.

User corrections. Turns where a user restates, denies or corrects something. Cheap to detect approximately, and the rate per conversation tracks memory failure rather than register drift — a different axis, and the one that catches wrong facts.

Repair phrases. Users asking the character to go back to how it was, or telling it that it is being odd. These are the highest-signal turns in your traffic and there are few of them, so they are worth reading individually — the only place in this post where reading is the recommendation.

Intervention rate. Moderation actions per conversation, tracked against depth. A rising rate at depth is a state observation, and interventions themselves feed a foreign register into the history.

Distance to failure. The conversational distance between a fact being stated and being needed. Clustering at a consistent distance is a window edge; scattering is something else.

Session resumption and message rates. Behavioural, coarse, and interpretable only as “something changed around here”. Use them to locate a time window to investigate, never as a quality measure in themselves.

The turn

THE TURN — production monitoring

  · Compare against a global baseline
                    → simple, and meaningless: conversations
                      diverge legitimately, so deviation proves
                      nothing.

  · Compare each conversation with itself
                    → deltas are comparable across a diverging
                      population, and the early window is free
                      ground truth.

  · Cheap statistics as proxies
                    → directional, not diagnostic. They tell
                      you where to look, never what is wrong.

  · Definition share per turn
                    → the mechanism measured directly, exact,
                      predictive, and comparable everywhere.

  · Logging assembled-state composition
                    → PAID EVERY TURN, in storage and a little
                      work per request, forever. It is the
                      cheapest thing here and still not free.

  · Monitoring changes nothing
                    → it tells you which conversations have
                      drifted. Every repair is still a tradeoff
                      against cost or memory.

Detection

The recursive question — how do you know the monitoring is right — has one practical answer: validate the signals against the complaints you already have.

Take reported drift, find the conversations, and check whether your deltas were elevated before the report. A signal that was not elevated is not a signal, however plausible it looks. This is a small, one-off piece of analysis and it separates the metrics worth alerting on from the ones worth ignoring, which is otherwise guesswork.

Then check the other direction: sample conversations with large deltas and no complaint. Some will have drifted without anyone minding, and the rate of that tells you how tight a threshold can usefully be. A monitor whose alerts are mostly conversations nobody complained about will be muted within a month.

Keep the population view rather than a single number. The useful reporting object is a distribution of drift deltas across conversations, bucketed by depth — it makes visible both the tail and whether a change you shipped moved the whole population or only the extremes. And correlate the same buckets with what the offline suite predicted, because a divergence between the two means your synthetic histories do not resemble real ones.

What this costs and what it doesn’t fix

Monitoring is the cheapest work in this whole subject: statistics over text you already store, plus a composition record per turn. It is still a per-turn cost, and the composition record is the part people skip because it has to be written at assembly time and cannot be reconstructed afterwards. Add it before you need it — the same argument as provenance on stored facts.

What it does not do is tell you what to change. Every response to a detected drift is one of the trades already on the table: carry less history and lose continuity, re-assert the definition and pay for it on every turn, rebuild the summary and lose specifics, pin more facts and pay a fixed charge. Monitoring moves you from finding out by complaint to finding out by measurement. The set of available repairs is exactly as lossy as it was.