Which definition produced a reply

A report comes in: the character was cold and clipped on Tuesday. You have the transcript, and reading it, they are right — those replies are wrong. You run the same messages today and get replies that are exactly on character. Nothing reproduces. The report goes into the pile with the other unreproducible ones, and three weeks later the same complaint arrives about a different conversation.

The transcript recorded the output. It recorded none of the inputs, and the inputs have all moved since.

A reply is a function of six things that change independently

What produced that Tuesday reply was an assembled input, and the assembled input was built from a definition version, a set of carried history, a summary at some generation depth, a set of extracted facts, a model, and a moderation configuration. Every one of those changes on its own schedule.

A transcript holds the user message and the reply. So the record you keep is the one variable you cannot learn anything from, and the six you would need are absent.

Worse, the state is not merely unrecorded — it is gone. The summary that produced Tuesday’s reply has been rebuilt since, possibly several times. The fact set has been written to. The definition has been edited. Even if you never changed a line of code, replaying that user message today assembles a materially different input, so the replay is not a test of anything. Reproduction is not hard here; it is impossible without a record, because the state is destroyed by normal operation.

This is the reason drift discussions go in circles. Two people looking at the same transcript cannot establish whether the character was misdefined, diluted by history, working from a bad stored fact, running on a different model, or intercepted — and each of those has a different fix.

Three ways to make a reply attributable

A per-turn state fingerprint. Store a small tuple alongside each turn: definition version identifier, state shape version, summary generation depth, counts of each state piece assembled, model identifier, moderation configuration version. It is a handful of short fields. That is enough to answer which configuration produced this for every reply you have ever sent, which converts an unfalsifiable report into a query. What it does not give you is replay: you know the definition version, so you can read it, but you cannot reconstruct the exact carried history.

Snapshot the assembled input. Full replay becomes possible, because you kept the thing that was sent. It is the only option that lets you re-run a turn and get the same reply. The cost is that you are storing a second copy of conversation content, at every turn, indefinitely — a decision that has to be made deliberately, with whatever retention rules you operate under, rather than arrived at by a debugging convenience that shipped. It is also the largest of these by a wide margin, and the volume grows with input size, which is the quantity you are already fighting.

An event log of state writes. Record every mutation to the state — each fact written, each summary rebuild, each definition version activation — rather than the assembled result. State at any past turn is then reconstructible by replaying the log to that point, without storing a copy of the input. It is the most elegant and the most demanding: every write path has to go through the log with no exceptions, and one back-door write makes the whole reconstruction untrustworthy without announcing itself.

The fingerprint is the one to build first because it is nearly free and it answers the most common question. The other two are answers to why, and they are only worth their cost once you already know which.

Attribution changes what a metric means

There is a second payoff that is easy to miss. Once every turn carries a version tuple, quality metrics can be grouped by version instead of by date.

Reporting consistency scores by day mixes populations: a definition change on Wednesday shows up as a gradual shift over the following week as conversations turn over, which reads as drift and is not. Drift is a proportion problem inside one conversation; a version change is a step function across the population. Grouped by date, these look identical. Grouped by version, they look nothing alike, and telling them apart is most of the diagnostic value.

It also makes a rollback decision answerable. Without version tags, “did the edit make it worse” is settled by argument. With them, the same probe scores for two definition versions sit next to each other, and the probes you were already running start producing evidence about production rather than about a test harness.

The turn

THE TURN — attribution

  · Store a version fingerprint per turn
                    → every reply becomes attributable to a
                      configuration. Cheapest useful record
                      here by a wide margin.

  · A fingerprint is not a replay
                    → you learn which definition and which
                      generation depth, never the history.

  · Snapshot the assembled input
                    → full replay, exact reproduction. Stores a
                      second copy of conversation content every
                      turn, and grows with the quantity you are
                      already trying to shrink.

  · Event log of state writes
                    → reconstruct any past state without
                      storing inputs. Requires that no write
                      ever bypasses the log.

  · Whichever you choose
                    → PAID EVERY TURN. A fingerprint is a
                      write per turn; a snapshot a copy per
                      turn; a log a write per mutation.

  · Moderation configuration version
                    → MODERATION SITS OUTSIDE. Record it, or an
                      intervention is indistinguishable from a
                      character failure in every report.

  · Attribution fixes nothing by itself
                    → it names the configuration responsible.
                      The defect is still the defect.

Detection

Group every quality metric by definition version by default, and by date only as a secondary view. This one reporting change surfaces more than any new instrument, because it separates step changes from decay without anyone reading a transcript.

Count turns whose fingerprint is incomplete. A turn that cannot say which definition version it used is a gap in the record, and the count should be zero. It will not be zero the first time you look, and the reasons it is not are usually paths nobody knew existed.

Track the distribution of definition versions in live traffic. Long-lived processes serving a cached definition mean an edit is not fully deployed when the deploy finishes. Without this measurement, that lag gets attributed to the edit not working.

Join complaints to fingerprints as a routine step, before any analysis. If a cluster of reports shares a version, it is a shipped defect. If it shares a generation depth, it is a summarisation cadence problem. If it shares nothing, it is dilution inside individual long conversations, which is a different investigation entirely.

What this costs and what it doesn’t fix

A fingerprint is a small write on every turn and a small amount of storage per turn, forever — modest, but recurring, and it belongs in the per-turn budget rather than in a one-time observability project. A snapshot is the same commitment multiplied by the size of your assembled input, which is the fastest-growing quantity in the system. An event log costs discipline on every write path, which is the kind of cost that erodes rather than one that shows up on a bill.

And none of it makes the character more consistent. Attribution converts unfalsifiable reports into findable defects, which is a prerequisite for fixing them and not a fix. The system that knows exactly which version produced a cold reply on Tuesday still produced a cold reply on Tuesday.