Testing a character before it ships

The definition change was reviewed by opening a session and having a conversation. It read well. In production it regressed two behaviours nobody was looking at, in conversations longer than anything the review touched.

A fresh session is the one condition your long-running users are never in, and a single sample of a probabilistic system is not a measurement. Both problems are fixable offline, before deploy, which makes this the cheapest evaluation you will ever do.

Consistency is a distribution, not a property

The same input produces different outputs. That is the mechanism working as designed, and it means “does the character follow this instruction” has no yes-or-no answer — it has a rate.

One consequence is that a passing spot-check is compatible with an instruction that holds most of the time and fails in a visible minority of cases. Another is that two consecutive reviews of the same definition can disagree, and the disagreement will be attributed to the change rather than to sampling.

So the unit of evaluation is a probe run many times, reported as a rate with a spread — and a change is an improvement only if the rates moved further than the spread. This is unremarkable as measurement practice and almost absent from how persona changes actually get reviewed.

Depth is the axis that gets skipped

The second failure is structural. Behaviour depends on how much history is in the input, so a probe on an empty conversation measures the best case and nothing else.

Synthetic history solves this adequately and cheaply. Generate filler turns of plausible conversation to a target depth, then run the probe. It does not need to be realistic; it needs to be the right size and roughly the right shape, because what you are testing against is the proportion of the input that is character definition.

Run every probe at several depths — short, medium, and past the point where your summarisation strategy has kicked in at least once. What comes out is not a pass rate but a curve, and the curve is the useful artifact: it tells you the depth at which each instruction stops binding. A change that raises the fresh-session rate and lowers the depth at which a constraint holds is a regression that a flat pass rate reports as an improvement.

Include the boundaries specifically. The turn immediately after a summary rebuild and the first turn of a resumed session are the two most fragile points in the whole system, and they are trivial to construct synthetically.

Golden transcripts rot; probes with checkable criteria do not

The intuitive way to test a character is to keep good transcripts and compare against them. It does not survive contact with a changing definition: every intentional change invalidates the golden set, so the set is either constantly regenerated — at which point it encodes current behaviour rather than desired behaviour — or it is stale and every diff is a false alarm.

What survives is a probe whose failure criterion is a property rather than a text. A probe should be a short setup plus a check that a specific violation did or did not occur, and the check should be mechanical wherever the violation is mechanically visible: a claim made, a fact stated, a name used, a topic answered that should have been declined. Those are the checks worth having, because they do not need judgement and do not rot.

For properties that genuinely need judgement — register, tone, whether the response reads as the same character — an automated judge is the only scalable option, and it comes with a caveat that is usually skipped. A judge is a model, it has variance, and its agreement with what you actually mean is unknown until you measure it. Keep a small hand-labelled set and score the judge against it, then re-score the judge whenever you change it. An unvalidated judge produces numbers that look like measurements and drift on their own schedule.

Derive the probe set from the definition itself, one probe per instruction, and add one whenever a line is added. That coupling is what makes it possible to later remove a line — provenance plus a probe is the only combination that makes a deletion safe, and without it a definition only ever grows.

The turn

THE TURN — offline evaluation

  · Review by having a conversation
                    → free, immediate, and measures one sample
                      of the best case.

  · Probe suite with mechanical checks
                    → repeatable, per-instruction, and does not
                      rot when the definition changes.

  · Run every probe at several depths
                    → yields the depth at which each instruction
                      stops binding, which a flat pass rate
                      hides entirely.

  · An automated judge for register
                    → scales the unmeasurable part, and must
                      itself be validated against a small
                      labelled set. VARIES BY MODEL.

  · Synthetic history in every probe
                    → PAID EVERY TURN of the suite: a
                      depth-graded suite costs a multiple of a
                      shallow one, on every run, forever.

  · Passing the suite is not consistency
                    → it is evidence at the depths and cases
                      you thought to construct.

Detection

The suite is the detection, so the question becomes how you know the suite is any good.

Track its false-negative rate against production. Every defect that reaches users and was not caught should produce a new probe, and the count of such escapes per release is the honest measure of coverage. A suite whose escape rate is not falling is not being maintained.

Check that probe results correlate with anything real. If suite scores move and production signals do not, the probes are measuring something you do not care about — which happens most often when probes are written to be easy to check rather than to match a behaviour anyone reported.

Report per-instruction rates, never an aggregate. An overall score is the mechanism by which three broken constraints hide behind fifty working ones, and the aggregate is also the number stakeholders will ask for.

Keep the suite’s own cost visible. It is turns, it runs on every change, and it grows with the definition. When it becomes slow enough that people skip it, its measured quality stops mattering.

What this costs and what it doesn’t fix

A depth-graded suite is a multiple of a shallow one in both work and cost — every probe carries synthetic history, and the deep runs are the expensive ones. That is a real recurring bill on the same per-turn arithmetic as production traffic, except you pay it on every change instead of on every message. Judge validation costs hand labelling, periodically, forever.

None of it makes a character consistent. It tells you where the definition stops binding, which lets you choose the failure rather than discover it. And it measures a synthetic conversation: real histories are messier, longer and shaped by users, so a suite is a lower bound on the failures you will see. What it reliably catches is regression — the change that quietly moved the depth at which something held — and that is most of what goes wrong.