A wrong fact outlives a missing one

A fact the extractor missed produces a character that does not know something. The user says it again and the gap closes. A fact the extractor got wrong produces a character that confidently knows something untrue, repeats it, builds on it, and gives the user no obvious way to correct it.

These two failures are usually discussed as one error rate. They are not comparable, and treating them as comparable is how a fact store ends up full of confident nonsense.

Why a wrong fact is durable

A stored fact is included in the input on every subsequent turn. That is the whole point of storing it, and it is also what makes an error self-sustaining.

The mechanism is reinforcement through output. Once the wrong fact is in the input, responses are written as though it were true. Those responses become history. If your continuity strategy also summarises history, the wrong fact is now attested twice — once in the store and once in a summary built from text that assumed it. At that point removing it from the store does not remove it from the conversation, and the character appears to remember something you deleted.

A missing fact is a gap with a natural repair path; a wrong fact is a claim with none. The user’s correction arrives as one recent statement against a fact that has been present on every turn for weeks, and whether the correction wins depends on the same proportions that govern everything else here.

Where wrong facts come from

Extraction errors are not random noise. They cluster in identifiable places, which means they can be measured and reduced without solving the general problem.

Statements about other people. Someone describes a friend’s job. The extractor stores it as the user’s job. This is the single most common category and it comes from extracting facts without extracting who they are about.

Hypotheticals, plans and wishes. “I might move next year” becomes a stored move. “I wish I still played” becomes a hobby. The tense and mood carry the meaning, and a short extracted phrase drops both.

Negations. “I don’t drink coffee” surviving as “coffee” is the failure mode that produces the most memorable errors, because the resulting behaviour is precisely inverted rather than merely wrong.

Irony and self-deprecation. Nothing available at the extraction step reliably distinguishes these from plain statement, and this is the category to accept rather than fix.

Inference. An extractor asked what the conversation implies will produce plausible facts nobody stated. These are the worst kind, because they are not traceable to any sentence a user wrote and so cannot be audited against the transcript.

The common thread is that every one of these is a fact that reads as well-formed. Malformed extractions are easy to filter; the dangerous output of an extractor is fluent.

What to do instead of trying to be accurate

Precision and recall trade against each other here, and the asymmetry above says which side to take: a fact store should prefer missing facts to wrong ones, which is the opposite of how extraction is usually tuned.

Three structural choices follow, each with its own loss.

Store the evidence with the fact. Keeping a pointer to the turn a fact came from makes every stored fact auditable and makes deletion decidable. The cost is storage and a schema that is no longer a flat list.

Store what the fact is about, always, even when the answer is “the user”. Without a subject field the other-people error is not detectable, only noticeable.

Require attestation before promotion. A fact mentioned once is a candidate; a fact restated in a separate session is a fact. This straightforwardly reduces wrong facts and equally straightforwardly delays right ones, so the character knows less in the first week than it could have. On a long-lived relationship product that is usually the correct trade, and on a product where the first session has to impress it is not.

None of these are accuracy improvements to the extractor. They are containment.

The turn

THE TURN — extraction errors

  · Extract aggressively
                    → the character knows more, sooner, and
                      the first session feels sharp.

  · Errors are not symmetrical
                    → a missed fact is a gap the user closes;
                      a wrong fact is included every turn and
                      is written into history as true.

  · Tune for precision instead
                    → fewer wrong facts, and a character that
                      knows less than it could for longer.

  · Evidence and subject per fact
                    → makes errors auditable and deletion
                      decidable; costs schema and storage.

  · Every stored fact, right or wrong
                    → PAID EVERY TURN it is included. You are
                      paying to repeat the error on every
                      message.

  · Deleting a wrong fact is not enough
                    → it survives in history and in any summary
                      built from history.

Detection

Do not measure extraction accuracy as a single number. Measure wrong-fact rate and missed-fact rate separately, because they move in opposite directions when you tune and a combined figure hides which one you made worse.

The cheapest live signal is user correction. Count turns where the user contradicts, denies or restates something the store holds — those turns are your wrong-fact detector, and they are already in your traffic. Rising correction rate after an extraction change is unambiguous, and it needs no labelled data.

Sample against the transcript rather than against judgement. For a random set of stored facts, check whether a specific turn supports the fact as written, including its subject and its mood. A fact with no supporting turn is an inference, and the proportion of inferred facts in your store is a number worth knowing.

Watch for facts that were deleted and then reappear. Reappearance means the fact is being re-extracted from history that was written under its influence, and that loop will not stop on its own.

Finally, look at what a fact’s presence changes. If the store holds facts that never affect a response, their error rate does not matter and their per-turn cost does — which is a cost question rather than an accuracy one.

What this costs and what it doesn’t fix

Containment costs storage, schema and delay. Evidence pointers and subject fields make the store larger than the facts alone; attestation makes the character slower to learn; audit sampling is ongoing work that produces no user-visible feature. Against that, every wrong fact avoided is a cost you were going to pay on every turn of that conversation for its lifetime.

What it does not fix is the failure that made extraction attractive in the first place. Extraction is blind to whatever it did not decide to keep, and tuning it toward precision makes it blinder — a deliberate move from one loss profile to another, not an escape from the set of available losses. A store with a low wrong-fact rate is a store that knows a small number of things well, and a character built on it will forget things a user clearly said. That is the better failure. It is still a failure.