Chapter 33

Failures That Look Like Success

Chapter 32 ended on an observation about its own material: every defect in it was silent. Nothing raised. Nothing logged. Each one produced a store that looked healthy, a call that reported success, a number that was perfectly plausible.

That is not a coincidence of one chapter. It is the dominant failure shape in this system, and it deserves its own. The dangerous failure in an evaluated system is not the one that breaks. It is the one that returns a normal-looking answer to a question nobody actually asked.

1. A wrong experiment still produces a number

Evaluate cross-lingual retrieval on a fixture that contains no cross-lingual pairs and the benchmark runs. The metric computes. The result file is well formed. What it is not is a measurement of cross-lingual retrieval.

Chapter 29's two rounds are the same shape: Korean queries with no Korean characters in them, and a harness reading a key the result never carries. Both produced confident numbers. Neither produced an error.

So the first rule of this chapter is that the absence of an error is not evidence. An evaluation has to check its own fixture before it reads any metric off it: that the pairs it claims to test actually exist, that the queries said to be in a language are in it, that the answers are inside the corpus being searched.

2. An arm that did not run

During the run of Chapter 30, the search response was recorded on every call. Eight calls fused both arms; four fell back to text alone with a read timeout on the vector arm.

If the system had simply returned the text results in those four cases without saying so, the search would have looked successful from the outside. And it would have been, for a text search. But the thing under evaluation was hybrid retrieval, and a hybrid result computed from one arm is not a hybrid result.

This is why the search response names which arms ran, and when an arm was attempted and failed, why. A degraded search and a complete one are indistinguishable from their results alone, so the result has to say. The honest statement after that run was not "the vector arm is back" but "the vector arm answers roughly two times in three, and which calls it answers is not under the caller's control."

3. A gate that never fired

The reranking gate of Chapter 16 runs a cross-encoder only when the two search arms disagree about first place, skipping the work when they already agree. For a while it fired zero times in seventy five queries.

The mechanism: the gate recognised agreement by a fused score, and the threshold it compared against was pinned to a module constant of 60, while the pool it was judging had been fused by production at 20 since an earlier change. The agreement it was looking for could not occur.

Nothing in the output said so, and this is the part worth sitting with. A gate that never fires prints a column identical to no reranking. A gate that correctly skips every query prints exactly the same column. Two opposite states, one output. The result file even recorded the other fusion setting in its own metadata while judging a pool built with this one.

It was found in September 2026, not by reading numbers but by making the gate state which fusion its pool came from instead of assuming one. Re-measured with the constant fixed, the gate skipped 11 of 75 queries and its numbers were identical to always reranking on every metric and every language axis, to three decimals and to the same interval. So the gate costs nothing and saves about fifteen percent of the reranking calls, which is what a working gate should look like. On cross-lingual queries it fires on all of them, and on same-language queries it skips most: it skips exactly the queries where reranking is worthless.

That last sentence is only sayable because the failure was found. Before it was, the same column was being read as evidence that the gate worked.

4. A status that misreported its own failure

Chapter 30 measured a composite tool returning "nothing found" five times out of five on one kind of node. The contract defines that status as the question having matched nothing. What had actually happened is that the question matched precisely and the loader refused to open the winner.

Those are different facts with different remedies, and the caller could not tell them apart. The truth was in an accompanying note, and a caller cannot branch on a note.

The fix made one case quieter, and that was measured rather than feared. Once the node could be loaded, one query whose ranking already preferred the wrong node stopped answering "nothing found" and started answering confidently about that wrong node, with no substitution reported. The failure moved from loud to silent, which is the wrong direction. It was deliberately not papered over: the ranking question touches every query and was left for its own evaluation, pinned meanwhile as a test that asserts the wrong answer and says why, so that whoever changes the ranking has to come and invert it.

5. A field accepted, dropped, and not reported

One write carried six relations, each with an evidence sentence. Read back immediately: the one permitted kind kept its sentence intact, and the other five came back empty. The response's warnings were empty and no partial flag was set.

So a writer following the house style, which asks for that sentence, gets no signal that the sentence went nowhere. The silent drop is the worse half of the finding, worse than the contradiction between two documents that caused it, because a contradiction on paper is discoverable and a dropped field is not.

6. A document stored truncated that reads as prose

This one is closest to home. On 16 September 2026 the English twin of Chapter 28 was stored cut off mid-sentence, partway through the fourth of its eight sections. The write reported success. The document opened. The text that was there was well formed, ended in a paragraph, and read exactly like prose. Nothing about it looked wrong, and the gap was found only by counting sections against the Korean source. It was closed by reconstructing the missing sections and confirming the twin read complete.

The general rule this yields is the one Part 4 keeps arriving at from different directions: a quietly truncated document is indistinguishable from a short one. So the store carries an integrity flag that says whether what came back is what was stored, and a read that cannot answer that question has to say so rather than answer from the text anyway. A write is not finished when it reports success. It is finished when it has been read back.

7. A filter that manufactured confidence

The subtlest case in the corpus, because nothing was broken.

Chapter 24 gave claims a validity window, and the read path was changed to return current claims by default. That is a filter, and a filter is supposed to change what is visible.

This one changed the answer. Two candidates had scored the same, and the tool had honestly answered "choose", because a tie is not a resolution. One of the two was historical. The filter removed it, the survivor stood alone, and the same question came back "resolved" with a confidence nothing had earned. One of six questions that had no answer in the graph was answered that way.

The fix was not to weaken the filter but to keep the tie: when a dropped claim had tied with the survivor, the tool refuses to resolve. The lesson generalises past this filter. Anything that removes candidates before a tie is broken can turn honesty into confidence by accident, and the only defence is to measure the honesty metric every time the read path changes.

8. An empty result is information; a substituted one is not

An empty search does not mean the system failed. The question may genuinely have asked for something that is not there, and saying so is a correct answer.

The real problem is a search that comes back thin and an Agent that picks something else rather than return nothing. Which is where the resolution principle of Chapter 18 earns its place:

When it is ambiguous, do not pick. Leave the result empty, and recover through another path if one exists. That is how a system keeps the difference between what it knows and what it inferred.

9. Recovery is part of what gets measured

Because of all of the above, an evaluation has to follow what happens after a failure as well as what happens instead of one. After an empty or wrong result comes recovery, and only then resolution.

Record whether recovery succeeded, how many extra calls it cost, and whether evidence was finally reached. Only then is "answered on the first try" distinguishable from "answered after five wrong turns", and only then does the number in Chapter 30's call column mean what it appears to mean.

Every failure in this chapter was found before something was built on it, or shortly after. That timing is not luck. It is what the ordering in the next chapter is for.

Continue with Chapter 34, Measurement Comes Before Decision.