Chapter 29

Measurement Needs Conditions

Chapter 28 closed Part 3 by saying that whether this design holds is not something to argue but something to measure. Part 4 is that measurement, and it is the only part of the book where the claims are numbers taken on a particular day under particular conditions.

Which is why it has to begin here. A number is the cheapest thing a system can produce. Producing one that means something takes more work than producing one at all, and the difference is entirely in the conditions.

1. A number with its conditions stripped off is not a result

Say a retrieval run scores 0.67 nDCG at ten. On its own that says nothing about whether it beats another run's 0.62. The two runs may have searched corpora of different sizes, and a corpus with fewer distractors makes retrieval easier and the score higher without the system being any better.

So Vinc's own rule, written into the evaluation plan, is blunt: every model runs against the same corpus, query set, answer set, chunking rules, retrieval depth, hardware class, thread count and evaluation script, and absolute scores from different fixtures are not compared. The M3 rebuild rehearsal repeats it over its own retrieval table: those values belong to that corpus and must not be set beside the earlier spikes.

What survives comparison is the difference between two arms inside one fixture. That is a much smaller claim than a leaderboard, and it is the one that holds.

2. The fixture is the part that is held still

A fixture is the fixed input set an evaluation runs against: the documents, the queries, the answers, the language mix, the depth. Fixing it is what turns an observation into a comparison.

For Vinc the language axes are the load-bearing part of the fixture, because the corpus is written in English by house style while the owner asks questions in Korean. Four axes are named and kept apart:

en     English query, English corpus
ko     Korean query, Korean corpus
ko2en  Korean query, English corpus
en2ko  English query, Korean corpus

ko2en is the one the product actually lives on. Collapse the four into one average and that axis disappears into the mean, which is a way of not measuring the thing that matters while appearing to measure everything.

3. Two rounds that measured the wrong thing and said nothing about it

Both of these happened here, and neither announced itself.

The Korean queries had no Korean in them. The first embedding measurement round drew its queries from document titles and filenames. Those are ASCII. So the queries labelled Korean contained zero Korean characters, and Korean retrieval had never been measured at all while its score was being read off the table. The round was thrown away and the queries rebuilt from body subheadings in the matching language.

The harness read a key that is never present. The first evaluation run of the M3 rebuild rehearsal reported recall 0.0 across every set. That was not a result. vector_search returns id, which is the chunk id, and the harness was asking for doc_id. Every lookup missed, every query scored zero, and zero is a perfectly plausible number for a system that has just changed its embedding model.

The shape is the same both times. A broken instrument does not produce an error; it produces a confident number. That is why Part 4 spends as much space on how things were measured as on what came out.

4. What has to be written down for the result to be re-readable

The evaluation plan lists what a candidate measurement records, and the list is longer than the metric:

• the model identifier and the exact artifact version
• the dimension
• the precision and the quantization method
• the export path and the runtime version
• the kernel the artifact requires, and the instruction support the machine actually has
• throughput in chunks per second, and encode latency
• peak resident memory and artifact size
• cold start behaviour
• rebuild time on the real corpus, and rollback cost
• the vector storage and index consequences
• the metric, the result, and if it failed, why

Take any line out and the number stops being re-interpretable a month later. The kernel line is there because of a specific loss described in Chapter 31: a quantized artifact that assumes an instruction extension the processor does not have gives back most of what the model gained, and nothing in the score says so.

5. Two measurements minutes apart beat two measurements days apart

The rehearsal measured its own rebuild path twice, in different sessions, and got 24.6 seconds and 4.42 seconds for comparable work. Rather than pick one, it recorded that two runs in different sessions under different machine load are not comparable, called the earlier figure a shape rather than a constant, and then ran both variants back to back on one machine against one source store to get a comparison that holds.

The same discipline shows up in the smaller checks. Competing processes were counted before any timing was taken. Index build time was reported separately from encode time, because a dimension change rebuilds the index and "faster" could otherwise be an artifact of index state. A self check confirmed that both sides of a model comparison were running the same production code, so implementation difference is excluded rather than assumed away.

6. The cheapest gate runs first

Not every check needs a fixture. On 7 September 2026 two new embedding candidates were screened against their model cards and this repository before any fixture was built, and both failed: one on licence, both on the absence of an official int8 artifact. Two of the evaluation plan's own stated premises were wrong as well.

A measurement that a reading could have prevented is a measurement that should not have been run. Ordering the gates by cost, cheapest first, is not a shortcut around evidence; it is what keeps the expensive evidence for the questions that need it.

7. A run states what it does not support

Every measurement document in this corpus carries a section naming its own limits, and those sections are load-bearing rather than decorative. The MCP run of Chapter 30 declares that it is one session, one rater, seven tasks and no second grader, and that nothing in it is a benchmark. It declares that its rater had read the earlier run before starting, so the absolute call counts are worth less than the difference between the two arms, which the contamination affects equally. It declares which of its variables were uncontrolled.

None of that weakens the run. It fixes what the run may be quoted for later, which is the only thing that stops a number from drifting into a claim it never made. Fix the fixture, verify, run, measure, state the limits, and only then compare.

Conditions settled, the first thing worth measuring is the thing an Agent actually pays: not how fast one call is, but how many calls a question costs. That is the next chapter.

Continue with Chapter 30, A Tool's Efficiency Is Measured in Calls.