Chapter 31

How Embedding Models Get Compared

Chapter 30 ended on a retrieval failure that had nothing to do with the tool: a Korean question could not reach an English title and an English question could not reach a Korean one, with the semantic arm live in both directions. That is a question about the space Chapter 17 described, and answering it means comparing embedding models.

Which sounds like picking the one with the better score. It is not, and the reason is that an embedding model is not a component of the retrieval system. It defines the semantic space the whole system searches in.

1. Four things change together

Swap the model and four things move at once: retrieval quality, compute cost, the shape of the storage that holds the vectors, and operating cost.

Choose on the first alone and the other three arrive later as surprises. That is why the comparison in this chapter is never a single number, and why Chapter 32 exists at all.

2. What the language axis actually cost

The corpus is English and Korean. The English-only model then in use had no Korean syllable tokens at all, so Korean text decomposed into letters and retrieval collapsed.

Korean almost doubles and English gives up about five percent. That trade is the decision, and stating it as a trade rather than as an improvement is the honest form. Nothing here says the multilingual model is better; it says what each column costs.

3. Quantization is part of the model, not a deployment detail

Read the third column again. The vendor's own int8 build of the same multilingual model gave back most of the Korean gain: 0.670 down to 0.472.

The cause is specific rather than inherent. That build is quantized without range reduction, and the runtime's own documentation warns that this saturates the intermediate accumulator on processors without a newer instruction extension. The machine measured has the older one. Rebuilding the same model with range reduction restores Korean to 0.669, which is parity with full precision.

So the deployed artifact is a self-built int8, not the published one. Range reduction is harmless on processors that do have the extension, so the choice does not depend on the hardware and the hardware never had to be surveyed. What a model can do is not what this runtime can do.

A model that supports int8 does not thereby give a given system int8 performance. This is the loss Chapter 29's kernel line exists to record, and it is the clearest case in the book of a number that is correct, reproducible, and about the wrong thing unless its conditions travel with it.

4. The chunk and the model window are different units

A storage chunk is about two thousand characters. The model window is five hundred tokens. One vector per chunk therefore encoded the beginning and silently discarded the rest: measured at 42.6 percent of Korean text and 12.3 percent of English.

Three ways out were measured against the same queries.

Averaging failed, and it was the shortcut worth wanting, because it would have made re-chunking and its migration unnecessary. It scores below doing nothing: distinct content inside one chunk smears together, and a truncated chunk with a sharp beginning beats a blurred whole.

The design that shipped is parent and child. The chunk stays at two thousand characters and carries several vectors, one per piece of at most five hundred tokens, and a hit on a child returns the parent. Quality equals re-chunking exactly, because the vector set is literally the same pieces encoded by the same model, and the price is a schema change instead of a data migration.

What that bought is in Chapter 12's territory. A Concept links to a chunk when every significant token of its title appears in that one chunk, so shrinking chunks makes the conjunction monotonically less likely: at five hundred tokens links fall 2.5 percent in English and 10 percent in Korean. The proposed remedy, testing the condition across adjacent chunks, was measured too and it does not restore links, it inflates them, by up to 137 percent. Over-linking points a Concept at chunks that barely discuss it, which destroys the feature it was meant to protect. No window was added.

Two smaller findings from the same work, both of which change how the code has to be written. Token counts are not additive, so a boundary chosen from character offsets can still exceed the limit and a re-encode loop is mandatory rather than defensive. And a character limit cannot serve both languages: at the same token budget the measured chunk sizes were 482 characters of Korean, 2,660 of English and 1,101 of mixed text.

5. What a different model did to the cross-lingual axis

The candidate measured against the shipped model was 1024-dimensional. Same store, same queries, same depth, only the model and the vectors changed:

No set regressed, and the axis Chapter 30 ended on moves from barely working to working. Query latency roughly doubles, from about 45 ms to about 110 ms at the median, which is far inside the read budget.

Those absolute values belong to that corpus and must not be set beside the table in section 2. What section 2 and this table share is not a scale but a lesson: the axis that separates two models is not visible in a combined score.

6. The thing measured is the artifact, not the checkpoint

A model card is not what runs. Every candidate is measured as the file that would actually deploy:

Read the peaks as an upper observation on one machine rather than a container prediction, because the inference runtime's arena grows opportunistically. The container-relevant floor is the artifact size, and there the gap is unambiguous: 570 MB against 2.27 GB, in a container with a one gibibyte limit. A quality comparison that stopped at the first column would have recommended something that does not fit.

7. Gates, in order of cost, rather than a leaderboard

A candidate is not adopted for winning one metric. Five gates stand in front of adoption: no regression on the load-bearing language axes, acceptable throughput and memory on the actual worker hardware, an acceptable storage contract for the dimension, reproducibility through the whole export and quantization and runtime path, and migration economics that justify a full re-embed.

The last gate is the one that turns this chapter into the next one. Everything above compares models. None of it prices the act of changing one.

Two notes before that. The cross-encoder placed after fusion is not deployed, for the reason given in Chapter 16. And the candidate model of sections 5 and 6 was adopted and is the one deployed now. Where this chapter's tables say shipped, they mean the 768-dimensional model in use when the measurements were taken.

Which raises the question this chapter has been deferring. Changing a model is not a setting. The next chapter prices it.

Continue with Chapter 32, The Cost of Changing a Model and a Storage Structure.