Chapter 16

How Memory Gets Found

Part 1 showed in what form memory is stored. Part 2 looks at how stored memory is found and how it is handed on.

When a question arrives, the system first has to find the memory that might bear on it. But a search result is not yet an answer. This chapter looks at what search does and where it stops.

1. Search is not one method

Full-text search is strong at finding a particular word or phrase. It confirms that the expression actually occurs in a document.

For Korean that strength needs one adjustment. Korean attaches particles to the end of a word, so an index that keeps words whole holds 승부는 and never matches a question that says 승부. Vinc's full-text index therefore stores Korean as overlapping pairs of syllables and splits the question the same way (on the desktop from 0.9.6), so 승부 meets 승부는 without a dictionary. English words are kept whole.

Vector search is strong at finding candidates that are close in meaning even when the wording differs. Ask the question in other words and it still brings back pieces from the same region of meaning.

The two do not compete. One finds matching wording, the other finds nearness of meaning. Vinc runs both, and the search response always says which arms ran. If the vector arm could not run, the answer comes from full text alone and says so, so that half an answer never looks like a whole one.

2. Two lists become one

Each arm returns its own ranked list, and the two lists do not share a scale. A full-text score and a distance in vector space are not the same kind of number; neither can be trusted as an absolute, and adding them means nothing.

So the lists are merged by rank, not by score. A piece near the top of both lists rises; a piece only one arm liked sinks. This is reciprocal rank fusion, and its one setting is how steeply the top of a list is rewarded over the rest. Vinc sets it steep on purpose. What goes on to the next stage is not everything either arm found but the few both arms put near the top.

In one case a list is set aside before the merge. Ask in Korean over English text and the full-text arm returns Korean pieces that share a syllable pair or two with the question and nothing else; merged by rank, they push the vector arm's right answer down. So when the vector arm has answered and none of the top full-text results holds even half of the question's terms, the full-text list is left out, and the response says so (keyword_gated). On the measured question sets no axis got worse, and Korean questions over English text gained the most. Questions asked in other words are where it costs something: there it often drops the full-text list, and only the meaning arm is left.

The easy alternative, re-fusing with the vector arm down-weighted, was measured too. It was worse than plain fusion on every axis, because that arm is exactly the signal that carries a question across languages.

3. A third reader, only when the two disagree

Neither arm reads the question and the candidate together. A cross-encoder does. It takes the pair and judges how well the one answers the other. That judgment beats either arm, and it costs far more per pair, so it cannot read everything.

So there is a gate. The third reader is called only when the two arms disagree about first place. When both put the same piece at the top, there is nothing to reorder. Measured on a set of human-written questions, the gate lost nothing: the questions it skipped were exactly the ones where reranking would have changed nothing.

Where the third reader earns its cost is across languages. A Korean question against English text is where the two arms disagree most and where reranking moves the result most. On natural questions in one language it barely moves at all.

Two more things came out of the measurement. The reranker has to read the whole piece. Reading only the opening of a piece shows barely half of a Korean piece, and Korean questions lose ground; splitting the piece into the same units the embedding uses removes the loss. And model size hardly mattered: a model five times smaller gave the same result. What sets the cost is not the model but the number of pairs read.

This reranker is not deployed. On 50 hand-written questions it raised nDCG@10 by +0.016, and the 95% confidence interval, [−0.063, +0.101], includes zero. Part 4 has the rest of the measurement.

4. Search narrows; resolution decides

That is search. From one question, two arms run, one list forms, and a third reader reorders it when needed. What remains at the end is candidates.

Finding a sentence with a high similarity score does not make that sentence the answer to this question. If several Concepts with similar titles came back, taking the top score is not safe: different memories can share similar wording.

Each candidate also carries the passage it was found by. The excerpt is the stretch of the piece that holds the most of the question's terms, not its first lines, because an Agent decides from the excerpt which document to open.

Search narrows where memory might be. Deciding which memory it is belongs to a different step, and that step needs rules of its own. The next two chapters are that step: Chapter 17 looks inside the vector arm, and Chapter 18 follows the candidates to a single Node.

Continue with Chapter 17, How Meaning Becomes Space.