Research and eval

Retrieval evaluation

Search combines keyword and vector search with RRF k = 20, indexes Korean as two-syllable pairs, and uses no reranker. Each choice was measured on a fixed store of 176 documents.

How search is set up

Step	Setting
Keyword search	Full-text search over chunks; Korean indexed as two-syllable pairs
Keyword gate	The keyword list leaves fusion when vector results exist and none of the top 20 keyword results holds half the query's terms
Vector search	Chunk vectors from the embedding model
Fusion	RRF, k = 20
Reranker	None

The fusion score of chunk d, where r_m(d) is its rank in search m starting from 1:

R R F (d) = \sum_{m \in {f t s, v e c}} \frac{1}{20 + r_{m} (d)}

Results

Set	Questions	nDCG@10	R@1	R@3	R@5
Canonical	118	0.733	0.619	0.771	0.822
Hand-written	75	0.781	0.640	0.800	0.853

Measured on the production setup as it runs, 2026-10-22: each document piece encoded alone, as the hub does, on a CPU with AVX-512 VNNI like the hub's. Against the 2026-10-19 measurement, which encoded pieces eight at a time, every metric moved within noise.

Scoring

Each question has one gold document. The top ten chunks are grouped into documents in order and the gold document's rank is scored.

Metric	Formula
R@k (k = 1, 3, 5)	$\frac{1}{\| Q \|} \sum_{q} 1 [0 < {r a n k}_{q} \leq k]$
MRR	$\frac{1}{\| Q \|} \sum_{q} \frac{1}{{r a n k}_{q}}$
nDCG@10	$\frac{1}{\| Q \|} \sum_{q} \frac{1}{\log_{2} ({r a n k}_{q} + 1)}$

A missed question scores 0. With one gold document the ideal DCG is 1, so nDCG@10 reduces to the term above.

Question sets

All sets run on a fixed store of 176 Korean and English documents.

Set	Questions	Source
Canonical	118 (60 same language, 28 Korean → English, 30 English → Korean)	Section headings of the gold document and their translations
Hand-written	75 (50 same language, 25 Korean → English)	Questions a person would type, sharing no three consecutive content words with any heading

Headings share words with the body text, so the canonical set favours keyword search.