Research and eval

Node resolution

On 2026-10-19, brief resolved 15 of 60 answerable questions to the right node, 2 to a wrong node, and 3 of 21 unanswerable questions to some node.

What brief returns

status	Meaning
`resolved`	Settled on one node and answers with it
`choose`	Returns up to five ranked candidates
`none`	Found nothing

Candidates come from title search, vector search over node embeddings, and graph search.

Resolution rule

brief resolves only when the top candidate beats the second by at least 2 matched tokens, either in its text (h) or in its title (t). There is no score threshold.

r e s o l v e d ⟺ h_{1} > 0 \land (h_{1} - h_{2} \geq 2 \lor t_{1} - t_{2} \geq 2)

A question identical to one node's title resolves to that node.
A lone candidate resolves only if h > 0. Vector similarity alone never resolves.
When every candidate came from vector search alone, none shares a word with the question, and even the nearest is farther than 0.4297, brief answers none.

Pass rate by step

1. In the candidate poolSearch returned the gold node51 / 60

2. In the list shownAmong the five candidates (R@5)51 / 60

3. Top threeWithin the top three (R@3)49 / 60

4. Ranked firstFirst in the list (R@1)45 / 60

5. Resolvedbrief answered with it15 / 60

Out of 60 answerable questions, measured 2026-10-19.

Most questions are lost at the last step: in 30 the gold node was ranked first but brief did not resolve. Of the 9 that missed the candidate pool, 7 were Korean questions.

Metrics

Q: answerable questions (60). U: unanswerable (21). R: questions brief resolved. rank(g): position of the gold node in the returned list, 0 if absent.

Metric	Formula	2026-10-19
R@1	$\frac{1}{\| Q \|} \sum_{q \in Q} 1 [r a n k (g_{q}) = 1]$	45/60
R@3	$\frac{1}{\| Q \|} \sum_{q \in Q} 1 [0 < r a n k (g_{q}) \leq 3]$	49/60
R@5	$\frac{1}{\| Q \|} \sum_{q \in Q} 1 [0 < r a n k (g_{q}) \leq 5]$	51/60
Correct resolutions	$\frac{\| {q \in Q \cap R : {\hat{y}}_{q} = g_{q}} \|}{\| Q \|}$	15/60
Wrong resolutions	$\frac{\| {q \in Q \cap R : {\hat{y}}_{q} \neq g_{q}} \|}{\| Q \|}$	2/60
Unanswerable resolved	$\frac{\| U \cap R \|}{\| U \|}$	3/21
Choose rate	$\frac{\| {q : {s t a t u s}_{q} = c h o o s e} \|}{\| Q \| + \| U \|}$	61/81
Precision of resolutions	$\frac{\| {q \in R : {\hat{y}}_{q} = g_{q}} \|}{\| R \|}$	15/20

R@k by language and node type

Question → node	n	R@1	R@3	R@5
English → English	27	20/27	23/27	25/27
English → Korean	10	10/10	10/10	10/10
Korean → Korean	10	4/10	4/10	4/10
Korean → English	13	11/13	12/13	12/13

Gold node type	n	R@1	R@3	R@5
Concept	30	21/30	23/30	24/30
Record	10	9/10	9/10	9/10
Decision	10	8/10	9/10	10/10
Episode	10	7/10	8/10	8/10

All three unanswerable resolutions came from questions about a fact never recorded on an existing topic. The 14 questions about a missing topic all got candidates.