Research and eval
Eval methods
What is evaluated, and the latest result for each stage.
A question passes three stages: search finds text, vinc_brief settles on a node, and an agent may repeat this over several steps. Each stage is evaluated separately so a failure can be traced to one stage.
1Retrieval evaluationDoes search rank the right document near the top?
2Node resolutionDoes brief settle on the right node?
3Agent task evaluationDoes an agent finish a multi-step question?
Latest results
| Stage | Metric | Result | Measured |
|---|---|---|---|
| Retrieval | nDCG@10 (canonical · hand-written) | 0.733 · 0.781 | 2026-10-22, 118 and 75 questions |
| Node resolution | Correct resolutions | 15 / 60 | 2026-10-19, 81 questions |
| Agent tasks | Answer accuracy | 60.7% (pilot) | September 2026, 35 questions × 3 runs |
The stages use different graphs and answer keys, so these figures cannot be combined.
Terms
- Resolved: brief settled on one node and answered with it.
- Choose: brief returned candidates instead of settling.
- Exposed: the right node appears in that candidate list. This is not counted as an answer.
Evaluation design
- Each question names a gold node, not a gold document.
- 21 of the 81 node questions have no answer in the graph. Resolving one counts as a failure.
- The evaluation graph is fixed by byte hash and encoder runtime, and a measurement is refused if the query encoder differs.
- Two measurements are compared question by question (paired bootstrap, exact McNemar test).
Caveats
- The node answer key was written by the same agent that built the resolution rule. A change that improves only this key is not applied.
- The hosted service does not store query text, so no figure here comes from real usage.
- Other products have not been measured, so there is no comparison.