Research and eval

Agent task evaluation

In the September 2026 pilot, an agent answered 60.7% of multi-step questions correctly.

What is evaluated

Questions that need several steps through the graph, answered by an agent using four read-only tools: vinc_brief, vinc_search, vinc_neighbors and vinc_path.

Type	Question	n	Accuracy
trace_decision	Why a decision was made	8	91.7%
direct	One property of one node	8	75.0%
collection	Every node meeting a condition	7	66.7%
path	How two nodes connect	6	33.3%
chain	Follow nodes one hop at a time	6	27.8%

Six to eight questions per type is too few to rank the types.

Pilot results

September 2026. 38 questions (3 unanswerable) on a separate graph, each run three times with claude-haiku-4-5, at most 10 tool calls per run.

Metric	Result
Answer accuracy (105 runs)	60.7%
Gold node visited	89.3%
Unanswerable answered with confidence	7 of 9 runs
Same answer in all three runs	14 of 38
Tool calls per run	2.85

The agent reached the gold node in 89.3% of runs but answered with it in 60.7%. On path questions every run reached the gold node.

Metrics

Metric	Formula
Answer accuracy	$\frac{1}{N} \sum_{i} 1 [n o r m ({\hat{a}}_{i}) = n o r m (a_{i}^{*})]$
Gold node visited	$\frac{1}{N} \sum_{i} 1 [g_{i} \in S_{i}]$
Answer consistency	$1 [{max}_{a} {c n t}_{q} (a) \geq ⌈ R / 2 ⌉]$
Tool calls per run	$\frac{1}{N} \sum_{i} s_{i}$

N: runs. â_i, a_i^*: the agent's answer and the gold answer. S_i: nodes visited in run i. R: runs per question. s_i: tool calls in run i.