Benchmarks
What was measured, and under what rule
These are the maintainer’s own runs, frozen in July 2026, on harnesses that are in the repository. Nobody outside the project has reproduced them. That is the first thing the table is for.
Conditions
- Maintainer-reported
- Frozen July 2026
- Harnesses open
- No independent reproduction yet
Read the tight-k row before the high-k row. Retrieving a large candidate set is a lenient ceiling. 72.6% at k=5 is the operational number. Evidence recall is not question-answering accuracy. Comparing this table with an LLM-judged score from another memory system would not rank either of them, which is why the comparison has no benchmark column.
Figures
| Figure | What it measures | Condition |
|---|---|---|
| 96.8% | LoCoMo evidence recall Source: benchmarks/results/locomo-invented-stack-engine-2026-07-13.json. Opens the repository in a new tab. | at k=150 · 10 conversations, 1,982 questions · MiniLM-384 |
| 72.6% | LoCoMo evidence recall Source: benchmarks/results/locomo-invented-stack-engine-2026-07-13.json. Opens the repository in a new tab. | at k=5 — the operational number |
| 97.6% | LongMemEval-S session recall Source: benchmarks/results/longmemeval-colab-v2-full-2026-07-04.json. Opens the repository in a new tab. | at k=8 · 488 of 500 |
| 97.4% | LongMemEval end-to-end QA Source: benchmarks/results/e2e-cert-paper-v2-2026-07-07.json. Opens the repository in a new tab. | 487 of 500 · locked run, gpt-4o reader and judge |
| 0.646 | BEIR SciFact nDCG@10 Source: benchmarks/results/paper-2026-07-09.json. Opens the repository in a new tab. | against Chroma 0.645 in a shared harness |
The figure that was withdrawn
99.0% LoCoMo evidence recall at k=150. The old figure expanded every retrieved turn by three neighbours on each side, then counted neighbours the engine never retrieved. A plain BM25 baseline also reaches about 99% under that protocol, so it distinguished nothing. It is not the headline any more, and it is not a number we will defend. Source: README.md:70 · CHANGELOG.md:49. Opens the repository in a new tab.