Resources
Benchmarks
Audited, reproducible numbers on real workloads.
Performance measured on real workloads. All numbers are audited and reproducible; the raw per-question hypotheses and judge labels are committed to the benchmarks directory.
LongMemEval
LongMemEval (ICLR 2025) is the standard benchmark for long-term conversational memory: 500 questions across six reasoning types, over long multi-session histories (~115K tokens per question). MenteDB scores on LongMemEval-S, all 500 questions, graded by the official judge (gpt-4o-2024-08-06, verbatim prompts).
| Category | Accuracy |
|---|---|
| Single-session (assistant) | 100.0% (56/56) |
| Single-session (user) | 97.1% (68/70) |
| Single-session (preference) | 93.3% (28/30) |
| Temporal reasoning | 92.5% (123/133) |
| Knowledge update | 91.0% (71/78) |
| Multi-session reasoning | 85.7% (114/133) |
| Overall (500 questions) | 92.0% (460/500) |
Multi-session reasoning is the weakest category: synthesizing facts scattered across many sessions is the hardest problem, and where the engine has the most room to grow. We report the full distribution rather than a single headline. Each conversation is ingested chronologically with timestamps; MenteDB extracts memories (gpt-4o-mini) into vectors, BM25, and graph. For each question it runs hybrid retrieval (BM25 + vector, RRF-merged, time-filtered to memories dated before the question) and answers with gpt-4o. Embeddings are text-embedding-3-small. Grading uses the official LongMemEval judge, unmodified.
Quality benchmarks (5/5 passing)
| Test | Result | Key Metric |
|---|---|---|
| Stale Belief | PASS | Superseded memories correctly excluded via graph edges |
| Delta Savings | PASS | 90.7% reduction in memory retrieval tokens over 20 turns |
| Sustained Conversation | PASS | 100 turns, 3 projects, 0% stale returns, 0.29ms insert |
| Attention Budget | PASS | U-curve ordering maintains 100% LLM compliance |
| Noise Ratio | PASS | 100% useful vs 80% naive, +20pp improvement |
LLM accuracy (62 cases)
MenteDB's cognitive layer uses LLM judgment for memory invalidation, contradiction detection, and topic canonicalization. A curated suite of 62 cases validates accuracy across providers.
| Provider | Invalidation (23) | Contradiction (24) | Topic (15) | Total |
|---|---|---|---|---|
| Anthropic Claude Sonnet 4 | 100% | 100% | 100% | 100% (62/62) |
| Ollama llama3.1 8b | 87% | 66.7% | 93.3% | 80.6% (50/62) |
| Ollama llama3.2 3B | 78.3% | 58.3% | 80% | 71% (44/62) |
Three-tier design: works without any LLM (heuristics only), works well with a free local model via Ollama, and achieves perfect accuracy with a cloud API.
Token efficiency
The context assembler uses purpose-built serialization formats instead of dumping raw JSON into context windows. Format comparison (25 memories, same content):
| Format | Tokens | vs Raw JSON |
|---|---|---|
| Raw JSON | 947 | baseline |
| Structured (markdown) | 576 | 1.6x fewer |
| Compact (pipe-delimited) | 414 | 2.3x fewer |
Multi-turn delta serving (20-turn conversation):
| Metric | Tokens |
|---|---|
| Full retrieval (cumulative) | 9,863 |
| Delta serving (cumulative) | 2,004 |
| Savings | 79.7% |
Delta serving only sends memories that changed since the last turn. Early turns have overhead from the delta header, but by turn 10+ savings exceed 85% per turn. Memory density: within a 4,096-token budget the compact format fits 223 memories vs 166 structured and 218 raw JSON; at 8,192 tokens it is 448 vs 333 vs 436.
Performance (Criterion)
| Benchmark | 100 | 1,000 | 10,000 |
|---|---|---|---|
| Insert (per memory) | 130us | 244us | 265us |
| Context assembly (per query) | 218us | 342us | 696us |
Both insert and context assembly stay well under a millisecond per operation, even at 10,000 memories. (A bulk load of 10,000 is 2.65s total, which is that same 265us each.)
Scale and embeddings
10K scale test (local Candle embeddings)
- Total memories: 10,006
- Engine search at 10K (no embed): 1.05ms
- Engine insert, full write pipeline (no embed): 13.9ms per memory
- Query embed (local Candle): 40ms
- Belief changes: 6/6 correctly tracked
- Stale beliefs returned: 0
Candle (local) vs OpenAI embeddings
- Candle retrieval accuracy: 62% (5/8)
- Engine search: 0.3ms with either provider (same code path)
- Query embed: 39ms local with Candle, a network round trip with OpenAI
- OpenAI cost: ~$0.02 per 1M tokens
These are self-host embedder options: Candle gives good quality for zero-config local use, OpenAI offers higher accuracy. The managed cloud uses Cohere Embed English v3 on Bedrock (1024 dims) so you never configure an embedder or bring a key; the self-hosted server can use the same model with --embedding-provider bedrock.
Performance targets (10M memories)
~50ns
Point lookup
~10us
Multi-tag filter
~5ms
k-NN similarity search
<50ms
Full context assembly
<1ms
Startup (mmap)