Resources

Benchmarks

Audited, reproducible numbers on real workloads.

llms.txt

Performance measured on real workloads. All numbers are audited and reproducible; the raw per-question hypotheses and judge labels are committed to the benchmarks directory.

LongMemEval

LongMemEval (ICLR 2025) is the standard benchmark for long-term conversational memory: 500 questions across six reasoning types, over long multi-session histories (~115K tokens per question). MenteDB scores on LongMemEval-S, all 500 questions, graded by the official judge (gpt-4o-2024-08-06, verbatim prompts).

CategoryAccuracy
Single-session (assistant)100.0% (56/56)
Single-session (user)97.1% (68/70)
Single-session (preference)93.3% (28/30)
Temporal reasoning92.5% (123/133)
Knowledge update91.0% (71/78)
Multi-session reasoning85.7% (114/133)
Overall (500 questions)92.0% (460/500)

Multi-session reasoning is the weakest category: synthesizing facts scattered across many sessions is the hardest problem, and where the engine has the most room to grow. We report the full distribution rather than a single headline. Each conversation is ingested chronologically with timestamps; MenteDB extracts memories (gpt-4o-mini) into vectors, BM25, and graph. For each question it runs hybrid retrieval (BM25 + vector, RRF-merged, time-filtered to memories dated before the question) and answers with gpt-4o. Embeddings are text-embedding-3-small. Grading uses the official LongMemEval judge, unmodified.

Quality benchmarks (5/5 passing)

TestResultKey Metric
Stale BeliefPASSSuperseded memories correctly excluded via graph edges
Delta SavingsPASS90.7% reduction in memory retrieval tokens over 20 turns
Sustained ConversationPASS100 turns, 3 projects, 0% stale returns, 0.29ms insert
Attention BudgetPASSU-curve ordering maintains 100% LLM compliance
Noise RatioPASS100% useful vs 80% naive, +20pp improvement

LLM accuracy (62 cases)

MenteDB's cognitive layer uses LLM judgment for memory invalidation, contradiction detection, and topic canonicalization. A curated suite of 62 cases validates accuracy across providers.

ProviderInvalidation (23)Contradiction (24)Topic (15)Total
Anthropic Claude Sonnet 4100%100%100%100% (62/62)
Ollama llama3.1 8b87%66.7%93.3%80.6% (50/62)
Ollama llama3.2 3B78.3%58.3%80%71% (44/62)

Three-tier design: works without any LLM (heuristics only), works well with a free local model via Ollama, and achieves perfect accuracy with a cloud API.

Token efficiency

The context assembler uses purpose-built serialization formats instead of dumping raw JSON into context windows. Format comparison (25 memories, same content):

FormatTokensvs Raw JSON
Raw JSON947baseline
Structured (markdown)5761.6x fewer
Compact (pipe-delimited)4142.3x fewer

Multi-turn delta serving (20-turn conversation):

MetricTokens
Full retrieval (cumulative)9,863
Delta serving (cumulative)2,004
Savings79.7%

Delta serving only sends memories that changed since the last turn. Early turns have overhead from the delta header, but by turn 10+ savings exceed 85% per turn. Memory density: within a 4,096-token budget the compact format fits 223 memories vs 166 structured and 218 raw JSON; at 8,192 tokens it is 448 vs 333 vs 436.

Performance (Criterion)

Benchmark1001,00010,000
Insert (per memory)130us244us265us
Context assembly (per query)218us342us696us

Both insert and context assembly stay well under a millisecond per operation, even at 10,000 memories. (A bulk load of 10,000 is 2.65s total, which is that same 265us each.)

Scale and embeddings

10K scale test (local Candle embeddings)

  • Total memories: 10,006
  • Engine search at 10K (no embed): 1.05ms
  • Engine insert, full write pipeline (no embed): 13.9ms per memory
  • Query embed (local Candle): 40ms
  • Belief changes: 6/6 correctly tracked
  • Stale beliefs returned: 0

Candle (local) vs OpenAI embeddings

  • Candle retrieval accuracy: 62% (5/8)
  • Engine search: 0.3ms with either provider (same code path)
  • Query embed: 39ms local with Candle, a network round trip with OpenAI
  • OpenAI cost: ~$0.02 per 1M tokens

These are self-host embedder options: Candle gives good quality for zero-config local use, OpenAI offers higher accuracy. The managed cloud uses Cohere Embed English v3 on Bedrock (1024 dims) so you never configure an embedder or bring a key; the self-hosted server can use the same model with --embedding-provider bedrock.

Performance targets (10M memories)

~50ns

Point lookup

~10us

Multi-tag filter

~5ms

k-NN similarity search

<50ms

Full context assembly

<1ms

Startup (mmap)