Benchmark
92%
LongMemEval accuracy
460 / 500 questions
LongMemEval (ICLR 2025) is the standard benchmark for long-term conversational memory: 500 questions across six reasoning types, over long multi-session histories (~115K tokens each). MenteDB ingests each conversation, then answers every question. Graded by the official judge, unmodified.
Results by category
We report the full distribution, including our weakest category rather than a single headline. Multi-session reasoning, synthesizing facts scattered across many sessions, is the hardest problem and where the engine has the most room to grow.
Single-session (assistant)56 / 56100.0%
Single-session (user)68 / 7097.1%
Single-session (preference)28 / 3093.3%
Temporal reasoning123 / 13392.5%
Knowledge update71 / 7891.0%
Multi-session reasoning114 / 13385.7%
Overall (500 questions)460 / 50092.0%
How we measured
- 1. Ingest. Each conversation session is added chronologically with its timestamp. MenteDB extracts memories (gpt-4o-mini) into vectors, BM25, and a graph.
- 2. Retrieve. For each question, hybrid search runs BM25 + vector, merged with Reciprocal Rank Fusion, time-filtered to memories dated before the question.
- 3. Answer. The retrieved context and the question go to gpt-4o via the LongMemEval reader prompt.
- 4. Grade. The official LongMemEval judge (gpt-4o-2024-08-06, verbatim prompts) scores each answer. Embeddings are text-embedding-3-small.
The raw per-question hypotheses and the judge labels are committed in the repo, so every number here is auditable.
Benchmark code + raw resultsScope & caveats
- Measured on LongMemEval-S, the 500-question set with ~115K-token histories. It is the original (V1) benchmark, not the newer V2 leaderboard.
- This is end-to-end answer accuracy (retrieve then answer), not retrieval recall alone.
- Our weakest category is multi-session reasoning (85.7%). We show it rather than hide it.
Memory that reasons, not just stores.