Core concepts
Sleeptime enrichment
The background pipeline that turns conversations into structured knowledge.
A 4-phase background pipeline that automatically converts raw conversation memories into structured knowledge. Named after the brain's process of consolidating memories during sleep, it triggers automatically after process_turn when enough new episodic memories accumulate and an LLM provider is configured. No extra API calls, it runs in the background.
The 4 phases
- Batch LLM extraction, scans recent episodic memories and uses the configured LLM to extract structured semantic facts and entity nodes. Raw conversation turns become precise, reusable knowledge.
- Entity linking, resolves duplicate and alias entities using rule-based matching combined with LLM disambiguation. "JS" gets linked to "JavaScript." Produces a clean entity graph.
- Community detection, groups related entities into communities by category (people, technologies, places, projects) and generates LLM summaries for each community.
- User model, synthesizes an always-available user profile from the accumulated semantic facts and community summaries. Surfaced on every
process_turncall.
Impact on retrieval
Enrichment results feed directly back into process_turn context retrieval:
- Semantic facts, precise, structured memories that score higher in hybrid search than raw conversation text
- Entity graph, connected entity nodes enable graph-aware retrieval and relationship traversal
- Community summaries, high-level topic summaries provide broader context without token overhead
- User profile, an always-available profile of the user's preferences, projects, and patterns
Consolidation & decay
Similar memories are merged into unified semantic knowledge. The engine clusters eligible memories (episodic, >24h old, accessed >2 times) by cosine similarity, then merges each cluster: deduplicating sentences, averaging embeddings, and creating Derived edges back to sources. Source memories are invalidated, not deleted, preserving full history. Decay ages salience so old, unused memories rank lower over time, and archival forgets cold memories flagged for deletion.
LLM required, provenance, idempotency
Enrichment requires an LLM provider (OpenAI, Anthropic, or Ollama). Without one, the engine works perfectly, enrichment simply does not run, and all other features (hybrid search, write inference, contradiction detection) are unaffected. The pipeline is idempotent, and all outputs are tagged with source:enrichment and connected to their source memories via Derived edges, so you can trace where any piece of extracted knowledge came from. Every derived memory also carries the attributes origin: "derived" and derived_kind (fact, entity, community or profile), plus any attributes named in MENTEDB_ENRICH_COPY_ATTRIBUTES, copied from its sources (a document id, for example).
On the self hosted server
- Watermark on disk. The time of the last run is saved in the database directory, so a restart or a reopened database continues from there instead of enriching everything again.
- Candidate types. Enrichment reads episodic memories by default;
MENTEDB_ENRICH_CANDIDATE_TYPES(for exampleepisodic,semantic) widens it. Derived memories are never candidates. - Its own worker pool.
POST /enrichruns onMENTEDB_ENRICH_WORKERSthreads (default1), separate from request handling, so a long LLM pipeline never holds up reads. - Reader model.
--llm-reader-modelsets the model for the judge steps (entity linking, community summaries, the user model);--llm-modelstays the extraction model.
Configure a provider under Self-hosted, or use Cloud where it is wired up for you.