Core concepts
Inside process_turn
Every step the turn pipeline runs, in order.
process_turn is a single call that runs the full conversation-turn pipeline. Instead of manually orchestrating embedding, retrieval, storage, and cognitive features, one call handles everything. Here is what runs, in order, on every turn.
The 14-step pipeline
- Embed query + speculative cache check
- Hybrid retrieval (HNSW vector + BM25 keyword)
- Pain signal matching
- Delta tracking (new/changed since last turn)
- Store episodic memory with project tagging
- Write-time inference (contradiction + obsolescence detection)
- Fact extraction + edge linking
- Action detection + proactive recall
- Auto-correction detection
- Sentiment analysis
- Phantom tracking + stream contradiction check
- Trajectory tracking + ghost memory inference
- Speculative cache update for predicted topics
- Auto-maintenance (decay, archival, consolidation on intervals)
- Sleeptime enrichment, if triggered, runs batch LLM extraction, entity linking, community detection, and user model generation in the background
It returns a result with context, stored IDs, pain warnings, detected actions, proactive recalls, contradiction count, sentiment score, and predicted topics. When an LLM provider is configured (OpenAI, Anthropic, or Ollama), sleeptime enrichment runs automatically after enough memories accumulate, extracting structured facts, linking entities, building community summaries, and synthesizing a user profile that all feed back into future context retrieval.
Write-time intelligence
Most memory tools store everything and retrieve by similarity, filling context windows with noise. MenteDB filters at write time: an LLM extracts only what matters (decisions, preferences, corrections, facts, entities), a quality filter rejects low-confidence extractions, deduplication checks embedding similarity, and contradiction detection flags conflicts before they hit storage.
Write inference
Every store()finds the 20 most similar existing memories and runs heuristic inference. The bands are mutually exclusive: similarity > 0.95 flags a contradiction for review (both memories stay valid), 0.85 to 0.95 supersedes the older memory (setting valid_until), and 0.6 to 0.85 creates Related edges. A superseded or corrected memory has its confidence halved, and that drop reaches the memories that depend on it (see belief propagation under Graph edges). No LLM calls, pure vector heuristics for zero-latency inference.
Cognitive systems
MenteDB is a cognitive engine, not just a memory store. These systems are wired into the core facade and run automatically.
Salience decay (automatic on retrieval)
Memory relevance decays over time using an exponential formula: decayed = salience × 2^(-Δt / half_life) + boost × ln(1 + access_count). Half-life defaults to 7 days (configurable). Frequently accessed memories resist decay. Final retrieval score blends 70% similarity with 30% decayed salience, and memories never decay below 0.01. Decay is worked out when it is read: the stored base is internal and never rewritten, and recall, archival, dashboard health and MQL all see the current decayed value. Pinned memories keep their stored value.
Pain registry
Tracks recurring failures as pain signals. When similar contexts arise in future queries, the engine surfaces warnings like “this went wrong before.” Pain signals decay over time and match on trigger keywords via record_pain() and get_pain_warnings().
Interference detection
Finds confusable memory pairs, embeddings similar enough to cause retrieval confusion, and returns an InterferencePair with similarity scores and auto-generated disambiguation hints. Use during context assembly to add clarifying notes or separate confusable memories.
Phantom tracking
Detects “phantom memories,” entities referenced in conversation but never stored. Scans content for mentions not in the entity registry, flagging knowledge gaps with priority levels (Low to Critical) that can be resolved once the missing knowledge is captured.
Speculative cache
Pre-fetches context for predicted topics using the trajectory tracker’s transition patterns. When a query matches a cached prediction (by keyword overlap or embedding similarity), the pre-assembled context returns instantly, eliminating a retrieval round-trip. Entries auto-evict after configurable staleness windows.
Trajectory tracking
Records conversation turns as a trajectory, tracking topic evolution, decision states, and open questions. Provides get_resume_context() for session resumption and predict_next_topics() for speculative pre-fetching. Transition patterns persist across sessions and reinforce on cache hits.