The big shift over the last year has been away from "give the model everything that ever happened" toward memory as a retrieval problem, much like RAG for documents.
Most production agent systems end up with something like this:
Conversation
│
▼
Event extraction
│
├── Short-term context (recent messages)
├── Episodic memory (important events)
├── Semantic memory (facts)
└── Procedural memory (preferences/rules)
│
▼
Vector + structured storage
│
▼
Retrieve only relevant memories
│
▼
Inject into next prompt
That keeps token costs almost constant even over months of use.
1. Separate memory into different types
This is probably the biggest architectural improvement.
Working memory
- last 10–50 turns
- always in context
- discarded naturally
Episodic memory
- "User solved OAuth issue by disabling proxy."
- "Conversation about vacation plans."
- timestamped
- searchable
Semantic memory
Example:
{
"name": "Alice",
"favorite_language": "Rust",
"company": "Acme"
}
These can overwrite previous values instead of accumulating forever.
Procedural memory
Instructions the agent has learned:
- prefers concise answers
- always uses metric units
- hates emojis
Those rarely change.
2. Don't embed every message
A common beginner mistake:
message 1
embed
message 2
embed
message 3
embed
...
Months later:
- millions of embeddings
- noisy retrieval
- expensive
Instead:
conversation
↓
summarizer
↓
3-10 durable memories
↓
embed only those
One hour of conversation might become:
User switched jobs.
Interested in FPGA design.
Planning a Japan trip in October.
Uses Arch Linux.
Four vectors instead of hundreds.
3. Store structured facts separately
Not everything belongs in a vector DB.
Good candidates for SQL/JSON:
preferences
favorite_editor = neovim
timezone = PST
company = OpenAI
Lookup:
SELECT *
FROM preferences
WHERE user_id=...
That's faster, cheaper, and deterministic.
Vectors are best for fuzzy recall.
4. Memory extraction instead of transcript storage
Instead of asking:
Save everything.
Ask:
Did anything worth remembering happen?
Example prompt:
Extract durable memories.
Only save information likely useful
weeks or months later.
Ignore greetings,
small talk,
temporary plans.
This dramatically reduces storage.
5. Retrieval budget
A lot of systems cap memory retrieval.
Example:
top 5 semantic memories
top 2 episodic memories
1 user profile
1 conversation summary
That's it.
Never dump 300 memories into the prompt.
6. Hierarchical summaries
Instead of:
100,000 messages
keep:
daily summaries
↓
weekly summaries
↓
monthly summaries
If needed:
October summary
↓
Week 2 summary
↓
Conversation 14
This is similar to how humans recall information.
7. Background memory creation
Instead of slowing every request:
User asks question
↓
Agent answers immediately
↓
Background worker:
- summarize
- detect facts
- embed
- update database
The user never waits for memory creation.
8. Hybrid retrieval
Many projects combine several retrieval methods.
Exact lookup
+
Keyword search
+
Vector search
+
Recency scoring
Score example:
score =
0.55 semantic similarity
+0.25 recency
+0.20 importance
This usually outperforms pure vector search.
9. Importance scoring
Every memory gets a weight.
favorite color
importance = 0.1
new employer
importance = 0.9
married
importance = 1.0
temporary lunch order
importance = 0.02
Low-value memories can expire automatically.
10. Local stack that works well
You don't need cloud infrastructure.
A common local setup:
SQLite
├── user profile
├── facts
├── metadata
└── conversation summaries
Qdrant or Chroma
└── embeddings
Local embedding model
└── bge-small
└── nomic-embed-text
└── jina embeddings
LLM
└── memory extraction
└── summarization
On modern hardware this can comfortably handle tens or hundreds of thousands of memories.
A practical pattern
One design I've seen scale well is to keep three stores:
Recent Context
--------------
Last 20 messages
(always included)
↓
Long-term Facts
---------------
SQLite
↓
Memory Search
-------------
Vector DB
When a new request arrives:
- Load the last ~20 messages.
- Fetch structured facts (name, preferences, projects, etc.).
- Search the vector index for the top 5–10 relevant memories.
- Build the prompt from those pieces.
The prompt size stays roughly constant whether the conversation is one day old or two years old.
Cost comparison
If you send 100,000 tokens of history with every request, your inference cost grows linearly over time. If you instead retrieve 5–10 relevant memories plus a compact user profile, prompt size stays nearly fixed. The ongoing costs shift to occasional embedding and summarization work, which is typically much cheaper than repeatedly paying to process the entire conversation.
For many personal or self-hosted agents, a local stack of SQLite + a lightweight vector database (like Qdrant or Chroma) + a local embedding model is enough to provide durable cross-session memory without significant infrastructure or recurring API costs. The key insight is that long-term memory should be selectively distilled and retrieved, not replayed in full.
