Memory Architecture for Long-Running Agents

Most agent frameworks treat memory as a solved problem: embed everything, retrieve the top-k chunks, stuff them into the prompt. This works fine for a chatbot that lives for one session. It falls apart completely for a personal agent that runs for months, accumulates thousands of memories, and needs to distinguish between "my dentist appointment is on Thursday" (important, time-sensitive) and "the user once mentioned they like Italian food" (useful background, low urgency).

After running humux as my own daily agent for over a year, I've settled on a four-tier memory architecture that handles retrieval, ranking, forgetting, and deduplication. Here's how it works and why each tier exists.

Why flat RAG breaks down

Naive retrieval-augmented generation has three failure modes at scale. First, the store grows without bound. After six months of daily use, you have tens of thousands of memories. Embedding search returns results, but relevance degrades because the embedding space gets crowded—semantically similar but factually contradictory memories compete for the same top-k slots. Second, there's no notion of importance. "User's mother's birthday is March 15" and "user asked about the weather once" get equal standing. Third, contradictions accumulate. The user's phone number changed, they moved cities, their project priorities shifted—but old memories still float around, and the retriever has no way to prefer the fresh one.

You need retrieval and ranking and garbage collection and deduplication. Four concerns, four tiers.

Tier 1: Lexical retrieval

Before reaching for embeddings, try BM25-style word overlap. It's fast, deterministic, and surprisingly effective for proper nouns, dates, and specific terms that embedding models often muddle. When someone asks "what's Marco's email?", lexical search on "Marco email" nails it. Embedding search might return memories about email in general.

The implementation is straightforward: SQLite FTS5 (full-text search) over the memory text column. No external service, no model inference, sub-millisecond. humux runs lexical and semantic retrieval in parallel and merges the candidate sets before scoring.

Tier 2: Semantic retrieval with composite scoring

For queries where exact words don't match—"what does the user do for work?" when the memory says "senior engineer at Acme Corp"—you need embeddings. humux uses fastembed to compute embeddings locally (no API calls, no data leaving the machine). The model runs on CPU, takes about 30MB of RAM, and embeds a sentence in under 10ms.

But cosine similarity alone isn't enough. Inspired by the Generative Agents paper, each memory carries metadata that feeds a composite score:

import math
from datetime import datetime, timezone

ALPHA = 1.0   # relevance weight
BETA  = 1.0   # importance weight
GAMMA = 1.0   # recency weight
DECAY = 0.995 # per-hour decay factor

def score_memory(
    relevance: float,   # cosine similarity or BM25 score, normalized 0-1
    importance: int,     # 1-10, set at extraction time
    last_accessed: datetime,
) -> float:
    now = datetime.now(timezone.utc)
    hours = (now - last_accessed).total_seconds() / 3600
    recency = math.pow(DECAY, hours)
    return ALPHA * relevance + BETA * (importance / 10) + GAMMA * recency

Three components, each pulling in a different direction. Relevance is the similarity to the current query. Importance is a 1–10 score assigned when the memory is created—a birthday rates 9, a casual mention of a restaurant rates 3. Recency decays exponentially: a memory accessed an hour ago gets a near-1.0 boost, one untouched for a month gets almost nothing. The decay factor of 0.995 means a memory loses half its recency score after about 139 hours (~6 days).

When a memory is retrieved and used in a response, its last_accessed timestamp updates. Important memories that keep getting used stay hot. Unimportant ones fade naturally.

The extraction pipeline

Memories don't appear from nowhere. After each conversation turn, the orchestrator asks the model to reflect on the exchange and emit structured operations:

REFLECT_PROMPT = """Given this conversation, extract memory operations.
Each operation is one of:
  ADD    — new fact worth remembering (set importance 1-10)
  UPDATE — corrects or refines an existing memory (reference by ID)
  DELETE — a memory is now known to be false or outdated
  NOOP   — nothing worth remembering

Return JSON: [{"op": "ADD", "text": "...", "importance": 7}, ...]
Only extract durable facts. Skip transient chitchat."""

The model sees the last few turns plus the candidate memories that were retrieved for context, which lets it issue UPDATE operations that reference existing memory IDs. This is how contradictions get resolved: the user says "I moved to Berlin", the model sees the old memory "user lives in Milan", and emits an UPDATE replacing it.

Tier 3: Forgetting

Even with good extraction, the store grows. Tier 3 is a background job that archives cold memories. The rule is simple: memories below an importance threshold whose recency score has decayed past a cutoff get moved to an archive table. They're not deleted—they can be resurrected if a query specifically matches—but they're excluded from the default retrieval pool.

In practice, this means a memory with importance 2 that hasn't been accessed in three weeks gets archived. A memory with importance 9 survives much longer. The threshold is tunable, but the defaults work well: most agents stabilize at a few hundred active memories after the first month, even with heavy daily use.

Tier 4: Hygiene

The extraction model isn't perfect. It will sometimes create a new memory that's a near-duplicate of an existing one. "User works at Acme" and "User is a senior engineer at Acme Corp" are semantically close enough (cosine similarity above 0.92) to be suspicious. The hygiene pass runs periodically, clusters memories by embedding similarity, and hands clusters to the model with a simple prompt: "These memories look similar. Merge them into one, keeping the most complete version, or mark as distinct if they're actually different."

This also catches phantom memories—ones that were extracted from the model's own output rather than from user input. A common failure mode is the model "remembering" a tool name that doesn't exist because it hallucinated one during a previous turn. The hygiene pass gives the model a second look at these in isolation, where they're easier to spot and delete.

The storage layer

All of this lives in SQLite. The schema is minimal:

CREATE TABLE memories (
    id          TEXT PRIMARY KEY,
    agent_slug  TEXT NOT NULL,
    scope       TEXT NOT NULL DEFAULT 'global',  -- or chat-specific
    text        TEXT NOT NULL,
    embedding   BLOB,          -- f32 vector, stored raw
    importance  INTEGER NOT NULL DEFAULT 5,
    created_at  TEXT NOT NULL,
    accessed_at TEXT NOT NULL,
    archived    INTEGER NOT NULL DEFAULT 0
);

CREATE VIRTUAL TABLE memories_fts USING fts5(text, content=memories);

The scope column is important: some memories are global to an agent ("user's timezone is CET"), while others are specific to a chat context ("in this group, refer to the project as Phoenix"). The retriever filters by scope before scoring, so chat-specific memories don't leak into unrelated conversations.

Embeddings are stored as raw float32 blobs. Cosine similarity is computed in Python with a simple dot product—numpy isn't even needed for the store sizes a personal agent deals with. At 10,000 memories with 384-dimensional embeddings, a brute-force scan takes about 2ms. A vector database would be overkill here.

Putting it together

The four tiers work in concert. On every turn: T1 and T2 retrieve candidates, composite scoring ranks them, the top memories enter the prompt context. After the response, the extraction pipeline produces ADD/UPDATE/DELETE operations. In the background, T3 periodically archives cold memories and T4 deduplicates the store. The net effect is a memory system that grows in quality over time rather than just in size. After a few months of use, the active memory set is dense with high-value facts and free of contradictions—which is exactly what you want for a personal agent that's supposed to know you.