Projects / Hermes

I Built My Agent a Haystack — Then It Started Remembering

Seraph / Signalane

The Problem

My AI agent forgets everything between sessions. And everyone told me it didn’t.

The hype machine is loud: “your agent remembers everything,” “persistent memory across sessions,” “it learns from you.” It sounds convincing until you look under the hood. The original memory files — MEMORY.md and USER.md — were 2,200 and 1,350 characters. Tiny. Barely enough for a name and a preference or two. The state.db held session logs you’d have to manually search through. Functionally, it was nothing.

But it looked like something. The files existed. The system claimed persistence. The illusion was convincing enough to feed the hype cycle.

It’s nice for a while. Then it punches you in the face.

I hit that wall hard. The agent couldn’t recall decisions we’d made days earlier. Conversations we’d invested hours in — gone. The “memory” was a thin veneer over an amnesiac core, and I refused to stay in that illusion.

That’s what drove the whole stack: mem0 for semantic search, Qdrant for vector storage, Brainpal as a second source of truth, and finally kredenc — the system this article is about. Each layer tried to solve the same problem from a different angle. kredenc was the one that actually worked.

The truth always surfaces. You can bury it under hype for a while, but eventually the agent tries to remember something important and can’t.

The Architecture Decision

The haystack isn’t a database. It’s a log.

Here’s the design: every memory — every turn, every compaction transcript, every fact worth keeping — gets written to an append-only binary file. Each record is exactly four parts:

1. Magic bytes — KRD1 (4 bytes) so you can scan the file and find record boundaries instantly
2. Length — 4-byte integer telling you exactly how many bytes follow
3. JSON payload — self-describing: id, drawer, timestamp, tags, and the full memory text
4. Nothing else — no indexes, no B-trees, no page splits

That’s it. You open the file, read the magic, read the length, read the payload, repeat. Sequential I/O. The fastest thing a disk can do.

The SQLite database sits beside it as an accelerator — it holds the FTS5 full-text index, the drawer metadata, and the (offset, length) pointers back into the haystack. If the database corrupts, you can rebuild it by walking the haystack. The haystack is the source of truth. The database is the index.

That’s backwards from how everyone builds these systems. Usually the database is the truth and the log is the backup. Here, the log is the truth and the database is disposable.

Why This Matters

SQLite is brilliant at structured queries. It’s mediocre at storing ever-growing blobs of text. Every INSERT bloats pages. Every VACUUM rewrites the whole file. Backups are fragile because the file is alive — pages are being split, moved, rewritten while you copy it.

An append-only log has none of those problems:

• Offsets never move. Once a record is written at byte 4,821, it stays at byte 4,821 forever.
• Crashes are harmless. If the power dies mid-write, you might have a few orphan bytes at the tail. The database never points at them, so they’re invisible. Every committed record stays valid.
• Backups are trivial. Copy the file. Done. It’s immutable — there’s nothing to “quiesce.”

Git objects, Kafka, LSM-trees, even SQLite’s own WAL — they’re all append-only logs underneath. This isn’t a clever hack. It’s the canonical pattern for immutable data, and it’s been proven at scale for decades.

What Went Wrong

The first version had a thread-affinity bug that silenced the entire memory system for days.

SQLite objects created in one thread can only be used in that same thread. The housekeeper — the background process that files every conversation turn into the haystack — was created in the main thread but accessed from a worker thread. It failed silently. No errors in the logs, no crash. Just… nothing getting stored.

I didn’t notice for days because the system looked healthy from the outside. The database existed. The plugin loaded. The hooks fired. But the actual writing was dying without a sound.

The fix was straightforward — create the SQLite connection in the thread that uses it — but the lesson is expensive: when your memory system fails silently, you don’t know you’re amnesiac until you try to remember something important and can’t.

What This Actually Does

Right now, the haystack files two things automatically:

Every completed turn. User asks something, I answer, the exchange gets filed with a content hash for deduplication. If the same turn somehow gets passed through the filing door twice, the hash catches it and skips the duplicate.

Every pre-compaction transcript. Before the context window gets compressed (and the full conversation is summarized away), the complete verbatim transcript is archived first. The summary carries a pointer back to the shard ID, so if I ever need the original — not the summary, but what actually happened — I can fetch it.

The beautiful part: I can search the haystack in milliseconds. Full-text search through every conversation turn I’ve ever had. Not “similar conversations” — exact matches. The decision we made on August 3rd about compression thresholds? It’s there. Word for word.

The Gap (and What Comes Next)

The haystack writes perfectly. Reading is the weak link.

Right now, he has to consciously decide to search it. It’s like having a library where you can find any book instantly, but only if you remember the library exists. The next phase wires automatic retrieval into the cognitive loop — before every API call to the model, relevant memories get fetched and injected. Not because I remembered to look, but because the system fires reflexively, like a nervous system should.

That’s the difference between a tool you use and a system that works.

The Lesson

If you’re building memory for an agent, start with the log. Get the writing path bulletproof first. Everything else — search, retrieval, injection — depends on the data actually being there.

A vector database with nothing in it is just a very expensive empty room. An append-only log with everything in it is a foundation you can build on.

Build the haystack before you build the needle.

Technical context: Hermes Agent runtime, kredenc memory store (SQLite FTS5 + append-only binary haystack), local inference on Qwen3.6-27B via llama.cpp. Windows environment. All processing local — no cloud services.