Architecture

This page describes how a Dakera server is built: the request path, storage backends and caches, vector indexes, the knowledge graph, background jobs, durability, cluster mode and encryption. It is for operators sizing and running a deployment and for developers who want to know what happens behind the API.

Dakera is a single Rust binary. It serves a REST API (port 3000) and a gRPC API (port 50051) over the same engine, runs its embedding, reranking and extraction models locally with ONNX Runtime, keeps memories tidy with background jobs (decay and AutoPilot), and can run as a replicated cluster with gossip membership and leader election.

Retrieval pipeline

System overview

What each layer contains

Memory lifecycle

A memory is created with an importance, loses importance while it goes unused, and is deleted only by an explicit forget or when its TTL expires. Decay alone never deletes anything.

Storage and caching

One storage backend holds the data, chosen with DAKERA_STORAGE:

Tiered storage (DAKERA_TIERED_STORAGE=true, with DAKERA_STORAGE=s3) puts a hot tier and a local-disk warm tier in front of S3 and moves records between them by how recently they were accessed. Since v0.12 the hot tier is a durable RocksDB store (DAKERA_HOT_TIER=memory keeps the old in-memory one); its writes are fsynced and survive S3 outages.

Caches sit in front of the store:

• In-process cache (always on): recently read vectors, bounded to 128 MiB.
• Redis (DAKERA_REDIS_URL): a cache shared by every replica, with invalidation between nodes.
• Disk cache (DAKERA_DISK_CACHE_DIR): a RocksDB read cache that survives restarts.

Reads check the in-process cache, then Redis, then the disk cache, then the store, filling the faster tiers on the way back. Writes go to the store first and then update the caches. A vector past its TTL counts as a miss in every cache.

Vector indexes

The engine chooses the index for each namespace; it is not selected per namespace or per request. A namespace at or below DAKERA_ANN_THRESHOLD (default 1000 vectors) is searched by an exact scan; above it the engine builds an HNSW graph.

IVF and SPFresh code is kept in the engine for snapshot decoding and benchmarks but is not routed: nothing selects it, and benchmarks found no size from 10,000 to 50,000 vectors where either beats HNSW. GET /v1/namespaces/{ns} and the admin index endpoints report the live index (flat or hnsw), the vectors indexed and the build time; POST /admin/indexes/rebuild rebuilds it. All indexes use SIMD-accelerated distance functions (cosine, Euclidean, dot product); an omitted distance_metric resolves to the namespace's metric. Opt-in late interaction (colbert-small) adds a fixed-dimensional-encoding first stage and a MaxSim rerank; see Search and ranking.

Knowledge graph

Dakera keeps a graph of memories next to the vector store. Edges are created when a memory is stored (by similarity and, when the namespace extracts entities, by shared entity), or explicitly through the API. There are five edge types. The graph is durable by default and lives under the data root (DAKERA_GRAPH_PATH sets another location; :memory: is the explicit in-memory choice), and edges are scoped by namespace, so one agent's links are never read or deleted by another:

The graph serves multi-hop and entity-centric retrieval and the cross-agent network. Query it with the dk knowledge commands or the knowledge graph routes.

Event bus & SSE

Dakera includes a built-in event bus that streams real-time notifications over Server-Sent Events (SSE). Subscribe to memory lifecycle events — stored, recalled, decayed, forgotten, consolidated, importance_updated and session events. The stream carries every agent's events and needs a global admin key. Useful for building dashboards, audit UIs, and agent coordination pipelines.

# Subscribe to memory events (all agents)
curl -N -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3000/v1/events/stream

AutoPilot

AutoPilot is a background task, on by default (DAKERA_AUTOPILOT_ENABLED=false turns it off), with two jobs:

• Deduplication, every 6 hours (DAKERA_AUTOPILOT_DEDUP_INTERVAL_HOURS), for candidates above a similarity of 0.93 (DAKERA_AUTOPILOT_DEDUP_THRESHOLD). It merges only true duplicates: the same content words in the same order and the same entity tags ("I have two kids" and "I have three kids" are not merged). The absorbed text, type and metadata are kept in _dakera_duplicates, and a permanent memory is never folded into one that expires.
• Consolidation, every 12 hours: clusters of three or more memories that share at least two tags, all with an importance below 0.3, become one semantic memory that records its sources (_dakera_consolidated). A memory with an attachment, a TTL or user metadata is never consumed.

Decay is a separate hourly pass (DAKERA_DECAY_INTERVAL_SECS, default 3600) that follows the configured curve from where the importance started; it no longer compounds. The density-based (DBSCAN) consolidation runs on request, through POST /v1/agents/{id}/consolidate. Trigger an AutoPilot cycle with POST /admin/autopilot/trigger. Passes share one lock (across replicas through Redis); in a cluster, AutoPilot and decay run on the elected leader only.

Retrieval pipeline

A recall goes through eight steps:

gRPC API

The gRPC API on port 50051 runs on the same engine as REST but covers raw vectors only:

# gRPC is on by default; these are the defaults
DAKERA_GRPC_ENABLED=true
DAKERA_GRPC_PORT=50051

# The gRPC port has no TLS of its own; terminate TLS or mTLS at a proxy

v0.12: every RPC except Health needs an API key in the call metadata (x-api-key or authorization: Bearer); calls are rate limited, time-bounded and audited like REST, and requests are validated like REST (top_k must be 1–10,000).

Backup & WAL

The flat storage backends are protected by Dakera's own write-ahead log under {data root}/wal (snapshots in {data root}/wal-snapshots), synced according to DAKERA_WAL_SYNC (everywrite, periodic:N or manual). Write and fsync errors are returned to the caller, concurrent writes share one fdatasync (group commit), and on restart the log is replayed. A checkpoint rotates and truncates the active segment, so deleted content stays in the log for at most one checkpoint interval and in at most 10 snapshots. The RocksDB hot tier of tiered storage fsyncs every write itself (DAKERA_ROCKSDB_SYNC, default true). A replay that dropped entries is reported as wal_replay under degraded in GET /health.

Stored formats are versioned: vector records, index snapshots, RocksDB, the log, the disk and Redis caches and imports carry or check a format version, and a record written by a newer build is refused or skipped and counted, never misread. The server locks its data root (.dakera.lock), so a second server on the same root refuses to start.

For point-in-time backups, use the admin API (see Deployment → Backup & Restore). Backups carry memories, sidecars, attachments and graph edges, can be encrypted (AES-256-GCM per file) and compressed (zstd), and a stored backup schedule now actually runs. A restore with "overwrite": true gives the exact point in time of the backup.

Import & export

An agent's memories move in and out of Dakera through GET /v1/export and POST /v1/import, in jsonl (default), csv, mem0 or zep format. Export carries plaintext content and TTLs; import is a multipart upload whose content is decoded to at most the 100,000-byte content limit and refused per item past it, and import progress is available at GET /v1/import/{job_id}/status. Both need read / write on the agent's namespace.

# Export an agent's memories
curl "http://localhost:3000/v1/export?agent_id=my-agent&format=jsonl" \
  -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  -o memories.jsonl

# Import memories from a file (multipart upload)
curl -X POST "http://localhost:3000/v1/import?agent_id=my-agent&format=jsonl" \
  -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  -F file=@memories.jsonl

Compaction and index maintenance

The segmented backend and the RocksDB stores compact in the background. Two admin endpoints exist for maintenance windows: POST /admin/namespaces/{namespace}/optimize rebuilds the namespace's ANN index and runs storage compaction where the backend supports it (only the segmented backend does; compaction is null otherwise), and POST /ops/compact runs the storage backend's compaction or answers 501.

curl -X POST http://localhost:3000/admin/namespaces/my-ns/optimize \
  -H "Authorization: Bearer $DAKERA_ROOT_API_KEY"
# {"success":true,"job_id":...,"message":...,"index":{"namespace":"my-ns","index_type":"hnsw","rebuilt":true,...},"compaction":null}

Distributed architecture

A Dakera cluster is a set of nodes that each hold a full copy of the data. Any node accepts writes: a write is committed locally, then broadcast to every peer and acknowledged to the client whatever the peers answered; a peer that does not acknowledge within the sync timeout gets the change later from a durable outbox. It is an availability-first (AP) design: both sides of a partition keep accepting writes, and on heal the later write of each id wins, deletes included.

There is no sharding or consistent-hash ring: capacity is that of one node, and a cluster adds availability, not capacity. Each S3-backed node needs its own bucket. See High Availability for cluster setup, failure modes, and operational procedures.

Encryption at rest

All memory content can be encrypted with AES-256-GCM authenticated encryption. Set DAKERA_ENCRYPTION_KEY to enable. Passphrases are stretched with PBKDF2-HMAC-SHA256 (100,000 iterations). Every sealed value is $enc$v2$<key_id>$…, with the namespace, record id and field as authenticated data, so a sealed value copied elsewhere does not decrypt. Data keys live in a replicated keyring wrapped under masters derived from DAKERA_ENCRYPTION_KEY; POST /admin/encryption/rotate-key (optionally with namespace) adds a key and re-seals in the background, and GET /admin/encryption/status reports progress. Old keys are kept, so reads never break mid-rotation and older backups restore. See Security.

Filter expressions

The vector routes take a metadata filter: query, batch query, count, bulk update, bulk delete, aggregate and unified query. Memory recall uses its own fields instead (tags, memory_type, min_importance, session_id, since / until).

{
  "filter": {
    "$and": [
      { "importance": { "$gt": 0.7 } },
      { "tags": { "$in": ["decision", "blocker"] } }
    ]
  }
}

Supported operators: $eq, $ne, $gt, $lt, $gte, $lte, $in, $nin, $and, $or, $exists, $regex, $glob, $contains, $icontains, $startsWith, $endsWith, $arrayContains, $arrayContainsAll, $arrayContainsAny.

Related: Core Concepts · High Availability · Why Dakera is written in Rust · How agent memory works