High availability

This page is for operators who run Dakera v0.12.3 on more than one node. It explains how a cluster replicates and elects a leader, the settings it needs, how to add, remove and upgrade nodes, and what to monitor.

A Dakera cluster uses gossip-based membership (SWIM over UDP), lease-based leader election and versioned replication: every node accepts writes and holds a full copy of the data. Nodes find each other through seed addresses; the elected leader runs the background jobs that rewrite stored data. Each node uses its own storage (on S3, its own bucket).

Cluster environment variables

How the cluster works in v0.12

Any node accepts writes. There is no leader-only write path. A write is committed on the node that took it, then sent to every peer's /internal/sync/* route and awaited (5 s per peer). The client is answered 200 whatever the peers said. A peer that did not acknowledge gets the change later from the node's durable outbox. Every node holds a full copy of the data; the design favors availability (both sides of a network partition keep accepting writes and converge when it heals).

Versions and tombstones

• Every replicated record carries a hybrid logical clock version (_dakera_hlc, never returned to clients; client metadata may not use that key, 400).
• Every delete, namespace delete, revoked API key and removed quota leaves a tombstone, kept for DAKERA_CLUSTER_TOMBSTONE_RETENTION_SECS (default 604800, 7 days). Set it above the longest outage of a node you expect to bring back without wiping it.
• A change is applied only if its version is newer than the stored record and any tombstone: last writer wins (highest version; on equal time the counter, then the node id). A write acknowledged to a client is never deleted by a node that never had it.
• Knowledge-graph edges are not versioned and converge to the leader's copy.

Clock drift

A change whose version is more than DAKERA_CLUSTER_MAX_CLOCK_DRIFT_MS (default 60000) ahead of the receiver's clock is refused and retried. A node whose own clock is that far ahead of more than half its peers refuses local writes (503) and reports degraded with clock_ahead_ms in /health until the clocks agree. Keep NTP running on every node.

Reconciliation

A (re)starting node merges completely with a live peer (the leader first) before it answers digests or stands for election; it waits up to two leases for a peer. Every 5 minutes each follower merges with the elected leader in both directions, by version, over every namespace either side holds, including API keys, quotas, namespace configs, sidecars and attachments, and the leader merges with every live follower. Per-namespace digests are cached, so an unchanged cluster costs nothing to compare. Each node persists since when its data is complete (data_since): a node whose data starts later than a record's can never make others delete it.

Leader election

• Lease-based with terms and one vote per term. The quorum is a majority of the whole membership (every member ever seen minus graceful leaves), and a lease is extended only by heartbeats a quorum acknowledged, so a minority partition stops leading within one lease.
• A candidate that has not finished its startup merge, or whose data starts later than the voter's, is refused.
• The leader runs the background jobs that rewrite stored data: decay, auto-pilot, consolidation and the re-embed jobs. TTL expiry runs on every node.
• DAKERA_ELECTION_AUTO_ELECT (default true) is a topology choice: set it to false on nodes that must never stand for election.

One timing governs failure detection, DAKERA_ELECTION_LEASE_MS (default 15000, accepted range 1000 to 600000; a value outside it is an invalid value). Every other timing derives from it:

A longer lease suits a WAN or a loaded network; a shorter one detects failures faster. Reconciliation runs every 5 minutes regardless.

Durable outbox

Broadcasts a peer has not acknowledged are written to DAKERA_CLUSTER_OUTBOX_DIR (default {data root}/cluster-outbox) before they count as queued, and replayed in order at startup. The outbox holds at most 64 MiB; the oldest entries are evicted when it is full and are then repaired by reconciliation. A change a peer refuses for good (a 4xx other than 408, 429, 401 and 403) is dropped, logged and repaired by reconciliation instead of blocking that peer. A write whose client went away after it was committed is still replicated.

Authentication

Cluster mode requires DAKERA_CLUSTER_SECRET (at least 16 characters, identical on every node) on every node, fresh install or upgraded (v0.12.1), and a stable DAKERA_CLUSTER_NODE_ID on a fresh install. A node without the secret exits with code 78 before it serves anything; dakera --check-config reports a missing or short secret. On v0.12.0 an upgraded node could start without it, which left POST /internal/sync/key/add open: anyone who reached the API port could mint a SuperAdmin key. An upgraded node without a node id generates one, keeps it in {data root}/cluster-node-id, and is degraded in /health (cluster_auth) until it is set. /internal/* requires the secret and answers 404 on a node that is not in cluster mode; gossip packets carry an HMAC-SHA256 tag with a send time and nonce, and replays are dropped. The secret travels in every peer request and traffic is not encrypted: use a private network or TLS between nodes. See Security.

One S3 bucket per node

Cluster nodes must not share one S3 bucket as their store. Every S3 node keeps a heartbeat marker (_dakera_cluster/nodes/<node id>.json, refreshed every 30 s). A node that finds another node's live marker (younger than 120 s) in its bucket refuses to boot on a bucket that began its life on v0.12 (exit 78) and starts with a topology warning and shared_store degraded on a bucket that already held data. Backups are written to the backup store of the node that took them (node_id in GET /admin/backups); a restore or download through another node answers 409 naming the owner, unless the nodes share one backup bucket.

Observing a cluster

GET /admin/cluster/status (global admin) reports cluster_id, state (healthy, degraded, initializing, maintenance), node_count, total_vectors, namespace_count, redis_healthy, the node's election view (state, current_term, leader_id, fencing_token, lease_remaining_ms, member_count, alive_members, quorum) and, in cluster mode, replication:

Compare the vector counts and replication across nodes to see whether the cluster has converged. Metrics: dakera_cluster_sync_outbox_entries, dakera_cluster_sync_failures_total, dakera_cluster_sync_outbox_dropped_total, dakera_cluster_sync_outbox_persist_failures_total, dakera_cluster_sync_refused_total, dakera_cluster_sync_superseded_total, dakera_cluster_sync_held_entries, dakera_cluster_replication_queue_dropped_total, dakera_cluster_reconcile_total, dakera_cluster_reconcile_namespaces_total, dakera_cluster_reconcile_pushed_namespaces_total, dakera_cluster_tombstones, dakera_cluster_tombstones_collected_total, dakera_cluster_clock_ahead_ms and dakera_cluster_clock_skew_refused_writes_total. /health degraded components that concern clusters: cluster_gossip, cluster_outbox, cluster_auth, cluster_sync_held and shared_store. The v0.12 alert rules cover held changes, dropped outbox entries and clock skew (Operations).

Maintenance mode

POST /admin/cluster/maintenance/enable with {"reason": "...", "node_ids": [...], "reject_requests": true, "duration_minutes": 30} puts the node that receives the request into maintenance mode. It is held in memory by that node alone (not replicated, ended by a restart); send the request to each node that should enter it. With reject_requests every request outside /admin, /v1/admin, /ops, /health, /metrics and /internal is answered 503 with Retry-After; gRPC calls other than Health fail UNAVAILABLE. POST /admin/cluster/maintenance/disable is 409 while maintenance jobs run unless force is set. GET /admin/cluster/maintenance shows the state.

Operating limits

• A node that is lost for good (never left gracefully) keeps counting toward the quorum, and there is no admin endpoint that removes a member. Stop nodes with SIGTERM so they announce a graceful leave.
• Run an odd number of nodes (3 or 5): a majority of 3 is 2, of 5 is 3.
• A cluster is a replicated topology: every node stores a full copy.

Scaling procedures

Adding a node

# 1. Create the node's own S3 bucket, then start the node with seeds pointing at the cluster
docker run -d \
  --name dakera-4 \
  -e DAKERA_CLUSTER_MODE=true \
  -e DAKERA_CLUSTER_NODE_ID=node-4 \
  -e DAKERA_CLUSTER_SECRET=$DAKERA_CLUSTER_SECRET \
  -e DAKERA_CLUSTER_SEED_NODES=dakera-1:7946,dakera-2:7946 \
  -e DAKERA_API_ADVERTISE=http://dakera-4:3000 \
  -e DAKERA_S3_BUCKET=dakera-4 \
  -e DAKERA_ROOT_API_KEY=$DAKERA_ROOT_API_KEY \
  -p 3303:3000 -p 7949:7946/udp \
  ghcr.io/dakera-ai/dakera:0.12.3

# 2. Verify the node joined the cluster
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3000/admin/cluster/nodes

# 3. The new node merges with the leader before it answers digests or stands for election
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3000/admin/cluster/status  # replication.startup_reconciled: true

Removing a node

# Stop it gracefully (SIGTERM) so it announces a leave: a node that is lost for good
# keeps counting toward the quorum, and no admin endpoint removes a member.
docker stop dakera-4 && docker rm dakera-4

Rolling upgrade procedure

Upgrade cluster nodes one at a time, so the cluster keeps serving throughout. A cluster that already runs v0.12.0 with DAKERA_CLUSTER_SECRET set upgrades to v0.12.1 this way, keeping the same secret. A v0.12.0 cluster without the secret, or a v0.11.108 cluster, has extra rules:

• Set DAKERA_CLUSTER_SECRET on every node and upgrade them in one window (v0.12.1). A cluster node without it refuses to start, and nodes with and without it do not see each other (v0.11 nodes send none). Generate one value (openssl rand -hex 32) and set it on every node; run dakera --check-config first.
  • Recommended — one window: stop every node, set the same secret on each, start them all on v0.12.1 or later. The cluster is unavailable for the restart only; no data is lost.
  • Rolling: upgrade one node at a time, each started on v0.12.1 or later with the secret. Until the last node is done the cluster is two groups that do not see each other: each elects its own leader, replicates only within itself and keeps serving from its own data, so a client may not see a write made in the other group. Once every node runs v0.12.1 or later with the secret they form one membership, elect one leader and reconcile every node's data by version. Keep this window to minutes.
• Rotating the secret needs the same window: nodes on the old and new values do not see each other until all have the new one.
• An upgraded node without DAKERA_CLUSTER_NODE_ID generates a stable id (cluster_auth degraded); set one on every node.
• Changes a v0.11 node has no route for (namespace and runtime configuration, attachments, record writes) wait in the sending node's durable outbox until that node runs v0.12.
• In an encrypted cluster (DAKERA_ENCRYPTION_KEY) v0.12 nodes hold changes carrying $enc$v2$ text for v0.11 peers (cluster_sync_held in /health) and deliver them once the peer is upgraded; clients reading from a v0.11 node do not see them until then.
• Give each node its own S3 bucket.
• To go back, wait until every node's outbox is empty, stop every node and roll back the whole cluster: Rolling back to v0.11. v0.11.108 ignores DAKERA_CLUSTER_SECRET.

Full procedure: Upgrading to v0.12.

Docker Compose cluster

The dakera-deploy repository has a three-node Compose file (docker/docker-compose.ha.yml, with Traefik, Redis and MinIO). It was written for v0.11: its nodes share one MinIO bucket, set no DAKERA_CLUSTER_SECRET, and set variables v0.12 does not read. A v0.12.1 node refuses each of these (the shared bucket and unknown names on a fresh install; the missing secret on every node). Before you start it with v0.12.1:

• give each node its own bucket (DAKERA_S3_BUCKET) and create the buckets;
• set the same DAKERA_CLUSTER_SECRET (at least 16 characters) and a distinct DAKERA_CLUSTER_NODE_ID on every node;
• run dakera --check-config with each node's environment and remove the names it reports as unknown.

# After adapting the file as above
git clone https://github.com/dakera-ai/dakera-deploy
cd dakera-deploy/docker
cp .env.ha.example .env.ha  # set your keys and secrets
docker compose -f docker-compose.ha.yml --env-file .env.ha up -d

# Traefik publishes the API on port 3100
curl -H "Authorization: Bearer $DAKERA_ROOT_API_KEY" \
  http://localhost:3100/admin/cluster/status

Kubernetes

The published Helm chart (dakera-helm) does not configure cluster mode: dakera.replicaCount scales one Deployment whose replicas share the same storage settings, and it sets no gossip seeds, cluster secret or per-node bucket. For a v0.12.1 cluster on Kubernetes, run one workload per node with its own DAKERA_CLUSTER_NODE_ID, bucket and stable address, the same DAKERA_CLUSTER_SECRET (from a Kubernetes Secret), the settings in the table above, and probes on /health/live and /health/ready.

Monitoring HA health

Key signals in a clustered deployment (all emitted by the server):

Gossip ports

The defaults suit most deployments. Change them for a port conflict or a multi-NIC host:

Disaster recovery

For cross-region disaster recovery:

• Backup to remote S3 — schedule daily backups to an S3 bucket in a different region via the admin API (see Deployment → Backup & Restore).
• Full cluster restore — provision new nodes, then restore a stored backup with POST /admin/backups/restore or upload a downloaded bundle with POST /admin/backups/upload (S3 storage only; global super_admin) on one node. The restored writes replicate to the other nodes.
• Recovery point — whatever was written after the last backup is lost. For a smaller window, back up more often or replicate the node buckets to another region.

Admin endpoints

Production checklist

• Odd replica counts — run 3 or 5 nodes for clean quorum math
• One S3 bucket per node — nodes must not share a bucket as their store
• DAKERA_CLUSTER_SECRET and a stable DAKERA_CLUSTER_NODE_ID on every node, on a private network or behind TLS — a node without the secret does not start (v0.12.1)
• NTP on every node — a node whose clock runs ahead refuses writes
• Stop nodes gracefully — a node lost for good keeps counting toward the quorum
• External load balancer — place outside the Dakera node pool
• Monitor all nodes — scrape /metrics on every node; alert on election.alive_members, the outbox (dakera_cluster_sync_outbox_entries) and clock skew
• Firewall gossip port 7946 — restrict to cluster nodes only (UDP), and the API port to the load balancer and peers
• Rolling upgrades — one node at a time, waiting for /health/ready and an empty outbox
• Cross-region backups — replicate to a separate S3 bucket for disaster recovery
• Test failover — periodically stop a node and verify automatic recovery