Troubleshooting

This page is for anyone running a Dakera v0.12.0 server who hits an error, a refused start or unexpected results. Each section names the symptom, the likely cause and the fix. Start with the health endpoint:

curl http://localhost:3000/health
# {"service":"dakera","status":"healthy","version":"0.12.3","unreadable_records":0,"corrupt_records":0,"degraded":[],"config_warnings":[]}

/health lists degraded components and config_warnings, and embed_migration progress while the post-upgrade re-embed runs. Use /health/live for liveness and /health/ready for readiness. Every field is described in Operations.

Server not starting

Check the container logs first: docker logs dakera --tail 50. Common causes:

• Port conflict: change DAKERA_PORT and the -p mapping.
• Volume permission error: the image runs as UID 1000; make the data volume (/data) writable by it.
• No API key: with authentication on (the default) and no key configured, the server exits. Set DAKERA_ROOT_API_KEY, or set DAKERA_AUTH_ENABLED=false for local development only.
• Exit code 78 with a list of variables on stderr: a fresh install refuses a variable with an unusable value or an unknown DAKERA_* name (the message suggests the closest real name). Fix it, run dakera --check-config to see every finding, or start once with DAKERA_CONFIG_LENIENT=1. A deployment that already holds data starts instead and lists the findings under config_warnings in /health.
• "locked by a running Dakera process": two servers on one data root. A server locks its data root (.dakera.lock); give each server its own.
• Refuses to start after changing DAKERA_MODEL: the store records the model that embedded it. From v0.12.1 a changed model no longer refuses the start: the store is re-embedded in the background, one agent namespace at a time, each answered with its own model until its turn (progress in /health embedding_model_change). Startup is still refused where the change cannot run in the background: DAKERA_TIERED=1, late-interaction scoring, the visual lane, or a recorded model this build does not know. There, start on a fresh store, or acknowledge with DAKERA_ALLOW_MODEL_CHANGE=1 and re-embed with POST /admin/namespaces/migrate-dimensions (on v0.12.0 this was the only way).
• Cluster node refuses to start (exit 78): every cluster node needs DAKERA_CLUSTER_SECRET (at least 16 characters, the same on every node), fresh install or upgraded (v0.12.1; on v0.12.0 an upgraded node started without it). A fresh install also needs DAKERA_CLUSTER_NODE_ID and refuses an S3 bucket another node already uses. dakera --check-config with the node's environment reports a missing or short secret (High availability).

503 while the server starts

The port answers while models download or load. /health/live returns 200. /health/ready returns 503 with "starting": true, a reason (what is loading) and downloads (per-file progress); /health returns 503 with "status": "starting", reason and downloads. Both carry Retry-After: 5, and every other request gets 503 with Retry-After: 5. Wait for /health/ready. A Docker or Kubernetes liveness probe pointed at /health should move to /health/live, with a start period long enough for a model download.

403, 413, 400 and gRPC errors after upgrading

• 403 on /admin/*, /ops/* or analytics: a key restricted to namespaces no longer reaches node-wide routes. Use a key with the scope and no namespace restriction. Backup download, upload and restore need a global super_admin key (Security).
• 413 QUOTA_EXCEEDED: namespace quotas are enforced in v0.12.0 (v0.11.108 ignored them). Raise the quota, or set its enforcement to soft or none, with PUT /admin/quotas/{namespace} (the default for all namespaces: PUT /admin/quotas/default).
• gRPC UNAUTHENTICATED: gRPC needs an API key in the call metadata (x-api-key or authorization: Bearer). Query also needs top_k between 1 and 10,000.
• smart_score values are lower: the score changed scale (no access-count term, recency counted from creation). Re-check any absolute threshold, including DAKERA_ABSTAIN_MIN_SMART_SCORE.
• A scheduled backup starts running: a stored, enabled schedule now runs, and /health config_warnings shows a backup_schedule entry. Disable it with POST /admin/backups/schedule and {"enabled": false}.
• Memory counts dropped and session counts rose (v0.12.1): totals count memories, not stored records — sentence sub-memories are no longer counted as memories — and session totals count ended sessions too. No data changed; storage, recall and quotas are unaffected. GET /v1/agents vector_count holds the figure memory_count used to report. Re-base dashboard and alert thresholds on the new figures.
• Sessions end by themselves (v0.12.2): the server ends a session idle for DAKERA_SESSION_IDLE_TIMEOUT_SECS (default 4 h) and records ended_reason: "idle"; sessions left open longer than that are closed on the first passes after the upgrade. Set the variable (or PUT /admin/config session_idle_timeout_secs) to 0, start sessions with idle_timeout_secs: 0, or keep a quiet session open with POST /v1/sessions/{id}/touch.
• 400 on writes that used to work (v0.12.2): validation is stricter on every memory write path. The message names the field and the rule; common causes are a client-supplied dakera-curated tag or _dakera_* metadata key, a caller id of the form mem_s + 24 hex, content over DAKERA_MAX_MEMORY_CONTENT_BYTES (bytes, not characters, now on update too), an agent id over 241 bytes, or a memory record written through the raw vector routes of an agent namespace (use /v1/memory/*). Invalid key namespace entries are refused too.
• 403 ending a session (v0.12.2): POST /v1/sessions/{id}/end checks Write scope first, so a Read key is refused for any id.
• A key with a foo* entry reaches nothing (v0.12.2): a key created before v0.12.2 keeps reading it as a literal name. Save its namespaces again with PATCH /admin/keys/{key_id} to turn it into a pattern; GET /v1/auth/whoami lists such entries in inert_namespaces.
• Agent memory listings got shorter (v0.12.2): GET /v1/agents/{id}/memories and /wake-up leave sentence sub-memories out; add include_derived=true to see them.
• The container health check fails after upgrading (v0.12.3): the images no longer contain curl. Use ["CMD", "dakera", "healthcheck", "/health/ready"]; checks from the host and Kubernetes httpGet probes are not affected.
• 421 HOST_NOT_ALLOWED (v0.12.3): authentication is off and the request named a dotted host. Add the name to DAKERA_ALLOWED_HOSTS (comma-separated, * = any), or turn authentication on.
• 422 creating a key or a backup (v0.12.3): these admin requests refuse fields they do not know; the message names the field. Common mistakes are namespace for namespaces and encrypted for encrypt.
• 429 with Retry-After under load (v0.12.3): the server is busy, not broken; retry after the given seconds (the SDKs do). A bulk client that queues long requests can raise DAKERA_ADMISSION_WAIT_SECS together with DAKERA_REQUEST_TIMEOUT.
• Duplicates are no longer merged (v0.12.3): AutoPilot deduplication is off by default. Set DAKERA_AUTOPILOT_DEDUP_MODE=merge, or report to see the duplicate groups first. A memory removed by mistake can be restored for 7 days with POST /v1/memory/deduplicated/{id}/restore.
• 403 on POST /ops/shutdown (v0.12.3): it needs a global super_admin key.

Error codes and states

Every error body is JSON: {"error", "code", "status"}, plus details where there is more to say and resource on a 404.

• 501 FEATURE_DISABLED: an opt-in route (attachments, transcription, image indexing, records) was called while its switch is off; details names the variable (DAKERA_ATTACHMENTS, DAKERA_VISION, DAKERA_RECORDS). Set it and restart. See Multimodal memory.
• 501 NOT_IMPLEMENTED: a configuration the server cannot serve, which no retry changes: late-interaction scoring on a model without a late-interaction recipe or with DAKERA_TIERED=1, or an invalid DAKERA_EXTRACTOR_* setting. details names the settings to change.
• 503 with Retry-After: every 503 carries the header. Causes: the server is starting; recall admission is full (at most two recalls per core, and at least four, run at once; the rest queue for up to 20 s); a media job or entity-extraction pass does not fit the memory budget (it waits up to 10 s); maintenance mode. Retry after the interval. For memory-budget refusals, raise the container memory limit or lower the load (dakera_errors_total{type="memory_budget"}).
• 413 PAYLOAD_TOO_LARGE: a body over DAKERA_MAX_BODY_SIZE (default 10 MiB), or an attachment over DAKERA_ATTACHMENT_MAX_BYTES (default 25 MiB). Attachment uploads and import files over the limit were 400 in v0.11.108.
• 404 JOB_NOT_FOUND on a transcription or image-index job: job ids are kept in memory, so after a restart the job is unknown. The memory a completed job stored is kept (use the memory_id the 202 returned); an unfinished job must be started again.
• Rejections before a handler runs use the same JSON body: ROUTE_NOT_FOUND (404), METHOD_NOT_ALLOWED (405), UNSUPPORTED_MEDIA_TYPE (415), REQUEST_TIMEOUT (408, after DAKERA_REQUEST_TIMEOUT, default 300 s).
• 504 QUERY_TIMEOUT: a query, recall or search ran past query_timeout_ms (set with PUT /admin/config; 0, the default, means no deadline).
• 403 CROSS_ORIGIN_REQUEST_REFUSED: with DAKERA_AUTH_ENABLED=false, a state-changing request from a browser page whose origin is neither loopback nor listed in DAKERA_CORS_ORIGINS is refused. SDKs and curl (no Origin header) are unaffected.
• 400 on a namespace PATCH: PATCH /v1/namespaces/{ns}/config merges and refuses unknown fields; PUT replaces the whole configuration.

Reading /health

• config_warnings is not empty: the server started with a setting it could not honor (an unknown DAKERA_* name, a value of the wrong shape, v0.11.108's reading of a boolean kept on an upgraded deployment, data at a pre-0.12 location, a stored backup schedule). It is informational and never degrades the node. dakera --check-config prints the same findings and exits 0 (would start) or 78 (would refuse) without starting anything.
• degraded lists a component: status is degraded and each entry has a component and a reason (reranker, redis_cache, disk_cache, backups, cluster_gossip, cluster_outbox, wal_replay, speech_to_text, vision_model, entity_extraction and others). The reason names the cause; the server log has it in full. The matching gauge is dakera_component_degraded{component}.
• cluster_auth degraded: an upgraded cluster node runs without DAKERA_CLUSTER_NODE_ID, on an id it generated once and keeps in {data root}/cluster-node-id. Set a stable node id and restart the node. From v0.12.1 a node without DAKERA_CLUSTER_SECRET does not start at all (on v0.12.0 it ran unauthenticated and was listed here).
• unreadable_records or corrupt_records above 0: the node skipped records written by a newer Dakera (upgrade the node) or records that do not decode (see dakera_storage_corrupt_records_total). advice says which.
• embed_migration present: stored memories are re-embedded once in the background after an upgrade (about 35 minutes per 10,000 memories on bge-large). Recall is served throughout. Progress: GET /admin/reembed/migration.
• speech_to_text or vision_model degraded: the model's first-load conversion did not fit the memory budget. Convert it outside the server with dakera models pull whisper or dakera models pull vision, or raise the memory limit (Models and Docker).

Authentication errors (401)

Authenticated requests need Authorization: Bearer <key> or X-API-Key: <key>. The authentication layer answers {"error": "authentication_error", "message", "code", "status"}, and code says why: AUTHENTICATION_REQUIRED (no key), INVALID_API_KEY or API_KEY_EXPIRED. A 403 is INSUFFICIENT_SCOPE (with details such as required: admin, actual: read) or NAMESPACE_ACCESS_DENIED.

No memories returned from recall

• Wrong agent_id: recall is scoped to one agent. Check that it matches the one used to store.
• Nothing stored yet: list the agent's memories with GET /v1/agents/{agent_id}/memories.
• min_importance too high: lower or remove the filter.
• A score threshold calibrated on v0.11: smart_score has a new scale in v0.12.0; re-check client-side cut-offs and DAKERA_ABSTAIN_MIN_SMART_SCORE.

Keyword search or short recalls return nothing after upgrading

After upgrading a deployment with data from v0.11.108 to v0.12.0, POST /v1/namespaces/<ns>/fulltext/search returns no hits and short keyword-style recalls return no memories, although the memories are stored. The full-text indexes built by v0.11 without stemming were kept as built, so a query word that differs from a stored one only by inflection (release / released) finds nothing.

From v0.12.1 there is no step. At startup every full-text index built with another analyzer than the configured one is re-analysed in the background, one namespace at a time, while it keeps serving (stem variants of a query term match meanwhile); a restart resumes, and each cluster node does its own. The log says Full-text index re-analysed under the configured analyzer for each one. An index holding documents that exist only in the index, or agent memories that cannot be opened at that moment, keeps its analyzer (logged).

On v0.12.0, rebuild them once with a global admin key (this still works on v0.12.1):

curl -X POST https://<your-dakera>/admin/fulltext/reindex \
  -H "x-api-key: <global admin key>" -H "content-type: application/json" \
  -d '{"rebuild": true}'

Omit namespace to cover every agent memory namespace, or add "namespace": "<ns>" for one. Only derived search indexes are rebuilt; memories are untouched. Afterwards, a keyword search with a common word returns hits. Compose and Kubernetes forms are in Upgrading to v0.12.

Slow requests

• First recall after a restart that followed a write: the ANN index is rebuilt (about 13 s at 10,000 memories). Without a write since the last save, the saved index loads.
• Find the slow stage: read dakera_recall_pipeline_stage_duration_seconds{stage} and dakera_ingest_stage_duration_seconds{stage} (p50 and p95 per stage) to see whether embedding, search, storage or rerank dominates, and GET /admin/slow-queries for the slowest recent queries.
• Rerank cost: reranking scores every candidate on the CPU; bound it per request with rerank_candidates (Search and ranking).
• Background work: the post-upgrade re-embed and the leader's decay and consolidation jobs share the CPU; GET /admin/background-activity shows what the decay, deduplication and consolidation jobs have done.

Measured figures and a measurement recipe are in Performance.

Connection refused

• Check the server is running: docker ps | grep dakera.
• From another machine, use the server's address, not localhost.
• Check firewall rules for port 3000 (REST) and 50051 (gRPC).

MCP server not connecting

• Use an absolute path to the MCP binary and make sure it is executable.
• Check that DAKERA_API_URL points at the running server and DAKERA_API_KEY holds a valid key.
• Fully restart Claude Desktop or Claude Code after editing the configuration.
• Run the MCP binary directly in a terminal to see its errors.

Embedding and model failures

• A model download fails or is slow: the images ship bge-large and the reranker, so a default start downloads nothing. Other models download on first use into /app/models; pre-fetch them with dakera models pull, or mount a volume at /app/models so they download once (Models and Docker).
• Out of memory: lower concurrency or raise the container limit. Media jobs and entity extraction reserve memory first and answer 503 with Retry-After rather than risk an out-of-memory kill.
• Inconsistent recall across nodes: every node of a cluster must run the same DAKERA_MODEL; vectors from different models are not comparable.

Docker networking

• An SDK in another container cannot reach the server: use the service name, not localhost, for example http://dakera:3000 on the same Docker network.
• Port not reachable from the host: check -p 3000:3000 and the host firewall.
• Container exits immediately: read docker logs dakera --tail 20; the usual causes are a missing DAKERA_ROOT_API_KEY and a configuration refused with exit code 78 (see above).

# Test connectivity between containers
docker exec my-app-container curl -s http://dakera:3000/health

Rate limiting (429 Too Many Requests)

• Read the Retry-After header and the X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset headers.
• The server-wide limit is DAKERA_RATE_LIMIT_RPS (default 100) with DAKERA_RATE_LIMIT_BURST (default 50), per node unless the nodes share Redis. A 429 whose message says "operations per minute" comes from a namespace's memory policy instead (rate_limit_stores_per_minute, rate_limit_recalls_per_minute).
• For bulk loads, use POST /v1/memories/store/batch or POST /v1/import rather than one request per memory.

Storage and data issues

• Disk space: watch the volume with your host monitoring; GET /v1/analytics/storage breaks storage down per namespace. POST /ops/compact compacts the storage backend on request (a backend without it answers 501).
• Data not persisting across restarts: mount a volume at the data root, for example -v /srv/dakera:/data. Without one, data lives only in the container layer.
• Importance decaying faster than expected: decay runs every DAKERA_DECAY_INTERVAL_SECS (on the leader in a cluster) (default 3600) with a half-life of DAKERA_DECAY_HALF_LIFE_HOURS (default 168); a namespace's memory policy can override it. There is no switch to turn decay off: DAKERA_DECAY_ENABLED was never read, and v0.12.0 warns about it.

# Read a namespace's memory policy (decay, TTL, deduplication, rate limits)
curl http://localhost:3000/v1/namespaces/my-namespace/memory_policy \
  -H "Authorization: Bearer $DAKERA_API_KEY"

API key issues

• 403 INSUFFICIENT_SCOPE: the key exists but lacks the scope. Scopes are read, write, admin and super_admin, each including the one before.
• Lost root key: the root key is DAKERA_ROOT_API_KEY. Restart the server with a new value; stored data is unaffected.
• A client ignores the key: SDKs take the key in the client constructor; the MCP server reads DAKERA_API_KEY from its own environment (in Docker, -e DAKERA_API_KEY=...).

Cluster issues

• Nodes return different results: replication is eventual. Compare replication.outbox_entries and the vector counts in GET /admin/cluster/status on each node; a node that missed changes gets them from the sender's durable outbox or from reconciliation (every 5 minutes). Check that every node runs the same DAKERA_MODEL.
• Nodes do not see each other after an upgrade: nodes with and without DAKERA_CLUSTER_SECRET, or with different values, do not see each other, and each group elects its own leader. A node answers a peer without the right secret with 401 and logs refused a node-to-node request without the cluster secret. Set the same secret on every node; once all run v0.12.1 or later with it they form one membership, elect one leader and reconcile by version. GET /admin/cluster/status on every node should list every other node as a healthy member and the same leader.
• Leadership changes often: on a slow or distant network, raise DAKERA_ELECTION_LEASE_MS (default 15000); every other cluster timing derives from it.
• Writes refused with 503 and clock_ahead_ms in /health: this node's clock is ahead of its peers beyond DAKERA_CLUSTER_MAX_CLOCK_DRIFT_MS (default 60000). Fix NTP on the node.
• After a network partition: both sides keep accepting writes and converge when the partition heals; the write with the highest version wins, deletes included. No manual recovery is needed.
• No leader: a leader needs a majority of every member ever seen, minus those that left gracefully. Stop nodes with SIGTERM so they announce a graceful leave.

Details: High availability.

Knowledge graph has no entities

• Extraction is off for the namespace: entity extraction is opt-in per namespace. Turn it on with PATCH /v1/namespaces/{ns}/config and {"extract_entities": true, "entity_types": ["person", "organization", "location"]}; it applies to memories stored from then on.
• Test extraction on a sample: POST /v1/memories/extract returns the entities found in a text without storing anything.
• The extraction model did not load: entity_extraction appears in /health degraded when the memory budget refused it.
• A different agent_id: graph edges belong to one agent's namespace; recall with the agent that stored the memories.

Backup and restore issues

• Creating a backup: POST /admin/backups with a global admin key (Operations).
• Restore failing: restore a stored backup with POST /admin/backups/restore or a downloaded bundle with POST /admin/backups/upload (S3 storage only), with a global super_admin key, and poll GET /admin/backups/restore/{restore_id}. One restore runs at a time (409). Serving is not paused, so restore in a maintenance window. With one S3 bucket per node, a restore on a node that does not hold the backup's files is 409 naming the node that does.
• Backup size: set "compression": "zstd" on POST /admin/backups (lz4 is a 400). "encrypt": true without DAKERA_ENCRYPTION_KEY is also a 400.
• Cross-version restore: a backup taken under v0.12.0 is restored by v0.12.0, then converted with dakera downgrade if you need v0.11.108 (Rolling back to v0.11).

TLS

Dakera does not terminate TLS, so it has no certificate settings. Certificate errors come from the reverse proxy or load balancer in front of it: check the proxy's certificate, and point clients at the proxy's HTTPS address (Security, Deployment).

Getting help

If these steps do not resolve the issue:

• Ask in the Dakera Discord for setup questions and quick help from the team and other users.
• Open an issue at github.com/Dakera-AI/dakera-deploy/issues with the server version and logs.
• Include the output of curl http://localhost:3000/health, dakera --check-config and docker logs dakera --tail 50. /health contains no paths or credentials; check the logs for secrets before you post them.
• Describe the deployment (single node, cluster, Kubernetes, bare metal) and the failing request.