Troubleshooting

Common issues and solutions when self-hosting Honcho

This page covers common issues you may encounter when self-hosting Honcho, what causes them, and how to fix them.

​

Startup Failures

​

Server won’t start: “Missing client for …”

ValueError: Missing client for Deriver: google

Cause: The server validates at startup that all configured LLM providers have API keys. If a provider is referenced in your configuration but the corresponding API key isn’t set, the server refuses to start.

Fix: Set the API keys for your configured providers. With the default configuration every feature, embeddings included, uses the openai transport, so one key is enough:

LLM_OPENAI_API_KEY=...    # Or an OpenRouter/other OpenAI-compatible key with per-feature BASE_URL overrides

You only need LLM_ANTHROPIC_API_KEY or LLM_GEMINI_API_KEY if you switch a feature’s MODEL_CONFIG__TRANSPORT to anthropic or gemini.

See the LLM Setup section for provider configuration. You can change which providers are used in your .env or config.toml (see Configuration Guide).

​

Server won’t start: “JWT_SECRET must be set”

ValueError: JWT_SECRET must be set if USE_AUTH is true

Cause: You enabled authentication (AUTH_USE_AUTH=true) but didn’t provide a JWT secret.

Fix: Generate a secret and set it:

python scripts/generate_jwt_secret.py
# Then set the output as:
AUTH_JWT_SECRET=<generated_secret>

Or disable authentication for local development: AUTH_USE_AUTH=false

​

Runtime Errors

​

API returns “An unexpected error occurred” on every request

Cause: This is almost always a database issue. The health endpoint (/health) will return {"status": "ok"} even when the database is unreachable because it doesn’t check the database connection. The actual error appears in the server logs.

Common causes and fixes:

1. Database is unreachable — Check that PostgreSQL is running and the DB_CONNECTION_URI is correct
2. Migrations haven’t been run — The server starts successfully without tables, but every API call will fail. Run:
   uv run alembic upgrade head
   
   In Docker:
   docker compose exec api uv run alembic upgrade head
   
3. pgvector extension not installed — The vector extension must be enabled in your database:
   CREATE EXTENSION IF NOT EXISTS vector;
   

How to diagnose: Check the server logs for the actual error. Look for:

• sqlalchemy.exc.OperationalError — database connection issue
• sqlalchemy.exc.ProgrammingError with “relation does not exist” — migrations not run
• psycopg.OperationalError — connection refused or authentication failed

​

Health check passes but API calls fail

The /health endpoint is a lightweight check that confirms the server process is running. It does not verify:

• Database connectivity
• That migrations have been run
• That LLM providers are reachable

To verify full functionality, try creating a workspace:

curl -X POST http://localhost:8000/v3/workspaces \
  -H "Content-Type: application/json" \
  -d '{"name": "test"}'

If this succeeds, your database connection and migrations are working.

​

Deriver not processing messages

Messages are stored but no observations, summaries, or representations are being generated.

Common causes:

1. Deriver isn’t running — In manual setup, the deriver is a separate process:

   uv run python -m src.deriver
   

   In Docker, it starts automatically via docker compose up.

2. Deriver can’t reach the database — Check deriver logs for connection errors. The deriver uses the same DB_CONNECTION_URI as the API server.

3. Missing LLM API key for deriver provider — By default the deriver uses the openai transport (LLM_OPENAI_API_KEY). Check deriver logs for API errors.

4. Processing backlog — With DERIVER_WORKERS=1 (default), high message volume can cause a backlog. Increase workers:

   DERIVER_WORKERS=4
   

​

Alternative Provider Issues

​

OpenRouter / custom provider not working

If calls to an OpenAI-compatible proxy fail:

1. Verify the endpoint and key are set. Use transport = "openai" with a base URL override:

   LLM_OPENAI_API_KEY=sk-or-v1-...
   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=https://openrouter.ai/api/v1
   

2. Check model names match the provider’s format. OpenRouter uses vendor/model format (e.g., anthropic/claude-haiku-4-5), not the raw model ID.

3. Ensure your model supports tool calling. The deriver, dialectic, and dream agents require tool use. Check the provider’s model page for tool calling support.

4. Check server logs for the actual error. API errors from the upstream provider will appear in Honcho’s logs with the HTTP status code and message body.

​

vLLM / Ollama not responding

1. Verify the model server is running and accessible from the Honcho process (or container):

   curl http://localhost:8000/v1/models   # vLLM
   curl http://localhost:11434/v1/models  # Ollama
   

2. In Docker, localhost inside a container doesn’t reach the host. Use host.docker.internal (macOS/Windows) or the host’s network IP:

   DERIVER_MODEL_CONFIG__OVERRIDES__BASE_URL=http://host.docker.internal:8000/v1
   

3. Structured output failures — vLLM’s structured output support is limited to certain response formats. If you see JSON parsing errors, check the deriver/dream logs for the raw response. See Deriver produces no observations below.

​

Deriver produces no observations

If messages are processed (the queue drains, no errors in logs) but peers never accumulate observations — and you’re using an OpenAI-compatible provider — the likely cause is that the provider doesn’t support OpenAI Structured Outputs (json_schema). The OpenAI backend requests json_schema by default; providers like Z.AI GLM and some Ollama/vLLM deployments either reject it or silently ignore it and return prose, which the deriver can’t parse into observations.

Fix: set STRUCTURED_OUTPUT_MODE=json_object on the deriver’s model config to request loose JSON mode, which injects the schema into the prompt instead:

DERIVER_MODEL_CONFIG__STRUCTURED_OUTPUT_MODE=json_object

This is a per-model-config setting on the OpenAI transport; set it on whichever features use the affected provider (e.g. DREAM_DEDUCTION_MODEL_CONFIG__STRUCTURED_OUTPUT_MODE).

​

Chat with response_format returns 422

A chat request with a response_format returns 422 with The model's answer did not match the requested response_format when the final answer still isn’t JSON matching the schema after retries. The usual cause is the same as above: a provider or proxy that accepts json_schema without enforcing it. For example, LiteLLM in front of Anthropic models emulates json_schema with a tool the model may decline to call, especially on the final answer of a tool loop.

Fix: set STRUCTURED_OUTPUT_MODE=json_object on the affected dialectic levels:

DIALECTIC_LEVELS__medium__MODEL_CONFIG__STRUCTURED_OUTPUT_MODE=json_object

This sets the level’s primary model only. If its fallback model also goes through the same proxy, set STRUCTURED_OUTPUT_MODE on the fallback config too.

Streaming chat requests (stream=true) don’t validate the answer, so they never return this 422: a non-JSON answer streams through with a 200.

​

Thinking budget errors with non-Anthropic providers

If you see errors like thinking budget not supported, invalid parameter, or silent failures where agents produce no output, one of your per-component *_MODEL_CONFIG__THINKING_BUDGET_TOKENS overrides is likely set to a value > 0 with a provider that doesn’t support Anthropic-style extended thinking. The built-in defaults do not set thinking budgets, so this only applies if you added those overrides yourself.

Fix: Set *_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0 for every component when using models that don’t support thinking:

DERIVER_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
SUMMARY_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DREAM_DEDUCTION_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DREAM_INDUCTION_MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__minimal__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__low__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__medium__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__high__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0
DIALECTIC_LEVELS__max__MODEL_CONFIG__THINKING_BUDGET_TOKENS=0

For OpenAI reasoning models, use *_MODEL_CONFIG__THINKING_EFFORT instead of *_MODEL_CONFIG__THINKING_BUDGET_TOKENS.

​

Database Issues

​

Connection string format

The connection URI must use the postgresql+psycopg prefix:

# Correct
DB_CONNECTION_URI=postgresql+psycopg://postgres:postgres@localhost:5432/postgres

# Wrong - will fail
DB_CONNECTION_URI=postgresql://postgres:postgres@localhost:5432/postgres
DB_CONNECTION_URI=postgres://postgres:postgres@localhost:5432/postgres

​

Checking migration status

# See current migration version
uv run alembic current

# See migration history
uv run alembic history

# Upgrade to latest
uv run alembic upgrade head

​

”permission denied to create extension “vector""

Cause: The role in DB_CONNECTION_URI is not a superuser and lacks privileges to create extensions. Honcho issues CREATE EXTENSION IF NOT EXISTS vector both before running migrations and again at server startup, so this fails in both places. IF NOT EXISTS does not help — Postgres checks the privilege before checking whether the extension exists.

Fix: Preinstall pgvector once with a privileged role (superuser, or rds_superuser on RDS) in the Honcho database:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE EXTENSION IF NOT EXISTS btree_gin;

The IF NOT EXISTS calls then short-circuit and the application role needs no extension privileges. This is the supported path for deployments where the platform, not the application, owns extension management.

You won’t hit this with docker compose — the bundled stack connects as postgres, a superuser, and its database service also creates the extension from database/init.sql on first boot. It comes up on managed Postgres, Kubernetes operators, and other setups where you bring your own database and its role is deliberately not a superuser.

​

”pg_trgm is not installed; skipping ix_messages_tenant_content_trgm”

Cause: Migrations couldn’t create the pg_trgm extension, so they skipped the trigram index on messages.content. Search still works, but ILIKE matching scans the workspace’s messages, which gets slow on large workspaces. (Older releases named this index ix_messages_content_trgm.)

Fix: Install the extension as a privileged role, then create the index (the migration is already recorded as applied, so re-running migrations won’t add it):

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX CONCURRENTLY ix_messages_tenant_content_trgm ON "<schema>".messages USING gin (tenant_id, workspace_name, content gin_trgm_ops);

Where messages is partitioned (the multi-tenant schema), CONCURRENTLY is refused on the parent: run CREATE INDEX ... ON ONLY "<schema>".messages, then CREATE INDEX CONCURRENTLY on each partition, then ALTER INDEX ... ATTACH PARTITION each one. The parent turns valid once every partition is attached.

​

”btree_gin is not installed; skipping ix_messages_tenant_content_fts”

Cause: Migrations couldn’t create the btree_gin extension, so they kept the previous message search indexes (ix_messages_content_gin and ix_messages_content_trgm), which index only the text. Search still works and still uses those, but on a multi-tenant (partitioned) database each text match reads every tenant’s matches in the partition before the tenant and workspace filters apply. The same cause skips ix_queue_pending_observers, the queue index that finds pending work fanning out to a peer within a workspace; lookups that would use it fall back to other queue indexes.

Fix: Install the extension as a privileged role, then create the indexes (the migration is already recorded as applied, so re-running migrations won’t add them), and drop the old text indexes once the new ones are valid:

CREATE EXTENSION IF NOT EXISTS btree_gin;
CREATE INDEX CONCURRENTLY ix_messages_tenant_content_fts ON "<schema>".messages USING gin (tenant_id, workspace_name, to_tsvector('english', content));
CREATE INDEX CONCURRENTLY ix_messages_tenant_content_trgm ON "<schema>".messages USING gin (tenant_id, workspace_name, content gin_trgm_ops);
CREATE INDEX CONCURRENTLY ix_queue_pending_observers ON "<schema>".queue USING gin (workspace_name, (payload -> 'observers')) WHERE processed = false AND (payload -> 'observers') IS NOT NULL;
DROP INDEX CONCURRENTLY ix_messages_content_gin;
DROP INDEX CONCURRENTLY ix_messages_content_trgm;

On a partitioned messages, build each index with the ON ONLY / per-partition / ATTACH PARTITION steps described above.

​

Migration waits a long time on “Building ix_messages_tenant_content_… concurrently”

Cause: A concurrent index build waits for every open transaction that has touched messages (including sessions left idle in transaction), and then for every transaction in the database still holding an older snapshot, such as a long-running query on any table. Either one holds it up, and the build itself takes time proportional to the size of messages, once per partition on a partitioned table. The migration gives up after 60 minutes on a build and leaves an invalid index, which the next run drops and rebuilds.

Fix: On large or busy databases, build the indexes before upgrading, with the statements above under the same names (and, on a partitioned table, the partition index named after the parent plus the partition suffix, e.g. ix_messages_tenant_content_fts_p7 on messages_p7). The migration then finds them and only drops the old indexes. To see what the build is waiting on, check pg_stat_activity for sessions in idle in transaction and for old xact_start or backend_xmin values.

​

Cache & Redis

​

Redis is optional

Redis is used for caching when CACHE_ENABLED=true (default: false). If Redis is unreachable, Honcho gracefully falls back to in-memory caching and logs a warning. This means:

• The server and deriver will still start and function normally
• Performance may be reduced under high load without Redis
• You do not need Redis for local development or testing

​

Redis connection issues

If you see Redis connection warnings in logs but CACHE_ENABLED=false, they can be safely ignored. If you want caching:

# Start Redis via Docker
docker run -d -p 6379:6379 redis:latest

# Configure Honcho
CACHE_ENABLED=true
CACHE_URL=redis://localhost:6379/0

​

Docker Issues

​

Docker build fails with permission errors

The Honcho Dockerfile uses BuildKit mount syntax and creates a non-root app user. Common build failures:

1. BuildKit not enabled

The Dockerfile uses RUN --mount=type=cache which requires Docker BuildKit. If you see syntax errors during build:

# Ensure BuildKit is enabled
DOCKER_BUILDKIT=1 docker compose build

Or add to your Docker daemon config (/etc/docker/daemon.json):

{ "features": { "buildkit": true } }

2. Permission denied during build or at runtime (Linux)

On Linux, AppArmor or SELinux can block Docker build operations and volume mounts. Symptoms include permission denied errors during COPY, RUN, or when the container tries to access mounted volumes.

# Check if AppArmor is blocking Docker
sudo aa-status | grep docker

# Temporarily test without AppArmor (for diagnosis only)
docker compose down
sudo aa-remove-unknown
docker compose up -d

For SELinux, add :z to volume mounts in docker-compose.yml:

volumes:
  - .:/app:z

3. Volume mount UID mismatch

The Dockerfile creates a non-root app user, but docker-compose.yml.example mounts .:/app which overlays the container filesystem with host-owned files. The app user inside the container may not have permission to read them.

If you see permission errors at runtime (not build time), you can either:

• Run without the source mount (remove - .:/app from volumes — the image already contains the code)
• Or fix ownership: sudo chown -R 100:101 . (matches the app user inside the container)

​

Containers start but API fails

1. Check container status: docker compose ps
2. Check API logs: docker compose logs api
3. Check database logs: docker compose logs database
4. Ensure migrations ran: docker compose exec api uv run alembic upgrade head

​

Port conflicts

If port 8000 is already in use:

# Check what's using the port
lsof -i :8000

# Or change the port mapping in docker-compose.yml
ports:
  - "8001:8000"  # Map to a different host port

​

Rebuilding after code changes

docker compose build --no-cache
docker compose up -d

​

CLI local stack (honcho start)

The CLI local stack lives under ~/.honcho/profiles/ (default profile local) and uses the published image ghcr.io/plastic-labs/honcho:latest. Logs: docker compose -p honcho-local logs. Full flags: CLI reference.

​

MISSING_LLM_KEY

Cause: No provider key in the environment or the profile .env.

Fix: Export LLM_OPENAI_API_KEY, LLM_ANTHROPIC_API_KEY, or LLM_GEMINI_API_KEY, or run honcho start --setup basic in a TTY (not with --json).

​

Timed out waiting for /health

Cause: The API container started but did not become ready within --timeout (default 180s).

Fix: Check docker compose -p honcho-local logs api and ... logs deriver. Increase --timeout. Confirm Docker is running. A first-time GHCR pull happens before this wait (during image pin) — if that step hung, look at Docker pull logs instead.

​

CLI still talks to api.honcho.dev

Cause: honcho start does not rewrite environmentUrl in ~/.honcho/config.json.

Fix: Prefix commands with HONCHO_BASE_URL=http://127.0.0.1:8000, or run honcho init --base-url http://127.0.0.1:8000 to persist local as the default. honcho status prints the one-shot hint.

​

Port already in use

The CLI remaps 8000/5432/6379 automatically unless you pinned them with --api-port / --db-port / --redis-port. Pass those flags if you need a specific host port.

​

Getting Help

If your issue isn’t covered here:

• Check the logs — most issues are diagnosed from server or deriver logs (docker compose logs for from-source compose; docker compose -p honcho-local logs for honcho start)
• GitHub Issues — Report bugs
• Discord — Join our community
• Configuration — See the Configuration Guide for all available settings