Deploy runbooks
Condensed runbooks per service, with the exact order of steps and the owner-go rules. The full originals are docs/deploy-api-coolify.md, docs/deploy-gateway-coolify.md, docs/deploy-worker-coolify.md, docs/deploy-worker-fleet.md, docs/deploy-factory.md, docs/deploy-sandbox.md and docs/deploy-quality.md.
Read this first: the file names say "coolify", and each runbook opens with "Every step below is UNVERIFIED". Production is CapRover on the core host (ADR 0004), and the api, gateway and worker were deployed and smoke-tested there. The runbook items that depend on a proxy (hop count, SSE buffering, proxy timeouts) and the restore drill are still unverified.
Owner-go rules
| Rule | Detail |
|---|---|
| A live change needs the owner's go, in the session that performs it | Apply, restart, teardown and release tags. A first go prompt that was denied by mistake was re-approved by the owner directly in that session |
| One service per apply | --tags apps --limit <host> -e service_filter=<name>, because the services are excluded_from_bulk_apps: true (ADR 0004) |
| Owner-only steps | Creating ECR repos with lifecycle policy, every secret write with the env-sync write token, pushing the vX.Y.Z tag, dispatching the mirror (digests chosen by the owner), merging the catalog PR and running the apply, DNS and certificates, tailnet ACL edits, namespace creation, key rotation |
| Merge the catalog PR right before the apply | A merged entry with no live app makes every fleet --tags apps run fail the drift gate. Keep the revert path ready and time-box with a cutoff |
| Never retype a digest | Extract it from the release run log and compare programmatically |
| No credentials in chat or output | Agent sessions may load the write token and cloud keys into a throwaway tmux shell only with an explicit in-session go, answering each env-sync confirmation after reading item and field names |
| Prove the deploy | The drift gate (read-only) and docker service ps, not the recap line |
Master order for a stack deploy
The app-02 throwaway test applied in the order postgres, temporal, gateway, worker, api with the migration right after postgres. The core host stack has no Temporal app because it uses the shared cluster. Worker-type apps get no health gate: check each with docker inspect or docker service ps before the next apply.
Postgres (agentforge-postgres)
- Mirror the upstream image into ECR by digest (
docker-releasemanual dispatch,postgres_ref,image_tag=vX.Y.Z,build_apps=false). - Seed the item
agentforge-postgreswithPOSTGRES_DB,POSTGRES_USER,POSTGRES_PASSWORD(generated withopenssl rand). - Catalog entry type
worker, one instance, with the named volumeagentforge-postgres-dataat/var/lib/postgresql/data(volumes cannot be added to a non-persistent CapRover app: delete and re-register once, ADR 0006). - Apply, then check
docker service ps. - Run migrations once (below). With a volume this is once per schema change, not once per restart.
Without a volume (the first deploy), a Postgres restart dropped the schema and the spend ledger, so the daily caps reset and migrations had to be repeated. See Persistence and backups.
Migrations
The server never migrates at boot. Run, from the api container (it already has DATABASE_URL):
docker exec <api container> node dist/migrate.js
Or as a one-shot container attached to the private network, with an env file (mode 0600, only DATABASE_URL, deleted afterwards):
docker run --rm --network <private overlay> --env-file <file> <api image> node dist/migrate.js
The second form is from the throwaway test, where the overlay was captain-overlay-network. The migration files are packages/db/drizzle/0000_init.sql to 0007_quality_submissions.sql. After the first deploy the database held 11 tables; migrations 0006_quality and 0007_quality_submissions have run nowhere. Migrations are applied by hand, so a schema-changing release has an explicit manual step after the apply.
Gateway (agentforge-gateway)
Checklist order, from docs/deploy-gateway-coolify.md and the app-03 findings:
- Image from the release (it carries the baked org policy and role policies in
/policies, and an empty/registry). - Replicas 1; stop grace at least 12 s.
- Data listener
GATEWAY_DATA_PORT(default 8787): reachable only from the runtime network; never public. - Control listener
GATEWAY_CONTROL_PORT(default 8788): bound to one internal address. On the core host this isGATEWAY_CONTROL_HOST=0.0.0.0together withGATEWAY_CONTROL_BIND_ANY=private-network-onlyand a gateway attached only toagentforge-net.0.0.0.0and::are otherwise refused, and the control host must differ fromGATEWAY_DATA_HOST. - Policy directory
GATEWAY_POLICY_DIR(image default/policies) is baked in. An invalid policy file refuses startup; a policy change needs a new image tag and a re-pin (ADR 0003). - MCP registry
GATEWAY_REGISTRY_DIR(image default/registry, empty is valid: no MCP tools, and a policy granting any tool then fails registration with 400). - Required env:
ANTHROPIC_API_KEY(the only holder),DATABASE_URL,GATEWAY_MODELS, control tokens, plus the CLI allowlists below. - Apply, then run the checks.
Env that the first smoke test showed is mandatory for the real CLI (the defaults deny the CLI's requests before authentication):
| Variable | Value on the core host | Why |
|---|---|---|
GATEWAY_ALLOWED_QUERIES | beta=true | The CLI sends POST /v1/messages?beta=true; the default is empty |
GATEWAY_ALLOWED_FIELDS | model,messages,system,max_tokens,temperature,top_p,top_k,stop_sequences,stream,metadata,tools,tool_choice,thinking,context_management,output_config | The default list lacks context_management and output_config |
GATEWAY_MODELS | claude-haiku-5-5,claude-sonnet-5-5 | Each model needs a price in src/pricing.ts |
GATEWAY_MAX_TOKENS_CAP | 8192 | Default |
GATEWAY_CONTROL_BIND_ANY | private-network-only | Only accepted value |
GATEWAY_CONTROL_HOST | 0.0.0.0 | The image has no default (found by the first smoke test) |
Checks after deploy:
- First log line
gateway listeningshows the hosts, ports, models andallowlistsFrozen. GET /healthon the data port returns exactly{"status":"ok"}.GET /control/v1/healthwith the worker token returns{status, auditWriteFailures, preAuthDenials}.auditWriteFailures > 0means audit rows are being lost (requests are still answered).- The control port is not reachable from the sandbox network or the public internet.
POST /mcpwithout a job token returns 401;GET /mcpreturns 405.
Refused at startup: UPSTREAM_BASE_URL other than https://api.anthropic.com, NODE_TLS_REJECT_UNAUTHORIZED=0, any NODE_USE_ENV_PROXY, NODE_EXTRA_CA_CERTS, SSL_CERT_FILE or SSL_CERT_DIR unless GATEWAY_ALLOW_EXTRA_CA=1, NODE_OPTIONS unless GATEWAY_ALLOW_NODE_OPTIONS=1, an invalid registry, and a partial or non-https IdP setting. Never set GATEWAY_TEST_UPSTREAM.
Core session worker (agentforge-worker)
- Image from the release, default stage. Replicas 1, stop grace at least 8 s.
- It refuses
ANTHROPIC_API_KEY(FL-CFG1). Model calls go through the gateway. - Env, same values as the API where noted:
| Variable | Value | Notes |
|---|---|---|
DATABASE_URL | app Postgres | same as the API |
GATEWAY_URL, GATEWAY_CONTROL_URL | http://<gateway service name>:8787 and :8788 | plain http on the private overlay; the service name must be in GATEWAY_HTTP_ALLOWED_HOSTS |
GATEWAY_CONTROL_TOKEN_WORKER_FILE | /run/secrets/agentforge-ctl-worker | path of a mounted 0400 file |
GATEWAY_MAX_OUTPUT_TOKENS | 8192 | never above the gateway's GATEWAY_MAX_TOKENS_CAP |
PER_REQUEST_CAP_USD | 2.00 | default since 2026-11-13; per session turn, so a multi-call sonnet turn at the 80k-character cap fits (the API's synchronous cap stays 0.5) |
CLIENT_DAILY_LIMIT_USD, GLOBAL_DAILY_LIMIT_USD | 2.00, 5.00 | production values set by the secrets generator |
TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE | shared cluster, agentforge-prod (default) | plaintext, ACL only |
TEMPORAL_TASK_QUEUE | unset (default agentforge-sessions) | the API must use the same |
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID | same as the API | check that the codecKeys fingerprints in the worker config log line equal the API's |
SESSION_TURN_HEARTBEAT_INTERVAL_MS | the API's value (default 10000) | both apps log it at startup; they must match |
WORKER_CONCURRENCY | 2 (default) | |
WORKER_ALSO_EXEC | true (default) | the core worker also polls agentforge-sessions-exec |
AGENTS_DISABLED, SONNET_MIN_RESERVATION_USD, BUDGET_FLOOR_USD, BUDGET_MARGIN, ALERT_WEBHOOK_URL | same as the API |
- Apply, then check the logs:
worker config(fingerprints, heartbeat interval, namespace, task queue,tls), thenworker running.docker exec <worker> cat /proc/1/commprintstini.
API (agentforge-api)
Order from docs/deploy-api-coolify.md:
- Replicas 1, restart policy always, stop grace at least 10 s.
- Container port 3000 is not published to a public interface.
TRUST_PROXY_HOPStrusts the rightmostX-Forwarded-Forentries without checking who sent them, so a direct path to the port lets a client choose its own rate-limit IP. - Environment (runtime secrets, never build variables):
| Variable | Value | Notes |
|---|---|---|
DATABASE_URL | app Postgres | required |
ANTHROPIC_API_KEY | not set | refused since FL-2 |
GATEWAY_URL, GATEWAY_CONTROL_URL, GATEWAY_CONTROL_TOKEN_API_FILE | as for the worker, with the api token file /run/secrets/agentforge-ctl-api | the gateway accepts the api token on /control/v1/jobs only for session-readonly jobs and lets it revoke only its own registrations; GATEWAY_CONTROL_TOKEN_WORKER_FILE is refused |
API_KEYS | JSON {"<client>":"<key>"} | client ^[a-z0-9-]{1,64}$, key at least 32 characters, no duplicate keys |
API_ADMIN_CLIENTS | comma list | each must be a key of API_KEYS; may read GET /v1/usage |
MODELS | comma list | required, non-empty |
TRUST_PROXY_HOPS | 1 behind one appending proxy | default 0 is for local dev; a startup warn line starting TRUST_PROXY_HOPS=0 appears when it is 0 in production |
PER_REQUEST_CAP_USD | 0.5 | default |
CLIENT_DAILY_LIMIT_USD, GLOBAL_DAILY_LIMIT_USD | 2.00, 5.00 on the core host (defaults 1 and 2) | |
PORT, API_PORT | 3000 | set by the generator; image default PORT=3000 |
Further defaults: MAX_CONCURRENCY 2, RATE_LIMIT_PER_MIN 30 (per client), AUTH_FAIL_LIMIT_PER_MIN 10 (per IP), SHUTDOWN_GRACE_MS 8000, STALE_RESERVATION_SEC 900 (minimum 600), STALE_CHECK_INTERVAL_MS 300000, SSE_PING_MS 15000 (minimum 1000), BUDGET_MARGIN 0.2, SONNET_MIN_RESERVATION_USD 0.5, BUDGET_FLOOR_USD 0.02.
- Image-set variables, leave unset:
NODE_ENV=production,HOST=0.0.0.0,PORT=3000,WORKSPACE_DIR=/workspace,HOME=/home/agentforge. - Workspace:
/workspacemounted read-only. Never pointWORKSPACE_DIRat/appor anything containingnode_modules; the workspace audit refuses it. - Migrations, then deploy checks:
curl -fsS "$API/health" # {"status":"ok"}
curl -fsS --oauth2-bearer "$KEY" "$API/v1/models" # the MODELS list
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/models" # 401 without the header
Proxy items still unverified: which proxy runs, whether it appends or replaces X-Forwarded-For, a proxy read timeout of at least 300 s plus margin (a request may run timeoutMs 300 s), and that POST /v1/agent/stream is not buffered or cut (the first event: message line must arrive at least 1 s before event: done; hold one stream open longer than 300 s idle except pings).
Smoke test
Run from the owner's shell with the key read from 1Password without printing. Print only status codes, the reply text, the error code and the cost.
scripts/deploy/smoke-app02.sh automates this for the throwaway host: env API_URL, API_KEY (never printed), optional MODEL (default claude-haiku-5-5), MAX_COST_USD (default 0.30), TIMEOUT_SEC (default 180), and EXPECT_DISABLED=1 for the kill-switch check (the turn must end with errorCode disabled and cost nothing). The kill-switch check was never run on the deployed stack.
Recorded results: smoke 1 on the throwaway host cost 0.00038208 USD; the core host runs cost 0.00037995 USD and 0.0003812 USD (cumulative owner spend 0.00076115 USD, 0 stale reservations, 0 running workflows).
Fleet exec worker (laptop, VPS)
Not connected to production yet. The runbook is docs/deploy-worker-fleet.md.
- Build the image locally (not released):
docker build -f apps/worker/Dockerfile --target exec -t agentforge-worker-exec . bash scripts/exec-image.test.sh agentforge-worker-exec(local checks, no infra).- Create the env file from the 1Password item
agentforge/worker-fleetonly (payload keys,GATEWAY_URL,TEMPORAL_ADDRESS, optional certs). Never a database credential, control token, model key or GitHub key. - Run it with
--restart unless-stoppedandWORKER_IDENTITYset to<short hostname>-exec. - Remove a host from the fleet by removing its tailnet tag.
Exec worker env:
| Variable | Required | Notes |
|---|---|---|
TEMPORAL_ADDRESS | yes | the shared cluster, port 7233 |
TEMPORAL_NAMESPACE | no | default agentforge-prod |
TEMPORAL_TASK_QUEUE | no | default agentforge-sessions-exec; never the core queue |
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID | yes | the same codec keys as core |
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAME | no | all three files or none |
TEMPORAL_ALLOW_PLAINTEXT | in production without TLS | only accepted value tailnet-only |
GATEWAY_URL | yes | https, loopback http, or http to a host in GATEWAY_HTTP_ALLOWED_HOSTS |
GATEWAY_HTTP_ALLOWED_HOSTS | no | exact hostnames, no wildcards |
GATEWAY_MAX_OUTPUT_TOKENS | no | default 8192 |
WORKER_IDENTITY | no | default <hostname>-exec |
WORKSPACE_DIR | no | default /workspace; must exist |
SESSION_TURN_HEARTBEAT_INTERVAL_MS | no | default 10000; must equal the API's value |
WORKER_CONCURRENCY | no | default 1 |
WORKER_SHUTDOWN_GRACE_MS | no | default 5000 |
The exec worker exits at startup, naming the variable but never its value, if it finds DATABASE_URL, ANTHROPIC_API_KEY, any GATEWAY_CONTROL_* or any GITHUB_APP_*.
Where session turns run (workflow patch fleet-split):
| Activity | Queue | Host |
|---|---|---|
prepareTurn | agentforge-sessions | core |
executeTurn | agentforge-sessions-exec | fleet exec workers, and by default core (WORKER_ALSO_EXEC=true) |
executeTurn fallback | agentforge-sessions-exec-core | core, same process as prepareTurn |
finishTurn | agentforge-sessions | core |
With WORKER_ALSO_EXEC=false, executeTurn waits SESSION_EXEC_SCHEDULE_TO_START_MS (API setting, min 1000, default 60000) for a fleet worker, then runs on the core fallback queue. A laptop that sleeps mid-turn misses its heartbeat; the workflow starts a new attempt (new requestId, new token), revokes the old token and abandons its ledger row, so the old CLI gets 401.
Owner-only steps before the fleet works: ACL tag and grants (done for Temporal, open for 8787), a gateway exposure path, the 1Password item, publishing the exec image, and rotating the model test key once the gateway path is live.
Factory worker and API factory routes
Not deployed. Roles and env are in docs/deploy-factory.md; the essentials:
| Deployment | Entry | Holds | Never holds |
|---|---|---|---|
| API | node dist/server.js, FACTORY_ENABLED | API_KEYS, Temporal client, Postgres | GitHub App keys, model key, docker socket |
| Factory worker | Dockerfile target factory, node dist/factory-main.js | two GitHub App private keys, gateway worker control token, docker CLI and socket, host git 2.50.1 or newer | ANTHROPIC_API_KEY (startup refuses) |
| Session worker | default target | as before | any GITHUB_APP_* (startup refuses) |
| Gateway | as above | model key, per-job tokens, factory-<role> policies | GitHub keys |
| Sandbox image | apps/sandbox-runner/Dockerfile, by digest in SANDBOX_IMAGE | only the per-job token | any other credential |
SANDBOX_RUNTIME=runc needs FACTORY_ALLOW_RUNC=throwaway-repo-only; use runsc for any repo that matters. The default FACTORY_MAX_JOB_USD is 0.50 (cap 2.00), FACTORY_MAX_ACTIVE_PER_CLIENT default 2, FACTORY_TASK_QUEUE default agentforge-factory. See Factory overview and Sandbox roles.
Quality (CodeCrucible) rollout
Nothing is enabled: with QUALITY_ENABLED unset the routes answer 404 and the core worker registers no quality activities. Rollout order, each live step needing the owner's go:
- Release tag carrying the quality code (it ships dormant). Migration
0006_quality.sqlis run by hand. - Core host: set
QUALITY_ENABLEDand the limits on the API and the core worker, mount the policy directory (needs bead.112merged, which it is onmain). - Exec host (PRE-0 and PRE-1 first): operator-cloned mirror, the quality image by digest, the exec worker.
- Baseline run for
packages/agent-coreon a protected-branch commit, then one verify of a hand-made branch that kills a seeded survivor; record numbers indocs/p2-findings.md. - CI enforcement (CC-4.2, bead
.109): an API key of aQUALITY_CLIENTSclient as a CI secret, a runner-to-API route,statuses: writeper job, and the required status (an owner action).
Preconditions, none decided yet except PRE-2:
| Id | Needed for | State |
|---|---|---|
| PRE-0 Temporal access (authorizer with claim mapper, dedicated cluster, or written sign-off) | any exec worker on the tailnet | deferred 2026-11-16 until a host exists |
PRE-1 sandbox host with runsc | spike S3, any non-local exec host | owner |
| PRE-2 Postgres volume on the core host | Phase 4 | done (2026-11-16) |
| PRE-3 read-only GitHub App key | any repository fetched from GitHub | owner |
Until PRE-0 and PRE-1 are settled only local development is supported: a Temporal dev server, QUALITY_RUNTIME=runc, QUALITY_ALLOW_RUNC=own-repo-only, a local mirror. The owner decided to hold the local dogfood (baseline plus one real verify of packages/agent-core) until after 10:00 local time on 2026-11-17. Rollback is stopping the exec worker; open runs then end error: exec_lost. See Quality overview and Exec worker and image.
Sandbox host settings that are deploy-time
| Setting | Value | Why |
|---|---|---|
| Gateway sysctls | net.ipv4.ip_forward=0, net.ipv4.conf.all.forwarding=0 | the gateway joins every job network and must not route between them |
| Host-side service ports | publish on 127.0.0.1 only | otherwise reachable from an --internal network |
| Sandbox image | by digest only | <repo>@sha256:<64 hex> |
Sources
docs/deploy-api-coolify.mddocs/deploy-gateway-coolify.mddocs/deploy-worker-coolify.mddocs/deploy-worker-fleet.mddocs/deploy-factory.mddocs/deploy-sandbox.mddocs/deploy-quality.mddocs/deploy-app02-test.md(order, migration, smoke, owner-only steps)docs/design-gateway-fleet-exposure.mddocs/p2-findings.md("App-03 production deploy and smoke test", "App-03 Postgres persistent volume")docs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.md,SUMMARY.mddocs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/LESSONS.mddocs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.mddocs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md,0005-gateway-publishes-no-port-at-launch.md,0006-postgres-persistent-volume.mdscripts/deploy/smoke-app02.sh,scripts/deploy/make-app03-secrets.shapps/api/src/config.ts,apps/api/src/migrate.ts