Skip to main content

Deploy runbooks

Condensed runbooks per service, with the exact order of steps and the owner-go rules. The full originals are docs/deploy-api-coolify.md, docs/deploy-gateway-coolify.md, docs/deploy-worker-coolify.md, docs/deploy-worker-fleet.md, docs/deploy-factory.md, docs/deploy-sandbox.md and docs/deploy-quality.md.

Read this first: the file names say "coolify", and each runbook opens with "Every step below is UNVERIFIED". Production is CapRover on the core host (ADR 0004), and the api, gateway and worker were deployed and smoke-tested there. The runbook items that depend on a proxy (hop count, SSE buffering, proxy timeouts) and the restore drill are still unverified.

Owner-go rules​

RuleDetail
A live change needs the owner's go, in the session that performs itApply, restart, teardown and release tags. A first go prompt that was denied by mistake was re-approved by the owner directly in that session
One service per apply--tags apps --limit <host> -e service_filter=<name>, because the services are excluded_from_bulk_apps: true (ADR 0004)
Owner-only stepsCreating ECR repos with lifecycle policy, every secret write with the env-sync write token, pushing the vX.Y.Z tag, dispatching the mirror (digests chosen by the owner), merging the catalog PR and running the apply, DNS and certificates, tailnet ACL edits, namespace creation, key rotation
Merge the catalog PR right before the applyA merged entry with no live app makes every fleet --tags apps run fail the drift gate. Keep the revert path ready and time-box with a cutoff
Never retype a digestExtract it from the release run log and compare programmatically
No credentials in chat or outputAgent sessions may load the write token and cloud keys into a throwaway tmux shell only with an explicit in-session go, answering each env-sync confirmation after reading item and field names
Prove the deployThe drift gate (read-only) and docker service ps, not the recap line

Master order for a stack deploy​

The app-02 throwaway test applied in the order postgres, temporal, gateway, worker, api with the migration right after postgres. The core host stack has no Temporal app because it uses the shared cluster. Worker-type apps get no health gate: check each with docker inspect or docker service ps before the next apply.

Postgres (agentforge-postgres)​

  1. Mirror the upstream image into ECR by digest (docker-release manual dispatch, postgres_ref, image_tag=vX.Y.Z, build_apps=false).
  2. Seed the item agentforge-postgres with POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD (generated with openssl rand).
  3. Catalog entry type worker, one instance, with the named volume agentforge-postgres-data at /var/lib/postgresql/data (volumes cannot be added to a non-persistent CapRover app: delete and re-register once, ADR 0006).
  4. Apply, then check docker service ps.
  5. Run migrations once (below). With a volume this is once per schema change, not once per restart.

Without a volume (the first deploy), a Postgres restart dropped the schema and the spend ledger, so the daily caps reset and migrations had to be repeated. See Persistence and backups.

Migrations​

The server never migrates at boot. Run, from the api container (it already has DATABASE_URL):

docker exec <api container> node dist/migrate.js

Or as a one-shot container attached to the private network, with an env file (mode 0600, only DATABASE_URL, deleted afterwards):

docker run --rm --network <private overlay> --env-file <file> <api image> node dist/migrate.js

The second form is from the throwaway test, where the overlay was captain-overlay-network. The migration files are packages/db/drizzle/0000_init.sql to 0007_quality_submissions.sql. After the first deploy the database held 11 tables; migrations 0006_quality and 0007_quality_submissions have run nowhere. Migrations are applied by hand, so a schema-changing release has an explicit manual step after the apply.

Gateway (agentforge-gateway)​

Checklist order, from docs/deploy-gateway-coolify.md and the app-03 findings:

  1. Image from the release (it carries the baked org policy and role policies in /policies, and an empty /registry).
  2. Replicas 1; stop grace at least 12 s.
  3. Data listener GATEWAY_DATA_PORT (default 8787): reachable only from the runtime network; never public.
  4. Control listener GATEWAY_CONTROL_PORT (default 8788): bound to one internal address. On the core host this is GATEWAY_CONTROL_HOST=0.0.0.0 together with GATEWAY_CONTROL_BIND_ANY=private-network-only and a gateway attached only to agentforge-net. 0.0.0.0 and :: are otherwise refused, and the control host must differ from GATEWAY_DATA_HOST.
  5. Policy directory GATEWAY_POLICY_DIR (image default /policies) is baked in. An invalid policy file refuses startup; a policy change needs a new image tag and a re-pin (ADR 0003).
  6. MCP registry GATEWAY_REGISTRY_DIR (image default /registry, empty is valid: no MCP tools, and a policy granting any tool then fails registration with 400).
  7. Required env: ANTHROPIC_API_KEY (the only holder), DATABASE_URL, GATEWAY_MODELS, control tokens, plus the CLI allowlists below.
  8. Apply, then run the checks.

Env that the first smoke test showed is mandatory for the real CLI (the defaults deny the CLI's requests before authentication):

VariableValue on the core hostWhy
GATEWAY_ALLOWED_QUERIESbeta=trueThe CLI sends POST /v1/messages?beta=true; the default is empty
GATEWAY_ALLOWED_FIELDSmodel,messages,system,max_tokens,temperature,top_p,top_k,stop_sequences,stream,metadata,tools,tool_choice,thinking,context_management,output_configThe default list lacks context_management and output_config
GATEWAY_MODELSclaude-haiku-5-5,claude-sonnet-5-5Each model needs a price in src/pricing.ts
GATEWAY_MAX_TOKENS_CAP8192Default
GATEWAY_CONTROL_BIND_ANYprivate-network-onlyOnly accepted value
GATEWAY_CONTROL_HOST0.0.0.0The image has no default (found by the first smoke test)

Checks after deploy:

  • First log line gateway listening shows the hosts, ports, models and allowlistsFrozen.
  • GET /health on the data port returns exactly {"status":"ok"}.
  • GET /control/v1/health with the worker token returns {status, auditWriteFailures, preAuthDenials}. auditWriteFailures > 0 means audit rows are being lost (requests are still answered).
  • The control port is not reachable from the sandbox network or the public internet.
  • POST /mcp without a job token returns 401; GET /mcp returns 405.

Refused at startup: UPSTREAM_BASE_URL other than https://api.anthropic.com, NODE_TLS_REJECT_UNAUTHORIZED=0, any NODE_USE_ENV_PROXY, NODE_EXTRA_CA_CERTS, SSL_CERT_FILE or SSL_CERT_DIR unless GATEWAY_ALLOW_EXTRA_CA=1, NODE_OPTIONS unless GATEWAY_ALLOW_NODE_OPTIONS=1, an invalid registry, and a partial or non-https IdP setting. Never set GATEWAY_TEST_UPSTREAM.

Core session worker (agentforge-worker)​

  1. Image from the release, default stage. Replicas 1, stop grace at least 8 s.
  2. It refuses ANTHROPIC_API_KEY (FL-CFG1). Model calls go through the gateway.
  3. Env, same values as the API where noted:
VariableValueNotes
DATABASE_URLapp Postgressame as the API
GATEWAY_URL, GATEWAY_CONTROL_URLhttp://<gateway service name>:8787 and :8788plain http on the private overlay; the service name must be in GATEWAY_HTTP_ALLOWED_HOSTS
GATEWAY_CONTROL_TOKEN_WORKER_FILE/run/secrets/agentforge-ctl-workerpath of a mounted 0400 file
GATEWAY_MAX_OUTPUT_TOKENS8192never above the gateway's GATEWAY_MAX_TOKENS_CAP
PER_REQUEST_CAP_USD2.00default since 2026-11-13; per session turn, so a multi-call sonnet turn at the 80k-character cap fits (the API's synchronous cap stays 0.5)
CLIENT_DAILY_LIMIT_USD, GLOBAL_DAILY_LIMIT_USD2.00, 5.00production values set by the secrets generator
TEMPORAL_ADDRESS, TEMPORAL_NAMESPACEshared cluster, agentforge-prod (default)plaintext, ACL only
TEMPORAL_TASK_QUEUEunset (default agentforge-sessions)the API must use the same
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_IDsame as the APIcheck that the codecKeys fingerprints in the worker config log line equal the API's
SESSION_TURN_HEARTBEAT_INTERVAL_MSthe API's value (default 10000)both apps log it at startup; they must match
WORKER_CONCURRENCY2 (default)
WORKER_ALSO_EXECtrue (default)the core worker also polls agentforge-sessions-exec
AGENTS_DISABLED, SONNET_MIN_RESERVATION_USD, BUDGET_FLOOR_USD, BUDGET_MARGIN, ALERT_WEBHOOK_URLsame as the API
  1. Apply, then check the logs: worker config (fingerprints, heartbeat interval, namespace, task queue, tls), then worker running. docker exec <worker> cat /proc/1/comm prints tini.

API (agentforge-api)​

Order from docs/deploy-api-coolify.md:

  1. Replicas 1, restart policy always, stop grace at least 10 s.
  2. Container port 3000 is not published to a public interface. TRUST_PROXY_HOPS trusts the rightmost X-Forwarded-For entries without checking who sent them, so a direct path to the port lets a client choose its own rate-limit IP.
  3. Environment (runtime secrets, never build variables):
VariableValueNotes
DATABASE_URLapp Postgresrequired
ANTHROPIC_API_KEYnot setrefused since FL-2
GATEWAY_URL, GATEWAY_CONTROL_URL, GATEWAY_CONTROL_TOKEN_API_FILEas for the worker, with the api token file /run/secrets/agentforge-ctl-apithe gateway accepts the api token on /control/v1/jobs only for session-readonly jobs and lets it revoke only its own registrations; GATEWAY_CONTROL_TOKEN_WORKER_FILE is refused
API_KEYSJSON {"<client>":"<key>"}client ^[a-z0-9-]{1,64}$, key at least 32 characters, no duplicate keys
API_ADMIN_CLIENTScomma listeach must be a key of API_KEYS; may read GET /v1/usage
MODELScomma listrequired, non-empty
TRUST_PROXY_HOPS1 behind one appending proxydefault 0 is for local dev; a startup warn line starting TRUST_PROXY_HOPS=0 appears when it is 0 in production
PER_REQUEST_CAP_USD0.5default
CLIENT_DAILY_LIMIT_USD, GLOBAL_DAILY_LIMIT_USD2.00, 5.00 on the core host (defaults 1 and 2)
PORT, API_PORT3000set by the generator; image default PORT=3000

Further defaults: MAX_CONCURRENCY 2, RATE_LIMIT_PER_MIN 30 (per client), AUTH_FAIL_LIMIT_PER_MIN 10 (per IP), SHUTDOWN_GRACE_MS 8000, STALE_RESERVATION_SEC 900 (minimum 600), STALE_CHECK_INTERVAL_MS 300000, SSE_PING_MS 15000 (minimum 1000), BUDGET_MARGIN 0.2, SONNET_MIN_RESERVATION_USD 0.5, BUDGET_FLOOR_USD 0.02.

  1. Image-set variables, leave unset: NODE_ENV=production, HOST=0.0.0.0, PORT=3000, WORKSPACE_DIR=/workspace, HOME=/home/agentforge.
  2. Workspace: /workspace mounted read-only. Never point WORKSPACE_DIR at /app or anything containing node_modules; the workspace audit refuses it.
  3. Migrations, then deploy checks:
curl -fsS "$API/health" # {"status":"ok"}
curl -fsS --oauth2-bearer "$KEY" "$API/v1/models" # the MODELS list
curl -s -o /dev/null -w '%{http_code}\n' "$API/v1/models" # 401 without the header

Proxy items still unverified: which proxy runs, whether it appends or replaces X-Forwarded-For, a proxy read timeout of at least 300 s plus margin (a request may run timeoutMs 300 s), and that POST /v1/agent/stream is not buffered or cut (the first event: message line must arrive at least 1 s before event: done; hold one stream open longer than 300 s idle except pings).

Smoke test​

Run from the owner's shell with the key read from 1Password without printing. Print only status codes, the reply text, the error code and the cost.

scripts/deploy/smoke-app02.sh automates this for the throwaway host: env API_URL, API_KEY (never printed), optional MODEL (default claude-haiku-5-5), MAX_COST_USD (default 0.30), TIMEOUT_SEC (default 180), and EXPECT_DISABLED=1 for the kill-switch check (the turn must end with errorCode disabled and cost nothing). The kill-switch check was never run on the deployed stack.

Recorded results: smoke 1 on the throwaway host cost 0.00038208 USD; the core host runs cost 0.00037995 USD and 0.0003812 USD (cumulative owner spend 0.00076115 USD, 0 stale reservations, 0 running workflows).

Fleet exec worker (laptop, VPS)​

Not connected to production yet. The runbook is docs/deploy-worker-fleet.md.

  1. Build the image locally (not released): docker build -f apps/worker/Dockerfile --target exec -t agentforge-worker-exec .
  2. bash scripts/exec-image.test.sh agentforge-worker-exec (local checks, no infra).
  3. Create the env file from the 1Password item agentforge/worker-fleet only (payload keys, GATEWAY_URL, TEMPORAL_ADDRESS, optional certs). Never a database credential, control token, model key or GitHub key.
  4. Run it with --restart unless-stopped and WORKER_IDENTITY set to <short hostname>-exec.
  5. Remove a host from the fleet by removing its tailnet tag.

Exec worker env:

VariableRequiredNotes
TEMPORAL_ADDRESSyesthe shared cluster, port 7233
TEMPORAL_NAMESPACEnodefault agentforge-prod
TEMPORAL_TASK_QUEUEnodefault agentforge-sessions-exec; never the core queue
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_IDyesthe same codec keys as core
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAMEnoall three files or none
TEMPORAL_ALLOW_PLAINTEXTin production without TLSonly accepted value tailnet-only
GATEWAY_URLyeshttps, loopback http, or http to a host in GATEWAY_HTTP_ALLOWED_HOSTS
GATEWAY_HTTP_ALLOWED_HOSTSnoexact hostnames, no wildcards
GATEWAY_MAX_OUTPUT_TOKENSnodefault 8192
WORKER_IDENTITYnodefault <hostname>-exec
WORKSPACE_DIRnodefault /workspace; must exist
SESSION_TURN_HEARTBEAT_INTERVAL_MSnodefault 10000; must equal the API's value
WORKER_CONCURRENCYnodefault 1
WORKER_SHUTDOWN_GRACE_MSnodefault 5000

The exec worker exits at startup, naming the variable but never its value, if it finds DATABASE_URL, ANTHROPIC_API_KEY, any GATEWAY_CONTROL_* or any GITHUB_APP_*.

Where session turns run (workflow patch fleet-split):

ActivityQueueHost
prepareTurnagentforge-sessionscore
executeTurnagentforge-sessions-execfleet exec workers, and by default core (WORKER_ALSO_EXEC=true)
executeTurn fallbackagentforge-sessions-exec-corecore, same process as prepareTurn
finishTurnagentforge-sessionscore

With WORKER_ALSO_EXEC=false, executeTurn waits SESSION_EXEC_SCHEDULE_TO_START_MS (API setting, min 1000, default 60000) for a fleet worker, then runs on the core fallback queue. A laptop that sleeps mid-turn misses its heartbeat; the workflow starts a new attempt (new requestId, new token), revokes the old token and abandons its ledger row, so the old CLI gets 401.

Owner-only steps before the fleet works: ACL tag and grants (done for Temporal, open for 8787), a gateway exposure path, the 1Password item, publishing the exec image, and rotating the model test key once the gateway path is live.

Factory worker and API factory routes​

Not deployed. Roles and env are in docs/deploy-factory.md; the essentials:

DeploymentEntryHoldsNever holds
APInode dist/server.js, FACTORY_ENABLEDAPI_KEYS, Temporal client, PostgresGitHub App keys, model key, docker socket
Factory workerDockerfile target factory, node dist/factory-main.jstwo GitHub App private keys, gateway worker control token, docker CLI and socket, host git 2.50.1 or newerANTHROPIC_API_KEY (startup refuses)
Session workerdefault targetas beforeany GITHUB_APP_* (startup refuses)
Gatewayas abovemodel key, per-job tokens, factory-<role> policiesGitHub keys
Sandbox imageapps/sandbox-runner/Dockerfile, by digest in SANDBOX_IMAGEonly the per-job tokenany other credential

SANDBOX_RUNTIME=runc needs FACTORY_ALLOW_RUNC=throwaway-repo-only; use runsc for any repo that matters. The default FACTORY_MAX_JOB_USD is 0.50 (cap 2.00), FACTORY_MAX_ACTIVE_PER_CLIENT default 2, FACTORY_TASK_QUEUE default agentforge-factory. See Factory overview and Sandbox roles.

Quality (CodeCrucible) rollout​

Nothing is enabled: with QUALITY_ENABLED unset the routes answer 404 and the core worker registers no quality activities. Rollout order, each live step needing the owner's go:

  1. Release tag carrying the quality code (it ships dormant). Migration 0006_quality.sql is run by hand.
  2. Core host: set QUALITY_ENABLED and the limits on the API and the core worker, mount the policy directory (needs bead .112 merged, which it is on main).
  3. Exec host (PRE-0 and PRE-1 first): operator-cloned mirror, the quality image by digest, the exec worker.
  4. Baseline run for packages/agent-core on a protected-branch commit, then one verify of a hand-made branch that kills a seeded survivor; record numbers in docs/p2-findings.md.
  5. CI enforcement (CC-4.2, bead .109): an API key of a QUALITY_CLIENTS client as a CI secret, a runner-to-API route, statuses: write per job, and the required status (an owner action).

Preconditions, none decided yet except PRE-2:

IdNeeded forState
PRE-0 Temporal access (authorizer with claim mapper, dedicated cluster, or written sign-off)any exec worker on the tailnetdeferred 2026-11-16 until a host exists
PRE-1 sandbox host with runscspike S3, any non-local exec hostowner
PRE-2 Postgres volume on the core hostPhase 4done (2026-11-16)
PRE-3 read-only GitHub App keyany repository fetched from GitHubowner

Until PRE-0 and PRE-1 are settled only local development is supported: a Temporal dev server, QUALITY_RUNTIME=runc, QUALITY_ALLOW_RUNC=own-repo-only, a local mirror. The owner decided to hold the local dogfood (baseline plus one real verify of packages/agent-core) until after 10:00 local time on 2026-11-17. Rollback is stopping the exec worker; open runs then end error: exec_lost. See Quality overview and Exec worker and image.

Sandbox host settings that are deploy-time​

SettingValueWhy
Gateway sysctlsnet.ipv4.ip_forward=0, net.ipv4.conf.all.forwarding=0the gateway joins every job network and must not route between them
Host-side service portspublish on 127.0.0.1 onlyotherwise reachable from an --internal network
Sandbox imageby digest only<repo>@sha256:<64 hex>

Sources​

  • docs/deploy-api-coolify.md
  • docs/deploy-gateway-coolify.md
  • docs/deploy-worker-coolify.md
  • docs/deploy-worker-fleet.md
  • docs/deploy-factory.md
  • docs/deploy-sandbox.md
  • docs/deploy-quality.md
  • docs/deploy-app02-test.md (order, migration, smoke, owner-only steps)
  • docs/design-gateway-fleet-exposure.md
  • docs/p2-findings.md ("App-03 production deploy and smoke test", "App-03 Postgres persistent volume")
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.md, SUMMARY.md
  • docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/LESSONS.md
  • docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md, 0005-gateway-publishes-no-port-at-launch.md, 0006-postgres-persistent-volume.md
  • scripts/deploy/smoke-app02.sh, scripts/deploy/make-app03-secrets.sh
  • apps/api/src/config.ts, apps/api/src/migrate.ts