Worker fleet
The worker fleet lets session turns run on hosts that are not the core host: laptops, extra VPSes. A fleet host runs an activities-only Temporal worker, the "exec" role, which holds no database, control-token, Anthropic or GitHub credential. It runs the Claude CLI for a turn using a per-turn job token that the core minted, and it talks to the gateway data port only.
Status as stated by the source documents: the code (FL-1 to FL-4, and the gateway control-listener setting from the exposure design) is built and tested locally. Nothing is deployed or verified against real infrastructure: not a fleet host, the tailnet, a real Anthropic key, or tailscale serve. Every item in the "Unverified" section below needs a probe or an owner step.
Why the turn was split
Before the split a session worker held DATABASE_URL (full read and write on sessions, ledger and the kill switch) and the worker control token. Moving such a worker off the core host would expose both, so the design (docs/design-worker-fleet.md, risk R-1) says never run session workers off the gateway host before the split. The split (FL-3) moves only the database-free part to the fleet.
| Activity | Queue | Runs on | Holds |
|---|---|---|---|
prepareTurn | agentforge-sessions | core worker (dist/main.js) | DATABASE_URL, worker control token |
executeTurn | agentforge-sessions-exec | fleet exec workers (dist/exec-main.js) and, by default, the core worker | payload keys, gateway data URL, the turn's job token |
executeTurn fallback | agentforge-sessions-exec-core | core worker | as core |
finishTurn | agentforge-sessions | core worker | as core |
The queue names are derived from the session queue by execTaskQueue(q) = q + '-exec' and execCoreTaskQueue(q) = q + '-exec-core'. Queues are per role, not per host: all fleet hosts poll the same exec queue. A host's identity is carried separately, by the Temporal worker identity (WORKER_IDENTITY) shown on the poller and in workflow history for audit.
Topology
What a fleet host can reach: the Temporal frontend and the gateway data port. It has no route to Postgres, the gateway control port or other services (a network policy requirement, below).
Scheduling and fallback
The workflow sends executeTurn to the fleet queue with a scheduleToStartTimeout of SESSION_EXEC_SCHEDULE_TO_START_MS (set on the API; default 60,000, minimum 1000). If no worker picks the task up in time it reruns the same call on the core fallback queue. Retries are not used for the exec activity (maximumAttempts: 1); a retry must go through prepareTurn to get a new request id and token.
Because of WORKER_ALSO_EXEC (default true) the core process also polls the fleet queue, so with no fleet host attached a turn starts at once and attached hosts simply share the load. With WORKER_ALSO_EXEC=false (fleet-only exec) a turn waits up to SESSION_EXEC_SCHEDULE_TO_START_MS before the core fallback runs it. The fallback poller always runs. With the default, the core process can run up to WORKER_CONCURRENCY exec turns on each of the two exec queues at once, so memory must be sized for that.
A fleet host that sleeps or dies mid-turn misses its heartbeat (SESSION_TURN_HEARTBEAT_TIMEOUT_MS, default 30,000); the workflow then starts a new attempt with a new request id and token. prepareTurn revokes the old token and abandons its ledger row, so the old CLI gets 401 (registration_stale) at its next gateway call.
The exec worker
apps/worker/src/exec-main.ts creates one Worker with createExecActivities and no workflow bundle on TEMPORAL_TASK_QUEUE (default agentforge-sessions-exec). loadExecConfig refuses to start when the environment contains any of these (it names the variable, never its value):
| Refused pattern | Reason |
|---|---|
DATABASE_URL | no database |
ANTHROPIC_API_KEY | no model key |
any name starting GATEWAY_CONTROL_ | no control token or URL |
any name starting GITHUB_APP_ | no GitHub credential |
An esbuild metafile check (FL-ISO1) verifies that the exec entry's bundle graph contains no pg, @agentforge/db or control-token code.
Exec environment
| Variable | Required | Default | Notes |
|---|---|---|---|
TEMPORAL_ADDRESS | yes | none | Temporal frontend of the core |
TEMPORAL_NAMESPACE | no | agentforge-prod | |
TEMPORAL_TASK_QUEUE | no | agentforge-sessions-exec | never the core queue |
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID | yes | none | same codec keys as core: the prompt and the job token cross Temporal encrypted |
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAME | no | none | all three files or none |
TEMPORAL_ALLOW_PLAINTEXT | in production without TLS | none | the only accepted value is tailnet-only |
GATEWAY_URL | yes | none | https, loopback http, or http to a host listed in GATEWAY_HTTP_ALLOWED_HOSTS |
GATEWAY_HTTP_ALLOWED_HOSTS | no | empty | comma list of exact hostnames, no wildcards |
GATEWAY_MAX_OUTPUT_TOKENS | no | 8192 | must not exceed the gateway cap |
WORKER_IDENTITY | no | <hostname>-exec | audit identity |
WORKSPACE_DIR | no | /workspace | must exist |
SESSION_TURN_HEARTBEAT_INTERVAL_MS | no | 10000 | must equal the API's value (cancel latency) |
WORKER_CONCURRENCY | no | 1 | |
WORKER_SHUTDOWN_GRACE_MS | no | 5000 | |
NODE_ENV | no | none | production triggers the plaintext rule |
Safety rules enforced at startup:
- Plaintext Temporal (FL-4): with
NODE_ENV=productionand no TLS files the exec worker refuses to start unlessTEMPORAL_ALLOW_PLAINTEXT=tailnet-onlyis set, documenting the decision that the only control is the private network's ACL. Set it only when the Temporal address is reachable from the host over the private overlay alone. The core worker does not refuse:runWorkeronly logs the warningTEMPORAL_TLS_* unset: plaintext connection to Temporalin production (TEMPORAL_ALLOW_PLAINTEXTis not part of its config). - Gateway URL (FL-4): plain
httptoGATEWAY_URLis allowed only to loopback or to a host named inGATEWAY_HTTP_ALLOWED_HOSTS, because the job token travels in thex-api-keyheader. List a host only when the path is an encrypted private overlay. The same rule applies to the core worker's and the API'sGATEWAY_URLandGATEWAY_CONTROL_URL. - Heartbeat interval must be at most the heartbeat timeout divided by 3 (checked in the workflow args).
Core worker additions
| Variable | Default | Meaning |
|---|---|---|
WORKER_IDENTITY | SDK default | passed to all of the core's Temporal workers |
WORKER_ALSO_EXEC | true | core also polls the fleet exec queue |
GATEWAY_HTTP_ALLOWED_HOSTS | empty | covers both gateway URLs |
PER_REQUEST_CAP_USD | 2 | session turn reservation cap |
Running an exec host
The runbook (docs/deploy-worker-fleet.md) describes two shapes. Both were only exercised locally.
- Laptop (macOS with OrbStack): build the
exectarget ofapps/worker/Dockerfile, run the local image check script, produce an env file from the dedicated fleet secret item, then run the container with a restart policy andWORKER_IDENTITYset to the short host name plus-exec. The CLI runs as a subprocess inside the container; no Docker socket is needed, and an injected read sees only the container. Stop withdocker stop(SIGTERM, grace, then the turn retries elsewhere). - VPS (Linux): the same image and env, or
node dist/exec-main.jsfrom apnpm deploytree with the env from a root-only file. The host needs outbound access to the Temporal frontend and the gateway data port only; no inbound port.
Removing a host from the fleet is done by removing its network tag, which cuts both Temporal and the gateway.
Network policy requirements
- One ACL tag for exec hosts, with grants only to the Temporal frontend port and the gateway data port (or the HTTPS port a
tailscale servefront end uses). No wildcard rules, no route to Postgres, the gateway control port or other services of the core host. - Temporal 1.23 OSS cannot restrict by task queue. A fleet host that reaches the frontend can poll any queue of any namespace and fail its tasks (risk R-5). The ACL is the only control until Temporal authentication is enabled.
Gateway exposure design
The gateway has two listeners: data (port 8787 by default: job tokens, model proxy) and control (port 8788: register and revoke jobs, requires a control token of at least 32 characters). Fleet hosts need only the data port from outside the box. A published Swarm port gives the whole gateway container an address on the shared ingress overlay, so a control listener bound to all interfaces would then be reachable from every other service that publishes a port; the control token would be the only remaining barrier. Launch therefore published nothing, which left fleet hosts unable to reach the gateway.
docs/design-gateway-fleet-exposure.md compares four options and records the owner's choice (2026-11-16):
| Option | What changes | Verdict |
|---|---|---|
| A. Bind the control listener to the private-network address | new gateway setting picks the interface address inside a CIDR; control never listens on ingress | chosen |
| B. Separate data-only edge service | a second service forwards the data path only | fallback if A fails its probe |
| C. Keep bind-any and rely on the token | publish the data port as is | rejected: removes the network layer of defence |
| D. Split into two gateway services | data and control in different containers | rejected: large refactor (shared in-memory registrations) |
As built (step 1)
GATEWAY_CONTROL_BIND_CIDR is implemented in apps/gateway/src/config.ts (controlHostFromCidr, called by loadConfig with an injectable interface lookup). It is not released or deployed.
- IPv4 only, prefix
/8to/32. An IPv6 CIDR, a malformed CIDR, a prefix shorter than/8or extra/parts are refused. Host bits are ignored. - Every IPv4 address of every interface is checked; network and broadcast addresses count as inside. Exactly one match becomes
GATEWAY_CONTROL_HOST. No match or several matches refuse startup. - It is mutually exclusive with
GATEWAY_CONTROL_BIND_ANYandGATEWAY_CONTROL_HOST; setting both refuses startup. A blank value counts as unset. - The derived address goes through every existing rule (one explicit IP, different from
GATEWAY_DATA_HOST). The data listener keeps all interfaces. - Error messages name the setting and the match count only, never the CIDR or the local addresses.
- The older rule remains: the control listener refuses the wildcard bind addresses (all IPv4 or all IPv6 interfaces) unless
GATEWAY_CONTROL_BIND_ANY=private-network-only(the only accepted value), to be set only when the gateway is attached solely to the private encrypted network.
Steps 2 to 5 (pin the subnet, change the deployment catalog, grant the network path, configure fleet hosts) are open. See Deploy runbooks and Gateway.
Threat model of the fleet
After the split a compromised fleet host can: read the prompts of turns it is handed; forge replies for them (finishTurn trusts the exec result, risk R-7); spend per token up to that row's reservation (at most PER_REQUEST_CAP_USD, only allowlisted models and paths, at most 360 s; daily caps are enforced on the core at admission); poll other queues and namespaces it can reach and fail core tasks (denial of service). It cannot touch Postgres, mint or revoke tokens, flip the kill switch, or reach Anthropic without a token.
Controls: network ACL tags, per-host Temporal credentials if authentication is later enabled, the gateway's pre-auth and per-token rate limits (GATEWAY_PREAUTH_PER_SEC), per-token spend caps through the ledger row, and audit through gateway rows and Temporal worker identity. Host kill switches: local, SIGTERM with grace; remote, remove the network tag; global, the database kill switch and AGENTS_DISABLED. See Threat model.
Phases
| Phase | Content | Status in the source |
|---|---|---|
| FL-1 | gateway-mode ClaudeRuntime | built |
| FL-2 | session worker and API use the gateway; keys refused (FL-CFG1) | built |
| FL-3 | turn split (prepareTurn, executeTurn, finishTurn) behind patched('fleet-split') | built, tested locally |
| FL-4 | exec role refuses plaintext in production, WORKER_IDENTITY, gateway URL rules, exec Docker target | built, tested locally |
| FL-5 | TEMPORAL_API_KEY_FILE mapped to the native connection API key (only if JWT auth is chosen) | not started |
| FL-6 | keyless per-host relay for factory and sandbox workers on a laptop | not planned |
Factory and sandbox workers on a laptop are out of scope: the sandbox driver connects the gateway container to each job network and uses its address as the base URL (so the gateway must be on the same Docker host), and such workers hold GitHub App keys and the database URL.
Unverified
- Everything against real infrastructure: Temporal over the private network from a fleet host, the gateway on the core host reachable from the fleet,
tailscale serve, MagicDNS inside OrbStack containers (use the tailnet address inTEMPORAL_ADDRESSif it fails), the ACL grants. - A real model: the end-to-end tests use a fake upstream. Turn cost and timing with a real Anthropic response are unmeasured.
- A fleet host that sleeps mid-turn beyond the heartbeat timeout (simulated locally by SIGKILL, FL-WF2).
- Temporal history size with long transcripts (see Sessions and workflows).
- Whether the CLI accepts an HTTPS base URL on a tailnet name, Docker's anti-spoof behaviour on a shared overlay, interface ordering inside Swarm tasks with several networks.
Sources
apps/worker/src/exec-main.ts,apps/worker/src/exec-config.ts,apps/worker/src/exec-activities.ts,apps/worker/src/env.tsapps/worker/src/worker.ts(runWorker: fallback, also-exec, quality workers),apps/worker/src/config.ts(WORKER_ALSO_EXEC,WORKER_IDENTITY)apps/worker/src/activities.ts(prepareTurn,finishTurn)packages/agent-workflows/src/workflows/shared.ts(execTaskQueue,execCoreTaskQueue,DEFAULT_EXEC_SCHEDULE_TO_START_MS)packages/agent-workflows/src/workflows/index.ts(execute,splitTurn)apps/api/src/config.ts(SESSION_EXEC_SCHEDULE_TO_START_MS)docs/design-worker-fleet.md,docs/design-gateway-fleet-exposure.md,docs/deploy-worker-fleet.md