Skip to main content

Worker fleet

The worker fleet lets session turns run on hosts that are not the core host: laptops, extra VPSes. A fleet host runs an activities-only Temporal worker, the "exec" role, which holds no database, control-token, Anthropic or GitHub credential. It runs the Claude CLI for a turn using a per-turn job token that the core minted, and it talks to the gateway data port only.

Status as stated by the source documents: the code (FL-1 to FL-4, and the gateway control-listener setting from the exposure design) is built and tested locally. Nothing is deployed or verified against real infrastructure: not a fleet host, the tailnet, a real Anthropic key, or tailscale serve. Every item in the "Unverified" section below needs a probe or an owner step.

Why the turn was split​

Before the split a session worker held DATABASE_URL (full read and write on sessions, ledger and the kill switch) and the worker control token. Moving such a worker off the core host would expose both, so the design (docs/design-worker-fleet.md, risk R-1) says never run session workers off the gateway host before the split. The split (FL-3) moves only the database-free part to the fleet.

ActivityQueueRuns onHolds
prepareTurnagentforge-sessionscore worker (dist/main.js)DATABASE_URL, worker control token
executeTurnagentforge-sessions-execfleet exec workers (dist/exec-main.js) and, by default, the core workerpayload keys, gateway data URL, the turn's job token
executeTurn fallbackagentforge-sessions-exec-corecore workeras core
finishTurnagentforge-sessionscore workeras core

The queue names are derived from the session queue by execTaskQueue(q) = q + '-exec' and execCoreTaskQueue(q) = q + '-exec-core'. Queues are per role, not per host: all fleet hosts poll the same exec queue. A host's identity is carried separately, by the Temporal worker identity (WORKER_IDENTITY) shown on the poller and in workflow history for audit.

Topology​

What a fleet host can reach: the Temporal frontend and the gateway data port. It has no route to Postgres, the gateway control port or other services (a network policy requirement, below).

Scheduling and fallback​

The workflow sends executeTurn to the fleet queue with a scheduleToStartTimeout of SESSION_EXEC_SCHEDULE_TO_START_MS (set on the API; default 60,000, minimum 1000). If no worker picks the task up in time it reruns the same call on the core fallback queue. Retries are not used for the exec activity (maximumAttempts: 1); a retry must go through prepareTurn to get a new request id and token.

Because of WORKER_ALSO_EXEC (default true) the core process also polls the fleet queue, so with no fleet host attached a turn starts at once and attached hosts simply share the load. With WORKER_ALSO_EXEC=false (fleet-only exec) a turn waits up to SESSION_EXEC_SCHEDULE_TO_START_MS before the core fallback runs it. The fallback poller always runs. With the default, the core process can run up to WORKER_CONCURRENCY exec turns on each of the two exec queues at once, so memory must be sized for that.

A fleet host that sleeps or dies mid-turn misses its heartbeat (SESSION_TURN_HEARTBEAT_TIMEOUT_MS, default 30,000); the workflow then starts a new attempt with a new request id and token. prepareTurn revokes the old token and abandons its ledger row, so the old CLI gets 401 (registration_stale) at its next gateway call.

The exec worker​

apps/worker/src/exec-main.ts creates one Worker with createExecActivities and no workflow bundle on TEMPORAL_TASK_QUEUE (default agentforge-sessions-exec). loadExecConfig refuses to start when the environment contains any of these (it names the variable, never its value):

Refused patternReason
DATABASE_URLno database
ANTHROPIC_API_KEYno model key
any name starting GATEWAY_CONTROL_no control token or URL
any name starting GITHUB_APP_no GitHub credential

An esbuild metafile check (FL-ISO1) verifies that the exec entry's bundle graph contains no pg, @agentforge/db or control-token code.

Exec environment​

VariableRequiredDefaultNotes
TEMPORAL_ADDRESSyesnoneTemporal frontend of the core
TEMPORAL_NAMESPACEnoagentforge-prod
TEMPORAL_TASK_QUEUEnoagentforge-sessions-execnever the core queue
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_IDyesnonesame codec keys as core: the prompt and the job token cross Temporal encrypted
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAMEnononeall three files or none
TEMPORAL_ALLOW_PLAINTEXTin production without TLSnonethe only accepted value is tailnet-only
GATEWAY_URLyesnonehttps, loopback http, or http to a host listed in GATEWAY_HTTP_ALLOWED_HOSTS
GATEWAY_HTTP_ALLOWED_HOSTSnoemptycomma list of exact hostnames, no wildcards
GATEWAY_MAX_OUTPUT_TOKENSno8192must not exceed the gateway cap
WORKER_IDENTITYno<hostname>-execaudit identity
WORKSPACE_DIRno/workspacemust exist
SESSION_TURN_HEARTBEAT_INTERVAL_MSno10000must equal the API's value (cancel latency)
WORKER_CONCURRENCYno1
WORKER_SHUTDOWN_GRACE_MSno5000
NODE_ENVnononeproduction triggers the plaintext rule

Safety rules enforced at startup:

  • Plaintext Temporal (FL-4): with NODE_ENV=production and no TLS files the exec worker refuses to start unless TEMPORAL_ALLOW_PLAINTEXT=tailnet-only is set, documenting the decision that the only control is the private network's ACL. Set it only when the Temporal address is reachable from the host over the private overlay alone. The core worker does not refuse: runWorker only logs the warning TEMPORAL_TLS_* unset: plaintext connection to Temporal in production (TEMPORAL_ALLOW_PLAINTEXT is not part of its config).
  • Gateway URL (FL-4): plain http to GATEWAY_URL is allowed only to loopback or to a host named in GATEWAY_HTTP_ALLOWED_HOSTS, because the job token travels in the x-api-key header. List a host only when the path is an encrypted private overlay. The same rule applies to the core worker's and the API's GATEWAY_URL and GATEWAY_CONTROL_URL.
  • Heartbeat interval must be at most the heartbeat timeout divided by 3 (checked in the workflow args).

Core worker additions​

VariableDefaultMeaning
WORKER_IDENTITYSDK defaultpassed to all of the core's Temporal workers
WORKER_ALSO_EXECtruecore also polls the fleet exec queue
GATEWAY_HTTP_ALLOWED_HOSTSemptycovers both gateway URLs
PER_REQUEST_CAP_USD2session turn reservation cap

Running an exec host​

The runbook (docs/deploy-worker-fleet.md) describes two shapes. Both were only exercised locally.

  • Laptop (macOS with OrbStack): build the exec target of apps/worker/Dockerfile, run the local image check script, produce an env file from the dedicated fleet secret item, then run the container with a restart policy and WORKER_IDENTITY set to the short host name plus -exec. The CLI runs as a subprocess inside the container; no Docker socket is needed, and an injected read sees only the container. Stop with docker stop (SIGTERM, grace, then the turn retries elsewhere).
  • VPS (Linux): the same image and env, or node dist/exec-main.js from a pnpm deploy tree with the env from a root-only file. The host needs outbound access to the Temporal frontend and the gateway data port only; no inbound port.

Removing a host from the fleet is done by removing its network tag, which cuts both Temporal and the gateway.

Network policy requirements​

  • One ACL tag for exec hosts, with grants only to the Temporal frontend port and the gateway data port (or the HTTPS port a tailscale serve front end uses). No wildcard rules, no route to Postgres, the gateway control port or other services of the core host.
  • Temporal 1.23 OSS cannot restrict by task queue. A fleet host that reaches the frontend can poll any queue of any namespace and fail its tasks (risk R-5). The ACL is the only control until Temporal authentication is enabled.

Gateway exposure design​

The gateway has two listeners: data (port 8787 by default: job tokens, model proxy) and control (port 8788: register and revoke jobs, requires a control token of at least 32 characters). Fleet hosts need only the data port from outside the box. A published Swarm port gives the whole gateway container an address on the shared ingress overlay, so a control listener bound to all interfaces would then be reachable from every other service that publishes a port; the control token would be the only remaining barrier. Launch therefore published nothing, which left fleet hosts unable to reach the gateway.

docs/design-gateway-fleet-exposure.md compares four options and records the owner's choice (2026-11-16):

OptionWhat changesVerdict
A. Bind the control listener to the private-network addressnew gateway setting picks the interface address inside a CIDR; control never listens on ingresschosen
B. Separate data-only edge servicea second service forwards the data path onlyfallback if A fails its probe
C. Keep bind-any and rely on the tokenpublish the data port as isrejected: removes the network layer of defence
D. Split into two gateway servicesdata and control in different containersrejected: large refactor (shared in-memory registrations)

As built (step 1)​

GATEWAY_CONTROL_BIND_CIDR is implemented in apps/gateway/src/config.ts (controlHostFromCidr, called by loadConfig with an injectable interface lookup). It is not released or deployed.

  • IPv4 only, prefix /8 to /32. An IPv6 CIDR, a malformed CIDR, a prefix shorter than /8 or extra / parts are refused. Host bits are ignored.
  • Every IPv4 address of every interface is checked; network and broadcast addresses count as inside. Exactly one match becomes GATEWAY_CONTROL_HOST. No match or several matches refuse startup.
  • It is mutually exclusive with GATEWAY_CONTROL_BIND_ANY and GATEWAY_CONTROL_HOST; setting both refuses startup. A blank value counts as unset.
  • The derived address goes through every existing rule (one explicit IP, different from GATEWAY_DATA_HOST). The data listener keeps all interfaces.
  • Error messages name the setting and the match count only, never the CIDR or the local addresses.
  • The older rule remains: the control listener refuses the wildcard bind addresses (all IPv4 or all IPv6 interfaces) unless GATEWAY_CONTROL_BIND_ANY=private-network-only (the only accepted value), to be set only when the gateway is attached solely to the private encrypted network.

Steps 2 to 5 (pin the subnet, change the deployment catalog, grant the network path, configure fleet hosts) are open. See Deploy runbooks and Gateway.

Threat model of the fleet​

After the split a compromised fleet host can: read the prompts of turns it is handed; forge replies for them (finishTurn trusts the exec result, risk R-7); spend per token up to that row's reservation (at most PER_REQUEST_CAP_USD, only allowlisted models and paths, at most 360 s; daily caps are enforced on the core at admission); poll other queues and namespaces it can reach and fail core tasks (denial of service). It cannot touch Postgres, mint or revoke tokens, flip the kill switch, or reach Anthropic without a token.

Controls: network ACL tags, per-host Temporal credentials if authentication is later enabled, the gateway's pre-auth and per-token rate limits (GATEWAY_PREAUTH_PER_SEC), per-token spend caps through the ledger row, and audit through gateway rows and Temporal worker identity. Host kill switches: local, SIGTERM with grace; remote, remove the network tag; global, the database kill switch and AGENTS_DISABLED. See Threat model.

Phases​

PhaseContentStatus in the source
FL-1gateway-mode ClaudeRuntimebuilt
FL-2session worker and API use the gateway; keys refused (FL-CFG1)built
FL-3turn split (prepareTurn, executeTurn, finishTurn) behind patched('fleet-split')built, tested locally
FL-4exec role refuses plaintext in production, WORKER_IDENTITY, gateway URL rules, exec Docker targetbuilt, tested locally
FL-5TEMPORAL_API_KEY_FILE mapped to the native connection API key (only if JWT auth is chosen)not started
FL-6keyless per-host relay for factory and sandbox workers on a laptopnot planned

Factory and sandbox workers on a laptop are out of scope: the sandbox driver connects the gateway container to each job network and uses its address as the base URL (so the gateway must be on the same Docker host), and such workers hold GitHub App keys and the database URL.

Unverified​

  • Everything against real infrastructure: Temporal over the private network from a fleet host, the gateway on the core host reachable from the fleet, tailscale serve, MagicDNS inside OrbStack containers (use the tailnet address in TEMPORAL_ADDRESS if it fails), the ACL grants.
  • A real model: the end-to-end tests use a fake upstream. Turn cost and timing with a real Anthropic response are unmeasured.
  • A fleet host that sleeps mid-turn beyond the heartbeat timeout (simulated locally by SIGKILL, FL-WF2).
  • Temporal history size with long transcripts (see Sessions and workflows).
  • Whether the CLI accepts an HTTPS base URL on a tailnet name, Docker's anti-spoof behaviour on a shared overlay, interface ordering inside Swarm tasks with several networks.

Sources​

  • apps/worker/src/exec-main.ts, apps/worker/src/exec-config.ts, apps/worker/src/exec-activities.ts, apps/worker/src/env.ts
  • apps/worker/src/worker.ts (runWorker: fallback, also-exec, quality workers), apps/worker/src/config.ts (WORKER_ALSO_EXEC, WORKER_IDENTITY)
  • apps/worker/src/activities.ts (prepareTurn, finishTurn)
  • packages/agent-workflows/src/workflows/shared.ts (execTaskQueue, execCoreTaskQueue, DEFAULT_EXEC_SCHEDULE_TO_START_MS)
  • packages/agent-workflows/src/workflows/index.ts (execute, splitTurn)
  • apps/api/src/config.ts (SESSION_EXEC_SCHEDULE_TO_START_MS)
  • docs/design-worker-fleet.md, docs/design-gateway-fleet-exposure.md, docs/deploy-worker-fleet.md