Skip to main content

Secrets and keys

This page names every secret in the system, who holds it, how it is delivered and which process refuses it. It contains no values. Operational steps (seeding, rotation commands) are in Secrets flow.

Inventory​

SecretHolder(s)VariableDeliveryRefused by
Model provider keygateway onlyANTHROPIC_API_KEYenvapi, core worker, factory worker, exec worker (startup error)
Gateway control token, api routegateway; apigateway: GATEWAY_CONTROL_TOKEN_API (env); api: GATEWAY_CONTROL_TOKEN_API_FILEgateway env; api swarm-secret filethe api refuses GATEWAY_CONTROL_TOKEN_WORKER_FILE; exec worker refuses any GATEWAY_CONTROL_*
Gateway control token, worker routegateway; core worker; factory workergateway: GATEWAY_CONTROL_TOKEN_WORKER (env); workers: GATEWAY_CONTROL_TOKEN_WORKER_FILEgateway env; worker swarm-secret fileexec worker
Job token (afj_…)gateway (hash only); one runtime for one turnsandbox: ANTHROPIC_API_KEY inside the container env file; exec: activity inputminted per turnexpires with the turn
Temporal payload keysapi, core worker, factory worker, exec workersTEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_IDenv (fleet: 1Password item via op inject)none (required)
Database credentialapi, core worker, factory worker, gatewayDATABASE_URLenvexec worker
IdP client secret (OBO, service identity)gatewayGATEWAY_IDP_CLIENT_SECRET (with GATEWAY_IDP_TOKEN_URL, GATEWAY_IDP_CLIENT_ID)envnone; unset in production (no IdP yet)
GitHub App private keysfactory workerGITHUB_APP_READ_PR_PRIVATE_KEY[_FILE], GITHUB_APP_PUSH_PRIVATE_KEY[_FILE]mounted 0400 file in productionexec worker refuses GITHUB_APP_*
API client keysapiAPI_KEYSenv
Postgres passwordpostgres servicePOSTGRES_PASSWORDenv

The gateway holds the only model key​

The owner decision behind this (ADR 0001 of the 2026-11-13 session) is that laptops and VPSes may run agent workers over the tailnet without the key or the database credential being copied to every machine. Enforcement in code:

ProcessCheckError text
apiANTHROPIC_API_KEY present at startup"the api never holds an Anthropic key (FL-CFG1)"
session workersame"the session worker never holds an Anthropic key (FL-CFG1)"
factory workersame"the factory worker never holds an Anthropic key (FX-CFG1)"
exec workerany of DATABASE_URL, ANTHROPIC_API_KEY, GATEWAY_CONTROL_*, GITHUB_APP_*"the exec worker holds no database, control-token, Anthropic or GitHub credential (FL-3)"

The gateway sends the key upstream only as x-api-key to the fixed upstream URL, with redirect: 'error', and refuses startup settings that would let a proxy or extra CA see it (NODE_TLS_REJECT_UNAUTHORIZED=0, NODE_USE_ENV_PROXY, NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, SSL_CERT_DIR, NODE_OPTIONS without their explicit allow flags). See Gateway.

Residual: gateway compromise equals key compromise (P2.G plan section 11). The key on the provider side has a spend limit; the owner decided not to rotate the earlier test key until the gateway path was live (docs/p2-findings.md).

The _FILE convention​

Secrets that must not sit in a process environment visible to other tools are read from a file whose path is in a variable ending in _FILE:

VariableReaderBehaviour
GATEWAY_CONTROL_TOKEN_API_FILEapps/api/src/runtime.tsreadFileSync(...).trim(); empty file is a startup error
GATEWAY_CONTROL_TOKEN_WORKER_FILEapps/worker/src/runtime.ts, factory configsame
GITHUB_APP_READ_PR_PRIVATE_KEY_FILE, GITHUB_APP_PUSH_PRIVATE_KEY_FILEapps/worker/src/factory/config.ts appKeyexactly one of the inline variable and the _FILE variant must be set

In production the control-token files are swarm secrets: each secret is one flat file under /run/secrets/<target>, mode 0400, delivered from a sibling SSM path /catesworks/<service>-files/prod/<KEY> (targets agentforge-ctl-api and agentforge-ctl-worker, key CTL_TOKEN). Each file holds the same value as the matching gateway env variable; the api never gets the worker token. The gateway reads its own control tokens and the model key from env.

Lesson from the app-03 deploy: the images run as uid 10001 while the role first mounted secrets as uid 1000, which gave EACCES; the secret uid must match the image (hostinger PR #102, ADR 0004).

Policy files are not secrets and are not delivered as swarm secrets: swarm secrets are flat, extensionless and owned by another uid, while the policy loader needs *.json or *.yaml names. The default policies are baked into the gateway image instead (ADR 0003).

Temporal payload codec keys​

Prompts, transcripts and job tokens cross Temporal inside activity inputs and results. The cluster is plaintext and reachable by anything on the tailnet ACL, so payloads are encrypted at the application level (packages/agent-workflows/src/codec.ts):

PropertyValue
CipherAES-256-GCM (node:crypto), 12-byte random IV per payload, 16-byte tag
Envelopethe whole payload (metadata and data) sealed as JSON; data is the IV, then the ciphertext, then the tag
Encoding markermetadata encoding = binary/encrypted, encryption-key-id = <keyId>
Key formatTEMPORAL_PAYLOAD_KEYS = JSON object mapping key id to base64 of 32 bytes
Current keyTEMPORAL_PAYLOAD_KEY_ID must name a key in the map
Rotationadd a new id, switch TEMPORAL_PAYLOAD_KEY_ID; every key in the map can still decode
Decodeunknown key id or a tampered payload throws (GCM authentication); non-encrypted payloads pass through
Startup logkeyFingerprints: per id, the first 12 hex chars of sha256 of the key (a consistency check, not secret)

Parse errors never contain key material. Every role that runs a Temporal client or worker (api, core worker, factory worker, exec worker) needs the same key map.

Delivery flow (roles)​

Rules recorded in the deploy docs:

  • secrets.manifest.yml (repo root) declares the env-sync services: agentforge-postgres, agentforge-gateway, agentforge-worker, agentforge-api, agentforge-worker-files, agentforge-api-files. It holds names only.
  • The env segment is prod only; a test segment would also be delivered and collide.
  • env-sync's prod gate (a TTY or --confirm) applies to every write; the write token is passed by variable name and never read by an agent.
  • scripts/deploy/make-app03-secrets.sh writes values through printf and redirection, never argv; it refuses the repo and shared temp dirs as output locations; it never talks to 1Password, AWS or the network.
  • The fleet item agentforge/worker-fleet holds payload keys, GATEWAY_URL, TEMPORAL_ADDRESS and per-host certs if any; never a database credential, control token, model key or GitHub key.

Unverified: the env-sync commands in docs/deploy-app02-test.md are marked UNVERIFIED there; real IdP secrets do not exist yet.

Logging and redaction​

packages/agent-core/src/redaction.ts scrubs every logged string (pino logMethod hook) and redacts known paths:

Pattern or pathReplacement
sk-ant-…[REDACTED]
afj_ + 43 chars[REDACTED]
sk-, pk-, rk- tokens of 16+ chars[REDACTED]
bearer tokensthe scheme word kept, the token replaced with [REDACTED]
api_key, secret, token, password assignmentsvalue replaced
paths apiKey, ANTHROPIC_API_KEY, env, headers.authorization (and one level nested)[REDACTED]

The gateway never logs headers or bodies; the audit table stores no tokens; subject and exchanged tokens are held in Buffers that are zeroed on delete (JS string copies live until GC, a documented shortcut in subject-token-cache.ts). The sandbox job token reaches the container only through a 0600 env file deleted once the container runs; the worker control token never enters a sandbox, an argv or a log line (in the local shape it is read inside the gateway container by docker exec).

Sources​

  • apps/gateway/src/config.ts, apps/gateway/src/proxy.ts (step 9), apps/gateway/src/identity/subject-token-cache.ts
  • apps/api/src/config.ts, apps/api/src/runtime.ts, apps/worker/src/config.ts, apps/worker/src/runtime.ts, apps/worker/src/exec-config.ts (EXEC_FORBIDDEN, loadExecConfig), apps/worker/src/factory/config.ts (appKey, loadFactoryConfig)
  • apps/worker/src/sandbox/control.ts (dockerExecControl), apps/worker/src/sandbox/sandbox-runtime.ts (env file)
  • packages/agent-workflows/src/codec.ts (parsePayloadKeys, keyFingerprints, AesGcmCodec)
  • packages/agent-core/src/redaction.ts, packages/agent-types/src/gateway-control.ts
  • secrets.manifest.yml, scripts/deploy/make-app03-secrets.sh (header)
  • docs/deploy-app02-test.md (secrets section), docs/deploy-worker-fleet.md, docs/design-worker-fleet.md
  • docs/p2-findings.md ("Production deploy: answers from the fleet sessions", "App-03 production deploy and smoke test")
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/ARCHITECTURE.md, docs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.md
  • docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0001-all-model-calls-through-the-gateway.md, docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0003-temporal-acl-only-and-baked-gateway-policy.md, docs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md