Secrets and keys
This page names every secret in the system, who holds it, how it is delivered and which process refuses it. It contains no values. Operational steps (seeding, rotation commands) are in Secrets flow.
Inventory
| Secret | Holder(s) | Variable | Delivery | Refused by |
|---|---|---|---|---|
| Model provider key | gateway only | ANTHROPIC_API_KEY | env | api, core worker, factory worker, exec worker (startup error) |
| Gateway control token, api route | gateway; api | gateway: GATEWAY_CONTROL_TOKEN_API (env); api: GATEWAY_CONTROL_TOKEN_API_FILE | gateway env; api swarm-secret file | the api refuses GATEWAY_CONTROL_TOKEN_WORKER_FILE; exec worker refuses any GATEWAY_CONTROL_* |
| Gateway control token, worker route | gateway; core worker; factory worker | gateway: GATEWAY_CONTROL_TOKEN_WORKER (env); workers: GATEWAY_CONTROL_TOKEN_WORKER_FILE | gateway env; worker swarm-secret file | exec worker |
Job token (afj_…) | gateway (hash only); one runtime for one turn | sandbox: ANTHROPIC_API_KEY inside the container env file; exec: activity input | minted per turn | expires with the turn |
| Temporal payload keys | api, core worker, factory worker, exec workers | TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID | env (fleet: 1Password item via op inject) | none (required) |
| Database credential | api, core worker, factory worker, gateway | DATABASE_URL | env | exec worker |
| IdP client secret (OBO, service identity) | gateway | GATEWAY_IDP_CLIENT_SECRET (with GATEWAY_IDP_TOKEN_URL, GATEWAY_IDP_CLIENT_ID) | env | none; unset in production (no IdP yet) |
| GitHub App private keys | factory worker | GITHUB_APP_READ_PR_PRIVATE_KEY[_FILE], GITHUB_APP_PUSH_PRIVATE_KEY[_FILE] | mounted 0400 file in production | exec worker refuses GITHUB_APP_* |
| API client keys | api | API_KEYS | env | |
| Postgres password | postgres service | POSTGRES_PASSWORD | env |
The gateway holds the only model key
The owner decision behind this (ADR 0001 of the 2026-11-13 session) is that laptops and VPSes may run agent workers over the tailnet without the key or the database credential being copied to every machine. Enforcement in code:
| Process | Check | Error text |
|---|---|---|
| api | ANTHROPIC_API_KEY present at startup | "the api never holds an Anthropic key (FL-CFG1)" |
| session worker | same | "the session worker never holds an Anthropic key (FL-CFG1)" |
| factory worker | same | "the factory worker never holds an Anthropic key (FX-CFG1)" |
| exec worker | any of DATABASE_URL, ANTHROPIC_API_KEY, GATEWAY_CONTROL_*, GITHUB_APP_* | "the exec worker holds no database, control-token, Anthropic or GitHub credential (FL-3)" |
The gateway sends the key upstream only as x-api-key to the fixed upstream URL, with redirect: 'error', and refuses
startup settings that would let a proxy or extra CA see it (NODE_TLS_REJECT_UNAUTHORIZED=0, NODE_USE_ENV_PROXY,
NODE_EXTRA_CA_CERTS, SSL_CERT_FILE, SSL_CERT_DIR, NODE_OPTIONS without their explicit allow flags). See
Gateway.
Residual: gateway compromise equals key compromise (P2.G plan section 11). The key on the provider side has a spend
limit; the owner decided not to rotate the earlier test key until the gateway path was live (docs/p2-findings.md).
The _FILE convention
Secrets that must not sit in a process environment visible to other tools are read from a file whose path is in a
variable ending in _FILE:
| Variable | Reader | Behaviour |
|---|---|---|
GATEWAY_CONTROL_TOKEN_API_FILE | apps/api/src/runtime.ts | readFileSync(...).trim(); empty file is a startup error |
GATEWAY_CONTROL_TOKEN_WORKER_FILE | apps/worker/src/runtime.ts, factory config | same |
GITHUB_APP_READ_PR_PRIVATE_KEY_FILE, GITHUB_APP_PUSH_PRIVATE_KEY_FILE | apps/worker/src/factory/config.ts appKey | exactly one of the inline variable and the _FILE variant must be set |
In production the control-token files are swarm secrets: each secret is one flat file under /run/secrets/<target>,
mode 0400, delivered from a sibling SSM path /catesworks/<service>-files/prod/<KEY> (targets agentforge-ctl-api and
agentforge-ctl-worker, key CTL_TOKEN). Each file holds the same value as the matching gateway env variable; the api
never gets the worker token. The gateway reads its own control tokens and the model key from env.
Lesson from the app-03 deploy: the images run as uid 10001 while the role first mounted secrets as uid 1000, which gave
EACCES; the secret uid must match the image (hostinger PR #102, ADR 0004).
Policy files are not secrets and are not delivered as swarm secrets: swarm secrets are flat, extensionless and
owned by another uid, while the policy loader needs *.json or *.yaml names. The default policies are baked into the
gateway image instead (ADR 0003).
Temporal payload codec keys
Prompts, transcripts and job tokens cross Temporal inside activity inputs and results. The cluster is plaintext and
reachable by anything on the tailnet ACL, so payloads are encrypted at the application level
(packages/agent-workflows/src/codec.ts):
| Property | Value |
|---|---|
| Cipher | AES-256-GCM (node:crypto), 12-byte random IV per payload, 16-byte tag |
| Envelope | the whole payload (metadata and data) sealed as JSON; data is the IV, then the ciphertext, then the tag |
| Encoding marker | metadata encoding = binary/encrypted, encryption-key-id = <keyId> |
| Key format | TEMPORAL_PAYLOAD_KEYS = JSON object mapping key id to base64 of 32 bytes |
| Current key | TEMPORAL_PAYLOAD_KEY_ID must name a key in the map |
| Rotation | add a new id, switch TEMPORAL_PAYLOAD_KEY_ID; every key in the map can still decode |
| Decode | unknown key id or a tampered payload throws (GCM authentication); non-encrypted payloads pass through |
| Startup log | keyFingerprints: per id, the first 12 hex chars of sha256 of the key (a consistency check, not secret) |
Parse errors never contain key material. Every role that runs a Temporal client or worker (api, core worker, factory worker, exec worker) needs the same key map.
Delivery flow (roles)
Rules recorded in the deploy docs:
secrets.manifest.yml(repo root) declares the env-sync services:agentforge-postgres,agentforge-gateway,agentforge-worker,agentforge-api,agentforge-worker-files,agentforge-api-files. It holds names only.- The env segment is
prodonly; atestsegment would also be delivered and collide. - env-sync's prod gate (a TTY or
--confirm) applies to every write; the write token is passed by variable name and never read by an agent. scripts/deploy/make-app03-secrets.shwrites values throughprintfand redirection, never argv; it refuses the repo and shared temp dirs as output locations; it never talks to 1Password, AWS or the network.- The fleet item
agentforge/worker-fleetholds payload keys,GATEWAY_URL,TEMPORAL_ADDRESSand per-host certs if any; never a database credential, control token, model key or GitHub key.
Unverified: the env-sync commands in docs/deploy-app02-test.md are marked UNVERIFIED there; real IdP secrets do not
exist yet.
Logging and redaction
packages/agent-core/src/redaction.ts scrubs every logged string (pino logMethod hook) and redacts known paths:
| Pattern or path | Replacement |
|---|---|
sk-ant-… | [REDACTED] |
afj_ + 43 chars | [REDACTED] |
sk-, pk-, rk- tokens of 16+ chars | [REDACTED] |
| bearer tokens | the scheme word kept, the token replaced with [REDACTED] |
api_key, secret, token, password assignments | value replaced |
paths apiKey, ANTHROPIC_API_KEY, env, headers.authorization (and one level nested) | [REDACTED] |
The gateway never logs headers or bodies; the audit table stores no tokens; subject and exchanged tokens are held in
Buffers that are zeroed on delete (JS string copies live until GC, a documented shortcut in
subject-token-cache.ts). The sandbox job token reaches the container only through a 0600 env file deleted once the
container runs; the worker control token never enters a sandbox, an argv or a log line (in the local shape it is read
inside the gateway container by docker exec).
Sources
apps/gateway/src/config.ts,apps/gateway/src/proxy.ts(step 9),apps/gateway/src/identity/subject-token-cache.tsapps/api/src/config.ts,apps/api/src/runtime.ts,apps/worker/src/config.ts,apps/worker/src/runtime.ts,apps/worker/src/exec-config.ts(EXEC_FORBIDDEN,loadExecConfig),apps/worker/src/factory/config.ts(appKey,loadFactoryConfig)apps/worker/src/sandbox/control.ts(dockerExecControl),apps/worker/src/sandbox/sandbox-runtime.ts(env file)packages/agent-workflows/src/codec.ts(parsePayloadKeys,keyFingerprints,AesGcmCodec)packages/agent-core/src/redaction.ts,packages/agent-types/src/gateway-control.tssecrets.manifest.yml,scripts/deploy/make-app03-secrets.sh(header)docs/deploy-app02-test.md(secrets section),docs/deploy-worker-fleet.md,docs/design-worker-fleet.mddocs/p2-findings.md("Production deploy: answers from the fleet sessions", "App-03 production deploy and smoke test")docs/sessions/2026-11-15-app03-deploy-and-smoke/ARCHITECTURE.md,docs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.mddocs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0001-all-model-calls-through-the-gateway.md,docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0003-temporal-acl-only-and-baked-gateway-policy.md,docs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md