Skip to main content

Secrets flow

This page covers the secrets manifest, the env-sync path from the password manager to the running container, swarm secrets and the _FILE convention, the generator script and its invariants, and rotation. It works at role level: no values, key ids, addresses or hostnames appear here.

The path of a secret​

Rules from the manifest header and the lessons:

  • 1Password is upstream; SSM is what the hostinger apps role delivers to each CapRover app.
  • Items live in vault app-secrets as <source>/<env>, env prod only (for example agentforge/agentforge-api/prod). env-sync appends the env segment to the SSM path, giving /catesworks/<service>/prod/<VAR>.
  • The apps role delivers every SSM parameter under the path, and env-sync pushes every field of an item. Each item therefore holds exactly the keys the service needs, no more. The manifest schema has no key list, so the names are kept as comments in secrets.manifest.yml.
  • Names only in the manifest, never values.

The manifest​

secrets.manifest.yml declares six services, all type: worker, each with one ssm target:

Service1Password sourceSSM pathVariables (names only)
agentforge-postgresapp-secrets/agentforge/agentforge-postgres/catesworks/agentforge-postgresPOSTGRES_DB, POSTGRES_PASSWORD, POSTGRES_USER
agentforge-gatewayapp-secrets/agentforge/agentforge-gateway/catesworks/agentforge-gatewayANTHROPIC_API_KEY, DATABASE_URL, GATEWAY_ALLOWED_FIELDS, GATEWAY_ALLOWED_QUERIES, GATEWAY_CONTROL_BIND_ANY, GATEWAY_CONTROL_HOST, GATEWAY_CONTROL_TOKEN_API, GATEWAY_CONTROL_TOKEN_WORKER, GATEWAY_MAX_TOKENS_CAP, GATEWAY_MODELS
agentforge-workerapp-secrets/agentforge/agentforge-worker/catesworks/agentforge-workerCLIENT_DAILY_LIMIT_USD, DATABASE_URL, GATEWAY_CONTROL_TOKEN_WORKER_FILE, GATEWAY_CONTROL_URL, GATEWAY_HTTP_ALLOWED_HOSTS, GATEWAY_MAX_OUTPUT_TOKENS, GATEWAY_URL, GLOBAL_DAILY_LIMIT_USD, PER_REQUEST_CAP_USD, TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_PAYLOAD_KEY_ID, TEMPORAL_PAYLOAD_KEYS
agentforge-apiapp-secrets/agentforge/agentforge-api/catesworks/agentforge-apiAPI_ADMIN_CLIENTS, API_KEYS, API_PORT, CLIENT_DAILY_LIMIT_USD, DATABASE_URL, GATEWAY_CONTROL_TOKEN_API_FILE, GATEWAY_CONTROL_URL, GATEWAY_HTTP_ALLOWED_HOSTS, GATEWAY_MAX_OUTPUT_TOKENS, GATEWAY_URL, GLOBAL_DAILY_LIMIT_USD, MODELS, PER_REQUEST_CAP_USD, PORT, TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_PAYLOAD_KEY_ID, TEMPORAL_PAYLOAD_KEYS
agentforge-worker-filesapp-secrets/agentforge/agentforge-worker-files/catesworks/agentforge-worker-filesCTL_TOKEN (the worker control token; same value as the gateway's GATEWAY_CONTROL_TOKEN_WORKER)
agentforge-api-filesapp-secrets/agentforge/agentforge-api-files/catesworks/agentforge-api-filesCTL_TOKEN (the api control token; same value as the gateway's GATEWAY_CONTROL_TOKEN_API; the api never gets the worker token)

Who may hold which secret:

Enforced refusals (startup, naming the variable, never its value):

AppRefusesReason
APIANTHROPIC_API_KEY, GATEWAY_CONTROL_TOKEN_WORKER_FILEmodel calls go through the gateway; the api token is scoped to session-readonly jobs
Session workerANTHROPIC_API_KEY, any GITHUB_APP_*FL-CFG1, FX-CFG1
Factory workerANTHROPIC_API_KEYFX-CFG1
Exec workerDATABASE_URL, ANTHROPIC_API_KEY, any GATEWAY_CONTROL_*, any GITHUB_APP_*fleet hosts hold no database, control, model or GitHub credential
Quality exec workerDATABASE_URL, GITHUB_APP_PUSH_*, GATEWAY_CONTROL_*, ANTHROPIC_API_KEY, the production payload key idno credential for anything it does not need

Swarm secrets and _FILE​

The gateway control tokens reach the api and the worker as files, not environment values. The mechanism (from the 2026-11-13 findings, verified in the host repo docs):

  • A swarm_secrets: entry in the catalog is { ssm: /catesworks/<svc>-files/prod/<KEY>, target: <name>, env: <X_FILE> }, one entry per file.
  • Each secret is one file /run/secrets/<target>, flat directory, mode 0400. No subdirectories, no other mount path.
  • The SSM parameter must be a sibling path (/catesworks/<svc>-files/prod/..., never under /catesworks/<svc>/), with a separate <svc>-files item on the 1Password side. That is why the manifest has agentforge-worker-files and agentforge-api-files.
  • The *_FILE variable must exist in the service's own SSM env set with the exact value /run/secrets/<target>; the role asserts it. On the core host: GATEWAY_CONTROL_TOKEN_WORKER_FILE=/run/secrets/agentforge-ctl-worker and GATEWAY_CONTROL_TOKEN_API_FILE=/run/secrets/agentforge-ctl-api.
  • target must not contain a dot (the schema rejects it).
  • SSM Standard caps a value at 4 KB (Advanced 8 KB); env-sync refuses multi-line values.
  • A change needs a new SSM value plus an apps run: a new versioned secret and a rolling update. A plain restart does not pick it up.
  • The owner uid matters. The images run as uid 10001, while the role first mounted secrets as uid 1000 and the first smoke test hit EACCES. Hostinger PR #102 added swarm_secret_user 10001 (ADR 0004).

_FILE conventions in the apps:

VariableHoldsUsed by
GATEWAY_CONTROL_TOKEN_WORKER_FILEpath to a 0400 file with the gateway worker control tokensession worker, factory worker
GATEWAY_CONTROL_TOKEN_API_FILEpath to a 0400 file with the gateway api control tokenAPI
GITHUB_APP_READ_PR_PRIVATE_KEY_FILE, GITHUB_APP_PUSH_PRIVATE_KEY_FILEpath to a GitHub App private key (exactly one of the inline key and _FILE; production uses _FILE, mode 0400)factory worker
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CAfile paths (all three or none)api, workers

Why the gateway policy is not a swarm secret: a policy directory cannot be delivered that way. The loader reads only files matching .yaml, .yml or .json directly in the directory, a secret target cannot contain a dot, and secrets arrive as uid 1000 while the gateway runs as uid 10001. The decision was to bake a default org policy into the image (ADR 0003, dossier 2026-11-13).

The generator script​

scripts/deploy/make-app03-secrets.sh builds the env-sync field directories for the core host stack. It supersedes make-app02-secrets.sh, which is kept only as history.

CommandEffect
make-app03-secrets.sh [DIR]Creates DIR (default ~/.agentforge-secrets/app03-secrets) and writes DIR/<service>/<VAR>; content is the value with no trailing newline, directories 0700, files 0600
make-app03-secrets.sh --check [DIR]Verifies an existing directory

Behavior:

  • It never talks to 1Password, AWS or the network. Output is service names, variable names and counts only. Values reach files through printf (a builtin) and redirection, never argv.
  • The Postgres password (hex, safe in a URL), the Temporal payload key, the control tokens and the owner API key are generated with openssl. ANTHROPIC_API_KEY comes from the environment or a silent prompt on the terminal and goes only into the gateway item.
  • It refuses to write under the repository or /tmp, /private/tmp, /var/tmp, /private/var/tmp, by logical and by physical path, and refuses to overwrite an existing directory.
  • The values it sets on the core host: models claude-haiku-5-5,claude-sonnet-5-5, GATEWAY_MAX_TOKENS_CAP=8192, CLIENT_DAILY_LIMIT_USD=2.00, GLOBAL_DAILY_LIMIT_USD=5.00, worker PER_REQUEST_CAP_USD=2.00, api PER_REQUEST_CAP_USD=0.5, API_ADMIN_CLIENTS=owner, api PORT and API_PORT 3000, namespace agentforge-prod, gateway and database addresses as srv-captain--<service> names.

Invariants the check function enforces:

CheckMeaning
File setexactly the listed keys per service, no empty file, no symlink, nothing unexpected (an extra file would be pushed)
Modesfiles 0600, directories 0700
DATABASE_URL equalin api, worker and gateway
Payload keys equalTEMPORAL_PAYLOAD_KEYS and TEMPORAL_PAYLOAD_KEY_ID in api and worker; TEMPORAL_NAMESPACE equal
Control tokensthe gateway's api token equals agentforge-api-files/CTL_TOKEN; the worker token equals agentforge-worker-files/CTL_TOKEN
Tokens differthe api and worker control tokens must not be equal
URL carries the passwordDATABASE_URL equals the URL built from the Postgres password
Bind guardGATEWAY_CONTROL_BIND_ANY equals private-network-only
No key leakapi and worker must not contain ANTHROPIC_API_KEY

CI runs only bash scripts/deploy/make-app02-secrets.test.sh (the superseded script); there is no CI test for the app-03 script. The open follow-up from the session lessons is to validate the generated key sets against the real config schemas in CI, because a missing set caused two deploy bugs (GATEWAY_ALLOWED_QUERIES and GATEWAY_ALLOWED_FIELDS were in the e2e scripts but not in the generator; GATEWAY_CONTROL_HOST appears only in the Dockerfile as a comment).

Seeding and teardown commands​

Per service, from the repository root, dry run first and read each output (<WRITE_VAR> is the name of the variable holding the write token, never the token):

env-sync-ts secret set-many --service <svc> --env prod --from-dir <DIR>/<svc> --create
env-sync-ts secret set-many --service <svc> --env prod --from-dir <DIR>/<svc> --create --apply \
--confirm app-secrets/agentforge/<svc>/prod --vault-token app-secrets=<WRITE_VAR>
env-sync-ts push --service <svc> --env prod
env-sync-ts push --service <svc> --env prod --apply
env-sync-ts diff --service <svc> --env prod

When every push and diff is clean, destroy the directory (find <DIR> -type f -exec rm -P {} + on macOS, shred -u on Linux, then rm -r <DIR>).

Credential-flow rules that came out of the 2026-11-13 session:

  • Agent sessions on the owner's laptop can load the write token and cloud keys into a throwaway tmux shell, but only with an explicit in-session go, answering each env-sync confirmation after reading the item and field names. The shell is killed afterwards.
  • Never print values. Guard hooks withhold outputs that mention words such as TOKEN or KEY, so filter to field names and counts.
  • An AWS identity in an agent shell may not read the registry account. Use the release run log for digests or the control-node identity.
  • The throwaway test removed 43 SSM parameters (12+9+3+8+11 for api, gateway, postgres, temporal, worker) in five console batches of at most 10 rows, then the owner archived the five 1Password items.

Rotation​

SecretProcedureEffect
Control tokensSet the new value in the gateway item and in the caller's -files item together (api or worker), apply, redeploy bothLive job registrations are lost on the gateway restart; callers re-register on retry
IdP client secret (GATEWAY_IDP_CLIENT_SECRET)Rotate at the IdP, update the secret, redeployCached exchanged and service tokens are in memory only and drop with the restart. The three IdP settings are all or none and https only
Model key (ANTHROPIC_API_KEY)Rotate in the Console, update the gateway item, redeployIn-flight streams finish on the old connection. The owner declined rotation of the current test key unless it can be done much faster than the earlier two-hour pace; it has a spend limit, and the owner will replace it once everything works
Job tokenNot rotated; per turn, short TTLIt is a CLI argument while a turn runs (--mcp-config), so treat argv as sandbox-visible
Postgres passwordNot documented as a procedureUnverified: it appears in DATABASE_URL of three items, so all four items change together
Temporal payload keysNot documented as a procedureUnverified: TEMPORAL_PAYLOAD_KEYS is a map by id and TEMPORAL_PAYLOAD_KEY_ID selects the encrypting key. The api and every worker, including exec workers, must hold the same map

Log scrubbing: a job token (prefix afj_) is replaced by [REDACTED] in logs by the redaction patterns (see Observability and alerts).

Quality exec worker keys​

The quality exec worker uses its own Temporal namespace (default agentforge-quality) and payload key. When the namespace is not agentforge-prod, QUALITY_PROD_PAYLOAD_KEY_ID is required and holds the production key id that startup refuses to hold: if the worker's own key id equals it, startup stops. This keeps the sandbox host from decrypting production payloads.

Sources​

  • secrets.manifest.yml
  • scripts/deploy/make-app03-secrets.sh (including the check function)
  • scripts/deploy/make-app02-secrets.sh, scripts/deploy/make-app02-secrets.test.sh
  • docs/p2-findings.md ("Production deploy: answers from the fleet sessions", sections A and B; "App-02 test teardown")
  • docs/deploy-gateway-coolify.md (section 5, token rotation)
  • docs/deploy-api-coolify.md, docs/deploy-worker-coolify.md, docs/deploy-worker-fleet.md, docs/deploy-factory.md, docs/deploy-quality.md
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.md, FOLLOWUPS.md
  • docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/LESSONS.md
  • docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0003-temporal-acl-only-and-baked-gateway-policy.md
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md
  • .github/workflows/ci.yml