Secrets flow
This page covers the secrets manifest, the env-sync path from the password manager to the running container, swarm secrets and the _FILE convention, the generator script and its invariants, and rotation. It works at role level: no values, key ids, addresses or hostnames appear here.
The path of a secret
Rules from the manifest header and the lessons:
- 1Password is upstream; SSM is what the hostinger
appsrole delivers to each CapRover app. - Items live in vault
app-secretsas<source>/<env>, envprodonly (for exampleagentforge/agentforge-api/prod). env-sync appends the env segment to the SSM path, giving/catesworks/<service>/prod/<VAR>. - The apps role delivers every SSM parameter under the path, and env-sync pushes every field of an item. Each item therefore holds exactly the keys the service needs, no more. The manifest schema has no key list, so the names are kept as comments in
secrets.manifest.yml. - Names only in the manifest, never values.
The manifest
secrets.manifest.yml declares six services, all type: worker, each with one ssm target:
| Service | 1Password source | SSM path | Variables (names only) |
|---|---|---|---|
agentforge-postgres | app-secrets/agentforge/agentforge-postgres | /catesworks/agentforge-postgres | POSTGRES_DB, POSTGRES_PASSWORD, POSTGRES_USER |
agentforge-gateway | app-secrets/agentforge/agentforge-gateway | /catesworks/agentforge-gateway | ANTHROPIC_API_KEY, DATABASE_URL, GATEWAY_ALLOWED_FIELDS, GATEWAY_ALLOWED_QUERIES, GATEWAY_CONTROL_BIND_ANY, GATEWAY_CONTROL_HOST, GATEWAY_CONTROL_TOKEN_API, GATEWAY_CONTROL_TOKEN_WORKER, GATEWAY_MAX_TOKENS_CAP, GATEWAY_MODELS |
agentforge-worker | app-secrets/agentforge/agentforge-worker | /catesworks/agentforge-worker | CLIENT_DAILY_LIMIT_USD, DATABASE_URL, GATEWAY_CONTROL_TOKEN_WORKER_FILE, GATEWAY_CONTROL_URL, GATEWAY_HTTP_ALLOWED_HOSTS, GATEWAY_MAX_OUTPUT_TOKENS, GATEWAY_URL, GLOBAL_DAILY_LIMIT_USD, PER_REQUEST_CAP_USD, TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_PAYLOAD_KEY_ID, TEMPORAL_PAYLOAD_KEYS |
agentforge-api | app-secrets/agentforge/agentforge-api | /catesworks/agentforge-api | API_ADMIN_CLIENTS, API_KEYS, API_PORT, CLIENT_DAILY_LIMIT_USD, DATABASE_URL, GATEWAY_CONTROL_TOKEN_API_FILE, GATEWAY_CONTROL_URL, GATEWAY_HTTP_ALLOWED_HOSTS, GATEWAY_MAX_OUTPUT_TOKENS, GATEWAY_URL, GLOBAL_DAILY_LIMIT_USD, MODELS, PER_REQUEST_CAP_USD, PORT, TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_PAYLOAD_KEY_ID, TEMPORAL_PAYLOAD_KEYS |
agentforge-worker-files | app-secrets/agentforge/agentforge-worker-files | /catesworks/agentforge-worker-files | CTL_TOKEN (the worker control token; same value as the gateway's GATEWAY_CONTROL_TOKEN_WORKER) |
agentforge-api-files | app-secrets/agentforge/agentforge-api-files | /catesworks/agentforge-api-files | CTL_TOKEN (the api control token; same value as the gateway's GATEWAY_CONTROL_TOKEN_API; the api never gets the worker token) |
Who may hold which secret:
Enforced refusals (startup, naming the variable, never its value):
| App | Refuses | Reason |
|---|---|---|
| API | ANTHROPIC_API_KEY, GATEWAY_CONTROL_TOKEN_WORKER_FILE | model calls go through the gateway; the api token is scoped to session-readonly jobs |
| Session worker | ANTHROPIC_API_KEY, any GITHUB_APP_* | FL-CFG1, FX-CFG1 |
| Factory worker | ANTHROPIC_API_KEY | FX-CFG1 |
| Exec worker | DATABASE_URL, ANTHROPIC_API_KEY, any GATEWAY_CONTROL_*, any GITHUB_APP_* | fleet hosts hold no database, control, model or GitHub credential |
| Quality exec worker | DATABASE_URL, GITHUB_APP_PUSH_*, GATEWAY_CONTROL_*, ANTHROPIC_API_KEY, the production payload key id | no credential for anything it does not need |
Swarm secrets and _FILE
The gateway control tokens reach the api and the worker as files, not environment values. The mechanism (from the 2026-11-13 findings, verified in the host repo docs):
- A
swarm_secrets:entry in the catalog is{ ssm: /catesworks/<svc>-files/prod/<KEY>, target: <name>, env: <X_FILE> }, one entry per file. - Each secret is one file
/run/secrets/<target>, flat directory, mode 0400. No subdirectories, no other mount path. - The SSM parameter must be a sibling path (
/catesworks/<svc>-files/prod/..., never under/catesworks/<svc>/), with a separate<svc>-filesitem on the 1Password side. That is why the manifest hasagentforge-worker-filesandagentforge-api-files. - The
*_FILEvariable must exist in the service's own SSM env set with the exact value/run/secrets/<target>; the role asserts it. On the core host:GATEWAY_CONTROL_TOKEN_WORKER_FILE=/run/secrets/agentforge-ctl-workerandGATEWAY_CONTROL_TOKEN_API_FILE=/run/secrets/agentforge-ctl-api. targetmust not contain a dot (the schema rejects it).- SSM Standard caps a value at 4 KB (Advanced 8 KB); env-sync refuses multi-line values.
- A change needs a new SSM value plus an apps run: a new versioned secret and a rolling update. A plain restart does not pick it up.
- The owner uid matters. The images run as uid 10001, while the role first mounted secrets as uid 1000 and the first smoke test hit
EACCES. Hostinger PR #102 addedswarm_secret_user10001 (ADR 0004).
_FILE conventions in the apps:
| Variable | Holds | Used by |
|---|---|---|
GATEWAY_CONTROL_TOKEN_WORKER_FILE | path to a 0400 file with the gateway worker control token | session worker, factory worker |
GATEWAY_CONTROL_TOKEN_API_FILE | path to a 0400 file with the gateway api control token | API |
GITHUB_APP_READ_PR_PRIVATE_KEY_FILE, GITHUB_APP_PUSH_PRIVATE_KEY_FILE | path to a GitHub App private key (exactly one of the inline key and _FILE; production uses _FILE, mode 0400) | factory worker |
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA | file paths (all three or none) | api, workers |
Why the gateway policy is not a swarm secret: a policy directory cannot be delivered that way. The loader reads only files matching .yaml, .yml or .json directly in the directory, a secret target cannot contain a dot, and secrets arrive as uid 1000 while the gateway runs as uid 10001. The decision was to bake a default org policy into the image (ADR 0003, dossier 2026-11-13).
The generator script
scripts/deploy/make-app03-secrets.sh builds the env-sync field directories for the core host stack. It supersedes make-app02-secrets.sh, which is kept only as history.
| Command | Effect |
|---|---|
make-app03-secrets.sh [DIR] | Creates DIR (default ~/.agentforge-secrets/app03-secrets) and writes DIR/<service>/<VAR>; content is the value with no trailing newline, directories 0700, files 0600 |
make-app03-secrets.sh --check [DIR] | Verifies an existing directory |
Behavior:
- It never talks to 1Password, AWS or the network. Output is service names, variable names and counts only. Values reach files through
printf(a builtin) and redirection, never argv. - The Postgres password (hex, safe in a URL), the Temporal payload key, the control tokens and the owner API key are generated with
openssl.ANTHROPIC_API_KEYcomes from the environment or a silent prompt on the terminal and goes only into the gateway item. - It refuses to write under the repository or
/tmp,/private/tmp,/var/tmp,/private/var/tmp, by logical and by physical path, and refuses to overwrite an existing directory. - The values it sets on the core host: models
claude-haiku-5-5,claude-sonnet-5-5,GATEWAY_MAX_TOKENS_CAP=8192,CLIENT_DAILY_LIMIT_USD=2.00,GLOBAL_DAILY_LIMIT_USD=5.00, workerPER_REQUEST_CAP_USD=2.00, apiPER_REQUEST_CAP_USD=0.5,API_ADMIN_CLIENTS=owner, apiPORTandAPI_PORT3000, namespaceagentforge-prod, gateway and database addresses assrv-captain--<service>names.
Invariants the check function enforces:
| Check | Meaning |
|---|---|
| File set | exactly the listed keys per service, no empty file, no symlink, nothing unexpected (an extra file would be pushed) |
| Modes | files 0600, directories 0700 |
DATABASE_URL equal | in api, worker and gateway |
| Payload keys equal | TEMPORAL_PAYLOAD_KEYS and TEMPORAL_PAYLOAD_KEY_ID in api and worker; TEMPORAL_NAMESPACE equal |
| Control tokens | the gateway's api token equals agentforge-api-files/CTL_TOKEN; the worker token equals agentforge-worker-files/CTL_TOKEN |
| Tokens differ | the api and worker control tokens must not be equal |
| URL carries the password | DATABASE_URL equals the URL built from the Postgres password |
| Bind guard | GATEWAY_CONTROL_BIND_ANY equals private-network-only |
| No key leak | api and worker must not contain ANTHROPIC_API_KEY |
CI runs only bash scripts/deploy/make-app02-secrets.test.sh (the superseded script); there is no CI test for the app-03 script. The open follow-up from the session lessons is to validate the generated key sets against the real config schemas in CI, because a missing set caused two deploy bugs (GATEWAY_ALLOWED_QUERIES and GATEWAY_ALLOWED_FIELDS were in the e2e scripts but not in the generator; GATEWAY_CONTROL_HOST appears only in the Dockerfile as a comment).
Seeding and teardown commands
Per service, from the repository root, dry run first and read each output (<WRITE_VAR> is the name of the variable holding the write token, never the token):
env-sync-ts secret set-many --service <svc> --env prod --from-dir <DIR>/<svc> --create
env-sync-ts secret set-many --service <svc> --env prod --from-dir <DIR>/<svc> --create --apply \
--confirm app-secrets/agentforge/<svc>/prod --vault-token app-secrets=<WRITE_VAR>
env-sync-ts push --service <svc> --env prod
env-sync-ts push --service <svc> --env prod --apply
env-sync-ts diff --service <svc> --env prod
When every push and diff is clean, destroy the directory (find <DIR> -type f -exec rm -P {} + on macOS, shred -u on Linux, then rm -r <DIR>).
Credential-flow rules that came out of the 2026-11-13 session:
- Agent sessions on the owner's laptop can load the write token and cloud keys into a throwaway tmux shell, but only with an explicit in-session go, answering each env-sync confirmation after reading the item and field names. The shell is killed afterwards.
- Never print values. Guard hooks withhold outputs that mention words such as TOKEN or KEY, so filter to field names and counts.
- An AWS identity in an agent shell may not read the registry account. Use the release run log for digests or the control-node identity.
- The throwaway test removed 43 SSM parameters (12+9+3+8+11 for api, gateway, postgres, temporal, worker) in five console batches of at most 10 rows, then the owner archived the five 1Password items.
Rotation
| Secret | Procedure | Effect |
|---|---|---|
| Control tokens | Set the new value in the gateway item and in the caller's -files item together (api or worker), apply, redeploy both | Live job registrations are lost on the gateway restart; callers re-register on retry |
IdP client secret (GATEWAY_IDP_CLIENT_SECRET) | Rotate at the IdP, update the secret, redeploy | Cached exchanged and service tokens are in memory only and drop with the restart. The three IdP settings are all or none and https only |
Model key (ANTHROPIC_API_KEY) | Rotate in the Console, update the gateway item, redeploy | In-flight streams finish on the old connection. The owner declined rotation of the current test key unless it can be done much faster than the earlier two-hour pace; it has a spend limit, and the owner will replace it once everything works |
| Job token | Not rotated; per turn, short TTL | It is a CLI argument while a turn runs (--mcp-config), so treat argv as sandbox-visible |
| Postgres password | Not documented as a procedure | Unverified: it appears in DATABASE_URL of three items, so all four items change together |
| Temporal payload keys | Not documented as a procedure | Unverified: TEMPORAL_PAYLOAD_KEYS is a map by id and TEMPORAL_PAYLOAD_KEY_ID selects the encrypting key. The api and every worker, including exec workers, must hold the same map |
Log scrubbing: a job token (prefix afj_) is replaced by [REDACTED] in logs by the redaction patterns (see Observability and alerts).
Quality exec worker keys
The quality exec worker uses its own Temporal namespace (default agentforge-quality) and payload key. When the namespace is not agentforge-prod, QUALITY_PROD_PAYLOAD_KEY_ID is required and holds the production key id that startup refuses to hold: if the worker's own key id equals it, startup stops. This keeps the sandbox host from decrypting production payloads.
Sources
secrets.manifest.ymlscripts/deploy/make-app03-secrets.sh(including thecheckfunction)scripts/deploy/make-app02-secrets.sh,scripts/deploy/make-app02-secrets.test.shdocs/p2-findings.md("Production deploy: answers from the fleet sessions", sections A and B; "App-02 test teardown")docs/deploy-gateway-coolify.md(section 5, token rotation)docs/deploy-api-coolify.md,docs/deploy-worker-coolify.md,docs/deploy-worker-fleet.md,docs/deploy-factory.md,docs/deploy-quality.mddocs/sessions/2026-11-15-app03-deploy-and-smoke/LESSONS.md,FOLLOWUPS.mddocs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/LESSONS.mddocs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/adr/0003-temporal-acl-only-and-baked-gateway-policy.mddocs/sessions/2026-11-15-app03-deploy-and-smoke/adr/0004-app03-dedicated-agentforge-host.md.github/workflows/ci.yml