Observability and alerts
AgentForge has no metrics stack or dashboards in the repository. Its observability is structured logs (pino), three read endpoints, a ledger-based spend report, one webhook alert, an audit table in Postgres and a kill switch. This page lists each signal, its exact fields and what is not wired.
Signals at a glance
Logs
All services use pino. createLogger in packages/agent-core/src/redaction.ts applies two layers to every line:
- Key-based
redactwith censor[REDACTED]on the pathsapiKey,*.apiKey,ANTHROPIC_API_KEY,*.ANTHROPIC_API_KEY,env,*.env,headers.authorization,*.headers.authorization. - Value scrubbing of every logged string (
scrubDeep, cycle-safe, depth 64) with these patterns:
| Pattern | Replaced with |
|---|---|
Provider API keys: the sk, dash, ant, dash prefix followed by key characters | [REDACTED] |
afj_[A-Za-z0-9_-]{43} (gateway job tokens) | [REDACTED] |
sk-, pk- or rk- followed by 16 or more token characters | [REDACTED] |
| the word bearer (any case) followed by a token | the word Bearer followed by [REDACTED] |
api_key, secret, token or password assignments (key=value or key: value) | the key and separator, then [REDACTED] |
logError logs only the scrubbed error text, with no stack and no raw object. Practical consequence: a probe file or test fixture that contains TOKEN= text is hidden by the redaction rule, so use another form.
API log lines
| Message | Level | Fields |
|---|---|---|
request | info | requestId, client, method, path, httpStatus, status (runtime status), costUsd, durationMs. One line per request; never headers, prompts or replies |
listening | info | port |
session config | info | temporal, namespace, taskQueue, tls, heartbeatIntervalMs, codecKeys (key fingerprints). Must match the worker's worker config line |
TRUST_PROXY_HOPS=0: ... | warn | host, nodeEnv. Appears when the value is 0 in production or on a non-loopback host; must be absent after the first deploy |
stale claude homes swept | info | removed |
agent run failed | warn | requestId, status, text |
stream ended | info | requestId, status, costUsd |
stream failed, unhandled error | error | scrubbed message, requestId |
temporal call settled after the deadline | warn | call, ok, with requestId, sessionId |
orphaned pending message re-signalled | info | messageId |
stale running turn: session closed | warn | messageId |
shutting down, stopped | info | signal, drained |
The request id has the form api/<uuid> and is returned in the x-request-id response header.
Gateway log lines
| Message | Fields |
|---|---|
gateway listening | dataHost, dataPort, controlHost, controlPort, models, allowlistsFrozen |
shutting down, stopped | signal, drained |
Worker log lines
| Message | Notes |
|---|---|
worker config | fingerprints, heartbeat interval, namespace, task queue, tls |
worker running | with alsoExec and quality flags |
worker stopped | on shutdown |
exec worker running, exec worker stopped | the fleet exec worker |
plaintext connection to Temporal (TEMPORAL_ALLOW_PLAINTEXT) | warn, exec worker without TLS |
TEMPORAL_TLS_* unset: plaintext connection to Temporal | warn, core worker |
Startup comparison to do after every deploy: the API's session config and the worker's worker config must show the same namespace, task queue, heartbeat interval and codecKeys fingerprints.
Health endpoints
| Endpoint | Auth | Returns | Meaning |
|---|---|---|---|
GET /health on the API | public | {"status":"ok"} | liveness only; touches no dependency |
GET /health on the gateway data port | none | exactly {"status":"ok"} | liveness |
GET /control/v1/health on the gateway control port | worker control token | {status, auditWriteFailures, preAuthDenials} | auditWriteFailures > 0 means audit rows are being lost while requests are still answered; preAuthDenials counts pre-auth denials |
Workers serve no port: their Docker HEALTHCHECK is a process liveness check (node -e process.exit(0), every 30 s), so a healthy worker container says nothing about Temporal connectivity. Check the worker running log line instead.
The usage endpoint
GET /v1/usage is the spend report. It needs a key whose client is in API_ADMIN_CLIENTS; any other client gets 404 (not_found), not 403.
Response (UsageResponse):
| Field | Meaning |
|---|---|
day | UTC day YYYY-MM-DD |
clients[] | client, spentUsd, inputTokens, cacheReadTokens, cacheWriteTokens, cacheHitRate (cacheRead / (input + cacheRead + cacheWrite), null when the denominator is 0) |
globalSpentUsd | total spend of all clients for the day |
staleReservations[] | requestId, client, amountUsd, createdAt: ledger rows still reserved longer than STALE_RESERVATION_SEC |
curl -fsS --oauth2-bearer "$ADMIN_KEY" "$API/v1/usage"
After a clean smoke run the expected state is owner spend equal to the sum of the turns and an empty staleReservations.
Stale reservations
A reservation counts in full against spend until settled. The stale check only reports; it never releases rows.
| Setting | Default | Limits |
|---|---|---|
STALE_RESERVATION_SEC | 900 | minimum 600 |
STALE_CHECK_INTERVAL_MS | 300000 |
Every interval the API logs one warn line (stale reservations) while the list is non-empty, with event: stale_reservations, count, oldestRequestId, oldestAgeSec, totalReservedUsd. It logs nothing when empty, and one error line (stale reservation check failed) on a query error.
Abandon (settling a stale row as aborted at its full reservation) is a workflow step before a retry, not an operator command.
Spend alerts
createAlertListener in packages/agent-core/src/alerts.ts runs after each settle.
| Property | Value |
|---|---|
| Thresholds | ALERT_THRESHOLDS = [0.5, 0.8] of GLOBAL_DAILY_LIMIT_USD |
| Log line | warn: global daily spend crossed alert threshold with threshold, spendUsd, limitUsd |
| Webhook | ALERT_WEBHOOK_URL (optional, URL), POST with content-type: application/json and body {threshold, spendUsd, limitUsd} |
| Timeout | 5 s |
| Delivery | at most once per threshold per UTC day; dedup is in alerts_sent (primary key day, threshold), so it survives restarts |
| Failure | the threshold is marked sent before the POST, so a failed or timed-out POST is not retried that day; the error log line is the only signal |
| Blocking | detached; a slow or failing webhook never delays or fails the request |
Monitor these log messages: alert webhook failed (network error or timeout) and alert webhook rejected (non-2xx, with status and threshold).
The quality runbook says alerts for the daily global sandbox limit at 80 percent and for any control failure go through ALERT_WEBHOOK_URL; that is Unverified and not wired yet.
The kill switch
The kill switch refuses every new request with disabled. It is on when either source is on, and it is treated as on when it cannot be read (fail closed).
| Source | How to set | Notes |
|---|---|---|
Config flag AGENTS_DISABLED | environment of api, worker, factory worker | parsed fail closed: unset means enabled; any set value other than 0 or false (trimmed, case-insensitive) disables agents. The runbooks set false to enable |
| Persisted setting | settings row with key = 'agents_disabled', value 1 or true (case-insensitive) disables; any other value or no row enables | read on every request, so it works without a redeploy |
Setting it from the database:
INSERT INTO settings (key, value) VALUES ('agents_disabled', '1')
ON CONFLICT (key) DO UPDATE SET value = '1';
The SQL above is built from the table definition (key primary key, value not null) and stores.ts; it has not been run against a real database in this documentation pass.
Behavior by component:
| Component | On disabled |
|---|---|
| Guarded runtime (api, worker) | writes a ledger row with status rejected_disabled, yields a terminal result disabled; the kill switch runs before admission, so a disabled request never reserves |
| Session turn | ends failed with errorCode: disabled and no spend; kill-switched or aborted turns may book up to 2.00 USD against the daily cap (over-count only) |
| Gateway data path | answers 503 (GV3 probe b) |
| Factory | disabled is a FactoryErrorCode |
| Quality | unsetting QUALITY_ENABLED makes the routes answer 404 and registers nothing; the database kill switch also stops admission and chunks |
The deployed kill-switch check (EXPECT_DISABLED=1 in the smoke script, which needs AGENTS_DISABLED=true in SSM and a redeploy) was never run on the deployed stack.
The other cost-containment levers, in order of speed: the gateway revoke path (DELETE /control/v1/jobs/:requestId kills a registration; live job tokens are in memory and a gateway restart drops them all), the daily limits (client 2.00 and global 5.00 USD on the core host), and the model key's Console spend limit.
Gateway audit
Every gateway decision writes a gateway_audit row (evidence only: no prompts, bodies, tokens or headers).
| Column | Meaning |
|---|---|
at, job_id, request_id, subject_id | when and for whom (subject_id is the IdP sub, never a token) |
target | model id or gateway tool name |
args_sha256 | hash of tool arguments, never the arguments |
kind | model, tool, context, control |
decision | allow, deny |
identity | obo, service, none |
reason | null only for allow model or tool rows; otherwise one of the audit reasons below |
http_status, input_tokens, output_tokens, latency_ms, cost_usd | outcome |
Audit reasons (AuditReason): field_not_allowed, content_not_allowed, content_type_not_allowed, body_too_large, invalid_json, model_not_allowed, max_tokens_invalid, thinking_budget_invalid, tool_type_not_allowed, beta_not_allowed, header_invalid, budget_exhausted, registration_stale, disabled, ledger_error, busy, rate_limited, upstream_error, upstream_bad_content_type, upstream_timeout, client_aborted, tool_not_granted, args_too_large, args_invalid, exchange_denied, schema_changed, result_too_large, server_request_denied, content_stripped, method_not_found, registered, revoked, registration_conflict, registration_unknown, context_recorded.
Useful operator queries (assembled from the schema; not run in this pass):
SELECT reason, count(*) FROM gateway_audit WHERE decision = 'deny' AND at > now() - interval '1 day' GROUP BY reason ORDER BY 2 DESC;
SELECT sum(cost_usd) FROM gateway_audit WHERE kind = 'model' AND decision = 'allow' AND at > now() - interval '1 day';
In the smoke runs the first failed turn was diagnosed from denials of ?beta=true and unlisted fields before authentication: see the gateway allowlist variables in Deploy runbooks. Retention is a manual 90-day delete (Persistence and backups).
Gateway rate-limit and denial counters in GET /control/v1/health are cumulative since process start and reset on a restart.
What is not wired
| Gap | Source |
|---|---|
| No uptime monitor, metrics or dashboards for AgentForge | not present in the repository |
Alerting and log rotation for the pg_dump log | docs/p2-findings.md |
Quality alerts through ALERT_WEBHOOK_URL | docs/deploy-quality.md ("not wired yet") |
| Worker liveness checks only prove the process | Dockerfile HEALTHCHECK |
/v1/usage has no history, only today (UTC) | spendByClientToday |
| Log shipping and retention | not configured in the repository; the hosts' container logs are the only copy (Unverified) |
Sources
packages/agent-core/src/redaction.ts(redact,SECRET_PATTERNS,createLogger,logError)packages/agent-core/src/alerts.ts,packages/agent-core/README.md("Spend alerts")packages/agent-core/src/kill-switch.tspackages/db/src/stores.ts(isDisabled),packages/db/src/schema.ts(settings,alertsSent,gatewayAudit)packages/agent-types/src/audit.ts,packages/agent-types/src/http.ts(UsageResponse)apps/api/src/app.ts(request log,/health,/v1/models,/v1/usage),apps/api/src/serve.ts,apps/api/src/stale.tsapps/gateway/src/serve.ts,apps/gateway/src/control.ts,apps/gateway/src/proxy.tsapps/worker/src/worker.ts,apps/worker/src/exec-main.ts,apps/worker/src/activities.tsdocs/deploy-api-coolify.md,docs/deploy-gateway-coolify.md,docs/deploy-quality.md,docs/deploy-app02-test.md(smoke and kill-switch steps)docs/p2-findings.md("App-03 Postgres volume restart proof and nightly pg_dump")scripts/deploy/smoke-app02.sh,scripts/gv3-standin-probe.mjs