Skip to main content

Observability and alerts

AgentForge has no metrics stack or dashboards in the repository. Its observability is structured logs (pino), three read endpoints, a ledger-based spend report, one webhook alert, an audit table in Postgres and a kill switch. This page lists each signal, its exact fields and what is not wired.

Signals at a glance​

Logs​

All services use pino. createLogger in packages/agent-core/src/redaction.ts applies two layers to every line:

  1. Key-based redact with censor [REDACTED] on the paths apiKey, *.apiKey, ANTHROPIC_API_KEY, *.ANTHROPIC_API_KEY, env, *.env, headers.authorization, *.headers.authorization.
  2. Value scrubbing of every logged string (scrubDeep, cycle-safe, depth 64) with these patterns:
PatternReplaced with
Provider API keys: the sk, dash, ant, dash prefix followed by key characters[REDACTED]
afj_[A-Za-z0-9_-]{43} (gateway job tokens)[REDACTED]
sk-, pk- or rk- followed by 16 or more token characters[REDACTED]
the word bearer (any case) followed by a tokenthe word Bearer followed by [REDACTED]
api_key, secret, token or password assignments (key=value or key: value)the key and separator, then [REDACTED]

logError logs only the scrubbed error text, with no stack and no raw object. Practical consequence: a probe file or test fixture that contains TOKEN= text is hidden by the redaction rule, so use another form.

API log lines​

MessageLevelFields
requestinforequestId, client, method, path, httpStatus, status (runtime status), costUsd, durationMs. One line per request; never headers, prompts or replies
listeninginfoport
session configinfotemporal, namespace, taskQueue, tls, heartbeatIntervalMs, codecKeys (key fingerprints). Must match the worker's worker config line
TRUST_PROXY_HOPS=0: ...warnhost, nodeEnv. Appears when the value is 0 in production or on a non-loopback host; must be absent after the first deploy
stale claude homes sweptinforemoved
agent run failedwarnrequestId, status, text
stream endedinforequestId, status, costUsd
stream failed, unhandled errorerrorscrubbed message, requestId
temporal call settled after the deadlinewarncall, ok, with requestId, sessionId
orphaned pending message re-signalledinfomessageId
stale running turn: session closedwarnmessageId
shutting down, stoppedinfosignal, drained

The request id has the form api/<uuid> and is returned in the x-request-id response header.

Gateway log lines​

MessageFields
gateway listeningdataHost, dataPort, controlHost, controlPort, models, allowlistsFrozen
shutting down, stoppedsignal, drained

Worker log lines​

MessageNotes
worker configfingerprints, heartbeat interval, namespace, task queue, tls
worker runningwith alsoExec and quality flags
worker stoppedon shutdown
exec worker running, exec worker stoppedthe fleet exec worker
plaintext connection to Temporal (TEMPORAL_ALLOW_PLAINTEXT)warn, exec worker without TLS
TEMPORAL_TLS_* unset: plaintext connection to Temporalwarn, core worker

Startup comparison to do after every deploy: the API's session config and the worker's worker config must show the same namespace, task queue, heartbeat interval and codecKeys fingerprints.

Health endpoints​

EndpointAuthReturnsMeaning
GET /health on the APIpublic{"status":"ok"}liveness only; touches no dependency
GET /health on the gateway data portnoneexactly {"status":"ok"}liveness
GET /control/v1/health on the gateway control portworker control token{status, auditWriteFailures, preAuthDenials}auditWriteFailures > 0 means audit rows are being lost while requests are still answered; preAuthDenials counts pre-auth denials

Workers serve no port: their Docker HEALTHCHECK is a process liveness check (node -e process.exit(0), every 30 s), so a healthy worker container says nothing about Temporal connectivity. Check the worker running log line instead.

The usage endpoint​

GET /v1/usage is the spend report. It needs a key whose client is in API_ADMIN_CLIENTS; any other client gets 404 (not_found), not 403.

Response (UsageResponse):

FieldMeaning
dayUTC day YYYY-MM-DD
clients[]client, spentUsd, inputTokens, cacheReadTokens, cacheWriteTokens, cacheHitRate (cacheRead / (input + cacheRead + cacheWrite), null when the denominator is 0)
globalSpentUsdtotal spend of all clients for the day
staleReservations[]requestId, client, amountUsd, createdAt: ledger rows still reserved longer than STALE_RESERVATION_SEC
curl -fsS --oauth2-bearer "$ADMIN_KEY" "$API/v1/usage"

After a clean smoke run the expected state is owner spend equal to the sum of the turns and an empty staleReservations.

Stale reservations​

A reservation counts in full against spend until settled. The stale check only reports; it never releases rows.

SettingDefaultLimits
STALE_RESERVATION_SEC900minimum 600
STALE_CHECK_INTERVAL_MS300000

Every interval the API logs one warn line (stale reservations) while the list is non-empty, with event: stale_reservations, count, oldestRequestId, oldestAgeSec, totalReservedUsd. It logs nothing when empty, and one error line (stale reservation check failed) on a query error.

Abandon (settling a stale row as aborted at its full reservation) is a workflow step before a retry, not an operator command.

Spend alerts​

createAlertListener in packages/agent-core/src/alerts.ts runs after each settle.

PropertyValue
ThresholdsALERT_THRESHOLDS = [0.5, 0.8] of GLOBAL_DAILY_LIMIT_USD
Log linewarn: global daily spend crossed alert threshold with threshold, spendUsd, limitUsd
WebhookALERT_WEBHOOK_URL (optional, URL), POST with content-type: application/json and body {threshold, spendUsd, limitUsd}
Timeout5 s
Deliveryat most once per threshold per UTC day; dedup is in alerts_sent (primary key day, threshold), so it survives restarts
Failurethe threshold is marked sent before the POST, so a failed or timed-out POST is not retried that day; the error log line is the only signal
Blockingdetached; a slow or failing webhook never delays or fails the request

Monitor these log messages: alert webhook failed (network error or timeout) and alert webhook rejected (non-2xx, with status and threshold).

The quality runbook says alerts for the daily global sandbox limit at 80 percent and for any control failure go through ALERT_WEBHOOK_URL; that is Unverified and not wired yet.

The kill switch​

The kill switch refuses every new request with disabled. It is on when either source is on, and it is treated as on when it cannot be read (fail closed).

SourceHow to setNotes
Config flag AGENTS_DISABLEDenvironment of api, worker, factory workerparsed fail closed: unset means enabled; any set value other than 0 or false (trimmed, case-insensitive) disables agents. The runbooks set false to enable
Persisted settingsettings row with key = 'agents_disabled', value 1 or true (case-insensitive) disables; any other value or no row enablesread on every request, so it works without a redeploy

Setting it from the database:

INSERT INTO settings (key, value) VALUES ('agents_disabled', '1')
ON CONFLICT (key) DO UPDATE SET value = '1';

The SQL above is built from the table definition (key primary key, value not null) and stores.ts; it has not been run against a real database in this documentation pass.

Behavior by component:

ComponentOn disabled
Guarded runtime (api, worker)writes a ledger row with status rejected_disabled, yields a terminal result disabled; the kill switch runs before admission, so a disabled request never reserves
Session turnends failed with errorCode: disabled and no spend; kill-switched or aborted turns may book up to 2.00 USD against the daily cap (over-count only)
Gateway data pathanswers 503 (GV3 probe b)
Factorydisabled is a FactoryErrorCode
Qualityunsetting QUALITY_ENABLED makes the routes answer 404 and registers nothing; the database kill switch also stops admission and chunks

The deployed kill-switch check (EXPECT_DISABLED=1 in the smoke script, which needs AGENTS_DISABLED=true in SSM and a redeploy) was never run on the deployed stack.

The other cost-containment levers, in order of speed: the gateway revoke path (DELETE /control/v1/jobs/:requestId kills a registration; live job tokens are in memory and a gateway restart drops them all), the daily limits (client 2.00 and global 5.00 USD on the core host), and the model key's Console spend limit.

Gateway audit​

Every gateway decision writes a gateway_audit row (evidence only: no prompts, bodies, tokens or headers).

ColumnMeaning
at, job_id, request_id, subject_idwhen and for whom (subject_id is the IdP sub, never a token)
targetmodel id or gateway tool name
args_sha256hash of tool arguments, never the arguments
kindmodel, tool, context, control
decisionallow, deny
identityobo, service, none
reasonnull only for allow model or tool rows; otherwise one of the audit reasons below
http_status, input_tokens, output_tokens, latency_ms, cost_usdoutcome

Audit reasons (AuditReason): field_not_allowed, content_not_allowed, content_type_not_allowed, body_too_large, invalid_json, model_not_allowed, max_tokens_invalid, thinking_budget_invalid, tool_type_not_allowed, beta_not_allowed, header_invalid, budget_exhausted, registration_stale, disabled, ledger_error, busy, rate_limited, upstream_error, upstream_bad_content_type, upstream_timeout, client_aborted, tool_not_granted, args_too_large, args_invalid, exchange_denied, schema_changed, result_too_large, server_request_denied, content_stripped, method_not_found, registered, revoked, registration_conflict, registration_unknown, context_recorded.

Useful operator queries (assembled from the schema; not run in this pass):

SELECT reason, count(*) FROM gateway_audit WHERE decision = 'deny' AND at > now() - interval '1 day' GROUP BY reason ORDER BY 2 DESC;
SELECT sum(cost_usd) FROM gateway_audit WHERE kind = 'model' AND decision = 'allow' AND at > now() - interval '1 day';

In the smoke runs the first failed turn was diagnosed from denials of ?beta=true and unlisted fields before authentication: see the gateway allowlist variables in Deploy runbooks. Retention is a manual 90-day delete (Persistence and backups).

Gateway rate-limit and denial counters in GET /control/v1/health are cumulative since process start and reset on a restart.

What is not wired​

GapSource
No uptime monitor, metrics or dashboards for AgentForgenot present in the repository
Alerting and log rotation for the pg_dump logdocs/p2-findings.md
Quality alerts through ALERT_WEBHOOK_URLdocs/deploy-quality.md ("not wired yet")
Worker liveness checks only prove the processDockerfile HEALTHCHECK
/v1/usage has no history, only today (UTC)spendByClientToday
Log shipping and retentionnot configured in the repository; the hosts' container logs are the only copy (Unverified)

Sources​

  • packages/agent-core/src/redaction.ts (redact, SECRET_PATTERNS, createLogger, logError)
  • packages/agent-core/src/alerts.ts, packages/agent-core/README.md ("Spend alerts")
  • packages/agent-core/src/kill-switch.ts
  • packages/db/src/stores.ts (isDisabled), packages/db/src/schema.ts (settings, alertsSent, gatewayAudit)
  • packages/agent-types/src/audit.ts, packages/agent-types/src/http.ts (UsageResponse)
  • apps/api/src/app.ts (request log, /health, /v1/models, /v1/usage), apps/api/src/serve.ts, apps/api/src/stale.ts
  • apps/gateway/src/serve.ts, apps/gateway/src/control.ts, apps/gateway/src/proxy.ts
  • apps/worker/src/worker.ts, apps/worker/src/exec-main.ts, apps/worker/src/activities.ts
  • docs/deploy-api-coolify.md, docs/deploy-gateway-coolify.md, docs/deploy-quality.md, docs/deploy-app02-test.md (smoke and kill-switch steps)
  • docs/p2-findings.md ("App-03 Postgres volume restart proof and nightly pg_dump")
  • scripts/deploy/smoke-app02.sh, scripts/gv3-standin-probe.mjs