Sessions and workflows
A session is a multi-turn conversation. Each turn is a separate model run, scheduled by a Temporal workflow and executed by activities on worker processes. Postgres is the system of record for sessions, messages and the spend ledger; Temporal decides when work runs and carries only ids, small enums and (since the fleet split) encrypted prompts and replies.
Three rules shape the design (stated in the source comments and the P2.T findings):
- Postgres decides every terminal outcome. Temporal cancellation, worker shutdown and timeouts only trigger an abort; the final write to a message row is fenced by
request_id. - Workflow code throws only Temporal failures. A plain
Errorkeeps a workflow task retrying forever and the workflow staysRUNNING; the workflow converts anything else intoApplicationFailure.nonRetryable('workflow_error'). - Errors thrown to Temporal carry fixed codes only, never prompt text or driver messages.
Data model
Constraints that implement the invariants: messages_session_id_seq_unique on (session_id, seq); the partial unique index messages_one_open_turn allows at most one message per session in pending or running; check constraints sessions_status_valid and messages_status_valid. createSession takes a per-client advisory lock (pg_advisory_xact_lock(hashtext('agentforge:sessions:' || client))) before counting open sessions against SESSION_MAX_OPEN_PER_CLIENT.
State machines
Message (turn) status
claimTurn only claims a message in pending or running whose attempts < MAX_TURN_ATTEMPTS (3) and whose session is open with no cancel flag. When no row qualifies and attempts are exhausted, the same transaction marks the message failed with error code agent_failed and returns nothing, which the activity reports as ignored.
Session status
closeSession in PgSessionStore is idempotent. The effective reason is cancelled whenever the cancel flag is set, and a cancelled close sets the flag, so every aborted message belongs to a flagged session. Open messages are rewritten: cancelled gives aborted with no error code, closed gives failed with conflict, failed gives failed with agent_failed. The caller then runs ledger.abandonPrefix('session-<id>/').
The workflow agentSessionWorkflow
Workflow type name agentSessionWorkflow, workflow id session-<sessionId>, default task queue agentforge-sessions (SESSION_TASK_QUEUE, overridden by TEMPORAL_TASK_QUEUE). The API starts it by type name and signals it by the string userMessage; it never imports workflow code (TMP-2), so the Temporal SDK's workflow package stays out of the API process.
Input
SessionWorkflowArgs, filled by the API from config:
| Field | API source | Default |
|---|---|---|
sessionId | new session uuid | |
maxTurns | SESSION_MAX_TURNS | 50 |
idleTimeoutMs | SESSION_IDLE_TIMEOUT_MS | 86,400,000 (24 h), minimum 1000 |
heartbeatTimeoutMs | SESSION_TURN_HEARTBEAT_TIMEOUT_MS | 30,000, minimum 300 |
heartbeatIntervalMs | SESSION_TURN_HEARTBEAT_INTERVAL_MS | floor(timeout / 3), must be at most timeout / 3 |
execScheduleToStartMs | SESSION_EXEC_SCHEDULE_TO_START_MS | 60,000, minimum 1000 |
turnRetryInitialIntervalMs | tests only | 5000 |
validArgs checks every field (wire data is untrusted); an invalid start fails with ApplicationFailure.nonRetryable('invalid_workflow_args') and calls no activity.
Control flow
Signal handling: the userMessage handler only touches an in-memory Set and queue and never throws. A message id is dropped if it was already seen or if seen.size >= maxTurns, so a redelivered or forged signal cannot start a second turn. The activities re-check everything in Postgres (a forged id is ignored by claimTurn).
Activities and their Temporal options
| Activity | Task queue | startToClose | Heartbeat | Retry |
|---|---|---|---|---|
prepareTurn | session queue | 30 s | none | initial 5 s, coefficient 2, maximumAttempts 4 (MAX_TURN_ATTEMPTS + 1) |
executeTurn (fleet) | <session queue>-exec | 360 s | heartbeatTimeoutMs, cancellation WAIT_CANCELLATION_COMPLETED | maximumAttempts: 1, scheduleToStartTimeout = execScheduleToStartMs |
executeTurn (core fallback) | <session queue>-exec-core | 360 s | as fleet | maximumAttempts: 1, no schedule-to-start timeout |
finishTurn | session queue | 30 s | none | initial 5 s, coefficient 2, maximum interval 5 min, no attempt limit |
failTurn | session queue | 30 s | none | as finishTurn |
closeSession | session queue | 30 s | none | as finishTurn |
runTurn (legacy) | session queue | 360 s | heartbeatTimeoutMs, WAIT_CANCELLATION_COMPLETED | as prepareTurn |
runTurn is the single pre-split activity. The workflow keeps it behind patched('fleet-split') so histories recorded before the split still replay and finish. The split path is prepareTurn, executeTurn, finishTurn.
WAIT_CANCELLATION_COMPLETED is required: with TRY_CANCEL the activity sees only "not found" and closeSession would run first (T0d finding).
The split turn
Details of each step:
prepareTurn(apps/worker/src/activities.ts):claimTurnreturnsrequestId = session-<sid>/<mid>#<attempts>from the database counter (never Temporal's attempt number) plus the completed history. It abandons earlier attempts' ledger rows and revokes their job tokens, because an earlier attempt's exec worker may still be running. It renders the transcript withrenderTranscriptand, ifprompt.length > SESSION_MAX_PROMPT_CHARS(default 80,000), writes the message asagent_limitwithout any runtime call. It reads the kill switch (AGENTS_DISABLEDorsettings.isDisabled(), failing closed on a read error) and, when disabled, records arejected_disabledledger row and writesdisabled. It reserves withledger.admitusing the per-request cap (PER_REQUEST_CAP_USD, worker default 2) and a floor ofmax(BUDGET_FLOOR_USD, modelMin)wheremodelMinismin(SONNET_MIN_RESERVATION_USD, cap)for model ids containingsonnet. A duplicate request id writesinvalid_request; a budget refusal recordsrejected_budgetand writesbudget_exhausted. Limits passed on aremaxTurns5,timeoutMs120,000 (agent-core defaults) andmaxBudgetUsd = reserved * (1 - BUDGET_MARGIN). It registers the job with the gateway with TTLtimeoutMs + 60000and role policysession-readonly. A registration failure settles the row at zero cost and throws the retryable failuregateway_unavailable.executeTurn(apps/worker/src/exec-activities.ts): no Postgres and no control token. It buildswithRedaction(withLimits(inner(jobToken))), runs with profilereadonly, heartbeats on a timer, and returns{status, text}(text only on success). Anabortedstatus caused by anything other than a Temporal cancel becomes the retryable failureturn_interrupted; a Temporal cancel rethrows theCancelledFailure(not retried).finishTurn: settles the ledger row first so a message shown as finished has a final ledger row. The exec result is not trusted for cost: it settles with zero tokens and cost 0, because a metered row costs its gateway debits and an unmetered row had no gateway call. Then the fenced write, revoke, re-abandon, and a detached spend-alert listener.
If executeTurn fails with a retryable ActivityFailure (worker lost, heartbeat timeout, turn_interrupted), the workflow sleeps initialInterval * 2^(attempt - 1) and loops back to prepareTurn, which starts a new attempt with a new request id and a new token. The loop allows MAX_TURN_ATTEMPTS + 1 passes so the last one reaches claimTurn's exhausted branch. A non-retryable failure or an exceeded count rethrows; the outer handler calls failTurn.
Fleet fallback
execute calls the fleet queue. If it fails with a SCHEDULE_TO_START timeout (no fleet worker picked it up within execScheduleToStartMs), it calls the same activity on the core fallback queue. The mechanics, defaults and operations are on Worker fleet.
Retries and idempotency
| Mechanism | Where | Effect |
|---|---|---|
(session_id, seq) unique and one-open-turn index | messages | at most one live turn per session |
request_id = session-<sid>/<mid>#<n> | claimTurn | every attempt is a distinct ledger row and a fence value |
admit returns duplicate | ledger | a replayed request id writes invalid_request instead of a second reservation |
abandonPrefix / abandon | ledger | earlier attempts are settled so a late zombie debit pays max(actual, reserved) |
| signal de-dup by message id | workflow | redelivery is a no-op |
fenced finishTurn update (WHERE request_id = $2 AND status = 'running') | Postgres | a stale attempt's result is discarded (ignored) |
MAX_TURN_ATTEMPTS = 3 | DB attempt cap | the Temporal option is one higher to reach the exhausted branch |
db_error | activities | any database failure inside an activity becomes a fixed retryable code |
prepareTurn retries can also create attempts: each Temporal retry of prepareTurn calls claimTurn again and takes a new attempt, so claimTurn bounds them.
Cancellation
Order matters (P2.T plan 7.4): the Postgres flag first, so a running turn's next fence tick aborts and new messages get 409; then the Temporal cancel; then the close-out check. If Temporal is unreachable the flag stays set and the API answers 503; a retry closes a dead workflow's session.
Inside runTurn (legacy path) a timer ticks every heartbeatIntervalMs: it heartbeats, then reads the fence with fenceState(messageId, requestId):
| Abort reason | Source | Outcome |
|---|---|---|
temporal_cancel | cancellation details cancelRequested | message aborted if the DB flag is also set, else no write (the workflow's close sets it); CancelledFailure thrown |
interrupted | worker shutdown, timeout, paused, reset, no details | retryable turn_interrupted; message stays running |
fence_lost | another attempt holds the message | retryable turn_interrupted |
session_cancel | DB flag set | message written aborted |
fence_query_failed | database error in the tick | fail closed: abort, retry |
Cancel latency is bounded by the heartbeat throttle. The worker sets maxHeartbeatThrottleInterval and defaultHeartbeatThrottleInterval to SESSION_TURN_HEARTBEAT_INTERVAL_MS (default 10,000), so the interval must match between the API (which sends it in the workflow args) and the worker. The T0d probe measured 4 configurations times 20 cancels, all within the throttle plus the interval.
On the split path executeTurn has no database access, so there is no fence tick: cancellation reaches it only through heartbeats, and an abandoned attempt's token dies at its next gateway call (the ledger row is abandoned, the gateway answers settled).
Temporal payload encryption
Every payload, including the workflow input, signals and activity arguments, is encrypted by AesGcmCodec in packages/agent-workflows/src/codec.ts:
- AES-256-GCM through
node:crypto, a random 12-byte IV per payload;data = iv(12) | ciphertext | tag(16); payload metadataencoding = binary/encryptedandencryption-key-id. - The whole payload (metadata and data) is sealed as a JSON envelope. Non-encrypted payloads pass through on decode; an unknown key id or tampered data throws.
TEMPORAL_PAYLOAD_KEYSis a JSON map{keyId: base64(32 bytes)};TEMPORAL_PAYLOAD_KEY_IDnames the encode key. Rotation is adding a key id. Services logkeyFingerprints(first 12 hex characters of each key's SHA-256) at startup so API and worker can be compared without exposing key material.- The failure converter (
DefaultFailureConverterwithencodeCommonAttributes: true) moves failure messages and stacks into encrypted attributes. The converter is loaded by path (failureConverterPath) because workflow code runs in the sandbox. TEMPORAL_ADDRESSwithout both key variables stops the API at startup.
Worker processes
runWorker (apps/worker/src/worker.ts) starts several Temporal workers on one connection and shuts them all down if one fails:
| Worker | Task queue | Has workflow bundle | Activities |
|---|---|---|---|
| main | TEMPORAL_TASK_QUEUE (default agentforge-sessions) | yes (workflow-bundle.js) | prepareTurn, finishTurn, failTurn, closeSession, runTurn |
| fallback | agentforge-sessions-exec-core | no | executeTurn |
also-exec (when WORKER_ALSO_EXEC is true, the default) | agentforge-sessions-exec | no | executeTurn |
quality (when QUALITY_ENABLED) | QUALITY_TASK_QUEUE | yes | CodeCrucible core activities |
| Variable | Default | Meaning |
|---|---|---|
WORKER_CONCURRENCY | 2 | maxConcurrentActivityTaskExecutions per worker |
WORKER_SHUTDOWN_GRACE_MS | 5000 | in-flight turns get this long after SIGTERM, then are cancelled with workerShutdown (treated as interrupted: retried, never aborted) |
SESSION_MAX_PROMPT_CHARS | 80,000 | rendered transcript cap |
SESSION_TURN_HEARTBEAT_INTERVAL_MS | 10,000 | heartbeat throttle, must equal the API's |
PER_REQUEST_CAP_USD | 2 | session turn reservation cap (the API's synchronous cap stays 0.5) |
WORKER_IDENTITY | SDK default | Temporal worker identity for audit |
The 80,000-character cap and the 2.00 USD default are explained in the source comments: one sonnet call through the gateway can debit up to about 0.44 USD at 8192 output tokens, so a smaller reservation yields a guaranteed 429.
Transcript rendering
renderTranscript(history, prompt) is a pure function of the stored strings in sequence order: no ids, attempt numbers or timestamps, fixed newline separators, no trimming, so equal histories render byte-identically. A first turn is the plain prompt. Otherwise the output is Previous conversation: followed by User: <prompt> and Assistant: <reply> lines for each completed turn, then Current message: and User: <prompt>.
Not verified
- Everything against production Temporal: the T0c link behaviour and escrow custody of payload keys are marked UNVERIFIED in
docs/p2-findings.md. - Temporal history growth with long transcripts: each split turn stores the prompt twice (prepare result and execute input), encrypted. At the 80,000-character cap and 50 turns that is up to about 8 MB per session (Temporal warns at 10 MB, fails at 50 MB). Not measured (
docs/deploy-worker-fleet.md).
Sources
packages/agent-workflows/src/workflows/index.ts(agentSessionWorkflow,splitTurn, activity options)packages/agent-workflows/src/workflows/shared.ts(constants,SessionActivities,SessionWorkflowArgs, queue helpers)packages/agent-workflows/src/codec.ts(AesGcmCodec,parsePayloadKeys,keyFingerprints)packages/agent-workflows/src/failure-converter.ts,packages/agent-workflows/src/index.ts(createDataConverter)apps/worker/src/activities.ts(createActivities:runTurn,prepareTurn,finishTurn,failTurn,closeSession)apps/worker/src/exec-activities.ts(createExecActivities)apps/worker/src/worker.ts(runWorker,runAll),apps/worker/src/config.ts,apps/worker/src/transcript.tsapps/api/src/sessions.ts,apps/api/src/temporal.ts,apps/api/src/config.tspackages/db/src/sessions.ts(PgSessionStore),packages/db/src/schema.ts(sessions,messages)docs/p2-findings.md(P2.T Step 0 summary table, T0d, T0e, T0f),docs/deploy-worker-fleet.md,docs/design-worker-fleet.md