Skip to main content

Sessions and workflows

A session is a multi-turn conversation. Each turn is a separate model run, scheduled by a Temporal workflow and executed by activities on worker processes. Postgres is the system of record for sessions, messages and the spend ledger; Temporal decides when work runs and carries only ids, small enums and (since the fleet split) encrypted prompts and replies.

Three rules shape the design (stated in the source comments and the P2.T findings):

  1. Postgres decides every terminal outcome. Temporal cancellation, worker shutdown and timeouts only trigger an abort; the final write to a message row is fenced by request_id.
  2. Workflow code throws only Temporal failures. A plain Error keeps a workflow task retrying forever and the workflow stays RUNNING; the workflow converts anything else into ApplicationFailure.nonRetryable('workflow_error').
  3. Errors thrown to Temporal carry fixed codes only, never prompt text or driver messages.

Data model​

Constraints that implement the invariants: messages_session_id_seq_unique on (session_id, seq); the partial unique index messages_one_open_turn allows at most one message per session in pending or running; check constraints sessions_status_valid and messages_status_valid. createSession takes a per-client advisory lock (pg_advisory_xact_lock(hashtext('agentforge:sessions:' || client))) before counting open sessions against SESSION_MAX_OPEN_PER_CLIENT.

State machines​

Message (turn) status​

claimTurn only claims a message in pending or running whose attempts < MAX_TURN_ATTEMPTS (3) and whose session is open with no cancel flag. When no row qualifies and attempts are exhausted, the same transaction marks the message failed with error code agent_failed and returns nothing, which the activity reports as ignored.

Session status​

closeSession in PgSessionStore is idempotent. The effective reason is cancelled whenever the cancel flag is set, and a cancelled close sets the flag, so every aborted message belongs to a flagged session. Open messages are rewritten: cancelled gives aborted with no error code, closed gives failed with conflict, failed gives failed with agent_failed. The caller then runs ledger.abandonPrefix('session-<id>/').

The workflow agentSessionWorkflow​

Workflow type name agentSessionWorkflow, workflow id session-<sessionId>, default task queue agentforge-sessions (SESSION_TASK_QUEUE, overridden by TEMPORAL_TASK_QUEUE). The API starts it by type name and signals it by the string userMessage; it never imports workflow code (TMP-2), so the Temporal SDK's workflow package stays out of the API process.

Input​

SessionWorkflowArgs, filled by the API from config:

FieldAPI sourceDefault
sessionIdnew session uuid
maxTurnsSESSION_MAX_TURNS50
idleTimeoutMsSESSION_IDLE_TIMEOUT_MS86,400,000 (24 h), minimum 1000
heartbeatTimeoutMsSESSION_TURN_HEARTBEAT_TIMEOUT_MS30,000, minimum 300
heartbeatIntervalMsSESSION_TURN_HEARTBEAT_INTERVAL_MSfloor(timeout / 3), must be at most timeout / 3
execScheduleToStartMsSESSION_EXEC_SCHEDULE_TO_START_MS60,000, minimum 1000
turnRetryInitialIntervalMstests only5000

validArgs checks every field (wire data is untrusted); an invalid start fails with ApplicationFailure.nonRetryable('invalid_workflow_args') and calls no activity.

Control flow​

Signal handling: the userMessage handler only touches an in-memory Set and queue and never throws. A message id is dropped if it was already seen or if seen.size >= maxTurns, so a redelivered or forged signal cannot start a second turn. The activities re-check everything in Postgres (a forged id is ignored by claimTurn).

Activities and their Temporal options​

ActivityTask queuestartToCloseHeartbeatRetry
prepareTurnsession queue30 snoneinitial 5 s, coefficient 2, maximumAttempts 4 (MAX_TURN_ATTEMPTS + 1)
executeTurn (fleet)<session queue>-exec360 sheartbeatTimeoutMs, cancellation WAIT_CANCELLATION_COMPLETEDmaximumAttempts: 1, scheduleToStartTimeout = execScheduleToStartMs
executeTurn (core fallback)<session queue>-exec-core360 sas fleetmaximumAttempts: 1, no schedule-to-start timeout
finishTurnsession queue30 snoneinitial 5 s, coefficient 2, maximum interval 5 min, no attempt limit
failTurnsession queue30 snoneas finishTurn
closeSessionsession queue30 snoneas finishTurn
runTurn (legacy)session queue360 sheartbeatTimeoutMs, WAIT_CANCELLATION_COMPLETEDas prepareTurn

runTurn is the single pre-split activity. The workflow keeps it behind patched('fleet-split') so histories recorded before the split still replay and finish. The split path is prepareTurn, executeTurn, finishTurn.

WAIT_CANCELLATION_COMPLETED is required: with TRY_CANCEL the activity sees only "not found" and closeSession would run first (T0d finding).

The split turn​

Details of each step:

  • prepareTurn (apps/worker/src/activities.ts): claimTurn returns requestId = session-<sid>/<mid>#<attempts> from the database counter (never Temporal's attempt number) plus the completed history. It abandons earlier attempts' ledger rows and revokes their job tokens, because an earlier attempt's exec worker may still be running. It renders the transcript with renderTranscript and, if prompt.length > SESSION_MAX_PROMPT_CHARS (default 80,000), writes the message as agent_limit without any runtime call. It reads the kill switch (AGENTS_DISABLED or settings.isDisabled(), failing closed on a read error) and, when disabled, records a rejected_disabled ledger row and writes disabled. It reserves with ledger.admit using the per-request cap (PER_REQUEST_CAP_USD, worker default 2) and a floor of max(BUDGET_FLOOR_USD, modelMin) where modelMin is min(SONNET_MIN_RESERVATION_USD, cap) for model ids containing sonnet. A duplicate request id writes invalid_request; a budget refusal records rejected_budget and writes budget_exhausted. Limits passed on are maxTurns 5, timeoutMs 120,000 (agent-core defaults) and maxBudgetUsd = reserved * (1 - BUDGET_MARGIN). It registers the job with the gateway with TTL timeoutMs + 60000 and role policy session-readonly. A registration failure settles the row at zero cost and throws the retryable failure gateway_unavailable.
  • executeTurn (apps/worker/src/exec-activities.ts): no Postgres and no control token. It builds withRedaction(withLimits(inner(jobToken))), runs with profile readonly, heartbeats on a timer, and returns {status, text} (text only on success). An aborted status caused by anything other than a Temporal cancel becomes the retryable failure turn_interrupted; a Temporal cancel rethrows the CancelledFailure (not retried).
  • finishTurn: settles the ledger row first so a message shown as finished has a final ledger row. The exec result is not trusted for cost: it settles with zero tokens and cost 0, because a metered row costs its gateway debits and an unmetered row had no gateway call. Then the fenced write, revoke, re-abandon, and a detached spend-alert listener.

If executeTurn fails with a retryable ActivityFailure (worker lost, heartbeat timeout, turn_interrupted), the workflow sleeps initialInterval * 2^(attempt - 1) and loops back to prepareTurn, which starts a new attempt with a new request id and a new token. The loop allows MAX_TURN_ATTEMPTS + 1 passes so the last one reaches claimTurn's exhausted branch. A non-retryable failure or an exceeded count rethrows; the outer handler calls failTurn.

Fleet fallback​

execute calls the fleet queue. If it fails with a SCHEDULE_TO_START timeout (no fleet worker picked it up within execScheduleToStartMs), it calls the same activity on the core fallback queue. The mechanics, defaults and operations are on Worker fleet.

Retries and idempotency​

MechanismWhereEffect
(session_id, seq) unique and one-open-turn indexmessagesat most one live turn per session
request_id = session-<sid>/<mid>#<n>claimTurnevery attempt is a distinct ledger row and a fence value
admit returns duplicateledgera replayed request id writes invalid_request instead of a second reservation
abandonPrefix / abandonledgerearlier attempts are settled so a late zombie debit pays max(actual, reserved)
signal de-dup by message idworkflowredelivery is a no-op
fenced finishTurn update (WHERE request_id = $2 AND status = 'running')Postgresa stale attempt's result is discarded (ignored)
MAX_TURN_ATTEMPTS = 3DB attempt capthe Temporal option is one higher to reach the exhausted branch
db_erroractivitiesany database failure inside an activity becomes a fixed retryable code

prepareTurn retries can also create attempts: each Temporal retry of prepareTurn calls claimTurn again and takes a new attempt, so claimTurn bounds them.

Cancellation​

Order matters (P2.T plan 7.4): the Postgres flag first, so a running turn's next fence tick aborts and new messages get 409; then the Temporal cancel; then the close-out check. If Temporal is unreachable the flag stays set and the API answers 503; a retry closes a dead workflow's session.

Inside runTurn (legacy path) a timer ticks every heartbeatIntervalMs: it heartbeats, then reads the fence with fenceState(messageId, requestId):

Abort reasonSourceOutcome
temporal_cancelcancellation details cancelRequestedmessage aborted if the DB flag is also set, else no write (the workflow's close sets it); CancelledFailure thrown
interruptedworker shutdown, timeout, paused, reset, no detailsretryable turn_interrupted; message stays running
fence_lostanother attempt holds the messageretryable turn_interrupted
session_cancelDB flag setmessage written aborted
fence_query_faileddatabase error in the tickfail closed: abort, retry

Cancel latency is bounded by the heartbeat throttle. The worker sets maxHeartbeatThrottleInterval and defaultHeartbeatThrottleInterval to SESSION_TURN_HEARTBEAT_INTERVAL_MS (default 10,000), so the interval must match between the API (which sends it in the workflow args) and the worker. The T0d probe measured 4 configurations times 20 cancels, all within the throttle plus the interval.

On the split path executeTurn has no database access, so there is no fence tick: cancellation reaches it only through heartbeats, and an abandoned attempt's token dies at its next gateway call (the ledger row is abandoned, the gateway answers settled).

Temporal payload encryption​

Every payload, including the workflow input, signals and activity arguments, is encrypted by AesGcmCodec in packages/agent-workflows/src/codec.ts:

  • AES-256-GCM through node:crypto, a random 12-byte IV per payload; data = iv(12) | ciphertext | tag(16); payload metadata encoding = binary/encrypted and encryption-key-id.
  • The whole payload (metadata and data) is sealed as a JSON envelope. Non-encrypted payloads pass through on decode; an unknown key id or tampered data throws.
  • TEMPORAL_PAYLOAD_KEYS is a JSON map {keyId: base64(32 bytes)}; TEMPORAL_PAYLOAD_KEY_ID names the encode key. Rotation is adding a key id. Services log keyFingerprints (first 12 hex characters of each key's SHA-256) at startup so API and worker can be compared without exposing key material.
  • The failure converter (DefaultFailureConverter with encodeCommonAttributes: true) moves failure messages and stacks into encrypted attributes. The converter is loaded by path (failureConverterPath) because workflow code runs in the sandbox.
  • TEMPORAL_ADDRESS without both key variables stops the API at startup.

Worker processes​

runWorker (apps/worker/src/worker.ts) starts several Temporal workers on one connection and shuts them all down if one fails:

WorkerTask queueHas workflow bundleActivities
mainTEMPORAL_TASK_QUEUE (default agentforge-sessions)yes (workflow-bundle.js)prepareTurn, finishTurn, failTurn, closeSession, runTurn
fallbackagentforge-sessions-exec-corenoexecuteTurn
also-exec (when WORKER_ALSO_EXEC is true, the default)agentforge-sessions-execnoexecuteTurn
quality (when QUALITY_ENABLED)QUALITY_TASK_QUEUEyesCodeCrucible core activities
VariableDefaultMeaning
WORKER_CONCURRENCY2maxConcurrentActivityTaskExecutions per worker
WORKER_SHUTDOWN_GRACE_MS5000in-flight turns get this long after SIGTERM, then are cancelled with workerShutdown (treated as interrupted: retried, never aborted)
SESSION_MAX_PROMPT_CHARS80,000rendered transcript cap
SESSION_TURN_HEARTBEAT_INTERVAL_MS10,000heartbeat throttle, must equal the API's
PER_REQUEST_CAP_USD2session turn reservation cap (the API's synchronous cap stays 0.5)
WORKER_IDENTITYSDK defaultTemporal worker identity for audit

The 80,000-character cap and the 2.00 USD default are explained in the source comments: one sonnet call through the gateway can debit up to about 0.44 USD at 8192 output tokens, so a smaller reservation yields a guaranteed 429.

Transcript rendering​

renderTranscript(history, prompt) is a pure function of the stored strings in sequence order: no ids, attempt numbers or timestamps, fixed newline separators, no trimming, so equal histories render byte-identically. A first turn is the plain prompt. Otherwise the output is Previous conversation: followed by User: <prompt> and Assistant: <reply> lines for each completed turn, then Current message: and User: <prompt>.

Not verified​

  • Everything against production Temporal: the T0c link behaviour and escrow custody of payload keys are marked UNVERIFIED in docs/p2-findings.md.
  • Temporal history growth with long transcripts: each split turn stores the prompt twice (prepare result and execute input), encrypted. At the 80,000-character cap and 50 turns that is up to about 8 MB per session (Temporal warns at 10 MB, fails at 50 MB). Not measured (docs/deploy-worker-fleet.md).

Sources​

  • packages/agent-workflows/src/workflows/index.ts (agentSessionWorkflow, splitTurn, activity options)
  • packages/agent-workflows/src/workflows/shared.ts (constants, SessionActivities, SessionWorkflowArgs, queue helpers)
  • packages/agent-workflows/src/codec.ts (AesGcmCodec, parsePayloadKeys, keyFingerprints)
  • packages/agent-workflows/src/failure-converter.ts, packages/agent-workflows/src/index.ts (createDataConverter)
  • apps/worker/src/activities.ts (createActivities: runTurn, prepareTurn, finishTurn, failTurn, closeSession)
  • apps/worker/src/exec-activities.ts (createExecActivities)
  • apps/worker/src/worker.ts (runWorker, runAll), apps/worker/src/config.ts, apps/worker/src/transcript.ts
  • apps/api/src/sessions.ts, apps/api/src/temporal.ts, apps/api/src/config.ts
  • packages/db/src/sessions.ts (PgSessionStore), packages/db/src/schema.ts (sessions, messages)
  • docs/p2-findings.md (P2.T Step 0 summary table, T0d, T0e, T0f), docs/deploy-worker-fleet.md, docs/design-worker-fleet.md