qualityAssessmentWorkflow
One assessment measures one package of one repository at one head commit, gates it against the stored baseline of the protected branch, and (only when the head is the protected tip) replaces that baseline. It is also the child workflow a verify runs twice, once for the base and once for the head (see verify workflow).
Code: packages/agent-workflows/src/workflows/quality.ts (qualityAssessmentWorkflow), contract in
quality-shared.ts, core activities in apps/worker/src/quality/core-activities.ts, exec activities in
apps/worker/src/quality/exec-activities.ts (bead CC-2.5, CC-2.4).
The workflow result is informational. Postgres holds the verdict; REST reads Postgres, never a workflow result (plan section 3, "Verdict").
Names and queues
Constant (quality-shared.ts) | Value | Used for |
|---|---|---|
QUALITY_TASK_QUEUE | agentforge-quality | Workflows and core activities (core worker) |
QUALITY_EXEC_TASK_QUEUE | agentforge-quality-exec | Shared exec queue: prepareWorkspace only |
qualityExecIdentityQueue(identity) | agentforge-quality-exec-<identity> | Everything after prepareWorkspace, on the worker that prepared the workspace |
QUALITY_ASSESSMENT_WORKFLOW_TYPE | qualityAssessmentWorkflow | Workflow type |
QUALITY_VERIFY_WORKFLOW_TYPE | qualityVerifyWorkflow | Workflow type |
qualityWorkflowId(runId) | quality-<runId> | Workflow id; also quality_runs.workflow_id |
The identity queue exists so that two pollers during a rolling deploy cannot split one run: the workspace tar lives on the disk of the worker that prepared it.
Limits (QUALITY_RUN_LIMITS)
| Key | Value | Meaning |
|---|---|---|
maxChunks | 20 | More chunks end the run partial (AC7) |
assessmentSandboxMs | 3,600,000 (3,600 s) | Sandbox time of one assessment; reaching it ends partial |
chunkMaxMs | 840,000 (840 s) | One chunk including Stryker's dry run (AC5) |
identityScheduleToStartMs | 60,000 | Identity-queue schedule-to-start; on timeout the workspace is prepared again |
maxReprepares | 2 | Re-prepares after a lost exec worker; the next loss is error: exec_lost |
maxChunkReportChars | 131,072 (128 KiB) | Characters of a chunk result's base64 gzip report |
heartbeatIntervalMs | 10,000 | Exec heartbeat interval |
heartbeatTimeoutMs | 30,000 | Exec heartbeat timeout |
verifySandboxMs | 7,200,000 | Verify only |
stabilityRuns, soloRuns | 3, 3 | Verify only |
maxTestIdentities | 20,000 | Verify only |
maxTreesChars | 1,500,000 | Verify only (under Temporal's 2 MiB payload limit) |
maxCreditCandidates | 50 | Verify only |
Workflow-local constants in quality.ts: ADMIT_HISTORY_LIMIT = 60, MAX_ADMIT_RETRY_MS = 60_000,
CHUNK_MARGIN_MS = 60_000.
Activity options
| Proxy | Queue | Timeouts | Retry | Cancellation |
|---|---|---|---|---|
core (proxyLocalActivities) | the workflow's worker | start-to-close 60 s | initial 1 s, backoff 2, max interval 1 min, no attempt limit (Postgres only, must eventually run) | default |
shared (prepareWorkspace) | agentforge-quality-exec | start-to-close 10 min, heartbeat 30 s | maximumAttempts: 1 | WAIT_CANCELLATION_COMPLETED |
onIdentity(identity, ms) | agentforge-quality-exec-<identity> | schedule-to-start 60 s, start-to-close per call, heartbeat 30 s | maximumAttempts: 1 | WAIT_CANCELLATION_COMPLETED |
WAIT_CANCELLATION_COMPLETED is required: with TRY_CANCEL the run would close before the container is removed. A retry
is disabled because it would land on the same identity; a lost worker is handled by preparing again.
Per-call start-to-close on the identity queue: analyze 10 min; runChunk chunkTimeoutMs + 60 s; cleanup 2 min.
End-to-end sequence
Phases in detail
1. Input validation
The workflow re-checks every wire field before any activity (validInput): runId a lowercase UUID, client 1 to 128
characters, repo owner/name, package 1 to 200 characters, headSha 40 lowercase hex, kind === 'assessment',
optional parentId a UUID. A second argument (QualityTestOptions) exists for tests only. Invalid input fails with the
non-retryable invalid_workflow_args. The admit activity parses the same input strictly again with
QualityAssessmentInput from packages/agent-types/src/quality.ts.
2. Admission (core)
- Admission runs under the Postgres advisory lock
hashtext('agentforge:quality:admission'), so two concurrent admissions cannot both see a free slot. A run id is also serialized byhashtext('agentforge:quality:run:' || id); a retried admission that already inserted the row returns it. - A run that is
busyhas no row, so nothing needs closing. The workflow waits with exponential backoff (1 s * 2^n, capped at 60 s) and, once its history exceeds 60 events, continues as new so the wait does not consume the 150-event budget. - A refused run is inserted already closed with its error code, so REST can read why. The workflow returns at once without any exec activity or close.
intentis stored asremediationfor every assessment; a verify stores its own intent.configSha256= sha256 ofJSON.stringify(policy.quality), stored on the run.
The concurrency and daily limits are described on REST and limits.
3. Prepare the workspace (shared exec queue)
prepareWorkspace runs on whichever exec worker polls agentforge-quality-exec. It fetches nothing over the network:
it reads the operator's bare mirror QUALITY_REPO_SOURCE=local:<dir> at <dir>/<owner>/<name>.git.
| Check, in order | Refusal |
|---|---|
repo has a quality policy and the mirror exists | repo_unavailable |
git cat-file -e <head>^{commit} | head_not_found |
git rev-parse --verify refs/heads/<protectedBranch>^{commit} | repo_unavailable |
git merge-base <head> <tip> equals the tip (exit 1, no common ancestor, also fails) | base_stale |
<head>:<package>/package.json exists | package_not_found |
Then it writes git archive --format=tar --mtime='1970-01-01 00:00:00 +0000' <head> (deterministic) to
<QUALITY_WORK_DIR>/<runId>/workspace.tar (directory mode 0700) with meta.json (repo, package, headSha,
baseSha = the tip, protectedBranch), and returns the worker's identity. The base is always the protected tip;
a caller can never name it (see anti-gaming).
4. Analysis (identity queue)
analyze runs the analysis container (node /opt/crucible/analyze.mjs <package> --mutate <glob>... --support <glob>...,
container limit ANALYZE_TIMEOUT_MS = 540 s inside the 10-minute activity timeout). It keeps the parsed
analysis.json in the run directory and returns the chunks with their cache keys. Any failure is
error: analysis_failed. Details: exec worker and image.
If the analysis lists more than 20 chunks, the run ends partial with finding
{ code: 'too_many_chunks', chunks, max: 20 } before any chunk runs (AC7).
5. Cache lookup (core)
cacheLookup is a single call for all chunks, so only one marker event enters history. It:
- re-checks the kill switch (
stop: disabled); - records the computed base once (
setRunRefs:base_shaset only while it is still 40 zeros, andprotected_branchstored only when base equals head, which is what later allows a baseline write); - stops with
internal_errorwhen the run is closed or a different base was already recorded; - re-checks the daily sandbox-second limits (
stop: budget_exhausted); - returns
hits[i]forcacheKeys[i]; reports stay in Postgres, only booleans enter history.
6. Chunks (identity queue)
Chunks run one at a time on the identity queue (each exec Worker has one activity slot). Limits are checked before
each chunk from the workflow's own sandbox counter; the database daily limits are checked only at admission and once in
cacheLookup (ADR 0009: per-chunk database checks would exceed the history budget).
7. Store and evaluate (core)
storeAndEvaluate decodes every report (cached ones from quality_chunk_cache, fresh ones from the payload), each
gunzipped with a cap of 8 MiB and parsed with the strict QualityChunkReport schema; any invalid one is
report_invalid (also when a cached row was evicted between lookup and store). Only parsed reports are cached
(putChunk, first write wins). Reports are merged (a function or mutant seen twice counts once), CRAP is computed per
function, and rows are upserted into quality_functions and quality_mutants.
If the run is partial (complete: false) it stops there: rows are stored, no gate and no baseline. Otherwise it
reads the baseline (readBaselineFunctions for repo, package, protected branch), runs evaluateGate with the stored
intent and the policy thresholds (timeouts count as killed, no_coverage as survived), and calls writeBaseline, which
is a no-op unless protected_branch was set (head equals the protected tip, AC15).
8. Close (finally, non-cancellable)
The finally block runs cleanup on the identity queue (2-minute timeout; errors ignored; a lost worker's tar goes with
the age sweep) and then the core close, both inside CancellationScope.nonCancellable. close writes the status,
verdict or error code, findings (capped to fit 64 KiB of jsonb, see data model), image digest and
counters once (closeRun updates only an open row), then deletes cache rows not hit for 30 days.
Run status lifecycle
queued exists only in the REST view (QualityRunView), never in quality_runs.status. inconclusive is a valid
status but is produced only by a verify (step 1); in the assessment code it would need a complete run without a
verdict, which storeAndEvaluate never returns.
Lost exec worker and exec_lost
isLoss(e) is true only for an ActivityFailure whose cause is a TimeoutFailure of type SCHEDULE_TO_START or
HEARTBEAT. Each loss re-prepares on the shared queue; after maxReprepares (2) the next loss ends the run
error: exec_lost. A re-prepared worker that has no analysis.json re-runs the analysis lazily inside runChunk
(trustedAnalysis); the same head, image and policy give the same trusted list. Concurrency 1 at admission keeps a
healthy worker from being reported lost through queueing (plan risk table, AC17).
docs/deploy-quality.md says that stopping the exec worker makes open runs end error: exec_lost. In the code, the
shared-queue prepareWorkspace proxy has no schedule-to-start timeout, so a re-prepare with no exec worker polling at all
would wait rather than fail. This was not exercised against a real Temporal server.
Cancellation (AC8)
A cancel during a chunk reaches the activity through its heartbeat; the activity aborts the container, waits for its
removal and rethrows the cancellation (heartbeating() in exec-activities.ts). The workflow's catch block sets
close = { status: 'cancelled' }, rethrows, and the finally block cleans up and closes the run cancelled. Tests:
packages/agent-workflows/test/quality.workflow.test.ts ("AC8 cancel during a chunk", "AC8 cancel during a re-prepare").
Error codes
QualityErrorCode (quality-shared.ts), stored in quality_runs.error_code when status = 'error':
| Code | Raised by |
|---|---|
disabled | admission or cacheLookup: kill switch set |
not_enabled | admission: repo has no quality policy |
budget_exhausted | admission (under lock) or cacheLookup: daily sandbox seconds used up |
exec_lost | third exec-worker loss |
base_stale | prepareWorkspace: merge base is not the protected tip |
head_not_found | prepareWorkspace: head commit not in the mirror |
package_not_found | prepareWorkspace: no package.json at the package path |
repo_unavailable | prepareWorkspace: no policy, no mirror, or no protected branch ref |
analysis_failed | analyze: container failed, timed out or wrote an invalid analysis.json |
report_invalid | storeAndEvaluate or verifyEvaluate: a report or trees payload failed its schema |
report_too_large | runChunk or the workflow: report over 128 KiB |
sandbox_failed | runChunk and verify exec activities: container failed or wrote no valid report |
internal_error | an exec activity or a child failed outright; also the default close |
Findings that accompany partial: too_many_chunks, sandbox_limit (with file), chunk_timeout (with file).
History budget (AC7)
At most 150 events for 20 chunks, under 3 MiB. How the code stays under it:
- core activities are local activities: one marker event each, inputs never enter history;
- the cache lookup is one call for all chunks;
- only chunk results (at most 128 KiB each) enter history; per chunk that is an activity's 3 events and one workflow task's 3;
- a busy admission continues as new after 60 events.
Measured: one workflow history of 20 chunks is 147 events (limit 150). Test:
packages/agent-workflows/test/quality.workflow.test.ts ("AC7 20 chunks with 128 KiB results stay under 150 history
events and 3 MiB", "AC7 a 21st chunk ends the run partial before any chunk runs").
Sources
packages/agent-workflows/src/workflows/quality.ts(qualityAssessmentWorkflow,admitted,execSession,isLoss,closeFor,finish,ADMIT_HISTORY_LIMIT,CHUNK_MARGIN_MS)packages/agent-workflows/src/workflows/quality-shared.ts(queue names,QUALITY_RUN_LIMITS,QualityErrorCode, activity contracts)apps/worker/src/quality/core-activities.ts(admit,cacheLookup,storeAndEvaluate,close,merge,capFindings,configSha256)apps/worker/src/quality/exec-activities.ts(prepareWorkspace,analyze,runChunk,cleanup,trustedAnalysis,ANALYZE_TIMEOUT_MS)packages/db/src/quality.ts(createRun,setRunRefs,writeBaseline,closeRun,PENDING_BASE_SHA,MAX_OPEN_CHILDREN)packages/agent-types/src/quality.ts(QualityAssessmentInput,QualityChunkReport,QualityStatus)packages/agent-workflows/test/quality.workflow.test.ts(test names)docs/sessions/2026-11-16-codecrucible-build/adr/0009-quality-admission-and-pending-submissions.md,SUMMARY.md(147 events)docs/deploy-quality.md(Operations and rollback)