Skip to main content

qualityAssessmentWorkflow

One assessment measures one package of one repository at one head commit, gates it against the stored baseline of the protected branch, and (only when the head is the protected tip) replaces that baseline. It is also the child workflow a verify runs twice, once for the base and once for the head (see verify workflow).

Code: packages/agent-workflows/src/workflows/quality.ts (qualityAssessmentWorkflow), contract in quality-shared.ts, core activities in apps/worker/src/quality/core-activities.ts, exec activities in apps/worker/src/quality/exec-activities.ts (bead CC-2.5, CC-2.4).

note

The workflow result is informational. Postgres holds the verdict; REST reads Postgres, never a workflow result (plan section 3, "Verdict").

Names and queues​

Constant (quality-shared.ts)ValueUsed for
QUALITY_TASK_QUEUEagentforge-qualityWorkflows and core activities (core worker)
QUALITY_EXEC_TASK_QUEUEagentforge-quality-execShared exec queue: prepareWorkspace only
qualityExecIdentityQueue(identity)agentforge-quality-exec-<identity>Everything after prepareWorkspace, on the worker that prepared the workspace
QUALITY_ASSESSMENT_WORKFLOW_TYPEqualityAssessmentWorkflowWorkflow type
QUALITY_VERIFY_WORKFLOW_TYPEqualityVerifyWorkflowWorkflow type
qualityWorkflowId(runId)quality-<runId>Workflow id; also quality_runs.workflow_id

The identity queue exists so that two pollers during a rolling deploy cannot split one run: the workspace tar lives on the disk of the worker that prepared it.

Limits (QUALITY_RUN_LIMITS)​

KeyValueMeaning
maxChunks20More chunks end the run partial (AC7)
assessmentSandboxMs3,600,000 (3,600 s)Sandbox time of one assessment; reaching it ends partial
chunkMaxMs840,000 (840 s)One chunk including Stryker's dry run (AC5)
identityScheduleToStartMs60,000Identity-queue schedule-to-start; on timeout the workspace is prepared again
maxReprepares2Re-prepares after a lost exec worker; the next loss is error: exec_lost
maxChunkReportChars131,072 (128 KiB)Characters of a chunk result's base64 gzip report
heartbeatIntervalMs10,000Exec heartbeat interval
heartbeatTimeoutMs30,000Exec heartbeat timeout
verifySandboxMs7,200,000Verify only
stabilityRuns, soloRuns3, 3Verify only
maxTestIdentities20,000Verify only
maxTreesChars1,500,000Verify only (under Temporal's 2 MiB payload limit)
maxCreditCandidates50Verify only

Workflow-local constants in quality.ts: ADMIT_HISTORY_LIMIT = 60, MAX_ADMIT_RETRY_MS = 60_000, CHUNK_MARGIN_MS = 60_000.

Activity options​

ProxyQueueTimeoutsRetryCancellation
core (proxyLocalActivities)the workflow's workerstart-to-close 60 sinitial 1 s, backoff 2, max interval 1 min, no attempt limit (Postgres only, must eventually run)default
shared (prepareWorkspace)agentforge-quality-execstart-to-close 10 min, heartbeat 30 smaximumAttempts: 1WAIT_CANCELLATION_COMPLETED
onIdentity(identity, ms)agentforge-quality-exec-<identity>schedule-to-start 60 s, start-to-close per call, heartbeat 30 smaximumAttempts: 1WAIT_CANCELLATION_COMPLETED

WAIT_CANCELLATION_COMPLETED is required: with TRY_CANCEL the run would close before the container is removed. A retry is disabled because it would land on the same identity; a lost worker is handled by preparing again.

Per-call start-to-close on the identity queue: analyze 10 min; runChunk chunkTimeoutMs + 60 s; cleanup 2 min.

End-to-end sequence​

Phases in detail​

1. Input validation​

The workflow re-checks every wire field before any activity (validInput): runId a lowercase UUID, client 1 to 128 characters, repo owner/name, package 1 to 200 characters, headSha 40 lowercase hex, kind === 'assessment', optional parentId a UUID. A second argument (QualityTestOptions) exists for tests only. Invalid input fails with the non-retryable invalid_workflow_args. The admit activity parses the same input strictly again with QualityAssessmentInput from packages/agent-types/src/quality.ts.

2. Admission (core)​

  • Admission runs under the Postgres advisory lock hashtext('agentforge:quality:admission'), so two concurrent admissions cannot both see a free slot. A run id is also serialized by hashtext('agentforge:quality:run:' || id); a retried admission that already inserted the row returns it.
  • A run that is busy has no row, so nothing needs closing. The workflow waits with exponential backoff (1 s * 2^n, capped at 60 s) and, once its history exceeds 60 events, continues as new so the wait does not consume the 150-event budget.
  • A refused run is inserted already closed with its error code, so REST can read why. The workflow returns at once without any exec activity or close.
  • intent is stored as remediation for every assessment; a verify stores its own intent.
  • configSha256 = sha256 of JSON.stringify(policy.quality), stored on the run.

The concurrency and daily limits are described on REST and limits.

3. Prepare the workspace (shared exec queue)​

prepareWorkspace runs on whichever exec worker polls agentforge-quality-exec. It fetches nothing over the network: it reads the operator's bare mirror QUALITY_REPO_SOURCE=local:<dir> at <dir>/<owner>/<name>.git.

Check, in orderRefusal
repo has a quality policy and the mirror existsrepo_unavailable
git cat-file -e <head>^{commit}head_not_found
git rev-parse --verify refs/heads/<protectedBranch>^{commit}repo_unavailable
git merge-base <head> <tip> equals the tip (exit 1, no common ancestor, also fails)base_stale
<head>:<package>/package.json existspackage_not_found

Then it writes git archive --format=tar --mtime='1970-01-01 00:00:00 +0000' <head> (deterministic) to <QUALITY_WORK_DIR>/<runId>/workspace.tar (directory mode 0700) with meta.json (repo, package, headSha, baseSha = the tip, protectedBranch), and returns the worker's identity. The base is always the protected tip; a caller can never name it (see anti-gaming).

4. Analysis (identity queue)​

analyze runs the analysis container (node /opt/crucible/analyze.mjs <package> --mutate <glob>... --support <glob>..., container limit ANALYZE_TIMEOUT_MS = 540 s inside the 10-minute activity timeout). It keeps the parsed analysis.json in the run directory and returns the chunks with their cache keys. Any failure is error: analysis_failed. Details: exec worker and image.

If the analysis lists more than 20 chunks, the run ends partial with finding { code: 'too_many_chunks', chunks, max: 20 } before any chunk runs (AC7).

5. Cache lookup (core)​

cacheLookup is a single call for all chunks, so only one marker event enters history. It:

  1. re-checks the kill switch (stop: disabled);
  2. records the computed base once (setRunRefs: base_sha set only while it is still 40 zeros, and protected_branch stored only when base equals head, which is what later allows a baseline write);
  3. stops with internal_error when the run is closed or a different base was already recorded;
  4. re-checks the daily sandbox-second limits (stop: budget_exhausted);
  5. returns hits[i] for cacheKeys[i]; reports stay in Postgres, only booleans enter history.

6. Chunks (identity queue)​

Chunks run one at a time on the identity queue (each exec Worker has one activity slot). Limits are checked before each chunk from the workflow's own sandbox counter; the database daily limits are checked only at admission and once in cacheLookup (ADR 0009: per-chunk database checks would exceed the history budget).

7. Store and evaluate (core)​

storeAndEvaluate decodes every report (cached ones from quality_chunk_cache, fresh ones from the payload), each gunzipped with a cap of 8 MiB and parsed with the strict QualityChunkReport schema; any invalid one is report_invalid (also when a cached row was evicted between lookup and store). Only parsed reports are cached (putChunk, first write wins). Reports are merged (a function or mutant seen twice counts once), CRAP is computed per function, and rows are upserted into quality_functions and quality_mutants.

If the run is partial (complete: false) it stops there: rows are stored, no gate and no baseline. Otherwise it reads the baseline (readBaselineFunctions for repo, package, protected branch), runs evaluateGate with the stored intent and the policy thresholds (timeouts count as killed, no_coverage as survived), and calls writeBaseline, which is a no-op unless protected_branch was set (head equals the protected tip, AC15).

8. Close (finally, non-cancellable)​

The finally block runs cleanup on the identity queue (2-minute timeout; errors ignored; a lost worker's tar goes with the age sweep) and then the core close, both inside CancellationScope.nonCancellable. close writes the status, verdict or error code, findings (capped to fit 64 KiB of jsonb, see data model), image digest and counters once (closeRun updates only an open row), then deletes cache rows not hit for 30 days.

Run status lifecycle​

queued exists only in the REST view (QualityRunView), never in quality_runs.status. inconclusive is a valid status but is produced only by a verify (step 1); in the assessment code it would need a complete run without a verdict, which storeAndEvaluate never returns.

Lost exec worker and exec_lost​

isLoss(e) is true only for an ActivityFailure whose cause is a TimeoutFailure of type SCHEDULE_TO_START or HEARTBEAT. Each loss re-prepares on the shared queue; after maxReprepares (2) the next loss ends the run error: exec_lost. A re-prepared worker that has no analysis.json re-runs the analysis lazily inside runChunk (trustedAnalysis); the same head, image and policy give the same trusted list. Concurrency 1 at admission keeps a healthy worker from being reported lost through queueing (plan risk table, AC17).

Unverified

docs/deploy-quality.md says that stopping the exec worker makes open runs end error: exec_lost. In the code, the shared-queue prepareWorkspace proxy has no schedule-to-start timeout, so a re-prepare with no exec worker polling at all would wait rather than fail. This was not exercised against a real Temporal server.

Cancellation (AC8)​

A cancel during a chunk reaches the activity through its heartbeat; the activity aborts the container, waits for its removal and rethrows the cancellation (heartbeating() in exec-activities.ts). The workflow's catch block sets close = { status: 'cancelled' }, rethrows, and the finally block cleans up and closes the run cancelled. Tests: packages/agent-workflows/test/quality.workflow.test.ts ("AC8 cancel during a chunk", "AC8 cancel during a re-prepare").

Error codes​

QualityErrorCode (quality-shared.ts), stored in quality_runs.error_code when status = 'error':

CodeRaised by
disabledadmission or cacheLookup: kill switch set
not_enabledadmission: repo has no quality policy
budget_exhaustedadmission (under lock) or cacheLookup: daily sandbox seconds used up
exec_lostthird exec-worker loss
base_staleprepareWorkspace: merge base is not the protected tip
head_not_foundprepareWorkspace: head commit not in the mirror
package_not_foundprepareWorkspace: no package.json at the package path
repo_unavailableprepareWorkspace: no policy, no mirror, or no protected branch ref
analysis_failedanalyze: container failed, timed out or wrote an invalid analysis.json
report_invalidstoreAndEvaluate or verifyEvaluate: a report or trees payload failed its schema
report_too_largerunChunk or the workflow: report over 128 KiB
sandbox_failedrunChunk and verify exec activities: container failed or wrote no valid report
internal_erroran exec activity or a child failed outright; also the default close

Findings that accompany partial: too_many_chunks, sandbox_limit (with file), chunk_timeout (with file).

History budget (AC7)​

At most 150 events for 20 chunks, under 3 MiB. How the code stays under it:

  • core activities are local activities: one marker event each, inputs never enter history;
  • the cache lookup is one call for all chunks;
  • only chunk results (at most 128 KiB each) enter history; per chunk that is an activity's 3 events and one workflow task's 3;
  • a busy admission continues as new after 60 events.

Measured: one workflow history of 20 chunks is 147 events (limit 150). Test: packages/agent-workflows/test/quality.workflow.test.ts ("AC7 20 chunks with 128 KiB results stay under 150 history events and 3 MiB", "AC7 a 21st chunk ends the run partial before any chunk runs").

Sources​

  • packages/agent-workflows/src/workflows/quality.ts (qualityAssessmentWorkflow, admitted, execSession, isLoss, closeFor, finish, ADMIT_HISTORY_LIMIT, CHUNK_MARGIN_MS)
  • packages/agent-workflows/src/workflows/quality-shared.ts (queue names, QUALITY_RUN_LIMITS, QualityErrorCode, activity contracts)
  • apps/worker/src/quality/core-activities.ts (admit, cacheLookup, storeAndEvaluate, close, merge, capFindings, configSha256)
  • apps/worker/src/quality/exec-activities.ts (prepareWorkspace, analyze, runChunk, cleanup, trustedAnalysis, ANALYZE_TIMEOUT_MS)
  • packages/db/src/quality.ts (createRun, setRunRefs, writeBaseline, closeRun, PENDING_BASE_SHA, MAX_OPEN_CHILDREN)
  • packages/agent-types/src/quality.ts (QualityAssessmentInput, QualityChunkReport, QualityStatus)
  • packages/agent-workflows/test/quality.workflow.test.ts (test names)
  • docs/sessions/2026-11-16-codecrucible-build/adr/0009-quality-admission-and-pending-submissions.md, SUMMARY.md (147 events)
  • docs/deploy-quality.md (Operations and rollback)