Skip to main content

qualityVerifyWorkflow

Verify answers one question about a pushed change: did the head make the package better or at least not worse, and are the kills it claims real? It measures the base (the protected tip) and the head as two child assessments, runs the head's tests unmutated several ways, compares the stored rows, and then re-kills and controls the kills that would earn credit. It implements plan section 4.3 (steps 1 to 9) and section 4.2 (static checks), beads CC-3.1 and CC-3.2.

Code: packages/agent-workflows/src/workflows/quality.ts (qualityVerifyWorkflow, creditSteps), the pure evaluation apps/worker/src/quality/verify.ts (evaluateVerify, sampleSize), the core activity verifyEvaluate in apps/worker/src/quality/core-activities.ts, and the exec activities stabilityRuns, characterize, readTrees, soloCheck, reKill, controls in apps/worker/src/quality/exec-activities.ts.

Verify activities are not registered in the exec entry point

apps/worker/src/quality-exec-main.ts registers prepareWorkspace on the shared queue and only analyze, runChunk and cleanup on the identity queue. stabilityRuns, characterize, readTrees, soloCheck, reKill and controls are implemented and tested (fake containers in apps/worker/test/quality/exec-verify.test.ts and evasion.test.ts, the real image in scripts/quality-verify.test.sh), but a verify started against the real exec worker entry would fail at step 1. Treat end-to-end verify through the deployed entry as not built.

Overview​

The verify limit (verifySandboxMs = 7,200 s) counts the verify's own sandbox time plus its children's. checkLimit() runs after each child, after step 1, after step 3 and after every step 5 to 7 activity; reaching it ends the verify partial with finding sandbox_limit, never passed.

Intent semantics​

intent is remediation (default; the change claims to alter no behaviour) or feature. It is self-declared, so it is stored on the run and carried in the verdict (passed:feature is never shown or recorded as a remediation pass; the quality_runs_verdict_valid constraint enforces it). It decides only how changed functions are judged. Functions whose source hash is unchanged get the full checks under both intents.

Checkremediationfeature
Characterization (step 3)all base testsbase tests whose covered functions are all unchanged
Per-function delta (step 4)every function in the changed filesunchanged functions only
Changed and new functionsdelta, and new ones meet the thresholdsmeet the absolute thresholds
Summed-complexity cap (step 4)appliesdoes not apply
Steps 1, 2, 5 to 9, all static checksapplyapply

In the code, step 3's feature scope is approximated: a base test file is excused when it is in the base killedBy of any mutant of a changed function (marked shortcut in verify.ts: Stryker gives files, not tests, so a test that covers a changed function without killing any of its mutants stays in scope).

Phase 1: admission, workspace, base​

Admission stores the verify's intent on its run row. The verify counts as one active top-level run; its children are admitted under it without counting (a parent has at most 2 open children). testTimeoutMs is the policy's chunk limit (min(chunkTimeoutMs, 840 s)), also used as the limit of one unmutated test run.

Phase 2: child assessments​

Children run in sequence so one exec worker never serves two runs at once. Each child is a full assessment with its own workspace, analysis, chunks and rows. The child's own verdict (its gate against the baseline) is not the verify's verdict; verify judges base against head itself.

Child outcomeVerify outcome
completedcontinue with the child's run id
error with code Xerror with code X
cancellederror: internal_error, finding child_cancelled (side, runId)
partial or inconclusivesame status, finding child_partial or child_inconclusive (side, runId)

Phase 3: steps 1 and 3, trees​

Step 1, stability. tests.mjs <package> runs the package's tests unmutated stabilityRuns (3) times in sequence (three runs at once would compete for the sandbox's 2 CPUs and hide timing flakes). A run counts as a pass only when it wrote a valid report with exitCode 0, no failedFiles and no failed test. Fewer than 3 of 3 ends the verify inconclusive with finding { code: 'unstable', passes, runs } (AC12). executed is the set of test identities (passed or failed, not skipped) executed in every run.

Step 3, characterization. The head workspace tar with the base revision's test files laid over it (later tar entries win on extract), run once. Only failures of base test files count (a file that failed to load is { file, name: '' }). A container timeout ends the verify partial (characterization_timeout); no valid report ends it error: sandbox_failed.

Trees. readTrees returns both package trees (package-relative path to text), each capped at 5,000 files and 16 MiB, and the payload at 1,500,000 base64 characters; larger ends the verify partial (trees_too_large). FOLLOWUPS notes that readTrees does not heartbeat.

Phase 4: first evaluation (static checks, steps 2, 3, 4, 8, 9, candidates)​

verifyEvaluate (core) refuses unless the verify run is open, the repo has a policy, and both children have parent_id equal to the verify and status completed (invalid_children). It gunzips the trees (cap 64 MiB of JSON), reads both children's quality_functions and quality_mutants, and calls evaluateVerify.

Step 2 compares sets: three hidden tests plus three new ones still fail. Identities are file plus describe titles and the test title joined with > (the form of scanFile and of tests.mjs).

Credit candidates (step 6)​

A head mutant is a candidate only if it is a head kill (killed, or timeout that was also timeout at base):

Candidate classRuleretained
Newly killedbase status was not a killfalse
Retained kill of a modified test fileits base killedBy includes a test file whose content changed between base and headtrue
Sample of retained kills in changed source filesthe rest of the retained kills whose source file changed, ranked by sha256(sampleSeed + "\0" + stableId), sampleSize(n) = min(n, max(10, ceil(n / 10))) takentrue
  • sampleSeed is a uuid4() drawn by the workflow after the head is fixed, so the change under test cannot predict the sample; the same seed is passed to both evaluations.
  • The "modified test file" rule is broader than the plan in the code (marked shortcut): any content change of a base test file counts, including one that only gained new tests. More candidates only cost more re-kills.
  • More than maxCreditCandidates (50): the evaluation lists none and returns overCap = { candidates, splitBy }, where splitBy is up to 50 test files ordered by how many candidates they kill. Unless already failed, the verify ends partial with finding too_many_candidates.

If the first evaluation already failed, verify closes failed without steps 5 to 7: credit can only remove kills, so a failure without it stands.

Phase 5: steps 5 to 7 (credit)​

StepExec activityWhat runsPass rule
5 SolosoloCheckOnly the set's test files (host-written test-files.json), unmutated, soloRuns (3) times in one fresh container3 of 3 exit 0; an empty set or a name that is not a head test file earns no credit
6 Re-killreKillThe trusted list's (span, replacement) for the stable id, written as a plain source edit; only the killedBy files run, no Stryker runtimenon-zero exit is confirmed; a container timeout also counts as confirmed; exit 0 is not_reproduced
7 ControlscontrolsComment-only edit plus two drawn rewrites (drawControls with a host seed from randomBytes(16)), each in its own fresh container, against the union of the function's killedBy filesevery control must exit 0; a failure is control_killed

reKill returns not_reproduced without running anything when the source file is missing, a test name is not a head test file, the stable id is not in the trusted list, or the span does not hold the mutant's original text. A step 5 to 7 container that ends without a valid report ends the verify error: sandbox_failed. Each activity's start-to-close is testTimeoutMs + 60 s (controls: three times that).

Phase 6: second evaluation and verdict​

verifyEvaluate runs again with credit = { outcomes, controls }. It writes each re-kill outcome (confirmed or not_reproduced) as rekill on the head's mutant rows copied into the verify run's own quality_mutants (the children are closed). evaluateVerify then:

  • keeps a candidate's kill only on confirmed; every other candidate counts as survived in the gate;
  • adds rekill_not_reproduced (blocking) for a retained kill that was not_reproduced, kill_not_credited (non-blocking) for any other uncredited candidate, solo_failed (non-blocking) per failing set;
  • adds controls (non-blocking: seed and kinds), control_killed (blocking) and controls_limited (non-blocking);
  • re-runs evaluateGate with the credited counts, so CRAP credit (step 8) sees only re-kill-confirmed kills.

Verdicts and findings​

Run statusVerdictMeaning
completedpassed:remediationNo blocking finding, judged as behaviour-preserving
completedpassed:featureNo blocking finding, changed functions judged on absolute thresholds
completedfailedAt least one blocking finding (the findings name it)
partialnoneA limit was reached; neither passed nor failed
inconclusivenoneThe unmutated suite was not stable
errornoneerror_code says why
cancellednoneCancelled

Blocking verify findings: every static code (directive_added, test_skip_added, test_removed, mutation_runtime_reference, code_moved_out_of_scope, quality_support_changed, mutant_count_changed), test_not_executed, characterization_failed, rekill_not_reproduced, control_killed, and the gate's survivors_rose, score_fell, below_threshold, crap_over, package_score_fell, complexity_cap. Non-blocking: kill_not_credited, solo_failed, controls, controls_limited, crap_credit_denied.

The plan's CI mapping (bead CC-4.2, not built): every verdict except passed:* (failed, partial, inconclusive, error) becomes a failing quality-verify commit status.

Cost bound​

At most 50 re-kill containers, plus one solo container per distinct killedBy set and three control containers per function, each running a few test files, all inside the 7,200 s verify limit (which also counts both children). Test: packages/agent-workflows/test/quality-verify.workflow.test.ts ("cost bound: steps 5 to 7 stop at the 7,200 s limit and end partial, never passed").

Tests​

BehaviourTest
Workflow paths with fake activities (flaky suite, limits, children, cancels, over-cap)packages/agent-workflows/test/quality-verify.workflow.test.ts
AC13 intent, AC14 deltas, AC15 no new debt, step 2, step 9, step 6 sample, credit appliedapps/worker/test/quality/verify.test.ts
Exec steps 1, 3, trees, tar helpersapps/worker/test/quality/exec-verify.test.ts
AC10 evasion fixtures, steps 5 to 7 exec contractapps/worker/test/quality/evasion.test.ts
Solo, re-kill, controls in the real imagescripts/quality-verify.test.sh (solo 3 of 3, re-kill confirmed with the new test and not_reproduced with the base tests, controls pass, in 195 s)

Sources​

  • packages/agent-workflows/src/workflows/quality.ts (qualityVerifyWorkflow, creditSteps, TEST_RUN_MARGIN_MS)
  • packages/agent-workflows/src/workflows/quality-shared.ts (QualityVerifyEvaluateArgs, QualityCreditCandidate, QualityVerifyCredit, exec contracts, QUALITY_RUN_LIMITS)
  • apps/worker/src/quality/verify.ts (evaluateVerify, sampleSize)
  • apps/worker/src/quality/core-activities.ts (verifyEvaluate, VerifyArgs)
  • apps/worker/src/quality/exec-activities.ts (stabilityRuns, characterize, readTrees, soloCheck, reKill, controls, materialize, runFiles)
  • apps/worker/src/quality-exec-main.ts (registered activities)
  • packages/quality-core/src/gate.ts, static.ts, rewrites.ts
  • packages/db/drizzle/0006_quality.sql (quality_runs_verdict_valid)
  • .omc/plans/codecrucible-mutant-hunter.md (sections 4.2, 4.3, CC-4.2)
  • docs/deploy-quality.md (local evidence), docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md