qualityVerifyWorkflow
Verify answers one question about a pushed change: did the head make the package better or at least not worse, and are the kills it claims real? It measures the base (the protected tip) and the head as two child assessments, runs the head's tests unmutated several ways, compares the stored rows, and then re-kills and controls the kills that would earn credit. It implements plan section 4.3 (steps 1 to 9) and section 4.2 (static checks), beads CC-3.1 and CC-3.2.
Code: packages/agent-workflows/src/workflows/quality.ts (qualityVerifyWorkflow, creditSteps), the pure evaluation
apps/worker/src/quality/verify.ts (evaluateVerify, sampleSize), the core activity verifyEvaluate in
apps/worker/src/quality/core-activities.ts, and the exec activities stabilityRuns, characterize, readTrees,
soloCheck, reKill, controls in apps/worker/src/quality/exec-activities.ts.
apps/worker/src/quality-exec-main.ts registers prepareWorkspace on the shared queue and only analyze, runChunk
and cleanup on the identity queue. stabilityRuns, characterize, readTrees, soloCheck, reKill and controls
are implemented and tested (fake containers in apps/worker/test/quality/exec-verify.test.ts and evasion.test.ts, the
real image in scripts/quality-verify.test.sh), but a verify started against the real exec worker entry would fail at
step 1. Treat end-to-end verify through the deployed entry as not built.
Overview
The verify limit (verifySandboxMs = 7,200 s) counts the verify's own sandbox time plus its children's.
checkLimit() runs after each child, after step 1, after step 3 and after every step 5 to 7 activity; reaching it ends
the verify partial with finding sandbox_limit, never passed.
Intent semantics
intent is remediation (default; the change claims to alter no behaviour) or feature. It is self-declared, so it
is stored on the run and carried in the verdict (passed:feature is never shown or recorded as a remediation pass; the
quality_runs_verdict_valid constraint enforces it). It decides only how changed functions are judged. Functions
whose source hash is unchanged get the full checks under both intents.
| Check | remediation | feature |
|---|---|---|
| Characterization (step 3) | all base tests | base tests whose covered functions are all unchanged |
| Per-function delta (step 4) | every function in the changed files | unchanged functions only |
| Changed and new functions | delta, and new ones meet the thresholds | meet the absolute thresholds |
| Summed-complexity cap (step 4) | applies | does not apply |
| Steps 1, 2, 5 to 9, all static checks | apply | apply |
In the code, step 3's feature scope is approximated: a base test file is excused when it is in the base killedBy of
any mutant of a changed function (marked shortcut in verify.ts: Stryker gives files, not tests, so a test that covers
a changed function without killing any of its mutants stays in scope).
Phase 1: admission, workspace, base
Admission stores the verify's intent on its run row. The verify counts as one active top-level run; its children are
admitted under it without counting (a parent has at most 2 open children). testTimeoutMs is the policy's chunk limit
(min(chunkTimeoutMs, 840 s)), also used as the limit of one unmutated test run.
Phase 2: child assessments
Children run in sequence so one exec worker never serves two runs at once. Each child is a full assessment with its own workspace, analysis, chunks and rows. The child's own verdict (its gate against the baseline) is not the verify's verdict; verify judges base against head itself.
| Child outcome | Verify outcome |
|---|---|
completed | continue with the child's run id |
error with code X | error with code X |
cancelled | error: internal_error, finding child_cancelled (side, runId) |
partial or inconclusive | same status, finding child_partial or child_inconclusive (side, runId) |
Phase 3: steps 1 and 3, trees
Step 1, stability. tests.mjs <package> runs the package's tests unmutated stabilityRuns (3) times in sequence
(three runs at once would compete for the sandbox's 2 CPUs and hide timing flakes). A run counts as a pass only when it
wrote a valid report with exitCode 0, no failedFiles and no failed test. Fewer than 3 of 3 ends the verify
inconclusive with finding { code: 'unstable', passes, runs } (AC12). executed is the set of test identities
(passed or failed, not skipped) executed in every run.
Step 3, characterization. The head workspace tar with the base revision's test files laid over it (later tar
entries win on extract), run once. Only failures of base test files count (a file that failed to load is
{ file, name: '' }). A container timeout ends the verify partial (characterization_timeout); no valid report ends it
error: sandbox_failed.
Trees. readTrees returns both package trees (package-relative path to text), each capped at 5,000 files and
16 MiB, and the payload at 1,500,000 base64 characters; larger ends the verify partial (trees_too_large). FOLLOWUPS
notes that readTrees does not heartbeat.
Phase 4: first evaluation (static checks, steps 2, 3, 4, 8, 9, candidates)
verifyEvaluate (core) refuses unless the verify run is open, the repo has a policy, and both children have
parent_id equal to the verify and status completed (invalid_children). It gunzips the trees (cap 64 MiB of JSON),
reads both children's quality_functions and quality_mutants, and calls evaluateVerify.
Step 2 compares sets: three hidden tests plus three new ones still fail. Identities are file plus describe titles and
the test title joined with > (the form of scanFile and of tests.mjs).
Credit candidates (step 6)
A head mutant is a candidate only if it is a head kill (killed, or timeout that was also timeout at base):
| Candidate class | Rule | retained |
|---|---|---|
| Newly killed | base status was not a kill | false |
| Retained kill of a modified test file | its base killedBy includes a test file whose content changed between base and head | true |
| Sample of retained kills in changed source files | the rest of the retained kills whose source file changed, ranked by sha256(sampleSeed + "\0" + stableId), sampleSize(n) = min(n, max(10, ceil(n / 10))) taken | true |
sampleSeedis auuid4()drawn by the workflow after the head is fixed, so the change under test cannot predict the sample; the same seed is passed to both evaluations.- The "modified test file" rule is broader than the plan in the code (marked
shortcut): any content change of a base test file counts, including one that only gained new tests. More candidates only cost more re-kills. - More than
maxCreditCandidates(50): the evaluation lists none and returnsoverCap = { candidates, splitBy }, wheresplitByis up to 50 test files ordered by how many candidates they kill. Unless already failed, the verify endspartialwith findingtoo_many_candidates.
If the first evaluation already failed, verify closes failed without steps 5 to 7: credit can only remove kills, so a
failure without it stands.
Phase 5: steps 5 to 7 (credit)
| Step | Exec activity | What runs | Pass rule |
|---|---|---|---|
| 5 Solo | soloCheck | Only the set's test files (host-written test-files.json), unmutated, soloRuns (3) times in one fresh container | 3 of 3 exit 0; an empty set or a name that is not a head test file earns no credit |
| 6 Re-kill | reKill | The trusted list's (span, replacement) for the stable id, written as a plain source edit; only the killedBy files run, no Stryker runtime | non-zero exit is confirmed; a container timeout also counts as confirmed; exit 0 is not_reproduced |
| 7 Controls | controls | Comment-only edit plus two drawn rewrites (drawControls with a host seed from randomBytes(16)), each in its own fresh container, against the union of the function's killedBy files | every control must exit 0; a failure is control_killed |
reKill returns not_reproduced without running anything when the source file is missing, a test name is not a head
test file, the stable id is not in the trusted list, or the span does not hold the mutant's original text. A step 5 to 7
container that ends without a valid report ends the verify error: sandbox_failed. Each activity's start-to-close is
testTimeoutMs + 60 s (controls: three times that).
Phase 6: second evaluation and verdict
verifyEvaluate runs again with credit = { outcomes, controls }. It writes each re-kill outcome
(confirmed or not_reproduced) as rekill on the head's mutant rows copied into the verify run's own
quality_mutants (the children are closed). evaluateVerify then:
- keeps a candidate's kill only on
confirmed; every other candidate counts as survived in the gate; - adds
rekill_not_reproduced(blocking) for a retained kill that wasnot_reproduced,kill_not_credited(non-blocking) for any other uncredited candidate,solo_failed(non-blocking) per failing set; - adds
controls(non-blocking: seed and kinds),control_killed(blocking) andcontrols_limited(non-blocking); - re-runs
evaluateGatewith the credited counts, so CRAP credit (step 8) sees only re-kill-confirmed kills.
Verdicts and findings
| Run status | Verdict | Meaning |
|---|---|---|
completed | passed:remediation | No blocking finding, judged as behaviour-preserving |
completed | passed:feature | No blocking finding, changed functions judged on absolute thresholds |
completed | failed | At least one blocking finding (the findings name it) |
partial | none | A limit was reached; neither passed nor failed |
inconclusive | none | The unmutated suite was not stable |
error | none | error_code says why |
cancelled | none | Cancelled |
Blocking verify findings: every static code (directive_added, test_skip_added, test_removed,
mutation_runtime_reference, code_moved_out_of_scope, quality_support_changed, mutant_count_changed),
test_not_executed, characterization_failed, rekill_not_reproduced, control_killed, and the gate's
survivors_rose, score_fell, below_threshold, crap_over, package_score_fell, complexity_cap. Non-blocking:
kill_not_credited, solo_failed, controls, controls_limited, crap_credit_denied.
The plan's CI mapping (bead CC-4.2, not built): every verdict except passed:* (failed, partial, inconclusive,
error) becomes a failing quality-verify commit status.
Cost bound
At most 50 re-kill containers, plus one solo container per distinct killedBy set and three control containers per
function, each running a few test files, all inside the 7,200 s verify limit (which also counts both children). Test:
packages/agent-workflows/test/quality-verify.workflow.test.ts ("cost bound: steps 5 to 7 stop at the 7,200 s limit and
end partial, never passed").
Tests
| Behaviour | Test |
|---|---|
| Workflow paths with fake activities (flaky suite, limits, children, cancels, over-cap) | packages/agent-workflows/test/quality-verify.workflow.test.ts |
| AC13 intent, AC14 deltas, AC15 no new debt, step 2, step 9, step 6 sample, credit applied | apps/worker/test/quality/verify.test.ts |
| Exec steps 1, 3, trees, tar helpers | apps/worker/test/quality/exec-verify.test.ts |
| AC10 evasion fixtures, steps 5 to 7 exec contract | apps/worker/test/quality/evasion.test.ts |
| Solo, re-kill, controls in the real image | scripts/quality-verify.test.sh (solo 3 of 3, re-kill confirmed with the new test and not_reproduced with the base tests, controls pass, in 195 s) |
Sources
packages/agent-workflows/src/workflows/quality.ts(qualityVerifyWorkflow,creditSteps,TEST_RUN_MARGIN_MS)packages/agent-workflows/src/workflows/quality-shared.ts(QualityVerifyEvaluateArgs,QualityCreditCandidate,QualityVerifyCredit, exec contracts,QUALITY_RUN_LIMITS)apps/worker/src/quality/verify.ts(evaluateVerify,sampleSize)apps/worker/src/quality/core-activities.ts(verifyEvaluate,VerifyArgs)apps/worker/src/quality/exec-activities.ts(stabilityRuns,characterize,readTrees,soloCheck,reKill,controls,materialize,runFiles)apps/worker/src/quality-exec-main.ts(registered activities)packages/quality-core/src/gate.ts,static.ts,rewrites.tspackages/db/drizzle/0006_quality.sql(quality_runs_verdict_valid).omc/plans/codecrucible-mutant-hunter.md(sections 4.2, 4.3, CC-4.2)docs/deploy-quality.md(local evidence),docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md