Anti-gaming: the threat model of the gate
A quality gate that an agent must pass is a target. CodeCrucible's design assumes the change under test is written by an adversary taking the cheapest path to a green verify, including deliberate cheat tests and a forged report written from test code. Plan section 4 is the source; every defence below is named with the code that implements it and the test or fixture that proves it.
Threat model
| In the model | Out of the model |
|---|---|
| An agent or a person writing the head commit: source, tests, helpers, config, setup files | A compromised exec host |
Test code that runs inside the test container and can write anything there, including a forged mutation.json or tests.json | The operator's policy files and the quality image |
| A caller choosing request fields: repo, package, head SHA, intent | The core worker and Postgres |
What the change under test cannot influence (plan 4.1)
| Asset | Enforcement | Code |
|---|---|---|
| Thresholds, globs, mutators, timeouts | Only in the operator's FactoryRepoPolicy.quality (strict zod) | packages/agent-types/src/factory.ts (QualityPolicy) |
| Stryker and Vitest config | Generated by the image scripts into /workspace/.agentforge/quality/ (a forbidden patch path) and passed by explicit path; the list run uses a host-only minimal Vitest config | apps/worker/quality-image/analyze.mjs, run.mjs; QUALITY_CONFIG_DIR in apps/worker/src/sandbox/contract.ts |
| Mutant list and ids | From the analysis container, which runs no repository test | analyze.mjs, buildAnalysis, assignStableIds |
| The base | Computed by the exec worker as git merge-base <head> <protected tip> and required to equal that tip | prepareWorkspace in apps/worker/src/quality/exec-activities.ts |
| Support, config and setup files | Any change is the blocking finding quality_support_changed | checkStatic rule 6 |
| Dependencies | node_modules of every workspace package replaced by links into the image | linkImageDependencies in common.mjs |
| Cache, baseline, verdict | Postgres, written only by core activities | apps/worker/src/quality/core-activities.ts, packages/db/src/quality.ts |
run.mjs and tests.mjs run the repository's tests with the package's own Vitest config from the head tree (the
comments say so). The defence is that any change to such a config file is quality_support_changed, so a passing verify
measured the base config. The plan's wording ("the head tree's config files are never loaded") is stronger than the
code.
Trusted list versus untrusted evidence
Everything the test container reports is evidence, not fact:
- a mutant absent from the trusted list is dropped (
onlyInReal), so a forgedthrow 0mutant earns nothing; - a trusted mutant missing from the report counts as
survived; killedByonly chooses which test files run in a re-kill; it never grants credit by itself;- coverage is capped by the share of mutants that were reached (
capCoverage), and CRAP credit needs the mutation threshold; tests.json(executed identities, pass or fail) is a tripwire: a forged executed list is caught because the hidden test's file is modified, so every retained kill of that file is re-killed.
Static tripwires (plan 4.2)
Seven blocking rules in checkStatic, run in verify over both trees. Rules 1, 2 and 4 are tripwires that catch careless
changes; the defence against deliberate evasion is the verify steps.
| Rule | Code | Defeated by (stated) | Real defence |
|---|---|---|---|
| 1 | directive_added | building the directive dynamically | step 6 re-kill |
| 2 | test_skip_added | t['sk'+'ip'], describe.each([]), if (false) it(...) | step 2 executed identities |
| 3 | test_removed | dynamic titles | step 2 |
| 4 | mutation_runtime_reference | key enumeration, built names | step 6 (no runtime in the re-kill) |
| 5 | code_moved_out_of_scope | renaming and editing while moving | (stated limit) |
| 6 | quality_support_changed | none: any change blocks | separate human-reviewed merge |
| 7 | mutant_count_changed | none for unchanged files |
Named gaming paths
The plan's table (section 4.4) with the defence as built and the test or fixture that proves it. AC10 cases are run "with rules 1, 2 and 4 disabled", i.e. their assertions do not rely on the tripwires.
| Gaming path | Defence | Proof |
|---|---|---|
| Cheat test detecting the mutation runtime (key enumeration, built names) | No Stryker runtime in the step 6 re-kill: the materialized plain edit runs only the killedBy files, so the cheat passes and earns no credit | AC10 (a), (b): apps/worker/test/quality/evasion.test.ts ("a test detecting the mutation runtime earns zero credited kills"); S1 spike (iii) exit 0 |
Forged report: fake mutants, throw 0 or syntax-error replacements, forged "still killed" | Only trusted-list tuples are credited; retained kills of modified test files are all re-killed | AC10 (g): "a forged throw 0 mutant is dropped; the kept kill of the weakened file is not reproduced: fails, 20 of 20" |
Forged executed-identity list or dry-run result (if (false) it(...) reported as run) | The hidden test's file is modified, so step 6 re-kills all its retained kills and fails | AC10 (h): "its 3 of 300 retained kills are re-killed and fail, 20 of 20" |
| A weakened base: an unmerged commit that removed assertions | The base is computed, never supplied; the request has no base field | AC10 (i), (k): "a head on an unmerged branch ... is measured against the protected tip", "a caller cannot name a base" |
| A head branched from an old protected commit and never rebased | Merge base must equal the protected tip, else base_stale | AC10 (l): "a head branched from an old protected commit and never rebased is base_stale" |
Declaring feature to skip checks | Unchanged functions keep every check; the verdict carries the intent; the DB constraint ties a pass to the stored intent | AC13: apps/worker/test/quality/verify.test.ts ("AC13 intent: price() and tax() in one file") |
| Helper weakened inside an existing test file | That file is modified, so all its retained kills are re-killed | step 6 rule, verify.test.ts ("every retained kill of a modified test file is a candidate") |
| Test hashing source text | Comment-only control | AC10 (c): evasion.test.ts ("tests hashing the source get control_killed in 20 of 20 draws") |
Test hashing fn.toString(), the AST, or a normalised token stream | Rewrite controls drawn per run with a host seed, first one structural | AC10 (d), (j): same test; a fully normalising hash test is a stated limit |
| Test that fails alone or flakes, sole killer of many mutants | Step 5 solo check: 3 of 3 alone, else no credit for any mutant of that set | AC10 (f): "its killedBy set fails the solo check (0 of 3); with its setup file it passes 3 of 3" |
Hidden tests (t['sk'+'ip'], describe.each([]), if (false) it('name')), with or without padding tests | Step 2: every base and static head identity must be executed (sets, not totals) | AC10 (e): "if (false) it('doubles') plus it('pads'): the base identity was not executed"; verify.test.ts step 2 |
Weakened helper, new vite.config.ts, new setup file | quality_support_changed blocks; it blocks a merge only if verify is a required CI status (CC-4.2, not built) | AC9 fixtures in packages/quality-core/test/static.test.ts |
| Coverage without assertions to lower CRAP | Step 8: coverage counts only when the function meets the mutation threshold; credit uses re-killed kills only | AC14: verify.test.ts ("higher coverage with a mutation score below the threshold earns no CRAP credit", "CRAP credit counts only re-kill-confirmed kills") |
| Splitting a complex function into tiny ones | complexity_cap on summed cyclomatic of changed files (remediation) | AC14: "a split raising summed complexity 20 percent fails; one lowering it passes" |
Fewer mutants through lookup tables, Math.sign, data-driven code | Plan: a non-blocking review finding at a 30 percent fall | Not implemented (stated limit in packages/quality-core/README.md) |
| Assertion removed so a test passes | Step 3 characterization (base tests against head source) and step 6 | AC11 honest path is not separately automated; verify.test.ts AC13 Y case and quality-verify.test.sh (not_reproduced with the base tests) |
A new Timeout instead of a kill | Step 9: a new timeout counts as survived and is never a candidate | verify.test.ts ("step 9: a new Timeout counts as survived") |
| Too many candidates to re-kill | Over 50 ends partial, never passed, naming the test files to split by | packages/agent-workflows/test/quality-verify.workflow.test.ts ("more than 50 candidates end partial") |
The AC10 matrix
The fixtures live in apps/worker/test/quality/evasion/fixtures.ts: a fixture package packages/p with price() and
its honest test, plus the cheat test files. evasion.test.ts runs the real exec activities (soloCheck, reKill,
controls, prepareWorkspace) on a git mirror of that package with an in-process stand-in for the test container, and
the pure evaluateVerify on the resulting credit. scripts/quality-verify.test.sh runs solo check, re-kill and
controls in the real quality image.
Seeds and unpredictability
| Draw | Who draws | When | Recorded |
|---|---|---|---|
| Step 6 sample of retained kills in changed files | workflow uuid4() (sampleSeed) | after the head is fixed, before the first evaluation | passed to both evaluations |
| Step 7 control rewrites | exec worker randomBytes(16) per function | at control time | seed in the controls finding, after the run |
The rewrite set is in this repository; only the draw is unpredictable. A test that fully normalises semantics-preserving rewrites survives the controls: a stated limit.
Stated limits
From docs/deploy-quality.md, packages/quality-core/README.md and the code's shortcut comments:
- Report forgery from test code is mitigated by re-kill and controls, not prevented.
- Coverage is untrusted and capped by the mutation results.
- Equivalent mutants are not excluded in v1; the policy threshold is where the operator leaves room for them.
!(a >= b)differs froma < bfor NaN orundefinedoperands; a test that pins that case getscontrol_killed.- A retained Stryker timeout with no killers costs credit (FOLLOWUPS).
- The
featurescope of step 3 uses killing test files, not covering tests (verify.tsshortcut). - Any content change of a base test file makes all its retained kills candidates (broader than the plan,
verify.tsshortcut). - Rules 1, 2 and 4 are tripwires; a function renamed and edited while moved out of scope is not matched by rule 5.
- The executed-identity list and the dry-run result come from the test container (plan changelog B).
- Enforcement in CI (required status, label approval for
feature) is not built (CC-4.2), so verify is voluntary.
Sources
.omc/plans/codecrucible-mutant-hunter.md(sections 2, 3 "Two kinds of container", 4.1 to 4.5, 5 AC9 to AC14, changelogs)apps/worker/src/quality/verify.ts(evaluateVerify, shortcut comments)apps/worker/src/quality/exec-activities.ts(prepareWorkspace,buildChunkReport,capCoverage,reKill,controls,soloCheck)apps/worker/quality-image/analyze.mjs,run.mjs,tests.mjs,common.mjspackages/quality-core/src/report.ts(joinTrustedList),static.ts(checkStatic),rewrites.ts(drawControls),gate.tspackages/quality-core/README.md(stated limits)apps/worker/test/quality/evasion.test.ts,evasion/fixtures.ts,verify.test.ts;packages/agent-workflows/test/quality-verify.workflow.test.ts;packages/quality-core/test/static.test.tsdocs/p2-findings.md(CodeCrucible Phase 0 spikes, S1)docs/deploy-quality.md(Operations and rollback, stated limits)docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md