Skip to main content

Anti-gaming: the threat model of the gate

A quality gate that an agent must pass is a target. CodeCrucible's design assumes the change under test is written by an adversary taking the cheapest path to a green verify, including deliberate cheat tests and a forged report written from test code. Plan section 4 is the source; every defence below is named with the code that implements it and the test or fixture that proves it.

Threat model​

In the modelOut of the model
An agent or a person writing the head commit: source, tests, helpers, config, setup filesA compromised exec host
Test code that runs inside the test container and can write anything there, including a forged mutation.json or tests.jsonThe operator's policy files and the quality image
A caller choosing request fields: repo, package, head SHA, intentThe core worker and Postgres

What the change under test cannot influence (plan 4.1)​

AssetEnforcementCode
Thresholds, globs, mutators, timeoutsOnly in the operator's FactoryRepoPolicy.quality (strict zod)packages/agent-types/src/factory.ts (QualityPolicy)
Stryker and Vitest configGenerated by the image scripts into /workspace/.agentforge/quality/ (a forbidden patch path) and passed by explicit path; the list run uses a host-only minimal Vitest configapps/worker/quality-image/analyze.mjs, run.mjs; QUALITY_CONFIG_DIR in apps/worker/src/sandbox/contract.ts
Mutant list and idsFrom the analysis container, which runs no repository testanalyze.mjs, buildAnalysis, assignStableIds
The baseComputed by the exec worker as git merge-base <head> <protected tip> and required to equal that tipprepareWorkspace in apps/worker/src/quality/exec-activities.ts
Support, config and setup filesAny change is the blocking finding quality_support_changedcheckStatic rule 6
Dependenciesnode_modules of every workspace package replaced by links into the imagelinkImageDependencies in common.mjs
Cache, baseline, verdictPostgres, written only by core activitiesapps/worker/src/quality/core-activities.ts, packages/db/src/quality.ts
A known gap in the config row

run.mjs and tests.mjs run the repository's tests with the package's own Vitest config from the head tree (the comments say so). The defence is that any change to such a config file is quality_support_changed, so a passing verify measured the base config. The plan's wording ("the head tree's config files are never loaded") is stronger than the code.

Trusted list versus untrusted evidence​

Everything the test container reports is evidence, not fact:

  • a mutant absent from the trusted list is dropped (onlyInReal), so a forged throw 0 mutant earns nothing;
  • a trusted mutant missing from the report counts as survived;
  • killedBy only chooses which test files run in a re-kill; it never grants credit by itself;
  • coverage is capped by the share of mutants that were reached (capCoverage), and CRAP credit needs the mutation threshold;
  • tests.json (executed identities, pass or fail) is a tripwire: a forged executed list is caught because the hidden test's file is modified, so every retained kill of that file is re-killed.

Static tripwires (plan 4.2)​

Seven blocking rules in checkStatic, run in verify over both trees. Rules 1, 2 and 4 are tripwires that catch careless changes; the defence against deliberate evasion is the verify steps.

RuleCodeDefeated by (stated)Real defence
1directive_addedbuilding the directive dynamicallystep 6 re-kill
2test_skip_addedt['sk'+'ip'], describe.each([]), if (false) it(...)step 2 executed identities
3test_removeddynamic titlesstep 2
4mutation_runtime_referencekey enumeration, built namesstep 6 (no runtime in the re-kill)
5code_moved_out_of_scoperenaming and editing while moving(stated limit)
6quality_support_changednone: any change blocksseparate human-reviewed merge
7mutant_count_changednone for unchanged files

Named gaming paths​

The plan's table (section 4.4) with the defence as built and the test or fixture that proves it. AC10 cases are run "with rules 1, 2 and 4 disabled", i.e. their assertions do not rely on the tripwires.

Gaming pathDefenceProof
Cheat test detecting the mutation runtime (key enumeration, built names)No Stryker runtime in the step 6 re-kill: the materialized plain edit runs only the killedBy files, so the cheat passes and earns no creditAC10 (a), (b): apps/worker/test/quality/evasion.test.ts ("a test detecting the mutation runtime earns zero credited kills"); S1 spike (iii) exit 0
Forged report: fake mutants, throw 0 or syntax-error replacements, forged "still killed"Only trusted-list tuples are credited; retained kills of modified test files are all re-killedAC10 (g): "a forged throw 0 mutant is dropped; the kept kill of the weakened file is not reproduced: fails, 20 of 20"
Forged executed-identity list or dry-run result (if (false) it(...) reported as run)The hidden test's file is modified, so step 6 re-kills all its retained kills and failsAC10 (h): "its 3 of 300 retained kills are re-killed and fail, 20 of 20"
A weakened base: an unmerged commit that removed assertionsThe base is computed, never supplied; the request has no base fieldAC10 (i), (k): "a head on an unmerged branch ... is measured against the protected tip", "a caller cannot name a base"
A head branched from an old protected commit and never rebasedMerge base must equal the protected tip, else base_staleAC10 (l): "a head branched from an old protected commit and never rebased is base_stale"
Declaring feature to skip checksUnchanged functions keep every check; the verdict carries the intent; the DB constraint ties a pass to the stored intentAC13: apps/worker/test/quality/verify.test.ts ("AC13 intent: price() and tax() in one file")
Helper weakened inside an existing test fileThat file is modified, so all its retained kills are re-killedstep 6 rule, verify.test.ts ("every retained kill of a modified test file is a candidate")
Test hashing source textComment-only controlAC10 (c): evasion.test.ts ("tests hashing the source get control_killed in 20 of 20 draws")
Test hashing fn.toString(), the AST, or a normalised token streamRewrite controls drawn per run with a host seed, first one structuralAC10 (d), (j): same test; a fully normalising hash test is a stated limit
Test that fails alone or flakes, sole killer of many mutantsStep 5 solo check: 3 of 3 alone, else no credit for any mutant of that setAC10 (f): "its killedBy set fails the solo check (0 of 3); with its setup file it passes 3 of 3"
Hidden tests (t['sk'+'ip'], describe.each([]), if (false) it('name')), with or without padding testsStep 2: every base and static head identity must be executed (sets, not totals)AC10 (e): "if (false) it('doubles') plus it('pads'): the base identity was not executed"; verify.test.ts step 2
Weakened helper, new vite.config.ts, new setup filequality_support_changed blocks; it blocks a merge only if verify is a required CI status (CC-4.2, not built)AC9 fixtures in packages/quality-core/test/static.test.ts
Coverage without assertions to lower CRAPStep 8: coverage counts only when the function meets the mutation threshold; credit uses re-killed kills onlyAC14: verify.test.ts ("higher coverage with a mutation score below the threshold earns no CRAP credit", "CRAP credit counts only re-kill-confirmed kills")
Splitting a complex function into tiny onescomplexity_cap on summed cyclomatic of changed files (remediation)AC14: "a split raising summed complexity 20 percent fails; one lowering it passes"
Fewer mutants through lookup tables, Math.sign, data-driven codePlan: a non-blocking review finding at a 30 percent fallNot implemented (stated limit in packages/quality-core/README.md)
Assertion removed so a test passesStep 3 characterization (base tests against head source) and step 6AC11 honest path is not separately automated; verify.test.ts AC13 Y case and quality-verify.test.sh (not_reproduced with the base tests)
A new Timeout instead of a killStep 9: a new timeout counts as survived and is never a candidateverify.test.ts ("step 9: a new Timeout counts as survived")
Too many candidates to re-killOver 50 ends partial, never passed, naming the test files to split bypackages/agent-workflows/test/quality-verify.workflow.test.ts ("more than 50 candidates end partial")

The AC10 matrix​

The fixtures live in apps/worker/test/quality/evasion/fixtures.ts: a fixture package packages/p with price() and its honest test, plus the cheat test files. evasion.test.ts runs the real exec activities (soloCheck, reKill, controls, prepareWorkspace) on a git mirror of that package with an in-process stand-in for the test container, and the pure evaluateVerify on the resulting credit. scripts/quality-verify.test.sh runs solo check, re-kill and controls in the real quality image.

Seeds and unpredictability​

DrawWho drawsWhenRecorded
Step 6 sample of retained kills in changed filesworkflow uuid4() (sampleSeed)after the head is fixed, before the first evaluationpassed to both evaluations
Step 7 control rewritesexec worker randomBytes(16) per functionat control timeseed in the controls finding, after the run

The rewrite set is in this repository; only the draw is unpredictable. A test that fully normalises semantics-preserving rewrites survives the controls: a stated limit.

Stated limits​

From docs/deploy-quality.md, packages/quality-core/README.md and the code's shortcut comments:

  • Report forgery from test code is mitigated by re-kill and controls, not prevented.
  • Coverage is untrusted and capped by the mutation results.
  • Equivalent mutants are not excluded in v1; the policy threshold is where the operator leaves room for them.
  • !(a >= b) differs from a < b for NaN or undefined operands; a test that pins that case gets control_killed.
  • A retained Stryker timeout with no killers costs credit (FOLLOWUPS).
  • The feature scope of step 3 uses killing test files, not covering tests (verify.ts shortcut).
  • Any content change of a base test file makes all its retained kills candidates (broader than the plan, verify.ts shortcut).
  • Rules 1, 2 and 4 are tripwires; a function renamed and edited while moved out of scope is not matched by rule 5.
  • The executed-identity list and the dry-run result come from the test container (plan changelog B).
  • Enforcement in CI (required status, label approval for feature) is not built (CC-4.2), so verify is voluntary.

Sources​

  • .omc/plans/codecrucible-mutant-hunter.md (sections 2, 3 "Two kinds of container", 4.1 to 4.5, 5 AC9 to AC14, changelogs)
  • apps/worker/src/quality/verify.ts (evaluateVerify, shortcut comments)
  • apps/worker/src/quality/exec-activities.ts (prepareWorkspace, buildChunkReport, capCoverage, reKill, controls, soloCheck)
  • apps/worker/quality-image/analyze.mjs, run.mjs, tests.mjs, common.mjs
  • packages/quality-core/src/report.ts (joinTrustedList), static.ts (checkStatic), rewrites.ts (drawControls), gate.ts
  • packages/quality-core/README.md (stated limits)
  • apps/worker/test/quality/evasion.test.ts, evasion/fixtures.ts, verify.test.ts; packages/agent-workflows/test/quality-verify.workflow.test.ts; packages/quality-core/test/static.test.ts
  • docs/p2-findings.md (CodeCrucible Phase 0 spikes, S1)
  • docs/deploy-quality.md (Operations and rollback, stated limits)
  • docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md