Skip to main content

CodeCrucible: the mutation-evidence quality gate

CodeCrucible is AgentForge's deterministic, model-free quality analyzer. It measures one package of one repository at one commit and records, per function: cyclomatic and cognitive complexity, coverage, CRAP, and a mutation score from StrykerJS runs. It then gates a pushed change against the protected branch ("no new debt") and, in a verify run, checks that every kill the change claims is real.

The defining idea is mutation evidence: a coverage number alone can be raised by tests without assertions, so CodeCrucible credits only mutants that a test actually kills, and it re-kills the important ones itself in a fresh container before it believes them.

Status: built locally, dormant

Everything on this section's pages is built and tested locally only (OrbStack, runc, a local git mirror). Nothing has run on a VPS, against the shared Temporal cluster or against GitHub. QUALITY_ENABLED is unset everywhere, so the REST routes answer 404 and the core worker registers no quality workflow or activity. Migrations 0006_quality.sql and 0007_quality_submissions.sql exist but have run nowhere. See status and limits.

Why it exists​

The source note (summarised in the plan .omc/plans/codecrucible-mutant-hunter.md, section 1) asks for an analyzer whose findings an agent must act on, with these properties:

RequirementHow v1 meets it
Two separate gates, not one composite scoreCRAP per function (C^2 * (1 - cov)^3 + C) and mutation score per function, both in evaluateGate
Mutation engine is StrykerJSStryker 10.1.1 with the Vitest runner, inside the quality image
Baseline plus "no new debt"quality_baselines rows written only by runs on the protected branch tip; deltas against them
Deterministic, model-independentpackages/quality-core is pure; no model call happens anywhere, so no gateway budget is touched
Guard-rails: the agent never edits rules, never weakens tests, killed mutants stay killedOperator-only policy, static checks, executed-identity check, materialized re-kill, controls (anti-gaming)

Scope cut (v1)​

ADR 0007 (docs/sessions/2026-11-16-codecrucible-and-gateway-cidr/adr/0007-codecrucible-v1-shape.md) fixes the v1 shape: "measure, gate, and verify a pushed change over REST". v1 does not close the agent loop: a person or an existing agent session reads the survivors, pushes a fix, then calls verify.

In v1Later (plan section 10)
packages/quality-core: complexity, CRAP, report schema, stable mutant ids, gate, tamper scanComposite score, Tree-sitter, other languages
Deep run: mutation testing of one package, chunked per file, plus one coverage run per chunk for CRAPA "fast" level as its own run type; exhaustive schedules
qualityAssessmentWorkflow, qualityVerifyWorkflow with re-kill and controlsFactory child workflow; survivors in the coder prompt
Postgres tables, baseline, "no new debt"Exclusions for equivalent mutants with approval; artifacts table; dashboards
REST /v1/quality/* in apps/apiMCP read facade (quality.get_report, get_function, get_survivors)
Database-free exec worker on a sandbox hostMore than one package per run; continue-as-new beyond 20 chunks
Dogfood on packages/agent-coreapps/worker, apps/gateway, packages/db

Alternatives rejected by ADR 0007: a separate always-on service on its own host; a custom Tree-sitter mutation engine; MCP-first exposure.

Design principles​

From plan section 2:

  1. The analyzer is deterministic and holds no model key. The agent never computes or stores a score.
  2. Untrusted code (repository tests) runs only in the sandbox. The process that drives Docker and reads sandbox output holds no database credential, no push key, no gateway control token and no agentforge-prod Temporal payload key.
  3. Everything that defines the measurement (rules, config, thresholds, scope, the mutant list, the cache) is produced by the host from the operator policy and the base commit, never by the test container.
  4. No new always-on service on the core host. One small database-free worker on a sandbox host is the price of principle 2.
  5. Measure before building, and ship the smallest thing one team can verify.

Architecture​

The plan calls this option "B-prime": a pure library, core activities inside the existing core worker, and a database-free exec worker on a dedicated sandbox host. It reuses the FL-3 worker split of the session runtime (see worker fleet).

The trust boundaries in that picture:

The two run types​

RunStarted byWhat it doesVerdict
Assessment (qualityAssessmentWorkflow)POST /v1/quality/runsMeasures the head commit of one package, gates it against the stored baseline, writes a new baseline when the head is the protected tippassed:remediation or failed (assessments always judge as remediation)
Verify (qualityVerifyWorkflow)POST /v1/quality/verifyRuns two child assessments (base and head), then steps 1 to 9 of plan 4.3: stability, executed identities, characterization, deltas, solo checks, materialized re-kills, controls, CRAP creditpassed:remediation, passed:feature or failed

Details: assessment workflow, verify workflow.

Component map​

PieceLocationHolds
Analysis librarypackages/quality-coreNothing (pure; imports only typescript, zod, node:crypto, node:zlib)
Wire types and limitspackages/agent-types/src/quality.ts, QualityPolicy in packages/agent-types/src/factory.ts, HTTP schemas in packages/agent-types/src/http.tsSchemas only
Workflowspackages/agent-workflows/src/workflows/quality.ts, quality-shared.tsWorkflow code, queue names, QUALITY_RUN_LIMITS
Core activitiesapps/worker/src/quality/core-activities.ts, apps/worker/src/quality/verify.tsDATABASE_URL (through the core worker)
Exec workerapps/worker/src/quality-exec-main.ts, apps/worker/src/quality/exec-activities.ts, exec-config.tsDocker socket, repo mirror, quality payload key
Quality imageapps/worker/quality-image/ (Dockerfile, analyze.mjs, run.mjs, tests.mjs, common.mjs)Stryker 10.1.1, Vitest 5.1.0, repo dependencies, base dist
Persistencepackages/db/src/quality.ts, migrations 0006_quality.sql, 0007_quality_submissions.sqlquality_runs, quality_functions, quality_mutants, quality_baselines, quality_chunk_cache, quality_submissions
RESTapps/api/src/quality.tsQUALITY_ENABLED, QUALITY_CLIENTS, limits

Pages in this section​

PageContent
Libraryquality-core: function discovery, qualified names, complexity, CRAP, coverage, report schema, trusted-list join, stable ids, gate evaluator, static checks, control rewrites, analysis out file
Assessment workflowqualityAssessmentWorkflow end to end: queues, admission, chunking, cache, history budget, partial and exec_lost
Verify workflowqualityVerifyWorkflow steps 1 to 9, intent semantics, credit candidates, verdicts
Exec worker and imageExec activities, workspace, sandbox argv, image contents, analyze.mjs, run.mjs, tests.mjs, config refusals
Anti-gamingThreat model, named gaming paths with defences and the tests that prove them
Data modelER diagram and every constraint of the quality tables
REST and limitsRoutes, admission, pending cap, daily limits, queued view, error codes
Status and limitsWhat is built, what is not, measured numbers, stated limits, open beads

Related pages elsewhere: sandbox, patch policy (the .agentforge/quality/ path is a forbidden patch path), deploy runbooks, decisions.

Sources​

  • .omc/plans/codecrucible-mutant-hunter.md (git-ignored plan; sections 1, 2, 3, 10 and the ADR draft)
  • docs/sessions/2026-11-16-codecrucible-and-gateway-cidr/adr/0007-codecrucible-v1-shape.md
  • docs/sessions/2026-11-16-codecrucible-build/README.md, SUMMARY.md
  • docs/deploy-quality.md (Roles, state line)
  • packages/quality-core/README.md
  • apps/api/src/app.ts (QUALITY_ENABLED registration), apps/worker/src/worker.ts (quality Worker)
  • packages/agent-workflows/src/workflows/quality.ts, quality-shared.ts