Exec worker and quality image
The exec worker is the only CodeCrucible process that touches Docker and repository code. It runs on a sandbox host (not built yet: precondition PRE-1), holds no database credential, no push key, no gateway control token and no model key, and talks to Temporal with the quality namespace's payload key only. It drives two kinds of container from one per-repository quality image.
Code: apps/worker/src/quality-exec-main.ts, apps/worker/src/quality/exec-config.ts,
apps/worker/src/quality/exec-activities.ts, image in apps/worker/quality-image/, sandbox argv in
apps/worker/src/sandbox/contract.ts and test-run.ts, out-file extraction in apps/worker/src/sandbox/extract.ts
(beads CC-2.1, CC-2.2, CC-2.4, CC-3.1, CC-3.2).
Process layout
- Two
Workerobjects share oneNativeConnection; each hasmaxConcurrentActivityTaskExecutions: 1and heartbeat throttle 10 s. Either worker stopping shuts the other down. - At start:
assertGitFloor(), thensweep()(run directories older thanRUN_DIR_MAX_AGE_MS= 6 h are removed, and leftoverafsbx-q-*containers and volumes of this owner from an earlier process instance are force-removed). - The test-container runner is a
SandboxTestRunnerwithgatewayContainer: 'none',maxOutputTokens: 1, the configured runtime and image, andowner=WORKER_IDENTITY(theagentforge.ownerlabel).
quality-exec-main.ts registers only prepareWorkspace (shared queue) and analyze, runChunk, cleanup (identity
queue). The verify activities stabilityRuns, characterize, readTrees, soloCheck, reKill and controls exist in
createQualityExecActivities but are not registered, so a verify against this entry point cannot run (see
verify workflow).
Configuration (exec-config.ts)
| Variable | Default | Rule |
|---|---|---|
WORKER_IDENTITY | <hostname>-exec | ^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,99}$; also the docker owner label and the identity-queue suffix |
TEMPORAL_ADDRESS | (required) | |
TEMPORAL_NAMESPACE | agentforge-quality | |
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAME | unset | mTLS; unset in NODE_ENV=production needs TEMPORAL_ALLOW_PLAINTEXT=tailnet-only |
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID | (required) | the quality namespace's payload key |
QUALITY_PROD_PAYLOAD_KEY_ID | unset | required when the namespace is not agentforge-prod; startup refuses if the payload key map holds that id |
QUALITY_REPO_SOURCE | (required) | local:<absolute dir> only; anything else throws GitHubSourceUnavailable (code github_source_unavailable) until PRE-3 |
QUALITY_IMAGE | (required) | SandboxImageRef: digest form only |
QUALITY_RUNTIME | runsc | runc or runsc |
QUALITY_ALLOW_RUNC | unset | must be own-repo-only when QUALITY_RUNTIME=runc |
QUALITY_WORK_DIR | /var/lib/agentforge-quality | must be an existing directory |
FACTORY_REPOS, FACTORY_POLICY_DIR | '', unset | at least one repo must have a policy with quality, else startup fails |
NODE_ENV | unset |
Config refusals (AC17)
Isolation test: apps/worker/test/quality/exec-iso.test.ts bundles dist/quality-exec-main.js with esbuild and asserts
the metafile holds no pg, no @agentforge/db, no core activities and none of the control-token markers
(/control/v1/jobs, GATEWAY_CONTROL_TOKEN_WORKER_FILE, controlToken, claimTurn, abandonPrefix, isDisabled,
connectionString); a positive control asserts dist/main.js has them. Config tests:
apps/worker/test/quality/exec-config.test.ts. The Docker target is quality-exec in apps/worker/Dockerfile
(CMD ["node","dist/quality-exec-main.js"], QUALITY_WORK_DIR=/var/lib/agentforge-quality, mode 0700); FOLLOWUPS
records that the exec worker has not run from that Docker target or against a real Temporal server.
Local-mirror mode
Without PRE-3 (a read-only GitHub App key) the exec worker reads only a bare mirror the operator clones with their own
credentials, at <dir>/<owner>/<name>.git. Local development, the integration tests and the AgentForge dogfood use it,
so PRE-3 gates only other repositories. All git calls use runGit with the mirror as cwd and a 300 s timeout.
The run directory
| File | Written by | Content |
|---|---|---|
<QUALITY_WORK_DIR>/<runId>/workspace.tar | prepareWorkspace | git archive --format=tar --mtime='1970-01-01 00:00:00 +0000' <head> of the whole repository (deterministic) |
meta.json | prepareWorkspace | repo, package, headSha, baseSha (the protected tip), protectedBranch |
analysis.json | analyze (or lazily by runChunk, reKill, controls) | the parsed analysis container output |
Missing workspace.tar or meta.json fails the activity non-retryably with workspace_missing. cleanup sweeps old
directories, removes this run's directory and every container and volume named afsbx-q-<runId>-*; it is idempotent.
Exec activities
| Activity | Queue | Container command | Out file | Result |
|---|---|---|---|---|
prepareWorkspace | shared | none (git only) | none | identity, base, protected branch, or a refusal code |
analyze | identity | node /opt/crucible/analyze.mjs <pkg> --mutate <g>... --support <g>... (limit 540 s) | analysis.json (kind quality-analyze) | image digest, sandbox ms, chunks with cache keys |
runChunk | identity | node /opt/crucible/run.mjs <pkg> <file>[:<start>-<end>] (limit timeoutMs, at most 840 s) | mutation.json (kind quality-run) | base64 gzip QualityChunkReport, or timeout, or failed |
cleanup | identity | none | none | void |
stabilityRuns | identity | node /opt/crucible/tests.mjs <pkg> x 3 | tests.json (kind quality-tests) | passes, runs, executed identities |
characterize | identity | tests.mjs <pkg> once, base test files overlaid | tests.json | failed base tests or timedOut |
readTrees | identity | none (git archive of both SHAs) | none | base64 gzip JSON of both trees |
soloCheck | identity | tests.mjs <pkg> --runs 3 with a host-written file list | tests.json (exit codes) | passes |
reKill | identity | tests.mjs <pkg> --runs 1 on the head with one materialized mutant | tests.json | confirmed or not_reproduced |
controls | identity | tests.mjs <pkg> --runs 1 per control (3 containers) | tests.json | seed, per-kind pass, limited |
All container activities heartbeat every heartbeatIntervalMs (validated 100 to 60,000 ms) and turn a Temporal cancel
into an abort of the container, then rethrow CancelledFailure once the runner returned.
runChunk in detail
buildChunkReport keeps the trusted mutants of the chunk's file (and line range), joins the report on the tuple
(onlyInReal is counted as dropped and discarded), assigns stable ids from the head source, cuts original and
replacement to 80 characters, fits killedBy into 4,096 bytes of JSON, and computes each function's coverage from the
report's coverage of that file (0 without one) capped by capCoverage. The raw Stryker report (which embeds whole source
files) never leaves the container's /tmp.
Sandbox argv
Quality containers use the P3 test-container argv (testRunArgs in contract.ts): the job container's hardening with
--network none, no --env-file, and --workdir /workspace. See sandbox for the shared runner.
| Flag | Value |
|---|---|
--network | none |
--ipc | private |
--user | 10001:10001 |
--read-only | root filesystem read-only |
--cap-drop | ALL |
--security-opt | no-new-privileges:true |
--pids-limit | 512 |
--memory, --memory-swap | 2g, 2g |
--cpus | 2 |
--ulimit | nofile=1024:1024 |
--init, --stop-timeout | yes, 5 |
--log-driver | none |
--runtime | QUALITY_RUNTIME (runsc default) |
--tmpfs | /home/agent 256 MiB (uid 10001), /tmp 512 MiB |
| workspace | a 1 GiB tmpfs volume mounted at /workspace, filled from the workspace tar through a keeper container |
| command prefix | timeout --signal=KILL <ceil(timeoutMs / 1000) + margin> (convenience only, not a control) |
SandboxTestRunner.run refuses a command that is not 1 to 64 plain argv tokens and a timeoutMs outside 1,000 to
840,000 (MAX_TEST_TIMEOUT_MS). Docker calls in removeDocker pass assertSafeArgv. AC6 (golden argv) is the
CC-2.2 test of this argv.
Out files
Each quality container leaves exactly one JSON file in /workspace/.agentforge/out, tarred out through the keeper with
tar -c -C /workspace/.agentforge/out --no-recursion -- <name> (no -h, so a symlink is recorded as a link, never its
target).
Kind (QUALITY_OUT_FILE_BY_KIND) | File |
|---|---|
quality-analyze | analysis.json |
quality-run | mutation.json |
quality-tests | tests.json |
extractOutFile reads the keeper's tar stream into memory (nothing is written to disk) and refuses anything but
exactly one regular-file entry with that exact name followed by the end-of-archive marker: a missing or empty archive, a
symlink or other type, any other name (including ./name or a .. path), a prefix field, pax or GNU long-name
headers, a second entry, or a file over QUALITY_OUT_MAX_BYTES (8 MiB).
Host config the commands read goes under QUALITY_CONFIG_DIR = /workspace/.agentforge/quality (a forbidden patch path,
see patch policy), so a change under test cannot supply it.
The quality image
apps/worker/quality-image/Dockerfile, one image per repository, tagged by the repository's lockfile hash and run by
digest only. Build from the AgentForge root:
docker build -f apps/worker/quality-image/Dockerfile --build-context repo=<repo dir> -t agentforge-quality:<lock hash> .
(AgentForge dogfood: --build-context repo=.; scripts/quality-image.test.sh builds and exercises it.)
| Item | Value |
|---|---|
| Base | node:24-bookworm-slim pinned by digest; pnpm@11.28.11 in build stages |
| Engine | Stryker core and vitest-runner 10.1.1, Vitest 5.1.0 with v8 coverage (from apps/worker/quality-image/package.json and its own lockfile) |
| Package store | a BuildKit cache mount, never a layer (S5: keeping it made the image 3.4 GB instead of 2.06 GB) |
| Repository | dependencies and the base revision's prebuilt dist in /opt/crucible/repo |
procps | Stryker ends test-runner process trees with ps |
HOME, XDG_* | on the /home/agent tmpfs; Vitest needs a writable home for its API token (S5) |
| Size | 1.79 GB measured (S5 prototype 2.06 GB; plan limit 3 GB) |
Scripts inside the image
All scripts run as uid 10001 with cwd = /workspace (the workspace tar). common.mjs holds the shared helpers; nothing
in it is read from the workspace.
Helper (common.mjs) | Behaviour |
|---|---|
linkImageDependencies() | gives every workspace package (depth at most 3, never inside node_modules) a fresh node_modules whose entries are symlinks to the image's read-only copy; dependencies, Stryker and Vitest always come from the image |
readTree(dir, max = 5000) | package-relative path to bytes; skips symlinks and the package-root node_modules; throws on a .git, .stryker-tmp or .agentforge directory or a nested node_modules at any depth |
workspaceDependencies(pkg) | transitive workspace: dependencies from dependencies, devDependencies, peerDependencies, optionalDependencies |
packageDir(pkg) | refuses absolute paths, .., backslashes, empty segments, a missing package.json or a symlinked package dir (exit 2) |
writeFresh(path, text) | removes the path first and writes with flag wx, never through a link the workspace held |
runStryker(config, cwd) | stryker run <config> by explicit path from the package directory |
| paths | CONFIG_DIR = .agentforge/quality, OUT_DIR = .agentforge/out, SCRATCH = /tmp/crucible, STRYKER_TMP = .agentforge/stryker-tmp; the Vitest runner plugin by absolute path to @stryker-mutator/vitest-runner/dist/src/index.js |
analyze.mjs (analysis container)
Runs no repository test and loads no repository Vitest or Vite config, setup file or plugin.
- Validate args (
<package dir> --mutate <glob>... [--support <glob>...]), prepare dirs, link image dependencies. - Read the package tree and every transitive workspace dependency tree before Stryker instruments anything in place.
- Scope = source files (
\.[cm]?[jt]sx?$, not*.test.*or*.spec.*) in the mutate globs (negations honoured), sorted; none is an error. - Write the host-only Vitest config
vitest.list.config.mjs(rootCONFIG_DIR, includeempty.test.mjs,globals: true, no setup files) and the one empty testit('empty', () => {});. - Write
stryker.list.json:testRunner: vitest,coverageAnalysis: off,inPlace: true,mutate: <scope>,vitest.related: false, JSON reporter to/tmp/crucible/list.json,concurrency: 1,timeoutMS: 30000,disableTypeChecks: true. - Run Stryker: every mutant survives the empty test, which enumerates the full trusted list.
buildAnalysis(...), check withparseAnalysisOutput(the exec worker's caps), writeanalysis.json.
run.mjs (test container, one chunk)
- Validate
<package dir> <file>[:<start>-<end>](a source file, safe relative path). - Write
stryker.run.json:coverageAnalysis: perTest,inPlace: true,mutate: [chunk],concurrency: 2(matches--cpus 2),timeoutMS: 30000,disableTypeChecks: true. The package's own Vitest config is used here; a change to it is the blocking findingquality_support_changed. - Run Stryker; normalise the report; refuse mutants of another file.
- Run Vitest once more with
--coverage.provider=v8 --coverage.reporter=json --coverage.include=<file>on the restored source for per-function coverage (CRAP). A failing coverage run leaves no coverage (counted as 0). - Check with
parseQualityReportand writemutation.json.
tests.mjs (unmutated runs)
tests.mjs <pkg>:vitest run --reporter=jsonwith the package's config; writes{ schemaVersion: 1, exitCode, tests: [{ file, name, status }], failedFiles }wherenameis the describe titles and title joined with>andstatusispassed,failedorskipped.tests.mjs <pkg> --runs <1-9>: runs only the files listed in the host-written.agentforge/quality/test-files.json(1 to 500 safe relative paths, each an existing regular file in the package), by absolute path,<runs>times; writes{ schemaVersion: 1, exits: [...] }.
The exec worker parses both forms with strict schemas (TestsReport, RunsReport) and treats them as untrusted
tripwires (see anti-gaming).
Local evidence
| Check | Command | Result |
|---|---|---|
| Image, analysis, one chunk | bash scripts/quality-image.test.sh | build 70 to 276 s, 1.79 GB; analyze 35 to 92 s: 417 trusted mutants in 8 chunks; src/budget.ts 96 mutants, 82 killed, 12 survived, 2 no coverage; 100 percent of tuples join the trusted list |
| Exec activities | bash scripts/quality-exec.test.sh <image> | prepare, analyze, runChunk, cleanup in 122 s; cleanup idempotent; nothing left behind |
| Verify steps 5 to 7 | bash scripts/quality-verify.test.sh <image> | solo 3 of 3, re-kill confirmed with the new test and not_reproduced with the base tests, controls pass, in 195 s |
All on an OrbStack laptop with runc under load. Not measured: runsc anywhere (spike S3), a real tailnet exec host, a
real GitHub fetch.
Sources
apps/worker/src/quality-exec-main.tsapps/worker/src/quality/exec-config.ts(QualityExecEnv,FORBIDDEN,parseRepoSource,GitHubSourceUnavailable,loadQualityExecConfig)apps/worker/src/quality/exec-activities.ts(ANALYZE_TIMEOUT_MS,RUN_DIR_MAX_AGE_MS,capCoverage,buildChunkReport,tarFiles,overlay,materialize,TestsReport,RunsReport,TEST_FILES_PATH,removeDocker,sweep, every activity)apps/worker/src/sandbox/contract.ts(runArgs,testRunArgs,QUALITY_OUT_FILE_BY_KIND,QUALITY_CONFIG_DIR,qualityOutArgs),test-run.ts(MAX_TEST_TIMEOUT_MS),extract.ts(QUALITY_OUT_MAX_BYTES,extractOutFile)apps/worker/quality-image/Dockerfile,common.mjs,analyze.mjs,run.mjs,tests.mjsapps/worker/Dockerfile(targetquality-exec)apps/worker/test/quality/exec-iso.test.ts,exec-config.test.tsdocs/deploy-quality.md(Environment, Local evidence),docs/p2-findings.md(CodeCrucible S5)docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md