Skip to main content

Exec worker and quality image

The exec worker is the only CodeCrucible process that touches Docker and repository code. It runs on a sandbox host (not built yet: precondition PRE-1), holds no database credential, no push key, no gateway control token and no model key, and talks to Temporal with the quality namespace's payload key only. It drives two kinds of container from one per-repository quality image.

Code: apps/worker/src/quality-exec-main.ts, apps/worker/src/quality/exec-config.ts, apps/worker/src/quality/exec-activities.ts, image in apps/worker/quality-image/, sandbox argv in apps/worker/src/sandbox/contract.ts and test-run.ts, out-file extraction in apps/worker/src/sandbox/extract.ts (beads CC-2.1, CC-2.2, CC-2.4, CC-3.1, CC-3.2).

Process layout​

  • Two Worker objects share one NativeConnection; each has maxConcurrentActivityTaskExecutions: 1 and heartbeat throttle 10 s. Either worker stopping shuts the other down.
  • At start: assertGitFloor(), then sweep() (run directories older than RUN_DIR_MAX_AGE_MS = 6 h are removed, and leftover afsbx-q-* containers and volumes of this owner from an earlier process instance are force-removed).
  • The test-container runner is a SandboxTestRunner with gatewayContainer: 'none', maxOutputTokens: 1, the configured runtime and image, and owner = WORKER_IDENTITY (the agentforge.owner label).
Registered activities

quality-exec-main.ts registers only prepareWorkspace (shared queue) and analyze, runChunk, cleanup (identity queue). The verify activities stabilityRuns, characterize, readTrees, soloCheck, reKill and controls exist in createQualityExecActivities but are not registered, so a verify against this entry point cannot run (see verify workflow).

Configuration (exec-config.ts)​

VariableDefaultRule
WORKER_IDENTITY<hostname>-exec^[a-zA-Z0-9][a-zA-Z0-9_.-]{0,99}$; also the docker owner label and the identity-queue suffix
TEMPORAL_ADDRESS(required)
TEMPORAL_NAMESPACEagentforge-quality
TEMPORAL_TLS_CERT, TEMPORAL_TLS_KEY, TEMPORAL_TLS_CA, TEMPORAL_TLS_SERVER_NAMEunsetmTLS; unset in NODE_ENV=production needs TEMPORAL_ALLOW_PLAINTEXT=tailnet-only
TEMPORAL_PAYLOAD_KEYS, TEMPORAL_PAYLOAD_KEY_ID(required)the quality namespace's payload key
QUALITY_PROD_PAYLOAD_KEY_IDunsetrequired when the namespace is not agentforge-prod; startup refuses if the payload key map holds that id
QUALITY_REPO_SOURCE(required)local:<absolute dir> only; anything else throws GitHubSourceUnavailable (code github_source_unavailable) until PRE-3
QUALITY_IMAGE(required)SandboxImageRef: digest form only
QUALITY_RUNTIMErunscrunc or runsc
QUALITY_ALLOW_RUNCunsetmust be own-repo-only when QUALITY_RUNTIME=runc
QUALITY_WORK_DIR/var/lib/agentforge-qualitymust be an existing directory
FACTORY_REPOS, FACTORY_POLICY_DIR'', unsetat least one repo must have a policy with quality, else startup fails
NODE_ENVunset

Config refusals (AC17)​

Isolation test: apps/worker/test/quality/exec-iso.test.ts bundles dist/quality-exec-main.js with esbuild and asserts the metafile holds no pg, no @agentforge/db, no core activities and none of the control-token markers (/control/v1/jobs, GATEWAY_CONTROL_TOKEN_WORKER_FILE, controlToken, claimTurn, abandonPrefix, isDisabled, connectionString); a positive control asserts dist/main.js has them. Config tests: apps/worker/test/quality/exec-config.test.ts. The Docker target is quality-exec in apps/worker/Dockerfile (CMD ["node","dist/quality-exec-main.js"], QUALITY_WORK_DIR=/var/lib/agentforge-quality, mode 0700); FOLLOWUPS records that the exec worker has not run from that Docker target or against a real Temporal server.

Local-mirror mode​

Without PRE-3 (a read-only GitHub App key) the exec worker reads only a bare mirror the operator clones with their own credentials, at <dir>/<owner>/<name>.git. Local development, the integration tests and the AgentForge dogfood use it, so PRE-3 gates only other repositories. All git calls use runGit with the mirror as cwd and a 300 s timeout.

The run directory​

FileWritten byContent
<QUALITY_WORK_DIR>/<runId>/workspace.tarprepareWorkspacegit archive --format=tar --mtime='1970-01-01 00:00:00 +0000' <head> of the whole repository (deterministic)
meta.jsonprepareWorkspacerepo, package, headSha, baseSha (the protected tip), protectedBranch
analysis.jsonanalyze (or lazily by runChunk, reKill, controls)the parsed analysis container output

Missing workspace.tar or meta.json fails the activity non-retryably with workspace_missing. cleanup sweeps old directories, removes this run's directory and every container and volume named afsbx-q-<runId>-*; it is idempotent.

Exec activities​

ActivityQueueContainer commandOut fileResult
prepareWorkspacesharednone (git only)noneidentity, base, protected branch, or a refusal code
analyzeidentitynode /opt/crucible/analyze.mjs <pkg> --mutate <g>... --support <g>... (limit 540 s)analysis.json (kind quality-analyze)image digest, sandbox ms, chunks with cache keys
runChunkidentitynode /opt/crucible/run.mjs <pkg> <file>[:<start>-<end>] (limit timeoutMs, at most 840 s)mutation.json (kind quality-run)base64 gzip QualityChunkReport, or timeout, or failed
cleanupidentitynonenonevoid
stabilityRunsidentitynode /opt/crucible/tests.mjs <pkg> x 3tests.json (kind quality-tests)passes, runs, executed identities
characterizeidentitytests.mjs <pkg> once, base test files overlaidtests.jsonfailed base tests or timedOut
readTreesidentitynone (git archive of both SHAs)nonebase64 gzip JSON of both trees
soloCheckidentitytests.mjs <pkg> --runs 3 with a host-written file listtests.json (exit codes)passes
reKillidentitytests.mjs <pkg> --runs 1 on the head with one materialized mutanttests.jsonconfirmed or not_reproduced
controlsidentitytests.mjs <pkg> --runs 1 per control (3 containers)tests.jsonseed, per-kind pass, limited

All container activities heartbeat every heartbeatIntervalMs (validated 100 to 60,000 ms) and turn a Temporal cancel into an abort of the container, then rethrow CancelledFailure once the runner returned.

runChunk in detail​

buildChunkReport keeps the trusted mutants of the chunk's file (and line range), joins the report on the tuple (onlyInReal is counted as dropped and discarded), assigns stable ids from the head source, cuts original and replacement to 80 characters, fits killedBy into 4,096 bytes of JSON, and computes each function's coverage from the report's coverage of that file (0 without one) capped by capCoverage. The raw Stryker report (which embeds whole source files) never leaves the container's /tmp.

Sandbox argv​

Quality containers use the P3 test-container argv (testRunArgs in contract.ts): the job container's hardening with --network none, no --env-file, and --workdir /workspace. See sandbox for the shared runner.

FlagValue
--networknone
--ipcprivate
--user10001:10001
--read-onlyroot filesystem read-only
--cap-dropALL
--security-optno-new-privileges:true
--pids-limit512
--memory, --memory-swap2g, 2g
--cpus2
--ulimitnofile=1024:1024
--init, --stop-timeoutyes, 5
--log-drivernone
--runtimeQUALITY_RUNTIME (runsc default)
--tmpfs/home/agent 256 MiB (uid 10001), /tmp 512 MiB
workspacea 1 GiB tmpfs volume mounted at /workspace, filled from the workspace tar through a keeper container
command prefixtimeout --signal=KILL <ceil(timeoutMs / 1000) + margin> (convenience only, not a control)

SandboxTestRunner.run refuses a command that is not 1 to 64 plain argv tokens and a timeoutMs outside 1,000 to 840,000 (MAX_TEST_TIMEOUT_MS). Docker calls in removeDocker pass assertSafeArgv. AC6 (golden argv) is the CC-2.2 test of this argv.

Out files​

Each quality container leaves exactly one JSON file in /workspace/.agentforge/out, tarred out through the keeper with tar -c -C /workspace/.agentforge/out --no-recursion -- <name> (no -h, so a symlink is recorded as a link, never its target).

Kind (QUALITY_OUT_FILE_BY_KIND)File
quality-analyzeanalysis.json
quality-runmutation.json
quality-teststests.json

extractOutFile reads the keeper's tar stream into memory (nothing is written to disk) and refuses anything but exactly one regular-file entry with that exact name followed by the end-of-archive marker: a missing or empty archive, a symlink or other type, any other name (including ./name or a .. path), a prefix field, pax or GNU long-name headers, a second entry, or a file over QUALITY_OUT_MAX_BYTES (8 MiB).

Host config the commands read goes under QUALITY_CONFIG_DIR = /workspace/.agentforge/quality (a forbidden patch path, see patch policy), so a change under test cannot supply it.

The quality image​

apps/worker/quality-image/Dockerfile, one image per repository, tagged by the repository's lockfile hash and run by digest only. Build from the AgentForge root:

docker build -f apps/worker/quality-image/Dockerfile --build-context repo=<repo dir> -t agentforge-quality:<lock hash> .

(AgentForge dogfood: --build-context repo=.; scripts/quality-image.test.sh builds and exercises it.)

ItemValue
Basenode:24-bookworm-slim pinned by digest; pnpm@11.28.11 in build stages
EngineStryker core and vitest-runner 10.1.1, Vitest 5.1.0 with v8 coverage (from apps/worker/quality-image/package.json and its own lockfile)
Package storea BuildKit cache mount, never a layer (S5: keeping it made the image 3.4 GB instead of 2.06 GB)
Repositorydependencies and the base revision's prebuilt dist in /opt/crucible/repo
procpsStryker ends test-runner process trees with ps
HOME, XDG_*on the /home/agent tmpfs; Vitest needs a writable home for its API token (S5)
Size1.79 GB measured (S5 prototype 2.06 GB; plan limit 3 GB)

Scripts inside the image​

All scripts run as uid 10001 with cwd = /workspace (the workspace tar). common.mjs holds the shared helpers; nothing in it is read from the workspace.

Helper (common.mjs)Behaviour
linkImageDependencies()gives every workspace package (depth at most 3, never inside node_modules) a fresh node_modules whose entries are symlinks to the image's read-only copy; dependencies, Stryker and Vitest always come from the image
readTree(dir, max = 5000)package-relative path to bytes; skips symlinks and the package-root node_modules; throws on a .git, .stryker-tmp or .agentforge directory or a nested node_modules at any depth
workspaceDependencies(pkg)transitive workspace: dependencies from dependencies, devDependencies, peerDependencies, optionalDependencies
packageDir(pkg)refuses absolute paths, .., backslashes, empty segments, a missing package.json or a symlinked package dir (exit 2)
writeFresh(path, text)removes the path first and writes with flag wx, never through a link the workspace held
runStryker(config, cwd)stryker run <config> by explicit path from the package directory
pathsCONFIG_DIR = .agentforge/quality, OUT_DIR = .agentforge/out, SCRATCH = /tmp/crucible, STRYKER_TMP = .agentforge/stryker-tmp; the Vitest runner plugin by absolute path to @stryker-mutator/vitest-runner/dist/src/index.js

analyze.mjs (analysis container)​

Runs no repository test and loads no repository Vitest or Vite config, setup file or plugin.

  1. Validate args (<package dir> --mutate <glob>... [--support <glob>...]), prepare dirs, link image dependencies.
  2. Read the package tree and every transitive workspace dependency tree before Stryker instruments anything in place.
  3. Scope = source files (\.[cm]?[jt]sx?$, not *.test.* or *.spec.*) in the mutate globs (negations honoured), sorted; none is an error.
  4. Write the host-only Vitest config vitest.list.config.mjs (root CONFIG_DIR, include empty.test.mjs, globals: true, no setup files) and the one empty test it('empty', () => {});.
  5. Write stryker.list.json: testRunner: vitest, coverageAnalysis: off, inPlace: true, mutate: <scope>, vitest.related: false, JSON reporter to /tmp/crucible/list.json, concurrency: 1, timeoutMS: 30000, disableTypeChecks: true.
  6. Run Stryker: every mutant survives the empty test, which enumerates the full trusted list.
  7. buildAnalysis(...), check with parseAnalysisOutput (the exec worker's caps), write analysis.json.

run.mjs (test container, one chunk)​

  1. Validate <package dir> <file>[:<start>-<end>] (a source file, safe relative path).
  2. Write stryker.run.json: coverageAnalysis: perTest, inPlace: true, mutate: [chunk], concurrency: 2 (matches --cpus 2), timeoutMS: 30000, disableTypeChecks: true. The package's own Vitest config is used here; a change to it is the blocking finding quality_support_changed.
  3. Run Stryker; normalise the report; refuse mutants of another file.
  4. Run Vitest once more with --coverage.provider=v8 --coverage.reporter=json --coverage.include=<file> on the restored source for per-function coverage (CRAP). A failing coverage run leaves no coverage (counted as 0).
  5. Check with parseQualityReport and write mutation.json.

tests.mjs (unmutated runs)​

  • tests.mjs <pkg>: vitest run --reporter=json with the package's config; writes { schemaVersion: 1, exitCode, tests: [{ file, name, status }], failedFiles } where name is the describe titles and title joined with > and status is passed, failed or skipped.
  • tests.mjs <pkg> --runs <1-9>: runs only the files listed in the host-written .agentforge/quality/test-files.json (1 to 500 safe relative paths, each an existing regular file in the package), by absolute path, <runs> times; writes { schemaVersion: 1, exits: [...] }.

The exec worker parses both forms with strict schemas (TestsReport, RunsReport) and treats them as untrusted tripwires (see anti-gaming).

Local evidence​

CheckCommandResult
Image, analysis, one chunkbash scripts/quality-image.test.shbuild 70 to 276 s, 1.79 GB; analyze 35 to 92 s: 417 trusted mutants in 8 chunks; src/budget.ts 96 mutants, 82 killed, 12 survived, 2 no coverage; 100 percent of tuples join the trusted list
Exec activitiesbash scripts/quality-exec.test.sh <image>prepare, analyze, runChunk, cleanup in 122 s; cleanup idempotent; nothing left behind
Verify steps 5 to 7bash scripts/quality-verify.test.sh <image>solo 3 of 3, re-kill confirmed with the new test and not_reproduced with the base tests, controls pass, in 195 s

All on an OrbStack laptop with runc under load. Not measured: runsc anywhere (spike S3), a real tailnet exec host, a real GitHub fetch.

Sources​

  • apps/worker/src/quality-exec-main.ts
  • apps/worker/src/quality/exec-config.ts (QualityExecEnv, FORBIDDEN, parseRepoSource, GitHubSourceUnavailable, loadQualityExecConfig)
  • apps/worker/src/quality/exec-activities.ts (ANALYZE_TIMEOUT_MS, RUN_DIR_MAX_AGE_MS, capCoverage, buildChunkReport, tarFiles, overlay, materialize, TestsReport, RunsReport, TEST_FILES_PATH, removeDocker, sweep, every activity)
  • apps/worker/src/sandbox/contract.ts (runArgs, testRunArgs, QUALITY_OUT_FILE_BY_KIND, QUALITY_CONFIG_DIR, qualityOutArgs), test-run.ts (MAX_TEST_TIMEOUT_MS), extract.ts (QUALITY_OUT_MAX_BYTES, extractOutFile)
  • apps/worker/quality-image/Dockerfile, common.mjs, analyze.mjs, run.mjs, tests.mjs
  • apps/worker/Dockerfile (target quality-exec)
  • apps/worker/test/quality/exec-iso.test.ts, exec-config.test.ts
  • docs/deploy-quality.md (Environment, Local evidence), docs/p2-findings.md (CodeCrucible S5)
  • docs/sessions/2026-11-16-codecrucible-build/FOLLOWUPS.md