Testing and CI
This page describes the test pyramid, the turbo tasks, the Docker image tests, the smoke app, the real-Docker scripts, the CI workflow jobs, the runner constraints and how flakes are handled. Convention source: CODING_STANDARDS.md section 13.
The test pyramid
| Layer | Runs where | Spends money | Trigger |
|---|---|---|---|
| Unit and integration (vitest) | CI and local | no | pnpm turbo run test |
| Static (typecheck, lint, format, boundaries, migration drift, gitleaks) | CI and local | no | CI |
| Image tests | CI (api, gateway, GV3) and local (the rest) | no | bash scripts/<name>.test.sh <image> |
| Real-Docker keyless end to end | local only, one agent at a time | no | scripts below |
| Real-key and real-GitHub runs | user-run only | yes (cents) | test:real, sandbox-rk, factory-gh |
Only the user runs real-key tests. Keyless tests are the default.
Test conventions
From CODING_STANDARDS.md section 13:
- Framework vitest with explicit imports (no globals). Tests live in
test/besidesrc/, import the package through../src/index.jsand other packages by package name. - Name each
describeafter the plan criterion it proves, prefix form first, for exampledescribe('P1-12 kill switch', ...). Titles state behavior and outcome. - Database tests run against real Postgres, never mocks: start it with
docker compose up -d --wait(host port 5433; CI uses a throwaway container on a random host port). Each test file creates and drops its own database withcreateTestDatabase(), each test resets withTRUNCATEinbeforeEach, and day-boundary cases pass explicitatdates. - Deterministic tests: injected clocks,
vi.useFakeTimers({ toFake: [...] })with an explicit list. Randomized runs only when the expectation is computed independently of event order (packages/db/test/concurrency.test.ts). - Every fail-closed path is tested: store down, throwing listener, invalid cost, unscrubbable event, stuck runtime.
- Real-key tests live in
test/real/*.real.test.tsand run only throughtest:real(cache: false). That config has aglobalSetupthat refuses before any test file loads, and each file repeats the guard.apps/smoke/test/guard.test.tsasserts the config split. - Structural invariants are tests too: dependency sets, SDK resolution, ESLint probes, the export list of
compose.ts. - Workflow replay tests run over recorded histories (
pnpm --filter @agentforge/agent-workflows test:replay). The Docker doc testapps/worker/test/sandbox-doc.test.tsfails when a flag the driver emits is missing fromdocs/deploy-sandbox.md.
Turbo tasks (turbo.json)
| Task | dependsOn | Cache | Notes |
|---|---|---|---|
build | ^build | outputs dist/** | tsc -b; the worker also runs build:bundle |
typecheck | ^build | default | tsc -p tsconfig.test.json |
test | ^build | default | vitest run |
lint | ^build, build | default | eslint . |
test:real | ^build | cache: false | only apps/smoke defines it |
Global settings: globalDependencies are pnpm-lock.yaml and pnpm-workspace.yaml; globalEnv is NODE_ENV, TEMPORAL_TEST_ADDRESS, TEMPORAL_TEST_NAMESPACE; globalPassThroughEnv is RUN_REAL, ANTHROPIC_API_KEY, DATABASE_URL, TEMPORAL_TEST_STARTUP_TIMEOUT_MS (secrets pass through without entering the hash).
Package scripts beyond the common four:
| Package | Script | Purpose |
|---|---|---|
@agentforge/gateway | test:gate (vitest.gate.config.ts) | GW8/GW8b against the RK-G4 recording; never part of test |
@agentforge/agent-workflows | test:replay | test/replay.test.ts over recorded workflow histories |
@agentforge/db | db:generate | drizzle-kit generate (used by the CI drift step) |
@agentforge/smoke | test:real | real-key harness, refuses without caps |
@agentforge/worker | start, start:factory, start:quality-exec | entries dist/main.js, dist/factory-main.js, dist/quality-exec-main.js |
Package boundaries are enforced by turbo boundaries with tags types, core, db, adapter, workflows, client, governance, mcp, context, each denying the tags listed in turbo.json. In CI the step is advisory (continue-on-error: true). packages/quality-core has no boundary tag yet (bead .110).
Cache proof: scripts/cache.test.sh checks that turbo task hashes change after a globalEnv change, a lockfile change and a source edit, and that an unchanged second run is FULL TURBO. It restores every file it touches and is not part of CI.
The smoke app
apps/smoke is the private harness that composes the guarded runtime over Postgres for real-key runs.
- Its default
testrun is keyless.test:real(vitest.real.config.ts) is the only entry that spends money, and aglobalSetuprefuses to start unless the preconditions hold. - The smoke cap is the highest daily limit a real-key run accepts, bounded by a fixed hard ceiling in
apps/smoke/src/compose.ts. Runbook defaults:DATABASE_URLset to the local container,CLIENT_DAILY_LIMIT_USD=1,GLOBAL_DAILY_LIMIT_USD=2,SMOKE_CLIENT_CAP_USD=1,SMOKE_GLOBAL_CAP_USD=2,AGENTS_DISABLED=0. Do not raise any limit or cap. - Its vitest config sets a 30 s test and hook timeout (commit
22aebab) aftercompose.test.tshit the 5 s default on a loaded runner.
The first real-key runbook is docs/p1-realkey-runbook.md; results are in docs/p1-findings.md and docs/p2-findings.md.
Image tests
Each script builds on a built image and cleans up after itself. Throwaway Postgres 16 containers run on private Docker networks.
| Script | Image under test | Asserts |
|---|---|---|
scripts/api-image.test.sh | agentforge-api | non-root, no secrets shipped, exactly one Claude CLI platform package for the libc and arch, migrations run, container turns healthy (API-16) |
scripts/gateway-image.test.sh | agentforge-gateway | non-root, no secrets, no Claude Agent SDK, refuses an unsafe config, healthy under tini (G-P2G-MP) |
scripts/gateway-gv3.test.sh | agentforge-gateway plus a stand-in runtime | on two --internal networks, no published ports, canary scans, SEC10b and SEC12; probes in scripts/gv3-standin-probe.mjs |
scripts/worker-image.test.sh | agentforge-worker | non-root, /app read-only, /workspace present, prebuilt workflow bundle, one glibc Claude CLI package, the core-bridge native binary loads, PID 1 is tini (TMP-13) |
scripts/exec-image.test.sh | worker exec target | non-root, PID 1 tini, CMD dist/exec-main.js, refuses DATABASE_URL, ANTHROPIC_API_KEY, GATEWAY_CONTROL_*, GITHUB_APP_* (FL-4) |
scripts/factory-image.test.sh | worker factory target | non-root, /app read-only, factory entry, host git 2.50.1 or newer with https, docker CLI, no model or GitHub key in the env, tini, private work dir |
scripts/sandbox-image.test.sh | agentforge-sandbox | sandbox image and driver checks (P2.5-1, P2.5-2) |
scripts/quality-image.test.sh | quality image | builds the image keyed by lockfile hash, runs analyze.mjs and one run.mjs chunk with the sandbox argv (CC-2.1) |
Build commands:
docker build -f apps/api/Dockerfile -t agentforge-api . && bash scripts/api-image.test.sh agentforge-api
docker build -f apps/gateway/Dockerfile -t agentforge-gateway . && bash scripts/gateway-image.test.sh agentforge-gateway
bash scripts/gateway-gv3.test.sh agentforge-gateway
docker build -f apps/worker/Dockerfile --target exec -t agentforge-worker-exec . && bash scripts/exec-image.test.sh agentforge-worker-exec
Real-Docker scripts
Local only (OrbStack on the laptop). Run one docker script at a time; use git add <own paths> when several agents share a tree.
| Script | What runs for real | What is scripted |
|---|---|---|
scripts/sandbox-e2e.test.sh | gateway image, Postgres, SandboxRuntime via composeRuntime, sandbox image and CLI | the model (scripts/p25/fake-upstream.mjs: RUN<<cmd>>RUN becomes a Bash tool use, STALL streams forever, BIGUSAGE reports huge input tokens) |
scripts/session-gateway-e2e.test.sh | session turns through the gateway image with only its baked /policies, Postgres, the worker's real config, gateway-mode ClaudeRuntime (FL-2) | the model |
scripts/fleet-split-e2e.test.sh | the real core worker and two real exec workers (no DATABASE_URL, no control token) on a local Temporal dev server (FL-3) | the model |
scripts/exec-container-e2e.test.sh | the exec worker as a container from the Linux image, over https to the gateway with a throwaway CA, real CLI (FL-E2E2) | the model |
scripts/factory-e2e.test.sh | gateway image, Postgres, sandbox image, factory activities and workflow on a Temporal dev server, LocalGitHost | the model and GitHub |
scripts/quality-exec.test.sh | exec activities with the real SandboxTestRunner and the quality image, packages/agent-core against a local bare mirror | nothing model-related (no model is used) |
scripts/quality-verify.test.sh <image> | verify steps 5 to 7 (solo check, materialized re-kill, controls) | nothing model-related |
scripts/sandbox-rk.test.sh | real model, real key; aborts when the ledger passes 1.00 USD (expected well under 0.10) | none |
scripts/factory-gh.test.sh | one factory job against real GitHub (throwaway repo and its fork), real model, real key | none |
scripts/audit-prt.sh | audits workflows of checked-out repos for the pwn-request pattern (pull_request_target, workflow_run, issue_comment referencing PR code) | not a test; FX-AUD1 |
Measured local timings (OrbStack, runc, machine under load): quality image build 70 to 276 s and 1.79 GB; analyze 35 to 92 s with 417 trusted mutants in 8 chunks; exec activities end to end 122 s; solo check plus re-kill plus controls 195 s.
CI workflows
Three workflow files under .github/workflows/.
ci.yml
Triggers: push to main, every pull request, and manual dispatch with input full (default true: run every task with --force instead of --affected, for gate evidence). Permissions contents: read.
Details of the build job:
--affectedusesTURBO_SCM_BASE(the PR base commit, or the previous tip on push). Without a resolvable base turbo warns and runs everything, which is safe but slower.fetch-depth: 0is needed because shallow clones silently fall back to run-all.- Postgres runs as a plain container (
docker run -d --name ci-postgres-<run id>-<attempt> ... -p 127.0.0.1::5432 postgres:16) becauseservices:works only on Linux runners and a fixed port can collide on this shared Mac. It waits for the log linedatabase system is ready to accept connectionstwice, because the image starts the server twice on first boot. If it never appears the step fails withpostgres did not become ready. - Node: the step detects Node 24.20 or newer on the runner and skips
setup-node(its cache save and restore has wedged runners) unless the runner lacks it. - The Temporal test environment downloads about 190 MB of binaries into
os.tmpdir()on first use. The runner keeps its tmpdir between runs, so the cache restore runs only when the binaries are missing, and the save runs only if the files exist after the run (--affectedmay skip every Temporal test). - Migrations drift:
pnpm --filter @agentforge/db db:generatemust print noError, leavepackages/db/drizzleunchanged bygit diff --exit-code, and leavegit status --porcelain packages/db/drizzleempty.
Details of the docker job: timeout-minutes: 30 (a warm run takes about 10), BuildKit with --progress=plain, and runner-state snapshots before and after (uptime, df -h /, docker system df, other docker or buildkit processes by name only, never command lines, because other jobs' build args can hold secrets GitHub does not mask).
Details of the gitleaks job: it uses the CLI on the host, not the container action (the action needs a paid license for organization repos, and a container mount of this runner's workspace is empty because OrbStack does not share /usr/local). It requires gitleaks version to equal 8.30.1, scans with --redact --log-opts=--all and GIT_CONFIG_GLOBAL=/dev/null, and then requires the scanned commit count to be greater than zero and at least the number of commits that carry a text hunk. gitleaks reports "no leaks found" with exit 0 when it reads no history, which is why the count guard exists.
docker-release.yml
Builds and mirrors images; see Release process.
docs-site.yml
The documentation site (website/) is its own pnpm project outside the repository workspace, so ci.yml never builds it. This workflow runs on pushes to main and pull requests that touch website/** or the workflow file, and on manual dispatch. Steps: checkout, pnpm, a Node 24.20 or newer check, pnpm install --frozen-lockfile, pnpm run check:diagrams (every Mermaid block must parse, because the browser renders them and docusaurus build alone would not notice), then pnpm run build (broken links and anchors fail it). Timeout 20 minutes, runner group ci-runners.
Runner constraints
| Constraint | Detail |
|---|---|
| One shared self-hosted runner group | ci-runners, labels self-hosted, macOS, ARM64, the owner's laptop. It also runs other sessions, local stacks and other products' jobs |
| Queueing | A busy or offline runner leaves runs queued. At the end of the 2026-11-16 build session five runs (c21f154 to 5039f88) were still queued behind a load average of about 51, so no commit since a7c4e2d was confirmed green by CI |
| Docker | OrbStack. It does not share /usr/local with containers, so mounting the workspace into a container sees an empty directory |
| Node | 24.20 or newer expected on the runner (engines.node >= 24) |
| Disk | A full runner disk made two release builds fail with no space left on device; the release workflow now refuses below 20 GB free |
| Architecture | The runner is ARM64; deploy images are linux/amd64, so the release platform is explicit |
| Secrets | The build job has no deployment secrets. Release auth is GitHub OIDC only |
Flake handling
Known CI-only traps and their fixes, from the dossiers:
| Trap | Fix |
|---|---|
CI=true changes typescript-estree to parse from disk, ignoring lintText input | Set TSESTREE_SINGLE_RUN=false in lint-probe tests; reproduce locally with CI=true |
| Node keep-alive reuses sockets the server closed after error responses | Use agent:false in raw-request test helpers |
| Shared CI Postgres plus parallel packages makes 5 s and 10 s vitest defaults time out | DB tests use 30 s and 60 s; never stop the shared Postgres in tests, inject faults with a pool wrapper |
agent-client ESLint test timeout under load | Run turbo with --concurrency=3 (later sessions used --concurrency=2) |
apps/smoke/test/compose.test.ts hit the 5 s default on a loaded runner, then cascaded (DROP DATABASE FORCE killed a running migration) | 30 s testTimeout and hookTimeout in apps/smoke/vitest.config.ts |
| Gateway timer test "a client that never reads: the total timer ends the call too" failed once in CI and passed 5 of 5 locally | UNVERIFIED flake on a shared runner; if it recurs treat it as a real regression of apps/gateway/src/control.ts and bisect with 97ddf69 |
Load average 50 to 64 (other sessions) made ESLint tests take 17 to 72 s, afterAll pool timeouts hit 6 to 10 API files, and "Start Postgres" print did not become ready in three runs | Files re-run alone passed. Run turbo with --concurrency=2, re-run single files, and check uptime before blaming code |
api-image Postgres readiness race | Wait for the ready line twice (8af6cc9) |
| Quiet CI job for twelve minutes | 30 minute timeout, plain BuildKit progress, runner snapshots |
Rules for evidence: a green agent report is not evidence. The lead re-runs on the merged main (turbo build, typecheck, test, lint, format:check, the db drift check) and the real-docker scripts where a bead has one. Wait for the final agent report with a SHA.
Local gate
The full gate that dossiers quote as "24/24" is:
pnpm turbo run build typecheck test lint --force
pnpm format:check
pnpm --filter @agentforge/db db:generate && git diff --exit-code packages/db/drizzle
Long scripts: use the foreground ctx_execute form with a 600 s timeout, and write a log file when the output matters (a background form can lose its output).
Sources
.github/workflows/ci.yml,.github/workflows/docker-release.yml,.github/workflows/docs-site.ymlturbo.json,package.json,pnpm-workspace.yaml,docker-compose.ymlCODING_STANDARDS.mdsection 13apps/*/package.json,packages/*/package.json(scripts)apps/gateway/vitest.gate.config.tsscripts/api-image.test.sh,gateway-image.test.sh,gateway-gv3.test.sh,gv3-standin-probe.mjs,worker-image.test.sh,exec-image.test.sh,factory-image.test.sh,sandbox-image.test.sh,quality-image.test.shscripts/sandbox-e2e.test.sh,session-gateway-e2e.test.sh,fleet-split-e2e.test.sh,exec-container-e2e.test.sh,factory-e2e.test.sh,quality-exec.test.sh,quality-verify.test.sh,sandbox-rk.test.sh,factory-gh.test.sh,audit-prt.sh,cache.test.sh,p25/fake-upstream.mjsdocs/p1-realkey-runbook.mddocs/sessions/2026-11-10-agentforge-p2b-to-p2g-s4/README.md,LESSONS.mddocs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/README.md,LESSONS.md,FOLLOWUPS.mddocs/sessions/2026-11-15-app03-deploy-and-smoke/SUMMARY.md,LESSONS.mddocs/sessions/2026-11-16-codecrucible-build/README.md,LESSONS.mddocs/deploy-quality.md(local evidence table)