Skip to main content

Testing and CI

This page describes the test pyramid, the turbo tasks, the Docker image tests, the smoke app, the real-Docker scripts, the CI workflow jobs, the runner constraints and how flakes are handled. Convention source: CODING_STANDARDS.md section 13.

The test pyramid​

LayerRuns whereSpends moneyTrigger
Unit and integration (vitest)CI and localnopnpm turbo run test
Static (typecheck, lint, format, boundaries, migration drift, gitleaks)CI and localnoCI
Image testsCI (api, gateway, GV3) and local (the rest)nobash scripts/<name>.test.sh <image>
Real-Docker keyless end to endlocal only, one agent at a timenoscripts below
Real-key and real-GitHub runsuser-run onlyyes (cents)test:real, sandbox-rk, factory-gh

Only the user runs real-key tests. Keyless tests are the default.

Test conventions​

From CODING_STANDARDS.md section 13:

  • Framework vitest with explicit imports (no globals). Tests live in test/ beside src/, import the package through ../src/index.js and other packages by package name.
  • Name each describe after the plan criterion it proves, prefix form first, for example describe('P1-12 kill switch', ...). Titles state behavior and outcome.
  • Database tests run against real Postgres, never mocks: start it with docker compose up -d --wait (host port 5433; CI uses a throwaway container on a random host port). Each test file creates and drops its own database with createTestDatabase(), each test resets with TRUNCATE in beforeEach, and day-boundary cases pass explicit at dates.
  • Deterministic tests: injected clocks, vi.useFakeTimers({ toFake: [...] }) with an explicit list. Randomized runs only when the expectation is computed independently of event order (packages/db/test/concurrency.test.ts).
  • Every fail-closed path is tested: store down, throwing listener, invalid cost, unscrubbable event, stuck runtime.
  • Real-key tests live in test/real/*.real.test.ts and run only through test:real (cache: false). That config has a globalSetup that refuses before any test file loads, and each file repeats the guard. apps/smoke/test/guard.test.ts asserts the config split.
  • Structural invariants are tests too: dependency sets, SDK resolution, ESLint probes, the export list of compose.ts.
  • Workflow replay tests run over recorded histories (pnpm --filter @agentforge/agent-workflows test:replay). The Docker doc test apps/worker/test/sandbox-doc.test.ts fails when a flag the driver emits is missing from docs/deploy-sandbox.md.

Turbo tasks (turbo.json)​

TaskdependsOnCacheNotes
build^buildoutputs dist/**tsc -b; the worker also runs build:bundle
typecheck^builddefaulttsc -p tsconfig.test.json
test^builddefaultvitest run
lint^build, builddefaulteslint .
test:real^buildcache: falseonly apps/smoke defines it

Global settings: globalDependencies are pnpm-lock.yaml and pnpm-workspace.yaml; globalEnv is NODE_ENV, TEMPORAL_TEST_ADDRESS, TEMPORAL_TEST_NAMESPACE; globalPassThroughEnv is RUN_REAL, ANTHROPIC_API_KEY, DATABASE_URL, TEMPORAL_TEST_STARTUP_TIMEOUT_MS (secrets pass through without entering the hash).

Package scripts beyond the common four:

PackageScriptPurpose
@agentforge/gatewaytest:gate (vitest.gate.config.ts)GW8/GW8b against the RK-G4 recording; never part of test
@agentforge/agent-workflowstest:replaytest/replay.test.ts over recorded workflow histories
@agentforge/dbdb:generatedrizzle-kit generate (used by the CI drift step)
@agentforge/smoketest:realreal-key harness, refuses without caps
@agentforge/workerstart, start:factory, start:quality-execentries dist/main.js, dist/factory-main.js, dist/quality-exec-main.js

Package boundaries are enforced by turbo boundaries with tags types, core, db, adapter, workflows, client, governance, mcp, context, each denying the tags listed in turbo.json. In CI the step is advisory (continue-on-error: true). packages/quality-core has no boundary tag yet (bead .110).

Cache proof: scripts/cache.test.sh checks that turbo task hashes change after a globalEnv change, a lockfile change and a source edit, and that an unchanged second run is FULL TURBO. It restores every file it touches and is not part of CI.

The smoke app​

apps/smoke is the private harness that composes the guarded runtime over Postgres for real-key runs.

  • Its default test run is keyless. test:real (vitest.real.config.ts) is the only entry that spends money, and a globalSetup refuses to start unless the preconditions hold.
  • The smoke cap is the highest daily limit a real-key run accepts, bounded by a fixed hard ceiling in apps/smoke/src/compose.ts. Runbook defaults: DATABASE_URL set to the local container, CLIENT_DAILY_LIMIT_USD=1, GLOBAL_DAILY_LIMIT_USD=2, SMOKE_CLIENT_CAP_USD=1, SMOKE_GLOBAL_CAP_USD=2, AGENTS_DISABLED=0. Do not raise any limit or cap.
  • Its vitest config sets a 30 s test and hook timeout (commit 22aebab) after compose.test.ts hit the 5 s default on a loaded runner.

The first real-key runbook is docs/p1-realkey-runbook.md; results are in docs/p1-findings.md and docs/p2-findings.md.

Image tests​

Each script builds on a built image and cleans up after itself. Throwaway Postgres 16 containers run on private Docker networks.

ScriptImage under testAsserts
scripts/api-image.test.shagentforge-apinon-root, no secrets shipped, exactly one Claude CLI platform package for the libc and arch, migrations run, container turns healthy (API-16)
scripts/gateway-image.test.shagentforge-gatewaynon-root, no secrets, no Claude Agent SDK, refuses an unsafe config, healthy under tini (G-P2G-MP)
scripts/gateway-gv3.test.shagentforge-gateway plus a stand-in runtimeon two --internal networks, no published ports, canary scans, SEC10b and SEC12; probes in scripts/gv3-standin-probe.mjs
scripts/worker-image.test.shagentforge-workernon-root, /app read-only, /workspace present, prebuilt workflow bundle, one glibc Claude CLI package, the core-bridge native binary loads, PID 1 is tini (TMP-13)
scripts/exec-image.test.shworker exec targetnon-root, PID 1 tini, CMD dist/exec-main.js, refuses DATABASE_URL, ANTHROPIC_API_KEY, GATEWAY_CONTROL_*, GITHUB_APP_* (FL-4)
scripts/factory-image.test.shworker factory targetnon-root, /app read-only, factory entry, host git 2.50.1 or newer with https, docker CLI, no model or GitHub key in the env, tini, private work dir
scripts/sandbox-image.test.shagentforge-sandboxsandbox image and driver checks (P2.5-1, P2.5-2)
scripts/quality-image.test.shquality imagebuilds the image keyed by lockfile hash, runs analyze.mjs and one run.mjs chunk with the sandbox argv (CC-2.1)

Build commands:

docker build -f apps/api/Dockerfile -t agentforge-api . && bash scripts/api-image.test.sh agentforge-api
docker build -f apps/gateway/Dockerfile -t agentforge-gateway . && bash scripts/gateway-image.test.sh agentforge-gateway
bash scripts/gateway-gv3.test.sh agentforge-gateway
docker build -f apps/worker/Dockerfile --target exec -t agentforge-worker-exec . && bash scripts/exec-image.test.sh agentforge-worker-exec

Real-Docker scripts​

Local only (OrbStack on the laptop). Run one docker script at a time; use git add <own paths> when several agents share a tree.

ScriptWhat runs for realWhat is scripted
scripts/sandbox-e2e.test.shgateway image, Postgres, SandboxRuntime via composeRuntime, sandbox image and CLIthe model (scripts/p25/fake-upstream.mjs: RUN<<cmd>>RUN becomes a Bash tool use, STALL streams forever, BIGUSAGE reports huge input tokens)
scripts/session-gateway-e2e.test.shsession turns through the gateway image with only its baked /policies, Postgres, the worker's real config, gateway-mode ClaudeRuntime (FL-2)the model
scripts/fleet-split-e2e.test.shthe real core worker and two real exec workers (no DATABASE_URL, no control token) on a local Temporal dev server (FL-3)the model
scripts/exec-container-e2e.test.shthe exec worker as a container from the Linux image, over https to the gateway with a throwaway CA, real CLI (FL-E2E2)the model
scripts/factory-e2e.test.shgateway image, Postgres, sandbox image, factory activities and workflow on a Temporal dev server, LocalGitHostthe model and GitHub
scripts/quality-exec.test.shexec activities with the real SandboxTestRunner and the quality image, packages/agent-core against a local bare mirrornothing model-related (no model is used)
scripts/quality-verify.test.sh <image>verify steps 5 to 7 (solo check, materialized re-kill, controls)nothing model-related
scripts/sandbox-rk.test.shreal model, real key; aborts when the ledger passes 1.00 USD (expected well under 0.10)none
scripts/factory-gh.test.shone factory job against real GitHub (throwaway repo and its fork), real model, real keynone
scripts/audit-prt.shaudits workflows of checked-out repos for the pwn-request pattern (pull_request_target, workflow_run, issue_comment referencing PR code)not a test; FX-AUD1

Measured local timings (OrbStack, runc, machine under load): quality image build 70 to 276 s and 1.79 GB; analyze 35 to 92 s with 417 trusted mutants in 8 chunks; exec activities end to end 122 s; solo check plus re-kill plus controls 195 s.

CI workflows​

Three workflow files under .github/workflows/.

ci.yml​

Triggers: push to main, every pull request, and manual dispatch with input full (default true: run every task with --force instead of --affected, for gate evidence). Permissions contents: read.

Details of the build job:

  • --affected uses TURBO_SCM_BASE (the PR base commit, or the previous tip on push). Without a resolvable base turbo warns and runs everything, which is safe but slower. fetch-depth: 0 is needed because shallow clones silently fall back to run-all.
  • Postgres runs as a plain container (docker run -d --name ci-postgres-<run id>-<attempt> ... -p 127.0.0.1::5432 postgres:16) because services: works only on Linux runners and a fixed port can collide on this shared Mac. It waits for the log line database system is ready to accept connections twice, because the image starts the server twice on first boot. If it never appears the step fails with postgres did not become ready.
  • Node: the step detects Node 24.20 or newer on the runner and skips setup-node (its cache save and restore has wedged runners) unless the runner lacks it.
  • The Temporal test environment downloads about 190 MB of binaries into os.tmpdir() on first use. The runner keeps its tmpdir between runs, so the cache restore runs only when the binaries are missing, and the save runs only if the files exist after the run (--affected may skip every Temporal test).
  • Migrations drift: pnpm --filter @agentforge/db db:generate must print no Error, leave packages/db/drizzle unchanged by git diff --exit-code, and leave git status --porcelain packages/db/drizzle empty.

Details of the docker job: timeout-minutes: 30 (a warm run takes about 10), BuildKit with --progress=plain, and runner-state snapshots before and after (uptime, df -h /, docker system df, other docker or buildkit processes by name only, never command lines, because other jobs' build args can hold secrets GitHub does not mask).

Details of the gitleaks job: it uses the CLI on the host, not the container action (the action needs a paid license for organization repos, and a container mount of this runner's workspace is empty because OrbStack does not share /usr/local). It requires gitleaks version to equal 8.30.1, scans with --redact --log-opts=--all and GIT_CONFIG_GLOBAL=/dev/null, and then requires the scanned commit count to be greater than zero and at least the number of commits that carry a text hunk. gitleaks reports "no leaks found" with exit 0 when it reads no history, which is why the count guard exists.

docker-release.yml​

Builds and mirrors images; see Release process.

docs-site.yml​

The documentation site (website/) is its own pnpm project outside the repository workspace, so ci.yml never builds it. This workflow runs on pushes to main and pull requests that touch website/** or the workflow file, and on manual dispatch. Steps: checkout, pnpm, a Node 24.20 or newer check, pnpm install --frozen-lockfile, pnpm run check:diagrams (every Mermaid block must parse, because the browser renders them and docusaurus build alone would not notice), then pnpm run build (broken links and anchors fail it). Timeout 20 minutes, runner group ci-runners.

Runner constraints​

ConstraintDetail
One shared self-hosted runner groupci-runners, labels self-hosted, macOS, ARM64, the owner's laptop. It also runs other sessions, local stacks and other products' jobs
QueueingA busy or offline runner leaves runs queued. At the end of the 2026-11-16 build session five runs (c21f154 to 5039f88) were still queued behind a load average of about 51, so no commit since a7c4e2d was confirmed green by CI
DockerOrbStack. It does not share /usr/local with containers, so mounting the workspace into a container sees an empty directory
Node24.20 or newer expected on the runner (engines.node >= 24)
DiskA full runner disk made two release builds fail with no space left on device; the release workflow now refuses below 20 GB free
ArchitectureThe runner is ARM64; deploy images are linux/amd64, so the release platform is explicit
SecretsThe build job has no deployment secrets. Release auth is GitHub OIDC only

Flake handling​

Known CI-only traps and their fixes, from the dossiers:

TrapFix
CI=true changes typescript-estree to parse from disk, ignoring lintText inputSet TSESTREE_SINGLE_RUN=false in lint-probe tests; reproduce locally with CI=true
Node keep-alive reuses sockets the server closed after error responsesUse agent:false in raw-request test helpers
Shared CI Postgres plus parallel packages makes 5 s and 10 s vitest defaults time outDB tests use 30 s and 60 s; never stop the shared Postgres in tests, inject faults with a pool wrapper
agent-client ESLint test timeout under loadRun turbo with --concurrency=3 (later sessions used --concurrency=2)
apps/smoke/test/compose.test.ts hit the 5 s default on a loaded runner, then cascaded (DROP DATABASE FORCE killed a running migration)30 s testTimeout and hookTimeout in apps/smoke/vitest.config.ts
Gateway timer test "a client that never reads: the total timer ends the call too" failed once in CI and passed 5 of 5 locallyUNVERIFIED flake on a shared runner; if it recurs treat it as a real regression of apps/gateway/src/control.ts and bisect with 97ddf69
Load average 50 to 64 (other sessions) made ESLint tests take 17 to 72 s, afterAll pool timeouts hit 6 to 10 API files, and "Start Postgres" print did not become ready in three runsFiles re-run alone passed. Run turbo with --concurrency=2, re-run single files, and check uptime before blaming code
api-image Postgres readiness raceWait for the ready line twice (8af6cc9)
Quiet CI job for twelve minutes30 minute timeout, plain BuildKit progress, runner snapshots

Rules for evidence: a green agent report is not evidence. The lead re-runs on the merged main (turbo build, typecheck, test, lint, format:check, the db drift check) and the real-docker scripts where a bead has one. Wait for the final agent report with a SHA.

Local gate​

The full gate that dossiers quote as "24/24" is:

pnpm turbo run build typecheck test lint --force
pnpm format:check
pnpm --filter @agentforge/db db:generate && git diff --exit-code packages/db/drizzle

Long scripts: use the foreground ctx_execute form with a 600 s timeout, and write a log file when the output matters (a background form can lose its output).

Sources​

  • .github/workflows/ci.yml, .github/workflows/docker-release.yml, .github/workflows/docs-site.yml
  • turbo.json, package.json, pnpm-workspace.yaml, docker-compose.yml
  • CODING_STANDARDS.md section 13
  • apps/*/package.json, packages/*/package.json (scripts)
  • apps/gateway/vitest.gate.config.ts
  • scripts/api-image.test.sh, gateway-image.test.sh, gateway-gv3.test.sh, gv3-standin-probe.mjs, worker-image.test.sh, exec-image.test.sh, factory-image.test.sh, sandbox-image.test.sh, quality-image.test.sh
  • scripts/sandbox-e2e.test.sh, session-gateway-e2e.test.sh, fleet-split-e2e.test.sh, exec-container-e2e.test.sh, factory-e2e.test.sh, quality-exec.test.sh, quality-verify.test.sh, sandbox-rk.test.sh, factory-gh.test.sh, audit-prt.sh, cache.test.sh, p25/fake-upstream.mjs
  • docs/p1-realkey-runbook.md
  • docs/sessions/2026-11-10-agentforge-p2b-to-p2g-s4/README.md, LESSONS.md
  • docs/sessions/2026-11-13-agentforge-app02-test-and-worker-fleet/README.md, LESSONS.md, FOLLOWUPS.md
  • docs/sessions/2026-11-15-app03-deploy-and-smoke/SUMMARY.md, LESSONS.md
  • docs/sessions/2026-11-16-codecrucible-build/README.md, LESSONS.md
  • docs/deploy-quality.md (local evidence table)