Testing and CI

CI scope and routing

Scope and routing

Scope logic lives in scripts/ci-changed-scope.mjs and is covered by unit tests in src/scripts/ci-changed-scope.test.ts. Ordinary manual dispatch skips changed-scope detection and makes the preflight manifest act as if every scoped area changed. The exact-head release_gate exception evaluates the fetched pull request merge tree and retains its macOS, iOS-build, and generated-native-locale decisions while still verifying native sources.

Affected pull requests, main pushes, and exact-head release_gate fallbacks run one required ios-build (smoke) phase: the existing pnpm ios:build command and Swift lint, with their Xcode, Swift, and Watch Rust tooling. They do not run Rust tests, lifecycle or Watch simulator tests, or screenshot capture. A failed, cancelled, or unexpectedly skipped smoke still fails openclaw/ci-gate.

Ordinary manual CI (workflow_dispatch with release_gate=false and release_scope=full), including full-scope release validation, retains separate Release device and Debug/native-test phases plus the full iPhone, iPad, and Watch screenshot matrix. The test phase runs the Rust engine, lifecycle, and Watch operation tests. The two device shards build their own simulator apps independently of ios-build; scenarios stay serial within each device, with Watch evidence in the iPad shard. A hosted reducer verifies the exact evidence union and provenance before publishing the canonical screenshot artifact. The final gate requires both build phases, both screenshot shards, and the reducer. Frozen compatibility targets retain their existing Debug-only full-manual contract and screenshot exclusion; npm-beta and npm-stable still defer native qualification. All iOS build phases and screenshot shards use GitHub-hosted macos-26. A pure iOS app change does not select macOS jobs by itself.

The iPad shard captures Watch from the Watch app compiled by its own fresh screenshot build, after all four iPad captures pass. It does not rebuild Watch or transfer simulator products between jobs. The standalone watch_screenshot lane still prepares and builds a fresh Watch app. A missing or invalid product fails capture without a rebuild fallback; completed iOS screenshots and result bundles remain available in the failure artifact.

Separate iOS and macOS Periphery workflows enforce a zero-findings dead-code policy. Each runs only when a non-draft pull request touches its native scan scope, or when manually dispatched.

The shared PR commenter reads each producer's fixed run title to distinguish report admission, explicit converted_to_draft cleanup, passive draft events, and manual runs. Reports require a live open PR at the same repository and head with draft status off; draft cleanup requires draft status on. Passive runs cannot publish or supersede reports. Newer eligible runs and attempts supersede older results, including while pending. A report-admitted run that successfully detects no scan scope can clear an existing comment; draft cleanup names draft status instead of claiming scope loss. The commenter rechecks PR state, repository, head, and draft status immediately before writing, but separate REST calls are not atomic.

Runs without recognized admission metadata are logged no-ops: they neither publish nor supersede. After rollout, a new pull-request source event is needed to produce an eligible run; rerunning an old unmarked run does not recover its original admission intent.

The iOS, macOS, and both shared OpenClawKit Periphery scans always use GitHub-hosted macos-26. This transfers four existing scan registrations from blacksmith-12vcpu-macos-26 to hosted capacity for eligible same-repository first attempts while preserving their scope, workloads, timeouts, artifacts, and rerun behavior.

  • CI workflow edits validate the Node CI graph, workflow linting, and the Windows lane (ci.yml executes it), but do not force iOS, Android, or macOS native builds by themselves; those platform lanes stay scoped to platform source changes.
  • Git-owner changes to its action, base-commit policy, projection generator, lifecycle tests and support, or named owner-adopting workflows such as Workflow Sanity, QA Profile Evidence, Mantis ref validation/installers/worktrees, Docs Sync Publish Repo, OpenClaw Performance, the Linux/macOS/npm-placeholder release admission jobs, and plugin ClawHub/npm publication select the existing macos-node and Windows lanes. These run native checkout ownership proof without selecting Swift, iOS, or Android jobs; Mac app and shared-native changes retain their existing Mac lanes.
  • macOS Swift runner budgets are 30 minutes per worker on GitHub-hosted macos-26, including automatic first attempts. Regular PR/main CI and PR release_gate dispatches run only tests: Swift lint, schema checks, the Talk opt-out build, shared package suite, standalone Swabble suite for current targets, and app coverage build/tests. Ordinary full-scope manual validation adds the independent release app build, moves lint/schema ownership to that phase, and retains health renders in tests. Every selected phase must pass the existing CI gate. At most two workers run concurrently; a failed phase does not cancel the other phase's diagnostics.
  • macOS fixture support changes select the existing Mac Node gate so the shared managed-command and concurrency owner receives Darwin proof independently of Swift/app changes.
  • macOS Swift build caches retain the original nanosecond timestamps and content hashes of their source inputs inside apps/macos/.build. The restore helper replays timestamps only for byte-identical regular files with matching permissions in the current input inventory; changed, missing, linked, or invalid entries keep their checkout metadata and invalidate through SwiftPM normally. The v6 archive keys include the phase, helper, toolchain, package graph, and source identities, with same-phase, same-graph prefix reuse. Each phase starts with a cold seed instead of restoring the former combined build archive, then records metadata immediately before its own trusted save. Both phases may restore the shared SwiftPM dependency cache; its sole eligible writer is tests in regular CI or release in full validation. Candidate cache trust is unchanged: cache-off validation compiles every selected phase cold. Historical targets retain their target-owned build commands.
  • Workflow Sanity runs actionlint, zizmor over all workflow YAML files, the composite-action interpolation guard, and the conflict-marker guard. The PR-scoped security-fast job also runs zizmor over changed workflow files so workflow security findings fail early in the main CI graph.
  • Docs on main pushes are checked by the standalone Docs workflow with the same ClawHub docs mirror used by CI, so mixed code+docs pushes do not also queue the CI check-docs shard. Pull requests and manual CI still run check-docs from CI when docs changed.
  • TUI PTY runs two built-CLI artifact canaries in build-artifacts: a local model roundtrip and a real Gateway connection. The complete suite is defined in test/vitest/vitest.tui-pty.config.ts; canonical pull-request fallbacks and manual/release full plans retain its core-runtime-tui-pty descriptor. CI consumes that descriptor only through the built-artifact selection flag, so the full suite has no executing matrix row; manual and release CI also run only the canaries. Canonical main push compaction omits the full descriptor while keeping the canaries.
  • SQLite session lifecycle runs the built-CLI migration, restart, compaction, cleanup, and session RPC proof only when the diff touches its direct storage/session owners or a reachable session path in the embedded runner. The build-artifacts verifier wave runs it against the runtime already built in that job, after the isolated startup-memory measurement. It overlaps independent readers on Blacksmith and stays serial on hosted runners; manual and release dispatches always select it when the target contains the proof.
  • CI routing-only edits, the small set of core-test fixtures the fast task runs directly, and narrow plugin contract helper edits use a fast Node-only manifest path: preflight, security-fast, and only the fast lanes the change touches — a single checks-fast-core CI-routing task, the plugin contract job, or both. That path skips build artifacts, Node 22 compatibility, channel contracts, full core shards, bundled-plugin shards, and additional guard matrices.
  • QA Smoke on pull requests runs only when the diff touches a QA-owned surface: the qa-lab harness, qa/ scenario data, the matrix/telegram channels the smoke profile drives, the docker packaging scripts, or the gate's own orchestration. Broad runtime changes (src, ui, packages, dependency manifests) no longer select the six-part smoke matrix per PR; every canonical main push and release validation still runs the full profile set, so runtime regressions surface one push after merge instead of taxing every PR with roughly five extra hosted-runner minutes.
  • Windows Node checks are scoped to Windows-specific process/path wrappers, npm/pnpm/UI runner helpers, package manager config, and the CI workflow surfaces that execute that lane; unrelated source, plugin, install-smoke, and test-only changes stay on the Linux Node lanes. Test-only changes to any explicit target in test:windows:ci:1 or test:windows:ci:2 also select the existing Windows lane; these package scripts own its test inventory.

The slowest Node test families are split or balanced so each job stays small without over-reserving runners:

  • Plugin contracts and channel contracts each retain two weighted process selections in one job, with the standard GitHub runner fallback. Frozen targets keep separate jobs.
  • Additional checks: Current targets combine export-collision, session accessor/transcript reader, SQLite transaction, and SQLite schema checks into one serial source-contract row, preserving each command and collecting every failure. Additional checks use five rows on push/PR and six on current-target dispatch; the SDK API report is allocated only for dispatch. Frozen targets retain their eight original rows and historical command fallbacks.
  • Core unit fast/support lanes run separately; unit-src, Control UI, and gateway-core each use three deterministic file-weighted stripes, while the security and media/UI companion configs retain their scoped whole-config support groups; core runtime infra splits into process, shared, hooks, secrets, and three cron domain shards.
  • Auto-reply runs as balanced workers, with the reply subtree split into agent-runner, commands, dispatch, session, and state-routing shards; dispatch further isolates core, delivery, and lifecycle entrypoints.
  • Agentic gateway/server (control-plane) configs split across chat, auth, model, HTTP/plugin, runtime, and startup lanes instead of waiting on built artifacts.
  • Normal CI packs only isolated infra include-pattern shards into deterministic bundles of at most 64 test files, reducing the Node matrix without merging non-isolated command/cron, stateful agents-core, or gateway/server suites. Heavy fixed suites stay on 8 vCPU while most bundled and lower-weight lanes use 4 vCPU. Previously promoted compact workloads retain 8-vCPU capacity through their semantic owners and selected files before packing. Timing or build-ownership changes cannot transfer that capacity to an unrelated row. Hosted stripes inherit their parent owner's capacity; whole named manual and release plans retain their existing routing.
  • Pull requests on the canonical repository reuse the changed-test resolver against the synthetic merged-tree diff. Precise core changes use bounded targeted jobs; plugin changes retain their owning configs and pack compatible envelopes into serial jobs. Each selected test file keeps its existing process isolation. The planner combines sibling tests with import-graph dependents and falls back to a metadata-complete compact plan for workspace package, package/lockfile, shared harness, split-config, renamed, or deleted changes, public extension-contract changes, tests with special shard setup, partially resolved or empty targets, oversized path or target plans, and planner errors. Nondist descriptors run as Node jobs; dist descriptors fold into the built-artifact boundary. The number of compact jobs follows the current measured weights. That PR fallback retains all sixteen tooling stripes, the isolated tooling shard, and the TUI PTY shard because scripts and PTY owners intentionally require their Go, dist, and environment metadata. Those timing-sensitive groups remain isolated in concurrency-one exclusive bins. Targeted plans always retain the full boundary gate because its repository scanners cannot be derived from imports.
  • When canonical pull requests fall back to compact planning, directly changed, existing tests owned by the release-only plugin shard remain as exact-file selections in the existing compact jobs. Canonical Vitest routing keeps unit-fast, contract, bundled, and E2E tests with their own suites; source files, directories, deleted tests, and live tests do not widen plugin coverage. The broad agentic-plugins sweep remains release-only, and push and manual CI plans are unchanged.
  • Canonical main pushes use a Blacksmith integration compact with nondist Node jobs plus the dist boundary descriptor. Former multi-config walls (CLI plus CLI-process, isolated plus fake-timers unit fast, and the logging/process/runtime-config trio) are split into per-config shards so no single group floors a lane. Real Node+TSX command tests belong to the isolated CLI-process catalog, so ordinary CLI tests do not prepare a runtime. The process catalog splits by complete file costs, with split sizing bounded below by its complete file costs and runtime prerequisite so an older aggregate timing cannot hide newly owned work. File packing also includes the prerequisite; runtime-consuming CLI children may share one preparation in the same serial job when their complete combined estimate fits the existing 150-second budget. Each child retains its selected files, isolated process and two-worker limit; deliberately separated fixed stripe families remain apart. The gateway process file stays alone because its cold proof already takes 200 seconds. CLI children retain the 150-second sizing target and their two-worker limit. Ordinary hybrid bins containing only non-build CLI children may combine up to 250 predicted seconds, with each original child still admitted separately below 150 seconds; the hosted runtime prerequisite itself has a 160-second floor. They omit the low-signal-per-push tooling and TUI PTY groups while retaining all product-runtime groups, including three file-weighted stripes apiece for unit-src, Control UI, and gateway-core. Blacksmith serial admission stays at 200 seconds for the large class and 276 seconds for the small class. Ordinary groups that can share two process slots admit 360 predicted aggregate seconds; a group already above its serial cap stays alone. Manual dispatches and Full Release Validation retain the full named per-shard matrix. No scheduled workflow currently runs that full Node suite; this is a known coverage-timing gap, not coverage supplied by this compact plan.
  • Doctor session and cron tests keep the two longest SQLite files in separate shards, with the remaining files together. Each shard runs complete files under the commands config's existing execution policy. The three owners cannot share a compact job; Blacksmith gives them 8-vCPU placement and its 200-second admission target. The two new owners charge the observed file body plus the full group overhead until the canonical timing refit has two successful main-run samples. Fixture sizes, assertions, and process isolation are unchanged.
  • Direct measurements for generated hosted stripes own their packing weights just like native groups. GitHub runs use hosted measurements and hybrid attempt-one runs use Blacksmith measurements; only unmeasured children divide the parent estimate. This keeps an expensive measured child from being packed beside extra work under an artificially small estimate while preserving its test partition, worker pins and per-child file execution policy.
  • Changed-plugin jobs run their independent config envelopes serially, including configs whose inner process chunks also run sequentially. The existing worker policy can then use the detected CPU budget without reserving capacity for overlapping plans; multi-target jobs retain their separate concurrency policy.
  • The sixteen tooling stripes use measured file weights to keep the Git owner, managed-process, worker-artifact, and transform-cache proofs in separate jobs. Go setup follows explicit selected files first. For whole-config tooling plans, only the ordinary tooling config owns the docs i18n Go tests; isolated and Docker catalogs do not. Unknown historical configs retain their original Go setup. The shard containing the unified declaration compiler fixture retains the existing larger runner; hosted splits carry that placement through each child's selected files before packing. Its synthetic compiler graph uses the existing 1,024 MB heap override, while production full builds keep their resource guard. Git owner cases each own an isolated checkout and process tree, so their real timeout and cleanup checks overlap at most two cases per runner. Worker transform proofs use a separate suite with its own fixture lifetime; late child cleanup cannot remove another suite's inputs. These changes preserve the test cases, deadlines, and process isolation; measured admission determines the job count within the existing hybrid row cap.
  • Blacksmith numbered tooling bins request the 32-vCPU class after packing while retaining their logical runner classes, names, file inventory, serial project/file execution and two-worker pins. In run 33689551111, the three slowest tooling bins received two CPUs and 8 GB of memory; their 314–342-second bodies set an eight-minute non-Windows wall. The larger request addresses that host-capacity mismatch without adding shards. Hosted and hybrid tooling placement is unchanged; the request needs native timing proof before claiming the eight-minute target.
  • Control UI browser projects greedily pack discovered test files using committed per-file timings, then cold-start basename hints, then source byte size. Bundled files default to at most two workers, bounded by the shared worker limit and local throttling; explicit Vitest CLI worker overrides remain available. Private source servers, real Gateways, and runtime-budget tests stay in a single-worker project; local whole-suite runs finish bundled work before starting that project. Per-file overhead is refitted from serial invocations; parallel invocations still update their file weights. Timing keys supply weights only: discovery still determines the complete test inventory, including new files and files without measurements.
  • Broad browser, QA, media, and miscellaneous plugin tests use their dedicated Vitest configs instead of the shared plugin catch-all. Include-pattern shards record timing entries using the CI shard name, so .artifacts/vitest-shard-timings.json can distinguish a whole config from a filtered shard.
  • The browser-extension Chromium bootstrap command prepares qaRuntime. It builds the native-host and relay JavaScript and runtime assets; the separate artifact job owns Control UI and plugin SDK declaration validation. The real Chromium flow and its assertions are unchanged.
  • The browser native-host launch test is a separate POSIX E2E case. Linux build-artifacts runs it explicitly after building or restoring dist, using OPENCLAW_E2E_USE_PREBUILT_DIST=1 so the test cannot start another build. Its JSON report must contain exactly the named passing assertion in the expected file, with one passed test and zero failures, pending tests, or todos; missing artifacts, skipped tests, and absent results fail. The workflow step skips only when a frozen historical checkout lacks the test file: that is unavailable historical proof, not coverage. Current checkouts with a missing file still fail. Changes to the case, its installation fixture, or its relay-key fixture select the artifact job even on test-only diffs; unrelated browser unit tests stay build-free. Manual CI uses the same artifact step, independently of the release-only plugin sweep.
  • Linux Node shard jobs persist Vitest's filesystem module cache through the upstream Actions cache API. On Blacksmith runners, official cache actions use Blacksmith's colocated cache backend instead of GitHub's, so cache entries are backend-local even when their keys match. Blacksmith CI shards are restore-only and unpack the protected Blacksmith seed into isolated runner-local roots. While the GitHub-hosted outage backend is active, every checks-node-* test shard, checks-ui, the ordinary sharded checks-ui-e2e job, both fast contract matrices, and the Vitest-running checks-fast-core tasks restore a separately published immutable transform seed from GitHub's backend. The composite action's single default-off restore-test-caches input keeps the expansion easy to disable without changing cache keys or writer policy; mixed fast-core rows enable it only for tasks that invoke Vitest. The real-Gateway UI job does not restore these test caches; native and Control UI i18n lanes do not invoke Vitest. The hybrid planner profile uses matching key contracts across two backend-local archives: attempt-1 Blacksmith rows read the Blacksmith seed, while hosted retries can read only a separately published GitHub seed. Ordinary CI jobs remain restore-only; the separate trusted warmer owns protected backend-local cache publication. The non-cancelling warmer follows main, runs daily, accepts manual or repository dispatch, and serializes per ref. Its Linux row follows OPENCLAW_CI_RUNNER_BACKEND and publishes only to the cache backend serving that selected runner. A hosted fallback therefore requires its own trusted hosted warmer seed; a successful Blacksmith publication does not populate GitHub's cache backend. Maintainers can select the existing hosted backend for an isolated trusted warmer without canceling a main writer. The Linux row launches each selected shard/config envelope through the normal runner in a fresh child process with concurrency one and --testNamePattern=(?!). Collection preserves the include patterns, environment, compiled imports and per-file cleanup while reusing the same serial cache leaf. Test bodies run in ordinary CI; imports reached only inside those bodies can remain cold until that run. The warmer finishes every selected envelope and saves the content-keyed transform and compile caches even when collection fails, then reports the failure after the cache saves; ordinary CI shard execution remains fail-fast. This prevents config-global state from leaking, avoids expanding filtered shards into whole configs, and retains transforms produced by the previous child. Setup computes the transform-input fingerprint once when transform caching is enabled and passes it to restore and generation validation; disabled caches do not scan the checkout. The fingerprint clears incompatible lockfile, package, tsconfig, and Vitest-config generations. Before publishing, the trusted warmer scans and prunes the transform cache to 75% after it exceeds 2 GiB, and the Node compile cache to 75% after it exceeds 1 GiB. Consumer jobs never prune the restored seed. Vitest hashes module id, source content, environment, and resolved transform config, so ordinary partial source changes keep unchanged entries warm while changed modules miss safely. Coarse restore prefixes bridge workflow runs; normal Actions cache LRU and inactivity eviction bound old immutable archives.
  • Trusted Blacksmith Linux Node jobs restore root node_modules, workspace importer trees (including plugin-local versions and links), and the workspace-local pnpm store from one immutable upstream Actions cache, which Blacksmith transparently serves from its colocated backend. Pnpm imports with hard links where the filesystem permits, and keeping the complete installed tree and store in one archive preserves those links. Pnpm's metadata cache lives beneath that same archived store root, so restored installs can verify supply-chain policy without depending on the producer's home directory. Source postinstall and build preparation leave pnpm-owned dependency trees intact. The key includes an explicit archive format, runner OS and architecture, the exact Node patch, and the semantic install-input fingerprint; there are no stale-prefix fallbacks. Manifests are canonicalized before hashing. The repository-owned openclaw metadata block and non-install scripts are excluded because pnpm and the audited direct root hooks do not read them, so runtime schema, publication metadata, formatting, and ordinary test/build script edits keep the dependency tree warm; unaudited lifecycle-hook drift fails closed until its source inputs join the fingerprint contract. Dependency, package-manager, hook-source, and lockfile changes always select a new immutable archive. Every exact restore runs frozen offline pnpm reconciliation, so an unchanged archive validates without registry access or importer relinking. If reconciliation fails, setup first clears every importer tree and rebuilds it offline from the restored store, then clears both modules and store and retries from the network rather than serving a partial tree. Setup then disables pnpm's redundant pre-run dependency check so install and frozen reconciliation remain the only dependency writers; shard commands must not launch concurrent implicit installs. The separate trusted warmer publishes the toolchain and exact dependency archives immediately after setup succeeds, before build and transform warming; preflight and downstream CI jobs are restore-only. Canonical pushes and same-repo pull requests opt into exact restores only on actual self-hosted runners, including hybrid attempt 1. An exact miss automatically falls back to the coarser pnpm store cache. Manual CI dispatches, fork pull requests, hosted lanes, and hosted retries use only that store cache. Cache restore/save failures are optimization misses rather than correctness failures, and normal branch scoping, LRU, and inactivity eviction bound obsolete archives. The former mutable dependency StickyDisk path was retired after repeated successful writers acknowledged commits that later runs still restored as empty filesystems.
  • Node shard and build-artifact jobs also restore Node's portable on-disk compile cache through immutable Actions caches. In GitHub-hosted outage mode, the hosted Vitest lane set above restores the separately published GitHub test-scope archive alongside its transform seed. Independent test and build namespaces keep their bytecode separate. The trusted warmer owns the protected test seed; ordinary build-artifacts and test jobs only restore caches. PR and ordinary test jobs only read protected snapshots, so feature-branch bytecode never enters the shared seed and PR traffic creates no cache archives. This reuses V8 bytecode for Node-loaded orchestration, build tooling, and external dependencies across different checkout paths, including when only part of the source graph changes. A maximum-size 2 GiB transform archive costs roughly 15–20 seconds to restore at about 125 MB/s; measured fast-contract transforms are roughly 21 seconds against an approximately 8-second restore, and broader cold imports reach roughly 100–143 seconds. The optimization should be reverted if measured savings fall below restore cost. Vitest child processes disable an inherited compile cache because coverage can be enabled inside dynamic configs and V8 coverage can lose source-position precision when scripts are deserialized from bytecode.
  • The existing Linux cache warmer publishes the native SDK declaration archive to the cache backend serving its selected runner before its full build. Hosted lint stripes and the dedicated package-boundary lane restore the matching archive from their runner's backend and validate native compiler inputs and complete outputs before reusing it. A Blacksmith warmer does not populate GitHub's cache backend, so hosted fallback requires a separately published hosted seed. After saving, the warmer removes its native SDK output so the subsequent packaged declaration cache describes the same tree as ordinary build consumers. Package-store contents and pnpm store-location bookkeeping are not compiler inputs; installed dependency bytes, resolution topology, explicit inputs, and compiler identity still invalidate stale declarations. The all-Blacksmith profile retains its read-only sticky-disk path. The Control UI and UI E2E jobs share a Linux Playwright Chromium archive keyed by the exact pinned Playwright version. The protected warmer also runs one standard hosted macos-15 row that installs dependencies and publishes only the pnpm store after an exact miss. It uses the existing OS, architecture, Node-version, package, and lockfile key; macOS CI remains restore-only. This row leaves exact dependency, build, transform, and compile caching disabled and runs no build or test warming. Pnpm owns pruning, while the before/after disk usage and saved archive size expose retained content; pruning does not impose a content-store size bound.
  • The build-artifact and Docker seed jobs restore the protected full-build cache through the shared Node setup action. Full, package, and ciArtifacts builds share scripts/write-plugin-sdk-entry-dts.ts: it stages the canonical public/private tsdown SDK declaration groups and caches each group independently. A hit restores into fresh staging; both cold and cached generations must contain every selected SDK entry and a complete relative declaration closure before publication to dist/. The SDK cache never adopts declarations from live dist/. Local plugin lint and package-boundary compilation use independent native declaration trees, not packaged declarations; see declaration ownership. The built Doctor plugin-index proof reuses that exact dist/ output instead of invoking the E2E harness's fallback TypeScript build a second time.
  • Full and package builds separately cache the AI, workspace-package, and remaining unified declarations. The unified runtime always rebuilds JavaScript before the shared declaration owner stages one base group and five plugin groups; the later SDK stage uses the same owner for its two groups. Both stages restore unchanged groups into private staging and compile only misses with the existing serial executor. They share one before/after input snapshot and validate every selected entry, successful compiler receipt, relative declaration edge, and shared-chunk owner before publication, including after package preparation clears dist. Conflicting shared bytes or changed consumed inputs fail before live declarations are written; cache records refresh only after successful publication. Canonical DTS configuration enables TypeScript stable type ordering so an unrelated literal allocation does not reorder an unchanged exported type in a rebuilt group. Full publication prunes obsolete declarations while preserving signed app bundles and Control UI assets; SDK publication owns only its flat entries and preserves other groups’ shared chunks. The AI and workspace-package steps also rebuild JavaScript on cache hits. Implicit and explicit declaration-enabled builds share those seeds; runtime-only profiles use an uncached graph and cannot publish declaration generations. Each declaration group hashes ordinary source bytes from its successful compiler Program, so edits to existing unconsumed tests, UI sources, and workflows retain its cache hit. The exact checkout-local .cache/vitest scratch root is excluded from resolution discovery; explicitly consumed inputs, installed aliases, and adjacent or nested paths still invalidate normally. It still validates inherited configuration, generator and package/plugin metadata, compiler identity, and resolution topology; consumed declaration dependencies and new resolution candidates invalidate the generation. Workspace-package changes conservatively invalidate the AI and package declaration caches. The protected warmer publishes build archives immediately after a successful full build, before unrelated test warming and pnpm maintenance. Every warmer attempt gets a new immutable archive key; coarse restore prefixes supply prior groups, and per-step signatures remain the sole content-validity owner. Rebuilt groups replace their complete owned cache trees, including obsolete bytes whose previous record is missing or invalid. GitHub's cache quota and inactivity eviction bound old generations; identical warmer attempts can publish separate archives. The weekly Node 22 lane instead publishes a 14-day artifact after successful main runs and restores only artifacts whose immutable producer identity resolves to that workflow on main, avoiding quota churn without allowing PR code to write a shared cache. Private-QA declarations are never persisted in Actions caches because cache namespaces are not confidentiality boundaries.
  • check-additional-boundaries runs the complete supplemental guard list (scripts/run-additional-boundary-checks.mts) with four concurrent child processes and per-check timings. Its 20 checks retain individual failures, deadlines and process cleanup. The shared four-rule focused scan runs once across all source roots; the narrower public lint commands remain available. Prompt snapshots run in their separate lane. Package-boundary compile/canary work stays together, and runtime topology architecture runs separately from the gateway watch coverage embedded in build-artifacts.
  • On the 32-vCPU self-hosted build runner, Gateway watch, channel tests, and the core support-boundary shard start together inside build-artifacts after dist/ and dist-runtime/ are already built. GitHub-hosted fallback runs keep Gateway watch serial so low-core contention cannot consume its readiness deadline. Full Node builds then verify Discord component attachment filenames through a serial public Gateway message action, checking the built revision and retaining the named-test JSON result; frozen targets that predate the case explicitly report unavailable proof. Both paths then run the two built TUI PTY artifact canaries alone.

The standalone UI suite runs three native Vitest shards through the same group executor and cache leaf as its bounded four-file seed in the trusted warmer. Each row retains the root Node worker limit of three; Chromium uses its project default. The shards preserve the complete four-project inventory and each project's isolation and cleanup policy. Every frozen target keeps its original singleton, unsharded test command. The window.open lint runs once in row one. The extra two jobs add two Blacksmith registrations per selected non-frozen run on Blacksmith routes, and none on hosted routes. They do not guarantee an eight-minute workflow: preflight, setup, queue time and other jobs still apply. Its native reporter records runtime CPU and memory facts, configured project workers, module diagnostics, and observed queue/end events. Browser pool logs record actual Chromium sessions. Project counts at run start describe discovery before sharding; completed module identities and final counts prove coverage. Event intervals are not scheduler-admission times; repeated environment/prepare durations must not be summed into wall time. These receipts do not establish transform-cache hits. Tooling stripes install Go only when their selected files include the docs translation test; historical whole-config plans retain their existing setup.

Shell-heavy macOS signing and elevation cases admit up to three cases per file, capped at Node's available parallelism. Independent checkout fixture tables retain their two-case limit. Each case owns its commands and temporary roots; cleanup joins its process tree and callback work before removing those inputs. Outer suites and the remaining checkout contract cases stay sequential.

Once admitted, canonical Linux CI permits up to 96 concurrent Node test jobs. The manifest separately enforces total-job budgets: 64 Node rows for canonical pushes and 120 for canonical PRs, including precise and plugin plans. GitHub also caps one job's combined outputs at 1 MiB measured in UTF-16, so preflight has 524,288 characters for every matrix together. Grouped Node rows list each striped test file explicitly. The manifest projects the five fields consumed by the shard runner, then uses gzip+base64 (groups_gzip_base64) when the target contains the codec. Historical targets without that capability receive the same projection through legacy groups JSON. Workflow tests keep the complete generated output under half of the cap. The smaller fast/check lanes remain capped at 12; Windows is capped at two and Android at two because those runner pools are narrower. Compact whole-config batches run with a 120-minute batch timeout, while include-pattern groups share the same bounded job budget.

Type-aware lint on CI runners with fewer than 8 CPUs or 24 GiB of RAM uses the existing Go compiler memory policy (GOGC=30, GOMEMLIMIT=3GiB) to reduce swap pressure. Explicit Go settings remain authoritative. The limit is soft and applies only to the lint child; declaration preparation retains its own policy.

Regular Android PR/main CI and PR release_gate dispatches use four rows: Play and Wear-shared unit tests/lint, third-party unit tests/lint, Wear unit tests/lint, and Kotlin lint for all four modules. Each phone flavor has its own source set and SensitiveFeatureConfig; apps/android/app/src/thirdParty/AndroidManifest.xml declares additional permissions and components. The Kotlin-lint row also compiles the benchmark when benchmark or Android build/dependency inputs change; missing or unusable changed-path data keeps that build.

Ordinary full-scope manual validation retains six Android rows: the three test rows, build-play, build-wear, and Kotlin lint. The build rows own lint without repeating it in test rows. build-play assembles both phone flavors and the Wear shared module and compiles the benchmark; build-wear assembles Wear. build-play-compat retains Play-only packaging for older frozen targets. GitHub-hosted build-play gets a 35-minute job budget for its three memory-bounded Gradle invocations; Blacksmith build-play and all other Android tasks retain 20 minutes. The budget follows the current attempt's runner route even when a retry reuses the original preflight matrix. Each current Gradle task has one protected sticky disk; PR jobs use disposable clones, while protected runs refresh content-addressed Gradle entries in place. Existing Android opt-in and npm qualification deferral are unchanged.

Robolectric resolves Android SDK artifacts outside Gradle's dependency cache, so every Android test-* task receives a workflow-owned Gradle init script that points test JVMs at a dedicated Maven-local repository. Actions cache restores are task-, platform-, and Android-contract-scoped; a prefix restore can seed a changed contract, but only a successful trusted run may publish the completed exact cache after a miss. Cold runs may download missing SDK artifacts, while warm runs reuse the exact archive. Build and lint tasks do not receive the Robolectric init script.

Remaining Blacksmith sticky-disk keys are deliberately bounded by supported task dimensions, never PR number, commit, run, branch, or dependency hash. Dependency, runtime transform, and compile caches use Actions cache instead because immutable archives expose verifiable restore/save results and avoid mutable snapshot-promotion failures. After a sticky key-version migration, add only the exact obsolete key, architecture, and region identities to .github/retired-sticky-disks.json, dispatch Sticky Disk Cleanup from main with the same dimensions and confirmation, verify deletion, then remove those entries. The workflow routes ARM identities to an ARM runner, rejects runner-region mismatches, uses Blacksmith's exact-key deletion action, and never deletes Docker builder caches or wildcard prefixes. Actions cache archives use normal LRU and inactivity eviction.

The check-dependencies shard runs Knip dependency, unused-file, and unused-export checks. Both guards enforce zero findings across production and full-tree scans, with no unused-file allowlist. The export guard also audits script entry exports. Production excludes test-support consumers; the full-tree and script scans include tests as consumers. Model intentional dynamic consumers in config/knip.config.ts, config/knip.all-exports.config.ts, or config/knip.scripts-exports.config.ts as appropriate. Each guard reports every scan outcome and fails if any scan fails. Historical targets run the export guard when they provide it and retain their older dead-code fallback otherwise.

Manual dispatches

Ordinary manual CI dispatches run the same job graph as normal CI but force every non-Android scoped lane on: Linux Node shards, bundled-plugin shards, plugin and channel contract shards, Node 22 compatibility, check-*, check-additional-*, built-artifact smoke checks, docs checks, Python skills, Windows, macOS, full iOS build/test and screenshot qualification, and Control UI/native app i18n. Their logical runner profile is always github, independent of the physical fallback selected by runs-on. Node 22 compatibility runs in Full Release Validation and manual dispatches only; push and pull request CI skip it. The exact-head release_gate fallback instead keeps the pull request's macOS, iOS smoke, and generated-native-locale scope without selecting iOS screenshots or native tests. Automatic source PRs and release gates verify native extraction inventory and Android/Apple localization safety without requiring translated or platform-generated output in the same PR. The serialized Native App Locale Refresh workflow rebuilds those artifacts in one isolated PR and enables exact-head auto-merge after required checks pass. Full native parity remains blocking for generated-artifact PRs, generated-scope release gates, ordinary manual CI, full-scope release validation, and release prep. Control UI locale parity remains advisory on automatic PR and main runs and blocking on manual/release CI. Standalone manual CI dispatches run Android only with include_android=true (the release_gate input also forces Android); full-scope release validation enables Android by passing include_android=true without setting release_gate; npm qualification scopes defer Android. Plugin prerelease static checks, the full agentic-plugins sweep, the full extension batch sweep, and plugin prerelease Docker lanes are excluded from CI. The Docker prerelease suite runs only when Full Release Validation dispatches the separate Plugin Prerelease workflow with the release-validation gate enabled.

PR baseline ratchets derive their comparison state from the checked-out synthetic merge tree and verify its head parent against the event head. The max-lines entry chains the environment-variable budget with the same fork-point ref before the assertion-safety check, so production source growth cannot first surface on main. Manual runs use a unique concurrency group so a release-candidate full suite is not cancelled by another push or PR run on the same ref. The optional target_ref input lets a trusted caller run that graph against a branch, tag, or full commit SHA while using the workflow file from the selected dispatch ref; ratchet baselines are compared with the target's merge base against the default-branch head resolved for that run. The release_gate input is an exact-SHA maintainer fallback for capacity-stalled PR CI: it requires target_ref to be a full commit SHA that matches the dispatched branch head and pull_request_number to identify the open PR whose merge tree is validated. Release-gate merge-tree lint uses the same five core stripes as hosted PR CI plus one extension stripe, so no single hosted runner owns the full type-aware lint workload.

bash
gh workflow run ci.yml --ref release/YYYY.M.PATCHgh workflow run ci.yml --ref main -f target_ref=<branch-or-sha> -f include_android=trueVALIDATION_SHA="<full-commit-sha>"gh workflow run full-release-validation.yml --ref main \  -f ref="$VALIDATION_SHA" \  -f expected_sha="$VALIDATION_SHA"

Gateway extended-stable runs npm preflight, Full Release Validation, and plugin npm release from extended-stable/YYYY.M.33; core publish consumes those three run IDs plus the validation attempt. release-ci/* evidence is invalid because publish binds every run to the canonical branch and release SHA. The tag publishes Gateway images and only the extended-stable* aliases; the path skips the regular orchestrator and its ClawHub, native-app, GitHub Release, website, and private dist-tag surfaces. See Monthly Gateway extended-stable publication for commands and recovery.

Windows Testbox Probe

The manual windows-testbox-probe.yml workflow keeps Windows/WSL probing and headless Windows CI on the selected runner_label. The run_windows_ci input (default false) requests both headless CI and a separate native Scheduled Task proof job on GitHub-hosted windows-2025. Neither job depends on the other, so their results remain independently visible; either requested proof failing fails the workflow.

For both proofs, set target_ref to an exact 40-character commit SHA. Both jobs check out that target, and native proof verifies checkout equality before running the lifecycle test. Native preflight runs before setup and requires an interactive Windows session. A noninteractive runner fails qualification rather than silently skipping proof. Selecting windows-2025 does not establish native qualification: the unchanged lifecycle assertions and cleanup must pass on the actual runner. Cleanup and diagnostic upload still run after failure, and retained evidence is removed only after cleanup and upload succeed.

Was this useful?
On this page

On this page