On this page
On this page
Testing and CI
CI runner classes
Runners
Testbox spending limits
The focused, routine, general, high-memory, ARM, build-artifact, and Windows Testbox
workflows share 32 concurrency slots. A GitHub-hosted admission job validates the lease ID, runner,
and runtime before the Blacksmith job becomes eligible. Dispatches wait in their
assigned slot without canceling an active lease. Each slot retains up to 100
pending requests using GitHub's queue: max; newer requests do not replace earlier
waiters. When a slot's queue is full, GitHub cancels additional requests. Ordering
follows when each job entered the slot queue, not workflow dispatch time. Hash
collisions can still leave other slots unused; this is a ceiling, not a promise
of 32 busy machines.
Admission expires 60 minutes after the workflow was created. This bounded queue
allowance lets waiting clients survive saturation beyond the former ten-minute
cutoff without admitting arbitrarily old abandoned requests. It is an operating
bound, not a measured queue percentile or proof that the client is still waiting:
the delegated warmup contract exposes no requester heartbeat to these workflows,
and SSH activity is only available after runner assignment. A request that waits
60 minutes or longer fails before checkout or hydration when its runner starts;
it can still incur runner startup cost. Abandoned requests younger than the bound
can still hydrate, so caller cleanup remains required. This does not remove the
queued job immediately or reset the clock when a runner is assigned.
Once the request passes that check, checkout and hydration do not recheck queue
age. They remain bounded by the job timeout and idle limit.
Stop an abandoned lease by its exact ID instead of leaving a warmup pending.
For a queued lease, also verify that its backing Actions run is terminal;
Blacksmith's completed lease status alone does not prove that the queued run
was canceled. Match the exact TESTBOX_ID in the admission job log before
canceling an abandoned run; its title or creation time is not enough. If that
task-owned run remains pending, use gh run cancel <run-id> --repo openclaw/openclaw
and confirm its terminal state. If the lease-to-run match is unavailable, report
the unresolved cleanup instead of canceling a guessed run.
Do not retry in a loop when the pool is full.
If the admission job cannot read its GitHub Actions run, it denies admission and
reports the HTTP status, a fixed error classification, and validated GitHub request
ID and rate-limit/retry headers when available. Error bodies are limited to 8 KiB
within the existing 15-second request deadline; raw response messages and bodies
are never logged. A bare 403 is reported as forbidden, not assumed to be a
permission or rate-limit failure. Use the request ID to investigate the original
GitHub response; a later successful request does not explain an earlier rejection.
Honor retry-after (seconds) before a later request. When
x-ratelimit-remaining=0, wait until x-ratelimit-reset (UTC epoch seconds).
Admission does not retry or change token permissions, and caller cleanup remains
required for a failed request.
Idle requests are capped at 15 minutes. The existing Testbox monitor continues to protect active SSH commands. Stop retained leases when their task finishes; the idle limit is not a substitute for caller cleanup.
| Testbox | Allowed runner | Default runtime | Maximum requested runtime |
|---|---|---|---|
| General runtime proof | 16-vCPU Ubuntu x64 | 60 minutes | 240 minutes |
| High-memory exception | 32-vCPU Ubuntu x64 | 240 minutes | 240 minutes |
| ARM proof | 16-vCPU Ubuntu ARM | 120 minutes | 120 minutes |
| Build artifacts | 16-vCPU Ubuntu x64 | 35 minutes | 35 minutes |
| Windows | 8- or 16-vCPU Windows | 75 minutes | 75 minutes |
Routine remote proof uses the 16-class. The explicit high-memory workflow can use only four of the shared 32 slots; it does not add four more slots. Normal leases can also occupy those slots, so hash collisions may queue high-memory work below its ceiling. Stop owned leases when done rather than requesting new IDs to evade a busy slot.
Use the 32-class only for a named command with measured memory need, a verified smaller-runner OOM, or a controlled comparison showing lower total billed cost. Record the evidence before allocation. The explicit full-suite Testbox PR gate retains this exception; ordinary remote commands and changed gates use 16-class. See remote proof for selection. A failed test, queue delay, or generic timeout does not justify promotion.
General proof defaults to 60 minutes including hydration. The existing
timeout_minutes input can request a shorter deadline or explicitly extend a
known long proof up to 240 minutes; larger values fail before allocation. Native Blacksmith warmup does not expose arbitrary
workflow inputs, and Crabbox --ttl does not enforce a Testbox lifetime.
The GitHub job timeout is the wall-clock limit.
The October 1 sizing audit favors 16-class over a blanket 8-class default. Across the prior seven days, 5,521 32-class runs had a 31-minute median lifetime and a 108-minute p95. The memory sampler returned 4,436 eligible runs; sampled median and p75 peaks were 6.2 and 12.8 GiB against the 8-class's observed 7.66 GiB. At those sampled peaks, total RAM minus available RAM was 6.8 and 13.4 GiB, so reclaimable cache does not explain away the smaller-class risk. The separate 6,199-run 16-class population also included 25 OOM-affected jobs; these populations are not a controlled size comparison.
Job duration includes hydration, commands, and idle retention. The sampled 31-minute job spent 30 minutes waiting in the Testbox monitor with a 30-minute idle setting. Do not treat lease percentiles as active-command percentiles or claim a measured timeout rate for the new policy. The 60-minute default is an operating choice; remeasure after the 15-minute idle cap is deployed before making 30 minutes or 8-class the general default.
These controls cover dispatches using the updated workflows in this repository.
The OpenClaw wrapper selects the workflow from main for Testbox run and
warmup, overriding configured refs and rejecting explicit historical refs.
Its source capsule still reconstructs the checkout being tested. Old wrappers,
direct historical-ref dispatches, other repositories, alternate workflows, and Windows check
workflows are outside the shared pool. Organization-wide concurrency, per-token
admission, SKU restrictions, and a hard spending stop require provider controls.
CI runner routing
Current extension package boundary checks use the 16-class. Frozen targets keep their historical 32-class runner and compiler policy.
Hosted Node 24 setup honors the workflow's existing NODE_VERSION pin when the
setup input is 24.x. This prevents runner-image refreshes from silently choosing
a different patch version. Explicit compatibility versions retain their own
selection, and self-hosted setup retains its existing range and toolchain-cache
policy. A restored cache key is not runtime proof: record the selected binary and
node -v when comparing runner backends.
Runner choice does not depend on the pull request author. Same-repository PRs use the configured backend. Fork first attempts, including first-time contributors, use the default Blacksmith routes when repository variables are unavailable; their Node planner uses Blacksmith timings and the same capacity promotion. Fork check jobs keep the logical GitHub stripe layout because forks cannot mount the trusted caches that size the all-in-one jobs. Runner placement is separate from that layout and cache authority. The maintainers accepted the added Blacksmith spend. Fork retries route to GitHub-hosted runners. The repository backend override cannot reroute fork first attempts when the variable is unavailable. Pushes and manual dispatches are unaffected. Cache trust is a separate, stricter boundary: exact dependency restores require a pull request from openclaw/openclaw, and ordinary CI never publishes the shared archives. The separate trusted warmer owns publication.
| Runner | Jobs |
|---|---|
ubuntu-24.04 |
The three ordinary upstream same-repository hybrid critical controls (preflight, selected pr-failure-report, and openclaw/ci-gate), hybrid preflight retries, check-docs in every mode (its ClawHub mirror clone is unauthenticated by design), security-fast outside hybrid first attempts, manual CI dispatch and other repository fallbacks, CodeQL security and quality scans, workflow-sanity, labeler, auto-response, the standalone Docs workflow, the whole Install Smoke workflow, all configurable CI jobs in github mode, and the remaining light lanes plus rerun Blacksmith lanes in hybrid mode. The GitHub/hybrid planner profile expands the Node matrix, QA Smoke to six parts, core oxlint across five stripes (two jobs on ordinary non-frozen hybrid push/PR runs), and current test-type checks across five core stripe jobs plus the central extensions/root/scripts tail. Extension/scripts lint plus optional UI and format checks stay in check-lint; frozen targets retain their historical three-job type layout. |
blacksmith-4vcpu-ubuntu-2404 |
Automatic upstream preflight first attempts with an unset, blacksmith, or runson backend, retained hybrid fork and qualification routes, hybrid first-attempt security-fast, native-i18n, checks-fast-core except QA Smoke CI, plugin/channel contract shards, most bundled/lower-weight Linux Node shards, check-* lanes except check-lint, selected check-additional-* shards, and skills-python |
blacksmith-8vcpu-ubuntu-2404 |
Current-main hybrid first-attempt four-part QA Smoke, retained heavy Linux Node suites and native compact rows previously requesting the 4-class, the checks-ui-e2e browser-extension row, boundary/extension-heavy check-additional-* shards except runtime topology architecture, and android |
blacksmith-16vcpu-ubuntu-2404 |
Main CI build artifacts, other automatic QA Smoke, Docker seed, Control UI E2E, current-target real-Gateway E2E, lint, dependencies, test types, core test-type stripes, extension package boundaries, and runtime topology architecture |
blacksmith-32vcpu-ubuntu-2404 |
Full CLI bins, memory-gated Node rows, two-child compact bins, Blacksmith tooling and agent-support bins, hybrid unified and SDK declaration compiler fixture bins, frozen-target real-Gateway E2E, separate npm release preflight and reusable release/E2E workflows, report-plugin-sdk-api-diff |
blacksmith-16vcpu-windows-2025 |
checks-windows |
xcode-27 |
All macOS Node, Swift, installer, companion, cache, cross-OS release, ordinary iOS build, and Periphery jobs use GitHub-hosted capacity. |
xcode-27-xlarge |
iOS screenshot shards, Store Release, its screenshot-only operation, and the reusable iOS Release E2E qualification job used by release and eligible full-manual CI. |
QA Smoke uses the 8-class only for upstream main pushes and schedules on a
first-attempt hybrid four-part plan with explicit current-target classification.
The complete 24-scenario profile and a cold private runtime build passed on two
usable CPUs under a 7-GiB memory cap without swap, skips, or OOM events; peak
memory was 5.96 GiB. Older targets, other profiles, manual and release runs, and
retries retain their existing placement. Scenario coverage, worker limits,
stagger, and deadlines are unchanged. Observe actual job duration and billing
after rollout; a lower runner rate alone does not establish the saving.
The table lists default placement. On eligible hybrid first attempts, the hosted budget can move security-fast and only the browser-extension E2E row to ubuntu-24.04; the default Blacksmith routes apply when optional admission is closed.
Control UI unit rows (checks-ui) stay on the existing Blacksmith 8-class for canonical hybrid/RunsOn first attempts, with the existing 20-minute timeout. Optional hosted admission does not move these shards. Explicit hosted routes, including retries and the github override, retain their 35-minute timeout. Coverage, workers, and per-test deadlines are unchanged.
Baseline ratchets and Node shards start independently after preflight; the final gate still requires the selected ratchets to pass. Standalone ratchets use the Blacksmith 4-class on same-repository hybrid first attempts, automatic main runs, and admitted qualification dispatches. check-plan uses the 16-class for shared compiler snapshot memory on upstream first attempts with an unset, blacksmith, or hybrid backend, including fork PRs. The GitHub override, retries, ordinary manual and frozen targets, RunsOn planners, and other repository contexts retain hosted planning. Fork core type stripes with an unset or blacksmith backend also use the 16-class on current automatic first attempts. Their logical GitHub profile, five-stripe coverage, and restore-only cache authority remain unchanged; hybrid health admission retains its existing placement rules. Compiler and lint rows still wait for their complete plan; eligible guards and dependency rows start after preflight. See admission measurements and cost.
Current upstream core lint stripes use the 16-class on automatic first attempts with an unset, blacksmith, or hybrid backend, including fork PRs. The same five fork rows still own their core and extension lint chunks; the remaining extension chunk stays in the combined check-lint row, which already uses the 16-class. Standalone check-lint-extensions-* rows are hybrid-only and retain their existing route. PR caches stay restore-only. Retries, ordinary dispatches, frozen targets, other repositories, and OPENCLAW_CI_RUNNER_BACKEND=github keep hosted core lint; existing RunsOn and qualification routes are unchanged.
RunsOn retains its hosted placement for standalone ratchets and check planning, including qualification dispatches.
Same-repository hybrid PR first attempts run check-plan on hosted
ubuntu-24.04 using the existing serial compiler discovery mode. The complete
snapshot remains too large for that runner. This trades compiler execution time
for avoiding a second Blacksmith assignment before type/lint rows can start;
the hosted planner counts in the existing admission budget. Forks, other
backends, retries, push/schedule and manual/release routing keep their existing
placement and discovery setting. Graph validation, selected rows and all check
commands are unchanged. See the measurement and qualification.
Healthy eligible main pushes and Windows-selected PRs can additionally offload seven check rows under the assignment guard. Main alone can then offload lint and central test types. Each decision consumes remaining capacity within the same 45-row limit; the original runner remains the fallback. Artifact builds retain the 16-class: their hosted maximum reached 898 seconds before preflight and gate overhead. See the routing measurements and qualification gaps.
Except for the temporary CodeQL exception below, macOS jobs use GitHub-hosted xcode-27, the generally available macOS 27 Apple Silicon image, except existing larger iOS jobs on xcode-27-xlarge. Native Swift builds/tests, ordinary CI iOS build phases and Periphery scans use Xcode 27. The native Swift lanes preserve hosted placement, job counts, worker caps, coverage, and deadlines; they add no Blacksmith registrations. The Swift source-language minimum remains 6.3. Native compatibility and complete job timings require proof on the new image.
iOS screenshot shards, Store Release, and its screenshot-only operation use xcode-27-xlarge on the same macOS 27 image as the reusable qualification job. On the 3-CPU, 7 GB xcode-27 runner, the iPad Pro 13-inch simulator exhausted memory: simctl and XCTest launches stalled for 60 to 100 seconds, failing iPad captures at launch or readiness. Screenshot capture uses stock simulators without SimSlim. It creates a fresh simulator for each device type, then shuts down and deletes it before starting the next; the screenshot-only job accepts a selected branch without release credentials. The qualification job uses the same larger runner for its Simulator and isolated Gateway. Stock qualification keeps the simulator unmodified. After the native build, each arm starts one isolated Gateway and prepares its setup handler through device.pair.setupStatus before booting a fresh simulator. The status call prunes expired completion records without issuing a credential; the setup code consumed by the app is issued after boot. The same Gateway serves the live pairing, chat, and relaunch case, then stops before the independent transcript-reader case on the same simulator. Both cases must pass within their individual deadlines.
The shared screenshot wrapper scans and mounts installed runtimes, ensures CI has the watchOS simulator runtime matching the selected Xcode SDK, downloading it only when unavailable, and logs the image, Xcode, and runtime inventory. Watch capture selects a supported device type from the latest available watchOS runtime, so an image without pre-created Watch devices still works. If CI device creation reports a missing Watch runtime path, the capture owner logs the disk-image inventory, deletes only that runtime's image, downloads the version matching the selected Xcode SDK, and verifies its mounted bundle path before retrying creation once. Missing or ambiguous image registrations, failed downloads, and failed retries stop capture with an error. The existing capture lifecycle still owns simulator creation and cleanup; the existing phone-only mode does not provision Watch runtimes.
The macOS critical-security CodeQL job temporarily stays on macos-26-intel with Xcode 26.6 because Swift build tracing is unsupported on macOS 27 and Xcode 27. This preserves Swift security coverage until upstream restores traced-build support. It continues to build the native app for ARM64. The native app keeps its macOS 26.2 deployment target, shared Swift packages keep macOS 15, and the Tauri companion keeps macOS 13.0 through its package configuration and explicit build environment.
The earlier hosted-routing validation used macos-26: repeated first attempts left the Blacksmith macOS jobs unassigned while other CI completed. In run 33616182173, the hosted retry assigned all three waiting Mac jobs within eight seconds; the Debug/simulator job passed in 15m31s. That historical result predates Xcode 27. Complete native evidence remains required for full manual qualification.
The Node test planner marks only shards that run the real native grep fixture.
Those Linux jobs install the ripgrep package when the selected runner image
does not provide it. Other Node shards do not pay that setup cost.
Current targets share one checkout/setup per fast contract family. The two weighted plugin selections still run as separate test:contracts:plugins processes; the two channel selections still run separate test:contracts:channels invocations, each retaining its four owning configs, four project slots and one worker per project. The envelopes run sequentially, and any nonzero exit stops the job before another envelope is admitted. Frozen targets keep their original matrix rows and execute one envelope per row. Runner routing, caches, worker budgets and aggregate-gate selection stay unchanged. In main run 33704083233, the separate plugin bodies totaled 94 seconds and the channel bodies 145 seconds; those sums support consolidation but are not measured combined durations.
The dependency warmer publishes the completed pnpm store immediately after setup,
before unrelated SDK, build, or transform work can fail. All cache-enabled Node
setups use the same workspace-local store path, including store-only readers;
Actions includes that path in cache compatibility. Pnpm's side-effects cache
carries native postinstall outputs such as Matrix crypto's binary and version
marker, so a compatible warm install skips the download. Cold caches and changed
native build inputs still require the upstream asset.
Linux setup uses hard links when that store is private to the job's workspace,
including store-only restores on hosted runners. This avoids copying cached
packages on filesystems without clone support. Shared or external stores and
ordinary macOS installs retain their configured import method; frozen-lockfile
reconciliation and lifecycle scripts still run.
Setup restores the configured store root before activating pnpm. The same
artifact contains the pinned pnpm wrapper and Linux native executable archives
under toolchain/, keyed by the complete packageManager pin. Bootstrap checks
their SHA-512 hashes in private staging before extraction; missing or invalid
archives fall back to the runner image, then the registry. Exact dependency
archives carry the same files, and dependency repair preserves them. Store v2
and exact-dependency v4 entries reseed once to include these bootstrap archives.
Cache publication still belongs to the existing warmer on each backend.
Node discovery scans only the toolcache's executable levels, avoiding bundled
npm dependency trees before selecting an already-installed runtime.
In hybrid mode, independent hosted dependency and code-warming jobs populate GitHub's cache backend. Blacksmith and GitHub-hosted cache archives remain separate. See cache ownership and seed selection for the full Linux and bounded hosted profiles.
Windows dependency-cache experiment
Normal Windows CI keeps dependency setup uncached. In the September 20 benchmark, median complete setup took 45.295s cold versus 52.738s with a restored store (+16.4%), despite reusing all 1,453 packages with zero downloads. Restoring the 763.8-MiB archive took 23.601/25.978/26.980s; the first sample spent 20.966s extracting it. Producer setup plus archive publication added 62.527s of job work.
All seven native jobs passed, but the workflow failed qualification: its reducer rejected a 24-KiB difference in reported total RAM. These measurements use the original receipts, with no assertion changes or reruns. The exactly RAM-matched subset had a 47.369s cold median versus 52.738s warm (+11.3%); this descriptive comparison does not replace the failed qualification. Revisit caching only when complete setup improves after accounting for restore, extraction, frozen reconciliation, and producer work.
Blacksmith runner capacity
Npm preflight retains blacksmith-32vcpu-ubuntu-2404. Main CI previously used
the same class for test types, core type stripes, and runtime-topology checks
to compensate for smaller delivered machines.
In the 2026-09-01 capacity check,
that label was the first measured class meeting the eight-CPU/24-GiB threshold
used by OpenClaw's parallel-check policy:
| Requested x64 Ubuntu 24.04 label | Observed CPUs | Observed RAM |
|---|---|---|
blacksmith-2vcpu-ubuntu-2404 |
2 | 7.66 GiB |
blacksmith-4vcpu-ubuntu-2404 |
2 | 7.66 GiB |
blacksmith-8vcpu-ubuntu-2404 |
2 | 7.66 GiB |
blacksmith-16vcpu-ubuntu-2404 |
4 | 15.42 GiB |
blacksmith-32vcpu-ubuntu-2404 |
8 | 30.95 GiB |
OS CPU count, affinity, Node, and CPU-time measurements agreed. Guest cgroup quotas were unlimited. The provider-side reason for the mismatch is unresolved; the table records observed capacity, not Blacksmith's advertised specifications or a guaranteed allocation. The check measured capacity, not whole-release speedup or billing equivalence.
Compact Node jobs retain the planner's 32-class request for two-child bins and Blacksmith tooling or agent-support owners. The 16-class delivered four CPUs and 15.42 GiB, forcing two-child plans to execute serially despite their unchanged aggregate packing budget. The 32-class is the measured allocation that meets the existing eight-CPU/24-GiB admission threshold. Packing, check names, timing identities, budgets and worker limits remain unchanged; actual capacity still gates overlap. This moves existing jobs between classes without adding runner registrations or hosted rows. Current-target real-Gateway E2E uses the 16-class; frozen or unclassified targets retain the 32-class, as described below. Native CI must establish the resulting execution and queue times.
Compact groups with a memory-gated worker allowance also request the 32-class,
including standalone serial bins. The isolated Gateway groups already request
eight workers with a 28-GiB memory floor and a two-worker fallback. On main run
35791016837, two 8-class rows received two CPUs and 7.66 GiB: their isolated
children took 413 and 369 seconds, and the complete jobs took 678 and 685 seconds.
The 32-class meets that existing allowance. Promotion happens after packing;
file membership, child ordering, memory checks and fallback workers remain owned
by the same planner and executor. Hosted routing still uses the workflow's trust
and retry rules. This adds no jobs or runner registrations.
Native compact rows that would request the 4-class now request the 8-class after packing. Both delivered two CPUs and 7.66 GiB in the capacity check, while the five-run September 21 sample showed a 125-second median assignment wait on the 4-class. Logical packing classes, child processes, worker limits, and hosted fallbacks stay unchanged. This avoids that queue at a higher per-minute rate; the combined packing and hosted-check changes must establish the net cost saving.
Current-target build-artifacts uses the existing 16-class. A controlled Testbox proof at 3ccc3710bd6 completed all eight job compute steps in 229.3 seconds (252.8 seconds including payload setup) on four CPUs and 15.42 GiB RAM, with a 12.59 GiB cgroup peak and no recorded OOM events. The proof retained the complete parallel verifier wave and passed final source, worker-generation cleanup, and memory-event checks. SDK declaration generation now uses one compiler program, with the existing child heap budget and native headroom. Frozen targets and missing target classifications now request the same 16-class; hosted fallbacks, job counts, concurrency, and deadlines are unchanged. The recorded measurements establish compute fit for that tested current target, not historical targets or a guaranteed full Actions duration.
Existing recommendation-based promotions from the 8-class remain for checks-node-compact-large-5 and checks-node-compact-large-9. Extension bundles follow the planner's runner metadata: the former bundle-16/bundle-25 overrides would attach old recommendations to different work after compaction. The former 32-to-16 overrides for checks-node-compact-small-3, checks-node-compact-small-4, and checks-node-compact-small-10 are removed so these rows retain their planner-owned parallel or tooling capacity. Numbered bins can contain different work across profiles and revisions; the retained compact recommendations use a 24-hour window and still need ownership-based replacement when those bins change. A later recommendation identified CPU saturation for build-artifacts on the 16-class. Current complete-job duration and memory headroom still need measurement; the earlier controlled proof retains its original source scope.
Current-target checks-ui-e2e-real-gateway rows use the 16-class. Frozen or
unclassified targets retain the 32-class. The
planner balances serial fixtures and audited parallel standalone files in the
first row; the second owns the remaining parallel files. The first row also owns
the desktop transport proof when selected by full manual/release validation or
a direct desktop-spec edit. Both rows retain their existing worker limits, build-before-test
ordering, hosted fallbacks, and test deadlines. Runtime-only preparation leaves
SDK declaration generation and validation with build-artifacts. The split adds
one job and possible registration when the lane is selected, while ordinary PRs
continue to omit it. Backend routing is unchanged. Exact-head
CI measures complete row walls, including setup and transport proof.
An earlier 16-class attempt exceeded the 20-minute Blacksmith budget. In run 35120538555, the 16-class delivered four CPUs and was canceled after 1,227 seconds while real-Gateway files continued passing; the last file passed 13 seconds before cancellation. Setup, artifact build and desktop proof consumed 6m31s before the browser suite. Capacity changes require complete-job qualification against the current preparation and inventory; the hosted routes and all test deadlines stay unchanged.
Blacksmith real-Gateway shards have a 30-minute job budget; hosted fallbacks retain 40 minutes. In run 38590548763, shard 1 exceeded the 20-minute job limit after 16 of its 17 files completed. Setup, build, and desktop transport proof consumed 6m44s before Vitest; the last passing file completed six seconds before cancellation. The remaining API-key file had no completed result, so this does not establish an individual test hang. The scoped allowance retains every test and subprocess deadline and needs a natural complete-job run to establish sufficiency.
In run 38546577399, the native second shard spent 3m13s preparing, including a 2m04s build, then 16m48.92s in Vitest. All 21 files and 48 tests passed, but those phases alone exceeded the former 20-minute budget without counting wrapper startup or cleanup. The extra ten minutes leave completion headroom for cleanup and artifact uploads.
The hosted budget covers setup as well as tests. In run 34707873095, attempt 2, 7m42s elapsed before the browser suite began; the 20-minute job limit then canceled a progressing suite before its widget cases. The hosted budget covers the complete setup, private artifact build, and test workload. Individual test and subprocess deadlines, test inventory, workers, routing, and concurrency limits apply unchanged on both routes. This adds no jobs or runner registrations and makes no claim that execution is faster.
The full CLI compact bin requests the 32-class after packing and uses measured
workers with fallbackMaxWorkers: 2. Serial self-hosted execution can admit
eight workers on the observed eight-CPU/30.95-GiB allocation. Hosted, frozen,
constrained, and overlapping execution retain the fallback. Process-only CLI
bins, serial companion execution, packing, and job counts keep their existing
policies. See Vitest worker sizing.
The earlier 16-class placement had a controlled 2026-09-10 Linux Testbox comparison at the same source, memory, starting caches, and case inventory: 12m20s–14m15s with two available CPUs versus 9m17s with four, both at two Vitest workers. All 307 files and 7,977 cases were retained. Those measurements do not describe the 32-class policy or include CI setup, queueing, or companion groups. The later CLI timing weight of 595 seconds and complete CLI config remain unchanged by the measured-worker cutover.
Backend routing still applies. Hybrid retries and fork retries retain their
hosted routes. Ordinary manual CI dispatches remain hosted in hybrid mode;
Full Release Validation's existing frozen-target lint exception remains separate.
Npm preflight uses the larger Blacksmith request by default and retains its
explicit use_github_hosted_runners option.
Ordinary iOS smoke CI builds the app and embedded Watch targets for the runner's architecture, using the same iPhone simulator for compilation and voice-cleanup tests. It omits compiler indexes, which CI does not consume, and finishes simulator preparation before XCTest launch. Smoke retains complete test results and logs but disables verbose system-diagnostic collection: Xcode 27 can spend 600 seconds collecting it after passing tests. Full manual validation retains universal simulator compilation, verbose diagnostics, the Release device build, and lifecycle/UI/Watch tests. Frozen targets keep their original build settings.
Current main-tier iOS tests also build the app and both test bundles once, then execute the existing voice, lifecycle, and two-process Access restart proof without rebuilding. They restore Xcode build products and SDK modules by runner architecture, Xcode and dependency graph, with incremental fallback across source changes. The shared Swift cache owner restores timestamps only for byte-identical current inputs; generated version metadata stays fresh. Successful scheduled main runs alone publish these archives, with a four-minute bounded optional save tail. Signing, test selectors, serial simulator execution, and full manual/frozen builds are unchanged. The standalone restart helper still builds its own products. The October 15 baseline took 35m51s, including 13m36s in the cold Xcode app build; the first natural run populates the new cache, and subsequent natural hits must establish the complete-job saving, including archive transfer.
Scheduled main CI uses the same automatic runner and dependency-cache policy as
main pushes. A runson configuration keeps the existing hybrid baseline for
scheduled work; only qualified paths opt into RunsOn. Manual dispatches stay
hosted under their existing policy, even when their dispatch ID resembles the
retired hourly marker. Native runner labels and worker limits are unchanged.
Runner backend modes
The macos-swift lane builds Swift tests once and runs each test once per job. The ordinary suite retains default-profile behavior; rendered Quick Chat tests follow in a fresh default-profile process, then AppState isolation tests run in a named-profile process through the same resource-owning launcher. Historical targets retain their original two partitions. Each launch owns a private home and disposable, unlocked default Keychain until the test process group and output pipes close. HOME and profile markers do not isolate macOS services; all partitions run only on the disposable credentialless macOS worker. Current launcher-capable targets bound Swift Testing parallelism to the runner's logical CPU count, capped at 12, for automatic runs, manual dispatches, and rerun attempts. Only frozen targets that predate the resource owner use the serial fallback. A failing test fails the job without an in-job retry. See native test safety.
The repository variable OPENCLAW_CI_RUNNER_BACKEND controls the runner backend for ci.yml:
| Value | Light lanes | Heavy lanes | Rerun behavior |
|---|---|---|---|
unset or blacksmith |
Blacksmith-first, with the existing manual-dispatch and fork fallbacks | Blacksmith-first, with the existing manual-dispatch and fork fallbacks | Existing behavior is unchanged |
github |
GitHub-hosted | GitHub-hosted | Every configurable job remains hosted |
hybrid |
Ordinary upstream same-repository critical controls and light lanes use hosted runners | Blacksmith on attempt 1; GitHub-hosted on github.run_attempt > 1 |
Rerunning a failed or stuck Blacksmith job automatically moves it to hosted capacity |
runson |
Hybrid baseline | Hybrid baseline, with pure cron child rows on RunsOn for eligible first attempts | Automatic Spot-interruption retries disabled; other reruns retain the hybrid fallback |
Configurable heavy lanes are build-artifacts and android. The macOS Swift and iOS build jobs use GitHub-hosted xcode-27 with Xcode 27; screenshot shards use xcode-27-xlarge. The focused macos-node lane uses GitHub-hosted xcode-27 in every backend mode, with the same test inventory and two-worker limit. Ordinary push and scheduled runs in openclaw/openclaw, and its same-repository PRs, use hosted ubuntu-24.04 for preflight, the selected pr-failure-report, and openclaw/ci-gate when OPENCLAW_CI_RUNNER_BACKEND=hybrid. Fork first attempts retain their existing Blacksmith preflight and gate; their reporter is already hosted. Runner groups remain honored. Outside the macOS migration, all workflow_dispatch qualification/release overrides, retries, and backend modes keep their existing routes. The logical planner profile, cache trust, steps and deadlines are unchanged. These controls remove three Blacksmith assignments per eligible PR and two per eligible main run, with no new jobs. They count in the existing hosted budget; this is mandatory routing, independent of optional offload admission. Hosted assignment can still delay completion; see measurements and estimated cost. security-fast uses Blacksmith only on eligible hybrid first attempts when the hosted budget cannot admit optional work, and stays hosted outside hybrid. It waits for preflight to count the selected hosted rows, and still executes after a preflight failure unless the workflow is canceled. Security hooks use pinned installed packages and local hook definitions, so they no longer initialize remote Git repositories. For ordinary same-repository hybrid runs, security-fast retains a Blacksmith control assignment only when optional hosted admission is closed. Existing conservative registration reservations remain unchanged; this routing change does not assert free organization-wide capacity. Hybrid sends the compact Node matrix, up to 80 compact rows plus separately appended plugin fallback rows, thirteen-row checks-ui-e2e matrix for targets with the named-project contract, the checks-ui-e2e-real-gateway lane that shares its serial Chromium workload, four-row QA Smoke matrix on upstream automatic runs (six rows for manual dispatches), the two-part Windows matrix, checks-ui, check-lint, check-test-types, the five check-test-types-core-* rows, check-dependencies, check-additional-extension-package-boundary, check-additional-runtime-topology-architecture, and report-plugin-sdk-api-diff to Blacksmith on attempt 1. Eligible two-child ordinary compact rows request blacksmith-32vcpu-ubuntu-2404; bins containing the full agentic-cli group request blacksmith-32vcpu-ubuntu-2404 after planning. Other compact-small rows retain blacksmith-4vcpu-ubuntu-2404, compact-large rows retain blacksmith-8vcpu-ubuntu-2404, and the planner's measured small queue-tail promotions retain their 8-vCPU labels. Within that set, only the browser-extension E2E row may move to hosted Ubuntu; together with security-fast, preflight admits at most two optional rows below the 45-row hosted limit. Every other configurable ci.yml lane stays hosted in hybrid, including the remaining lint/check rows, docs, and Python skills. Separate Opengrep workflows remain GitHub-hosted.
The dispatch-only Plugin SDK API report requests the 32-class. Recent 8-class jobs exposed two CPUs and about 8 GiB of memory, below the parallel-render floor. Hosted report attempts also ran serially and exceeded the 20-minute job budget. The measured 32-class exposed eight CPUs and about 32 GiB, satisfying the existing eight-CPU/24-GiB floor for two concurrent 6-GiB render heaps. The resource guard, deadline, hosted override, and other rows remain unchanged. Main-shaped qualification dispatches retain this exact-head SDK report alongside push coverage and use the admitted runner backend. They use the full validation tier; the main validation tier is reserved for ordinary same-revision manual or scheduled validation.
RunsOn qualification
The opt-in runson profile derives its plan from hybrid. It extracts the three
core-runtime-cron-parallel-* children into one serial job on c8i.8xlarge
with 32 vCPUs, 64 GiB RAM, ubuntu24-full-x64, and an 80 GB gp3 root. The
current 258-file cron inventory retains its two-worker job and group ceilings.
The three source Blacksmith jobs retain their other children. Hybrid owns the
shared serial-tail splits and measured nine-to-four tooling packing, preserving
child contracts, workers and deadlines. Broad-PR Node /compact counts are
98 /60 on hybrid and 99 /61 on RunsOn, within unchanged caps. No NVMe,
sticky disk, warm pool, or test-inventory change is enabled.
The exact-head comparison passed cron on Spot, Blacksmith, and GitHub in
396, 432, and 672 seconds, respectively. Both native workflows still failed
and exceeded fifteen minutes; see the
measured routing costs and remaining qualification gaps.
The repository backend value runson admits this route only on the first
attempt of an upstream same-repository PR. The repository variable
remains unchanged during qualification. A maintainer can instead dispatch
ci.yml with runner_backend=runson, release_gate=true,
pull_request_number, and target_ref set to the full current PR head SHA.
The workflow branch must be that PR's current branch and head, and the
existing maintainer admission must pass. The qualification adds two identical
cron controls, on Blacksmith and GitHub, with the same pinned Node version and
two-worker ceiling. They count against the final Node cap (101/130 for this
inventory) and are absent from normal PR plans. A qualification must select cron
tests; otherwise preflight fails before allocating comparison runners. Qualification uses the profile’s
normal downstream placement, read-only cache admission, and lint partitions.
Its preflight remains hosted until authorization succeeds and is counted in
the hosted budget. Release-only lint and minimum-Node compatibility jobs stay
with ordinary manual validation. Ordinary manual dispatches and
untrusted or unrelated targets cannot use this override.
For a main-shaped measurement, set ci_shape=main with the same exact-head
PR admission and choose runner_backend=hybrid or runson. This uses push
coverage, the 70-row Node cap, Node runtime, and ordinary main proof/native
selection. It omits PR extension fallback and the two cron comparison controls;
it does not add full-manual release-only work. Raw GitHub event/ref still own
trust, concurrency, cache publication and provenance. The initial preflight is
hosted and included in the measured wall. Ordinary dispatches retain
ci_shape=default and their existing behavior. Explicit hybrid qualification
with the default shape uses PR coverage.
RunsOn uses one opaque label per row, with a unique run/row identifier:
The existing GitHub App handles this label route. It does not require a new AWS
login from the operator, though the expired operator SSO session prevents current
administrative, selected-AZ price, and teardown verification. The public AWS feed
supplies a regional Spot reference without authentication: $0.6586/hour for this
type in us-east-1, fetched September 22, 2026, at 06:02:13 UTC. See the
price source, timestamp, and measured allocation estimate.
No interactive login is part of qualification.
spot=true retains the provider's automatic on-demand fallback when Spot
capacity is unavailable. retry=false explicitly disables automatic
interruption reruns: native retry=when-interrupted can rerun failed jobs and
their dependents twice, after the entire failed workflow attempt completes.
That recovery has not been measured inside the original 900-second wall, so
an interrupted qualification can fail. Manual reruns retain the hybrid hosted
fallback and do not establish Spot-recovery performance. See the
provider's retry contract and
routing costs and qualification gaps.
This opt-in route makes no interruption-safe fifteen-minute claim.
Hybrid hosted assignment guard
Automatic upstream hybrid first attempts inspect recent hosted assignment before admitting additional checks. Main pushes qualify directly; PRs must select Windows coverage and a full compact Node plan containing at least 500 predicted seconds in one serial row. The admitted hosted checks completed within 549 seconds including setup in the native PR trial, with at least 289 seconds of slack before the final Node job. The serial prediction is an admission floor; actual setup, assignment delay, and workflow completion still require measurement. Precise plans, shorter serial rows, and aggregate two-slot estimates do not qualify. The five Windows shards alone no longer establish the old two-shard latency floor. Fast-only scopes, frozen targets, manual dispatches, retries, and other repositories retain their existing routing.
scripts/lib/ci-hybrid-hosted-health.mts uses preflight's read-only Actions permission. One ten-second deadline bounds at most four requests: the newest ten main push runs created within the past day, then the first hundred attempt-bound jobs from up to three fresh eligible runs. The server-side date filter prevents an unrestricted history listing from omitting recent matches. It samples only known hosted jobs whose sole dependency is preflight. Wait begins at the later of job creation and successful preflight completion, excluding dependency delay. Evidence must be within thirty minutes; any assignment wait of at least sixty seconds, missing assigned samples, malformed response, or API failure keeps the new offloads on Blacksmith. The step reports the reason, sample count, and maximum wait without exposing request diagnostics.
Dependencies, core type stripes, and runtime topology can use optional hosted capacity within the eligible PR workload's slack. Extension package boundaries retain Blacksmith because hosted cold-archive runs exceeded 22 minutes; their compiled receipts and dependency links are not yet portable across checkout roots. Admission retains the previous boundary-row reservation so this change does not expand other offloads. Main additionally admits lint and central types. In the September 22 five-run sample, the largest independent hosted check took 664 seconds; artifact builds reached 898 seconds and therefore retain Blacksmith. These observations replace the earlier projections made against a thirty-minute main objective. The current qualification target is a complete run within fifteen minutes, including preflight, assignment, setup, and the gate; routing estimates alone do not establish it. Ordinary same-repository hybrid preflight uses hosted Ubuntu; fork and dispatch overrides retain their existing routes. The admission guard now rejects waits at sixty seconds to match that target; API and job deadlines remain unchanged.
This is an admission decision for future jobs. A hosted stall beginning after the snapshot can still delay admitted work, and existing hosted jobs remain exposed. The check itself can add up to ten seconds to preflight. It never changes repository variables, migrates an already queued job, or retries failed work.
Hybrid is the normal degraded-capacity mode. If Blacksmith is down: rerun the failed or stuck heavy job; it lands on hosted automatically. During a full Blacksmith outage, record whether OPENCLAW_CI_RUNNER_BACKEND is set and its current value, then switch to the github fallback:
The github override also routes Full Release Validation orchestration, npm qualification, live QA, performance, package Telegram, OpenWebUI, and release runtime-pair jobs to GitHub-hosted Ubuntu. Existing explicit hosted-runner inputs remain supported. Runner placement changes; coverage, artifact identities, approvals, worker limits, and timeouts do not. Already-running jobs are not moved. Keep OpenWebUI disk requirements and performance baseline hardware differences in mind when interpreting hosted results.
Hosted ci.yml paths use the same setup exercised by manual dispatches and fork pull requests. Fork PRs use the logical github check profile even when repository variables are unavailable, so broad core lint and test-type workloads retain hosted stripes instead of falling back to oversized all-in-one jobs that need trusted caches. Fork first attempts use Blacksmith Node labels when the backend variable is unavailable, so node_runner_backend uses the Blacksmith default and they get its measured timings and capacity promotion; fork retries plan Node shards with github. Failed-job-only fork retries keep their first-attempt matrix on hosted runners, where the shard executor clamps overlapping children to one. Frozen targets opt into this event-aware profile through hosted-runner-profile-contract-v1; targets without the marker retain their historical workload shape. Blacksmith-only Docker and sticky-disk steps are skipped, dependency setup uses the ordinary Actions pnpm-store cache, and low-memory Android builds use separate Gradle processes. Hybrid attempt-1 Blacksmith Node and plateau lanes restore the exact workspace dependency archive from the trusted warmer. Eligibility uses the actual runner environment, so hosted lanes and retries stay on the ordinary store cache. The exact key includes the resolved Node patch, OS, architecture, and semantic dependency inputs; a different runner image safely misses and follows the existing store-install path. The Node toolchain itself is also cached through that API: Blacksmith's image tracks an older runner-images snapshot whose toolcache Node patches (measured 2026-08-16: 20.20.0, 22.22.0, 24.13.0) sit just under this repo's engines floor, so every job otherwise re-downloads Node from nodejs.org. Restores are prefix-keyed and saves carry the resolved patch, because an exact-key hit suppresses the post-job save and would pin the first payload forever once the floor advanced past it. A restored payload below the floor is rejected, pruned, and replaced. Vitest transform and Node compile caches still use the upstream Actions cache API; their Linux-only runner.os != 'Windows' conditions do not exclude Blacksmith labels, and the trusted warmer alone publishes each backend-local protected seed. The warmer's selected runner route determines whether that publication reaches Blacksmith's cache or GitHub's cache. Core oxlint keeps five fixed hosted stripes with one lint thread per process. The large agents, Gateway, infrastructure, and UI targets run in separate processes within their assigned stripe so native semantic caches are released before neighboring targets run. This isolation applies to explicit core-stripe invocations. Automatic full lint, including the published Git updater preflight, retains its original five aggregated core Programs even when CI environment variables are inherited. Ordinary non-frozen hybrid push/PR runs group stripes 1+2 and 3+4+5 sequentially across two jobs. Eligible first attempts and admitted qualification dispatches use the Blacksmith 16-class for both packed rows; hosted assignment placed them on the measured critical path. The first row took 621 seconds on the 8-class, including 568 seconds of lint; the second later exceeded its 15-minute job limit while running stripes 3, 4, and 5. Both retain four actual CPUs and the existing memory policy for larger bounded checks. Other admitted unpaired stripes use the 8-class. The aggregate gate follows the critical-control routing above: ordinary same-repository hybrid runs use hosted Ubuntu, while existing fork, dispatch, qualification, release and other backend overrides remain. The available GitHub override, retries, ordinary manual dispatches, and other repositories retain hosted routing. A failing stripe stops its row. The github profile, frozen targets, manual dispatches, and release gates retain five jobs; GitHub plugin stripes keep their existing owners. Plugin lint ownership is described below; script lint and optional UI and format checks stay in the existing check-lint row. Extension type-aware lint discovers extensions/tsconfig.json for plugin tests and helpers, retaining imported dependencies and shared ambient declarations without adding unrelated core/UI/package source roots. Plugin production files keep their existing package-boundary projects. Eligible core-source and test changes in pull requests reuse the local changed-check selector. On current targets with stripe support on github or hybrid, each of the five check-test-types-core-* rows validates the complete core graph boundary, then compiles the changed tests' consuming graphs that belong to its assigned stripe. Stripe membership is assigned before narrowing, so empty intersections stay empty and the union preserves every selected consumer. Ambiguous compiler ownership or a removed test selects every registered core test graph across those same five stripes. The additional-boundary lane transfers its core graph check to these required type rows, and check-test-types owns only the extensions/root/scripts tail. Full runs use the same five stripes without consumer narrowing. Frozen targets retain two paired rows for stripes 1+2 and 3+4, with stripe 5 in the central row. Targets without stripe support and the all-Blacksmith profile keep the full central path. Each core type row preserves at most two concurrent compiler children and one builder per child. Core checks retain the standalone resource policy; the remaining type commands retain their existing environment. A failed boundary or compiler stops its row before another command starts.
The agents-sessions graph owns root src/agents/session-*.test.ts(x) tests and the complete src/agents/sessions/, src/agents/session-maintenance/, src/agents/main-session-recovery/, src/agents/subagents/, and src/config/sessions/ test subtrees, keeping session runtime, recovery, subagent, and storage tests together. The agents-tools graph also owns shell/tool tests, nested sandbox tests, managed-worktree tests, and the complete security and secrets test subtrees. The agents-other graph owns the remaining nested agents test subtrees and every root-level model catalog, selection, fallback, and auth test (src/agents/model-*.test.ts(x)), plus a few listed root-level prepared-runtime and workspace tests; agents-root keeps the remaining root-level agents tests. The config-cli graph owns configuration, hook, and CLI tests except the separately owned session, daemon, cron, program, and update subtrees. The cli-update graph owns src/cli/update-cli/ and root src/cli/update*.test.ts(x) tests; commands owns src/cli/program/. Root-level Gateway session tests belong to gateway-root; gateway-server owns root-level server*.test.ts(x) tests and the complete src/gateway/server/ test subtree. The gateway-methods graph owns src/gateway/server-methods/; gateway-other owns src/gateway/worker-environments/, the node-side src/node-host/ and src/worker/ test subtrees, and the remaining nested Gateway tests. The infra graph owns infrastructure and media tests; state-logging owns audit, state, logging, and shared tests plus the root-level src/infra/sqlite-* and src/infra/state-migrations* tests. The plugin-sdk graph owns the complete src/plugin-sdk/ and src/channels/ test subtrees; messaging owns auto-reply and outbound infrastructure tests. The services-cron graph owns cron CLI (the nested src/cli/cron-cli/ tests and root src/cli/cron*.test.ts(x) tests) and cron tests; services owns daemon CLI, daemon, process, system-agent, skills, and infrastructure update tests. New graph splits are appended to the shared registry so existing stripe assignments stay stable. The core-test boundary guard requires every test root exactly once across the registered graphs and reports graphs above 720 roots; the root budget fails local checks and is a warning in GitHub Actions. The inventory regression test warns at the same limit so a rebalance gets scheduled. The graphs keep their existing stripe assignments and compiler concurrency; moved tests follow their new graph's stripe. Changed-test selection still checks every consuming graph.
The ui-e2e-chat graph owns root-level ui/src/e2e/chat*.test.ts(x) tests. The ui-e2e graph retains the other UI E2E tests and ui/src/test-helpers/control-ui-e2e*.test.ts(x) harness tests. Both retain the UI bootstrap and Solid JSX compiler settings. The new graph is appended to the canonical registry, preserving existing stripe assignments and the five CI type-check rows.
Android admits all four rows together on upstream Blacksmith push and PR first attempts, including fork PRs. Hosted routes, retries, manual dispatches, and schedules retain two. Android Play and ThirdParty rows run their unit tests. In the normal four-row tier, Wear also owns third-party app lint, while the Kotlin-lint row owns Play app and Wear-shared lint. Every row keeps separate, sequential Gradle processes and shares one UTC build timestamp through the existing openclawBuildTimestamp property when it has multiple native invocations, avoiding BuildConfig regeneration solely because a later command starts. Phone test classes on Blacksmith run in at most four isolated JVMs, bounded by available processors, with the existing 1 GiB heap per JVM. Each JVM receives its share of the available processors so its GC and internal pools do not multiply the runner CPU budget. Classes remain sequential within each JVM; hosted tests keep one JVM. Full manual and historical compatibility task inventories are unchanged.
Task-scoped Blacksmith sticky disks retain the Gradle user home across dependency changes. Gradle owns content-addressed invalidation and periodic expiry of unused caches and wrapper distributions; CI does not erase the warm home when a dependency fingerprint changes. Protected pushes remain the only snapshot writers, and pull requests consume read-only clones. Dependency resolution remains online so a changed dependency or cold snapshot can populate missing artifacts.
On hosts with less than 24 GiB RAM, serial plugin lint runs use eight-directory chunks by default. Automatic Linux CI uses sixteen-directory chunks with at least four CPUs and 15 GiB of verified physical and cgroup memory capacity, including ancestor limits. Unknown or smaller capacity, local runs, Windows, explicit plugin stripes, and explicit serial selections retain eight-directory chunks. This amortizes repeated type-graph startup while covering every plugin and root source file. Outside Windows, explicit full-speed or parallel overrides keep the previous unsplit workload. The Windows chunk-size override remains Windows-only. Lint prepares only the SDK declaration tree; the separate package TypeScript boundary check still prepares the SDK and plugin declarations.
For current targets using the github profile, plugin lint chunks follow a fixed split across six existing jobs. Each of the five core-lint jobs runs its core Program, then its plugin chunks; check-lint owns the sixth stripe, script lint, and formatting. Current hybrid targets use three hosted extension-lint rows for complete lint plans, preserving the two packed core-lint jobs and leaving scripts, optional UI checks, and formatting in check-lint. The same eight-directory chunks are redistributed by the existing stripe selector and remain separate serial processes; each row prepares the SDK once before its chunks. This removes three checkouts, dependency setups, and SDK preparation invocations without adding a producer job or changing the SDK compiler budget. Targeted lint selections retain their existing six-stripe ownership. When the installed planner falls back to complete lint, it reports the reduced row count to the existing failure observer; its six-row preflight reservation remains conservative. This avoids adding declaration preparation to the heaviest core stripe. Every extension stripe uses the existing eight-directory chunks, one lint thread, and serial semantic Programs. Restored boundary artifacts still pass the existing content-freshness checks; a miss or changed input rebuilds them. Preflight counts its selected rows within the unchanged hosted admission limit; any newly affordable offload moves existing selected work and adds no coverage. Frozen hybrid targets, Blacksmith, release gates, and historical targets without extension-stripe support retain their existing plugin-lint ownership.
The compact Node planner keeps separate Blacksmith and standard 4-core hosted timing ownership. The github profile and serial hybrid jobs admit 210 predicted seconds per job, including shared runtime preparation. Eligible ordinary hybrid bins use the larger Blacksmith capacity policy below; file partitions, process envelopes and explicit worker pins stay intact. GitHub applies a 1.6x median scaling fallback only to unmeasured groups. Hybrid splits groups using the slower of the first-attempt Blacksmith estimate and the hosted retry estimate against the unchanged 150-second ceiling. This prevents a faster retry estimate from leaving a slow first attempt indivisible. Its attempt-1 packing applies the existing 0.87 scale to Blacksmith estimates, using committed measurements before the cold-start hints. These calibrated predictions remain separate from the recorded wrapper wall times. Refits can change the number and composition of compact jobs without changing runner policy. Direct sampled hints cover the doctor and cron-service outliers. In hybrid, the unmeasured agentic-gateway-core-3 tail retains its 140-second fallback within the applicable serial or parallel admission budget.
Hybrid compact jobs containing test/scripts/write-unified-entry-dts.test.ts request blacksmith-32vcpu-ubuntu-2404 after packing. In run 35186413697, the unchanged 120-second cache and cold-output test took 168.96 seconds on a two-CPU allocation from the 8-vCPU class; every measured compiler phase returned successfully. This promotion preserves job names, file partitions, two-worker pins, serial execution, compiler invocations, and deadlines. GitHub mode and hybrid retries retain hosted routing. Final exact-head CI must verify timing on the larger allocation.
The Blacksmith profile reuses the existing file partitioner for three measured serial outliers: chat/session control-plane tests, the third Gateway core group, and infrastructure storage/state tests. Their complete file inventory, config ownership, worker pins, build prerequisites and complete timing-history floors remain intact. Agent support stays one larger-runner job. On the captured 2026-09-02 inventory, these exceptions add three compact jobs while plugin consolidation removes twenty-two. The resulting broad-PR projection is 103 Node jobs; the last measured run used 121. Source inventory has changed between those observations, so actual CI must establish the final count and wall-time improvement.
GitHub pull-request and push packing checks for a single exchange between two existing bins before opening another anchor job. This avoids stranding time when one bin reaches the ten-group limit. Both replacement bins and the incoming group must pass the original admission rules, including time, prerequisite and sibling-family constraints. Pull-request anchor placement stays independent of hosted tooling inventory; Blacksmith and hybrid plans retain their existing packing.
Ordinary non-Windows CI targets eight minutes; Windows must still pass but sits outside this latency objective. The 210-second expanded packing budget is an estimate, not a job timeout or a measured workflow result. Preflight, checkout/setup, queueing and actual test walls all count.
Dynamic child timings bind to the configs, environment, complete parent file inventory, and ordered child allocation. Human shard names stay readable in logs; wrapper timing spans use the membership key. A parent total can be reconstructed only from every part of one matching allocation, so partial samples from different partitions cannot be combined. Every child retains its file-weighted share of the current parent estimate, and a matching child sample can raise that floor. GitHub admission uses hosted measurements; hybrid admission uses scaled Blacksmith measurements while either profile can require a wider shared split. Failed and timeout-and-retry samples are excluded from refreshes.
Whole-config groups with registered file listers (CLI processes, agent support, gateway methods, runtime config, isolated unit fast) split into file-weighted hosted stripes. CLI file weights use serial file-boundary intervals from successful main runs, so the longest process fixture is separated from the remaining files without overcounting concurrent cases. Agent support and chat also carry relative file weights to separate their slowest owners; support anchors include median case-body sums from three successful main runs. The subprocess-heavy tooling family uses sixteen file-weighted stripes based on 2,412 seconds of observed serial work in run 33364935118. Tooling file weights include import/setup time; a shared runtime prerequisite is charged once to the stripe that owns its consumers. Compact admission charges the strongest prerequisite once per job (100 seconds for runtime, 104 seconds for private QA, with the hosted scaling fallback). Mixed explicit file groups keep their runtime consumers together and stripe the remaining test work separately; the fixed build cost does not create extra stripes for files that need no build. GitHub tooling retains its packing of remaining files into the existing 150-second budget. A single packing pass applies that complete cost and the existing sibling-separation rules; no later rebalance can invalidate an admitted cap. Hosted child splits retain nonempty file partitions and recompute build ownership for each child. Indivisible files may exceed the target and retain their truthful prediction.
If a GitHub pull-request compact plan exceeds the 96-row cap, the planner repartitions hosted tooling tails once before applying the same packing rules. Full-size chunks stay intact; only the final divisible tail is split into parts of at most half the existing 150-second budget, allowing tails from different families to share a job. The splitter regenerates membership-bound timing identities and retains measured floors, build costs, runner requirements, worker pins, and serial isolation. The final 96-row guard still rejects plans that do not fit. Blacksmith and hybrid planning are unchanged.
When agent-support membership changes, its native fallback retains 479 seconds from complete main-run spans of 478.25, 450.21, and 418.13 seconds in runs 33537556582, 33537739443, and 33543106647. The hosted fallback remains 253 seconds. Hybrid keeps that observed native fallback without applying the older whole-suite scale; matching child samples can still raise the file-weighted parent floor. This kept the measured 241-file inventory in four hybrid parts when the old 240-file generation no longer matched. On the Blacksmith profile, the whole support group now requests blacksmith-16vcpu-ubuntu-2404; its files, process envelope and resource-derived worker limit stay unchanged.
Compact descriptor counts and predicted maxima vary with the committed measurements; the serial TUI PTY dist descriptor keeps its indivisible measured wall. In github mode compact jobs run hosted; hybrid attempt 1 uses each row's 4-vCPU, 8-vCPU, 16-vCPU or 32-vCPU Blacksmith label, while hybrid retries use hosted runners. Every fresh Control UI E2E plan for a target with the named-project contract uses twelve combined weighted shards plus one browser-extension row, across backend profiles, attempts and frozen targets. Eligible Control UI rows request the 16-class; the browser-extension row keeps the 8-class unless admitted to hosted Ubuntu by the bounded hybrid plan. Targets without the named-project contract retain the same combined Control UI command and their historical width: three shards on the Blacksmith planner profile or thirteen on hosted profiles, plus the browser-extension row. Failed-job-only reruns retain their previously emitted UI matrix, including older six-shard plans; rerunning preflight selects the current width. Two/one-worker project limits, the 25-minute timeout, max-parallel 14 and the conservative registration ceiling remain unchanged. QA Smoke uses its existing four-part plan on normal upstream hybrid first attempts, removing two repeated checkouts, dependency setups and private runtime builds. Blacksmith profiles retain four parts; GitHub profiles and freshly planned hybrid retries, manual targets and runs with missing attempt metadata retain six. Failed-job-only retries preserve the original matrix width. Hybrid first attempts use Blacksmith; GitHub mode and hybrid retries use hosted runners. All scenarios, separate channel runs, concurrency limits, stagger and deadlines remain unchanged; native timings must establish the effect on completion time. QA's planner reserves the final part's observed roughly two-minute Matrix rider before greedily assigning primary scenarios, keeping that separate run from becoming the tail. Windows uses the complete two package-script inventories as input to scripts/lib/ci-windows-test-plan.mts, which emits up to five disjoint whole-file rows. Current measured inputs need five rows to stay below the 420-second estimate; four predict 489 seconds each; five predict 412 seconds each. Shared project grouping reduces repeated Vitest starts, and the runtime-consuming files share one preparation row. Each row retains one project process at a time, with the runtime prerequisite prepared before readers start. The Blacksmith 16-class delivers four native CPUs and 14 GiB and uses four Vitest workers; hosted fallback retains one worker. Windows invocations explicitly enable file parallelism for their selected files. The single-file package-contract and fake-timer selections retain their own one-worker budgets. See the worker-count measurements for the prior two-row measurements and source identities. The worker-artifact fixture remains shared within its unsplit file. Historical targets without this planner retain two package-script rows. max-parallel: 5 permits overlap; actual Blacksmith/hosted capacity and queue time must be measured because the older two-runner capacity observation does not establish five available slots. A hybrid retry retains the emitted inventory on hosted windows-2025. Expect slower individual builds on standard 4-core hosted runners. Blacksmith's runner-registration budget is irrelevant for hosted jobs, but GitHub-hosted concurrency limits apply.
The two package-script input inventories keep small projects together: general unit, Gateway, infrastructure, runtime config, media, and CLI selections are in the first inventory; fast unit, plugin SDK, shared core, and agent support selections are in the second. That alignment removed eight repeated project processes from the earlier two-job plan while preserving all 133 explicit targets. The five-row planner consumes their union and regroups execution by shared project metadata rather than preserving the two input parts as job boundaries. Tooling remains divisible by whole file because its compiler fixtures dominate execution; the extension catch-all retains plugin-owned process boundaries. Windows still proves native path casing and aliases, PATH/shim resolution, spawn and process-tree cleanup, scheduled-task services, SQLite/file-locking and publication behavior, plus shared CLI, package, and runtime-import smoke.
Mac Node coverage uses two disjoint package-script parts on the existing runner labels. The elevation lifecycle, skills and remaining runtime projects share the first part so their subprocess consumers reuse one prepared worker generation; native artifact, packaging and checkout tooling share the second. All 26 files remain selected exactly once. The aggregate test:macos:ci command runs both parts; historical targets with three part scripts retain three jobs, and targets without part scripts retain one complete job. Project execution stays serial, and the existing two-worker, CPU-clamped three-case Mac fixture and two-case checkout limits are unchanged. In hourly run 38823827091, the first two-part allocation spent 247 and 465 seconds in tests, with separate 64- and 90-second compilations of the same worker input graph. Grouping its runtime consumers removes the second compilation and transfers the other runtime projects to the shorter row. The same run spent 137–140 seconds per Node setup, compared with 68–118 seconds in the prior three-row baseline; unchanged tooling files also ran slower, so total job time is not a controlled comparison of partition shape. Native hourly timings establish the resulting wall time.
The OpenClawKit package lane creates one fresh scratch directory for its Talk-disabled build and default-trait test suite. SwiftPM replans and recompiles trait-dependent modules when enabling Talk, while retaining dependency resolution and checkout work within that job. No products are copied from an earlier run. Both the no-ElevenLabs opt-out guard and the complete default-trait suite remain; cleanup runs after success, failure, cancellation, or an unselected suite. In hourly run 38817599758, the second fresh graph repeated 19 dependency operations over 16.9 seconds before compilation. A local compile-only replay using one scratch eliminated those repeated operations and verified all three production modules recompiled with Talk enabled. Hosted elapsed savings come from the next natural hourly.
After recovery, restore the value recorded before switching to the GitHub fallback. For example, restore a previous hybrid setting with:
Delete the variable only if it was previously unset; deletion selects the default Blacksmith-first routing:
ci.yml does not check Blacksmith or mutate this variable. Hybrid fallback is per job and activates only when a coordinator reruns the workflow or selected failed jobs.