CI runner classes

Runners

Runner choice follows contributor trust, not whether a pull request came from a fork. Every runs-on expression admits Blacksmith only when github.event.pull_request.author_association is OWNER, MEMBER, COLLABORATOR, or CONTRIBUTOR, so a fork pull request from someone who has already landed a commit is routed exactly like a maintainer pull request. FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE, and MANNEQUIN stay on GitHub-hosted runners, which are free for public repositories, so an unreviewed author cannot spend Blacksmith capacity. Maintainers report CONTRIBUTOR here because org membership is concealed; keep CONTRIBUTOR in that list or maintainer pull requests lose Blacksmith. Pushes and manual dispatches are unaffected. Cache trust is a separate, stricter boundary: exact dependency restores require a pull request from openclaw/openclaw, and ordinary CI never publishes the shared archives. The separate trusted warmer owns publication.

The automatic iOS smoke, full-manual build phases, and screenshot shards always use GitHub-hosted macos-26. Repeated first attempts left the Blacksmith macOS jobs unassigned while other CI completed. In run 33616182173, the hosted retry assigned all three waiting Mac jobs within eight seconds; the Debug/simulator job passed in 15m31s. Starting on that verified image removes the wait-before-retry path. The Xcode pin, ordinary pnpm-store cache, and matrix caps are unchanged; complete native evidence remains required for full manual qualification.

The Node test planner marks only shards that run the real native grep fixture. Those Linux jobs install the ripgrep package when the selected runner image does not provide it. Other Node shards do not pay that setup cost.

Current targets share one checkout/setup per fast contract family. The two weighted plugin selections still run as separate test:contracts:plugins processes; the two channel selections still run separate test:contracts:channels invocations, each retaining its four owning configs, four project slots and one worker per project. The envelopes run sequentially, and any nonzero exit stops the job before another envelope is admitted. Frozen targets keep their original matrix rows and execute one envelope per row. Runner routing, caches, worker budgets and aggregate-gate selection stay unchanged. In main run 33704083233, the separate plugin bodies totaled 94 seconds and the channel bodies 145 seconds; those sums support consolidation but are not measured combined durations.

Blacksmith runner capacity

Npm preflight, test types, core type stripes, and runtime-topology checks request blacksmith-32vcpu-ubuntu-2404 to compensate for smaller delivered machines. In the 2026-09-01 capacity probe, that label was the first measured class meeting the eight-CPU/24-GiB threshold used by OpenClaw's parallel-check policy:

OS CPU count, affinity, Node, and CPU-time measurements agreed. Guest cgroup quotas were unlimited. The provider-side reason for the mismatch is unresolved; the table records observed capacity, not Blacksmith's advertised specifications or a guaranteed allocation. The probe measured capacity, not whole-release speedup or billing equivalence.

The larger requests compensate for that observed allocation. Size workers from nproc/available parallelism and memory rather than the label. This promotion preserves existing worker limits, explicit budgets, matrix sizes, and max-parallel: a 32-vCPU label does not authorize 32 workers. Promoting these existing jobs adds no runner registrations. Repeat the capacity probe and reassess sizing after Blacksmith's allocation changes.

Backend routing still applies. Hybrid retries and untrusted pull requests retain their hosted routes. Ordinary manual CI dispatches remain hosted in hybrid mode; Full Release Validation's existing frozen-target lint exception remains separate. Npm preflight uses the larger Blacksmith request by default and retains its explicit use_github_hosted_runners option.

Runner backend modes

The macos-swift lane builds Swift tests once and runs each test once per job. The ordinary suite retains default-profile behavior; AppState isolation tests run afterward in a separate named-profile process through the same resource-owning launcher. Each launch owns a private home and disposable, unlocked default Keychain until the test process group and output pipes close. HOME and profile markers do not isolate macOS services; both partitions run only on the disposable credentialless macOS worker. Current launcher-capable targets bound Swift Testing parallelism to the runner's logical CPU count, capped at 12, for automatic runs, manual dispatches, and rerun attempts. Only frozen targets that predate the resource owner use the serial fallback. A failing test fails the job without an in-job retry. See native test safety.

The repository variable OPENCLAW_CI_RUNNER_BACKEND controls the runner backend for ci.yml:

Configurable heavy lanes are build-artifacts and android. The macOS Swift, iOS build, and screenshot jobs always use macos-26. The focused macos-node lane uses the existing GitHub-hosted macos-15 image in hybrid mode, with the same test inventory and two-worker limit. openclaw/ci-gate always uses ubuntu-24.04: its Bash-only result aggregation needs no checkout or dependency setup. This removes one Blacksmith registration from previously eligible runs without adding jobs or changing the required check. Hosted runner assignment can still delay completion. preflight uses GitHub-hosted Ubuntu in hybrid mode, with the same logical planner profile and cache trust; unset or blacksmith keeps its existing route. This avoids the measured first-attempt preflight queue on Blacksmith, but hosted assignment can also queue and needs native measurement. security-fast uses Blacksmith only on eligible hybrid first attempts and stays hosted outside hybrid. Security hooks use pinned installed packages and local hook definitions, so they no longer initialize remote Git repositories. Budget one control-job registration per eligible Blacksmith run or eligible hybrid first attempt; the github override remains unchanged. Hybrid sends the compact Node matrix, up to 80 compact rows plus separately appended plugin fallback rows, seven-row checks-ui-e2e matrix for non-frozen targets with the named-project contract, the checks-ui-e2e-real-gateway lane that shares its serial Chromium workload, four-row QA Smoke matrix on canonical automatic runs (six rows for manual dispatches), the two-part Windows matrix, checks-ui, check-lint, check-test-types, the two check-test-types-core-* rows, check-dependencies, check-additional-extension-package-boundary, check-additional-runtime-topology-architecture, and report-plugin-sdk-api-diff to Blacksmith on attempt 1. Eligible multi-group ordinary compact rows request blacksmith-32vcpu-ubuntu-2404. Other compact-small rows retain blacksmith-4vcpu-ubuntu-2404, compact-large rows retain blacksmith-8vcpu-ubuntu-2404, and the planner's measured small queue-tail promotions retain their 8-vCPU labels. Every other configurable ci.yml lane stays hosted in hybrid, including the core-lint jobs, the remaining lint/check rows, docs, and Python skills. Separate Opengrep workflows remain GitHub-hosted.

Hybrid is the normal degraded-capacity mode. If Blacksmith is down: rerun the failed or stuck heavy job; it lands on hosted automatically. During a full Blacksmith outage, record whether OPENCLAW_CI_RUNNER_BACKEND is set and its current value, then enable the github circuit breaker:

gh variable set OPENCLAW_CI_RUNNER_BACKEND --repo openclaw/openclaw --body github

The github override also routes Full Release Validation orchestration, npm qualification, live QA, performance, package Telegram, OpenWebUI, and release runtime-pair jobs to GitHub-hosted Ubuntu. Existing explicit hosted-runner inputs remain supported. Runner placement changes; coverage, artifact identities, approvals, worker limits, and timeouts do not. Already-running jobs are not moved. Keep OpenWebUI disk requirements and performance baseline hardware differences in mind when interpreting hosted results.

Hosted ci.yml paths use the same setup exercised by manual dispatches and fork pull requests. Fork PRs are forced into the logical github planner profile even when repository variables are unavailable, so broad core lint and test-type workloads retain hosted stripes instead of falling back to oversized all-in-one jobs. Frozen targets opt into this event-aware profile through hosted-runner-profile-contract-v1; targets without the marker retain their historical workload shape. Blacksmith-only Docker and sticky-disk steps are skipped, dependency setup uses the ordinary Actions pnpm-store cache, and low-memory Android builds use separate Gradle processes. Hybrid attempt-1 Blacksmith Node and plateau lanes restore the exact workspace dependency archive from the trusted warmer. Eligibility uses the actual runner environment, so hosted lanes and retries stay on the ordinary store cache. The exact key includes the resolved Node patch, OS, architecture, and semantic dependency inputs; a different runner image safely misses and follows the existing store-install path. The Node toolchain itself is also cached through that API: Blacksmith's image tracks an older runner-images snapshot whose toolcache Node patches (measured 2026-08-16: 20.20.0, 22.22.0, 24.13.0) sit just under this repo's engines floor, so every job otherwise re-downloads Node from nodejs.org. Restores are prefix-keyed and saves carry the resolved patch, because an exact-key hit suppresses the post-job save and would pin the first payload forever once the floor advanced past it. A restored payload below the floor is rejected, pruned, and replaced. Vitest transform and Node compile caches still use the upstream Actions cache API; their Linux-only runner.os != 'Windows' conditions do not exclude Blacksmith labels, and the trusted warmer alone publishes each backend-local protected seed. The warmer's selected runner route determines whether that publication reaches Blacksmith's cache or GitHub's cache. Core oxlint keeps five deterministic hosted Programs with one lint thread each. Ordinary non-frozen hybrid push/PR runs group stripes 1+2 and 3+4+5 sequentially across two jobs. A failing stripe stops its row. The github profile, frozen targets, manual dispatches, and release gates retain five jobs; GitHub plugin stripes keep their existing owners. Plugin lint ownership is described below; script lint and optional UI and format checks stay in the existing check-lint row. Extension type-aware lint discovers extensions/tsconfig.json for plugin tests and helpers, retaining imported dependencies and shared ambient declarations without adding unrelated core/UI/package source roots. Plugin production files keep their existing package-boundary projects. Eligible test-only pull requests reuse the local changed-check selector: check-test-types validates the complete core graph boundary once, then compiles every graph that consumes the changed tests. Preflight omits both check-test-types-core-* jobs before runner allocation, and the additional-boundary lane transfers its core graph check to this required central row. Ambiguous compiler ownership or a removed test falls back to all 16 graphs in the central row. Pushes, manual and frozen targets, shared or mixed changes, and unsupported targets retain the full path. For full runs with stripe support on github or hybrid, the two check-test-types-core-* rows run stripes 1+2 and 3+4 sequentially, and check-test-types runs stripe 5 before the extensions/root/scripts tail. Targets without stripe support and the all-Blacksmith profile keep the full central path. Each core type row preserves at most two concurrent compiler children and one builder per child. Core checks retain the standalone resource policy; the remaining type commands retain their existing environment. A failed boundary or compiler stops its row before another command starts.

The infra type graph owns infrastructure and logging tests. The core-test boundary guard requires every test root exactly once across the 16 graphs, with at most 720 roots per graph.

On hosts with less than 24 GiB RAM, serial plugin lint runs use the same eight-directory chunks as Windows. This bounds each type-aware process while covering every plugin and root source file. Outside Windows, explicit full-speed or parallel overrides keep the previous unsplit workload. The Windows chunk-size override remains Windows-only. Lint prepares only the SDK declaration tree; the separate package TypeScript boundary check still prepares the SDK and plugin declarations.

For current targets using the github profile, plugin lint chunks are divided deterministically across six existing jobs. Each of the five core-lint jobs runs its core Program, then its plugin chunks; check-lint owns the sixth stripe, script lint, and formatting. Every selected chunk retains its original arguments and runs sequentially under the existing resource and artifact-ownership limits. This removes the single-job serial tail without adding jobs or increasing chunk sizes. Hybrid, Blacksmith, release gates, and historical targets without extension-stripe support retain their existing plugin-lint ownership.

The compact Node planner keeps separate Blacksmith and standard 4-core hosted timing ownership. The github profile and serial hybrid jobs admit 210 predicted seconds per job, including shared runtime preparation. Eligible ordinary hybrid bins use the larger Blacksmith capacity policy below; file partitions, process envelopes and explicit worker pins stay intact. GitHub applies a 1.6x median scaling fallback only to unmeasured groups. Hybrid splits groups using the slower of the first-attempt Blacksmith estimate and the hosted retry estimate against the unchanged 150-second ceiling. This prevents a faster retry estimate from leaving a slow first attempt indivisible. Its attempt-1 packing applies the existing 0.87 scale to Blacksmith estimates, using committed measurements before the cold-start hints. These calibrated predictions remain separate from the recorded wrapper wall times. Refits can change the number and composition of compact jobs without changing runner policy. Direct sampled hints cover the doctor and cron-service outliers. In hybrid, the unmeasured agentic-gateway-core-3 tail retains its 140-second fallback within the applicable serial or parallel admission budget.

The Blacksmith profile reuses the existing file partitioner for three measured serial outliers: chat/session control-plane tests, the third Gateway core group, and infrastructure storage/state tests. Their complete file inventory, config ownership, worker pins, build prerequisites and complete timing-history floors remain intact. Agent support stays one larger-runner job. On the captured 2026-09-02 inventory, these exceptions add three compact jobs while plugin consolidation removes twenty-two. The resulting broad-PR projection is 103 Node jobs; the last measured run used 121. Source inventory has changed between those observations, so actual CI must establish the final count and wall-time improvement.

Ordinary non-Windows CI targets eight minutes; Windows must still pass but sits outside this latency objective. The 210-second expanded packing budget is an estimate, not a job timeout or a measured workflow result. Preflight, checkout/setup, queueing and actual test walls all count.

Dynamic child timings bind to the configs, environment, complete parent file inventory, and ordered child allocation. Human shard names stay readable in logs; wrapper timing spans use the membership key. A parent total can be reconstructed only from every part of one matching allocation, so partial samples from different partitions cannot be combined. Every child retains its file-weighted share of the current parent estimate, and a matching child sample can raise that floor. GitHub admission uses hosted measurements; hybrid admission uses scaled Blacksmith measurements while either profile can require a wider shared split. Failed and timeout-and-retry samples are excluded from refreshes.

Whole-config groups with registered file listers (CLI processes, agent support, gateway methods, runtime config, isolated unit fast) split into file-weighted hosted stripes. CLI file weights use serial file-boundary intervals from successful main runs, so the longest process fixture is separated from the remaining files without overcounting concurrent cases. Agent support and chat also carry relative file weights to separate their slowest owners; support anchors include median case-body sums from three successful main runs. The subprocess-heavy tooling family uses sixteen file-weighted stripes based on 2,412 seconds of observed serial work in run 33364935118. Tooling file weights include import/setup time; a shared runtime prerequisite is charged once to the stripe that owns its consumers. Compact admission charges the strongest prerequisite once per job (100 seconds for runtime, 104 seconds for private QA, with the hosted scaling fallback). Mixed explicit file groups keep their runtime consumers together and stripe the remaining test work separately; the fixed build cost does not create extra stripes for files that need no build. GitHub tooling retains its packing of remaining files into the existing 150-second budget. A single packing pass applies that complete cost and the existing sibling-separation rules; no later rebalance can invalidate an admitted cap. Hosted child splits retain nonempty file partitions and recompute build ownership for each child. Indivisible files may exceed the target and retain their truthful prediction.

If a GitHub pull-request compact plan exceeds the 80-row cap, the planner repartitions hosted tooling tails once before applying the same packing rules. Full-size chunks stay intact; only the final divisible tail is split into parts of at most half the existing 150-second budget, allowing tails from different families to share a job. The splitter regenerates membership-bound timing identities and retains measured floors, build costs, runner requirements, worker pins, and serial isolation. The final 80-row guard still rejects plans that do not fit. Blacksmith and hybrid planning are unchanged.

When agent-support membership changes, its native fallback retains 479 seconds from complete main-run spans of 478.25, 450.21, and 418.13 seconds in runs 33537556582, 33537739443, and 33543106647. The hosted fallback remains 253 seconds. Hybrid keeps that observed native fallback without applying the older whole-suite scale; matching child samples can still raise the file-weighted parent floor. This kept the measured 241-file inventory in four hybrid parts when the old 240-file generation no longer matched. On the Blacksmith profile, the whole support group requests blacksmith-32vcpu-ubuntu-2404; its files, process envelope and resource-derived worker limit stay unchanged.

Compact descriptor counts and predicted maxima vary with the committed measurements; the serial TUI PTY dist descriptor keeps its indivisible measured wall. In github mode compact jobs run hosted; hybrid attempt 1 uses each row's 4-vCPU, 8-vCPU or 32-vCPU Blacksmith label, while hybrid retries use hosted runners. For targets with the named-project contract, Control UI E2E uses six combined weighted shards on non-frozen Blacksmith and hybrid first attempts and twelve on other freshly planned runs, plus one browser-extension row. Frozen targets without the named-project contract retain the same combined Control UI command and their historical width: three shards on the Blacksmith planner profile or thirteen on hosted profiles, plus the browser-extension row. QA Smoke uses its existing four-part plan on normal canonical hybrid first attempts, removing two repeated checkouts, dependency setups and private runtime builds. Blacksmith profiles retain four parts; GitHub profiles and freshly planned hybrid retries, manual targets and runs with missing attempt metadata retain six. Failed-job-only retries preserve the original matrix width. Hybrid first attempts use Blacksmith; GitHub mode and hybrid retries use hosted runners. All scenarios, separate channel runs, concurrency limits, stagger and deadlines remain unchanged; native timings must establish the effect on completion time. QA's planner reserves the final part's observed roughly two-minute Matrix rider before greedily assigning primary scenarios, keeping that separate run from becoming the tail. Windows runs two jobs with disjoint, project-aligned explicit test lists on every backend. This partitions the complete Windows-specific inventory without applying Vitest --shard to project-local single-file selections, which Vitest rejects. The split width is pinned to two because blacksmith-8vcpu-windows-2025 admits exactly two concurrent jobs (measured on run 31865243804): a third part queues behind a finished one, while a single lane serialized the whole 226-second body onto the wall and made Windows the slowest job in every run that scheduled it. The Windows cron process-identity proof uses the existing runtime prerequisite owner before starting its real Gateway; it does not require the broader E2E declaration and private-QA builds. Worker-artifact cases use independent per-test fixture lifetimes with at most two concurrent cases; the case that observes process-wide preparation logs remains sequential. The transform-cache cases share part 2 with the PowerShell installer, checkout-owner, and SDK declaration tests; the original worker suite stays in part 1. Jobs requesting the existing Blacksmith Windows class admit at most two project processes, each with one Vitest worker, after the shared runtime prerequisite completes. The requested label does not verify available CPUs or RAM; native proof must cover concurrent fixture memory and cleanup. The largest selected project starts first, using its exact recorded timing when available and the same advisory file weights as the CI planner otherwise; generated worker artifacts and per-project caches retain their existing ownership. Hosted fallbacks keep serial project execution. A hybrid retry reruns both parts on hosted windows-2025, slower but bounded. Expect slower individual builds on standard 4-core hosted runners. Blacksmith's runner-registration budget is irrelevant for hosted jobs, but GitHub-hosted concurrency limits apply.

Mac Node coverage uses three disjoint package-script parts on the existing runner labels. The elevation lifecycle suite owns one part; native artifact and packaging proofs own the second; checkout and the remaining platform projects own the third. The aggregate test:macos:ci command runs every part, and historical targets without part scripts retain one complete job. Project execution stays serial, tooling stays one file at a time, and the existing CPU-clamped three-case Mac fixture limit and two-case checkout limit are unchanged. Retained native costs place about 152 seconds of test work in the longest part, but setup and runner admission still require native measurement before claiming an eight-minute workflow.

After recovery, restore the value recorded before enabling the circuit breaker. For example, restore a previous hybrid setting with:

gh variable set OPENCLAW_CI_RUNNER_BACKEND --repo openclaw/openclaw --body hybrid

Delete the variable only if it was previously unset; deletion selects the default Blacksmith-first routing:

gh variable delete OPENCLAW_CI_RUNNER_BACKEND --repo openclaw/openclaw

ci.yml does not probe Blacksmith or mutate this variable. Hybrid fallback is per job and activates only when a coordinator reruns the workflow or selected failed jobs.

Related

• Install overview
• Release channels