CI pipeline

Blacksmith Linux x64 jobs now request their measured physical size. The planner retains its historical profiles and timing keys: 4/8/16/32 profiles select 2/2/4/8-vCPU runners at admission, raised when a group requires more memory. Windows, ARM, hosted routing, coverage, and concurrency remain unchanged. See runner allocation evidence.

CI continues during Full Release Validation; the legacy release-priority variable does not pause workflow admission. See deferred CI recovery for runs already deferred by older workflow revisions.

Gateway server tests use the same 16-file process bound in local full-suite runs and CI config/include selections. This retires the shared worker before retained module graphs exhaust a 16 GB runner on Bun. The CI shard keeps its original file membership, runtime selection, and outer timing key; the project runner executes its chunks sequentially. Native --shard and suite-wide state or output options retain their single-invocation semantics.

The existing live-cache release job has a 30-minute budget for three sequential eight-minute checks on one runner: stored cache floors, transport prefixes, and agent-turn prefixes. See prompt-cache regression coverage.

Native video smoke coverage uses four shards of four providers. Each provider has a ten-minute operation timeout plus 30 seconds of test overhead; each shard has a 50-minute job budget, leaving eight minutes for setup. These shards keep full-mode video testing disabled.

The Linux App Linux companion and Windows installer jobs allow 105 minutes for source staging, packaging, and native proof. Their canonical core builds use the existing OPENCLAW_DOCKER_PACKAGE_BUILD_TIMEOUT_MS override with a 60-minute allowance: cold declarations exceeded the default 45-minute package deadline. A Linux run completed its package declarations in 38 minutes before the deadline interrupted the following declaration phase; a complete Windows installer job previously took 78 minutes. The remaining job budget covers dependency installation, Rust compilation, desktop tests, and installer packaging. The override stays on each source-build step; release-workflow budgets and package-integrity checks are unchanged.

Precise PR plans can run the complete source-boundary gate first in an existing short, serial 8-class tooling job. Both configs retain separate test processes; a 90-second boundary reservation must fit the existing 150-second test budget. Other runners, preparation modes, parallel jobs, and full validation keep their standalone boundary ownership.

Broad PRs retain their compact selected-owner Node plan when time-based splitting would exceed the 130-row matrix cap. See Node test lanes.

Agent auth-store admission and configured model-owner changes explicitly select the Gateway agent-config readiness suite. Its lazy startup path is not fully visible to the bounded import graph; the owner watch preserves overlap coverage.

PR Node plans also select the QA scenario catalog's reference check when a changed path is absent from the checkout, including the removed side of a rename. This catches stale YAML code references without selecting the full QA smoke lane. Removed UI test paths also select the UI runtime inventory check so its literal Node-driven browser list cannot silently retain deleted tests. Both checks use the existing bounded Node planner and leave native and real-Gateway tiers unchanged.

Added tests outside the ordinary suite roots select the full-suite ownership inventory, including rename destinations. The policy watch uses the PR base to distinguish additions from existing-file edits, which add no inventory work. See selection and routing for the scope.

Gateway close and run-shutdown owner changes likewise select the Gateway detached-execution shutdown and late-admission suites. server.close() reaches those owners through the kernel's lazy shutdown runtime, beyond the bounded

This page is an index. CI is documented on nine pages, one per reader job. Open the page that matches your task.

Storage-engine PRs, scheduled main runs, and manual CI validations include three hosted PostgreSQL 17 conformance shards, required by the CI gate when selected. They run in parallel with separate databases and unchanged three-worker test limits, with a 30-minute job budget each. The conformance runner models sequential configuration processes, three parallel file workers within each process, and the shared runtime preparation paid by each job. Committed config/postgres-conformance-timings.json measurements price file work, configuration startup, and preparation; they never select the test inventory. Cold file scheduling follows Vitest's file-size order. The allocator minimizes predicted tail time, then total work, so preparation consumers can share one job. Unknown targets retain a default weight. Inspect any assignment with --shard 1/3 --list (or 2/3, 3/3). Refresh measurements from successful natural runs as the suite changes. The latest two file samples track recent workload changes; configuration and preparation costs use observed medians. These are estimates, including inferred import/hook costs from verbose logs, not exact per-file profiler measurements. Refresh weights when adding expensive files: the unknown-target default preserves coverage but can underestimate their cost enough to unbalance a shard. Rebalancing adds no jobs or per-job setup. The combined inventory matches the local full suite; a failure in one shard leaves the others running and fails the aggregate gate.

The PostgreSQL Scheduled Qualification workflow (postgres-conformance-weekly.yml) runs the same three conformance shards on main with digest-pinned images: PostgreSQL 17 daily at 07:17 UTC, and PostgreSQL 16 and 18 every Monday at 06:43 UTC. Manual dispatch accepts versions=all (the default), pg16, pg17, or pg18. PostgreSQL 17 uses the same image pin as PR CI. The daily run adds three hosted jobs at roughly 25 minutes each (75 runner-minutes per day); Monday's weekly run retains six jobs. All runs use the same measured shard allocator with a 50-minute job budget and at most three jobs running at once. PR CI keeps its existing path selection and three PostgreSQL 17 shards. All versions start with max_locks_per_transaction=512 for the conformance workload. Scheduled failures are visible in Actions through the workflow's failed jobs, with no additional notifications; the PR failure classifier continues to compare against hourly main CI's matching PostgreSQL 17 jobs.

Full hybrid extension lint packs the same standard chunks into three existing rows, sharing setup and SDK preparation within each row. Targeted plans and frozen routes retain their existing layout; see runner profiles.

Full hybrid PR checks share central formatting and scripts lint with core lint stripe 1 when those are the complete central command inventory. The existing 16-class runner executes them sequentially within its 15-minute deadline, removing one checkout and dependency setup. Additional central checks, targeted core stripes, and plans without another required check row retain their separate lint job. Central lint still runs after a core lint failure and fails the shared job if it fails.

Default fork first attempts run their existing core lint stripes on Blacksmith16, retaining the same core and extension chunk assignments and restore-only caches. Retries and the GitHub override remain hosted; see runner placement.

Automation admission filters known no-op events before runner allocation and concurrency, keeping automation on GitHub-hosted runners.

PR jobs and independent checks run to completion after a failure, including forks and retries. Setup, planning, build dependencies, cancellation, and watchdog deadlines still stop work that cannot safely proceed. Failing runs therefore take longer and use more runner time; successful runs keep the same selected checks, job names, and concurrency. pr-failure-report compares every failed or timed-out job with the newest completed scheduled main CI run and publishes a pre-existing, new, or unknown classification in its summary and openclaw/ci-gate, with machine-readable JSON output. When GitHub reports only successful or skipped prerequisites, the reporter records that no failures need classification without enumerating job pages. The CI gate still checks selected lanes and rejects unexpected skips. Reasons identify matched signatures or job-name-only matches. The existing known-main-red allowance and gate result stay unchanged. Detected build-verifier or lint-mode scope differences are unknown; other reasons say when selected scope was not compared. See job ordering and failure reporting.

Add ci:full before a PR run to select every checks-ui unit shard, all checks-ui-e2e shards (Control UI and browser extension), and every checks-ui-e2e-real-gateway shard. These use the same UI inventories and shard budgets as scheduled main CI, including the release-only exclusions. Editing a release-only test selects that test; full manual/release runs retain the complete inventory, including desktop resize proof. Unlabeled PRs keep change-selected UI coverage; the label does not expand unrelated platform lanes.

Current UI unit/browser CI retains three native shards with three workers each, complete discovery, runtime ownership, and first-shard WebKit selection. Two-shard trials did not reduce measured runner minutes and exceeded the critical-path budget.

Compact mocked Control UI E2E uses six 16-class jobs with four workers each, matching the four observed CPUs per runner and retaining 24 aggregate worker slots. Private source-server projects stay serial. The conditional browser-extension row, noncompact plans, and real-Gateway worker limits retain their existing rules. The Mantis chat recording runs in its dedicated evidence workflow and full release validation, or when its test changes; routine CI keeps the equivalent chat lifecycle assertions without recording the same scenario again. Sidebar cached-row parity screenshots and JSON require OPENCLAW_CAPTURE_UI_PROOF=1; the parity assertions still run during routine CI. Browser suites that only exercise the rendered UI borrow the invocation's production bundle. Private source servers are reserved for source-module interception, custom build identities, or development-server contracts.

The mocked Control UI sharder keeps four measured source fixtures that need compiled subprocesses together in one existing shard. Their combined timing weight participates in normal packing, so preparation is shared without adding jobs or changing test selection, isolation, or worker limits. The group is an explicit scheduling hint; a new subprocess borrower can add preparation to another shard until the measured group is refreshed.

UI fixtures import pure protocol constants and data builders directly, so mocked browser cases do not load backend stores or Gateway policy. The shared E2E cleanup hook reads the canonical admission state through its thin state module; policy and logging remain owned by the Gateway admission module.

When artifact reuse is enabled, service-worker-update and phone stale-build fixtures run in build-artifacts. Each copies the verified production UI artifact and refreshes its own build identity before serving it. Local, compatibility, and frozen targets retain their ordinary preparation path.

Real protocol-mismatch recovery runs in the prebuilt real-Gateway pool, sharing that job's Gateway and current UI artifacts while retaining its incompatible protocol UI build. It no longer makes a mocked browser shard prepare a Gateway. Isolated child-Gateway fixtures share the existing two-worker parallel pool; in-process global Gateway fixtures remain serial. Both parallel phases start their longest measured files first without crossing the serial phase barrier.

The exact Doctor plugin-index and SQLite sessions/transcripts built-CLI proofs use the runtime-only compiled-worker registry. The local TUI file does the same when its existing built-CLI mode is enabled; source TUI and broad, mixed, or custom selections retain the full registry.

When the artifact producer selects both channels and core support checks with matching compiler heap options, they share one full compiled-worker generation. Both test commands still run concurrently with their original configurations, worker limits, and separate transform caches. The generation remains owned until both commands finish; either failure fails the producer without canceling its sibling. Single selections, differing compiler heap options, and older targets without the shared runner retain their independent commands. Changes to the shared runner select both checks in the artifact producer on current targets, while the changed Node plan defers its boundary coverage to that producer. This opt-in does not enable unrelated proof-tier checks.

Full and runtime-only compiled workers reserve separate live/cache slots, so runtime checks cannot replace a retained full generation. The protected warmer keeps its prebuild full seed for source-only Node shards and publishes the full generation already prepared by postbuild collection separately. Only selected source verifiers in the artifact producer restore that postbuild seed, keyed by the existing Node toolchain identity. A missing archive keeps their uncached path; an input, topology, runtime, or exact-output-path mismatch still compiles fresh. The producer creates no shared cache archive.

Real-Gateway UI shards have a 30-minute whole-job budget on the normal Blacksmith path. Successful 16-class jobs reached 19:04, leaving too little margin under the former 20-minute limit. Hosted and retry paths retain their 40-minute budget; runner size, worker count, shard split, and individual test deadlines are unchanged. Current targets overlap desktop transport proof with the one-worker serial UI phase. The planner balances serial files across the two existing runners; each runner keeps those files serial and joins cleanup before qualified fixtures use its existing two-worker ceiling. The serial and qualified phases share one prepared worker generation when both run; the desktop carrier keeps its separate lifecycle, and the shared generation is disposed only after both borrowers join. One-worker overrides, desktop-only selections, and frozen targets retain sequential execution.

First-hop compatibility uses a 3,200-second container budget and a 3,500-second lane budget, based on hosted 4-vCPU measurements with a slow-host margin. The release self-upgrade job gives first-hop lanes weight two at npm limit five, admitting at most two concurrently. It allows 210 minutes for three waves of six source versions, the survivor, and setup. Authenticated update restart uses a 3,420-second container budget, a 62-minute lane budget, and a lane-specific 1,500-second command timeout. Its dedicated recovery chunk allows 75 minutes; the remaining OpenAI package chunk allows 60 minutes. See release-path chunks.

For the published-upgrade regression gate, see selection and routing, runner budgets, and Package Acceptance baselines. Weekly validation is listed under Update Migration.

Main-tier and selected manual/release CI run the published-driver update cell. One GitHub-hosted Linux job runs the latest stable npm updater against the candidate package with two synthetic agents. The Docker cell retains its twenty-minute budget, followed by a fifteen-minute native systemd survivor step using the same candidate and published 2026.10.9. Both results are included in openclaw/ci-gate; the broader main/release Docker survivor remains separate.

Full main CI and cache warming are hourly by default; OPENCLAW_CI_ON_PUSH=true restores their existing per-push admission. CodeQL, Workflow Sanity, and CI's security-fast keep their existing main-push scopes. Docs-only main pushes still skip the CI workflow and push-triggered cache warming. The cache warmer publishes dependencies independently of long builds and maintains a bounded hosted seed in hybrid mode. Every admitted upstream main run exercises one published-driver × candidate Docker upgrade; ordinary manual/release validation adds the other five Docker seed lanes. QA Smoke, real-Gateway browser checks, and named process proofs retain their selected main coverage and manual/release validation. Pull requests and exact-head PR fallback dispatches run static correctness gates, owner-bounded tests, transitive import consumers, protected regressions, and a six-file runtime smoke set. Node rows target at most 150 estimated test seconds. Single files and indivisible standard groups can exceed that target; setup, builds, and queues are separate from test time. Node shards selecting sandbox container E2E cases prepare the Docker sandbox image when the runner does not already have it. Missing or unbounded runtime selection fails preflight instead of falling back to every test. Windows, browser, Docker, QA Smoke, packaging, contract, and extension families opt in through their existing owners; individual built-process proofs have independent owner flags. Full static fallback does not widen them. The PR-exempt integration tier retains measured slow tests in hourly main and Full Release Validation, with PR opt-in when their tests or subjects change. The existing Plugin Prerelease workflow owns complete extension runtime coverage hourly and in Full Release Validation; normal CI selects affected extension owners on PRs. Windows retains its complete inventory across five measured file shards on hourly main and ordinary manual/release validation; Windows-owner PRs retain that complete inventory.

Hourly iOS retains ios-build (tests) with Rust, voice, native Access, and focused lifecycle coverage. Successful current iOS builds run SwiftLint and SwiftFormat once through their required Xcode prebuild phases; a standalone fallback retains both checks after failed or skipped builds and for historical or frozen targets. Debug builds select their simulator before compiling, overlap its boot and SimSlim preparation with compilation, and join preparation before either focused simulator test group. Preparation failures or a timed-out join fail the job with its log; test selection and build settings stay unchanged. Current smoke builds prepare testing products once with build-for-testing; the voice cleanup and lifecycle groups separately reuse those products with test-without-building. Historical targets and other phases retain their existing build actions. Managed attachment UI/export, Watch operation, and Watch delivery UI suites retain every assertion in full manual/release validation. Main-tier simulator builds use the native architecture without indexing or verbose test diagnostics; logs and xcresult bundles remain available. A coalesced scheduled iOS cancellation can leave openclaw/ci-gate green with a notice delegating iOS proof to a later scheduled job; it does not validate the canceled revision, and the workflow can still be canceled. Genuine failures remain red. Screenshot capture runs for its own changed inputs and full manual/release validation. See scope selection and capacity for the coverage trade-off.

iOS Store Release reuses its locked Fastlane gems through ruby/setup-ruby's Bundler cache for trusted main upload jobs. The action keys gems by runner OS and architecture, Ruby, bundle configuration, and the lockfile; it still runs a frozen install on a hit and installs normally on a miss or cache-transfer failure. Generated Bundler configuration stays in the runner's temporary directory, and its absolute gem path lets the detached release worktree reuse the installed bundle without copying gems or dirtying the source checkout. Both mobile release entry points separately log elapsed time for the clean-checkout check and the current-main fetch, including failed checks, without logging arguments or credentials.

For a non-publishing notes/archive contention comparison, run node scripts/bench-ios-release-note-overlap.mjs --root CLEAN_SOURCE --plan PLAN.json --output IGNORED_DIR --samples 5 with OPENAI_API_KEY available. The plan schema is documented in the helper; freeze the source SHA and release identity before running. After warming the archive and notes generator, it interleaves warm serial and overlapping samples using the same generated inputs and isolated compilation caches. Each archive starts with fresh DerivedData while retaining compilation replay and package download caches to model a fresh release runner with restored caches. Results include timings, task counts, compilation replay hits/misses, and resource samples. Archives are unsigned: this does not validate signing, export, upload, or hosted-runner performance.

Store and TestFlight archives share an optional Xcode compilation cache, separated by Xcode build, runner OS, and architecture. The release job restores the latest matching compiler storage and saves an immutable run-specific snapshot after a successful release. Each transfer has a three-minute limit and cannot fail the release. Compiler input hashes own invalidation across source changes; signed products and DerivedData are never saved in this cache. Existing clean archive, signing, export, and IPA validation remain required.

Current iOS builds restore three independent input caches: verified Mermaid assets, SwiftPM source packages and binary artifacts, and the Watch RTC Cargo registry and compiled simulator library. Only trusted main push and scheduled runs save them; PRs restore only. Frozen targets retain their original cold path. Mermaid validates source and output hashes before copying resources. Its key covers the renderer's complete locked dependency graph, including workspace sources and optional build dependencies, so unrelated root dependency upgrades reuse the assets. SwiftPM downloads use apps/ios/build/SourcePackages, shared with screenshot capture through the same exact cache key and Xcode's -clonedSourcePackagesDirPath option. Keys cover Xcode, package manifests, and available Package.resolved files; automatic resolution remains enabled (the generated iOS project currently has no tracked lockfile). Watch keys cover Xcode, architecture, target mappings, the pinned Rust toolchain, lockfile, crate sources, and iOS build settings; restores first try the exact key, then fall back to the same OS, architecture, and Xcode prefix so unrelated iOS project edits can reuse existing slices. The build phase verifies the library checksum and input fingerprint before reuse, and otherwise runs the locked Cargo build. Caching the finished slice avoids rebuilding the Rust standard library when a fresh runner installs rust-src with new timestamps. The hourly Watch engine test shares registry downloads, while its host Debug products stay out of the simulator Release cache.

Current iOS Debug builds log CPU count, memory, machine model, booted simulators, and timestamps immediately around Xcode execution. The read-only hardware and simulator checks each have a five-second limit; unavailable diagnostics do not block the build. These markers distinguish simulator-query delays from Xcode startup, package resolution, and compilation.

Shared OpenClawKit Periphery scans restore the same verified Watch RTC libraries published by trusted main CI, without saving caches. Both Apple consumers build their complete index in a separate timed step with streamed output retained as build.log in the consumer artifact. iOS uses a fresh run-owned index; Periphery analyzes that exact index without rebuilding. Scan scope and the shared dead-code intersection stay unchanged. Cache misses retain the locked Cargo build.

iOS screenshot capture, release qualification, Store Release, and its screenshot-only operation use larger hosted capacity. Screenshot capture uses stock simulators and creates and cleans up one at a time. Before creating each device, it requires ten continuous seconds with at least 30% idle host CPU, using qualification's existing five-minute headroom deadline. A host that stays busy fails before the next simulator starts; boots and captures are not retried. The screenshot-only operation can validate a selected branch without signing or uploading a release. Debug screenshot fixtures temporarily record Mermaid renderer phases, elapsed times, cancellation, fixed failure categories, and numeric WebKit error codes in mermaid-renderer.log. Document identifiers correlate these events with bundled-resource response, data, and completion callbacks, using fixed resource categories and byte counts. The JavaScript invocation returns its entry timestamp through the existing evaluation callback without awaiting the render promise. Native bridge arrival and context/job rejection events distinguish missing delivery from rejected results. With screenshot diagnostics enabled, Fastlane retains each device's trace in SnapshotLogs before simulator cleanup on success or failure. Collection runs after XCTest with a ten-second deadline; it does not stream logs during capture. Diagram text, SVG, URLs, and error descriptions are excluded. Remove this incident instrumentation after the intermittent renderer fallback is diagnosed. The pairing, chat, and native Overview tests retain their existing assertions and deadlines.

Main CI screenshot capture installs the Mermaid renderer workspace and its dependencies with a frozen lockfile, retaining pnpm's root dependencies for the existing native helpers. It skips the general pnpm store cache and unrelated workspace installs. Project generation still builds Mermaid assets on a cold checkout. Screenshot capture restores the same verified Watch RTC simulator libraries published by trusted main CI, without saving that cache. Its Swift package sources and binary downloads consume the regular iOS build's apps/ios/build/SourcePackages cache. Trusted main push and hourly builds publish the downloads; screenshot jobs restore only. Screenshot preparation deletes all screenshot DerivedData, while downloads live outside that directory. Automatic package resolution and fresh compilation remain enabled. The screenshot build omits compiler index data while retaining the normal Debug configuration and all screenshot products. Historical and frozen targets skip both caches, and cache transfer failures retain the cold build path. Store Release's screenshot-only validation consumes these same two caches without saving them, and reports the verified Watch RTC reuse count after capture. Its upload operations retain their separate release caches.

iOS and Android releases check out the exact selected commit at depth one and gather their release plans there. The release entry point creates a detached worktree for notes and uploads. Release-note inventory deepens the exact source and recorded public baseline only until their comparison has complete history, without unrelated branches, tags, or historical file contents. TestFlight attempts that are unchanged or deferred for review or processing skip that extra checkout and history transfer. First releases and comparisons without relevant file changes need no extra ancestry. Release-note reads hydrate historical contents as needed; full local clones remain full. Source identity and main-ancestry checks remain unchanged.

iOS and Android upload jobs use a filtered frozen install for the Mermaid renderer and native protocol workspaces. Pnpm retains root dependencies for release helpers while skipping unrelated workspace installs. The release entry point reuses these dependency directories in its detached source worktree. Ordinary local releases and qualification retain their existing full-workspace installation. Run node scripts/bench-mobile-release-dependencies.mjs --samples 5 from a committed checkout to compare full and filtered installs, cold and warm stores, and regenerated native assets without uploading.

Both release destinations share a separate Watch RTC device-library cache, populated only by successful trusted main release runs. It retains both arm64 and arm64_32 device slices without sharing the simulator cache. Restores try the exact key first, then the same OS, architecture, and Xcode prefix so unrelated iOS project edits can reuse existing slices. The existing build phase verifies each slice's input fingerprint and checksum before reuse; a miss runs the locked Cargo build. Cache transfer failures do not block a release.

The same release job caches Xcode's SourcePackages downloads separately from ordinary CI, keyed by runner platform, architecture, Xcode, package manifests, and available lockfiles. Both destinations use the canonical $RUNNER_TEMP/ios-release-recovery/source project path, which determines Xcode's default DerivedData directory. Automatic package resolution and clean compilation remain enabled; build products and screenshot DerivedData are excluded. Only successful release jobs save downloads, and cache transfer failures retain the cold resolution path. Changing the canonical project path requires a new cache namespace.

Release qualification disables Spotlight indexing on the disposable runner's Data volume before checkout. After Simulator boot, it requires ten seconds of at least 30% idle host CPU within five minutes before issuing the pairing credential. This keeps first-launch background work out of the Gateway and XCTest deadlines. Local qualification uses the same CPU readiness check and leaves host indexing settings to the operator.

Android screenshot-input PRs and ordinary full manual CI run the existing phone and Wear store capture script in one hosted Ubuntu job. The final CI gate requires capture to succeed; unit-test-only and documentation changes omit it. See the job graph for capture evidence and scope details.

Android Store Release and Internal distribution use the Blacksmith 32-class for their shared build/upload job. A controlled cold-build comparison measured eight usable CPUs on this class and reduced median signing preflight plus phone/Wear bundle time from 621 to 217 seconds. This trades paid compute for shorter release runs; qualification and ordinary screenshot CI retain hosted Ubuntu. See runner evidence and cost.

Both Android release destinations restore the shared Gradle user-home cache read-only, reusing dependency downloads, compiled scripts, transforms and wrapper distributions from main CI. A cache miss or transfer failure keeps the cold build path. Release jobs do not save the archive or enable Gradle task-output caching. Toolchain setup runs before mobile dependency installation so cache-key hashing does not traverse the installed dependency tree. The shared toolchain action's gradle-cache-mode override leaves Android SDK caching off for these jobs; other callers inherit their existing cache-mode.

Android's Gradle daemon uses Parallel GC with the existing 3 GiB heap. A controlled phone/Wear release comparison with warm dependencies and fresh build outputs reduced median bundle time from 158.78 to 149.10 seconds (6.09%), with lower Java CPU time and peak memory. Task selection, lint, R8, signing, and upload behavior stay unchanged.

Android Release E2E adds one hosted Ubuntu KVM job to exact-source, full-scope manual validation when Android is selected, matching the iOS qualification selection. The final CI gate requires its success. Manual Android Store Release also requires qualification before the signing/upload job; internal distributions and Gateway APK publication retain their existing paths. It validates phone onboarding/chat/relaunch against a pinned stable Gateway, native transcript and keyboard behavior, and Wear native input, chat, history, restarts, and reply-reader behavior through a local raw-message transport at the Google Data Layer boundary. The production wire codec, phone bridge/controller, Gateway, and response/event path still execute. Real Google capability discovery, listener-service delivery, and Play services package/signature enforcement remain separate physical-device smoke coverage. Built debug APK hash, signature, package, version, and listener declaration checks remain in the gate; Release/R8, store signing, and upload remain owned by the existing release lane. See the Android release validation contract for evidence, rerun, and coverage details. Standard SDK images and build dependencies are downloaded during setup; no Google account, companion APK, or paired-device image is provisioned.

Eligible core-source and core-test PRs use targeted type checks when every selected compiler input exists in the checkout. Deleted Markdown files and UI stylesheets do not prevent narrowing for accompanying code changes. GitHub and hybrid profiles distribute the selected consumers across their existing core stripes; the Blacksmith profile checks them in the central row. Ambiguous ownership and deleted compiler inputs keep the full type-check coverage.

Numeric test-budget records retain full guard coverage without widening an otherwise scoped source/test compiler and lint selection.

Selected singleton compiler rows share setup in pairs within the existing two-compiler limit. Full stripes and rows carrying root partitions stay unchanged.

Scoped PR plans inspect the four root-test partitions separately and select every partition that consumes a changed input, including type-only imports. A change used only by tooling tests checks that partition; shared setup can select all four. Full checks retain the aggregate root alias. Missing or duplicate compiler inventories retain full coverage. Selected partitions keep their original stripe identities when four admitted core rows are available; smaller plans run only the selected partitions serially in the central row.

Preflight passes the complete changed-path manifest between steps as a local JSON file, so large PRs do not lose test-planning inputs to Actions output or environment size limits. Frozen targets that predate this transport retain their bounded JSON output contract. Missing or invalid inputs still reject current PR Node planning.

The Testbox check workflow requests the Blacksmith 16-class for routine dispatched proof, with a 60-minute total-job deadline including hydration. The explicit high-memory 32-class workflow retains 240 minutes for memory-heavy full-suite gates. The outer GitHub deadline can terminate active SSH commands; the separate 15-minute idle limit does not extend it. PR hydration checks stay on hosted Ubuntu; individual test deadlines remain unchanged.

All five lease workflows allow up to 60 minutes from dispatch to admission and runner startup, then reject expired requests before checkout and hydration. This queue allowance does not extend running-job or idle deadlines; see Testbox spending limits.

Full GitHub and hybrid type checks run the five core stripes independently, retaining two compiler children per job. The last four rows then each run one root-test partition serially, leaving extension tests and scripts in the central row. Narrow plans reuse four already-selected rows when available; smaller selections retain central root checking. This adds no jobs or compiler overlap. Current hybrid full runs use three hosted extension-lint jobs; targeted layouts retain six stripe identities. Trusted hybrid first attempts retain both packed core-lint rows on the Blacksmith 16-class. Ordinary upstream hybrid push and scheduled runs, plus same-repository PRs, place preflight, the selected pr-failure-report, and openclaw/ci-gate on hosted ubuntu-24.04; fork first attempts and all dispatch, qualification, release, and other backend routes retain their existing overrides. The mandatory controls count toward the existing hosted-row budget. See critical-path assignment evidence. Frozen targets keep their earlier layout; see static checks.

Additional checks and narrow-PR guards and dependency scans start directly after preflight. Ordinary PRs selecting Madge or Kysely divide guards into check-guards and check-guards-architecture, with the same runner routing and no compiler-plan wait. The manifest retains every check once, and both selected rows must pass. Other events retain their existing rows. Guards retain the exact comparison base and shared check commands; compiler and lint rows wait for their selected graphs. Known full compiler selections skip discovery while retaining the core graph boundary in an existing required owner; see pipeline ordering.

Changed compiler planning queries each config's complete file membership with --listFilesOnly, using tsrs by default and tsgo on unsupported hosts or when OPENCLAW_TYPECHECKER=tsgo. Type-only consumers and dependency declarations remain included; the selected compiler's bundled library directory is excluded because it describes compiler identity, not repository membership. Config expansion and shard/buildinfo validation still use tsgo's --showConfig. Queries run serially without opening native API snapshots. Upstream first-attempt check-plan jobs use the 16-class when the backend is unset, blacksmith, or hybrid, including fork PRs. Fork type stripes also use the 16-class with an unset or blacksmith backend; their logical GitHub profile and restore-only cache policy stay intact. Existing hybrid health admission, the explicit GitHub override, retries, and frozen routing remain in effect.

The extension package boundary row has a 40-minute job budget for SDK preparation, all selected plugin compiles, input-receipt validation, the required negative canary, and cleanup. Hosted four-CPU runs spent about 19 minutes in the compile command alone; one completed compile and canary but exceeded the former 20-minute whole-job deadline. Release validation on hosted ubuntu-24.04 later spent about 6 minutes preparing and 23 minutes compiling 128 plugins, passed both checks, and was cancelled at the former 30-minute deadline. Optional hybrid hosted overflow retains this row on Blacksmith: compiled receipt archives include checkout-specific paths and links, and hosted cold runs exceeded 22 minutes. Explicit hosted overrides, retry and trust fallbacks remain available. Other jobs keep their existing admission thresholds; the hosted row total counts only the rows actually offloaded. Compiler concurrency, coverage and cache guards are unchanged.

The Plugin SDK API diff row also has a 30-minute job budget. Two hosted runs took 722 and 725 seconds to render the base revision, then reached the former 20-minute deadline while rendering the candidate. The budget allows both full revision renders plus setup and cleanup. Rendering stays serial below eight CPUs or 24 GiB of memory; larger hosts retain two render workers. Other additional-check rows retain 20 minutes.

Core lint discovers separate source and UI TypeScript projects, retaining shared ambient declarations and imported dependencies. The source project also includes src/**/*.test-support.cjs; unrelated JavaScript files are not added as roots. See local checks.

Runtime topology checks inherit the existing Go memory defaults, with caller overrides and the full architecture check sequence retained.

Full Release Validation's Docker seed child uses the 16-class Blacksmith runner when no release runner group is configured, prepares the existing smoke package, and retains serial weighted lane admission. Hosted outage overrides and retries keep their recovery route. Ordinary manual dispatches retain hosted serial execution. All six lanes remain selected. The three long, unfitted hosted test rows (core-runtime-config, agentic-cli-process, and agentic-control-plane-agent-chat) have a 90-minute job cap until complete timing observations allow the release planner to split them. The targeted update-restart-auth lane has a 62-minute budget and a 75-minute job cap. Release-path migration matrices isolate channel switching and each published baseline on separate runners when multiple baselines are selected; per-host weighted resource limits and exact-candidate artifact checks remain unchanged. Package Acceptance admits its long standalone upgrade lanes before expanded scenario jobs and orders pinned baseline cohorts newest first, without changing its matrix cap or scenario coverage.

Android native resource preparation uses the Mermaid renderer's filtered dependency install, including optional build tooling. Pnpm retains root dependencies but omits unrelated plugin packages; Gradle still builds the assets and runs the selected native tests and lint. Historical targets keep their compatibility path.

Android phone tests use up to four isolated JVMs on Blacksmith and retain Gradle-owned cache expiry. The same four normal rows split phone tests from app lint: Wear owns Wear tests and lint plus third-party app lint, and Kotlin lint owns Play/shared lint. Upstream Blacksmith push and PR first attempts, including forks, overlap all four rows; the GitHub override, retries, manual dispatches, schedules, and other repositories retain two. All test and lint tasks remain selected.

macOS Swift CI runs the app and independent package suites in separate native phases, retaining every test and the existing concurrency limits. Ordinary native phases retain a 30-minute budget; full nonpublishing app-host recovery proof has 75 minutes to build and stage the exact runtime, exercise Retry, and verify a fresh-process relaunch.

Native test builds retain coverage and source-line backtraces while omitting IDE indexes and full debugger type metadata. Local development builds keep their normal debug settings.

The macOS app test phase installs the full workspace dependencies and builds the CLI runtime before Swift tests that use the shared exec policy owner. Other native phases keep the filtered Mermaid and protocol dependency install. Policy fixtures exercise the real local CLI rather than building it on their first request.

Short hybrid jobs use a 40-row base threshold and 45-row hosted admission limit, with unchanged coverage and Blacksmith fallback when optional work does not fit.

Additional hybrid check offloads require fresh hosted assignment evidence. Eligible PRs can move dependency, core type and topology checks; main pushes can also move lint and central types within the same hosted row limit. Artifact builds retain Blacksmith because their measured hosted tail leaves no room for the 15-minute routing objective.

Windows keeps its complete explicit test inventory in five measured project-aligned shards for main and release validation. Windows-owner PRs retain the complete family; unrelated PRs omit it.

Real-Gateway browser checks use job budgets matched to their selected runner. They restore the protected full-build cache and still run private-QA preparation. The Control UI step can reuse public build content because backend-only private plugins do not change its output. Native UI declarations, assets, source plugin discovery, and Vite-consumed inputs remain part of its validated cache identity. Existing UI test bodies do not invalidate an otherwise unchanged bundle; their complete path inventory still invalidates new or removed glob matches.

Control UI, repo E2E, and native live browser CI restore the Chromium revision pinned by the selected target's installed Playwright package, with separate OS/architecture cache keys and no fallback prefixes. The protected-main Vitest cache warmer publishes the browser cache in its short dependency job for both Linux backends; PR and release jobs remain restore-only. A cache miss still installs the managed browser, and current targets use the installer's --require-playwright-chromium mode rather than substituting system Chrome. Historical targets retain their Playwright installer and Linux dependency setup. Browser startup diagnostics include provider, page, WebSocket, and Chromium process events to diagnose a session-readiness timeout even when it is reported only after unrelated unit work finishes.

The first checks-ui shard also runs a curated WebKit overlay and composer-focus project when changed paths touch its interaction owners, their styles, or the subset in ui/vitest.config.ts. The manifest owns that selection. WebKit runs sequentially after the existing UI tests, with the same three-worker limit and no additional runner or matrix row. Scheduled runs without changed paths and frozen targets omit it. The selected row installs the pinned Playwright WebKit browser and Linux dependencies. Run the same project explicitly with OPENCLAW_UI_WEBKIT=1 pnpm --dir ui test --project=webkit --maxWorkers=3; ordinary UI test runs remain Chromium-only. CDP-dependent modal media and accessibility-tree suites stay on Chromium until their browser-neutral replacements land.

The WebKit project requires every selected assertion to pass, with no expected-failure exceptions. Shared fixtures wait for usable overlay content, inspect native adopted stylesheets, and establish keyboard traversal independently of Safari's pointer-focus behavior.

Linux baseline ratchets and native grep tests reuse an existing rg or download the checksum-pinned ripgrep 15.2.0 release directly into the runner's temporary directory. Setup does not use apt, sudo, package-index refreshes, or package-manager locks. The small archive download has bounded retries and transfer timeouts; unsupported architectures and checksum failures fail the job.

Browser extension CI launches the installed, patched Chrome MCP dependency directly, on Node and on the pinned Bun fork.

When an ordinary PR, push, or scheduled run already selects the public artifact build, current-target Control UI locale checks share that job's checkout and dependency setup. Source verification and locale parity run before generators, with their result settled after the existing build diagnostics. Source failures remain blocking; generated-locale drift retains its strict or advisory policy. Standalone selections, dispatches, frozen targets, and compatibility targets keep the separate locale job.

When the public artifact build is already selected, current-target browser extension E2E and the selected production deployment fixtures run directly in build-artifacts, before mutable built-artifact verifiers. They reuse that exact checkout's public build without an extra checkout, runtime archive, or runner. Native-policy and published-driver consumers wait for this browser proof with the rest of the producer job. Standalone browser selections and frozen targets retain their local runtime build. Real-Gateway fixtures retain their separate private-QA preparation. The native-host, Doctor and SQLite built-CLI verifiers opt into the existing validated worker cache to reuse their identical runtime-only compiled worker graph sequentially within the producer job; TUI retains its separate heap configuration and worker preparation.

Build, QA and test orchestration restore the same protected Node compile cache. The trusted warmer populates build tools before collecting test imports. Scheduled and push warming use ordinary CI's Bun policy, retaining the runtime owner's Node-only exceptions; explicit workflow dispatches prime both runtimes for full validation. Optional full Node runs may start colder between manual dual-runtime warmings. Both Linux cache backends retain the same seven Control UI seed files on Node and the pinned Bun fork, and hosted tooling and contract seeds keep their existing runtime owners. Ordinary CI remains restore-only.

Scheduled current-main artifact builds also publish their completed build-all cache without another declaration compile. The full warmer keeps its release profile; PRs and manual runs do not publish this archive.

In-process Gateway test configs use exclusive plan admission within existing packed jobs.

Changed-owner Node rows use existing file/group timing evidence with a 150-test-second admission target and the existing 130-row PR cap. Complete files, standard configs, and worker policies remain intact; predicted test seconds are separate from measured CI wall time. Compatible selected serial jobs share setup after singleton worker limits are fixed. Combined jobs retain the shortest contributing timeout and the existing test-work budget; complete and scheduled plans keep their original layout. Known indivisible-file costs remain a floor when selected subsets are priced; older complete-group measurements cannot cap those costs. Process-heavy worktree and updater suites carry explicit case-cost weights so they do not share a row on the default per-file estimate.

Plugin-sensitive PRs select their owner tests, transitive import consumers, protected regressions, and policy watches. The existing Plugin Prerelease workflow retains complete extension coverage hourly and in Full Release Validation; see Node test lanes.

Roomy serial Blacksmith Node jobs use measured Vitest worker sizing, with existing hosted, frozen-target, and overlapping-plan limits.

Source-only Linux Node shards can reuse content-validated compiled workers from the protected warmer; fixed preparation costs remain separate from test execution and runner capacity.

Changed-target shards whose executed file routes use the E2E config prepare the private-QA runtime once before launching test children, even when selection maps those files to a registered unit-suite owner. Preparation builds runtime JavaScript and assets without global declaration emission; the AI package test separately prepares its required declarations. Only a successful preparation step enables prebuilt consumption, so the children reuse its JavaScript, assets, and freshness stamps instead of starting another full build.

Vitest transform-cache generations include direct Vitest configs at every depth and the shared test/vitest/ inputs, except the two database-worker membership inventories, unit-fast discovery inventory, and the E2E config factory. Each project fingerprints its declaring config and imported config dependencies; the E2E factory also supplies its canonical file when loaded through a wrapper. E2E-only config edits therefore invalidate E2E transforms while preserving unrelated unit transforms. Shared compiler, helper, dependency, and schema inputs retain their invalidation. Generations exclude the generated .ci-harness checkout so CI consumers and the protected warmer hash the same source inputs. Node bytecode caching remains enabled for ordinary Vitest runs; Vitest owns the worker-level coverage safeguard described in local testing.

Transform keys also include each project's dependency optimizer directory. This prevents cached UI imports from mixing separate projects' Lit instances when a focused run and a full run share the persistent cache.

Linux PR tests use Bun for compatible unit lanes and Control UI Vitest selections. Audited synchronous unit-fast tests can use Bun's native runner; changed test or setup bytes return to Vitest. Full Release Validation retains complete Node coverage plus qualified Bun coverage; see test runtime selection. Both runtimes group uncached, non-isolated UI files by environment in batches to reduce worker restarts while retaining native shard ownership and worker budgets. Bun UI workers also use an allocation-based collection trigger to preserve memory headroom across the three native shards. Main runs UI on Node, so a PR changing the runtime owner (scripts/lib/ci-test-runtime.mts) selects the three existing UI unit shards on Bun; see scope selection.

Frozen-target CI loads its Node shard planner, planning helpers, and measured costs from the pinned workflow_sha checkout. Test discovery and execution still use the candidate source, so current shard budgets do not replace release bytes.

The npm/ClawHub release decision treats normal CI tests, plugin prerelease, cross-OS, performance, and QA lanes as advisory recorded evidence. Artifact, install-smoke, survivor, all first-hop compatibility, pack/npm qualification, package-integrity, and target-resolution proofs remain required. Aggregators follow required inputs; identity and provenance verification still apply.

For publication, the sealed manifest supplies the SDK evidence digest, per-package npm decisions, and any approved OPENCLAW_RELEASE_STABLE_SOAK_WAIVER text. Explicit publisher inputs override those defaults; historical manifests retain their existing input contract. SDK API changes still need an operator-supplied acknowledgement, and the sealed waiver applies only while the repository variable still holds the same text. Publishers still validate live authority, artifact bytes, and registry state at the mutation boundary.

Flaky tests never block npm/ClawHub publication: record advisory failures and investigate their owners without waiting for a green rerun. A passing replay alone does not prove a fix. Required artifact, install, update, target, and provenance proofs remain enforced. Native app publication is fully decoupled from npm/ClawHub, GitHub finalization, and main closeout; report each platform's readiness separately. The approximately 20-minute validation and one-hour publication targets require hosted timing evidence before being claimed.

Release closeout refreshes hosted full-release shard costs with node --import ./scripts/tsx.mjs scripts/ci-shard-timings-refresh.mts --run <ci-child-run-id>. The generator records successful hosted job walls, including setup, in the existing config/ci-test-timings.json store. Release keys stay separate from compact CI spans and survive the daily refit. The hosted full planner splits measured rows above 12 minutes after file bundling, retaining exact coverage and worker settings. Complete split generations keep subsequent plans from recombining expensive work. Agents-support, CLI-process, Gateway chat, Gateway methods, and runtime-config owners retain their completed hosted cost when files are added or removed, until a complete observation covers the new inventory. Partial generations never supply that floor. Their full-validation rows use the existing -hosted-N split, while compact main and PR routing retain their existing policy. An indivisible over-budget test fails planning with its owner named; unmeasured rows still need native timing evidence before claiming the 20-minute objective.

Full Release Validation's exact-target UI job retains the current three native shards for both runtimes. Historical compatibility targets keep their original unsharded package command; see UI job budgets.

Set the repository variable OPENCLAW_RELEASE_RUNNER_GROUP to reserve a runner group for Full Release Validation and its artifact, validation, and reusable worker jobs. The Release Publish parent and its dispatched publish children read the same variable. It selects the group; it does not provision runners or increase concurrency limits. Missing group capacity queues jobs. Shared workflows receive an optional runner_group from their release caller, including docker-release.yml and vercel-container-registry-publish.yml from Release Publish; docker-image-refresh.yml, ordinary CI, scheduled performance, and unrelated reusable callers retain their routing. Approval and credentialed publish jobs (npm trusted publishing, ClawHub, Docker, and GitHub App-backed release dispatch) keep their default GitHub-hosted labels, and the hourly plugin npm preview routes only when Release Publish dispatches it. The runner count, matrix caps, and default labels do not change.

To reserve capacity outside ordinary PR/main pools:

1. Create an org runner group with Linux runners labelled ubuntu-latest/ubuntu-24.04, plus the Windows/macOS labels used by validation.
2. Grant openclaw/openclaw access to the group.
3. Set OPENCLAW_RELEASE_RUNNER_GROUP to the group name; unset it to release the reservation and restore ordinary routing.

Linux runners for jobs that set semantic-checks: true also require:

• systemd as PID 1 and an active systemd-logind service.
• Noninteractive sudo access to check logind, enable linger for the runner user, and start that user's systemd manager.
• cgroup v2 with memory and swap accounting, delegated to the user manager.

These requirements apply to custom release groups as well as ordinary runners. The shared setup action starts the user manager and verifies a real 64 MiB scope with swap disabled and group OOM termination before checks run. Unsupported runners fail setup; provision these capabilities before assigning the validation labels, or unset the release-group override to restore ordinary routing. Setup qualifies the backend; it does not itself limit later lint or compiler commands.

Full Release Validation starts source-only children alongside artifact producers after admission and reuse selection. Candidate consumers start as soon as the candidate is verified, while npm qualification and independent validation can continue; see the release procedure.

Release-dispatched validation children add one best-effort hosted receipt job each, up to seven per full campaign and none for ordinary PR/main CI. It retains job results independently of parent completion. With reuse_evidence=true, each dispatch checks bounded prior receipts for its exact target and inputs, including candidate bytes, and adopts only verified successful children. Other roles still dispatch. Failed, cancelled, and active parents can supply green children; the current parent seals and revalidates each immutable selection. Discovery adds no jobs or Blacksmith registrations and falls back to fresh work on a miss.

Auto-reply reply tests run files in parallel with two workers per compact group. Their planner uses separate parallel timing identities; until those have measurements, serial group costs are divided by the effective worker count, with single-file groups retaining their full cost.

The Gateway isolated/database-worker cohort keeps its two-worker budget, including roomy serial Blacksmith and hybrid jobs, to leave cold startup headroom within existing test deadlines. Other packed groups retain their existing caps.

Commands tests share the existing worker budget across independent files. The Doctor session SQLite cases are split by operation while preserving the complete repair and recovery coverage; see shard weights.

The complete startup corpus uses eight state test files so existing workers can share its release/config matrix. Its explicit fallback prepares the runtime once and uses up to four workers, capped by available CPU parallelism; historical frozen targets retain their legacy process layout with CPU-bounded admission.

Where each section moved

Every section heading from the previous single-page version keeps its anchor here, so an existing link such as /ci#pipeline-overview still resolves. Each entry points at the page that now holds the content.

• Pipeline overview
• Job ordering and failure reporting
• Control UI size budgets
• Watching pull request CI
• Recover an existing PR run first
• PR context and evidence
• Checkout ownership
• Scope and routing
• Measured shard weights
• ClawSweeper activity forwarding
• Manual dispatches
• Windows Testbox Check
• Runners
• Blacksmith runner capacity
• Runner backend modes
• Runner registration budget
• Surface ratchets
• Local equivalents
• OpenClaw Performance
• Vitest paired benchmark
• Full Release Validation
• Live and E2E shards
• Package Acceptance
• Jobs
• Candidate sources
• Suite profiles
• Legacy compatibility windows
• Examples
• Install smoke
• Local Docker E2E
• Tunables
• Reusable live/E2E workflow
• Release-path chunks
• Plugin Prerelease
• QA Lab
• CodeQL
• Security categories
• Platform-specific security shards
• Critical Quality categories
• Maintenance workflows
• Dependency Audit
• Docs Agent
• Duplicate PRs After Merge
• Local check gates and changed routing
• Config baseline count ratchet
• Testbox validation

Related

• Tests
• Scripts
• Maturity scorecard
• Install overview
• Release channels

Ordinary PR iOS smoke keeps app and test-bundle compilation while selecting its two simulator groups by source owner. With neither group selected, simulator preparation is skipped. OPENCLAW_CI_IOS_SIMULATOR_FULL=true restores full PR execution; selection and routing documents the owners and full hourly/release coverage.

Android's existing android-access-native emulator rows retain their packaged Access-crypto proof and exercise Gateway WebSocket and widget HTTPS clients on Android Conscrypt. Each client must pass matching leaf-pin and platform-root controls, reject a wrong pin and wrong hostname before an origin request, and reject the verified incident OV G3C intermediate in both trust modes. The report reducer requires all twelve TLS class/method identities exactly once, separately from the existing Access case and packaged-library checks.

These rows use the Android toolchain action's filtered dependency install for native resource generation. They skip the full workspace pnpm store cache so the hosted runner has space to unpack the Android system image.

Platform-root incident cases require either exact Conscrypt key rejection or OkHttp's specific blocklist rejection after verifying the raw peer chain and hostname. They do not depend on Android downloading a platform blocklist update.

These secretless, disposable emulators temporarily import a synthetic localhost root and the verified incident root into copied APEX CA stores in the zygote mount namespaces before launching the test app. Stock roots and SELinux remain intact. Cleanup stops/uninstalls the task app, unmounts the copied stores, verifies the full original-store manifests and SELinux state, and always disposes the emulator and deletes its AVD without a snapshot. Failed restoration is preserved as contaminated runner evidence; that emulator must never be reused. Reports include fixture identities, before/after manifests and cleanup/disposal receipts. This adds no jobs, runner tiers or fanout and never changes host trust. Cleanup bounds device commands and trust restoration so an unavailable emulator cannot prevent disposal or report upload. The emulator boot log is copied into the reports and its last 100 lines are printed before device cleanup. A cleanup timeout remains a failure even when the owned emulator is killed and its AVD deleted.

Adaptive observation pilot

The adaptive CI observation pilot samples existing unit-fast Vitest invocations after runtime selection. It adds no matrix rows, runner registrations or skipped tests. Its isolated dependency cache and artifact upload add measurable overhead; projected selection savings are not realized savings.