--- summary: Default background computer use via a two-provider seam (CUA + Peekaboo) behind one typed computer.act v2 contract, app-owned TCC, cloud-gateway/multi-node ready. title: Computer use plan read_when: - Implementing or reviewing computer.act v2, the node provider seam, or the CUA/Peekaboo adapters - Changing macOS embedded driver spawning, provider selection UX, or managed driver artifacts - Extending Peekaboo or CUA integration surfaces --- ## Status Active campaign, started 2026-08-13. Owner-directed (steipete): build autonomously end to end, live-test everything, ship fully. Tracker table below is the source of truth and is updated as work lands. Derived from RFC 0025 (openclaw/rfcs `rfcs/0025-default-pluggable-computer-use.md`, RomneyDa) with the owner rulings recorded under Decisions. Coordinate with RFC author before each wave lands (check live PRs to avoid collisions with in-flight maintainer work). ## Problem OpenClaw's computer use is foreground-only everywhere: the macOS fulfiller drives the shared cursor via embedded Peekaboo + CGEvent primitives, and the `cua-computer` plugin uses only CUA's global desktop scope (`scope:"desktop"`, no `delivery_mode`). Both CUA and Peekaboo now have real background window-scoped input (no cursor move, no focus steal, no Space switch), semantic observation (AX tree + screenshot + element refs), and structured verification. None of that reaches the model. The current CUA adapter maps 8 of 49 driver tools; Peekaboo's MCP surface (25 tools, background-default clicks, `see` frame binding, `verify_state`) is not reachable through `computer.act` at all. ## Goals - One typed `computer.act` v2 contract (portable actions: window/element/browser targets, `deliveryMode: background|foreground`, verification envelope) carried over the existing node protocol. Cloud gateway + many paired desktop nodes works unchanged — provider processes live on the node. - Two first-class macOS providers behind one adapter seam in the app-owned TypeScript node worker: CUA (`cua-driver serve --embedded`, app-spawned, TCC via responsibility chain) and Peekaboo (`peekaboo mcp` or in-process PeekabooAutomationKit). Settings picker; exact capability advertisement per provider; no per-call fallback between them. - Windows/Linux via the same bundled plugin: CUA daemon + MCP proxy supervised by the node host; Windows companion PR in `openclaw/openclaw-windows-node`. - Model guidance (skill profile) teaching the background-first ladder: observe window -> background element action -> background pixel -> foreground -> desktop, driven by structured `effect`/refusal results. - Live-tested at every stage; UI changes ship with screenshots/video. ## Non-goals - Raw provider MCP passthrough to the model (rejected in RFC 0025; breaks multi-node caching, arming, and skills). - Reimplementing driver internals in TS/Swift. - Recording/replay + browser-isolation parity for Peekaboo (optional capability families; CUA-only at first). - Wayland-beyond-upstream, Windows elevated/UIAccess targets in v1. ## Decisions (owner rulings, 2026-08-13) 1. **No backward compatibility.** v2 replaces the v1 `computer.act` payload in place under the same command names. Old node + new gateway (or reverse) gets a typed `COMPUTER_CONTRACT_MISMATCH` rejection — a visible outcome, not silent degradation. No dual wire contract, no v1 preservation lane. (Supersedes RFC delivery rule "shipped v1 wire contract remains green".) 2. **Production LOC is a hard constraint.** The provider seam replaces the duplicate per-plugin command registration; the CUA plugin refactor deletes the desktop-scope-only branches it obsoletes; Peekaboo adapter reuses the same mapping core (frame binding, refusal codes, queueing) rather than forking it. Target: net-new production LOC bounded by deletions elsewhere; every wave reports its delta. 3. **Peekaboo is ours to change.** Gaps close upstream in Peekaboo (middle/triple click, hold/mouse-down/up, `get_cursor_position`; browser tool shape may align toward the v2 browser family) instead of adapter warts. 4. **Bundle CUA on macOS** (38 MB universal binary, re-signed with our Developer ID inside OpenClaw.app Resources — upstream's recommended embedding shape; avoids Gatekeeper friction). Windows/Linux use digest-pinned managed download per RFC OC-10A. 5. **Both providers ship together on macOS** with a Settings picker (CUA recommended / Peekaboo). Provider selection is node-local; switch ends the active execution and rotates provider generation. 6. **`invoke_menu` joins the v2 action union** (both providers support it; RFC omitted it). Recording + browser families are optional capability families — a provider without them is first-class, not degraded. ## Architecture ```text Model ── one computer tool (typed v2 actions, capability-filtered) Gateway (cloud OK) ── node.invoke("computer.act") over node protocol Desktop node node-host provider seam (one registration for screen.snapshot + computer.act) ├─ CUA adapter ── cua-driver mcp ──socket── cua-driver serve --embedded │ (macOS: app-spawned, TCC responsibility chain; Win/Linux: node-host │ supervised, interactive session required) └─ Peekaboo adapter ── peekaboo mcp / PeekabooAutomationKit (macOS only) ``` Key invariants (all existing, preserved): frame/observation binding (`displayFrameId` -> v2 `observationId`+`elementRef`, generation-scoped), dangerous command arming, pairing approval, serialized actions, model-only screenshots, cancellation releases held input. TCC: OpenClaw.app probes and holds Accessibility + Screen Recording at startup; app-spawned driver children inherit via the macOS responsibility chain — no new permission UX. Provider capability truth (verified in source, 2026-08-13): - CUA 0.19.3: 49 MCP tools; background delivery per-call with typed refusals (`background_unavailable`, `background_occluded`, `background_uipi_blocked`, `off_space_or_ax_unresolved`); SkyLight per-pid posting on macOS, synthetic pointer injection on Windows, XTest/libei on Linux (Wayland cannot target background windows — capability-gated). Embedded host mode, inherited IPC (#2410, PR 2545) and the protected-consent adapter (#2411, PR 2578) are all contained in v0.13.1+ (verified via tag containment) — the pinned 0.19.3 needs no bump. - Peekaboo v4: 25 MCP tools; background-default clicks (AX-action-first, pid-routed events, window-routed pointer); `see` = screenshot + element map + `reference_id` (maps to observation binding); `verify_state` structured predicates without focus; app/window/menu/dialog/space management; CDP browser tool into user Chrome with explicit connect consent. Missing vs v2: middle/ triple click, held input, cursor position, recording. ## Workstreams and tracker Waves follow RFC 0025's implementation plan, compressed by the no-compat ruling. Status: `todo | in-progress | pr | landed | blocked`. | ID | Work | Repo | Status | Notes | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | W0-FIX | Parity matrix + pinned fixtures: 49 CUA tools + 25 Peekaboo tools classified against the v2 union; recorded tools/list + result fixtures | openclaw | landed | landed #123469 (d9646ad): 58 CUA + 26 Peekaboo tools, +1059 test-only LOC | | W0-PIN | Pick + pin CUA release with embedded/inherited-IPC support; dependency bump (needs Dependency Guard approval) | openclaw | landed | resolved: 0.19.3 already contains #2410+#2411 | | W1-PROTO | v2 types: action union (+`invoke_menu`), target union, capability descriptor, result envelope, closed error codes; **replaces** v1 params in place | openclaw | landed | landed #123544 (d19c755) | | W1-TOOL | Built-in computer tool v2: capability-filtered schema, observation refs, result projection, no auto-retry | openclaw | landed | landed #123544: v1 651 / v2 742 tokens | | W1-SEAM | Node-host provider seam: one registration, provider selection, generation, lifecycle close path; absorbs cua-computer's direct registration | openclaw | landed | landed #123509 (848a7e3): seam+contract, prod +348/-217 | | W2-CUA | CUA plugin refactor onto seam: full v2 mapping (window/element/background), session-per-execution, deletes desktop-scope-only adapter | openclaw | landed | landed #123604 (2af5eca): full v2 adapter, prod +1129/-41 | | W2-MAC | macOS app: bundle + re-sign driver, direct `serve --embedded` spawn, private socket handoff to worker, TCC restart handling | openclaw | landed | landed #123635 (19ace68): app-owned daemon + picker + orphan reaping, live-proven | | W2-PKB | Peekaboo adapter on the same seam (macOS): see/click/type/press/set_value/verify_state/app/window/menu mapping | openclaw | todo | new vs RFC (was v1-only) | | W2-PKB-UP | Peekaboo upstream: middle/triple click, hold_key + mouse down/up in BackgroundInputDriver, get_cursor_position; optional browser-shape alignment | Peekaboo | todo | owner-approved | | W2-UX | Settings -> Computer Use provider picker + readiness checklist (both apps: macOS now, Tauri Linux later) | openclaw | todo | RFC OC-10B slice; owns picker screenshots (W2-MAC shipped without them; app instance lock at `/tmp/openclaw-UID-app-instances` can block a fresh profile launch) | | W3-GATE | Integration gate: live vertical on macOS (both providers) + Linux X11 (CUA): observe window -> background element click -> verify | openclaw | todo | RFC OC-8 | | W3-SKILL | Version-pinned skill profile: background-first ladder, result precedence, no CLI/daemon instructions | openclaw | todo | RFC OC-9D | | W4-BROWSER | CUA browser family (isolated profile first; existing-profile gated on consent adapter) | openclaw | todo | RFC OC-9B; optional family | | W4-REC | CUA recording/resources family with node-owned roots | openclaw | todo | RFC OC-9C; optional family | | W4-WIN | Windows companion CUA host PR | openclaw-windows-node | todo | RFC WIN-1; after W1-SEAM/W2-CUA | | W4-ART | Managed artifacts: Win/Linux digest-pinned download, atomic update + rollback | openclaw | todo | RFC OC-10A | | W5-SEC | Security closure: high-risk action classification, socket ownership audit, hostile-arg tests | openclaw | todo | RFC OC-10C | | W5-ACC | Packaged cross-platform acceptance + default-provider rollout | openclaw | todo | RFC OC-11/12 | Production LOC ledger (updated per landed PR): net target ≤ +1500 for the whole campaign excluding tests/fixtures, funded by deleting the v1-only branches, the duplicate registration path, and the eight-tool adapter. ## Live testing matrix Every wave carries live proof; unit fixtures alone never advance the tracker. - **macOS both providers**: dev gateway (isolated `OPENCLAW_STATE_DIR`, own port) - signed local OpenClaw.app build on one of the owner's Macs. Scenario: background-click + type into a non-frontmost TextEdit window while frontmost app keeps focus; assert frontmost app unchanged, cursor position unchanged, `verify_state`/`effect` confirms the edit. Video via screen recording for UX-visible changes. - **Linux CUA**: Crabbox Xvfb host (recipe proven in PR #117205): node registers command pair, real screenshot + frame id, background window action against xterm/gtk test app; Wayland smoke on Sway only. - **Windows**: openclaw-windows-node companion on a Windows box/VM (Parallels lab); Session 0 rejection probe over SSH. - **Multi-node**: two paired Macs + cloud gateway; explicit `node` selection, frame tokens do not cross nodes, provider generation rotates on app restart. - **Skew probe** (no-compat ruling): old app + new gateway must produce the typed contract-mismatch error in the tool result, not silence. ## Risks - CUA prerelease churn: everything pins to one accepted release + fixtures; version skew refuses closed. - v2 union token cost: measured in W1-TOOL; split into tool family only if a provider cannot consume one discriminated union reliably. - Private-socket embedding: owner-only dir, 0600, ownership validated, pre-existing path rejected; upgrade to inherited IPC when the pinned release supports it. - Dependency Guard approvals (CUA bump, new Peekaboo dep surface) interrupt autonomy; batch them per wave. - Collision with RFC author's in-flight work: check live PRs before each wave.