Computer use plan

Status

Active campaign, started 2026-08-13. Owner-directed (steipete): build autonomously end to end, live-test everything, ship fully. Tracker table below is the source of truth and is updated as work lands. Derived from RFC 0025 (openclaw/rfcs rfcs/0025-default-pluggable-computer-use.md, RomneyDa) with the owner rulings recorded under Decisions. Coordinate with RFC author before each wave lands (check live PRs to avoid collisions with in-flight maintainer work).

Problem

OpenClaw's computer use is foreground-only everywhere: the macOS fulfiller drives the shared cursor via embedded Peekaboo + CGEvent primitives, and the cua-computer plugin uses only CUA's global desktop scope (scope:"desktop", no delivery_mode). Both CUA and Peekaboo now have real background window-scoped input (no cursor move, no focus steal, no Space switch), semantic observation (AX tree + screenshot + element refs), and structured verification. None of that reaches the model. The current CUA adapter maps 8 of 49 driver tools; Peekaboo's MCP surface (25 tools, background-default clicks, see frame binding, verify_state) is not reachable through computer.act at all.

Goals

• One typed computer.act v2 contract (portable actions: window/element/browser targets, deliveryMode: background|foreground, verification envelope) carried over the existing node protocol. Cloud gateway + many paired desktop nodes works unchanged — provider processes live on the node.
• Two first-class macOS providers behind one adapter seam in the app-owned TypeScript node worker: CUA (cua-driver serve --embedded, app-spawned, TCC via responsibility chain) and Peekaboo (peekaboo mcp or in-process PeekabooAutomationKit). Settings picker; exact capability advertisement per provider; no per-call fallback between them.
• Windows/Linux via the same bundled plugin: CUA daemon + MCP proxy supervised by the node host; Windows companion PR in openclaw/openclaw-windows-node.
• Model guidance (skill profile) teaching the background-first ladder: observe window -> background element action -> background pixel -> foreground -> desktop, driven by structured effect/refusal results.
• Live-tested at every stage; UI changes ship with screenshots/video.

Non-goals

• Raw provider MCP passthrough to the model (rejected in RFC 0025; breaks multi-node caching, arming, and skills).
• Reimplementing driver internals in TS/Swift.
• Recording/replay + browser-isolation parity for Peekaboo (optional capability families; CUA-only at first).
• Wayland-beyond-upstream, Windows elevated/UIAccess targets in v1.

Decisions (owner rulings, 2026-08-13)

1. No backward compatibility. v2 replaces the v1 computer.act payload in place under the same command names. Old node + new gateway (or reverse) gets a typed COMPUTER_CONTRACT_MISMATCH rejection — a visible outcome, not silent degradation. No dual wire contract, no v1 preservation lane. (Supersedes RFC delivery rule "shipped v1 wire contract remains green".)
2. Production LOC is a hard constraint. The provider seam replaces the duplicate per-plugin command registration; the CUA plugin refactor deletes the desktop-scope-only branches it obsoletes; Peekaboo adapter reuses the same mapping core (frame binding, refusal codes, queueing) rather than forking it. Target: net-new production LOC bounded by deletions elsewhere; every wave reports its delta.
3. Peekaboo is ours to change. Gaps close upstream in Peekaboo (middle/triple click, hold/mouse-down/up, get_cursor_position; browser tool shape may align toward the v2 browser family) instead of adapter warts.
4. Bundle CUA on macOS (38 MB universal binary, re-signed with our Developer ID inside OpenClaw.app Resources — upstream's recommended embedding shape; avoids Gatekeeper friction). Windows/Linux use digest-pinned managed download per RFC OC-10A.
5. Both providers ship together on macOS with a Settings picker (CUA recommended / Peekaboo). Provider selection is node-local; switch ends the active execution and rotates provider generation.
6. invoke_menu joins the v2 action union (both providers support it; RFC omitted it). Recording + browser families are optional capability families — a provider without them is first-class, not degraded.

Architecture

Model ── one computer tool (typed v2 actions, capability-filtered)  Gateway (cloud OK) ── node.invoke("computer.act") over node protocol    Desktop node      node-host provider seam (one registration for screen.snapshot + computer.act)        ├─ CUA adapter ── cua-driver mcp ──socket── cua-driver serve --embedded        │    (macOS: app-spawned, TCC responsibility chain; Win/Linux: node-host        │     supervised, interactive session required)        └─ Peekaboo adapter ── peekaboo mcp / PeekabooAutomationKit (macOS only)

Key invariants (all existing, preserved): frame/observation binding (displayFrameId -> v2 observationId+elementRef, generation-scoped), dangerous command arming, pairing approval, serialized actions, model-only screenshots, cancellation releases held input. TCC: OpenClaw.app probes and holds Accessibility + Screen Recording at startup; app-spawned driver children inherit via the macOS responsibility chain — no new permission UX.

Provider capability truth (verified in source, 2026-08-13):

• CUA 0.19.3: 49 MCP tools; background delivery per-call with typed refusals (background_unavailable, background_occluded, background_uipi_blocked, off_space_or_ax_unresolved); SkyLight per-pid posting on macOS, synthetic pointer injection on Windows, XTest/libei on Linux (Wayland cannot target background windows — capability-gated). Embedded host mode, inherited IPC (#2410, PR 2545) and the protected-consent adapter (#2411, PR 2578) are all contained in v0.13.1+ (verified via tag containment) — the pinned 0.19.3 needs no bump.
• Peekaboo v4: 25 MCP tools; background-default clicks (AX-action-first, pid-routed events, window-routed pointer); see = screenshot + element map + reference_id (maps to observation binding); verify_state structured predicates without focus; app/window/menu/dialog/space management; CDP browser tool into user Chrome with explicit connect consent. Missing vs v2: middle/ triple click, held input, cursor position, recording.

Workstreams and tracker

Waves follow RFC 0025's implementation plan, compressed by the no-compat ruling. Status: todo | in-progress | pr | landed | blocked.

Production LOC ledger (updated per landed PR): net target ≤ +1500 for the whole campaign excluding tests/fixtures, funded by deleting the v1-only branches, the duplicate registration path, and the eight-tool adapter.

Live testing matrix

Every wave carries live proof; unit fixtures alone never advance the tracker.

• macOS both providers: dev gateway (isolated OPENCLAW_STATE_DIR, own port)
  • signed local OpenClaw.app build on one of the owner's Macs. Scenario: background-click + type into a non-frontmost TextEdit window while frontmost app keeps focus; assert frontmost app unchanged, cursor position unchanged, verify_state/effect confirms the edit. Video via screen recording for UX-visible changes.
• Linux CUA: Crabbox Xvfb host (recipe proven in PR #117205): node registers command pair, real screenshot + frame id, background window action against xterm/gtk test app; Wayland smoke on Sway only.
• Windows: openclaw-windows-node companion on a Windows box/VM (Parallels lab); Session 0 rejection probe over SSH.
• Multi-node: two paired Macs + cloud gateway; explicit node selection, frame tokens do not cross nodes, provider generation rotates on app restart.
• Skew probe (no-compat ruling): old app + new gateway must produce the typed contract-mismatch error in the tool result, not silence.

Risks

• CUA prerelease churn: everything pins to one accepted release + fixtures; version skew refuses closed.
• v2 union token cost: measured in W1-TOOL; split into tool family only if a provider cannot consume one discriminated union reliably.
• Private-socket embedding: owner-only dir, 0600, ownership validated, pre-existing path rejected; upgrade to inherited IPC when the pinned release supports it.
• Dependency Guard approvals (CUA bump, new Peekaboo dep surface) interrupt autonomy; batch them per wave.
• Collision with RFC author's in-flight work: check live PRs before each wave.