Extending the QA stack

Repo-backed seeds

Seed assets live in qa/:

• qa/scenarios/index.yaml
• qa/scenarios/<theme>/*.yaml

Identity-sensitive channel changes use the isolated channel-participant-identity-inspection QA Channel flow. It drives a real ephemeral Gateway and mock provider, then inspects admitted runs with the same openclaw audit --run ... --explain JSON and human surfaces operators use. The flow includes lifecycle-owned restart and a row-count check for rejected pre-run ingress.

These are intentionally in git so the QA plan is visible to both humans and the agent.

qa-lab stays a generic YAML scenario runner. Each scenario YAML file is the source of truth for one test run and should define:

• top-level title
• scenario metadata
• optional category, capability, lane, and risk metadata in scenario
• docs and code refs in scenario
• optional plugin requirements in scenario
• optional gateway config patch in scenario
• executable top-level flow for flow scenarios, or scenario.execution.kind / scenario.execution.path for Vitest and Playwright scenarios

The reusable runtime surface that backs flow stays generic and cross-cutting. For example, YAML scenarios can combine transport-side helpers with browser-side helpers that drive the embedded Control UI through the Gateway browser.request interface without adding a special-case runner.

Scenario files should be grouped by product capability rather than source tree folder. Keep scenario IDs stable when files move; use docsRefs and codeRefs for implementation traceability.

The baseline list should stay broad enough to cover:

• DM and channel chat
• thread behavior
• message action lifecycle
• cron callbacks
• memory recall
• model switching
• subagent handoff
• repo-reading and docs-reading
• one small build task such as Lobster Invaders

Native tool discovery proof

A flow can declare a model-specific configured runtime route:

execution:  kind: flow  liveConfiguredRuntime:    id: codex    model: openai/gpt-5.5

This route applies only in live-frontier when both --model and --alt-model select the declared model. The launcher prepares singleton and partitioned runs through the same runtime route. Other selections keep their general model route; explicit forced runtimes and runtime-pair cells retain their own preparation. The metadata does not override either model choice or product tool loading.

The native discovery lane selects openai/gpt-5.5 for both model slots. Its Codex catalog supports native tool search without a code_mode_only override. Provider identity alone does not establish that capability. Shared searchable fixtures require linked tool_search receipts only in Codex execution cells; direct OpenClaw cells still prove tool calls and results. sessions_spawn remains the always-direct control and does not require discovery receipts.

Provider mock lanes

qa suite has two local provider mock lanes:

• mock-openai is the scenario-aware OpenClaw mock. It remains the default fixed-response mock lane for repo-backed QA and parity gates.
• aimock starts an AIMock-backed provider server for experimental protocol, fixture, record/replay, and chaos coverage. It is additive and does not replace the mock-openai scenario dispatcher.

The scenario-aware mock answers Activity recap requests separately from agent turns. Recaps quote conversation text as data, so they cannot trigger scenario tools or enter the scenario request evidence returned by /debug/requests.

For an IPv6 loopback server, run pnpm openclaw qa mock-openai --host ::1. The printed URL includes brackets, such as http://[::1]:<port>; use that URL when configuring a client. QA Lab also brackets IPv6 hosts in its listen and advertised URLs. Pass the bare address to --host.

Provider-lane implementation lives under extensions/qa-lab/src/providers/. Each provider owns its defaults, local server startup, gateway model config, auth-profile staging needs, and live/mock capability flags. Shared suite and gateway code routes through the provider registry instead of branching on provider names.

Transport adapters

qa-lab owns a generic transport interface for YAML QA scenarios. qa-channel is the synthetic default. crabline starts separate local provider servers and runs OpenClaw's normal channel plugins against their provider-shaped REST and streaming boundaries; it does not use Crabline's fixture-level local mock providers. live is reserved for real provider credentials and external channels.

At the architecture level, the split is:

• qa-lab owns generic scenario execution, worker concurrency, artifact writing, and reporting.
• The transport adapter owns gateway config, readiness, inbound and outbound observation, transport actions, and normalized transport state.
• YAML scenario files under qa/scenarios/ define the test run; qa-lab provides the reusable runtime surface that executes them.

Adapter shutdown and failure hooks

The optional QA runner hooks follow one host-owned teardown order:

1. cleanup() stops new fixture actions while retaining credential authority and ownership of pending writes. Observers needed for final receipts can remain active through Gateway shutdown. Bound transport requests themselves; a scenario deadline does not settle an already-dispatched request.
2. The host stops the Gateway and confirms process shutdown.
3. captureBeforeGatewayCleanup() snapshots final native receipts before temporary Gateway state is removed. A successful capture runs once per Gateway lifetime. A thrown error retains the runtime evidence and fails teardown; it does not prevent post-stop fixture cleanup.
4. cleanupAfterGatewayStop() settles pending writes and owned fixture cleanup, then releases the credential lease. The host withholds this hook if Gateway shutdown is unconfirmed. Report failures rather than claiming successful cleanup.

Omitted hooks perform no adapter-specific work; normal host teardown still runs. Errors are accumulated across cleanup phases. Keep unknown remote write outcomes explicit instead of inferring ownership or replaying writes.

An optional whenUnhealthy promise resolves, rather than rejects, with a terminal error. The host aborts active flow admission; the adapter still owns settlement during cleanup. Adapters that omit it retain their explicit health checks and scenario deadlines.

Adding a channel

Adding a channel to the YAML QA system requires the channel implementation plus a scenario pack that exercises the channel contract. For smoke CI coverage, add the matching Crabline local provider server and expose it through the crabline driver.

Do not add a new top-level QA command root when the shared qa-lab host can own the flow.

qa-lab owns the shared host mechanics:

• the openclaw qa command root
• suite startup and teardown
• worker concurrency
• artifact writing
• report generation
• scenario execution
• compatibility aliases for older qa-channel scenarios

Runner plugins own the transport contract:

• how openclaw qa <runner> is mounted beneath the shared qa root
• how the gateway is configured for that transport
• how readiness is checked
• how inbound events are injected
• how outbound messages are observed
• how transcripts and normalized transport state are exposed
• how transport-backed actions are executed
• which trusted module preloads the Gateway runtime needs
• how transport-specific reset or cleanup is handled

The minimum adoption bar for a new channel:

1. Keep qa-lab as the owner of the shared qa root.
2. Implement the transport runner on the shared qa-lab host interface.
3. Keep transport-specific mechanics inside the runner plugin or channel harness.
4. Mount the runner as openclaw qa <runner> instead of registering a competing root command. Runner plugins should declare qaRunners in openclaw.plugin.json and export a matching qaRunnerCliRegistrations array from a lightweight qa-runner-api.ts surface. Installed plugins using the shipped runtime-api.ts contract remain supported through 2026-10-01 while authors migrate. Keep runner execution behind lazy entrypoints. An optional adapterFactory exposes the transport to shared scenarios without changing the command's existing scenario catalog. Same-channel partitions are serial unless the factory declares that every instance owns isolated credentials or disposable servers, Gateway state, and artifact paths. Module-backed flow scenarios additionally require adapterFactory.supportsModuleFlows: true; those factories must return adapters that implement prepareFlow.
5. Author or adapt YAML scenarios under the themed qa/scenarios/ directories.
6. Use the generic scenario helpers for new scenarios.
7. Keep existing compatibility aliases working unless the repo is doing an intentional migration.

The decision rule is strict:

• If behavior can be expressed once in qa-lab, put it in qa-lab.
• If behavior depends on one channel transport, keep it in that runner plugin or plugin harness.
• If a scenario needs a new capability that more than one channel can use, add a generic helper instead of a channel-specific branch in suite.ts.
• If a behavior is only meaningful for one transport, keep the scenario transport-specific and make that explicit in the scenario contract.

Scenario helper names

Preferred generic helpers for new scenarios:

• waitForTransportReady
• waitForChannelReady
• injectInboundMessage
• injectOutboundMessage
• waitForOutboundMessage
• waitForNoTransportOutbound
• getTransportSnapshot
• readTransportMessage
• readTransportTranscript
• formatTransportTranscript
• resetTransport

Compatibility aliases remain available for existing scenarios - waitForQaChannelReady, waitForNoOutbound, formatConversationTranscript, and resetBus - but new scenario authoring should use the generic names. Use the shared waitForOutboundMessage for outbound checks instead of adding transport- or channel-specific outbound wait aliases.