QA reporting

Reporting

qa-lab exports a Markdown protocol report from the observed bus timeline. The report should answer:

• What worked
• What failed
• What stayed blocked
• What follow-up scenarios are worth adding

For the inventory of available scenarios - useful when sizing follow-up work or wiring a new transport - run pnpm openclaw qa coverage (add --json for machine-readable output). When choosing focused proof for a touched behavior or file path, run pnpm openclaw qa coverage --match <query>. The match report searches scenario metadata, docs refs, code refs, coverage IDs, plugins, and provider requirements, then prints matching qa suite --scenario ... targets. Generated commands preserve declared channel-driver requirements and separate scenarios with different driver requirements. Without a driver requirement, non-QA channels use live and qa-channel keeps its default driver.

Every qa suite run writes top-level qa-evidence.json, qa-suite-summary.json, and qa-suite-report.md artifacts for the selected scenario set. Scenarios that declare execution.kind: vitest or execution.kind: playwright run the matching test path and also write per-scenario logs. Scenarios that declare execution.kind: script run the evidence producer at execution.path through node --import tsx (with ${outputDir} and ${scenarioId} expanded in execution.args); the producer writes its own qa-evidence.json, whose entries are imported into the suite output and whose artifact paths are resolved relative to that producer qa-evidence.json. When qa suite is reached through qa run --qa-profile, the same qa-evidence.json also includes the profile scorecard summary for the selected taxonomy categories.

qa confidence-report keeps productImpact and qaImpact annotations in their own Markdown table cells, collapsing whitespace for display. The JSON summary preserves the annotation values, including internal line breaks.

Treat coverage output as a discovery aid, not a gate replacement; the selected scenario still needs the right provider mode, live transport, Multipass, Testbox, or release lane for the behavior under test. For scorecard context, see Maturity scorecard.

For character and style checks, run the same scenario across multiple live model refs and write a judged Markdown report:

pnpm openclaw qa character-eval \  --model openai/gpt-5.6-luna,thinking=medium,fast \  --model openai/gpt-5.2,thinking=xhigh \  --model openai/gpt-5,thinking=xhigh \  --model anthropic/claude-opus-4-8,thinking=high \  --model anthropic/claude-sonnet-4-6,thinking=high \  --model zai/glm-5.1,thinking=high \  --model moonshot/kimi-k2.5,thinking=high \  --model google/gemini-3.1-pro-preview,thinking=high \  --judge-model openai/gpt-5.6-sol,thinking=xhigh,fast \  --judge-model anthropic/claude-opus-4-8,thinking=high \  --blind-judge-models \  --concurrency 16 \  --judge-concurrency 16

The command runs local QA gateway child processes, not Docker. Character eval scenarios should set the persona through SOUL.md, then run ordinary user turns such as chat, workspace help, and small file tasks. The candidate model should not be told that it is being evaluated. The command preserves each full transcript, records basic run stats, then asks the judge models in fast mode with xhigh reasoning where supported to rank the runs by naturalness, vibe, and humor. Use --blind-judge-models when comparing providers: the judge prompt still gets every transcript and run status, but candidate refs are replaced with neutral labels such as candidate-01; the report maps rankings back to real refs after parsing.

Candidate runs default to high thinking, with medium for GPT-5.6 Luna and xhigh for older OpenAI eval refs that support it. Override a specific candidate inline with --model provider/model,thinking=<level>; inline options also support fast, no-fast, and fast=<bool>. --thinking <level> still sets a global fallback, and the older --model-thinking <provider/model=level> form is kept for compatibility. OpenAI candidate refs default to fast mode so priority processing is used where the provider supports it. Pass --fast only when you want to force fast mode on for every candidate model. Candidate and judge durations are recorded in the report for benchmark analysis, but judge prompts explicitly say not to rank by speed. Candidate and judge model runs both default to concurrency 16. Lower --concurrency or --judge-concurrency when provider limits or local gateway pressure make a run too noisy.

When no candidate --model is passed, the character eval defaults to openai/gpt-5.6-luna, openai/gpt-5.2, openai/gpt-5, anthropic/claude-opus-4-8, anthropic/claude-sonnet-4-6, zai/glm-5.1, moonshot/kimi-k2.5, and google/gemini-3.1-pro-preview. When no --judge-model is passed, the judges default to openai/gpt-5.6-sol,thinking=xhigh,fast and anthropic/claude-opus-4-8,thinking=high.