OpenClawDocs
On this page

On this page

View as MarkdownView this page as plain textOpen in ChatGPTAsk questions about this pageOpen in ClaudeAsk questions about this page

Testing and CI

Live model smoke (profile keys)

Live: model smoke (profile keys)

Live model tests are split into two layers so failures are isolated:

  • "Direct model" tells you whether the provider/model can answer at all with the given key.
  • "Gateway smoke" tells you whether the full gateway+agent pipeline works for that model (sessions, history, tools, sandbox policy, etc.).

The curated model lists on Live: model matrix live in src/agents/test-helpers/live-model-dynamic-candidates.ts and change over time; treat the arrays there as the source of truth, not this page.

MiniMax M3 uses minimax/MiniMax-M3 as its default provider/model reference.

Explicit provider/model selections use the agent's shared model resolver, including configured models and accepted bundled aliases. Ambiguous model-only selectors continue to enumerate matching discovered models. Invalid selectors and requested models that cannot be resolved are reported before checks run. Provider allowlists still narrow the selection, and intentional model caps still limit how many candidates run. Resolving every applicable selector proves availability, not inference for models omitted by a cap.

Layer 1: Direct model completion (no gateway)

  • Test: src/agents/models.profiles.live.test.ts
  • Goal:
    • Enumerate discovered models
    • Use getApiKeyForModel to select models you have creds for
    • Run a small completion per model (and targeted regressions where needed)
  • How to enable:
    • pnpm test:live (or OPENCLAW_LIVE_TEST=1 if invoking Vitest directly)
    • Set OPENCLAW_LIVE_MODELS=modern, small, or all (alias for modern) to actually run this suite; otherwise it skips, so pnpm test:live on its own stays focused on gateway smoke.
  • How to select models:
    • OPENCLAW_LIVE_MODELS=modern runs the curated high-signal priority list (see Live: model matrix)
    • OPENCLAW_LIVE_MODELS=small runs the curated small-model priority list
    • OPENCLAW_LIVE_MODELS=all is an alias for modern
    • or OPENCLAW_LIVE_MODELS="openai/gpt-5.6-luna,anthropic/claude-opus-4-6,..." (comma allowlist)
    • Local Ollama small-model runs default to http://127.0.0.1:11434; set OPENCLAW_LIVE_OLLAMA_BASE_URL only for LAN, custom, or Ollama Cloud endpoints.
    • Modern/all and small sweeps default to their curated-list length as a cap; set OPENCLAW_LIVE_MAX_MODELS=0 for an exhaustive selected-profile sweep or a positive number for a smaller cap.
    • Exhaustive sweeps use OPENCLAW_LIVE_TEST_TIMEOUT_MS for the whole direct-model test timeout. Default: 60 minutes.
    • Direct-model checks run with 20-way parallelism by default; set OPENCLAW_LIVE_MODEL_CONCURRENCY to override.
  • How to select providers:
    • OPENCLAW_LIVE_PROVIDERS="google,google-gemini-cli" (comma allowlist)
  • Where keys come from:
    • By default: profile store and env fallbacks
    • Set OPENCLAW_LIVE_REQUIRE_PROFILE_KEYS=1 to enforce profile store only
  • Why this exists:
    • Separates "provider API is broken / key is invalid" from "gateway agent pipeline is broken"
    • Contains small, isolated regressions (example: OpenAI Responses/Codex Responses reasoning replay + tool-call flows)

Layer 2: Gateway + dev agent smoke (what "@openclaw" actually does)

  • Test: src/gateway/gateway-models.profiles.live.test.ts
  • Goal:
    • Spin up an in-process gateway
    • Create/patch an agent:dev:* session (model override per run)
    • Iterate models-with-keys and assert:
      • "meaningful" response (no tools)
      • a real tool invocation works (read check)
      • optional extra tool checks (exec+read check)
      • OpenAI regression paths (tool-call-only -> follow-up) keep working
  • Check details (so you can explain failures quickly):
    • read check: the test writes a nonce file in the workspace and asks the agent to read it and echo the nonce back.
    • exec+read check: the test asks the agent to exec-write a nonce into a temp file, then read it back.
    • image check: the test attaches a generated PNG (cat + randomized code) and expects the model to return cat <CODE>.
    • Implementation reference: src/gateway/gateway-models.profiles.live.test.ts and test/helpers/live-image-probe.ts.
  • How to enable:
    • pnpm test:live (or OPENCLAW_LIVE_TEST=1 if invoking Vitest directly)
  • How to select models:
    • Default: the curated high-signal (modern) priority list
    • OPENCLAW_LIVE_GATEWAY_MODELS=small runs the curated small-model list through the full gateway+agent pipeline
    • OPENCLAW_LIVE_GATEWAY_MODELS=all is an alias for modern
    • Or set OPENCLAW_LIVE_GATEWAY_MODELS="provider/model" (or comma list) to narrow
    • Modern/all and small gateway sweeps default to their curated-list length as a cap; set OPENCLAW_LIVE_GATEWAY_MAX_MODELS=0 for an exhaustive selected sweep or a positive number for a smaller cap.
  • How to select providers (avoid "OpenRouter everything"):
    • OPENCLAW_LIVE_GATEWAY_PROVIDERS="google,google-gemini-cli,openai,anthropic,zai,minimax" (comma allowlist)
  • Tool + image checks are always on in this live test:
    • read check + exec+read check (tool stress)
    • image check runs when the model advertises image input support
    • Flow (high level):
      • Test generates a tiny PNG with "CAT" + random code (test/helpers/live-image-probe.ts)
      • Sends it via agent attachments: [{ mimeType: "image/png", content: "<base64>" }]
      • Gateway parses attachments into images[] (src/gateway/server-methods/agent.ts + src/gateway/chat-attachments.ts)
      • Embedded agent forwards a multimodal user message to the model
      • Assertion: reply contains cat + the code (OCR tolerance: minor mistakes allowed)
Was this useful?