Writing and adding tests

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

import { afterEach } from "vitest";import { useAutoCleanupTempDirTracker } from "../helpers/temp-dir.js"; const tempDirs = useAutoCleanupTempDirTracker(afterEach); it("uses a temp workspace", () => {  const workspace = tempDirs.make("openclaw-example-");  // use workspace});

useAutoCleanupTempDirTracker(afterEach) intentionally exposes no manual cleanup method - Vitest owns cleanup after each test. Older lower-level helpers (makeTempDir, cleanupTempDirs, createTempDirTracker) still exist for tests that have not migrated; avoid new usage of them and avoid new bare fs.mkdtemp* calls unless a test is explicitly verifying raw temp-dir behavior. When a bare temp dir is genuinely needed, add an auditable allow comment with a reason:

// openclaw-temp-dir: allow verifies raw fs cleanup behaviorconst workspace = fs.mkdtempSync(prefix);

node scripts/report-test-temp-creations.mjs reports new bare temp-dir creation and new manual shared-helper usage in added diff lines, without blocking existing cleanup styles. It follows the same test-path classification as scripts/changed-lanes.mjs and skips the shared helper implementation itself. check:changed runs this report for changed test paths as a warning-only CI signal (GitHub warning annotations, not failures).

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like "agent reliability evals":

• Agent admission, run-ID responses, and abort requests through the real Gateway with a mock OpenAI provider (src/gateway/gateway.test.ts).
• End-to-end wizard flows that validate session wiring and config effects (src/gateway/gateway.test.ts).

What's still missing for skills (see Skills):

• Decisioning: when skills are listed in the prompt, does the agent pick the right skill (or avoid irrelevant ones)?
• Compliance: does the agent read SKILL.md before use and follow required steps/args?
• Workflow contracts: multi-turn scenarios that assert tool order, session history carryover, and sandbox boundaries.

Future evals should stay deterministic first:

• A scenario runner using mock providers to assert tool calls + order, skill file reads, and session wiring.
• A small suite of skill-focused scenarios (use vs avoid, gating, prompt injection).
• Optional live evals (opt-in, env-gated) only after the CI-safe suite is in place.

Adding regressions (guidance)

When you fix a provider/model issue discovered in live:

• Add a CI-safe regression if possible (mock/stub provider, or capture the exact request-shape transformation)
• If it's inherently live-only (rate limits, auth policies), keep the live test narrow and opt-in via env vars
• Prefer targeting the smallest layer that catches the bug:
  • provider request conversion/replay bug -> direct models test
  • gateway session/history/tool pipeline bug -> gateway live smoke or CI-safe gateway mock test
• SecretRef traversal guardrail:
  • src/secrets/exec-secret-ref-id-parity.test.ts derives one sampled target per SecretRef class from registry metadata (listSecretTargetRegistryEntries()), then asserts traversal-segment exec ids are rejected.
  • If you add a new includeInPlan SecretRef target family in src/secrets/target-registry-data.ts, update classifyTargetClass in that test. The test intentionally fails on unclassified target ids so new classes cannot be skipped silently.