Web fetch

web_fetch does a plain HTTP GET and extracts readable content (HTML to markdown or text). It does not execute JavaScript. For JS-heavy sites or login-protected pages, use the Web Browser instead.

Basic HTML extraction preserves boundaries around block content such as quotes, figure captions, and collapsible sections.

HTML cleanup excludes text hidden by recognized inline styles, including scale transforms that collapse the X or Y axis, such as scale(0, 0) and scale3d(0, 0, 0). Nonzero scales and a zero Z axis alone remain visible. Basic HTML extraction keeps definition-list terms and values on separate lines.

Quick start

Enabled by default, no configuration needed:

await web_fetch({ url: "https://example.com/article" });

Tool parameters

Result

web_fetch returns a closed structured result with these fields:

• Request metadata: url, finalUrl, status, extractMode, and extractor
• Optional response metadata: contentType, title, and warning (omitted when absent)
• Wrapped content metadata: externalContent, truncated, length, rawLength, fetchedAt, tookMs, and text
• Optional cached: true on a cache hit
• Optional spill: { path, chars, truncated? } when truncated content was written to a private temporary file; truncated is present only when that file contains partial source content

length is the wrapped text length. rawLength is the extracted content length before external-content wrapping.

text, title, and warning share maxChars, including their content wrappers. Titles and warnings retain at most 256 characters each, with a combined wrapped allowance of at most 512 characters and half of maxChars. Warnings take priority over titles; unused metadata space stays available to the body. truncated is also true when metadata is shortened or omitted; metadata-only truncation does not create a body spill file.

JSON framing and protocol metadata are additional overhead. contentType, extractor, and fetchedAt are capped at 256, 128, and 64 characters respectively. URLs stay whole for follow-up requests: a returned redirect/provider URL longer than 2,048 characters falls back to the requested url and marks the result truncated. The caller's requested URL is preserved.

How it works

Set tools.web.fetch.cacheTtlMinutes: 0 to bypass OpenClaw's fetch cache for both reads and writes. A positive value limits reuse by the current request's TTL; cached entries still expire at their original deadline. Provider-side caching, such as Firecrawl's maxAgeMs, is configured separately.

Response decoding honors a byte-order mark first, then the HTTP Content-Type charset, then an XML declaration or HTML charset metadata. Without a supported charset, it falls back to UTF-8. ASCII-compatible documents that declare UTF-16 in their XML or HTML metadata are decoded as UTF-8 instead; correctly declared legacy encodings such as Windows-1252 remain supported.

Progress updates

web_fetch emits a public progress line only when the fetch is still pending after five seconds:

Fetching page content...

Fast cache hits and quick network responses finish before the timer fires, so they never show a progress line. Canceling the call clears the timer. The progress line is channel UI state only and never contains fetched page content.

OpenClaw passes cancellation to fallback providers. Providers that honor the signal can stop their requests; core rejects late results even when a provider ignores cancellation. Already-canceled calls reject even when a cached result exists. If cancellation occurs during fetching, fallback processing, or connection cleanup, OpenClaw rejects the call instead of returning success or adding a result to the fetch cache.

Config

{  tools: {    web: {      fetch: {        enabled: true, // default: true        provider: "firecrawl", // optional; omit for auto-detect        maxChars: 20000, // default output chars; capped by maxCharsCap        maxCharsCap: 20000, // hard cap for maxChars param        maxResponseBytes: 750000, // max download size before truncation (32000-10000000)        timeoutSeconds: 30,        cacheTtlMinutes: 15,        maxRedirects: 3,        useTrustedEnvProxy: false, // let a trusted HTTP(S) env proxy resolve DNS        readability: true, // use Readability extraction        userAgent: "Mozilla/5.0 ...", // override User-Agent        headers: {          // optional; every value is treated as sensitive          "X-Routing-Target": "staging",        },        ssrfPolicy: {          dangerouslyAllowPrivateNetwork: false, // broad private-network opt-in; keep false by default          allowedHostnames: ["internal.example"], // narrow exact host exception          allowRfc2544BenchmarkRange: true, // opt-in for trusted fake-IP proxies using 198.18.0.0/15          allowIpv6UniqueLocalRange: true, // opt-in for trusted fake-IP proxies using fc00::/7        },      },    },  },}

Firecrawl fallback

If Readability extraction fails, web_fetch can fall back to Firecrawl for bot-circumvention and better extraction:

{  tools: {    web: {      fetch: {        provider: "firecrawl", // optional; omit for auto-detect from available credentials      },    },  },  plugins: {    entries: {      firecrawl: {        enabled: true,        config: {          webFetch: {            // apiKey: "fc-...", // optional; omit for keyless starter access            baseUrl: "https://api.firecrawl.dev",            onlyMainContent: true,            maxAgeMs: 172800000, // cache duration (2 days)            timeoutSeconds: 60,          },        },      },    },  },}

plugins.entries.firecrawl.config.webFetch.apiKey is optional and supports SecretRef objects. Legacy tools.web.fetch.firecrawl.* config auto-migrates to plugins.entries.firecrawl.config.webFetch via openclaw doctor --fix.

Current runtime behavior:

• tools.web.fetch.provider selects the fetch fallback provider explicitly.
• If provider is omitted, OpenClaw auto-detects the first ready web-fetch provider from configured credentials. Non-sandboxed web_fetch can use installed plugins that declare contracts.webFetchProviders and register a matching provider at runtime. The official Firecrawl plugin provides this fallback today.
• Sandboxed web_fetch calls allow bundled providers plus installed providers whose official npm or ClawHub provenance is verified. Today that permits the official Firecrawl plugin; third-party external fetch plugins stay excluded.
• If Readability is disabled, web_fetch skips straight to the selected provider fallback. If no provider is available, the request fails.

Custom request headers

Set tools.web.fetch.headers when your deployment needs extra request metadata on outbound fetches, such as a routing or service-injection header that steers traffic to a gateway you control.

{  tools: {    web: {      fetch: {        headers: {          "X-Routing-Target": "${WEB_FETCH_ROUTING_TARGET}",        },      },    },  },}

Behavior worth knowing:

• Values are plain strings and support ${VAR} environment substitution like any other config string. Structured SecretRef values are not accepted.
• Headers apply only to the direct web_fetch request. Provider fallbacks such as Firecrawl call their own API and never receive these headers.
• Entries are validated when the request is built, not at config load, so one bad entry is dropped while the rest still apply. Config load stays permissive on purpose: rejecting the config over a single header-name typo would disable the whole surface. Every dropped entry is logged by name.
• Dropped names:
  • Accept, Accept-Language, and User-Agent belong to the fetch and readability contract. Use tools.web.fetch.userAgent for the user agent.
  • Framing and hop-by-hop names such as Content-Length, Transfer-Encoding, Connection, and Upgrade, which a request either rejects outright or ignores.
  • Names that are not valid HTTP tokens, such as "X Routing Target".
• Dropped values: bytes a request cannot carry (CR, LF, NUL, or any character above U+00FF). Missing environment variables are reported by config loading; the global $${VAR} escape remains available when the literal ${VAR} text is intentional.
• Two entries whose names differ only in case collapse to the later entry, so a request never carries a comma-joined value the receiving gateway cannot parse. The dropped name is logged without either value. If the later entry is unusable, neither value is sent.
• Rejection happens before the cache key is computed, so the key always matches the bytes actually sent: changing a header that is really sent partitions the fetch cache, while adding one that gets dropped does not.
• When a redirect crosses origins, the guarded-fetch safe allowlist is applied. Routing headers outside that list are dropped; standard safe headers such as Cache-Control, Content-Type, and Range are preserved.

Trusted env proxy

If your deployment requires web_fetch to go through a trusted outbound HTTP(S) proxy, set tools.web.fetch.useTrustedEnvProxy: true.

In this mode, OpenClaw still applies hostname-based SSRF checks before sending the request, but it lets the proxy resolve DNS instead of doing local DNS pinning. Enable this only when the proxy is operator-controlled and enforces outbound policy after DNS resolution.

Saved output

When fetched content exceeds the inline limit, web_fetch includes a spill.path and a Full output location. Use read with that path and bounded offset/limit arguments to retrieve the omitted content. Saved output retains the untrusted-content warning and is capped at 2,000,000 characters; spill.truncated indicates an incomplete saved response.

Local Docker and Podman sandboxes save this output in their existing private input-staging area, accessible through the sandbox reader. Cached saved paths remain scoped to the requesting session and sandbox workspace. Recovery also works when guest workspace access is read-only: the trusted input-staging owner prepares the file without granting guest write access.

Other backends, including remote-shell sandboxes, need a prepared input-staging capability before saved output is available. Without it, the inline result reports that full output is unavailable, makes no staging write, and advertises no host-only path. An unsuccessful save is not cached.

Limits and safety

• maxChars is clamped to tools.web.fetch.maxCharsCap (default 20000)
• Response body is capped at maxResponseBytes (default 750000, clamped to 32000-10000000) before parsing; oversized responses are truncated with a warning
• Private/internal hostnames are blocked
• tools.web.fetch.ssrfPolicy.allowedHostnames allows exact trusted hosts while leaving other private/internal targets blocked
• tools.web.fetch.ssrfPolicy.blockedHostnames denies exact hosts and wildcard subdomains before DNS and allow rules, including on redirects and with private-network access enabled. For example, ["tracker.example.com", "*.ads.example.com"] blocks those hosts without requiring a full allowlist. Wildcards do not match the apex; add ads.example.com separately to block it. Matching ignores case and trailing dots; use ASCII/Punycode for internationalized domains. Empty or unset adds no denials
• tools.web.fetch.ssrfPolicy.dangerouslyAllowPrivateNetwork broadly permits private-network targets; enable it only when model-selected URLs are trusted in this deployment
• tools.web.fetch.ssrfPolicy.allowRfc2544BenchmarkRange and tools.web.fetch.ssrfPolicy.allowIpv6UniqueLocalRange are narrow opt-ins for trusted fake-IP proxy stacks; leave them unset unless your proxy owns those synthetic ranges and enforces its own destination policy
• Redirects are checked and limited by maxRedirects (default 3)
• tools.web.fetch.headers values are redacted from exposed config and debug captures, sent to the initial fetched host, and retained on redirects only when the existing guarded-fetch policy allows them
• useTrustedEnvProxy is an explicit opt-in and should only be enabled for operator-controlled proxies that still enforce outbound policy after DNS resolution
• web_fetch is best-effort -- some sites need the Web Browser

Tool profiles

If you use tool profiles or allowlists, add web_fetch or group:web:

{  tools: {    allow: ["web_fetch"],    // or: allow: ["group:web"]  (includes web_fetch, web_search, and x_search)  },}

Related

• Web Search -- search the web with multiple providers
• Web Browser -- full browser automation for JS-heavy sites
• Firecrawl -- Firecrawl search and scrape tools