Gateway logging

For a user-facing overview (CLI + Control UI + config), see /logging.

OpenClaw has two log surfaces:

• Console output - what you see in the terminal.
• File logs - JSON lines written by the gateway logger.

At startup, the Gateway logs the resolved default agent model plus the mode defaults that affect new sessions:

agent model: openai/gpt-6-astra (thinking=medium, fast=on)

thinking comes from the default agent, model params, or the global agent default. When unset it shows medium. fast comes from the default agent or the model's fastMode params.

If a plugin reload supersedes startup plugin loading, the model line, loaded-plugin summary, and channel warnings use the replacement configuration and plugin metadata.

Completed startup phase: lines include rssMb (resident memory for the whole process, including workers) and heapUsedMb (used JavaScript heap in the main isolate). Both values are in MiB and are sampled when the phase finishes, so service journals can attribute startup memory growth before readiness. They are point-in-time readings, not peak memory measurements.

Before readiness, plugin load details are logged when loading and registration take at least 250 ms or start a pooled worker. workersStarted counts confirmed pool worker starts caused by that plugin's preparation, module evaluation, and registration; workerScripts breaks those starts down by script. Concurrent recovery work is excluded, as are unmanaged workers created directly by plugins. These are starts, not live-worker or retained-memory counts; background work that starts after the plugin's load detail is emitted is not included. OPENCLAW_GATEWAY_STARTUP_TRACE=1 also logs the faster, worker-free plugin loads.

Environment selection reports imports, config reading, recovery (including its config-health database read), guarding, and runtime-path pinning separately. These sub-phases appear during ordinary foreground Gateway startup without enabling full startup tracing. Parent timings include their nested sub-phases; do not add them together. Config-health timing includes any private database snapshot preparation and cleanup required by that read.

The same lines include workers (live tracked worker isolates) and workerHeapSamples (sampled workers/live workers). When samples are available, workerHeapMb sums their total heap capacity in MiB; workerHeapSource=cached identifies the existing diagnostics samples, which may be up to 60 seconds old. Missing or stale samples are excluded, so partial coverage is not the total heap of all workers. Startup logging never requests new worker samples.

File-based logger

• Default rolling log files are under /tmp/openclaw/ (one file per day), dated by the gateway host's local timezone. The default profile uses openclaw-YYYY-MM-DD.log. Named profiles use openclaw-<profile>-YYYY-MM-DD.log (for example, openclaw-dev-YYYY-MM-DD.log). If that directory is unsafe or unwritable (wrong owner, world-writable, a symlink), OpenClaw falls back to a user-scoped os.tmpdir()/openclaw-<uid> path instead. On Windows it always uses that OS-tmpdir fallback.
• Active log files rotate at logging.maxFileBytes (default: 100 MB). Rotation keeps up to five numbered archives (.1 through .5), and continues to write a fresh active file.
• Configure the log file path and level via ~/.openclaw/openclaw.json: logging.file, logging.level.
• The file format is one JSON object per line.

With config hot reload enabled, changes to logging.level, logging.file, and logging.maxFileBytes apply to the next log record, including records from long-lived channel loggers. Queued records finish writing to their original file. Explicit logger-level overrides, such as Baileys verbosity, remain in effect.

Startup guidance to keep local CLI/SDK and Gateway versions aligned uses info; it does not indicate that a version mismatch was detected. In-memory auth storage initialization does not emit synchronous-storage deprecation warnings. Explicit calls to deprecated storage APIs still warn with their replacements.

Subsystem file logs omit call-site metadata (_meta.path) for trace, debug, info, and warn records, including raw() lines, to avoid capturing and parsing a stack on every routine message. error and fatal records retain it. Other than info, all levels retain call-site metadata while diagnostics are enabled and an internal log-record consumer is subscribed, preserving OTLP code locations. info records, including raw() lines, remain stack-free with diagnostic consumers; their severity, message, structured attributes, and trace context are still exported. This follows diagnostic enablement and subscriptions on the next record, including for existing subsystem loggers. Log messages, structured fields, and error stacks supplied by callers are unchanged.

Talk, realtime voice, and managed-room code paths use the shared file logger for bounded lifecycle records intended for operational debugging and OTLP log export. Transcript text, audio payloads, turn ids, call ids, and provider item ids are never copied into the log record.

Discord realtime voice keeps session lifecycle transitions at info; audio chunks and transcript deltas use debug. Model-fetch starts and successful responses also use debug, regardless of duration. Non-2xx responses remain at info; transport failures remain warnings. The existing model transport diagnostic flags promote transport details to info when enabled.

Successful secret egress forwarding, including credential substitution, uses debug. Refusals remain at info, and certificate failures remain warnings. Their structured fields record the proxy outcome without request payloads or credentials; see secret egress proxy. Code Mode and Tool Search catalog preparation summaries also use debug. Prometheus metrics count egress outcomes, catalog preparations, and model calls independently of log levels.

Per-request GitHub API timing records also use debug, including fetches over 500 ms. Transport failures and rate-limit errors retain their normal handling.

Plugin host cleanup failures during session reset or deletion emit one plugins/cleanup warning per failure, including the operation, plugin, hook, and error. Check the reported cleanup error before retrying. Successful cleanup stays quiet, and failures remain best effort. Console diagnostics use the shared secret redactor; plain-text console fields are capped at 2,048 characters.

Plugin replacement logs bounded retained-reference snapshots when work queues and warns when its drain deadline expires. Structured file records include the acquisition owner fields that were available when each reference was created: agentId, sessionKey, runId, and serviceId. These raw facts identify the work holding the previous generation; a missing owner stays unknown. Normal log egress applies secret redaction, and support bundles omit the nested owner details. Plain-text timeout warnings include the bounded structured fields.

The Control UI Logs tab tails this file via the gateway (logs.tail). The CLI does the same:

openclaw logs --follow

If a tail read observes that the active file has disappeared, the Control UI clears its previous records and follows the recreated file. Missing files still return an empty tail. Filesystem read errors, including a log path that points to a directory, remain visible while the Control UI keeps the last successfully read records as stale data.

Verbose vs. log levels

• File logs are controlled exclusively by logging.level.
• --verbose only affects console verbosity (and WS log style) - it does not raise the file log level.
• To capture verbose-only details in file logs, set logging.level to debug or trace.
• Embedded-run continue_normal decisions log at debug. Retry, profile-rotation, model-fallback, and error decisions remain warnings.
• Trace logging also includes diagnostic timing summaries for selected hot paths, such as plugin tool factory preparation. See /tools/plugin#slow-plugin-tool-setup.

SQLite session writes

Failed SQLite session writes include a bounded, redacted error summary in their structured file-log record, with cause and error-code details when available. Long summaries are truncated. The record retains its write timing and store fields.

SQLite snapshot cleanup

Failed removal of a temporary read-only SQLite snapshot is recorded once by its cleanup owner in the structured file log, with the owned path, removal operation, and filesystem error code when available. These diagnostics do not write to subprocess stdout or stderr, so a successful read keeps its result and a failed update retains its original error detail. Existing required-cleanup failures remain errors.

Slow agent database opens

A completed physical agent-database open taking at least one second emits slow OpenClaw agent database open. The record retains total elapsed time and the open, validation, configuration, schema, and registration phases. For a yielded integrity check, it also includes integrityGateMs and integrityGateOutcome (healthy or failed). The gate includes the check plus any driver waits, scheduling, and ownership revalidation. When admission uses a separate integrity Worker, its lifetime is included. This does not isolate native-check or CPU time.

When required-index validation completes, canonicalIndexMs reports the subsequent synchronous inspection and any repair or rechecks, and repairedIndexCount counts indexes successfully repaired by that operation. A healthy initial integrity check can still require an index-definition repair. A failed initial check can be recovered by a successful repair. Fields are absent when their stage does not run, including the yielded-check fields for a fresh empty database. These details cover portions of validation, not extra time to add to it. The summary is emitted only at registration. Earlier failures and live cache hits produce no summary. The details add no index names or database contents.

Slow cron list pages

A cron list page taking at least one second emits cron: slow list page through its existing logger, subject to the file log level. The structured record names operation: "cron.listPage" and reports elapsedMs, waitToCallbackMs, callbackMs, and completionDelayMs, plus available source, matched, and returned row counts, the outcome, and emitter pid, threadId, and isMainThread. Fast pages emit no such record.

These are wall times, not CPU time: waiting includes scheduling delays, callback time includes awaited work, and completion delay covers settlement after the callback finishes. Each source page is measured separately. Caller visibility filtering runs inside the page callback; delivery previews remain outside it. Existing trace context is retained when present. Emitter identity identifies the logging process/isolate, not the owner of work awaited by the callback. The diagnostic adds no job identifiers, job contents, or request parameters.

Slow cron list requests

With diagnostics.enabled active, a cron.list handler taking at least one second emits cron: slow list request through the Gateway logger. The record uses the existing request trace/span and reports elapsedMs plus fixed phaseDurationsMs for setup, listing, projection, optional previews, response, and handlerExit. Unentered phases are absent.

sourcePageMs and sourcePageCount aggregate source-page calls, including failed calls. returnedCount appears once a page is selected. scopeAttemptCount is zero for direct lists and one for scoped lists. Visibility filtering, sorting, revision calculation, and pagination share one locked source operation. For scoped lists, scopeProcessingMs is listing time minus source-page time, covering work outside that operation. These components are already included in the listing phase and must not be added to it again.

The bounded branch fields are compact, previewsRequested, and scopeApplied. previewsRequested describes the selected response mode, not whether execution reached that phase. handlerOutcome is returned or threw. responseOutcome is none, ok, error, or threw for the handler's response callback. Its response phase measures that synchronous callback, and handlerExit ends at the handler's own cleanup boundary. Neither proves socket delivery or client receipt. Outer RPC diagnostics retain those separate outcomes.

All durations are wall time, including awaits and scheduling, not CPU time. Fast requests and requests with diagnostics disabled emit no summary. The record adds no job identifiers, content, query strings, targets or error text, and does not change individual slow-page warnings or response payloads.

Slow Codex catalog pages

With diagnostics and warning logging enabled, a Codex catalog page taking at least one second emits slow Codex catalog page producer. Its existing phase totals distinguish client acquisition, request waiting, and page processing. diagnosticEpoch and operationId identify the page observation; listOperationId links its originating logical list when available.

controlWaitersV1 is a JSON-encoded array joining sampled page waits to the physical client and JSON-RPC attempt. Decode the string with JSON.parse to read its tuples. It keeps the first two and latest two completed waiter summaries. controlWaitersOmitted counts summaries excluded by the bounds. Each entry has these positions:

Times are rounded process-local monotonic milliseconds, comparable within the same process. A later waiter retains the original attempt and possible-write times. Waiter outcomes distinguish resolved, native-error, timed-out, aborted, authority-rejected, local-failed, and client-closed. Wire outcomes are retained-pending, native-ok, native-error, ingress-rejected, correlation-closed, or not-written.

A possible write does not prove native acceptance. A joined waiter does not mean another request was sent, and a timed-out waiter can leave the wire attempt pending. Later wire settlement is not promised after the page observation closes. These records contain no query, cursor, path, title, authentication data, or raw error. Existing bounds remain 64 active observations, 60 warnings per minute, 28 metadata keys, and 2,048 bytes. Missing or omitted summaries are unavailable evidence, not zero activity; durations do not attribute native CPU or client receipt.

Console capture

The CLI captures console.log/info/warn/error/debug/trace, writes them to file logs, and still prints to stdout/stderr.

console.trace() keeps its redacted stack in every console style, including forced stderr output. File capture records it once at trace level, subject to the configured file log level.

Tune console verbosity independently:

• logging.consoleLevel (default info)
• logging.consoleStyle (pretty | json). When unset, output is pretty on a TTY and the automatic compact style otherwise. compact is no longer a settable value. openclaw doctor --fix maps a stored one to pretty.

Redaction

File logs store raw records under the Gateway account with owner-only (0600) permissions. Redaction runs when logs widen their audience: logs.tail (including the CLI and Control UI), console output, diagnostic/support bundles, and remote or OTLP export. Each output redacts independently; direct reads of the file do not. Protect raw log files like the agent/state databases and review them before sharing.

Tool arguments, tool results, and session transcripts are not secret-masked. This includes source-code output and direct reads of .env files. Tool payloads still receive size limits, image/base64 handling, and structural normalization. Configuration snapshots, audit output, command-line diagnostics, and credential-bearing URLs retain their masking.

• Egress log and diagnostic redaction is always enabled.
• logging.redactPatterns: array of regex strings (replaces the default string list). Built-in structural protections for form bodies, structured authorization headers, and bare AWS secret access keys always apply.
  • Use raw regex strings (auto gi), or /pattern/flags for custom flags.
  • Matches are masked keeping the first 6 + last 4 chars (values >= 18 chars). Shorter values become ***.
  • Defaults cover common key assignments, CLI flags, JSON fields, bearer headers, PEM blocks, popular vendor token prefixes, and payment credential field names (card number, CVC/CVV, shared payment token, payment credential).

JSON console output and log tails preserve valid JSON while masking decoded values and serialized record context. Later rules see earlier masks. Structured credential fields use full masks; matched numbers, booleans, and null become the JSON string "***". Log tails retain built-in credential patterns when custom patterns are configured.

Formatted health failures, Doctor Gateway connection notes, diagnostics exports, and captured provider/WebSocket console output retain redaction. logging.redactPatterns supplies deployment-specific patterns at these egress boundaries. Log IDs, sizes, and outcomes rather than message text, prompts, tool arguments, or tool results. WebSocket agent-event summaries contain character counts instead of assistant text or tool metadata.

Gateway WebSocket logs

The gateway prints WebSocket protocol logs in two modes:

• Normal mode (no --verbose): failed RPC results (ok=false), slow calls (>= 1,000ms), and parse errors remain visible at the default log level. Successful responses below one second, including large session lists, log at debug. Set logging.level: "debug" to retain them in file logs or logging.consoleLevel: "debug" to print them.
• Verbose mode (--verbose): prints all WS request/response traffic.

The validated connect handshake also participates in the existing Gateway RPC metrics as method="connect", including handler duration, first-response latency, response bytes, and delivery failures. Handler time includes completion work after the hello write; first-response time ends at the transport's write callback, not at browser receipt. These are server wall times, including awaited work. A validated connect rejected by restart or suspension admission reports a rejected dispatch and its response; it has no handler sample because authentication was never entered.

Connect response logs follow the same slow-response threshold and include authMs (credentials through authentication), attachMs (device authorization, identity, scopes, and session attachment), helloMs (hello payload preparation), and sendMs (serialization and transport write callback). Unreached phases are absent. Bootstrap redemption or admission work between payload preparation and writing can make total response time exceed the sum of these fields. No payload fields or credentials are included. Failed connects also record failurePhase and the existing admission failureCause so a rejection before attachment remains attributable.

The attach interval also includes these sub-phase wall times on connect response logs: deviceAuthorizationMs (pairing policy, lookup and token issuance), nodePreparationMs (node admission and pairing capture), identityMs (profile admission), scopeGrantMs (scope/role policy and the identity-elevation audit log), operatorDeviceMs (accepted device credential lookup), and sessionRegistrationMs (remaining client preparation, access binding and transport handoff). Nested fields deviceLookupMs, deviceTokensMs, profileEnsureMs and profileAuthorityMs isolate pairing and profile operations, including cache hits and worker queue/lock waits. Nested times overlap their parent interval; do not add them to the total. These are elapsed times, so event-loop stalls during awaited work also count. Unused or unreached sub-phases are absent.

Profile preparation further reports profileAccessIdentityMs (Cloudflare Access verification or verified-assertion reuse), profileGitHubMetadataMs (GitHub public metadata lookup or reuse), profileBindingReadMs (the current local account/email or login binding), and profileWriteMs (canonical profile ensure/sync, which may be a no-op). profileAuthorityMs covers local authority acquisition; profileProjectionMs covers the resulting profile and recipient projection. These nested timings contain no identity, assertion, or profile contents. Warm account/email lookups use the existing profile catalog and its mutation settlement; standalone callers and login aliases retain worker reads. Expired external verification still blocks admission when roles or access policy require it. Optional avatar adoption remains after hello.

The hello interval includes snapshotMs (presence, session defaults, update details, and cached health) and helloDescriptorsMs (method and event descriptors). These are nested wall times, not additional phases. Model catalog publication runs after hello and is outside these intervals. Hello replies remain uncompressed even when the connection negotiates permessage-deflate, because they can contain credentials.

With diagnostics.enabled: true and warning logging enabled, sessions.list handlers and sessions.subscribe snapshot handlers taking at least one second also emit slow session list. The operation field identifies which request produced the record. The record includes process/thread identity, the request trace, and row counts: selectedRowCount, dirtyRowCount, materializedRowCount, and reusedRowCount. The latter two distinguish selected rows refreshed during this request from selected rows already resident when it began. Dirty counts describe pending owner work at the start of the request.

sessions.messages.subscribe requests taking at least one second emit slow session messages subscribe with operation, elapsedMs, and phaseDurationsMs. Timing starts in the router, before the handler, and separates projectionReadiness, accessFacts, handlerPreparation, retainedReadAdmission, replayPreparation, and observerCommit from response and cleanup work. Ordinary subscriptions prepare committed access facts without waiting for display-row refreshes; incognito reads retain their exact-row preparation. Approval subscriptions still prepare and validate the authoritative replay before acknowledging. Records contain no session keys or message content.

The materialize phase measures the wait for session-row projection readiness. In-flight catalog renewals no longer block lists or descriptions once a catalog is loaded: reads use the current catalog while its replacement loads in the background, then rows refresh with the new catalog. Startup still waits for the first catalog. Renewals that retain identical catalog content do not dirty resident rows.

Concurrent lists share metadata preparation and selected-row materialization. Selected pages yield to the event loop between bounded materialization batches, including when stored facts are already cached. A long materialize wait does not by itself indicate an event-loop stall or repeated work for each caller.

Profile and run-registry publications refresh their derived display facts without rereading session entries. Worker environment and placement publications refresh only the selected rows' worker facts on their next presentation. Stored session writes publish exact keys; broad list notifications do not schedule an all-row drain. Sidebar preferences, talk.realtime.model, and other projection-neutral config commits retain session rows. Agent identity edits refresh display facts on presentation; store admission reconciles physical generations and retains unchanged rows. Session policy, roster, sharing, and adopted model catalog changes still refresh affected live rows before lists respond. Archived rows stay cold until selected.

Transcript-only row refreshes use a one-second window per resident session: the first notification refreshes promptly, and further notifications collapse into a trailing refresh. These pending notifications are not dirty rows until that refresh is due. Transcript freshness can therefore lag by up to one window; optional previews still wait for idle background backfill. Metadata, lifecycle, catalog, and topology publications continue to invalidate immediately. Transcript notifications do not invalidate parents or children: relationships, inherited model settings, and subagent activity have their own metadata or registry publications.

Records report phase totals, synchronous selection/row time, and yieldWaitMs/yieldCount for awaiting shared projection readiness. These waits can include coalesced work shared with other callers. Phase totals include their wait intervals; do not add the detailed counters to those totals again. handlerElapsedMs starts before parameter validation and excludes admission before the handler. The response phase includes the synchronous response callback. These are elapsed durations, not CPU time or proof of client receipt. No query text or session contents are included.

The same record includes fractional-millisecond current-thread CPU measurements for synchronous work: prepareThreadCpuMs, rowThreadCpuMs, and responseThreadCpuMs. Preparation covers resident selection, filtering, and sorting after projection readiness. Row CPU includes presentation and final list construction. Both intervals finish before the response callback is measured; response CPU excludes network waits. Measurements finish before this diagnostic record is published or logged. Projection readiness waits, background materialization, intervening microtasks, and worker CPU are not included. These are selected inclusive CPU intervals, including same-thread native work and garbage collection, not SQL-only CPU or a complete request CPU total.

Unvisited measurements are omitted. Each request retains its own selection, presentation, and response CPU without inheriting shared background work. If a CPU counter read fails, all CPU fields are omitted for that request; its result and elapsed diagnostics are preserved. Existing activation and the one-second warning threshold are unchanged, so missing slow records do not account for CPU consumed by faster requests.

chat.send records slow chat send <N>ms stage=request when authorization, input admission, and acknowledgement take at least one second. Phase durations appear in the message text, including authority, run admission, attachments, persistence, and response. A separate stage=startup ack=<N>ms record attributes post-acknowledgement preparation to workspace setup, reply initialization, skill preparation, and authoring. worktree covers pending workspace creation; ordinary sandbox readiness remains inside inclusive preparation. replyInitialization includes routing, admission-ticket waits, workspace bootstrap, media processing, and session initialization until the prepared-session callback. Initial session row-reader waits are included in that interval, rather than timed separately. snapshot measures the agent-run transcript-start reader preparation, nested inside preparation. Startup measurement ends when the agent run starts, or when dispatch exits before starting one. Accepted steering instead reports stage=steer; a deferred follow-up reports stage=queued. These records include waits for the active run to commit the input at its next boundary, including the steering receipt timeout. They use the same one-second threshold and emit at most one post-acknowledgement record per send; they are not startup latency. Rejected steering that starts a new run retains stage=startup. These records do not contain message text or session identifiers. Nested phases can overlap; their sum is not the request duration. The Prometheus exporter records the same fixed phases for fast and slow sends when diagnostics are enabled. Message formatting is deferred until the one-second threshold is reached.

Catalog lists additionally expose fixed request-stage observations through the existing diagnostic event stream and Prometheus exporter. These include initial/final projection readiness, provider or coalesced waits, and synchronous planning/final-delivery thread CPU. Unlike slow logs, these observations include requests below one second when an interested trusted consumer is active. They do not change warning thresholds, startup-phase history, request behavior or diagnostic collection settings.

WS log style

openclaw gateway supports a per-gateway style switch:

• --ws-log auto (default): normal mode is optimized. Verbose mode uses compact output.
• --ws-log compact: compact output (paired request/response) when verbose.
• --ws-log full: full per-frame output when verbose.
• --compact: alias for --ws-log compact.

# optimized (connections, errors, and slow responses)openclaw gateway # show all WS traffic (paired)openclaw gateway --verbose --ws-log compact # show all WS traffic (full meta)openclaw gateway --verbose --ws-log full

Normal logging emits one client connected record per admitted operator or node connection, including CLI and backend clients. It includes the connection ID, client ID and display name, mode, role, authentication method, version, platform, build, remote address, and authenticated user when present. This replaces the separate authenticated user connected and webchat connected records; those records overlapped and did not count ephemeral clients. Request content and credentials are not included.

Compare these records with the connect RPC counter when investigating connection churn. The counter includes failed handshakes; client connected records admission before the hello response is sent. Client metadata is caller-declared, so it identifies the claimed client kind, not a verified process. For external pollers, reuse an authenticated Gateway connection for repeated requests. Gateway-hosted plugins should use api.runtime.gateway.request or api.runtime.nodes, which dispatch in-process; see Gateway and node runtime.

Console formatting (subsystem logging)

The console formatter is TTY-aware and prints consistent, prefixed lines. Subsystem loggers keep output grouped and scannable:

• Subsystem prefixes on every line (e.g. [gateway], [canvas], [tailscale]).
• Subsystem colors (stable per subsystem, hashed from the name) plus level coloring.
• Color when output is a TTY or the environment looks like a rich terminal (TERM/COLORTERM/TERM_PROGRAM). Respects NO_COLOR and FORCE_COLOR.
• Shortened subsystem prefixes: drops a leading gateway/, channels/, or providers/ segment, then keeps at most the last 2 remaining segments (e.g. channels/turn/execution displays as turn/execution). Known channel subsystems (telegram, whatsapp, slack, etc.) always collapse to just the channel name.
• Sub-loggers by subsystem (auto prefix + structured field { subsystem }).
• logRaw() for QR/UX output (no prefix, no formatting).
• Console styles: pretty | json (compact is applied automatically off-TTY and is not a settable value).
• Console log level is separate from file log level (file keeps full detail when logging.level is debug/trace).
• WhatsApp message bodies log at debug (use --verbose to see them).

This keeps file logs stable while making interactive output scannable.

Related

• Logging
• OpenTelemetry export
• Diagnostics export
• openclaw logs — tail Gateway logs over RPC from the CLI