Gateway protocol talk, config, and agent methods

RPC method families for Talk and TTS, secrets, config, update, and wizard flows, and agent and workspace helpers.

Talk and TTS

• talk.catalog returns the read-only Talk provider catalog for speech, streaming transcription, and realtime voice: standard provider ids, registry aliases, labels, configured state, an optional group-level ready result, exposed model/voice ids, standard modes, transports, brain strategies, and realtime audio/capability flags, without returning provider secrets or mutating global config. Optional sessionKey uses the existing session visibility checks and selects that session's agent for transcription and realtime readiness; unknown or inaccessible sessions are rejected. Without it, the catalog retains the configured Talk agent. Speech readiness uses the shared speech configuration, matching talk.speak and tts.speak. Chat clients pass their current session key; Talk settings omit it. Current gateways set ready after applying runtime provider selection; treat its absence as unverified on older gateways. Catalog reads serve the last successful snapshot while one background refresh checks provider readiness. Configuration, plugin, credential, and secret revisions trigger refresh; unchanged catalogs refresh on demand after 30 seconds. Readiness is advisory: session creation validates current credentials and policy. Before the first snapshot is ready, the RPC returns retryable UNAVAILABLE with details.reason: "talk-catalog-warming" and retryAfterMs; clients should retry within a bounded startup window. The Control UI retries for up to ten seconds. On the first snapshot, a provider readiness failure marks only that provider unconfigured, preserving healthy providers. A successful refresh can publish after the five-second diagnostic deadline; cold callers continue receiving the warming response until the first snapshot is ready. Failed refreshes retain the previous snapshot. Provider hooks that cannot be canceled retain their single-flight slot until they settle. Slow refresh logs identify completed and waiting phases without provider payloads.
• talk.config returns the effective Talk config payload from the Gateway's applied runtime and matching source snapshot; pending or rejected file edits appear only after a successful config reload. includeSecrets requires operator.talk.secrets (or operator.admin).
• talk.session.create (operator.talk) creates a gateway-owned Talk session for realtime/gateway-relay, transcription/gateway-relay, or stt-tts/managed-room. For stt-tts/managed-room, non-admin callers that pass sessionKey must also pass spawnedBy for scoped session-key visibility; unscoped sessionKey creation and brain: "direct-tools" require operator.admin.
• talk.session.appendAudio appends base64 PCM input audio to gateway-owned realtime relay and transcription sessions.
• talk.session.cancelOutput stops assistant audio output, primarily for VAD-gated barge-in in gateway relay sessions. Send the turnId from the current audio event's talkEvent; the result is applied, stale, or idle.
• talk.session.submitToolResult completes a provider tool call emitted by a gateway-owned realtime relay session. The request waits for any asynchronous completion signal exposed by the provider bridge; failed submissions keep the linked run active and do not emit a successful tool-result event. Pass options: { willContinue: true } for interim tool output or options: { suppressResponse: true } when the provider bridge advertises suppression support and the result should not start another response.
• talk.session.steer sends active-run voice control into a gateway-owned agent-backed Talk session: { sessionId, text, mode? }, where mode is status, steer, cancel, or followup; omitted mode is classified from the spoken text. It selects only work bound to that logical voice call, not another call sharing the connection and agent session.
• talk.session.close closes a gateway-owned relay, transcription, or managed-room session and emits terminal Talk events. For realtime relays, disposition: "detach" closes the transport while accepted agent work continues and its answer remains in the conversation; "abort" (the default) cancels that work. Control UI, Android phone, iOS, and macOS failure cleanup detaches, and the Android phone app also detaches when it leaves the foreground; explicit Stop aborts. Detach preserves the original agent answer even if speech was already recorded, because speech carries no per-consult delivery receipt. Send disposition only when hello-ok.features.capabilities advertises talk-session-close-disposition; older Gateways reject unknown fields, so clients without that capability retain the legacy aborting close. For transcription, omitted or false discard finalizes captured audio; discard: true drops buffered audio and cancels pending finalization without publishing its transcript. A discard request can supersede a close that is still finalizing.
• talk.mode sets/broadcasts the current Talk mode state for WebChat/Control UI clients.
• talk.client.create creates or resumes a client-owned realtime provider session using webrtc or provider-websocket while the gateway owns credentials, instructions, tool policy, and the returned voiceSessionId. Clients pass sessionKey and reuse voiceSessionId when replacing the provider transport during one call. Clients that negotiate gateway-control-v1 keep WebRTC media direct but move the provider control channel and tool lifecycle to the Gateway.
• talk.client.transcript appends one finalized { role, text } item to the normal agent session. The required entryId prevents duplicate transcript messages when a request is retried within the same voiceSessionId.
• talk.client.close closes the logical voice session after pending transcript writes. Repeating a close request has no additional effect. Closing may deliver a mutation-only call digest to the session's last non-WebChat channel.
• talk.client.toolCall lets client-owned realtime transports forward provider tool calls to gateway policy. The first supported tool is openclaw_agent_consult; clients get runId, agentId, and resolved agentSessionKey and wait for normal chat lifecycle events before submitting the provider-specific tool result. Use the returned target for chat.abort and chat.history; keep the original key for voice-session requests. Voice-bound actions use ordinary tool policy and host-authenticated approvals, without an additional spoken grant. Transcripts and model-supplied confirmation fields do not authorize actions. Policy and executable checks apply to final arguments, including each nested Code Mode call.
• talk.client.steer sends session-scoped active-run voice control for client-owned realtime transports. The gateway resolves owned active work from sessionKey, without a voice call ID, and returns a structured accepted/rejected result instead of silently dropping steering. Provider-attached Gateway controls are call-scoped instead.
• talk.event is the single Talk event channel for realtime, transcription, STT/TTS, managed-room, telephony, and meeting adapters.
• talk.speak synthesizes speech through the active Talk speech provider.
• tts.status returns TTS enabled state, active provider, fallback providers, and provider config state.
• tts.providers returns the visible TTS provider inventory.
• tts.enable and tts.disable toggle TTS prefs state.
• tts.setProvider updates the preferred TTS provider.
• tts.convert runs one-shot text-to-speech conversion.
• tts.speak (operator.write) renders non-empty text with the configured general TTS provider chain and returns one whole clip inline as audioBase64, plus provider and optional outputFormat, mimeType, and fileExtension metadata. Unlike tts.convert, it does not return a Gateway-local path; unlike talk.speak, it does not require a Talk provider. Text above tts.maxTextLength returns INVALID_REQUEST; synthesis failures return UNAVAILABLE.

Relay output cancellation

For interruptible providers, applied confirms that output was cleared locally, not that the provider has finished the interrupted response. If confirmation takes more than one second, microphone input resumes while the Gateway continues discarding that response's audio, assistant transcripts, and tool calls. Only its provider boundary or a provider continuity reset releases that output owner. If the response remains unfinished for another 30 seconds, the session fails instead of admitting stale output.

While an interrupted response still owns output, cancelling a newer input-only turn returns stale, including when turnId is omitted. Once the provider starts the newer response, its own turn can be cancelled normally. Intentionally non-interruptible providers ignore barge-in and end the session on an explicit output stop. talk.session.close always ends the session.

Secrets, config, update, and wizard

• secrets.reload re-resolves active SecretRefs and atomically publishes owner-aware runtime state. Eligible owner failures can publish as cold or stale degradation with warningCount; strict or unmapped failures reject the reload and preserve the active snapshot.
• secrets.resolve resolves command-target secret assignments for a specific command/target set.
• secrets.store.list (operator.admin) returns team-scoped metadata and values only for kind: "env" entries. kind: "secret" entries use a distinct result shape with no value field; there is no reveal method.
• secrets.store.set and secrets.store.delete (operator.admin) create/update or soft-delete one team-scoped entry. After a successful write, the Gateway refreshes the active secrets runtime only when the name is referenced by a store SecretRef in the active source config.
• config.get returns the current on-disk config snapshot, opaque authored hash covering root-file bytes and captured include identities and content, resolved configRevisionHash, and optional appliedConfigHash for the resolved revision accepted by the active Gateway runtime.
• config.set writes a validated config payload.
• config.patch merges a partial config update. Destructive array replacement requires the affected path in replacePaths; nested arrays under array entries use [] paths such as agents.entries.*.skills.
• config.apply validates + replaces the full config payload.
• config.schema returns the live config schema payload used by Control UI and CLI tooling: schema, uiHints, version, generation metadata, plugin + channel schema metadata when loadable. It includes title / description metadata from the same labels/help text as the UI, including nested object, wildcard, array-item, and anyOf / oneOf / allOf composition branches when matching field documentation exists.
• config.schema.lookup returns a path-scoped lookup payload for one config path: normalized path, a shallow schema node, matched hint + hintPath, optional reloadKind, and immediate child summaries for UI/CLI drill-down. reloadKind is one of restart, hot, or none (src/config/schema.ts) and mirrors the gateway config reload planner for the requested path. Lookup schema nodes keep the user-facing docs and common validation fields (title, description, type, enum, const, format, pattern, numeric/string/array/object bounds, additionalProperties, deprecated, readOnly, writeOnly). Child summaries expose key, normalized path, type, required, hasChildren, optional reloadKind, plus the matched hint / hintPath.
• update.run admits an update and returns its durable runId; acknowledgment is not completion, and callers with a session can include continuationMessage so startup resumes one follow-up agent turn through the restart continuation queue. Supported package-manager and Git-checkout updates from the control plane use the detached CLI updater. The Gateway keeps serving during validation and closes before its runtime files are replaced. A started handoff returns ok: true with result.reason: "managed-service-handoff-started" and handoff.status: "started". A second concurrent update.run handled by the same Gateway process returns ok: false with result.reason: "managed-service-handoff-already-running" and handoff.status: "already-running"; its continuation is not accepted, so the caller can retry after the active update completes. Standalone CLI updaters and replacement Gateway processes are outside this process-local guard. Unavailable or failed handoffs return ok: false with managed-service-handoff-unavailable or managed-service-handoff-failed, plus handoff.command when a manual shell update is required. Unavailable means OpenClaw lacks a safe supervisor boundary or durable service identity, such as OPENCLAW_SYSTEMD_UNIT for systemd. During a started handoff, the restart sentinel may briefly report stats.reason: "restart-health-pending"; the continuation waits for verification of the restarted Gateway.
• update.run acknowledgment fields distinguish notification delivery from update completion: ackDelivered reports delivery, ackQueued reports acceptance by the notification owner, and optional acknowledgement contains the notice text. Read the durable run to determine the update outcome.
• Durable run JSON from update.runs.get, update.runs.list, and update status can include admission: { owner: "candidate" | "installed", protocol?, candidateVersion?, checks?, fallbackReason? }. Each check has name, status (ok, warn, or refuse), and optional detail. This projects origin.admission from the existing ledger JSON; origin.candidateAdmission retains the bounded candidate verdict, reasons, warnings, and facts. Older records omit these fields. See candidate-owned admission for ownership and fallback behavior.
• update.status refreshes and returns the latest update restart sentinel, including the post-restart running version when available.
• wizard.start, wizard.next, wizard.status, and wizard.cancel expose the onboarding wizard over WS RPC. Each answered step uses the current wizard.next caller's authority through inference verification, including after progress replies. Device revocation still rejects the probe.

Agent and workspace helpers

• agent accepts image attachments and offloads them by default when the selected model is text-only, using the same image-understanding pipeline as chat.send. The configured image-understanding model receives the image, and the selected primary receives its description. The primary model, per-request model selection, and prompt/context overrides are preserved. Image-capable primaries continue receiving images inline. The shared attachment limits and per-attachment outcome markers still apply. If no image description can be produced, the request returns a clear error; configure agents.defaults.imageModel or an image-capable entry in tools.media.models, or repair the configured provider before retrying.
• agents.list returns gateway-visible agent entries, including effective model/runtime metadata and optional semantic kind (agent or system). Entries with recorded creation provenance also include createdVia (operator, agent, or claw), nullable creatorAgentId, and millisecond createdAt; entries without provenance omit those fields. Clients advertise the agent-kind handshake capability to receive the complete typed roster; clients without it keep the legacy selector-safe roster without system rows. Kind-aware clients exclude system rows from ordinary selectors while retaining them in diagnostic views. Older v4 gateways may return rows without kind.
• agents.create, agents.update, and agents.delete manage agent records and workspace wiring.
• claws.monitors (operator.admin, rate-limited as a control-plane write for all phases) supports Claw removal. Every request includes binding: { configPath, statePath, cronStorePath } for the local profile, checked against the serving owner. { phase: "inspect", agentId, binding } returns at most two corroborated config-owned monitor snapshots, each with id, name, enabled, agentId, null ownerAgentId, storeKey, declarationKey, and revision. { phase: "quiesce", agentId, operationId, monitors, binding } validates the current deletion journal and exact consented snapshots before cancelling scheduled work. { phase: "drain", agentId, operationId, binding } also requires applied agent removal and monitor convergence. Successful quiescence or drainage returns { drained: true }; incomplete drainage returns UNAVAILABLE after a five-second wait. The operation id must match the live journal in the serving Gateway's state; it is not standalone cleanup authority. Extra request fields are rejected.
• agents.files.list, agents.files.get, and agents.files.set manage the bootstrap workspace files exposed for an agent. agents.files.get and agents.files.set return the file's content hash (SHA-256 hex of the on-disk bytes, the same token sessions.files.set uses). agents.files.set accepts an optional expectedHash for an existing file or expectedMissing: true for a file that was absent when read; these preconditions cannot be combined. A changed hash or an already-created file refuses the write with an INVALID_REQUEST error whose details.type is agent_file_conflict. When available, details.currentHash carries the current hash; re-read the file to rebase. Conditional creation requires the workspace host's atomic exclusive-create capability and fails without writing if that capability is unavailable. Optional expectedWorkspace binds the save to the exact workspace returned by the read; a changed workspace refuses the write with the same conflict type before any mutation. Omit it to retain workspace-unbound writes. Once admitted, a write stays on its captured workspace even if the agent is reconfigured while it finishes. Omitting all preconditions keeps the unconditional overwrite.
• audit.activity.list returns the versioned metadata-only activity ledger; audit.run.inspect discovers execution ids or inspects one exact execution identity context; audit.list remains the compatibility-safe run/tool RPC.
• agents.workspace.list and agents.workspace.get (operator.read) expose read-only, paginated browsing of an agent's workspace directory for clients in the trusted operator domain described in Operator scopes. Requests accept workspace-relative paths only; reads stay confined to the realpathed workspace root (symlink and hardlink escapes rejected), size-capped, and limited to UTF-8 text plus common image types (base64). Workspace directory listings and file searches report an error when a filename is not valid UTF-8; rename that entry on the host before retrying. Responses do not expose the host workspace path. There are no write operations in this namespace.
• transcripts.list (operator.read) lists durable meeting captures newest first. Optional limit accepts 1–200 (default 50); providerId filters the source. The sessions result includes selectors, provider/source locators, times, active state, utterance counts, participants, summary availability, optional model/heuristic provenance, and an overview preview capped at 280 characters. Source locators expose only providerId, accountId, guildId, channelId, and meetingUrl, never free-form metadata.
• transcripts.get (operator.read) accepts selector and optional includeUtterances. It returns the session and stored summary, including its saved Markdown; requested utterances are sanitized and bounded by the capture limit of 2,000. Missing summaries omit summary rather than generating notes. Both transcript methods read across one trusted Gateway domain, like agents.workspace.*; separate domains are required for reader isolation. They do not export files or change capture state. See Transcripts CLI.
• artifacts.download accepts optional transport: "http" for clients with a reachable Gateway HTTPS origin. Inline transcript artifacts then return a short-lived, connection-bound URL for raw-byte downloads; omitted transport preserves base64 over the existing connection. WebSocket availability alone does not establish HTTP reachability. Inline payloads too large for one WebSocket response are refused with artifact_download_unsupported; use transport: "http". See HTTPS artifact downloads.
• artifacts.list, artifacts.get, and artifacts.download expose transcript-derived artifact summaries and downloads for an explicit sessionKey or runId scope. Run queries retain the stored session selected for that run, even if the current main-session alias points elsewhere; explicit session selectors resolve current aliases. Both enforce current access to the selected session and only return transcript media with matching provenance; unsafe or local URL sources return unsupported downloads instead of fetching server-side. Set messageRole: "assistant" on list, get, and download to select assistant-delivered artifacts, including delivery mirrors, and exclude uploaded inputs and raw tool observations. Omit the filter to keep all-role discovery. Combine it with runId for a particular run; a run ID alone does not identify generated output. Older Gateways reject the new filter; clients must surface that rejection instead of retrying without it.
• artifacts.list also accepts type: "image", optional limit (1–4, default 4), and an opaque cursor for recent image discovery. Each page reads up to 32 transcript messages with a 1 MiB budget, newest first, and reads an oversized message whole; nextCursor continues into older messages even when a sparse page has no images. Inline images up to 256 KiB return a bounded data-URL preview in image.url. Referenced inline images also return a downloadable artifact_transcript_image_ id; above the preview bound they are reference-only without image.url, and their bytes load through artifacts.download. Unreferenced inline images above the bound are omitted and the page reports omittedOversized: true; open the session to view them. URL-based image summaries keep image.url from structured content, stored uploaded media facts, or rendered Markdown references. Nonmanaged URL images use preview_ ids, source: "session-transcript-preview", and unsupported downloads: these are preview references, not identifiers for artifacts.get or artifacts.download. The Control UI uses the existing authenticated media routes and session media policy for local sources and the artifact download owner for managed media and inline transcript images. Discovery never restores cold transcripts or rebuilds their projections; open the session to restore unavailable previews. Cursors expire after 15 minutes, belong to the originating connection, agent, session, and query, and reject transcript resets. Requests without type preserve the complete artifact listing; limit and cursor require the image filter.
• Complete artifact listings reuse bounded summary metadata in the transcript worker while the SQLite transcript revision is unchanged. Session-access discovery uses the metadata reader; admitted sharing and transcript reads retain the history reader and its summaries between authorized requests. Appends (including text-only appends), rewrites, branch changes, and resets invalidate the result on the next read; access checks still run for each request. Listing does not stat artifact files or retain their contents in the summary cache.
• environments.list and environments.status (operator.read) remain available without cloud-worker profiles and preserve gateway-local and node environment discovery. environments.list also accepts an optional runtimeId from callers with operator.write. That request adds one Gateway-owned requiredNodeCommand result to each connected node when the runtime requires a node command. Its closed state is invocable, pending-approval, undeclared, or unauthorized; it never exposes the node's full pending declaration. Node environments include the durable sessionHost identity used to keep a known offline host visible, while current connected inventory is authoritative over that history. Missing identity means false. Exact bounded { total, available } worker slots are live-only and omitted offline; worker-turn admission consumes a slot, while node-backed remote-exec does not. Configured profile summaries expose their bounded, fixed-order executionModes array plus the existing singular executionMode primary/default display projection. Current clients select profiles only by membership in executionModes. Configured cloud workers and durable records left by earlier profiles add worker metadata with the configured profileId, providerId, optional leaseId, state, ageMs, optional idleMs, and attachedSessionIds. Worker lifecycle states are requested, provisioning, bootstrapping, ready, attached, idle, draining, destroying, destroyed, failed, and orphaned. A connected node may also include workerBundle: { status: "installed", version } or workerBundle: { status: "missing" }. This optional observation is reconnect-scoped and reports validation of one Gateway-retained bundle; it is not launch authority. The public result never exposes the bundle hash, Gateway namespace, node filesystem path, receipt, or protocol-feature details.
• environments.list with { projection: "profiles" } reads only the configured profile catalog, including provider-authored machine and operating-system choices. It returns environments: [] and omits profiles when none are advertised. Worker and paired-device inventory is not read, so inventory failures do not prevent profile discovery. Omitting projection keeps the full inventory behavior. Including runtimeId still requires operator.write, even with the profile projection.
• environments.create ({ profileId, idempotencyKey }) provisions an environment from a configured plugin provider profile; retries with the same key reuse the durable operation. Direct creation without a session does not select an execution mode, so the provider uses its intentional default; Crabbox prepares worker-turn. environments.destroy ({ environmentId }) requests teardown of a durable worker environment; repeating the request has no additional effect. Both require operator.admin, are control-plane writes, and return the same environment summary shape used by status responses.
• environments.prepare ({ profileId, projectPath }, operator.admin) admits a project build without a session and returns { environmentId, preparationKey, reused }. It is a control-plane write gated on provider startup. The project must be a local Git checkout; the configured provider must support project preparation. A matching unconsumed build or reserve is reused. Known errors retain details.code: profile_not_found, invalid_profile, and invalid_project map to INVALID_REQUEST; capacity maps to UNAVAILABLE. Other failures return a generic UNAVAILABLE without provider details. Prepared summaries expose only preparation: { purpose: "reserve" | "build", key }. Use environments.destroy to cancel. See Build on demand for pool policy and setup authorization.
• worker.desktop.observe ({ environmentId, control? }, operator.admin) starts or reuses the environment's desktop forward and returns { transport, wsPath, expiresAtMs, control, vncPassword? }. wsPath carries a single-use 60-second token for the Gateway's desktop observer WebSocket; reconnecting requires a fresh observe call. Environments with an observable desktop advertise worker.desktop: true in environments.list. The method is advertised only when the cloudWorkers.desktop lab is enabled. See Cloud workers.
• desktop.release ({ wsPath }, operator.admin) abandons an unclaimed result from desktop.observe or worker.desktop.observe on the original requesting Gateway connection. It returns { released } and releases only that ticket's pending node stream. Unknown, expired, already-claimed, and other-connection tickets return released: false; existing viewers and control ownership are unchanged. Clients should retain cleanup responsibility until RFB authentication and release late results when their presenter has closed.
• agent.identity.get returns the effective assistant identity for an agent or session.
• agent.wait waits for a run to finish and returns the terminal snapshot when available.