diagnostics-prometheus plugin. It listens to trusted internal diagnostics and renders a Prometheus text endpoint at:
text/plain; version=0.0.4; charset=utf-8, the standard Prometheus exposition format.
For traces, logs, OTLP push, and OpenTelemetry GenAI semantic attributes, see OpenTelemetry export.
Quick start
diagnostics.enabled: true is required. Without it, the plugin still registers the HTTP route but no diagnostic events flow into the exporter, so the response is empty.Metrics exported
| Metric | Type | Labels |
|---|---|---|
openclaw_run_completed_total | counter | channel, model, outcome, provider, trigger |
openclaw_run_duration_seconds | histogram | channel, model, outcome, provider, trigger |
openclaw_model_call_total | counter | api, error_category, model, outcome, provider, transport |
openclaw_model_call_duration_seconds | histogram | api, error_category, model, outcome, provider, transport |
openclaw_model_tokens_total | counter | agent, channel, model, provider, token_type |
openclaw_gen_ai_client_token_usage | histogram | model, provider, token_type |
openclaw_model_cost_usd_total | counter | agent, channel, model, provider |
openclaw_tool_execution_total | counter | error_category, outcome, params_kind, tool |
openclaw_tool_execution_duration_seconds | histogram | error_category, outcome, params_kind, tool |
openclaw_harness_run_total | counter | channel, error_category, harness, model, outcome, phase, plugin, provider |
openclaw_harness_run_duration_seconds | histogram | channel, error_category, harness, model, outcome, phase, plugin, provider |
openclaw_message_processed_total | counter | channel, outcome, reason |
openclaw_message_processed_duration_seconds | histogram | channel, outcome, reason |
openclaw_message_delivery_total | counter | channel, delivery_kind, error_category, outcome |
openclaw_message_delivery_duration_seconds | histogram | channel, delivery_kind, error_category, outcome |
openclaw_queue_lane_size | gauge | lane |
openclaw_queue_lane_wait_seconds | histogram | lane |
openclaw_session_state_total | counter | reason, state |
openclaw_session_queue_depth | gauge | state |
openclaw_memory_bytes | gauge | kind |
openclaw_memory_rss_bytes | histogram | none |
openclaw_memory_pressure_total | counter | level, reason |
openclaw_telemetry_exporter_total | counter | exporter, reason, signal, status |
openclaw_prometheus_series_dropped_total | counter | none |
Label policy
Bounded, low-cardinality labels
Bounded, low-cardinality labels
Prometheus labels stay bounded and low-cardinality. The exporter does not emit raw diagnostic identifiers such as
runId, sessionKey, sessionId, callId, toolCallId, message IDs, chat IDs, or provider request IDs.Label values are redacted and must match OpenClaw’s low-cardinality character policy. Values that fail the policy are replaced with unknown, other, or none, depending on the metric.Series cap and overflow accounting
Series cap and overflow accounting
The exporter caps retained time series in memory at 2048 series across counters, gauges, and histograms combined. New series beyond that cap are dropped, and
openclaw_prometheus_series_dropped_total increments by one each time.Watch this counter as a hard signal that an attribute upstream is leaking high-cardinality values. The exporter never lifts the cap automatically; if it climbs, fix the source rather than disabling the cap.What never appears in Prometheus output
What never appears in Prometheus output
- prompt text, response text, tool inputs, tool outputs, system prompts
- raw provider request IDs (only bounded hashes, where applicable, on spans — never on metrics)
- session keys and session IDs
- hostnames, file paths, secret values
PromQL recipes
Choosing between Prometheus and OpenTelemetry export
OpenClaw supports both surfaces independently. You can run either, both, or neither.- diagnostics-prometheus
- diagnostics-otel
- Pull model: Prometheus scrapes
/api/diagnostics/prometheus. - No external collector required.
- Authenticated through normal Gateway auth.
- Surface is metrics only (no traces or logs).
- Best for stacks already standardized on Prometheus + Grafana.
Troubleshooting
Empty response body
Empty response body
- Check
diagnostics.enabled: truein config. - Confirm the plugin is enabled and loaded with
openclaw plugins list --enabled. - Generate some traffic; counters and histograms only emit lines after at least one event.
401 / unauthorized
401 / unauthorized
`openclaw_prometheus_series_dropped_total` is climbing
`openclaw_prometheus_series_dropped_total` is climbing
A new attribute is exceeding the 2048-series cap. Inspect recent metrics for an unexpectedly high-cardinality label and fix it at the source. The exporter intentionally drops new series instead of silently rewriting labels.
Prometheus shows stale series after a restart
Prometheus shows stale series after a restart
The plugin keeps state in memory only. After a Gateway restart, counters reset to zero and gauges restart at their next reported value. Use PromQL
rate() and increase() to handle resets cleanly.Related
- Diagnostics export — local diagnostics zip for support bundles
- Health and readiness —
/healthzand/readyzprobes - Logging — file-based logging
- OpenTelemetry export — OTLP push for traces, metrics, and logs