RTC Agent includes a complete observability infrastructure: Prometheus for metrics collection, Grafana for dashboard visualization, and Alertmanager for alert notifications. This page documents all available metrics, dashboards, and pre-configured alert rules.
All Prometheus metrics are exposed via the GET /metrics endpoint (text/plain format), following the {namespace}_{subsystem}_{name} naming convention with a unified rtc_ prefix.
Automatically collected for all SQL operations via a GORM plugin.
Metric Type Labels Description rtc_db_queries_totalCounter operation (select/insert/update/delete/raw), status (success/error)Total database queries rtc_db_query_duration_secondsHistogram operationDatabase query duration distribution (buckets: 1ms ~ 4.096s)
💡 These metrics help identify slow queries and database bottlenecks. The operation label distinguishes CRUD operation types.
Tracks work items from publish to completion.
Metric Type Labels Description rtc_queue_publish_totalCounter status (success/error)Total work items published rtc_queue_claim_totalCounter status (success/empty/error)Total work items claimed rtc_queue_complete_totalCounter status (success/error)Total work items completed rtc_queue_cancel_totalCounter reasonTotal work items cancelled rtc_queue_wait_duration_secondsHistogram — Wait time (publish to claim) rtc_queue_processing_duration_secondsHistogram — Processing time (claim to completion)
💡 Queue metrics are essential for troubleshooting Worker load and task latency. wait_duration reflects scheduling efficiency; process_duration reflects processing capacity.
Monitor circuit breaker state for WebFetch and other modules.
Metric Type Labels Description rtc_circuitbreaker_state_changes_totalCounter name, from_state, to_stateTotal state changes rtc_circuitbreaker_stateGauge name, state (closed/open/half_open)Current state (1=active, 0=inactive)
Monitor real-time connections, RPC calls, and channel subscriptions.
Metric Type Labels Description rtc_centrifuge_connections_totalCounter status (connected/disconnected)Total connection events rtc_centrifuge_disconnect_reason_totalCounter reason (slow/normal/error)Disconnect reason distribution rtc_centrifuge_rpc_requests_totalCounter method, status (success/error)Total RPC calls rtc_centrifuge_rpc_duration_secondsHistogram methodRPC call duration distribution (buckets: 1ms ~ 4s) rtc_centrifuge_subscriptions_totalCounter channel_type (user/topic/live)Total channel subscriptions rtc_centrifuge_connecting_duration_secondsHistogram status (success/error)JWT verification duration (OnConnecting phase)
Metric Type Labels Description rtc_auth_failures_totalCounter type (jwt/oauth/centrifuge), reason (expired/invalid/missing/signature/claims)Total authentication failures
💡 A sudden spike in auth failure rate may indicate a brute-force attack or misconfiguration.
Track core business events (Session lifecycle and message sending).
Metric Type Labels Description rtc_session_created_totalCounter — Total sessions created rtc_session_closed_totalCounter reason (normal/error/stopped_by_parent)Total sessions closed (by reason) rtc_messages_sent_totalCounter type (user/assistant/system)Total messages sent (by type)
Track Turn (AI reasoning round) execution.
Metric Type Labels Description rtc_turn_totalCounter work_kind, statusTotal turns executed (by work kind and status) rtc_turn_duration_secondsHistogram — Turn execution duration distribution (buckets: 0.1s ~ 204.8s)
💡 A Turn is a complete AI reasoning round (from LLM call to tool execution completion). work_kind distinguishes different work types, status distinguishes success/failure.
Track LLM API calls and token consumption.
Metric Type Labels Description rtc_llm_tokens_totalCounter model, type (input/output)Total LLM tokens consumed rtc_llm_request_duration_secondsHistogram — LLM API request duration distribution (buckets: 0.5s ~ 256s) rtc_llm_http_requests_totalCounter — Total LLM HTTP requests rtc_llm_http_request_duration_secondsHistogram — LLM HTTP request duration distribution rtc_llm_http_tokens_totalCounter — Total LLM HTTP transferred tokens
💡 rtc_llm_tokens_total is the core metric for cost accounting. The type label distinguishes input vs output tokens; combined with the model label, you can calculate per-model consumption ratios.
Metric Type Labels Description rtc_interrupt_totalCounter reasonTotal interrupts (by reason)
Track RTC Checkpoint save/restore operations.
Metric Type Labels Description rtc_checkpoint_operations_totalCounter — Total checkpoint operations (save/restore) rtc_checkpoint_data_bytesCounter — Total checkpoint data size (bytes)
Track multimodal attachment (images, text files) processing.
Metric Type Labels Description rtc_attachment_tokens_totalCounter — Total attachment tokens consumed (tokens injected into LLM after image base64 encoding) rtc_attachment_build_duration_secondsHistogram — Attachment build duration distribution rtc_attachment_operations_totalCounter — Total attachment operations rtc_attachment_budget_truncated_totalCounter — Total attachment budget truncations (attachments truncated due to budget overflow)
Track script execution.
Metric Type Labels Description rtc_script_executions_totalCounter — Total script executions rtc_script_execution_duration_secondsHistogram — Script execution duration distribution rtc_script_result_size_bytesHistogram — Script result size distribution rtc_script_code_size_bytesHistogram — Script code size distribution
💡 Script metrics complement Script Observability logs — metrics are for trend monitoring, logs are for detailed investigation.
Metric Type Labels Description rtc_recovery_stale_turns_recovered_totalCounter — Total stale turns recovered (stuck turns automatically recovered after server restart)
Metric Type Labels Description rtc_ratelimit_rejected_totalCounter — Total OAuth2 rate limit rejections rtc_ip_ratelimit_rejected_totalCounter — Total IP rate limit rejections
💡 A spike in rate limit metrics may indicate a brute-force attack or abnormal client behavior.
Metric Type Labels Description rtc_goroutines_by_stateGauge state (running/runnable/syscall/waiting, etc.)Number of goroutines by state (sampled every 10 seconds)
💡 Complementary to go_goroutines (total count), this metric helps pinpoint the specific state of goroutine leaks — e.g., a large number of waiting goroutines typically indicates lock contention or IO blocking.
Metric Type Labels Description rtc_oss3_requests_totalCounter operation (PutObject/GetObject/…), statusTotal S3 requests rtc_oss3_request_duration_secondsHistogram operationS3 request duration distribution rtc_oss3_quota_usage_bytesGauge — Current quota usage (bytes) rtc_oss3_multipart_uploads_activeGauge — Active multipart uploads rtc_oss3_backend_errors_totalCounter operation, error_typeBackend (MinIO) error count rtc_oss3_upload_bytes_totalCounter bucketTotal bytes uploaded (by bucket) rtc_oss3_download_bytes_totalCounter bucketTotal bytes downloaded (by bucket) rtc_oss3_instant_upload_totalCounter type (db_hit/minio_repair/multipart)Total instant upload hits (by type) rtc_oss3_instant_upload_repair_error_totalCounter — Total instant upload DB repair failures rtc_oss3_consistency_violation_totalCounter type (db_has_minio_missing/minio_has_db_missing)Total DB/MinIO consistency violations rtc_oss3_orphaned_record_totalCounter operation (delete_failed/copy_failed)Total orphaned records (backend deleted, DB delete failed) rtc_oss3_orphaned_quota_commits_totalCounter — Total orphaned quota commits (upload succeeded, quota commit failed, reconciliation needed) rtc_oss3_quota_commit_retry_totalCounter — Total quota commit retries (transient Redis errors)
💡 OSS3 metrics cover the complete object storage lifecycle, including request statistics, quota management, instant upload optimization, consistency checks, and error tracking. instant_upload metrics help evaluate deduplication optimization effectiveness, and consistency_violation metrics monitor data consistency between DB and MinIO.
Metric Type Labels Description http_requests_totalCounter code, methodTotal HTTP requests (with status codes) http_request_duration_secondsHistogram — HTTP request duration distribution http_request_size_bytesHistogram — HTTP request body size distribution http_response_size_bytesHistogram — HTTP response body size distribution http_in_flight_requestsGauge — Current number of in-flight HTTP requests
Track built-in WebFetch tool (web scraping) requests.
Metric Type Labels Description rtc_agent_webfetch_requests_totalCounter statusTotal WebFetch requests rtc_agent_webfetch_errors_totalCounter error_typeTotal WebFetch errors rtc_agent_webfetch_duration_secondsHistogram domain_typeWebFetch request duration distribution rtc_agent_webfetch_cache_hits_totalCounter — Total WebFetch cache hits rtc_agent_webfetch_cache_misses_totalCounter — Total WebFetch cache misses
Metric Type Description go_goroutinesGauge Current Goroutine count go_memstats_heap_alloc_bytesGauge Heap memory usage (bytes) process_start_time_secondsGauge Process start time
The distributed deployment includes pre-configured Grafana (port 23001, default credentials admin/admin). The following dashboards are loaded automatically via Provisioning:
Dashboard Description Key Panels RTC Agent Core business dashboard Active sessions, Turn execution (duration/success rate), LLM calls (token consumption/request duration/HTTP transfer), Script execution (count/duration/result size), Interrupt/Checkpoint/Attachment metrics, Recovery stats; WebSocket/Centrifuge connections & RPC, Queue lifecycle, Auth & Security, Rate limiting, Circuit breaker state, Goroutine state distribution, Database queries, Business events (Session/messages) OSS3 Overview Object storage overview Upload/download throughput, Quota usage, Multipart upload status, Backend health MinIO Overview MinIO backend details Disk usage, S3 request rate, Node status HTTP Server HTTP layer monitoring Request rate, Latency distribution, Status code distribution Go Runtime Go runtime Goroutine count, GC pause times, Memory usage System Health Watchdog System health watchdog Composite health score, Resource usage trends Logs Log aggregation (Loki) Structured log queries, Error log filtering Error Feedback Overview Error feedback overview Error message creation rate, Reactive Compact status
Access Grafana: http://localhost:23001 (distributed deployment) or your custom port.
The distributed deployment includes pre-configured Alertmanager alert rules via etc/dev/prometheus/alerts.yml. Alerts are organized into the following groups:
Alert Condition Severity Description ServerDownup == 0 for 30scritical Service unreachable HighHTTPErrorRate5xx error rate > 5% for 2m warning HTTP error rate anomaly HighHTTPLatencyp95 latency > 2s for 5m warning HTTP latency too high LLMHighErrorRateLLM error rate > 10% for 2m warning LLM service anomaly HighTokenConsumptionRateToken consumption > 100k/s for 1m warning Token consumption anomaly HighReasoningTokenRatioReasoning token ratio > 50% for 5m warning Model configuration may be wrong GoroutineLeakGoroutines > 5000 for 10m warning Possible Goroutine leak HighMemoryUsageHeap memory > 1GB for 5m warning Memory pressure ContainerRestarting> 2 restarts in 10min critical Frequent container restarts ScriptExecutionHighErrorRateScript error rate > 10% for 2m warning Script execution anomaly ScriptExecutionSlowScript p95 latency > 30s for 5m warning Slow script execution
Alert Condition Severity Description StaleTurnRecoveryActiveStale turn recovery within 15min warning Auto-recovery triggered StaleTurnRecoveryHighRateRecovery rate > 0.1/s for 5m warning Many turns stuck ErrorMessageRateSpikeError message creation rate > 0.5/s warning Error message spike ReactiveCompactExhaustedAll Level-3 compactions failed critical Auto-recovery mechanism exhausted ErrorFeedbackSLABreachP99 latency > 10s warning Error feedback SLA breach
Alert Condition Severity CentrifugeHighErrorDisconnectRateerror disconnects > 20% warning CentrifugeRPCHighErrorRateRPC error rate > 10% warning CentrifugeRPCHighLatencyRPC p95 latency > 1s warning
Alert Condition Severity QueueHighWaitDurationWait time p95 > 30s warning QueueHighProcessingDurationProcessing time p95 > 5min warning QueueHighCompleteFailureRatecomplete failure rate > 10% warning QueueHighCancelRateCancel rate > 20% warning
Alert Condition Severity AuthFailuresSpikeAuth failures > 1/s for 2m warning
Alert Condition Severity CircuitBreakerOpenCircuit breaker in open state for 1m critical
Alert Condition Severity DBHighQueryLatencyQuery p95 latency > 1s warning DBHighQueryErrorRateQuery error rate > 5% warning DBUnavailableQuery error rate > 90% critical
Alert Condition Severity SessionHighErrorCloseRateSession error close rate > 20% warning
Alert Condition Severity MinIODownService unreachable for 30s critical MinIODiskUsageHighDisk usage > 85% warning MinIODiskUsageCriticalDisk usage > 95% critical MinIOHighErrorRateS3 request error rate > 5% warning MinIOHighLatencyS3 p95 latency > 5s warning MinIOOfflineNodesNodes offline critical MinIOQuotaExceededBucket usage > 10GB warning
Alert Condition Severity OSS3EndpointDownS3 endpoint unreachable critical OSS3HighUploadErrorRateUpload error rate > 5% warning OSS3HighDownloadLatencyDownload p95 > 5s warning OSS3QuotaNearLimitUser quota usage > 90% warning OSS3MultipartUploadBacklogActive multipart uploads > 1000 warning OSS3BackendErrorBackend error rate anomaly warning
Alert Condition Severity PostgresDownServer unreachable (possible dependency failure) critical PrometheusTargetMissingScrape target unreachable warning
In production, the /metrics endpoint requires Basic Auth — otherwise the endpoint is disabled:
password : " ${METRICS__PASSWORD} " # Configure via environment variable
⚠️ Production enforcement : Without credentials configured, the endpoint remains accessible in development (for local debugging) but is disabled in production.
Debug endpoints (pprof, goroutines) are disabled by default (enabled: false). To enable them, you must also configure authentication:
enabled : true # Default false; must be explicitly enabled
password : " ${DEBUG__PASSWORD} "
⚠️ In production, debug endpoints are disabled when authentication is not configured. Even if enabled: true, they will be turned off without authentication configuration.
The Server automatically enables the following security protections in production:
Mechanism Description OAuth2 IP Rate Limiting OAuth2 endpoints (/oauth2/authorize, /oauth2/token, /oauth2/refresh) are rate-limited to 5 req/s per IP with a burst of 10, preventing brute-force attacks HSTS Header Enforces HTTPS transport, preventing protocol downgrade attacks Permissions-Policy Header Restricts browser API access, reducing the attack surface Debug Endpoints Default Disabled Debug endpoints (pprof, goroutines) are disabled by default; must be explicitly enabled and configured with authentication Redirect URI Whitelist Production enforces OAuth2 redirect_uri whitelist validation, preventing open redirect attacks OSS3 SecretAccessKey Static Encryption Object storage SecretAccessKey is encrypted at rest in the database, preventing plaintext leakage Trusted Proxy Support Supports configuring trusted proxies to ensure audit logs and rate limiting correctly identify the real client IP
The distributed deployment comes with pre-configured Alertmanager (port 29093). Alert rules are loaded via Prometheus’s rule_files. To customize alert rules, edit etc/dev/prometheus/alerts.yml.
