Backend Architecture
RTC Agent consists of two Go services: the Main Server handles AI reasoning orchestration, context management, real-time communication, tool call scheduling, and object storage; the Admin-server is an independent management service providing administrator authentication, user management, and other functions, integrated with the Main Server via RFC 8693 Token Exchange.
Service Layers
Section titled “Service Layers”| Layer | Responsibility | Key Modules |
|---|---|---|
| 🌐 Access Layer | Protocol adaptation, authentication & authorization | WebSocket Gateway, OAuth2 Handler, Token Exchange |
| 📋 Use Case Layer | Business orchestration, RPC processing | Session / Message / Turn / RTC use cases |
| 🤖 Domain Layer | AI reasoning, state management | Agent Engine, Context Management, Memory System |
| 💾 Infrastructure Layer | Data persistence, message passing, object storage | PostgreSQL, Redis, Centrifuge, MinIO/S3 |
| 🔐 Admin-server | Administrator authentication, user management | JWT signing (RS256/ES256/EdDSA), JWKS endpoint, Admin UI |
Authentication System: The Main Server uses OAuth2 authorization code flow for end users; Admin-server uses email/password login for service administrators. Administrators obtain JWTs from Admin-server, which can be recognized by the Main Server via RFC 8693 Token Exchange. See Authentication Flow and HTTP API.
Core Components
Section titled “Core Components”WebSocket Gateway
Section titled “WebSocket Gateway”The Gateway is the entry point for frontend-backend communication, managing all WebSocket connections and RPC routing.
| Responsibility | Description |
|---|---|
| Connection Management | Handles WebSocket connection establishment, authentication, heartbeats, and disconnection |
| RPC Routing | Dispatches the 17 RPC methods to their corresponding use case handlers |
| Event Push | Pushes events generated by the Agent to the frontend via Centrifuge |
| RTC Relay | Forwards AI tool call requests to the frontend and receives execution results |
17 RPC Methods by Category:
| Category | Methods |
|---|---|
| Session | v1.session.list, v1.session.get, v1.session.open, v1.session.close, v1.session.update, v1.session.fork, v1.session.compact |
| Message | v1.message.send, v1.message.list, v1.message.get |
| Turn | v1.turn.list, v1.turn.get, v1.turn.stop |
| RTC | v1.rtc.list, v1.rtc.get, v1.rtc.update_status, v1.rtc.submit_result |
See WebSocket RPC.
Agent Engine
Section titled “Agent Engine”The Agent engine is the core of AI reasoning — assembling prompts, calling the LLM, handling tool calls, and managing the reasoning loop.
| Capability | Description |
|---|---|
| Reasoning Loop | Call LLM → Process response → Tool call → Continue reasoning, until the final reply is generated |
| Built-in Tools | File operations (ls/read/write/grep/find/script), sub-agent management (subAgent/listSubAgent/stopSubAgent), user interaction (askUser), goal management (goal), loop tasks (loop), etc. |
| Multimodal Understanding | Processes file attachments in user messages — images are loaded from OSS, preprocessed (EXIF correction, resizing, compression), and injected as base64 into LLM Vision; text files are UTF-8 validated, truncated if needed, XML-wrapped, and appended to Content (see below) |
| Streaming Output | Pushes LLM output to the frontend in real-time |
| Error Feedback | Automatically classifies errors when a Turn fails, generating structured ErrorContent messages pushed to the frontend (see below) |
| Sub-Agents | Complex tasks are automatically decomposed, with multiple specialized sub-agents working in parallel (see below) |
Error Feedback
Section titled “Error Feedback”When a Turn execution fails, the system automatically classifies the error and generates a structured error message, which the frontend can render as a user-friendly prompt with a retry button:
| Error Category | Description | Typical Scenario |
|---|---|---|
api | API call errors | Model returns an error, rate limit |
context | Context-related | Context too long, compression fails |
network | Network errors | Connection timeout, DNS failure |
permission | Permission errors | Authentication fails, session ownership mismatch |
stream | Stream processing errors | SSE interrupted, stream parsing fails |
system | System errors | Internal exception, database error |
timeout | Timeout errors | Reasoning timeout, tool execution timeout |
tool | Tool call errors | Tool doesn’t exist, parameter validation fails |
💡 Rate limit protection: Each session can have at most 20 error messages per hour, preventing error storms. The
debug.show_raw_errorsconfiguration controls whether raw error details are exposed to the frontend.
See ErrorContent Protocol Definition.
Sub-Agent Mechanism
Section titled “Sub-Agent Mechanism”When task complexity exceeds the capacity of a single reasoning pass, the Agent engine automatically decomposes the task and creates sub-agents for parallel processing:
- Independent context: Each sub-agent has its own session and context, no interference
- Parallel execution: Multiple sub-agents can call the LLM and tools simultaneously
- Result aggregation: The main Agent collects results from all sub-agents and generates the final reply
Context Management
Section titled “Context Management”Context management is responsible for building the complete prompt sent to the LLM, and automatically compressing it when conversations become too long.
Prompt Assembly
Section titled “Prompt Assembly”Compression Strategy
Section titled “Compression Strategy”When conversation length approaches the token limit, the system uses progressive compression:
| Strategy | Trigger | Compression Method | Cost |
|---|---|---|---|
| Microcompact | After tool calls | Cleans up earlier tool results, keeping only the most recent 5 | Zero |
| Auto Compact | Token reaches threshold | LLM summarizes earlier conversations (9-part structured summary) | One LLM call |
| Reactive Compact | Still over limit after Auto Compact | Progressive multi-level compression (3 levels, from zero cost to aggressive trimming) | Zero ~ one LLM |
| Session Memory Compact | When Auto Compact is triggered | Directly uses Session Memory as summary | Zero |
💡 Performance optimization: Strategic Cache Breakpoints set cache breakpoints at key positions in the message list, improving LLM prompt cache hit rate from 0% to 75% after compression, reducing input cost by approximately 69%.
See Context Management.
Memory System
Section titled “Memory System”A dual-layer memory architecture that gives AI both short-term and long-term memory.
Session Memory
Section titled “Session Memory”Independently maintained per session, used for long-conversation context compression:
| Category | Description |
|---|---|
| decision | Decisions and choices made by the user |
| context | Background information for the current task |
| progress | Completed work and progress |
| issue | Encountered problems and blockers |
| learnings | Lessons learned from the task |
- Capacity: Max 20 entries, approximately 12K tokens
- Extraction: Dual-track — background Agent automatic extraction + Agent proactive saving
- Usage: Source for Session Memory Compact summaries; 5 entries injected into context per conversation turn
User Memory
Section titled “User Memory”Cross-session long-term memory, storing user preferences and historical facts:
| Category | Description |
|---|---|
| user | User identity (role, expertise, preferences) |
| feedback | User feedback on working style |
| project | Ongoing projects, goals, constraints |
| reference | External resource pointers (URLs, docs, tickets) |
- Importance levels: low / medium / high / critical
- Capacity: Max 1000 entries
- Extraction: Agent proactive saving only
- Retrieval: Keyword full-text search + importance weighting → top 5
See Memory System.
RTC Handler
Section titled “RTC Handler”The RTC handler manages the full lifecycle of tool calls — from creating RTC records to saving checkpoints, pausing Turns, waiting for results, and resuming reasoning.
| Feature | Description |
|---|---|
| Checkpoint | Saves the current reasoning state to Redis, with a 24-hour TTL |
| Crash Recovery | Can recover from a checkpoint after server restart |
| Serial Execution | RTC calls within the same session are strictly serialized to avoid file conflicts |
| 100% Delivery | Frontend result submission retries indefinitely; idempotency is guaranteed |
Work Mode Permission Matrix
Section titled “Work Mode Permission Matrix”Different work modes have different confirmation strategies for tool calls:
| Tool Type | manual | edit | plan | auto | bypass |
|---|---|---|---|---|---|
| Read-only tools (ls/read/grep/find) | ✅ Auto | ✅ Auto | ✅ Auto | ✅ Auto | ✅ Auto |
| Write tools (write) | ⚠️ Confirm | ✅ Auto | ⚠️ Confirm | ✅ Auto | ✅ Auto |
| script tool | ⚠️ Confirm | ⚠️ Confirm | ⚠️ Confirm | ⚠️ Confirm | ✅ Auto |
See Work Modes.
Object Storage Layer
Section titled “Object Storage Layer”The OSS3 package provides S3-compatible object storage abstraction for managing user-uploaded file attachments (images, text, etc.).
Backend Interface
Section titled “Backend Interface”rtc-oss3.Backend defines S3-compatible object storage operations:
| Operation Category | Methods | Description |
|---|---|---|
| Basic Operations | PutObject / GetObject / DeleteObject / HeadObject | Object CRUD |
| Batch/Range | DeleteObjects / GetObjectRange / ListObjects / CopyObject | Batch delete, range download, list, copy |
| Multipart Upload | CreateMultipartUpload / UploadPart / CompleteMultipartUpload / AbortMultipartUpload / ListParts | Large file multipart upload |
| Presigned URLs | PresignGet / PresignPut | Generate presigned URLs (server-side proxy upload/download) |
| Lifecycle | HealthCheck / Close | Health check, resource cleanup |
💡 The current implementation is a MinIO backend, but the interface design is compatible with all S3-protocol storage services including Alibaba Cloud OSS and AWS S3. Switching backends only requires changing the constructor—no business code modifications needed.
HTTP API
Section titled “HTTP API”| Endpoint | Authentication | Purpose |
|---|---|---|
POST /api/credentials/temporary | JWT | STS temporary credentials — clients obtain temporary S3 credentials for direct upload/download |
POST /api/presigned-url | JWT | Presigned URLs — server generates signed GET/PUT URLs with configurable expiry (default 1 hour, max 7 days) |
PUT/GET/DELETE/HEAD /{bucket}/{key} | SigV4 signature | S3-compatible API — clients use temporary credentials to directly manipulate objects |
POST /{bucket}?delete | SigV4 signature | Batch delete objects |
| Multipart upload operations | SigV4 signature | Large file multipart upload (?uploads, ?partNumber, ?uploadId) |
💡 Two upload modes: (1) STS temporary credentials mode — clients obtain temporary AK/SK/Token, then use standard AWS SDK to directly operate S3; (2) Presigned URL mode — server generates signed URLs, clients PUT/GET directly without SDK.
File Attachment Processing
Section titled “File Attachment Processing”File attachments (FileAttachment) in user messages undergo end-to-end processing before being injected into LLM context.
| File Type | Processing Pipeline | Injection Method |
|---|---|---|
| Image (JPEG/PNG/BMP/TIFF/SVG) | OSS read → Magic bytes detection → EXIF auto-orientation → Resize to 2000x2000 bounding box → Multi-level JPEG compression (80→60→40→20) → base64 encoding (≤ 5MB) | MultiContent Image part, directly understood by LLM Vision |
| Text | OSS read → UTF-8 validation → Truncation if over limit (256KB, UTF-8 safe boundary) | XML-wrapped <file_content name="..." type="...">, appended to Content |
| Other | Skipped (logged warning) | — |
💡 Parallel loading: Multiple file attachments are loaded in parallel using goroutines, reducing latency in multi-file scenarios. WebP/GIF formats are rejected due to decoding library limitations—users should convert to JPEG/PNG.
Admin Dashboard
Section titled “Admin Dashboard”The Admin Dashboard is a single-page application (SPA) built with Ant Design Pro, served as static assets by the admin-server. Administrators access the dashboard via browser, authenticate with email and password, and manage users and system state.
| Feature | Description |
|---|---|
| Tech Stack | React + Ant Design Pro + Umi routing |
| Authentication | Email/password login, Access Token + Refresh Token dual-token mechanism with auto-refresh |
| Token Security | Access Token stored in memory only (XSS prevention), Refresh Token via httpOnly cookie or secure storage |
| RTC Agent Integration | After login, automatically mounts <rtc-agent> as a global floating widget using Token Exchange mode (admin JWT → RTC JWT) |
| Deployment | SPA static assets are embedded into admin-server binary at compile time, accessible via /admin/ path |
💡 RTC Agent Widget: After admin login, RTC Agent floats above the dashboard as a global widget, allowing administrators to interact with AI directly from the admin interface. The widget uses Token Exchange mode to automatically exchange the admin-server JWT for a Main Server RTC JWT—no separate login required.
Real-Time Communication Layer
Section titled “Real-Time Communication Layer”| Channel | Purpose | Characteristics |
|---|---|---|
| Topic | State change events (Session, Message, Turn, RTC) | Persisted, offset tracking, offline recovery |
| Live | Streaming output intermediate chunks | Non-persistent, Redis PUB/SUB, low latency, lossy |
Offset Mechanism & Offline Recovery
Section titled “Offset Mechanism & Offline Recovery”- Brief disconnection (< 5 seconds): Missing events are automatically pushed after reconnection
- Long disconnection: Client detects offset gap and proactively pulls history
- Epoch change: After server restart, epoch changes; client resets offset and pulls full state
Next Steps
Section titled “Next Steps”- Frontend Architecture — Learn about the Web Components system
- Architecture Overview — Return to the architecture panorama
- Remote Tool Calling — Learn about the full RTC tool call mechanism
- WebSocket RPC — Learn the detailed RPC method definitions