Sockless LLM Router

llama.cpp's own server only speaks a subset of the OpenAI API. LLM Router adds full translation layers on top, exposing OpenAI-compatible (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/models), Claude-compatible (/v1/messages), and Ollama-compatible (/api/chat, /api/generate, /api/tags, /api/embed, /api/ps) endpoints in front of the same backends — including a stateful OpenAI Responses API, bidirectional tool/function-call translation across all three protocols, per-request reasoning effort, and real-time reasoning/thinking-token streaming.

In practice, this means tools that only support one provider's API can still reach a local model. For example, pointing Visual Studio's GitHub Copilot at its Ollama model provider and aiming it at LLM Router lets Copilot drive a local llama.cpp-hosted model — even though llama.cpp itself doesn't speak Ollama's API.

A single live view of the whole inference fleet: every server with its engine, loaded model, and port, plus live slot usage and KV-cache context fill pulled from each backend. Requests, tokens, and average latency for the last 24 hours sit alongside a recent-activity feed, and one click starts, stops, or edits any server.

Register llama.cpp and Strata servers side by side, each bound to a managed engine install, with per-server logs and start, stop, restart, edit, and delete actions. Every backend runs under its own lightweight wrapper process, so the router itself can restart or upgrade without killing running models — and models can be live-swapped on a running server.

Background health checks route around failed backends, and requests are capped at each server's slot count with context-aware queuing, so a busy model queues instead of overflowing its KV cache.

No more hunting for the right llama-server binary. The Engines page installs official llama.cpp releases for your backend, or compiles from source using built-in recipes for CPU, CUDA, Vulkan, SYCL, HIP/ROCm, OpenVINO, CANN, MUSA, and Metal — or your own custom recipe.

Each install tracks its version, size, and which servers use it. Check for updates against upstream and update in place: servers stay bound to their build, so they always know which engine version they're running.

Strata runs large Mixture-of-Experts models such as Qwen3.8-Flash-Next on consumer GPUs by offloading dormant experts to system RAM. LLM Router installs Strata releases with their CUDA, CUDA 12, or AMD engine (checked against GitHub's published SHA-256), creates the Python environment, and prepares each model on its first start — with progress shown right in the server status.

Strata servers then behave just like llama.cpp ones: the same presets, routing, slot-based concurrency, and dashboard telemetry. Installs can be updated, rolled back, or added from an existing Strata folder.

Presets are reusable launch configurations covering the full llama.cpp option set — sampling, context, GPU/threading, flash attention, speculative decoding, LoRA, multimodal projectors, chat-template kwargs, and more — with a live VRAM estimate as you edit. The router auto-starts or restarts the server that owns a preset, so the requested model is always available.

Model aliases with * wildcards (e.g. gpt-4o or claude-*) map clients with hardcoded model ids onto local presets, and a fallback preset chain takes over when a preset's server is errored or at capacity.

Servers that share a GPU can be placed in a memory group with a VRAM budget. Before a model loads, the router stops the least-recently-used idle servers in the group until the new model's estimated footprint fits — so you can keep many models on call without manually juggling what's loaded.

Servers can also unload automatically after a set number of idle minutes, freeing the GPU for games or other work.

A local registry of GGUF models, grouped by repository, with size, status, and GGUF metadata inspection. Search and download straight from the Hugging Face Hub with live progress — split models download as a single entry — or import and scan existing files on disk. Multimodal projectors are detected automatically, and models can be updated in place.

Chat with any preset directly from the dashboard and see per-reply time to first token, prompt and generation speed, cached tokens, and speculative-decoding acceptance. Built-in prompts cover quick checks, long generations, long prompts, and code.

Turn on Record runs to save each reply with a snapshot of the preset's launch settings, tag runs with a label like fa=on ub=1024, and repeat each message to average out noise. The Run History page groups runs by preset, label, and settings hash so different configurations can be compared side by side and exported as CSV.

Real-time charts — pushed over SignalR, no polling — for prompt processing speed, token generation speed, context-window usage, and request latency, filterable by time range (1h/6h/24h/7d), plus per-API-key usage with a token breakdown.

Every request routed through the gateway is logged with protocol, endpoint, model, latency, and token counts — filterable and searchable for debugging and auditing, with timestamps shown in local time or UTC.

Issue scoped API keys to gate the OpenAI, Responses, Claude, and Ollama endpoints — restrict a key to specific models, disable it without deleting it, or regenerate it on demand. Enforcement is a single toggle in Settings and stays off by default for local-only use.

Detects every GPU and NPU on the host via WMI and shows VRAM, vendor, and which compute runtimes (CUDA, Vulkan, SYCL, OpenVINO) are likely available — with a one-click disable that flips the device off in Windows Device Manager for every application, not just the router.

Tune request logging (full payload capture, retention period with hourly auto-cleanup) and gateway behavior — API key enforcement, queue size and timeout — plus an optional dual-port mode that splits routing traffic onto its own port, separate from the admin UI.

Each release ships two Windows (x64) builds:

• Self-contained (…-win-x64-self-contained.zip) — includes the .NET runtime; just unzip and run.
• Framework-dependent (…-win-x64.zip) — a much smaller download that needs the .NET 10 runtime installed.

Unzip and start LR.Application.exe to run it from a console, or run the bundled install-service.ps1 as Administrator to install it as a Windows Service. The database is created on first start; servers, engines, presets, and models are all set up from the dashboard.