server
Local inference flexibility. The RPC distributed model server handles requests seamlessly when the router tier is set to local.
Backend Agnostic
The hirn server doesn't force you into a single execution engine. It provides a unified RPC interface that seamlessly connects to multiple powerful backends:
- llama.cpp: For highly optimized CPU/GPU offloading on consumer hardware.
- vLLM: For high-throughput serving on larger VRAM setups.
- exo / cactus: For experimental and specialized local routing architectures.
Distributed VRAM Management
Running large models often requires more VRAM than a single machine holds. The server intelligently pools and manages VRAM across multiple GPUs or even multiple devices on your local network, treating your home lab as a single unified inference cluster.
Zero Setup
Starts alongside your agent automatically. No complicated Docker configurations or Python environments required for standard operation.
Cross-Platform
Natively optimized for Windows, Linux, and macOS, providing bare-metal performance regardless of your OS setup.
Auto-Scaling
Dynamically unloads idle models from VRAM to free up system memory, only loading them when a router request comes through.