Intelligent LLM gateway and VRAM-aware router for Ollama, llama.cpp, and OpenAI with semantic caching and auto-failover.
150·LLM Ops·Open source
Alternative to
tightwad
Open sourceSelf-hosted mixed-vendor GPU inference cluster manager with speculative decoding proxy for faster LLM serving.
33MIT
Mixed-vendor GPU inference cluster manager with speculative decoding
Pros
- +Pools heterogeneous GPUs (CUDA, ROCm, CPU) into one cluster
- +Speculative decoding accelerates inference up to 2-3x
- +OpenAI-compatible API endpoint, drop-in replacement
- +Supports multiple modes: combined, proxy, multi-drafter consensus
Cons
- −Requires technical setup and hardware management
- −Network latency can limit speedup in some configurations
- −Limited to llama.cpp backends and specific model support
Target audience: Developers and hobbyists with spare GPUs who want to build a cost-effective, self-hosted LLM inference cluster.
Related tools
Other open-source tools that share tags with tightwad.
Run Hermes Agent and Claude Code locally on llama.cpp with zero API costs.
29·Self-Hosted AI·Open source
Alternative to
LocalAI
MCPMIT
Open-source AI engine to run LLMs, vision, voice, image, video on any hardware, no GPU required.
49k·Self-Hosted AI·Open source
Alternative to
93+ production-ready AI projects with tutorials on LLMs, RAG, and agents.
37k·Developer Tools·Open source
QwenPaw
MCPApache-2.0
QwenPaw: Your personal AI assistant, deployable locally or in the cloud, with memory, multi-agent support, and multi-cha
34k·AI Agents·Open source
Alternative to
Horizon
MCPMIT
AI-powered news radar generating daily bilingual briefings in English & Chinese.
9k·Other·Open source