Running frontier-class models typically requires an expensive fleet of H100 GPUs and hundreds of gigabytes of VRAM. For most developers and researchers, the hardware barrier for a 744B parameter model like GLM-5.2 seems impossible to overcome.
However, the Colibri engine changes the math entirely. It proves that you don’t need a data center to interact with massive Mixture-of-Experts (MoE) architectures; you just need a smart way to manage your disk I/O.
The Solution: Streaming Experts from Disk
The brilliance of Colibri lies in how it handles the 744B parameters of the GLM-5.2 model. Because a MoE architecture only activates a fraction of its weights per token, Colibri keeps the essential dense layers resident in your RAM while streaming the experts on demand from your NVMe drive.
By utilizing a per-layer LRU cache and leveraging the OS page cache as an L2, the engine maintains high efficiency. This approach is a massive leap forward compared to running frontier MoE models via vLLM, as it prioritizes accessibility on much smaller hardware footprints.
Technical Implementation
The engine is impressively lightweight, consisting of a single C file (approximately 2,400 lines) with no external dependencies or Python runtime required. It implements several advanced techniques to keep the memory footprint low:
- MLA Attention: Uses q/kv-LoRA and interleaved partial RoPE to reduce the KV-cache by 57x.
- Native MTP: Implements multi-token prediction speculative decoding to draft tokens efficiently.
- DeepSeek-style Routing: Uses a sigmoid router with shared experts for optimized token processing.
Performance and Hardware Reality
It is important to be clear: this is not built for high-speed throughput, but for accessibility. While the ‘cold’ decode speed is limited by your disk’s random read performance, the ability to run a model of this scale on a machine that costs less than a single H100 fan is revolutionary.
| Metric | Value (WSL2/NVMe) |
|---|---|
| Model Size on Disk (int4) | ~370 GB |
| Resident RAM (Dense, int4) | 9.9 GB |
| Peak RSS during chat | ~20 GB |
| Cold Decode Cost | ~11 GB disk reads/token |
| Disk Ceiling (Random) | ~1 GB/s |
When using hardware with higher bandwidth, such as an Apple M5 Max with unified memory, you can see much more usable latency, reaching upwards of 1.06 tokens per second.
Getting Started
You can find the full implementation and architecture details in the Colibri GitHub repository. If you want to test the weights immediately, the pre-quantized 4-bit version is available on Hugging Face.
Ready to experiment with local frontier models? Clone the repo and start streaming experts today!
Supporting the project
colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you’d like to support its development (better test hardware translates directly into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can:
- ⭐ star the repo and share it;
- 🐛 open issues with benchmark numbers from your hardware;
- 💬 reach out via GitHub issues if you’d like to sponsor development or donate hardware.