- 2-bit experts with FP4 recovery: Routed experts are compressed to a sign-symmetric 2-bit codebook using custom SM120 SASS kernels, with a runtime FP4 tier that restores precision where it matters most.
- Tiered expert residency: When the model exceeds VRAM, weights move to pinned host RAM, and eventually to an NVMe pack file. This turns your GPU into an intelligent expert cache.
- A rebuilt serving base: The patch includes MTP speculative decoding, an NVFP4 KV cache (352 B/token), and agent-ready tool parsing.
This level of optimization is critical when you are scaling beyond limits with MoE architectures like LongCat.
How 2-Bit Experts Maintain FP4 Quality
Naive 2-bit quantization usually destroys model performance due to sign asymmetry. The researchers found that by forcing a sign-symmetric {-4, -1, 1, 4} codebook, they could eliminate the degenerate loops and bias accumulation that typically plague low-bit models.
The system uses an adaptive recovery method:
- 2-bit expert planes: The default fast path for inference.
- FP4 delta cache: A background process that keeps ‘hot’ experts at higher precision in VRAM.
- Confidence gate: If a decoding step has low confidence, the system re-runs it using the FP4 experts to ensure accuracy.
Deploying frontier Mixture-of-Experts (MoE) models usually requires a massive enterprise-grade cluster. For developers working with consumer Blackwell hardware, such as the RTX 5090 or workstation RTX PRO 6000, official checkpoints for models like GLM-5.2 (753B) are physically impossible to load because they simply cannot fit into available VRAM.
The vLLM-Moet implementation changes this reality. By using a 7.4k-line patch for vLLM v0.24.0, it enables massive models to run on hardware that would otherwise be completely overwhelmed.
The Solution: Three Pillars of Optimization
This breakthrough isn’t achieved through simple compression, but through a sophisticated three-pronged approach to memory and compute management.
- 2-bit experts with FP4 recovery: Routed experts are compressed to a sign-symmetric 2-bit codebook using custom SM120 SASS kernels, with a runtime FP4 tier that restores precision where it matters most.
- Tiered expert residency: When the model exceeds VRAM, weights move to pinned host RAM, and eventually to an NVMe pack file. This turns your GPU into an intelligent expert cache.
- A rebuilt serving base: The patch includes MTP speculative decoding, an NVFP4 KV cache (352 B/token), and agent-ready tool parsing.
This level of optimization is critical when you are scaling beyond limits with MoE architectures like LongCat.
How 2-Bit Experts Maintain FP4 Quality
Naive 2-bit quantization usually destroys model performance due to sign asymmetry. The researchers found that by forcing a sign-symmetric {-4, -1, 1, 4} codebook, they could eliminate the degenerate loops and bias accumulation that typically plague low-bit models.
The system uses an adaptive recovery method:
- 2-bit expert planes: The default fast path for inference.
- FP4 delta cache: A background process that keeps ‘hot’ experts at higher precision in VRAM.
- Confidence gate: If a decoding step has low confidence, the system re-runs it using the FP4 experts to ensure accuracy.
Performance Benchmarks: GLM-5.2 (753B)
The following table demonstrates how the 753B parameter model can be served across different hardware configurations using this tiered approach.
| Hardware Config | Mode | Max Context | Throughput |
|---|---|---|---|
| 4× RTX PRO 6000 (TP4) | 2-bit + MTP k=2 | 256K | 105 tok/s |
| 4× RTX PRO 6000 (TP4) | 2-bit + FP4 Delta | 128K | 83–85 tok/s |
| 2× RTX PRO 6000 (TP2) | Three-tier + NVMe | 128K | 28–32 tok/s |
Implementation Guide
Getting this running requires building the specific SM120-compatible Docker image. This process handles the complex kernel compilation required for the 2-bit GEMM operations.
# Clone the repository
git clone https://github.com/kacper-daftcode/vLLM-Moet && cd vLLM-Moet
# Build the Blackwell-optimized image
DOCKER_BUILDKIT=1 docker build -f Dockerfile.sm120-v024 -t vllm-moet-sm120:v024 .Once deployed, you can serve models like DeepSeek-V4-Flash (159B) on a single RTX 5090. This brings frontier-class intelligence much closer to being accessible for local development and advanced RAG pipelines without needing a data center.
Final Thoughts
The vLLM-Moet project represents a massive leap for local AI, proving that with clever kernel engineering and tiered memory management, the gap between consumer hardware and cloud-scale frontier models is closing faster than we expected.
Ready to push your hardware to the limit? Clone the repo and start testing these weights on your Blackwell setup today!
