TurboQuant

PROBLEM: LLM inference costs $10k/month because KV cache eats up memory.

SOLUTION: I squeezed it 8.5 times. The quality is the same.

PROOF:

256MB → 30MB

8.48x faster

$10k → $1.2k per month

HOW: Orthogonal conversion from Google DeepMind is used.

RESULT: Works with Mixtral, DeepSeek, Qwen. MIT license. Free.

github.com/RemizovDenis/turboquant

Questions? I'm here.

TurboQuant-MoE v0.3.0 released! • Up to 15.4× KV-cache compression • Cross-layer delta + 3-bit PolarQuant Serious VRAM killer for MoE models (Mixtral, DeepSeek, Qwen etc.)