Compress your LLM's KV cache 33x with zero training
python
dev.to
Running out of GPU memory at long context lengths? The KV cache grows linearly with sequence length — at 128K tokens, a 7B model accumulates over 60 GB of KV state. That's more than a single A100. I built NexusQuant, a library that compresses the KV cache 10-33x at inference time. No training, no calibration data, no model changes. Before # OOM at 32K tokens on a 24GB GPU output = model.generate(input_ids, max_new_tokens=512) After from nexusquant import nexus