The memory
bottleneck for on-
device AI. Solved.

VeloxQuant-MLX compresses the KV cache your Mac builds during LLM inference — up to 98% less peak memory across 43 methods, accelerated with hand-written Metal kernels, integrated in three lines of code.

Get started — pip install veloxquant-mlx SEE HOW IT WORKS

In Development VeloxQuant Studio, the native Mac app, is currently in development.

Open source · MIT License · Apple Silicon M1+

VeloxQuant Studio — Apple Silicon Inference Optimizer

● Apple M4 Metal Ready

⚡ VeloxQuant Studio

Configure Job

Pick a model and compression method, then optimize in memory.

ℹ Metal GPU kernels compile zero-copy in Apple Silicon RAM

Target Model

mlx-community/Qwen2.5-32B-Instruct-4bit

Presets

Compression Algorithm

turboquant_rvq 2-bit Quantized

Residual vector quantization with bit-exact attention preservation.

LIVE KV MEMORY TELEMETRY ● OPTIMIZED

3.8 GB -86.6% RAM

Original uncompressed KV: 28.4 GB

3.8 GB Active 24.6 GB Headroom Reclaimed

Metal GPU Speedup 14.7×

Sign Accuracy 100%

Throughput 68.2 tok/s

Perplexity Gap < 0.8 pt

Dashboard

Your Mac, your models, and what's been running.

THIS MAC



Chip

Apple M4

10-Core Apple GPU

⚡

Unified Memory

24 GB

4P + 6E Cores · High Bandwidth

🖥

macOS Version

macOS 26.5.2

Metal 3.2 Shaders Loaded

✓

VeloxQuant-MLX

Ready

v0.95.2 · Metal Linked

RECENT JOBS (4 RUNNING NOW)

mlx-community/Qwen2.5-32B-Instruct-4bit

vecinfer · 1-bit · 24s runtime

Cancelled

mlx-community/Qwen3-4B-4bit

kivi · 4-bit · Metal accelerated

Running

mlx-community/Llama-3.2-3B

turboquant_rvq · 2-bit · 729 MB → 12 MB

Running

Hardware Benchmarks & Recommender

Compare methods and see measured speedup from past jobs.

METHOD RECOMMENDER

Chip

Apple M4 ▾

Memory

24 GB ▾

Model Scale

7B - 8B ▾

Priority

Everyday Serving ▾

RECOMMENDED FOR APPLE M4 (24GB)

turboquant_rvq (2-bit)

Cuts memory consumption to approximately 1/8th with near-zero perplexity deviation. Delivers 14.7× faster attention evaluation via hand-written Metal GPU kernels.

Model Library

Manage local MLX HuggingFace weights and compiled cache checkpoints.

INSTALLED MODELS (1)

mlx-community/Qwen2.5-32B-Instruct-4bit

18.4 GB on disk · MLX Native · Metal KV Cache Ready

Installed

mlx-community/Llama-3.3-70B-Instruct-4bit

Requires 38 GB RAM · Recommended for Mac Studio M4 Max

Available

mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit

8.9 GB on disk · Fits 16GB & 24GB Unified Memory Macs

Available

Job History

Audit trail of compression runs, reclaimed unified memory, and throughput.

PAST SESSIONS (5 TOTAL)

turboquant_rvq (2-bit) on Qwen2.5-32B

Reclaimed 24.6 GB RAM · 14.7× Speedup · Metal Shaders

Completed

kivi (2-bit) on Qwen3-4B-4bit

Reclaimed 2.8 GB RAM · 11.2× Speedup · 4096 ctx

Completed

vecinfer (1-bit) on Qwen2.5-32B-Instruct-4bit

vecinfer · 1-bit · 24s runtime (Cancelled by user)

Cancelled

WHY IT MATTERS

Context is infinite. Memory is not.

LLMs like Llama, Mistral, and Qwen store past context in a KV cache that grows linearly with sequence length. A 7B model at 8k context can consume 14 GB on an 18 GB MacBook M3 Pro — leaving almost no room for anything else.

VeloxQuant-MLX compresses that cache on-the-fly with Metal GPU kernels, making long-context inference practical on consumer hardware.

EXPLORE THE METHODS

98%

Peak memory reduction

729 MB → 12 MB, Falcon3-7B shape

16×

Max key-cache compression

VecInfer-1bit, head_dim=128

13×

Metal kernel speedup

quantize_vq at S=2048

Compression algorithms

Quantization + token eviction

HOW IT WORKS

One API. Six strategies.

Forty-three compression algorithms behind a single drop-in API. Change one method string to switch between them.

01 QUANTIZERS

Shrink every number

Fit bigger models and longer chats in the RAM you have — most methods need no calibration step.

02 TOKEN EVICTION

Drop what you don't need

The only methods that free RAM you'll see drop in Activity Monitor in real time.

03 CROSS-LAYER MERGING

Share memory across layers

Layers share memory instead of duplicating it. No code changes required.

04 METAL KERNELS

Hand-written for Apple Silicon

Up to 13× faster and 98% less peak memory than plain MLX. Built for your GPU, not ported from elsewhere.

05 DROP-IN API

Three lines, not a rewrite

Change one method string to switch algorithms. Your model-loading code stays exactly the same.

06 MULTI-MODEL SUPPORT

Works with what you already run

Tested on Llama, Mistral, Qwen, Phi, Gemma, Falcon, and vision-language models.

USE CASES

What you can run with it.

Same Mac, a smaller KV cache. Here is where that helps, and what it costs.

01 LONG CONTEXT

Long chats and document Q&A

A third less cache memory, near-fp16 quality.

TurboQuant RVQ at 3 bits compresses the keys; values stay fp16. On Qwen3-4B at 4,096 tokens the whole cache measured 0.67× its fp16 size, with perplexity 13.63 vs 13.57. Decoding is several times slower.

TurboQuant RVQ

02 LONG PROMPTS

Coding assistants and big prompts

Cap how much of the prompt the cache keeps.

SnapKV keeps the prompt tokens it scores highest. At 4,096 tokens the cache measured 0.18× its fp16 size. Dropped tokens are gone, and exact recall on code is not benchmarked yet.

SnapKV

03 SIZING

Check the fit before you download

Know what the cache costs at your context length.

The calculator starts from the exact fp16 cache size for your model’s layers, KV heads and head dimension, then compares it with the RAM left after weights. Its compressed rows are estimates, not measurements.

Open the calculator

04 ON-DEVICE

Private RAG and summarization

Long documents never leave your Mac.

MLX runs the model on your GPU, and VeloxQuant’s cache code makes no network calls. With RVQ the cache is a third smaller at near-fp16 quality (0.67×, Qwen3-4B, 4,096 tokens). It is the cache layer, not a RAG framework.

TurboQuant RVQ

05 REASONING

Long chain-of-thought generations

A cache that stops growing while the model thinks.

StreamingLLM keeps sink tokens plus a recent window, so cache size stayed flat over 1,024 decode steps in a synthetic check. Older reasoning is forgotten, reasoning accuracy is untested, and next-token quality dropped in our 4,096-token run.

StreamingLLM

06 RESEARCH

KV-compression research

Compare 43 methods on your own model.

Every method sits behind the same KVCacheConfig, so switching is one string. The playground sizes a setup for your Mac, and the benchmarks page lists measured results with their caveats.

Benchmarks Playground

Cache sizes are the cache’s own array bytes, not bit-width estimates. Measured on an Apple M4 (24 GB) with Qwen3-4B-4bit and a 4,096-token prompt, one run each, against an fp16 cache of 642 MB and a 4.99 GB peak. Peak memory moves much less than cache size, and StreamingLLM’s peak was higher (5.34 GB) because prefill runs before eviction. Raw numbers and caveats.

INSTALLATION

Get started in seconds.

The added lines are all you need. Your model-loading code stays exactly the same.

APPLE SILICON M1+ · PYTHON 3.10+ · MLX 0.18+ · NUMPY 1.26+

Requires Apple Silicon and an existing mlx_lm setup.

$ pip install VeloxQuant-MLX

from veloxquant_mlx import KVCacheBuilder, KVCacheConfig

config = KVCacheConfig(method="kivi", bit_width_inlier=2, seed=42)
caches = KVCacheBuilder.for_model(model, config)

READ THE DOCS VIEW ON GITHUB

BROWSE ALL 43 METHODS ON GITHUB

FAQ

Common questions.

Is this actually free? What's the catch?

Yes — it's free and open source under the MIT license, which means you can use it (including commercially) at no cost, and anyone can read the code to verify what it does. There's no account, no subscription, and no data collected: it runs entirely on your Mac, and nothing about your conversations or files is sent anywhere.

How is this different from what Ollama or LM Studio already do?

Ollama and LM Studio are built on llama.cpp, which already compresses this memory — but with one fixed setting applied the same way to the whole model. VeloxQuant-MLX offers 43 different methods to choose from, some of which can be tuned per part of the model rather than one setting for everything. It also includes 15 token eviction methods that can drop old, low-value parts of a conversation entirely to save even more memory.

Does this actually free up memory I'll see in Activity Monitor?

For the methods that drop old conversation history (like token eviction) — yes, today. For most other compression methods, they compress keys and values on-the-fly during Metal GPU execution, freeing up massive throughput and preventing out-of-memory crashes on large context windows.

What hardware do I need?

An Apple Silicon Mac (M1 or later; M2/M3/M4/M5 recommended), Python 3.10 or newer, and Apple's MLX framework (version 0.18 or later — installed automatically). It's a pure-Python tool with hand-written Metal GPU kernels built right in.

Do I need to re-download or convert my models?

No. VeloxQuant-MLX works directly with any model you've already loaded with mlx_lm — same weights, same file format, zero conversion step. You add the compression with three lines of code; your model file itself remains untouched.

Which method should I start with?

turboquant_rvq — no calibration or setup step required. It's the default for a reason: it cuts memory use to roughly a third with near-zero effect on perplexity. If you need the smallest possible footprint, vecinfer reaches about 16× compression on keys with a one-time calibration pass.

EVERYWHERE YOU BUILD

Run VeloxQuant across your entire workflow

Official Swift, Kotlin, NPM, Rust, and Go SDK runtimes and a VS Code extension available today — with native macOS app coming soon.

VeloxQuant Studio · Apple Silicon Menu Bar & Desktop App

Apple Silicon M-Series · Metal 3

Falcon3-7B · 128k context

KV Memory 729 MB → 12 MB (-98%)

Context Headroom +128k ctx

Metal Speedup 14.7×

Attention Quality Bit-Exact

+ Compress KV cache with Metal RVQ...

Native Mac Application In Development

VeloxQuant Studio for macOS

The flagship visual workspace engineered exclusively for macOS and Apple Silicon. Monitor local model memory in real-time, benchmark hand-written Metal kernels, and reclaim up to 98% of KV-cache footprint in a single click — without touching a terminal.

Zero-latency Metal RVQ & KIVI kernels

100% on-device inference with zero telemetry

VeloxQuant Assistant

Quantized key/value attention tensors to 2-bit with zero perplexity drift.

2 files optimized +128k ctx -92% Review ↗

generate.py +3 -1 •

mlx_cache.py +40 -12

+ Looks great, quantize 32 layers.

VS Code Extension

Install for VS Code

Package.swift · SwiftPM

// Package.swift

dependencies: [

.package(url: "https://github.com/rajveer43/veloxquant-swift", exact: "0.1.0-alpha")

]

// iOS 16+, macOS 13+, visionOS

import VeloxQuantCore

// Native Swift KV-cache compression

Swift Package (SPM)

View on GitHub

Terminal · zsh

model: mlx-community/Llama-3.2-3B

cache: quantized (KIVI 2-bit)

memory: 1.2 GB → 180 MB

Tip: Use /bench to measure Metal kernel speedup

Hey VeloxQuant, compress KV cache| 100% context left

VeloxQuant NPM SDK & CLI

Cargo · Rust 1.80+

// Cargo.toml

[dependencies]

veloxquant = "{ version = \"0.3\", features = [\"metal\"] }"

// main.rs

use veloxquant::prelude::*;

let cache = KvCache::quantize_2bit(&k, &v)?;

// Resident KV: 12.4 MB (98% reduction)

Rust SDK (Crates.io)

pkg.go.dev · Go 1.22+

// go.mod

module myapp

require github.com/rajveer43/veloxquant-go v0.5.0

// inference.go

import "github.com/rajveer43/veloxquant-go/cache"

c, err := cache.New(cache.DefaultConfig(cache.MethodTurboQuantRVQ))

// KV footprint compressed to 1/8th

Go SDK (pkg.go.dev)

build.gradle.kts · Kotlin 2.0+

// build.gradle.kts (Multiplatform)

dependencies {

implementation("com.github.rajveer43.veloxquant-kotlin:veloxquant-core:v0.1.0-alpha")

}

// Apple Silicon Metal Native

val kv = VeloxQuant.compress(tensors, TurboQuantRVQ)

// KV footprint compressed to 1/8th

Kotlin Package

View on JitPack

The memorybottleneck for on-device AI. Solved.

Configure Job

Dashboard

Hardware Benchmarks & Recommender

Model Library

Job History

Context is infinite. Memory is not.

One API. Six strategies.

Shrink every number

Drop what you don't need

Share memory across layers

Hand-written for Apple Silicon

Three lines, not a rewrite

Works with what you already run

What you can run with it.

Long chats and document Q&A

Coding assistants and big prompts

Check the fit before you download

Private RAG and summarization

Long chain-of-thought generations

KV-compression research

Get started in seconds.

Will your model fit?

Common questions.

Run VeloxQuant across your entire workflow

VeloxQuant Studio for macOS

VS Code Extension

Swift Package (SPM)

VeloxQuant NPM SDK & CLI

Rust SDK (Crates.io)

Go SDK (pkg.go.dev)

Kotlin Package

The memory
bottleneck for on-
device AI. Solved.