TurboQuant

For PTQ to work, the quantizer needs to see what activations look like in practice. Standard practice is to feed 256–2048 sequences from a corpus that resembles the deployment distribution:

# Typical calibration source: C4, WikiText, or a domain corpus
from datasets import load_dataset
calib = load_dataset("allenai/c4", split="train", streaming=True)
samples = []
for ex in calib.take(512):
    tokens = tokenizer(ex["text"], truncation=True, max_length=2048)
    samples.append(tokens["input_ids"])

# Pass these through the model to gather:
#  - per-tensor activation max/min
#  - per-channel activation distributions
#  - second-order info (Hessian) for GPTQ-style methods

Domain mismatch is the most common quantization bug: calibrating on Wikipedia and deploying on code produces noticeable quality loss. Best practice — calibrate on a sample of the actual deployment traffic when possible.

Each weight matrix has a 2D shape (out_features, in_features). The scale factor can be applied at different granularities:

The overhead is the metadata — extra bytes for scales and (for asymmetric) zero points stored alongside the int4 weights. AWQ uses g=128 with one fp16 scale per group; GPTQ uses g=128 with both scale and zero-point.

Naive quantization quantizes every layer independently. GPTQ does layer-by-layer with error compensation:

def gptq_layer(W, calibration_inputs):
    # 1. Forward pass to get this layer's input X
    X = run_forward_to_layer(model, layer_idx, calib)

    # 2. Compute Hessian H = X.T @ X (input second moment)
    H = X.T @ X

    # 3. Quantize column-by-column
    for j in range(W.shape[1]):
        q = quantize_int4(W[:, j])
        err = W[:, j] - dequantize(q)
        # Spread error to remaining columns weighted by Hessian
        W[:, j+1:] -= err.unsqueeze(1) * H[j, j+1:] / H[j, j]
        W[:, j] = dequantize(q)
    return W

This is roughly equivalent to OBQ (Optimal Brain Quantization) without the full inverse Hessian, made fast via Cholesky factorization. The per-layer cost is dominated by H = X.T @ X, which is parallel-friendly on a GPU.

GGUF is the lingua franca for desktop/edge deployments — single-file, mmap-friendly, supports many bit widths within one file. AWQ and GPTQ are the GPU-server formats. MLX is Apple-only but increasingly popular for laptops. Most quantization toolkits support multiple exports from the same calibrated state.

Throughput numbers are approximate single-stream decode for Llama-3 70B at batch=1. Bigger batches further amortize the weight load and tilt toward compute-bound, where fp16 closes the gap.

• Perplexity on WikiText-2: classic, fast, but only measures language modeling. Aim for <1% increase vs fp16.
• MMLU, ARC, HellaSwag: knowledge / reasoning benchmarks. Should be within 1-2 points.
• Domain eval: if you'll deploy on code, run HumanEval. If chat, run MT-Bench.
• Output-distribution comparison: KL divergence between fp16 and quantized model logits on a held-out set.
• Long-context degradation: quantization often hurts more at >8k tokens. Test specifically.

# Quick PPL eval
from lm_eval.tasks import wikitext
ppl_fp16 = evaluate(fp16_model, wikitext)
ppl_int4 = evaluate(int4_model, wikitext)
delta = (ppl_int4 - ppl_fp16) / ppl_fp16
assert delta < 0.01, f"Quantization regression: +{delta*100:.1f}%"