LLM Quantization

Quantization maps a real number x to an integer q via a scale (and optionally a zero point):

# Symmetric (zero point = 0)
scale = max(|w|) / 127
q     = round(w / scale)             # int8 in [-127, 127]
w_hat = q * scale                    # dequantized value

# Asymmetric (better for ReLU outputs that are all positive)
scale = (max(w) - min(w)) / 255
zp    = round(-min(w) / scale)       # zero point
q     = round(w / scale + zp)        # uint8 in [0, 255]
w_hat = (q - zp) * scale

One scale per tensor is cheapest but loses range. Per-channel uses one scale per output row of a weight matrix — the standard. Per-group further subdivides each row into groups of 64 or 128, with a separate scale per group. AWQ and GPTQ both use group size 128.

# LLM.int8 mixed-precision matmul:
# 1. Detect outlier columns of activations (> 6.0 absolute value)
# 2. Compute outlier columns in fp16 (small fraction)
# 3. Compute non-outlier in int8 (vast majority)
# 4. Sum results

mask_outlier = (|x|.max(dim=0) > 6.0)
y_int8  = x[:, ~mask_outlier].int8() @ W[~mask_outlier].int8()
y_fp16  = x[:, mask_outlier]         @ W[mask_outlier]
y       = y_int8.dequantize() + y_fp16

Why outliers matter: a few feature dimensions in transformers carry huge magnitudes (the classic 1.7% of features have ~10× higher activations). Standard int8 forced a giant scale factor across all features, destroying precision for the small ones. Splitting them out preserves the small features.

GPTQ (Frantar et al. 2023) does layer-by-layer quantization, using second-order info (the Hessian) to pick which weights to quantize first and how to compensate for error in the remaining weights. Output: about 0.3% perplexity loss for int4 on Llama-2 70B.

# GPTQ pseudocode for one layer
H = compute_hessian(calibration_data)           # input outer-product
for col in weight_cols(W):
    q_col = quantize_int4(col)
    err   = (col - dequant(q_col))
    # Update remaining columns to compensate for this error
    W[:, col+1:] -= err * H[col, col+1:] / H[col, col]

AWQ (Activation-aware Weight Quantization, Lin et al. 2023) takes a different angle: not all weights matter equally. The 1% of weights connected to the largest-magnitude activations dominate output quality. Scale those weights up before quantization (and scale activations down to compensate), so the precision goes where it counts.

ExLlama / ExLlamaV2 are inference engines focused on int4 throughput on consumer GPUs (RTX 3090/4090). They use a custom CUDA kernel that fuses dequantization with matmul and supports mixed bit widths per tensor (some layers at 6-bit, some at 4-bit) for accuracy/speed tuning.

NVIDIA's H100 introduced two 8-bit floating-point formats:

fp8 vs int8: floating point preserves dynamic range. Integer 8-bit struggles with outliers in attention activations; fp8's exponent absorbs them naturally. Combined with per-tensor scaling (one scale factor per tensor, computed on-the-fly), fp8 inference matches fp16 to within 0.5% on most benchmarks — and runs 2× faster on H100.

Frameworks: TensorRT-LLM, vLLM (recent), Transformer Engine (NVIDIA's fp8 training library).

Microsoft Research, 2024. Replace every weight with a value in 1. log₂(3) ≈ 1.58 bits per weight.

# Quantization-aware training, not post-hoc
def bitnet_quantize(W):
    scale = mean(|W|)
    W_q   = round_clip(W / scale, -1, +1)        # ternary
    return W_q, scale

# Forward pass: matmul becomes adds and subtractions
# (no multiplications for the weight matmul itself)
# Activations stay int8.

The trick: BitNet is trained from scratch with quantization-aware training (QAT), not retrofitted onto a fp16 model. Surprising result: at 3B+ parameters, BitNet b1.58 matches fp16 LLM perplexity. Below 3B, it noticeably underperforms.

Hardware implication: ternary weights enable matmul-free inference (lookup tables and adders only). No real production silicon yet, but FPGA and custom ASIC interest is high.

Practitioners almost always use PTQ. QAT is reserved for extreme bit widths (≤2) where PTQ collapses, or for end-to-end edge deployments where model size dictates everything.