Allocating Quantization Precision Under a Memory Budget

Why People Wanted It

A local model has to fit before its capabilities are useful. The weight-size target here is below 20 GB, but reducing precision indiscriminately can change the predictions that reasoning and agentic tasks depend on.

So the real question was never just “how few bits can we use?” It was “which tensors can lose precision cheaply, and how do we check the resulting model?” This module concerns weight allocation; block drafting and verification are a separate decoding mechanism.

The Core Move

Treat each tensor’s storage format as a decision variable, then minimize prediction disagreement subject to a weight-memory budget.

Let $a_i$ be the format assigned to weight unit $i$, $m_i(a_i)$ its stored size, and $B$ the weight budget. The allocation problem can be written schematically as

$$\min_a D_{\mathrm{cal}}(a) \quad\text{subject to}\quad \sum_i m_i(a_i)\le B.$$

Here $D_{\mathrm{cal}}$ measures disagreement between the quantized model and the BF16 reference on calibration examples. BF16 is the reference floating-point representation; Q4_K, Q5_K, and Q6_K are the candidate K-quant formats. Stored size includes representation overhead, not just a nominal bit width multiplied by the number of weights.

Quiz

What is the central allocation rule behind Greedy-KL?

Keep all attention tensors at the highest precision.

Lower precision where calibration KL rises least for each byte saved.

Choose precision using runtime token acceptance alone.

Assign every tensor the same number of bits.

$ check

Why Greedy-KL Spends Bits Where They Matter

Here is the deeper machinery: Greedy-KL turns the constrained allocation into a sequence of local decisions, starting from Q6_K and ranking precision reductions by calibration KL increase per byte saved.

For a candidate reduction of unit $i$, define

$$\Delta D_i=D_{\mathrm{cal}}(a^{(i)})-D_{\mathrm{cal}}(a), \qquad \Delta M_i=M(a)-M(a^{(i)}).$$

The candidate assignment $a^{(i)}$ differs from the current assignment $a$ at that unit. A reduction saves memory when $\Delta M_i>0$. Its local cost is

$$r_i=\frac{\Delta D_i}{\Delta M_i}.$$

A smaller ratio means less measured prediction damage for each byte recovered. After reducing precision, remaining capacity is used to upgrade the most sensitive Q4_K tensors to Q5_K.

KL divergence compares probability distributions, rather than just whether the highest-scoring token changes. For intuition, one conventional direction is

$$D_{\mathrm{KL}}(p\Vert q)=\sum_v p(v)\log\frac{p(v)}{q(v)},$$

where $p$ is the reference distribution and $q$ the quantized distribution. This illustrates the disagreement measure; the ratio-based allocation rule does not require us to assume a particular implementation of the calibration aggregation.

The useful granularity is smaller than a fused tensor. Query, key, value, and output-gate slices are exposed independently, as are feed-forward gate and up projections. Along with attention output, feed-forward down, token embedding, and output projection, these form 418 weight units. No tensor role is fixed at high precision.

This is a local heuristic, not a guarantee of the best global assignment. Can a search over complete assignments make a better use of the same budget?

Why EvoPress Searches Whole Assignments

Now the version for readers who want the search-level distinction: EvoPress evolves precision assignments under a memory constraint, using KL divergence from BF16 as its objective rather than committing only to a sequence of locally ranked reductions.

Its search space uses the same Q4_K, Q5_K, and Q6_K choices. The selected profile assigns 51, 130, and 237 units to those formats respectively, all with group size 32. A group shares quantization metadata; changing group size therefore changes more than the precision label alone.

The selected assignment places the token embedding at Q4_K and the output projection at Q5_K, totaling 19.65 GB of model weights. That is an allocation result, not an instruction to always quantize embeddings more aggressively than outputs.

A static schedule offers a different engineering tradeoff. Q4KM follows llama.cpp’s subset-of-layers upcasting pattern for value, feed-forward down, and output projections. This variant uses Q5_K for the output projection and group size 128 for its Q6_K tensors and Q5_K output projection. Those choices accommodate the deployment representation and reduce scale overhead.

A searched assignment can spend memory according to calibration sensitivity; a static assignment avoids that search. But which prediction metric should decide between them?

Why Calibration Needs a Downstream Check

Here is the technical distinction behind evaluation: matching a reference distribution and assigning probability to observed tokens are related objectives, but they are not identical.

For held-out next-token examples $(x_t,y_t)$, negative log-likelihood is

$$\mathrm{NLL}(a)=-\frac{1}{T}\sum_{t=1}^{T}\log p_a(y_t\mid x_t), \qquad \Delta\mathrm{NLL}(a)=\mathrm{NLL}(a)-\mathrm{NLL}(\mathrm{BF16}).$$

Unlike KL-based search, this score looks at the probability of the observed token, not disagreement across the entire reference distribution. Held-out text and code $\Delta\mathrm{NLL}$ select candidates for downstream reasoning and agentic evaluations.

EvoPress has the lowest validation loss in the reported comparisons, while Q4KM has the smallest footprint. They are selected for 32 GB and 24 GB VRAM deployments respectively. The selected EvoPress weight size is not a claim about total runtime memory: weights and device capacity are different quantities.

The downstream comparison covers 15 tasks. Average accuracy is 75.7% for BF16, 75.6% for EvoPress, 75.3% for Greedy-KL, and 74.8% for Q4KM. These aggregate results check the allocation choices without making them interchangeable for every workload. What matters next is the behavior of the chosen assignment on the intended application.

Minimal Causal Code

This plain PyTorch-oriented loop makes the allocation state visible. It is an illustrative budget allocator, not a reproduction of search scheduling or quantization kernels. The supplied measurement functions must actually quantize and evaluate each candidate; format labels alone do not determine byte size or quality.

import torch

@torch.no_grad()
def allocate(n_units, budget_bytes, calibration_kl, stored_bytes):
    # 0 = Q4_K, 1 = Q5_K, 2 = Q6_K.
    # calibration_kl(profile) evaluates causal next-token
    # distributions against BF16 and returns a scalar tensor.
    # stored_bytes(profile) includes quantization metadata.
    profile = torch.full((n_units,), 2, dtype=torch.long)
    loss = calibration_kl(profile)
    size = stored_bytes(profile)

    while size > budget_bytes:
        best = None
        for i in range(n_units):
            if profile[i].item() == 0:
                continue
            trial = profile.clone()
            trial[i] -= 1
            trial_size = stored_bytes(trial)
            saved = size - trial_size
            if saved <= 0:
                continue
            trial_loss = calibration_kl(trial)
            ratio = (trial_loss - loss).item() / saved
            if best is None or ratio < best[0]:
                best = (ratio, trial, trial_loss, trial_size)
        if best is None:
            raise ValueError("No remaining reduction can meet budget")
        _, profile, loss, size = best  # Commit the reduction.

    while True:
        best = None
        for i in range(n_units):
            if profile[i].item() != 0:
                continue
            trial = profile.clone()
            trial[i] = 1  # Spend spare capacity on Q4_K -> Q5_K.
            trial_size = stored_bytes(trial)
            if trial_size > budget_bytes:
                continue
            trial_loss = calibration_kl(trial)
            benefit = (loss - trial_loss).item()
            if benefit > 0 and (best is None or benefit > best[0]):
                best = (benefit, trial, trial_loss, trial_size)
        if best is None:
            break
        _, profile, loss, size = best  # Commit the upgrade.

    return profile

Candidate evaluations here recompute calibration scores for clarity. The visible state transition is the important part: change one unit’s format, measure the cost, and commit only the selected assignment.

What You Gain, and What You Lose

Mixed precision turns weight compression into an explicit allocation problem. It can preserve more detail where measured sensitivity is high, rather than requiring every tensor to use the same format.

The cost is calibration, search, format support, and deployment bookkeeping. A KL objective is a proxy for behavior, not a guarantee of reasoning quality. Static Q4KM and searched EvoPress also occupy different memory-quality points: the smallest profile is not the one with the lowest validation loss.

Finally, fitting the weights does not by itself establish generation latency or total runtime memory. Quantization allocation answers where to spend storage precision; speculative decoding answers a separate question about proposing and verifying future tokens.