Transformers can now load a quantized model file straight from llama.cpp's format, skipping the convert-then-quantize step entirely. The part worth checking before you trust it: on most setups, loading that file gives you llama.cpp's quantized storage, not its quantized math.

That distinction decides whether this is useful to you. If you wanted smaller downloads and one less CLI step, it delivers. If you wanted llama.cpp's speed on a 4-bit model, the fast path is narrow: it currently applies to Apple Silicon (Metal/MPS) with a supported architecture, and everything else falls back to dequantization.

The step this removes

The old path for running a quantized open model had three moves. Download the full-precision weights, run convert_hf_to_gguf.py to turn them into GGUF (GPT-Generated Unified Format, llama.cpp's single-file weight container), then run llama-quantize to squeeze them down to 4 or 5 bits. Only then could llama.cpp load the thing.

The new path is one call. Point from_pretrained at a Hub repo and the specific .gguf filename inside it, and Transformers pulls that file directly.

Load a GGUF checkpoint and check what dtype you actually got
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "unsloth/Qwen3.5-4B-GGUF"
gguf_file = "Qwen3.5-4B-Q4_K_M.gguf"

tok = AutoTokenizer.from_pretrained(repo, gguf_file=gguf_file)
model = AutoModelForCausalLM.from_pretrained(repo, gguf_file=gguf_file)

print(model.dtype)
print(model.model.layers[0].self_attn.q_proj.weight.dtype)
print(model.model.layers[0].self_attn.q_proj.weight.element_size(), "bytes/param")

Run that and read the last three lines carefully. That output is the whole article.

What load time does to the weights

GGUF stores weights in packed low-bit blocks: groups of values sharing a scale factor, crammed well below one byte per parameter. llama.cpp's kernels compute directly against those packed blocks. That is where its memory bandwidth advantage comes from.

Transformers keeps that packed layout only in the narrow case where a ggml kernel applies: Metal (MPS) devices, with a compatible quantization type and a supported architecture. Outside that case it unpacks the blocks at load time into a standard PyTorch dtype so the weights fit into ordinary nn.Linear layers. Your 4-bit file expands back into a float dtype in memory, and every matrix multiply after that runs at full precision. The legacy loader, which handles most of the older architectures (Llama, Mistral, Qwen2, Phi3, Falcon and friends), always dequantizes.

So on the fallback path the savings are real but narrow. Smaller download, smaller disk, faster fetch over a bad connection. No quantized compute, no bandwidth win at inference time.

Which quant formats actually load today

Not every scheme llama.cpp can produce is guaranteed to hit the fast path. The kernels cover specific block layouts, and llama.cpp has accumulated a lot of layouts over the years.

Safe first tests:

  1. Q4_K_M, the format Hugging Face suggests starting with. It is also the most widely published format on the Hub.
  2. Q5_K_M or Q6_K if you have more memory to spend. These K-quants use mixed block sizes with per-block scales.
  3. Anything else, expect to verify.

Legacy types and importance-matrix-tuned quants (IQ variants, where the quantization is guided by activation statistics collected on calibration data) are where you hit gaps. Hugging Face's own guidance when a checkpoint does not work is to open an issue with the checkpoint and your use case so support gets prioritized, which is a polite way of saying coverage is incremental.

Benchmarking against a llama.cpp baseline

Measure both, not one. The number that matters is wall clock from process start to first token, because that folds in download, dequantization, and warmup.

Time-to-first-token, both paths, same file
#!/usr/bin/env bash
set -euo pipefail

MODEL="$HOME/models/Qwen3.5-4B-Q4_K_M.gguf"
PROMPT="Explain a B-tree in two sentences."

echo "== llama.cpp =="
/usr/bin/time -p llama-cli -m "$MODEL" -p "$PROMPT" -n 1 --no-warmup 2>&1 \
  | tail -n 4

echo "== transformers =="
/usr/bin/time -p python - <<'PY' 2>&1 | tail -n 4
from transformers import AutoModelForCausalLM, AutoTokenizer
r, f = "unsloth/Qwen3.5-4B-GGUF", "Qwen3.5-4B-Q4_K_M.gguf"
tok = AutoTokenizer.from_pretrained(r, gguf_file=f)
m = AutoModelForCausalLM.from_pretrained(r, gguf_file=f)
ids = tok("Explain a B-tree in two sentences.", return_tensors="pt")
m.generate(**ids, max_new_tokens=1)
PY

Now the interpretation trap. Hugging Face reports generation throughput gains alongside this feature, and those gains are real, but part of them comes from changes to the generate loop rather than from the weights. Hugging Face's own charts isolate that: they keep the layer kernels enabled and show the generation-loop changes as a separate bar. Those loop changes are not GGUF-specific. Crediting a whole benchmark to "GGUF speed" attributes to the file format a win that partly belongs to the loop around the model.

If you report numbers, report both: load time, which GGUF genuinely affects, and generation speed, where you need to say which path you were on.

What this is not good for yet

Current support targets a single interactive conversation on Apple Silicon. There is no generate_batch path, so nothing here helps you serve concurrent requests. Reaching for this instead of vLLM because it sounds faster is a mistake you will discover under load.

The honest fit: local testing, quick model swaps, and pulling a community quant off the Hub without keeping a llama.cpp build around just to convert files.

The loader saves you a conversion step, and only sometimes a compute engine. Load one Q4_K_M checkpoint Monday, print model.dtype and the per-parameter byte count, and let that output decide whether this replaces llama.cpp in your setup or just sits next to it.