#Quantization

Loading llama.cpp Quants in Transformers: What Load Time Does

Transformers can load llama.cpp GGUF files directly. Outside the narrow Metal fast path it dequantizes at load time, so you get the disk savings without the quantized kernels.