transformers Supports Direct Loading of GGUF Quantized Models, with Local Inference Performance Near That of llama.cpp
Transformers now runs llama.cpp quants
AI BriefingHugging Face announced that transformers can now run GGUF quantized models directly. Pass gguf_file to from_pretrained to load GGUF checkpoints from the Hub and reuse ggml’s Metal kernels.
Source: Hugging Face Community Blog · huggingface.co
Event follow-up1
View all event coverage- transformers Supports Direct Loading of GGUF Quantized Models, with Local Inference Performance Near That of llama.cppHugging Face Community Blog · September 22