GGUF vs AWQ vs EXL3 (and EXL2): which quantization format should you use?

Quantization stores a model's weights in fewer bits, so it fits in less memory. The format is how those smaller weights are packaged, and it decides which apps can load the file. Pick the wrong one and the download simply will not open.

I read each project's own documentation on 4 October 2026: the GGUF spec, the llama.cpp, ExLlamaV3, ExLlamaV2, AWQ and AutoAWQ repos, and vLLM's hardware table. I have not run a speed shootout for this piece, so you will not find tokens-per-second claims here. What you get is what each format is, where it runs, and whether it is still maintained.

What is GGUF?

GGUF is a file format for storing a model so llama.cpp and other GGML-based programs can run it. The GGUF spec calls it "a binary format that is designed for fast loading and saving of models." Everything needed to load the model sits in one file: the weights, the tokenizer and the settings.

It replaced GGML's older file formats (GGML, GGMF and GGJT). The spec lists its goals: single-file deployment, room to add new features without breaking old files, and fast loading through memory mapping. A file name like Qwen3.8-27B-Q4_K_M.gguf tells you the model, its size and its quant type.

People also ask what the letters stand for. The spec itself does not spell it out. It simply calls GGUF the successor to GGML's older formats, so I won't invent an expansion for you.

The quant type sets the size. Our VRAM engine uses about 4.85 bits per weight for Q4_K_M, a figure we checked against real files. For Qwen3.8 27B that is about 15.7 GiB of weights, against 51.7 GiB at full 16-bit. You can see that math for any tracked model on its cost to run page, or read which level to choose in our Q4 vs Q5 vs Q8 guide.

Where does each format run?

GGUF runs almost anywhere. AWQ and EXL3 are GPU formats. llama.cpp lists backends for NVIDIA (CUDA), AMD (HIP), Apple Silicon (Metal), Intel GPUs (SYCL), Vulkan and plain CPUs. ExLlamaV3 needs CUDA. vLLM runs AWQ on recent NVIDIA cards, Intel GPUs and x86 CPUs, but not on AMD GPUs.

The GGUF bit range is our reading of real files, not a spec limit. The smallest community quants of big models land near 1.3 to 1.5 bits per weight, and Q8_0 sits at 8.5 bits per weight once scales are counted.

GGUF vs AWQ: which should I use?

GGUF for one person on one machine, AWQ for a GPU server that answers many users. GGUF works in Ollama and LM Studio and on any hardware. AWQ is built for server engines like vLLM, which batch many requests at once. On a laptop or a Mac, GGUF is the practical choice.

AWQ stands for Activation-aware Weight Quantization. It comes from MIT's Han Lab, whose repo describes it as "low-bit weight quantization (INT3/4)" and notes an MLSys 2024 Best Paper award. The idea is that a small share of weights matter most, and AWQ protects those when it rounds the rest down.

One practical change: the popular AutoAWQ tool is "officially deprecated and will no longer be maintained," per its README. It says vLLM adopted it into llm-compressor, and that MLX-LM now supports AWQ on Macs. So AWQ files keep working, but new conversions go through vLLM's tooling.

If you are choosing a server engine as well, our Ollama vs llama.cpp vs LM Studio vs vLLM comparison covers that decision.

EXL3 vs GGUF: is EXL3 worth it?

Only if you run NVIDIA GPUs and want control over exact size. EXL3 lets you pick any target bitrate, say 3.75 bits per weight, to fill your card to the last gigabyte. GGUF gives you fixed steps (Q4_K_M, Q5_K_M and so on) but runs on everything. Most home users should stay on GGUF.

EXL3 is the format of ExLlamaV3. Its README says EXL3 is "a streamlined variant of QTIP," a research method from Cornell's RelaxML group. Conversion needs only the original model and a target bitrate. The README says it takes a couple of minutes for small models and up to a few hours for 70B-plus models on one high-end consumer GPU.

ExLlamaV3 also adds 2 to 8-bit cache quantization, tensor and expert parallelism across several GPUs, and CPU offloading so large MoE models run on limited VRAM. Its recommended server is TabbyAPI, which speaks the same API as OpenAI. Install needs PyTorch with CUDA 12.4 or later.

Is EXL2 dead?

Archived, at least for now. The ExLlamaV2 README opens with: "This project is archived for now. Development continues on ExLlamaV3." Its last release was v0.3.2 in July 2025. Existing EXL2 files still load in ExLlamaV2, but for anything new, use EXL3 or GGUF.

That changes older advice, including an earlier version of this page. If you are weighing the engines rather than the formats, our ExLlamaV2 vs llama.cpp guide covers the trade-offs. Bear the archive notice in mind when you read it.

So which format should you download?

Start with a GGUF at Q4_K_M in Ollama or LM Studio. Move to AWQ only when you serve a model to several people through vLLM on a supported GPU. Try EXL3 if you own NVIDIA cards and want to fit the biggest model possible. Avoid starting anything new on EXL2.

The size matters more than the format for whether a model fits. Our VRAM calculator gives the number for your card. If you are on 12 or 16 GB, our list of the best MoE models for 12 to 16 GB GPUs shows what fits, almost all of it as GGUF. New to all this? Start with the beginner's guide.

The answer for most readers is still the boring one: GGUF at Q4_K_M, which is what Ollama and LM Studio hand you by default. The formats around it have moved, so it is worth knowing which ones are still maintained before you spend an afternoon converting a model.