--- title: "GGUF Quantization Explained: What Q4_K_M, Q8_0 and IQ2 Really Mean (and Which One to Download)" url: "https://www.geeklord.com/gguf-quantization-explained/" author: "Shobhit Kumar Prabhakar" date: "2026-11-05" modified: "2026-11-05" reading_time_minutes: 12 word_count: 3155 schema_type: "BlogPosting" categories: ["AI"] tags: ["Generative AI", "How-To", "LLM", "Open Source AI", "Tips & Tricks"] --- # GGUF Quantization Explained: What Q4_K_M, Q8_0 and IQ2 Really Mean (and Which One to Download) **TL;DR** - **What it is:** quantization stores each weight of an AI model in fewer bits (8, 6, 5, 4, 3 or 2 instead of 16). The file gets smaller and faster, and a little precision is lost. - **Reading the name:** in Q4_K_M, Q4 is about 4 bits per weight, _K is the modern block-based K-quant scheme, and _M is the medium tier of extra precision for sensitive layers. - **The default:** Q4_K_M cuts an 8B model from 16.1 GB to about 4.9 GB with a quality loss most people never notice. - **What breaks first:** maths, code, tool calling and long-context recall degrade well before casual chat does. - **Rule of thumb:** a bigger model at Q4 usually beats a smaller model at Q8. Check your GPU with our free [LLM VRAM calculator](https://www.geeklord.com/tools/llm-vram-calculator/). Open any model page on Hugging Face and you are hit with a wall of files: Q8_0, Q6_K, Q5_K_S, Q4_K_M, IQ3_XXS, Q2_K. They look like version numbers. They are not. Each name is a **recipe**: how many bits each weight was squeezed into, which packing scheme was used, and how much extra precision the most sensitive parts of the model were given. This guide unpacks that recipe with live, interactive demos: what the letters and numbers mean, what each step down in bits costs you, which tasks fall apart first, and how to pick the right file for your GPU. The one-sentence version: **fewer bits means smaller and faster, with rounding error that first nibbles at hard tasks (maths, code, reasoning) long before casual chat notices anything.** Watch a weight signal get quantized precision **8 bits**levels **256**avg. error **0.2%**size vs FP16 **50%** ** Original weights (FP16)** Quantized approximation** Error introduced The animation cycles through bit widths. Each weight's exact value is snapped to the nearest of 2bits allowed levels (the orange steps). At 8 bits the steps hug the curve. At 2 bits only 4 levels exist and the shape is barely recognisable. A whole model is this, repeated billions of times. ## 1. The core idea: rounding, billions of times A neural network is mostly a giant list of numbers called **weights**. Normally each weight is stored as a 16-bit float (FP16 or BF16), precise to about 3 decimal digits. Quantization re-stores every weight using far fewer bits by **snapping it to the nearest value on a coarse grid**. With *b* bits you get only 2b grid points: 8 bits gives 256 levels, 4 bits gives 16, and 2 bits gives just 4. Every snapped weight carries a small **rounding error**. One error is nothing. But a model performs billions of multiply-adds for every token it generates, and the errors add up, nudging the final scores (logits) that decide the next token. Nudge one far enough and the model picks a different word, and from there the whole answer can drift. Try it: these are 18 realistic weight values. Drag the bit width and watch the error grow. Bit width 4 bits*1 bit (crude) to 8 bits (precise) Allowed levels 16 2bits grid points Mean error 0 average gap to original Worst error 0 single worst-snapped weight Storage per weight 25% of the FP16 original Two things to notice. First, the error does not grow evenly. Going from 8 to 4 bits is gentle, but each step below 4 roughly **doubles** the grid spacing, so 3 and especially 2 bits hurt disproportionately. Second, even at 4 bits most bars still look right. Models carry enormous redundancy, which is exactly why 4-bit is the famous sweet spot. ## 2. How to read a GGUF filename GGUF is the file format used by llama.cpp and every tool built on it: Ollama, LM Studio, Jan, GPT4All, KoboldCpp and more. Popular uploaders such as bartowski, unsloth and mradermacher all follow the same naming scheme, and every part of the name is a decision about size versus quality: some-model-8B-Q4_K_M.gguf Q **Quantized.** The weights were compressed from FP16 or BF16. A file with no Q (labelled F16 or BF16) is the uncompressed original. 4 **Nominal bits per weight.** Q8 is about 8 bits, Q4 about 4, Q2 about 2. Real averages run a bit higher (Q4_K_M is about 4.85 bits per weight) because scales and a few high-precision tensors add overhead. _K **K-quant**, the modern block-based scheme (section 4). Weights are packed in blocks that share scale factors. Older files use _0 or _1 instead, and a leading I (as in IQ3) marks importance-matrix quants. _M **Size tier: S, M or L** (small, medium, large). Same headline bits, but a different number of sensitive tensors promoted to higher precision. M is the balanced default. ### The other labels you will meet - **Q8_0**: legacy 8-bit and near-lossless. Still the "safe" download and common for embedding models. In the old scheme, _0 stores one scale per block and _1 stores a scale plus a minimum (offset), which fits lopsided ranges slightly better. - **IQ2_XXS, IQ3_S, IQ4_XS**: "importance" quants. Instead of a plain grid they use learned codebooks plus an importance-matrix calibration pass. They squeeze usable quality out of bit widths where K-quants fall apart, at some cost in speed on older hardware. IQ4_XS is a great pick when Q4_K_M is just slightly too big. - **i1-Q4_K_M**: the i1 prefix (used by mradermacher) means the file was made with an imatrix. Same size as the plain version, usually a little better. - **UD-Q4_K_XL**: Unsloth Dynamic quants. Bit widths are chosen per tensor (important tensors high, the rest low). Often the best quality per GB at a given size. - **MXFP4**: a 4-bit floating-point format with shared block scales, made popular by OpenAI's gpt-oss models, which ship natively in it. Roughly 4.25 bits per weight. - **BF16 / F16**: no quantization at all. The reference everything is measured against, and about twice the size of Q8. - **GPTQ, AWQ, EXL2, NF4, MLX 4-bit**: not GGUF labels but the same idea in other ecosystems (Transformers, vLLM, ExLlamaV2, bitsandbytes and Apple MLX). The bit-width logic in this guide applies to all of them. ## 3. The ladder: every common quant, sized and rated Concrete numbers make the trade-off real. Below is an **8B-parameter model** (Llama 3 class) at every common setting. "Perplexity change" is how much worse the model's next-token prediction gets versus the FP16 original on standard test text. Perplexity is the field's basic quality gauge (lower is better), and changes under about 0.1 are essentially invisible in use. | Quant | Bits / weight | File size (8B) | Size vs FP16 | Perplexity change | Verdict | | --- | --- | --- | --- | --- | --- | | F16 | 16.0 | 16.1 GB | | 0 (reference) | Reference | | Q8_0 | 8.5 | 8.5 GB | | +0.00 | Near-lossless | | Q6_K | 6.6 | 6.6 GB | | +0.01 | Excellent | | Q5_K_M | 5.7 | 5.7 GB | | +0.02 to 0.05 | Excellent | | Q5_K_S | 5.5 | 5.6 GB | | +0.03 to 0.06 | Very good | | Q4_K_M ★ | 4.85 | 4.9 GB | | +0.05 to 0.10 | The sweet spot | | Q4_K_S | 4.6 | 4.7 GB | | +0.07 to 0.13 | Very good | | IQ4_XS | 4.25 | 4.4 GB | | +0.07 to 0.14 | Very good | | Q4_0 | 4.55 | 4.7 GB | | +0.10 to 0.20 | Older scheme | | Q3_K_L | 4.3 | 4.3 GB | | +0.15 to 0.30 | Noticeable | | Q3_K_M | 3.9 | 3.9 GB | | +0.20 to 0.40 | Noticeable | | Q3_K_S | 3.4 | 3.4 GB | | +0.35 to 0.60 | Rough edges | | Q2_K | 3.2* | 3.2 GB | | +0.9 to 1.5 | Heavy damage | | IQ2_XXS | 2.4 | 2.7 GB | | +1.0 to 1.8 | Last resort | Sizes are typical published GGUF file sizes for Llama 3 8B class models, rounded. Perplexity ranges reflect llama.cpp's published measurements on Llama-family models. Exact values vary by model and test set, so treat them as orders of magnitude, not lab results. *Q2_K's average is pulled up by its high-precision tensors, and its bulk weights really are 2-bit. ★ marks the default recommendation for most people. **The pattern to remember:** the quality curve is not a slope, it is a cliff with a long flat top. From FP16 down to Q4_K_M you give up almost nothing you can perceive. Below Q4 the cost accelerates. Q3 is a compromise you accept to fit a bigger model, and Q2 is a model-shaped souvenir of the original. Compare Q3_K_L with Q4_K_S: nearly the same file size, but the 4-bit file is clearly better. And prefer a **bigger model at Q4 over a smaller model at Q8**: a 14B at Q4_K_M will generally out-think an 8B at Q8_0, because extra parameters add capability faster than extra precision does. ## 4. Inside a K-quant: blocks, scales and VIP tensors Rounding every weight onto one global grid would be terrible, because weight ranges differ wildly from place to place. K-quants fix this with **blocks**: weights are processed in super-blocks of 256, split into sub-blocks of 32. Each sub-block gets its own **scale factor** (and the scales themselves are quantized), so the grid stretches or shrinks to fit the local range. A block of small weights gets a fine grid, a block with big values gets a coarse one, and outliers stop ruining their neighbours. One K-quant super-block: 256 weights in 8 sub-blocks of 32. Each sub-block stores 32 low-bit codes plus one shared scale. At inference time a weight is rebuilt as code × scale, which is cheap enough to do on the fly. The moving highlight is the read head: every generated token streams through every block of every layer. Because generation is limited by memory bandwidth, a file that is 3 times smaller is often close to 3 times faster. ### So what do _S, _M and _L actually change? Not all weights are equally important. The embedding table, the attention projections and the final output head are unusually sensitive, and rounding them hurts far more than rounding a mid-layer feed-forward block. The K-quant tiers are simply **three answers to "how many of those VIP tensors get promoted to a higher-precision quant?"** The base bit width stays the same, only the promotions differ. #### _S SMALL Q4_K_S, 8B ≈ 4.7 GB *************** Fewest promotions. Saves about 200 MB over _M on an 8B. Worth it only when you are just barely over a VRAM limit. #### _M MEDIUM, DEFAULT Q4_K_M, 8B ≈ 4.9 GB **************** The balanced recipe: several sensitive tensors are stored at Q6_K instead of Q4. This is the file most repos lead with, and the right default. #### _L LARGE Q3_K_L, 8B ≈ 4.3 GB **************** Most promotions. It exists mainly to rescue Q3: a Q3_K_L spends its extra bytes exactly where 3-bit hurts most. At Q4 and above, _M already covers the important ones. The bars show the idea, the share of tensors that get high-precision treatment, not an exact tensor count. The precise promotion list is defined per quant type in llama.cpp and varies a little by model architecture. ## 5. What breaks first (it isn't chat) Quantization damage is not spread evenly. Tasks that only need the gist (chat, summaries, everyday questions) are deeply redundant and shrug off rounding. Tasks that need **exact token choices chained over many steps** amplify every small error: one wrong digit early in a calculation, one missing brace in code, one wrong tool argument, and the whole chain fails. Pick a task and a bit width: Task **Casual chat Summarising RAG / Q&A on docs Code generation Maths and reasoning Tool calling and agents Low-resource languages Long-context recall Bit width 4 bits (Q4_K_M)* Expected reliability vs the FP16 original An illustrative rule of thumb drawn from published quant benchmarks and community testing, not one measured dataset. Individual models vary, and big models degrade more gracefully than small ones at the same bit width. ### Why reasoning is fragile Chat answers are overdetermined*: dozens of tokens would be fine at each step, so a nudged score still lands on a good word. A maths proof or a function call is the opposite. The correct next token is often one specific token out of 100,000+, and errors compound across a long chain. That is why a model can feel "98% as good" in conversation yet be visibly worse at GSM8K-style maths or multi-step code. ### Why size buys forgiveness An 8B model at Q3 and a 70B model at Q3 are not in the same situation. Larger models store knowledge with more redundancy, so random rounding noise cancels out instead of piling up. The practical rule: quantize big models aggressively, small models gently.** For a 3B model, Q4 is the floor for serious work, while a 70B at Q3_K_M still performs respectably. ## 6. Related ideas that complete the picture ### Perplexity, the quality yardstick Perplexity measures how surprised a model is by real text: roughly, how many tokens it is effectively choosing between at each step. Lower is better. It is blunt (it will not tell you a model got worse at coding), but it is cheap, repeatable and tracks overall damage well, which is why every quant table is anchored to it. ### The imatrix: calibrating what matters Good quantizers do not round blindly. They first run sample text through the model and record which weights actually matter, building an **importance matrix**. Sensitive weights get finer grids, and disposable ones absorb more error. This is why two files with the same label from different uploaders can differ in quality. ### KV cache: the other memory Weights are not the only thing in memory. During a chat the model caches keys and values for every token of context, and at 32K tokens on a mid-size model that can rival the weights. llama.cpp and Ollama can quantize this cache to 8 or 4 bits. It costs a little long-context accuracy and saves a lot of memory. Different memory, different trade-off. ### Why smaller is also faster Generating one token means reading essentially every weight once. On consumer GPUs the bottleneck is memory bandwidth, not arithmetic, so tokens per second scale almost inversely with file size. A Q4 model is not just the one that fits your 8 GB card, it is roughly twice as fast as its Q8 twin on the same hardware. ### MoE models: total vs active Mixture-of-Experts models (Qwen3 30B-A3B, gpt-oss-120b, DeepSeek) list two numbers. **Total parameters** decide how much memory you need, because every expert must be loaded. **Active parameters** decide speed, because only a few experts run per token. That is why a 30B MoE can feel as fast as a 3B model once it fits. ### Macs and unified memory Apple Silicon Macs share one pool of memory between CPU and GPU, so a 64 GB Mac can hold models no 24 GB graphics card can. By default macOS lets the GPU use roughly two-thirds to three-quarters of that pool. Memory bandwidth (from about 100 GB/s on base chips to over 500 GB/s on Max chips) then decides the speed. ## 7. Will it fit on your GPU? The quick maths: **file size ≈ parameters × bits per weight ÷ 8**. An 8B model at Q4_K_M (4.85 bits) is about 8 × 4.85 ÷ 8 ≈ 4.9 GB. Then add room for the KV cache and runtime, usually 0.5 to 2 GB at everyday context lengths and much more at 32K and beyond. If the total fits inside your VRAM with a little headroom, the whole model runs on the GPU at full speed. If it does not, layers spill into system RAM and generation slows down sharply. **Try the free LLM VRAM calculator.** Pick your GPU or Mac, model size, quant and context length. It estimates memory, speed in tokens per second, and the best quant that fits fully on your hardware. [Open the calculator](https://www.geeklord.com/tools/llm-vram-calculator/) ## 8. How to download exactly the quant you want Most tools pick a quant for you, usually Q4_K_M. Here is how to choose one yourself in the most popular tools. **Ollama**: add the quant to the model tag, or pull any GGUF straight from Hugging Face. ``` ollama run llama3.1:8b-instruct-q4_K_M ``` ``` ollama run hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q5_K_M ``` **llama.cpp**: the -hf flag downloads and runs a repo, with the quant after the colon. ``` llama-server -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M ``` **Hugging Face CLI**: download only the file you need instead of the whole repo. ``` hf download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF --include "*Q4_K_M.gguf" --local-dir ./models ``` **LM Studio**: search for the model, open the download options and pick the quant from the list. LM Studio also marks which files should fit fully on your GPU. **Quantize the KV cache** to fit longer context. In Ollama (set before starting the server): ``` OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve ``` In llama.cpp (flash attention is required for a quantized V cache): ``` llama-server -m model.gguf -c 32768 -fa on -ctk q8_0 -ctv q8_0 ``` Older llama.cpp builds take -fa without a value. Model names and repos above are examples, so swap in the model you want. ## 9. Which quant should you download? A 30-second guide 1. **Start at Q4_K_M.** It is the best balance of size, speed and quality for almost everyone. 2. **Have spare memory? Go up.** Q5_K_M or Q6_K for coding, maths and agents. Q8_0 if it fits with room to spare and you want peace of mind. 3. **Slightly too big? Step sideways first.** Try IQ4_XS or Q4_K_S, or cut the context length, before dropping to Q3. 4. **Choose a bigger model at Q4 over a smaller one at Q8.** Parameters beat precision. 5. **Use Q3 only to fit a much bigger model**, and treat Q2 as a curiosity, not a tool. 6. **Prefer imatrix (i1 or IQ) and dynamic (UD) builds** from well-known uploaders when both are on offer. ## Frequently asked questions What does Q4_K_M mean? It is a 4-bit K-quant at the medium tier. Weights average about 4.85 bits each, are packed in blocks with shared scales, and a set of sensitive tensors is kept at higher precision (Q6_K). For an 8B model the file is about 4.9 GB, versus 16.1 GB unquantized. Is Q4_K_M or Q5_K_M better? Q5_K_M is slightly more accurate and about 16% larger. If both fit fully on your GPU, take Q5_K_M, especially for code and maths. If only Q4_K_M fits fully on the GPU, it will be faster and the quality difference is small. Is Q8_0 worth it over Q6_K? Rarely. Q6_K is already within measurement noise of the original for most models. Q8_0 makes sense when memory is plentiful, for small models (3B and under), and for embedding models. What is the difference between K-quants and I-quants? K-quants use simple block-wise grids with scales and are fast everywhere. I-quants (IQ) use learned codebooks and an importance matrix, which gives better quality at very low bit widths (2 to 3 bits) but can run slower on CPUs and older GPUs. At 4 bits and above, the two are close. How much VRAM do I need for a 7B or 8B model? About 5 to 6 GB at Q4_K_M with a normal context length, so an 8 GB card runs it fully on the GPU. A 12 GB card can take Q8_0 or a 14B model at Q4_K_M. Use the [VRAM calculator](https://www.geeklord.com/tools/llm-vram-calculator/) for your exact setup. Does quantization make a model faster? Yes. Token generation is limited by how fast weights can be read from memory, so a smaller file usually means proportionally more tokens per second, as long as the whole model stays on the GPU. **The rule of thumb to walk away with:** Q8_0 if you have room, Q5_K_M if you want headroom, Q4_K_M as the default, Q3 only to fit something bigger, and Q2 never for work you care about. File sizes and perplexity figures are typical values for Llama-family GGUF models and vary by model. The reliability scores in section 5 are illustrative, not benchmark measurements. --- *Published on GeekLord.com - Original URL: https://www.geeklord.com/gguf-quantization-explained/*