Quantization

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Cover image for Why your quantized LLM loses its MTP heads and how to keep them

Alan West

May 27

Why your quantized LLM loses its MTP heads and how to keep them

#machinelearning #llm #python #quantization

5 min read

MxGuru

May 20

The Best Result This Week Was a Failed Prediction — Phase-3a Doesn't Transfer

#quantization #hsaq #methodology #granite

1 min read

MxGuru

May 20

Two Localizers, Both Wrong: Bounding a Quantization Cost That Wouldn't Close

#quantization #hsaq #methodology #granite

1 min read

MxGuru

May 20

When the Sensitivity Metric Lies: A Drift-Inversion Smoking Gun in Mixed-Precision LLM Quantization

#quantization #hsaq #awq #granite

8 min read

Patrick Hughes

May 13

GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8 — Which to Pick (2026)

#llamacpp #gguf #quantization #localai

4 min read

Vilius

May 9

1-bit, 545 megabytes, zero API keys — local AI that beats GPT-5.4

#ai #llm #local #quantization

2 min read

Aman Sachan

Apr 30

KVQuant: Run 70B LLMs on 8GB RAM with KV Cache Quantization

#python #llm #quantization

1 min read

Aman Sachan

Apr 30

KVQuant: Run 70B LLMs on 8GB RAM with 4-bit KV Cache Quantization

#python #llm #quantization #optimization

1 min read

Cover image for Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

Alan West

Apr 18

Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

#machinelearning #llm #quantization #ai

5 min read

Denis Lavrentyev

Apr 13

GIMP's Posterization: Simple Quantization vs. Median Cut for Better Visuals

#gimp #posterization #quantization #mediancut

8 min read

plasmon

Apr 8

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

#llm #quantization #vram #localllm

8 min read

ANKUSH CHOUDHARY JOHAL

Apr 27

Postmortem: How a Quantization Error in Llama 3.2 7B Caused Incorrect Code Suggestions for 500 Users

#postmortem #quantization #errors #llama

13 min read

Jean

May 8

Chasing 16MB: My Parameter Golf Journey and What I Learned the Hard Way

#parametergolf #tinyllm #aiexperimentation #quantization

3 min read

Scott Everitt

Mar 30

Building a Vector Database That Never Decompresses Your Vectors

#vectordatabase #quantization #turboquant #go

16 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Forem

# quantization

Why your quantized LLM loses its MTP heads and how to keep them

The Best Result This Week Was a Failed Prediction — Phase-3a Doesn't Transfer

Two Localizers, Both Wrong: Bounding a Quantization Cost That Wouldn't Close

When the Sensitivity Metric Lies: A Drift-Inversion Smoking Gun in Mixed-Precision LLM Quantization

GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q8 — Which to Pick (2026)

1-bit, 545 megabytes, zero API keys — local AI that beats GPT-5.4

KVQuant: Run 70B LLMs on 8GB RAM with KV Cache Quantization

KVQuant: Run 70B LLMs on 8GB RAM with 4-bit KV Cache Quantization

Traditional Quantization vs 1.58-Bit Ternary Models: A Practical Comparison

GIMP's Posterization: Simple Quantization vs. Median Cut for Better Visuals

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

Postmortem: How a Quantization Error in Llama 3.2 7B Caused Incorrect Code Suggestions for 500 Users

Chasing 16MB: My Parameter Golf Journey and What I Learned the Hard Way

Building a Vector Database That Never Decompresses Your Vectors