This is a Plain English Papers summary of a research paper called 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11). If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter. This is a Plain English Papers summary of a research paper called 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) . If you like these kinds of analysis, join AIModels.fyi AIModels.fyi or follow us on Twitter Twitter . The inefficiency nobody questions Large language models have become too big to fit anywhere convenient. Llama 3.1 405B needs 810GB of storage. Most consumer GPUs have 24GB of memory. Even data center GPUs top out at 80GB. When a model doesn't fit, practitioners face brutal choices: accept the 10-100x slowdown of offloading weights to CPU and transferring them over PCIe, or reduce precision through quantization and accept accuracy loss. Both paths feel inevitable. Both are wrong. The assumption baked into modern AI infrastructure is that neural network weights are arbitrary numbers that need full precision. So we store them in BFloat16, a format designed for general floating-point computation. It allocates bits uniformly: one for sign, eight for exponent, seven for mantissa. This is sensible for arbitrary numbers. It's nonsensical for LLM weights, which follow predictable patterns. This paper reveals a simple fact: LLM weights aren't random. They're highly structured. And that structure creates massive wasted space in how we store them. Why weights aren't as random as we think The key to understanding the inefficiency is entropy, a measure of how much information is actually contained in a set of values. When values are uniform, entropy is high and storage space is tight. When values are skewed, entropy is low and storage space is wasted. BFloat16 allocates the same number of bits to each component regardless of how much information that component actually carries. The sign bit, the exponent, and the mantissa all get fixed allocations. But trained neural networks don't distribute their values uniformly across these components. Figure 1 showing BFloat16 bit allocation and entropy across sign, exponent, and mantissa components in various LLMs Figure 1 showing BFloat16 bit allocation and entropy across sign, exponent, and mantissa components in various LLMs The left side of Figure 1 shows the bit layout of BFloat16. The right side shows the actual Shannon entropy of each component across different models. Notice that exponents have dramatically lower entropy than their allocated space. The left side of Figure 1 shows the bit layout of BFloat16. The right side shows the actual Shannon entropy of each component across different models. Notice that exponents have dramatically lower entropy than their allocated space. The exponent values in LLM weights cluster heavily around a few common values. The mantissa and sign bits distribute more evenly. This creates a mismatch: exponents are doing little information work but consuming eight full bits, while sign and mantissa bits use their allocations more efficiently. Figure 8 displaying relative frequency distributions of sign, exponent, and mantissa values Figure 8 displaying relative frequency distributions of sign, exponent, and mantissa values Figure 8 quantifies this skewness. Notice the exponent distribution is heavily concentrated. Some exponent values appear 10x more frequently than others. This is where the inefficiency lives. Figure 8 quantifies this skewness. Notice the exponent distribution is heavily concentrated. Some exponent values appear 10x more frequently than others. This is where the inefficiency lives. The measurement is not theoretical. It's baked into every trained model. A weight that represents a common exponent value and a weight that represents a rare exponent value both take up eight bits, even though one carries far more information. That's the slack. Dynamic-Length Float: the elegant answer The solution is as old as information theory itself. Huffman coding, invented in 1952, assigns shorter bit sequences to frequent values and longer sequences to rare ones. The total message gets shorter because you're not wasting bits on frequent events. Applied here: compress only the exponents. Leave sign and mantissa alone. Figure 2 showing the DFloat11 format with variable-length exponents and fixed-length sign/mantissa Figure 2 showing the DFloat11 format with variable-length exponents and fixed-length sign/mantissa Figure 2 shows the layout of DFloat11. Only exponents are compressed via Huffman coding. Sign and mantissa bits remain fixed-length, like they were in BFloat16. Figure 2 shows the layout of DFloat11. Only exponents are compressed via Huffman coding. Sign and mantissa bits remain fixed-length, like they were in BFloat16. The result is DFloat11. The name comes from the fact that weights compress from 16 bits to approximately 11 bits on average. "Lossless" means the decompressed weight is bit-for-bit identical to the original. No approximation. No accuracy loss. No numerical drift. You can decompress and get back exactly what you stored. This distinction matters enormously. Quantization, the standard compression approach for neural networks, trades precision for size. You reduce bit width, accept some loss of accuracy, and hope the model still works. DFloat11 makes a different bet: structure lets you eliminate waste without sacrificing fidelity. The approach is universal. It works on any LLM already in BFloat16 format. No retraining. No fine-tuning. No model-specific engineering. Making decompression fast enough Elegance on paper means nothing if decompression takes longer than the compression saves. This is where engineering discipline separates theory from practice. Variable-length codes create a fundamental problem: you can't know where the next code starts until you've decoded the previous one. A CPU would traverse a Huffman tree bit by bit, checking nodes and following branches. This is inherently serial. On a GPU, where parallelism is everything, this becomes a bottleneck. The authors solved this with hierarchical lookup tables. The key insight is that a Huffman tree can be decomposed into smaller subtrees, each compact enough to fit entirely in GPU SRAM (the fast on-chip memory). Decoding becomes a series of array lookups instead of tree traversals. Many threads can perform lookups in parallel without contention. Figure 3 showing how the Huffman tree decomposes into hierarchical LUTs stored in GPU SRAM Figure 3 showing how the Huffman tree decomposes into hierarchical LUTs stored in GPU SRAM Figure 3 visualizes the hierarchy. The full Huffman tree (left) breaks into smaller lookup tables (right) that fit in SRAM. Decoding becomes a sequence of fast array operations rather than tree navigation. Figure 3 visualizes the hierarchy. The full Huffman tree (left) breaks into smaller lookup tables (right) that fit in SRAM. Decoding becomes a sequence of fast array operations rather than tree navigation. The GPU kernel itself runs in two phases. First, threads read variable-length encoded data in parallel and perform LUT lookups. Lightweight auxiliary variables track read/write positions across threads without expensive synchronization. Second, threads synchronize and write decompressed weights to GPU memory. The design overlaps computation and memory operations where possible. Decompression happens at the transformer block level, just-in-time. Rather than decompressing the entire model into memory upfront, the kernel decompresses weights as they're needed for each block of computation. This minimizes the memory footprint at any moment. Figure 6 showing latency breakdown during inference for DFloat11 and BFloat16 Figure 6 showing latency breakdown during inference for DFloat11 and BFloat16 Figure 6 breaks down where latency goes during inference. Decompression adds a small overhead to each forward pass. The investment in custom kernels paid off: the cost is amortized across the speed gains. Figure 6 breaks down where latency goes during inference. Decompression adds a small overhead to each forward pass. The investment in custom kernels paid off: the cost is amortized across the speed gains. Figure 7 comparing throughput and latency of decompression on GPU versus CPU-to-GPU transfer Figure 7 comparing throughput and latency of decompression on GPU versus CPU-to-GPU transfer Figure 7 answers a direct question: is it faster to decompress on the GPU or just transfer uncompressed weights from the CPU? Decompression wins decisively. GPU decompression is faster than PCIe transfer, even for the same data. Figure 7 answers a direct question: is it faster to decompress on the GPU or just transfer uncompressed weights from the CPU? Decompression wins decisively. GPU decompression is faster than PCIe transfer, even for the same data. This matters because it proves the engineering was necessary and effective. Without these optimizations, decompression would be the bottleneck. With them, you get the compression benefit for free. What this actually buys you The results are concrete and reproducible across multiple models: Llama 3.3, Qwen 3, Mistral 3, FLUX.1. Model size reduction: approximately 30% across the board. Since this is lossless compression, the decompressed output is byte-identical to the original. No accuracy drift. No behavioral changes. The model works exactly as it did before. Model size reduction Throughput compared to alternatives: when a model doesn't fit on a single GPU, the current standard practice is to offload some weights to CPU and transfer them during inference. DFloat11 achieves 2.3x to 46.2x higher throughput than this approach, depending on model size and GPU memory constraints. Larger models see bigger wins because more layers require offloading. Throughput compared to alternatives Figure 4 comparing throughput and latency for DFloat11-compressed models versus BFloat16 with CPU offloading Figure 4 comparing throughput and latency for DFloat11-compressed models versus BFloat16 with CPU offloading Figure 4 shows throughput (top row) and latency (bottom row) across different models. DFloat11 is a horizontal line at the top of the throughput chart. BFloat16 with CPU offloading drops precipitously as the model grows larger. Figure 4 shows throughput (top row) and latency (bottom row) across different models. DFloat11 is a horizontal line at the top of the throughput chart. BFloat16 with CPU offloading drops precipitously as the model grows larger. Memory headroom and longer contexts: LLM inference doesn't just need to store weights. It needs to store the KV cache, the running history of key and value vectors from all previous tokens. This grows with both the number of layers and the context length. By freeing up 30% of GPU memory from model weights, that space becomes available for the KV cache. Memory headroom and longer contexts Figure 5 showing GPU memory consumption comparison between BFloat16 and DFloat11 models Figure 5 showing GPU memory consumption comparison between BFloat16 and DFloat11 models Figure 5 visualizes memory usage. The compressed model (blue) uses 30% less space for weights, leaving room for a larger KV cache (orange). This enables 5.7x to 14.9x longer context lengths with the same GPU memory. Figure 5 visualizes memory usage. The compressed model (blue) uses 30% less space for weights, leaving room for a larger KV cache (orange). This enables 5.7x to 14.9x longer context lengths with the same GPU memory. The practical impact is striking: Llama 3.1 405B, an 810GB model, now runs losslessly on a single node with 8x80GB GPUs. Full inference. Exact outputs. No tricks. This was impossible before. A separate direction of research explores dynamic memory compression for retrofitting LLMs for accelerated inference, which tackles similar constraints through different mechanisms. The broader landscape of lossless compression techniques for LLM tensors shows that this is an active area, with multiple groups converging on the idea that we can do better than fixed-bit representations. dynamic memory compression for retrofitting LLMs for accelerated inference lossless compression techniques for LLM tensors The real insight This paper works because it answers a question nobody was asking: what if we stopped pretending LLM weights were arbitrary? What if we measured the actual information content and built storage around that instead of around historical conventions? The insight is that bit-for-bit fidelity doesn't require wasted bits. You can compress without approximating. And when you do, the practical gains compound: smaller models, faster inference, longer sequences, lower cost per token. This is what happens when you look hard at where inefficiency actually lives.

Quant

Turns Out 30% of Your AI Model Is Just Wasted Space

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Can Large Language Models Develop Gambling Addiction?

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Can Large Language Models Develop Gambling Addiction?

The Noonification: Use This 7-Step McKinsey Framework to Solve Any Problem (1/10/2023)

The Noonification: A Taxonomy of Inclusiveness (1/11/2024)

The Noonification: What is the InfiniteNature-Zero AI Model? (11/19/2022)

10 Ways AI Has Changed Our Lives

100 Days of AI, Day 8: Experimenting With Microsoft's Semantic Kernel Using GPT-4

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps