New MXFP4 LLM Format Loses to Older Q4_K_M in Speed Tests

The Unexpected Speed Test Results

The landscape of running large language models (LLMs) locally is defined by a constant trade-off: model size versus speed and accuracy. Newer quantization formats often promise better performance, aiming to squeeze more efficiency from hardware. However, a recent test pitting OpenAI's MXFP4 format against the older Q4_K_M format on a MacBook M-series chip yielded surprising results. The supposedly faster, newer format, MXFP4, was outperformed by Q4_K_M, delivering tokens at 1.8 times the speed of its predecessor.

The specific test involved generating 200 tokens using the same prompt on the same laptop. Q4_K_M achieved a speed of 4.7 tokens per second, completing the generation in 44 seconds. In stark contrast, MXFP4 managed only 2.6 tokens per second, taking 71 seconds to complete the identical task. This outcome directly contradicts the marketing claims and expectations surrounding MXFP4, which was designed with improved efficiency in mind.

The impetus for this test was initially to compare different quantization levels (Q4, Q5, Q8) of the same model. However, the logistical hurdle of downloading multiple large GGUF files over a home network led the tester to pivot to comparing two different quantization formats of potentially similar models. The goal remained to understand the impact of quantization on inference speed, but the specific formats chosen led to an unexpected revelation about the current state of LLM quantization.

Methodology and Hardware

To ensure a fair comparison, the test adhered to strict controls. The hardware used was a single laptop equipped with an M-series Apple Silicon chip. This standardization is crucial because LLM inference performance is highly dependent on the underlying hardware architecture, particularly the efficiency of its neural processing units and memory bandwidth. Using the same machine for both tests eliminates hardware as a variable.

The prompt used for the generation was identical for both Q4_K_M and MXFP4. This ensures that the computational workload required to process the input and generate the output sequence was the same. Variations in prompt complexity can significantly impact token generation speed, so consistency here is paramount.

Each format underwent three trials. This repetition helps to mitigate the impact of temporary system fluctuations or background processes that might momentarily affect performance. By averaging the results across multiple runs, a more reliable and representative performance metric is established. The reported speeds (4.7 tokens/sec for Q4_K_M and 2.6 tokens/sec for MXFP4) are derived from these averaged trials, providing a robust comparison.

Understanding the Formats: Q4_K_M vs. MXFP4

Q4_K_M represents an established quantization method, often found in the popular GGUF (GPT-Generated Unified Format) files used for running LLMs locally. This format typically uses integer quantization, striking a balance between model size reduction and maintaining a reasonable level of accuracy. The 'K_M' suffix often denotes specific optimizations within the 4-bit quantization scheme, aiming for better performance characteristics than simpler 4-bit methods.

MXFP4, on the other hand, is a newer format introduced by OpenAI. It leverages microscaling floating-point (FP) representations. The intention behind such formats is to offer a finer granularity in representing model weights, potentially allowing for more compact models or faster computation on hardware optimized for floating-point operations. Theoretically, MXFP4 should offer advantages in either speed or accuracy, or both, compared to older integer-based formats like Q4_K_M, especially on hardware that excels at FP arithmetic.

The core difference lies in how the model's weights are stored and processed. Integer quantization, like in Q4_K_M, uses whole numbers, which are generally faster to compute on most CPUs and some specialized hardware. Floating-point representations, as used in MXFP4, offer a wider range of values and precision but can be more computationally intensive, depending on the hardware's FP capabilities and the specific implementation.

Possible Explanations for the Discrepancy

Several factors could explain why the newer MXFP4 format underperformed. One primary suspect is the optimization level of the inference engine and hardware drivers for each format. While MXFP4 might be theoretically superior, the software stack—including the LLM runtime (like llama.cpp) and the underlying GPU/NPU drivers—may not yet be as highly optimized for this newer format as it is for the more established Q4_K_M.

Think of it like a brand new, highly efficient engine design. If the car's chassis, transmission, and tires aren't yet engineered to take full advantage of that engine's potential, it might not outperform an older, well-understood engine in a perfectly tuned vehicle. The surrounding software and hardware ecosystem plays a critical role.

Another possibility relates to the specific implementation of MXFP4. The 'microscaling FP' approach could introduce overheads in the quantization or dequantization process during inference that negate its theoretical benefits on certain hardware. Apple's M-series chips have a unified memory architecture and specialized Neural Engines, and the way MXFP4 interacts with these specific components might be less efficient than Q4_K_M's established interaction patterns.

Furthermore, the test was conducted on a single laptop model. Performance can vary significantly across different hardware configurations, even within the same family of chips. The specific memory bandwidth, core configurations, and thermal management of the test machine could all influence the outcome. It's possible that on different hardware, perhaps with more robust FPUs or different memory subsystems, MXFP4 might show its intended advantages.

Implications for Local LLM Users

This test highlights a crucial point for anyone running LLMs locally: benchmarks and marketing claims should always be verified with real-world testing on your specific hardware. The performance of quantization formats is not a universal constant; it is deeply tied to the software stack and hardware architecture.

For developers and users seeking the fastest inference speeds on consumer hardware, particularly Apple Silicon, the current data suggests that sticking with well-established formats like Q4_K_M might still yield superior results. This doesn't mean MXFP4 is inherently flawed, but rather that its widespread adoption and optimization across various platforms may still be in its early stages.

As LLM technology rapidly evolves, the performance characteristics of different formats can shift quickly. Ongoing development in inference engines and hardware support will likely lead to improvements for newer formats like MXFP4. However, for immediate deployment and performance-critical applications, empirical testing remains the most reliable path to determining the optimal configuration.