ExecuTorch MLX Delegate Boosts LLM Performance on Macs

Running large language models (LLMs) locally on Macs has become increasingly accessible, yet optimizing PyTorch models for Apple Silicon and executing them with lightweight runtimes remains an active area of development. A new contender, ExecuTorch's experimental MLX delegate, released in May 2026, promises significant performance gains. This delegate allows PyTorch models to leverage Apple Silicon GPUs. The author tested ExecuTorch 1.3.1 against PyTorch's native MPS (Metal Performance Shaders) backend using the Qwen3-0.6B model, focusing on decode throughput and the effects of 4-bit quantization.

The results are compelling. Using BF16 precision, PyTorch MPS achieved 41.8 tokens/s. The MLX delegate, also with BF16, nearly tripled this performance to 134.8 tokens/s. The most dramatic improvement came with 4-bit quantization (INT4) using the MLX delegate, which pushed decode throughput to 188.9 tokens/s. This represents a 4.52x speedup compared to PyTorch MPS BF16.

Beyond speed, the MLX INT4 delegate also delivered substantial file size savings. The quantized model was 71.8% smaller than its BF16 counterpart. However, this aggressive quantization was not without consequence. Testing with three simple prompts revealed that INT4 altered the generated output in two out of the three cases, indicating a trade-off between performance and output fidelity.

Understanding the ExecuTorch MLX Delegate

ExecuTorch is an open-source project designed to enable the execution of trained PyTorch models across a range of devices, including desktops, mobile phones, and embedded systems. Its core function is to provide a portable and efficient runtime environment for deploying machine learning models outside of the typical Python development ecosystem. The MLX delegate is a specific extension within ExecuTorch that targets Apple Silicon hardware (M1, M2, M3 series chips). It leverages the MLX framework, Apple's own high-performance machine learning library for unified memory on Apple Silicon, to accelerate model inference.

The delegate acts as an intermediary, translating PyTorch operations into a format that MLX can efficiently process on the Apple GPU. This offloading of computation from the CPU to the GPU, combined with MLX's optimized kernels and memory management for Apple Silicon, is the primary driver of the observed performance improvements. Unlike PyTorch's MPS backend, which is more general-purpose for Metal, the MLX delegate aims for deeper integration and optimization specific to the MLX framework and Apple's hardware architecture.

Performance Benchmarks: Qwen3 on M1 Max

The benchmark focused on the decode throughput of the Qwen3-0.6B model, a relatively small but capable LLM. The tests were conducted on an M1 Max chip, a high-performance processor found in certain MacBook Pro and Mac Studio models.

  • PyTorch MPS (BF16): 41.8 tokens/s
  • ExecuTorch MLX (BF16): 134.8 tokens/s (3.2x faster than MPS)
  • ExecuTorch MLX (INT4): 188.9 tokens/s (4.52x faster than MPS BF16)

These figures highlight the significant advantage of using the MLX delegate, particularly when combined with 4-bit quantization. The ability to achieve such high throughput on local hardware opens up new possibilities for running sophisticated LLMs without relying on cloud-based APIs, which can incur costs and introduce latency.

The Impact of 4-bit Quantization

Quantization is a technique used to reduce the precision of model weights and activations, typically from 32-bit floating-point numbers to lower bit representations like 16-bit or 8-bit integers, and in this case, 4-bit integers. The primary benefits are reduced memory footprint and faster computation, as lower-precision arithmetic operations are generally quicker and require less memory bandwidth.

The 4.52x speedup and the 71.8% file size reduction achieved with INT4 quantization are substantial. For developers deploying LLMs on resource-constrained devices or aiming for maximum performance, 4-bit quantization is an attractive option. However, this performance boost comes at the cost of potential accuracy degradation.

In this experiment, the Qwen3 model with INT4 quantization exhibited noticeable changes in output for two out of three simple test prompts. While the specific nature of these changes isn't detailed, it implies that for tasks requiring high factual accuracy or nuanced language generation, INT4 might not be suitable without further fine-tuning or architectural adjustments. The BF16 version, while slower, maintained output fidelity comparable to the original PyTorch model.

What Does This Mean for Developers?

The ExecuTorch MLX delegate represents a promising step towards efficient on-device LLM inference for Apple Silicon users. It offers a clear path to significantly accelerate PyTorch models, making it more feasible to run powerful LLMs locally. The trade-off with 4-bit quantization is a critical consideration: developers must weigh the dramatic performance and size benefits against the potential impact on output quality. For applications where absolute precision is paramount, sticking with higher precision formats like BF16 or FP16 might be necessary. However, for many use cases, such as real-time text generation, summarization, or code completion where minor output variations are acceptable, the INT4 quantized models running via the MLX delegate could offer an unparalleled local experience.

The fact that this is an experimental delegate, released in May 2026, suggests that it is still evolving. Future iterations may bring further optimizations, improved quantization techniques, or broader model support. The success of this delegate also points to a growing trend of specialized runtimes and delegates designed to extract maximum performance from specific hardware architectures, moving beyond more general-purpose inference engines.

What remains to be seen is how this performance scales with larger models and more complex inference tasks. While Qwen3-0.6B is a good starting point, testing with state-of-the-art models will be crucial to understanding the full potential and limitations of the ExecuTorch MLX delegate for widespread developer adoption.