Introducing the Benchmark
A new benchmark has emerged to evaluate the performance of several small, CPU-based Text-to-Speech (TTS) models. The comparison, shared on Reddit's r/MachineLearning, focuses on objective Mean Opinion Scores (MOS) derived from the UTMOS scoring system. The primary motivation for this benchmark was to include Kyutai's Pocket TTS, a model architecturally distinct from others in its class, and to provide a head-to-head comparison with established models.
The evaluation aims to assist developers and researchers in selecting appropriate TTS solutions when GPU acceleration is not an option or when a lightweight, CPU-deployable model is required. This is particularly relevant for edge computing scenarios, embedded systems, or applications where cost and resource constraints limit the use of powerful GPUs.
Models Under Scrutiny
The benchmark rigorously tested four distinct TTS models, each representing a different architectural approach and scale:
- Kokoro (82M parameters): Implemented in both PyTorch and ONNX Runtime, this model is inspired by the StyleTTS2 architecture. Its performance is a key reference point for StyleTTS2-based approaches on CPU.
- Supertonic (3 models): This entry utilizes a Vector Estimator backbone and was tested at two different configurations: 2 and 5 flow-matching steps. The number of steps directly impacts computational load and potentially the quality of the generated speech.
- Inflect-Nano-v1 (4.6M parameters): Positioned at the very small end of the spectrum, this model follows a FastSpeech-style architecture. Its extremely low parameter count makes it a candidate for highly resource-constrained environments, but its quality is often a trade-off.
- Pocket TTS (~100M parameters): Developed by Kyutai, this model stands out due to its streaming Language Model (LM) architecture operating over Kyutai's Mimi neural audio codec. This unique approach aims to balance quality with efficient streaming capabilities, a departure from many traditional TTS pipelines.
Methodology and Setup
The benchmark was conducted on a specific hardware configuration to ensure consistency. The test system featured an Intel Xeon 8272CL CPU with 4 cores and 15.6GB of RAM. Crucially, CUDA (GPU acceleration) was disabled at the environment level, forcing all computations onto the CPU. For models utilizing ONNX Runtime, the ONNX sessions were explicitly pinned to the CPUExecutionProvider.
The testing matrix was designed to cover a range of scenarios. Six different configurations were tested, encompassing six distinct text lengths ranging from a concise 12 characters to a lengthy 1712 characters. To account for initial overhead and ensure stable performance measurements, five timed repetitions were performed for each text length and model configuration after an initial discarded warmup run. This resulted in a total of 180 individual runs across all tested models and conditions.
The primary metric for evaluation was the objective Mean Opinion Score (MOS) obtained using UTMOS. Specifically, the utmos22_strong model was employed to score every generated WAV file. UTMOS is a learned MOS predictor that aims to correlate highly with human perceptual quality judgments, providing an objective, reproducible measure of speech naturalness and quality.
Performance Insights and Observations
While the full dataset of scores is available through the original source, preliminary observations highlight the trade-offs inherent in CPU-based TTS. Models with lower parameter counts, like Inflect-Nano, are expected to exhibit faster inference times but may sacrifice audio quality. Conversely, larger models like Kokoro and Pocket TTS, despite being optimized for CPU, will likely have higher latency but potentially better fidelity.
Pocket TTS's unique streaming LM architecture over a neural codec presents an interesting case. Its performance will be critical in determining whether this novel approach can compete with more traditional, albeit potentially slower, methods on CPU. The benchmark's focus on objective MOS scores allows for a direct comparison of perceived quality, abstracting away from subjective human listening panels which are time-consuming and less reproducible.
The inclusion of both PyTorch and ONNX Runtime for Kokoro is also significant. ONNX Runtime often provides performance optimizations for CPU inference, and observing its gains over native PyTorch could inform deployment strategies. The detailed setup, including specific CPU, RAM, and the disabling of CUDA, provides a clear blueprint for others to replicate or adapt the benchmark for their own evaluations.
This benchmark serves as a valuable resource for anyone needing to deploy TTS on devices without readily available GPUs. It provides empirical data on how different architectural choices and model sizes translate into tangible performance metrics on commodity hardware. The use of UTMOS offers a standardized, objective measure of quality, making the results directly comparable and actionable.
The broader implication is the continued viability and development of efficient TTS models capable of running on a wide range of hardware. As edge AI and on-device processing gain traction, benchmarks like this are crucial for guiding development and adoption of the most suitable technologies.
What remains to be seen is how these objective scores translate to specific use cases. While UTMOS is a strong proxy for human perception, real-world applications might have varying tolerances for latency, specific phonetic inaccuracies, or prosodic nuances that a single MOS score might not fully capture.