ML Predictors Need Dataset Size, Not Model Size, for Accuracy

The Data-Accuracy Nexus in ML Predictors

Determining the optimal number of design experiments for machine learning (ML) predictors is a perennial challenge in engineering and scientific research. Traditionally, engineers might consider model complexity or the number of design variables as primary factors influencing the required dataset size. However, recent insights suggest a different paradigm: the smoothness of the response surface is the critical determinant of achievable accuracy, directly correlating with the required dataset size.

This perspective challenges conventional wisdom, which often focuses on the intricate architecture of ML models themselves. While model sophistication plays a role in capturing complex relationships, its impact on prediction accuracy in design exploration is secondary to the fundamental characteristic of the data it learns from. The core idea is that if the underlying phenomenon being modeled exhibits a smooth response to changes in design variables, fewer data points are needed to accurately map that response. Conversely, a highly non-linear, erratic, or noisy response surface necessitates a larger dataset to ensure the ML model can reliably interpolate and extrapolate across the design space.

Think of it like mapping a landscape. If the terrain is mostly flat with gentle rolling hills, a few well-placed elevation markers will give you a good overall picture of the topography. But if the landscape is a jagged mountain range with deep canyons and sheer cliffs, you'll need many more measurements to accurately represent its complex contours. The 'smoothness' of the response surface is analogous to the gentle hills, while a 'rough' surface is like the mountain range.

Understanding Response Surface Smoothness

The concept of response surface smoothness is crucial. In design of experiments (DOE), the goal is often to understand how changing input parameters (design variables) affects an output metric (response). This relationship can be visualized as a surface, where the height at any point represents the response value for a given combination of design variables. A smooth response surface implies that small changes in input variables lead to small, predictable changes in the output. This predictability makes it easier for an ML model to learn the underlying function.

Conversely, a rough or highly non-linear response surface indicates that small changes in input variables can lead to large, abrupt, or unpredictable changes in the output. This could be due to complex interactions between variables, threshold effects, or inherent noise in the system being modeled. To accurately capture such a surface, an ML model needs to be trained on a significantly larger and more diverse set of data points to avoid overfitting to noise or missing critical non-linearities.

The implication for practitioners is clear: before diving deep into model selection or hyperparameter tuning, engineers should first analyze the expected or observed smoothness of their response surface. Techniques for characterizing this smoothness can guide decisions about the scale of experimental design and the expected accuracy achievable with a given dataset size.

Dataset Size vs. Model Size

The distinction between dataset size and model size is fundamental. A larger dataset, with more experimental runs or observations, provides more information about the system's behavior across the design space. This richer information allows ML models, even simpler ones, to generalize better and achieve higher accuracy. A complex model, with many parameters or layers, has a higher capacity to learn intricate patterns. However, without sufficient data, a complex model is prone to overfitting – it learns the training data, including its noise and idiosyncrasies, too well, leading to poor performance on new, unseen data.

The research highlights that the accuracy ceiling is often determined by the data's ability to represent the true underlying function. If the function is inherently smooth, even a relatively simple model trained on an adequate dataset can achieve high accuracy. If the function is complex and rough, even the most sophisticated model will struggle unless fed a vast amount of data that adequately samples this complexity.

This finding is particularly relevant in fields like semiconductor design, where experiments can be time-consuming and expensive. Optimizing the number of design experiments directly impacts resource allocation. Prioritizing the generation of a dataset that sufficiently captures the response surface's characteristics, rather than solely focusing on building the most complex predictor, can lead to more efficient and effective design exploration.

Practical Implications for Design Experiments

For engineers and data scientists involved in design experimentation, this insight shifts the focus. Instead of debating the merits of a convolutional neural network versus a gradient boosting machine, the primary question becomes: "How well does my experimental design sample the response surface?" This requires a deeper understanding of the physical or chemical processes being modeled.

Key considerations include:

• Understanding the System: Leverage domain knowledge to anticipate the expected smoothness of the response. Are there known non-linearities or interaction effects?
• Exploratory Data Analysis: After initial experiments, analyze the data for signs of roughness. Are there sudden jumps in performance? Are interactions between variables strong and complex?
• Adaptive Sampling: If the response surface appears rough, consider adaptive sampling strategies. These methods focus subsequent experiments on regions where the model's uncertainty is high or where significant non-linearities are suspected.
• Data Augmentation: When possible, explore methods to augment existing data or conduct additional experiments in critical regions, rather than solely increasing model complexity.

The takeaway is that achieving high accuracy in ML predictors for design tasks is less about the ML model's internal architecture and more about the quality and quantity of data used to train it, specifically how well that data captures the true, underlying response surface. Investing in a well-designed experimental strategy that generates a representative dataset is paramount.