Low-Capacity Networks Bridge Categorical Data Gap for Continuous Scoring

The Challenge of Sparse Data in Scoring

In many real-world scenarios, obtaining precise, continuous numerical scores for evaluation is hampered by the availability of only categorical labels. This is particularly common in domains where human judgment is involved, such as content moderation, sentiment analysis, or quality assessment. For instance, a reviewer might categorize an image as 'good,' 'average,' or 'poor,' but a nuanced score on a scale of 1 to 10 would offer much richer information for downstream analysis, model training, or performance benchmarking. Traditional supervised learning methods often struggle here, as they typically require paired continuous labels for regression tasks. The scarcity of granular data forces practitioners to either accept the limitations of categorical outputs or resort to expensive and time-consuming manual annotation efforts to generate continuous labels.

The core problem is that categorical data, by its nature, represents discrete bins of information. Converting these bins into a continuous spectrum without additional data can lead to information loss or introduce arbitrary assumptions. For example, simply assigning numerical values like 1, 2, and 3 to 'poor,' 'average,' and 'good' assumes an equal distance between these categories, which is rarely the case in practice. This post explores a novel approach that leverages the power of neural networks, specifically low-capacity (or under-parameterized) ones, to infer continuous scores directly from categorical training data.

Leveraging Low-Capacity Networks for Score Inference

The proposed method hinges on the surprising effectiveness of under-parameterized neural networks in learning representations that generalize well, even when the network capacity is theoretically insufficient to perfectly memorize the training data. In this context, an under-parameterized network, when trained on categorical labels, is encouraged to find a simpler, underlying structure that best explains the relationships between the categories and the features of the data points. Instead of simply classifying an input into one of the predefined categories, the network is trained to output a continuous value. This continuous output is then interpreted as a proxy for a more granular score.

Consider a scenario where you have images of product quality, labeled as 'defective,' 'acceptable,' or 'premium.' A standard classification network would learn to distinguish between these three classes. However, by framing this as a regression problem and training an under-parameterized network, the network is compelled to learn a representation where 'defective' items map to lower continuous values, 'acceptable' items to intermediate values, and 'premium' items to higher values. The network's internal weights and biases learn to project the input features into a one-dimensional space that inherently captures the ordinal relationship between the categories.

The mathematical underpinning involves training a neural network with a single output neuron, typically with a linear or sigmoid activation function, to predict a continuous score. The loss function used is a standard regression loss, such as Mean Squared Error (MSE), comparing the network's predicted continuous score against a target value. The crucial element is the choice of network architecture. By deliberately keeping the network 'shallow' or 'narrow' – meaning it has fewer layers or fewer neurons per layer than would be strictly necessary to classify the data perfectly – the network is prevented from simply learning hard decision boundaries for each category. Instead, it must learn a smoother mapping that reflects the underlying continuous quality.

The Mathematics Behind the Inference

Let $X$ be the input features (e.g., image pixels, text embeddings) and $Y_{cat}$ be the categorical label (e.g., 'defective', 'acceptable', 'premium'). We want to learn a function $f(X)$ that outputs a continuous score $s \in \mathbb{R}$. The training data is a set of pairs $(X_i, Y_{cat,i})$.

We train a neural network $N_{\theta}$ with parameters $\theta$. The network has an architecture designed to be under-parameterized. The output layer produces a single continuous value, $s_i = N_{\theta}(X_i)$.

The objective is to minimize a loss function $L$ that penalizes the difference between the predicted score $s_i$ and some target value. Since we only have categorical labels, we need to assign target values. A common approach is to map the categories to ordered numerical values, say $T(Y_{cat})$. For example, if $Y_{cat} \in \{$'defective', 'acceptable', 'premium'$\}$, we could map them to targets $T = \{1, 2, 3\}$. However, this is an oversimplification. A more nuanced approach acknowledges that the true continuous scores for these categories likely form ranges, not single points.

The power of the under-parameterized network lies in its ability to learn a mapping that respects the ordinal nature of the categories without being constrained by fixed target points. The MSE loss, $L = \frac{1}{N} \sum_{i=1}^{N} (s_i - T_i)^2$, drives the network to produce scores that are ordered according to the categories. The under-parameterization acts as a form of implicit regularization, preventing overfitting to the categorical labels and encouraging the discovery of a more generalizable continuous score.

The surprising detail here is not the use of neural networks, but the deliberate choice of *under-capacity*. Typically, we aim for sufficient capacity to model complex functions. Here, limiting capacity forces the network to find the simplest possible continuous representation that aligns with the ordered categories. This is analogous to fitting a simple line through a few points that are known to trend upwards, rather than trying to fit a high-degree polynomial that might capture noise.