The Limits of Traditional NLP: A Case Study in Author Identification

The rapid evolution of Natural Language Processing (NLP) often centers on deep learning models like transformers. However, a recent experiment on Kaggle's Spooky Author Identification task demonstrates that classical NLP techniques, when applied rigorously and in combination, can still achieve remarkable performance. This exploration dives deep into the capabilities of traditional methods, moving from fundamental text representations to sophisticated ensemble techniques.

The Spooky Author Identification dataset presents a challenging classification problem: given a short text excerpt, predict its author from a set of well-known gothic literature figures: Edgar Allan Poe, H.P. Lovecraft, and Mary Shelley. These authors have distinct writing styles, but their works also share thematic and linguistic commonalities, making the task non-trivial. The experiment aimed to systematically evaluate how far one could push these classical methods before needing more advanced deep learning architectures.

Feature Engineering and Representation Survey

The journey began with an examination of various text representation methods. Each method captures different aspects of the text, and understanding their strengths and weaknesses is crucial for building effective models. The survey included:

  • Bag-of-Words (BoW): This is perhaps the most fundamental NLP representation. It treats each document as a collection of its words, disregarding grammar and even word order but keeping track of frequency. While simple, it forms a solid baseline.
  • TF-IDF (Term Frequency-Inverse Document Frequency): An improvement over basic BoW, TF-IDF weights words not just by their frequency in a document but also by how rare they are across the entire corpus. This helps to highlight important, distinguishing words while downplaying common words like "the" or "a".
  • BM25: A ranking function used by search engines, BM25 is a more sophisticated variant of TF-IDF. It considers term frequency within a document, inverse document frequency, and document length, providing a more nuanced score for term importance.
  • Word Embeddings (Word2Vec, FastText): These methods represent words as dense vectors in a continuous vector space. Words with similar meanings are located closer to each other in this space. Word2Vec learns embeddings by predicting context words, while FastText also considers subword information (character n-grams), making it robust to out-of-vocabulary words and capturing morphological nuances. For sentence or document representation, these word vectors are typically averaged or aggregated.

The experiment meticulously evaluated these representations, noting how each contributed to the model's ability to discern authorial style. For instance, TF-IDF and BM25 excel at capturing keyword importance, which is often indicative of an author's preferred vocabulary and subject matter. Word embeddings, on the other hand, can capture semantic relationships and stylistic nuances that pure word counts might miss. The choice of representation directly impacts the feature space fed into the subsequent classification models.

Visualisation of TF-IDF scores for key terms across different author texts.

Baselines and Initial Models

With representations in hand, the next step involved establishing strong baselines. The experiment employed two primary classical models:

  • Naive Bayes (NB) with TF-IDF: A probabilistic classifier that applies Bayes' theorem with a strong (naive) independence assumption between features. When combined with TF-IDF features, it's a robust and fast baseline for text classification tasks. It often performs surprisingly well, especially when word choice is a strong predictor.
  • Support Vector Machines (SVM) with TF-IDF: SVMs are powerful algorithms for classification, finding an optimal hyperplane that separates data points of different classes. SVMs, particularly with a linear kernel, are effective on high-dimensional sparse data like TF-IDF vectors.

These models served as crucial benchmarks. They established a performance floor and provided insights into the discriminative power of the TF-IDF representation. The results from these baselines indicated that classical methods were already capturing significant signals related to authorial style.

Advanced Techniques: Stacking Ensembles

To push performance further, the experiment moved to ensemble methods, specifically stacking. Stacking combines predictions from multiple diverse base models to create a more robust and accurate final prediction. The core idea is that different models learn different aspects of the data, and by intelligently combining their outputs, the ensemble can often outperform any single model.

The process involved training several base models, each potentially using different feature representations (e.g., a Naive Bayes model on TF-IDF, an SVM on Word2Vec averages, a Logistic Regression on BM25 scores). The predictions of these base models on a validation set were then used as input features for a meta-model (or blender). This meta-model, often a simple model like Logistic Regression or a Gradient Boosting Machine, learns how to best combine the base model predictions. The experiment explored stacking various combinations of the previously surveyed representations and base classifiers.

The surprising detail here is not just the effectiveness of stacking, but the degree to which it amplified the performance of already strong classical models. By training a meta-model to learn the patterns in the base models' errors and successes, the ensemble could correct for individual model weaknesses. This approach is akin to assembling a team of specialists: each expert (base model) has their strengths, but a skilled manager (meta-model) can leverage their collective intelligence to solve problems no single expert could tackle alone.

Diagram illustrating the architecture of a stacked ensemble model for text classification.

Performance and Conclusions

The experiment culminated in a stacked ensemble model that achieved highly competitive results on the Spooky Author Identification task. The performance gains realized through stacking demonstrated the power of combining diverse learning algorithms and feature representations. This suggests that even with the advent of deep learning, carefully engineered classical NLP pipelines, especially when augmented by ensemble methods, remain potent tools for many text classification problems.

The research indicates that for tasks where stylistic features, vocabulary usage, and sentence structure are strong discriminators, classical NLP methods can achieve performance that rivals or even surpasses simpler deep learning baselines. The ability to interpret features (like TF-IDF scores) and the computational efficiency of many classical models make them attractive options, particularly when computational resources are constrained or when interpretability is a priority. This work serves as a valuable reminder that the foundation of NLP—feature engineering and robust classical algorithms—continues to hold significant relevance.

What remains to be fully explored is the precise point at which the complexity and data requirements of deep learning models become unequivocally necessary for surpassing the best classical approaches in author identification and similar stylistic classification tasks. This experiment suggests that the threshold might be higher than commonly assumed.