Text Vectorization: Turning Words into Numbers

Computers work with numbers, not text. To handle language, a Natural Language Processing (NLP) system must convert words, sentences, or documents into numerical data. Usually, this means turning them into vectors—ordered arrays of numbers. This process is called text vectorization.

A vector is a mathematical summary of a piece of text. The details and meaning behind the numbers depend on which vectorization method is used, but all serve a common purpose: to translate language into something a machine can process.

For example, imagine building a program to filter spam emails. The program can't directly understand words like "WINNER" or "sale." Every word must be mapped to a numerical representation. Text vectorization provides this bridge between human language and machine computation. Without it, tasks like sentiment analysis, topic modeling, and machine translation would be impossible.

The Bag-of-Words Model Explained

One of the simplest and most intuitive text vectorization techniques is the Bag-of-Words (BoW) model. The name itself offers a clue: imagine taking all the words in a document, throwing them into a bag, and then counting how many times each word appears. The order of words is completely disregarded; only their frequency matters.

Let's break down how this works with an example. Consider these two short sentences:

  • Sentence 1: "The cat sat on the mat."
  • Sentence 2: "The dog chased the cat."

First, we need to build a vocabulary of all unique words present in our corpus (in this case, these two sentences). The unique words are: "The", "cat", "sat", "on", "the", "mat", "dog", "chased". After removing duplicates and potentially standardizing case (e.g., converting "The" to "the"), our vocabulary becomes: ["the", "cat", "sat", "on", "mat", "dog", "chased"].

Visual representation of a vocabulary list derived from text documents.

Next, for each sentence, we create a vector where each element corresponds to a word in the vocabulary. The value of each element in the vector is the count of that word in the sentence. For Sentence 1 ("The cat sat on the mat."), assuming our vocabulary is ["the", "cat", "sat", "on", "mat", "dog", "chased"], the BoW vector would be [2, 1, 1, 1, 1, 0, 0].

For Sentence 2 ("The dog chased the cat."), the BoW vector would be [2, 1, 0, 0, 0, 1, 1].

This process transforms unstructured text into structured numerical data that machine learning algorithms can readily consume. The vector represents the document's content based on word occurrences.

Advantages and Disadvantages of Bag-of-Words

The BoW model's primary strength lies in its simplicity and ease of implementation. It provides a straightforward way to quantify text data, making it suitable for basic tasks like document classification or clustering where the presence and frequency of words are strong indicators.

However, BoW has significant limitations. The most critical is its inability to capture word order or context. In the example above, "The cat sat on the mat" and "The mat sat on the cat" would have identical BoW vectors, despite having vastly different meanings. This is akin to having a bag of ingredients but no recipe—you know what you have, but not how to combine it.

Another drawback is the high dimensionality of the resulting vectors, especially with large vocabularies. Many words might appear only once or twice across a vast corpus, leading to sparse vectors (vectors with many zeros). This can increase computational cost and memory requirements for machine learning models.

Beyond Basic Counts: TF-IDF

To address some of BoW's shortcomings, particularly the issue of common words dominating the representation, we often use Term Frequency-Inverse Document Frequency (TF-IDF).

TF-IDF is a weighting scheme that not only counts how often a word appears in a document (Term Frequency or TF) but also considers how rare that word is across the entire corpus (Inverse Document Frequency or IDF).

The intuition behind TF-IDF is that words that appear frequently in a specific document but rarely in others are more informative and discriminative. For example, the word "algorithm" might be very important in a document about computer science, but if it appears in almost every document in a general corpus, its IDF score will be low, reducing its overall TF-IDF weight.

The TF-IDF score for a word in a document is calculated as:

TF-IDF(t, d, D) = TF(t, d) * IDF(t, D)

Where:

  • TF(t, d) is the Term Frequency of term 't' in document 'd'.
  • IDF(t, D) is the Inverse Document Frequency of term 't' across the corpus 'D'.

The IDF is typically calculated as:

IDF(t, D) = log( |D| / (df(t) + 1) )

Where:

  • |D| is the total number of documents in the corpus.
  • df(t) is the number of documents containing the term 't'. The '+1' is added to prevent division by zero if a term is not present in any document.

By using TF-IDF, we get vectors that better represent the semantic importance of words within a document relative to the entire collection of documents, moving beyond simple word counts.

Other Text Vectorization Methods

While BoW and TF-IDF are foundational, modern NLP employs more sophisticated techniques:

  • Word Embeddings (e.g., Word2Vec, GloVe, FastText): These methods represent words as dense, low-dimensional vectors in a continuous vector space. Unlike BoW or TF-IDF, word embeddings capture semantic relationships between words. Words with similar meanings will have similar vector representations. For instance, the vectors for "king" and "queen" might be close in this space, and the relationship "king" - "man" + "woman" might approximate "queen". These models are typically trained on massive text corpora.
  • Sentence and Document Embeddings (e.g., Doc2Vec, Sentence-BERT): These extend the concept of word embeddings to entire sentences or documents, creating a single vector that encapsulates the meaning of a larger piece of text.
  • Transformer-based models (e.g., BERT, GPT): These state-of-the-art models generate contextualized word embeddings. This means the vector representation of a word can change depending on its surrounding words in a sentence, effectively addressing the context-blindness of BoW.

Each of these methods offers different trade-offs in terms of computational complexity, performance, and the richness of the semantic information captured. For developers starting in NLP, understanding Bag-of-Words and TF-IDF is crucial as they form the bedrock upon which more advanced techniques are built.