From Bag of Words to Word2Vec:
Fixing the Semantic Gap

The Problem with Common Words

Because Bag of Words only cares about raw frequency, it gets completely distracted by filler words.

Imagine you have a highly technical article about AI. If you run it through a Bag of Words model, the highest-scoring words won't be "LLMs" or "Data." The highest-scoring words will almost certainly be: "the," "is," "and," "of," and "to."

Even if we manually remove "stop words" from our dictionary, BoW has a fundamental flaw: it cannot measure a word's uniqueness across a whole library of documents.

Enter TF-IDF: Term Frequency & Squashing

To fix this, we need a better formula. We start with Term Frequency (TF): How many times does this word appear in this specific document?

But we don't just use raw counts. If a word appears 1000 times, it doesn't mean the document is 1000x more relevant than a document where it appears once. Instead, data scientists use a technique called logarithmic squashing.

The formula is TF = log₁₀(count(t, d) + 1).
If the word "the" appears 1,000 times, the math makes it log₁₀(1001) = 3.0.
If "data" appears 10 times, it becomes log₁₀(11) = 1.04.
Suddenly, that massive, misleading gap is heavily reduced!

Inverse Document Frequency (IDF)

The second half of the magic is IDF. It asks: How rare is a word across ALL documents?

The formula is IDF = log₁₀(N / dfₜ) (where N is total documents, and dfₜ is how many documents contain the word).

If "the" appears in every single document, the math results in a multiplier of exactly 0. We take our Term Frequency and multiply it by IDF. Mathematically, TF-IDF automatically and ruthlessly neutralizes words that appear everywhere, shining a spotlight on the rare keywords that give text its true meaning.

The Curse of Dimensionality & Semantic Gap

TF-IDF was a massive upgrade, but it still shares a fundamental flaw with Bag of Words: It relies entirely on exact spelling matches. It doesn't understand synonyms.

Take the phrases "The puppy sprinted" and "The dog ran." As humans, we know these mean the exact same thing. But to TF-IDF, because there are zero overlapping words, the mathematical similarity between these two sentences is 0%.

Furthermore, to track a 100,000-word dictionary, you need an array with 100,000 slots for every single sentence, most of which are completely empty zeros. This is known as the Curse of Dimensionality.

The Solution: Word2Vec

By 2013, researchers realized that counting words was a dead end for true comprehension. A famous linguist named John Rupert Firth once said:

"You shall know a word by the company it keeps."

Instead of creating massive lists of zeros, what if a neural network learned to predict a word based on its neighbors? This led to the creation of Word2Vec. This was a huge shift—instead of treating words as separate, unrelated things, AI finally started understanding how their meanings connect to each other.

But how exactly do you turn "meaning" into a math equation? Stay tuned for Part 2...