A Few Interview Questions I Like (1): Word Vectors
In interviews I sometimes drop a few questions that look almost too basic. The point is not to trap anyone — it is to see quickly whether a concept has been internalized, or whether it still lives only as a name.
This is the first piece in the series. We start from a foundation of modern NLP: word vectors.
The core property compresses into one sentence: words that are close in meaning or concept are also close in vector space. Fruits like apple, banana, and watermelon cluster together; places like Beijing, Shanghai, and Hangzhou do too. So where does that property come from? Coincidence, or is there a reason?
Here is how I see it: the answer needs both a linguistic regularity and an algorithmic design. Neither is enough alone.
When I was learning word2vec, I kept hearing the phrase distributional hypothesis. For years the usual Chinese rendering — 「分布式假设」— was so opaque that I never really felt what it meant. Eventually I decided it was pointing at a distributional regularity in language: a word’s appearances are tied to the contexts it sits in; sentences and words are not thrown together at random. After “I want to eat…” you almost always get something in the food category, not a place name or something else.
I asked Claude to build a small demo of the center–context relationship:
Interactive — distributional regularity
Slide a window over a corpus and the semantics surface on their own
The thirteen lines below are a tiny corpus. A fixed-width window starts at the first word and steps forward, one token at a time; at every stop, the words inside the window become the center word's "context." That is the only rule — no model, no vectors. Watch the three fruit lines and the three city lines: once the sweep finishes, the contexts each group collected are nearly the same set.
This interactive needs JavaScript. After the window finishes the corpus, the contexts collected for "apple" are ate, an, This, tastes, sweet — and "banana" / "watermelon" collect almost the same list — while "Beijing" collects will, visit, housing, prices, duck, tastes. Same-type words share about five contexts; cross-type sharing is near zero. The gap comes entirely from the corpus itself; nobody labeled "this is a fruit, that is a city."
This is a toy corpus written to make the mechanism visible. Real text is much messier, but the window rule is exactly this simple — it is how word2vec draws its training samples. You can change the radius: smaller favors tight collocations; larger leans toward topical relatedness.
Second, the algorithm matters — otherwise why don’t language models, or other word-vector models, put the same emphasis on this property? Skip-gram / CBOW line up with the linguistic story above, so the property shows up only when the two meet.
Saying it is “about the algorithm” is a bit abstract. What skip-gram actually does is wire that window into gradient descent: every stop of the window emits a few pairs, and those pairs are the training samples. And the “neither alone is enough” claim can be tested directly — leave the algorithm untouched, shuffle the word order in the corpus, and see what is left.
Interactive — the algorithmic half
Same window, wired into skip-gram
Still the same window — except now every stop hands each "center → context" pair to gradient descent, pushing the center vector to predict those context words more easily. That is all of skip-gram's input, and all of what it does. The six cards below keep each word's homework on the page; the bar at the bottom is its cosine with "apple." Read them side by side: how the corpus-given questions line up with the training-given geometry. Then switch to the control: leave the algorithm alone, shuffle only the word order.
This interactive needs JavaScript. It really trains a skip-gram in the page: every window stop emits "center → context" samples, and negative sampling with SGD pushes the model's confidence on those pairs toward 1. Because "apple" and "banana" are repeatedly asked to predict the same words (ate, an, This, tastes, sweet), their vectors have nowhere to go but toward each other: after eighty epochs, cos(apple, banana) is 1.00 and cos(apple, Beijing) is about 0.12. Shuffle the corpus and retrain with the same algorithm and epochs — the two numbers become hard to tell apart. Linguistic regularity decides which words share homework; the training objective turns "same homework" into "same direction." You need both.
Vectors are 4-D here: real models have hundreds of dimensions; this is enough to show the effect and still fast. The number under each context word is the model's current confidence; with negative sampling it does not converge to 1 — watch the trend. Use “Train all” for the result, or “Step” / “Jump to next target” to see how samples form. The RNG seed is fixed, so every refresh looks the same. The control still converges: fitting is not the same as learning a semantic representation.