2018 · Representations in Context
The word stays the same. Its vector does not.
Word2Vec gives every word one permanent vector. That means bank beside a river and bank holding a loan are literally identical inside the model. ELMo changed the lookup into a computation: read the sentence in both directions, then build each word's representation from what surrounds it.
This laptop-sized demonstration keeps that conceptual move and strips away the large bidirectional network. It starts from real co-occurrence vectors learned from the sonnets, then blends the target with the current ±2-word context. The static vector is unchanged; the contextual result bends toward the words present now.
thy fair face is bright
a fair judgement and honest mind
The first twelve dimensions of two context-conditioned vectors. Different neighbours pull the same static word in different directions.
Try it — change the context
The context rule is fixed. ELMo learns how syntax and meaning should change a vector; this demo averages neighbouring evidence.
The corpus is tiny. Rare meanings have little statistical support.
A window is not a sentence. Long-range dependencies still fall outside ±2 words.