The NLP Journey — finale
One line of Shakespeare, handed to every machine in this museum, in the order they were invented.
Every page here runs on the same 154 sonnets. That started as a convenience — one corpus to load, one tokenizer to argue with. It ended up being the only controlled experiment in the building: two dozen techniques, eighty years apart, all staring at identical text.
So here is the smallest version of that experiment. One line goes to each machine in turn, oldest first. Nobody gets a different example, and nothing below is written down in advance — every result on this page is computed in your browser as you scroll, from the same corpus and the same core.js files the individual pages use. If a technique embarrasses itself, you are watching it happen live.
The tokenizer lowercases everything and drops the apostrophe, so summer’s arrives as summers and the question mark never arrives at all. Eight tokens. That is the whole of what any of these machines will ever see of the most quoted line in English poetry.
Pattern-match the input, reflect the pronouns, hand it back as a question. Sixty years of hindsight say this is a parlour trick, but it was the first program most people ever spoke to.
The oldest idea in the building: record what word followed what. Ask it what comes after compare and it recites the five words it has ever seen there, with no opinion about which.
Widen the key from one word to two. Now the question is not “what follows compare” but “what follows shall i” — and the list gets shorter and much better aimed.
Same five candidates, but each now carries how often it actually occurred. In this corpus compare is rare enough that the five are perfectly tied — an honest answer that a bigger corpus would break apart.
Both improvements at once: two words of context, and a number on every candidate. Watch what it puts on the table when you give it shall i.
A tagger that judges each word on its spelling alone, in isolation, with no idea what surrounds it.
The same job done to the whole sentence at once, so a word’s neighbours get a vote. On a line this plain it lands in the same place as the baseline — the machinery only announces itself on ambiguous words.
A third opinion, from a model that scores whole label paths against overlapping features. Three taggers, one line, and they do not agree — which is the most useful thing on this page.
Where each of the eight sits in the corpus’s frequency ranking. The spread across one short line is the whole law in miniature.
How many yes/no questions it takes to pin down the next letter — first knowing nothing, then knowing the letter before it, then the two before it.
Misspell the line’s least common word and ask the corpus to repair it, counting the insertions, deletions and substitutions between strings.
Weight each word by how distinctive it is, then score all 154 sonnets. The first machine in this museum with any hope of finding the line’s home.
Which words keep company with ours far more often than chance would explain.
Trained on Shakespeare’s 154 sonnets against Elizabeth Barrett Browning’s 44, then shown the line cold and asked whose it is.
Describe a word by the company it keeps, then look for words described the same way. Nobody told it anything about seasons or weather.
Start from bare letters and repeatedly glue together the most frequent neighbouring pair. Here is our line dissolving into subwords, then reassembling as the merges accumulate.
Stop tallying word pairs and learn a vector for every word instead, so that unseen contexts can borrow from similar seen ones. Trained here, in this tab, for a few seconds.
Same destination as the co-occurrence vectors, opposite road: predict the context instead of counting it, then throw the predictions away and keep the weights.
A network reading one character at a time, carrying a running memory. It has never been told that words exist. Watch what a few hundred training steps buy.
Squeeze the line through a fixed-size memory and ask for it back in reverse. The capacity is four slots; the line is eight words long. You can see exactly where it stops being able to keep up.
Every word looks at every other word and decides how much each one matters. Each row sums to 1. Darker means more attention.
The word shall appears in our line and again eight lines later in the same sonnet. Give it a fresh vector in each place and the two no longer match.
Look the line up first, then speak only from what came back. The retrieval is TF-IDF from 1972; the arrangement is the modern part.
Between them, the machines on this page recovered a fair amount. The line’s grammar, three ways, with a disagreement. Its home address in the corpus, found twice by two different centuries of method. Its author, picked correctly out of two. The observation that day keeps company with night, arrived at by a program that has never been outside. The pieces summers breaks into when you make it earn its characters.
And every one of them did it by counting. The 1913 chain counted word pairs; the 2018 embeddings counted the same co-occurrences into a great many more dimensions. Nothing on this page was ever told that a sonnet is a poem, that a poem is addressed to someone, or that the someone is being told they will outlast the weather.
That gap is not a flaw in any particular technique here. It is the reason the last era of this journey is mostly concept pages: what closes it is not a cleverer algorithm but roughly a billion times more text, which is the one thing a laptop cannot supply.
One line is not an evaluation. Everything above is a single anecdote per technique, chosen because it is famous and because it happens to sit in the training corpus. A technique that looks foolish here might be excellent on a different sentence, and vice versa. For numbers you can actually compare, the held-out scoreboard on the homepage is the honest instrument.
The line is in the training data. Every model on this page has read Sonnet 18, most of them many times. TF-IDF finding sonnet 18 is retrieval working correctly, not comprehension; the models that generate text are quoting neighbours they have memorised. That is a fair demonstration of what each method does, and not at all a test of generalisation.
The small ones are handicapped by the corpus. 17,600 words is a rounding error as training data. The tied probabilities, the failed lookups, the tagger disagreements — some of that is the algorithm and some is simply not having enough text, and this page cannot always tell you which.