The NLP Journey — reference

Glossary & References

The recurring words of the journey, defined in plain language — and the papers behind every stop.

Every explainer in this museum defines its terms as it goes, so no page requires this one. But some words recur from stop to stop — token, embedding, parameter — and it helps to have them all in one place, said once, carefully. Two sentences each; no equations. Where a term has a good encyclopedia article behind it, a Wikipedia ↗ link follows the definition for anyone who wants the longer version.

Terms

attention
A mechanism that lets every token in a sequence look directly at every other token and decide, with a learned weight, how much each one matters for the job at hand. It replaced the idea of squeezing a whole sentence through a single memory, and it is the core primitive of the Transformer. See it run: Attention. Wikipedia
backpropagation
The bookkeeping that makes neural networks trainable: after the network makes a prediction, the error is passed backwards through every layer to work out how much each parameter contributed to it. Those per-parameter blame assignments are the gradients that gradient descent then follows. Wikipedia
bag of words
Treating a text as an unordered pile of words — keeping what occurs and how often, discarding the order entirely. It sounds like vandalism, but for tasks like search and classification (TF-IDF, Naive Bayes) the pile alone carries a surprising amount of signal. Wikipedia
bit
The unit of information: one bit is the answer to one perfectly balanced yes/no question. Saying a letter carries 4.1 bits of entropy means pinning it down takes, on average, about four such questions. Where it comes from: Entropy & the Guessing Game. Wikipedia
context window
How much preceding text a model is allowed to see when predicting what comes next — one word for a simple Markov chain, two for a trigram model, hundreds of thousands of tokens for a modern assistant. Most of this journey is the story of making the usable context longer without the statistics falling apart. Wikipedia
corpus
The body of text a technique learns from — here, Shakespeare's 154 sonnets and Elizabeth Barrett Browning's 44 (corpora/). Latin for "body"; plural corpora, a word linguists refuse to give up. Wikipedia
Conditional Random Field (CRF)
A discriminative sequence model that scores complete label paths given the observed input. Unlike an HMM it does not model how tags generated words, so overlapping clues can vote together. A partition function normalizes all possible paths into a conditional probability. Decode one: Linear-Chain CRF. Wikipedia
cosine similarity
A score for how similar two vectors are in direction, ignoring their length: 1 means pointing the same way, 0 means unrelated. It is the standard way to ask "how alike are these two words?" once words have become vectors. In action: Co-occurrence Word Vectors. Wikipedia
embedding
A learned vector that stands for a word (or token), placed so that words used in similar ways end up near each other. Early embeddings stored one vector per word; contextual embeddings compute a new one for every occurrence. Famous at Word2Vec; made contextual at ELMo. Wikipedia
entropy
The average surprise of a source, measured in bits — how unpredictable the next letter or word really is, given what you know. It is a ceiling on prediction: no model can beat the entropy of what it models. The full story: Entropy & the Guessing Game. Wikipedia
epoch (and iteration)
One full pass through the training data; an iteration is one update step, usually on a small slice of it. "Trained for 15 epochs" means the network saw every training example 15 times.
gradient descent (and gradient)
The gradient says, for each parameter, which direction of change would reduce the current error and by how much; gradient descent is simply taking a small step in that direction, millions of times. Nearly everything neural in this museum — and every frontier model — is trained this way. Wikipedia
held-out data
Text deliberately kept away from training, so a model can be graded on material it has never seen — the only honest test of whether it learned patterns rather than memorising examples. This repository holds out every 5th sonnet (Naive Bayes, the scoreboard); the gap between training and held-out scores is the classic symptom of memorisation. Wikipedia
precision, recall, and F1
Precision asks “when the model predicted this label, how often was it right?” Recall asks “of all real examples of this label, how many did it find?” F1 is their harmonic mean, falling sharply when either side is weak. A confusion matrix shows the underlying counts: actual labels in rows, predicted labels in columns. See the imbalance: Naive Bayes. Wikipedia
hidden state
A vector a recurrent network carries with it as it reads, updated after every token — its working memory of everything so far. The word "hidden" just means it is internal: not input, not output. Watch one in motion: RNN; then gate it in LSTM & GRU. Wikipedia
gate (LSTM / GRU)
A learned valve, expressed as a number from 0 to 1, that controls how much information enters, survives, or leaves recurrent memory. LSTM uses input, forget, and output gates; GRU uses update and reset gates. Open the valves: LSTM & GRU. Wikipedia
loss
The single number training tries to shrink: how wrong the model's predictions currently are, averaged over the training data. For language models the loss is essentially surprise — low loss means the actual next word rarely surprises the model — which is why it converts directly into perplexity. Wikipedia
n-gram
A run of n consecutive tokens: "shall i" is a bigram (n = 2), "shall i compare" a trigram (n = 3). Counting which n-grams occur, and what follows them, was the workhorse of statistical language modelling for half a century. See N-gram Markov Chain. Wikipedia
parameter
One learned number inside a model — a knob training is allowed to turn. This repository's neural LM has about 17,000; a frontier model has hundreds of billions, which is what "large" in large language model counts. Wikipedia
perplexity
A model's average uncertainty, expressed as an effective number of choices: perplexity 50 means that, on average, the model is as unsure as if it were picking among 50 equally likely words. It is entropy un-logged, and lower is better. The journey's scoreboard ranks every generative stop by it. Wikipedia
probability distribution
A full set of options with a probability for each, summing to exactly 1 — "the / 0.21, a / 0.09, my / 0.07, …" is a distribution over next words. Every language model in this museum, from weighted Markov chain to frontier model, is at heart a machine for producing one of these. Wikipedia
smoothing
The art of never letting a statistical model say "impossible": reserving a little probability for words and transitions that never appeared in training, because the training data is a sample, not the language. Without it, one unseen word makes a whole text's probability zero. Used throughout Naive Bayes and the scoreboard. Wikipedia
softmax
The standard recipe for turning a row of raw scores into a probability distribution: exponentiate each score (so bigger differences matter more), then divide by the total so everything sums to 1. It is the last step of nearly every neural language model — the moment scores become a bet on the next word. Wikipedia
token (and tokenization)
The unit a model actually reads — in this repository usually a lowercased word, in modern models a subword chunk like fa + irest. Tokenization is the (surprisingly consequential) act of cutting raw text into these units. Why the word isn't the obvious unit: Byte Pair Encoding. Wikipedia
Transformer
The 2017 neural architecture built almost entirely out of attention layers, processing every token in parallel instead of one at a time. Every frontier model — GPT, Claude, Gemini, Llama — is a Transformer at heart. The concept page: The Transformer. Wikipedia
vector
A plain list of numbers — [0.3, −1.2, 0.7, …] — treated as a point (or an arrow) in a space with one dimension per number. The great trick of modern NLP is representing words, sentences, and documents as vectors, so that "similar meaning" becomes the measurable "nearby in space". Wikipedia
vocabulary (and <unk>)
The fixed set of tokens a model knows; anything outside it is out-of-vocabulary and gets mapped to a single stand-in token, written <unk> for "unknown". Small vocabularies are a real handicap — one of the honest asterisks on the scoreboard — and subword tokenization (BPE) was invented largely to abolish them. Wikipedia

References

The papers (and one novel-in-verse) behind the stops, in order of appearance on the timeline. None are required reading — every explainer stands alone — but each is where its idea entered the world.