Curriculum Roadmap
Every topic by track, ordered by the prerequisite graph. Start where a track begins and follow its prerequisites.
Explore the full dependency graph, or browse by track below.
Foundations
Text as Data
Track completeText as a statistical object: the heavy-tailed laws of vocabulary and subword tokenization as a compression problem. The root of the dependency graph.
Zipf's and Heaps' Laws: The Statistics of Vocabulary
Why a vocabulary's rank–frequency curve and its growth curve are one power law seen twice — and why a straight line through a log–log plot is the wrong way to measure it
Every corpus obeys two regularities: the r-th most frequent word occurs roughly in proportion to r^(−s) (Zipf's law), and n tokens contain roughly n^β distinct word types (Heaps' law). We show they are one fact seen twice. The rank–frequency curve is the tail distribution of type counts turned sideways, which ties the rank exponent to the spectrum exponent. Zipf's law is the maximum-entropy distribution at fixed mean log-rank. Karlin's theorem proves that i.i.d. Zipf sampling with s > 1 makes the expected vocabulary grow as Γ(1 − 1/s)(Cn)^(1/s), so β = 1/s, with an error of at most about one word type. The Pitman–Yor process, the generator behind the site's finance corpus, plants both laws through a single discount d; we derive its exact expected type count, whose θ/d offset matters at corpus scale. Estimating the exponent is harder than drawing a line: log–log least-squares slopes swing with the fitting window, so we develop the discrete power-law maximum-likelihood estimator. A finance case study uses a corpus with a known answer: its full text reproduces the familiar real-text mismatch between s and β, generic estimators miss the planted discount, and the exact form of the Pitman–Yor growth law recovers it.
Subword Tokenization as Compression: BPE, WordPiece, and Unigram LM
Three ways to spell an open vocabulary with a closed one — greedy merges, likelihood merges, and a probabilistic lattice — compared as codes, and tested against a morphology we planted
A word vocabulary is closed, but the vocabulary of text keeps growing, so any fixed word list meets words it has never seen. Subword tokenizers spell every word from a fixed inventory of pieces. We make that precise: a vocabulary encodes every string exactly when it contains the alphabet, and a word has exponentially many segmentations, so choosing one is a search problem. Byte-pair encoding merges the most frequent adjacent pair; we prove each merge saves exactly its replacement count and show, with a nine-letter counterexample, that greedy merging is not optimal (the optimal problem is APX-complete, a cited result). WordPiece merges by the gain in unigram likelihood, which we derive exactly: its leading term is the pair's pointwise mutual information weighted by its count. The Unigram LM scores whole segmentations; we prove Viterbi decoding, the forward–backward lattice, and that EM never lowers the likelihood. All three are codes, and a two-part description length picks a vocabulary between letters and words. A finance case study tests the tokenizers on a corpus whose rare words were built from known prefixes, roots, and suffixes: compression and morphology agree only partly.
Language Modeling
Track completeAssigning probabilities to sequences: n-gram estimation and smoothing, perplexity as cross-entropy, and the move to neural language models.
n-Gram Language Models and Smoothing: From Laplace to Kneser–Ney
Every smoother is a guess about the words we have not seen — read three ways as empirical Bayes: a Dirichlet prior fit to the data, Robbins' formula on word counts, and Kneser–Ney as a hierarchical Pitman–Yor posterior
An n-gram language model predicts the next word from the few before it, and counting is its obvious estimator. Counting fails on the first sentence it has not seen: a third of our held-out trigrams never occur in training, and the maximum-likelihood model gives them probability zero. Every smoother is a guess about that unseen mass, and we read the classical guesses three ways as empirical Bayes. Additive smoothing is a Dirichlet posterior mean whose strength we fit by Type-II maximum likelihood. Good–Turing's adjusted count is Robbins' formula with word types in place of insurance policy holders, and we prove its missing-mass identity exactly. Katz backoff, absolute discounting (with a proof that Ney's discount estimate is an upper bound on the leave-one-out optimum), and Kneser–Ney follow, the last derived from a marginal constraint whose unique solution is the continuation count. Finally, interpolated Kneser–Ney is exactly the predictive of a hierarchical Pitman–Yor language model in which every word sits at one table, and a Gibbs-sampled version of that model predicts best of all. A finance case study asks what each estimate is an estimate of: Good–Turing forecasts the new-word rate of the same text continued to within two percent, and misses by a factor of four to twenty-one on anything else.
Neural Language Models: From Bengio's NPLM to Recurrent Networks
Replace counts with coordinates: a softmax over learned embeddings smooths by geometry, backpropagation is one reverse sweep of the chain rule, and a recurrent state lifts the trigram's horizon — as far as products of Jacobians let gradients reach
A count-based model estimates each context's next-word distribution separately; a neural language model gives every word a learned vector, makes each context a point, and lets a softmax read a distribution off the point. We prove what that buys and what it costs, on two networks written from scratch in NumPy: Bengio's neural probabilistic language model and an Elman recurrent network. Log-probabilities are Lipschitz in the context's coordinates, so unseen contexts are predicted like nearby seen ones — on held-out tokens whose word pair training never saw, the network beats Kneser–Ney by 0.79 nats. Training is maximum likelihood: the softmax is the categorical exponential family, every stationary point matches the model's average prediction to the training frequencies, and so a word never seen as a target, the unknown-word symbol, has no maximum-likelihood fit at all. Backpropagation is reverse accumulation on a computational graph, checked against complex-step derivatives. A softmax recurrent network is always a proper distribution over sentences; its gradients through time are products of Jacobians, which we bound, and on the trained network every per-step factor exceeds one while the product still decays. An LSTM's cell path carries a product of forget gates instead. A softmax over a d-dimensional state has rank at most d + 2. Every comparison is an exact arithmetic code in bits per character: the recurrent network with dynamic evaluation codes the held-out text in 79,614 bits against Kneser–Ney's 98,345, and on public EDGAR filings the spelling of unknown words alone costs more than lzma's whole file.
Perplexity, Cross-Entropy, and Compression
Held-out log-loss is a file size: an arithmetic coder turns any language model into a compressor within two bits of it — which makes perplexity a branching factor, bits per character its honest unit, and the unknown-word convention a flattering lower bound
Perplexity is usually read as a score. This topic reads it three more ways and proves each reading. As a forecast, held-out log-loss is a cross-entropy, and the logarithm is the score that rewards reporting one's true belief. As a branching factor, perplexity counts equally likely choices — exactly so by the asymptotic equipartition property, which we prove with the typical set's size. As a file size, an arithmetic coder turns any sequential model into a code within two bits of the model's negative log-probability: we prove McMillan's inequality, price coding with the wrong model at its KL divergence, and run an exact big-integer arithmetic coder on the whole held-out finance corpus, which becomes one codeword of 98,345 bits, 0.69 bits above the model's own figure. The file-size reading settles which unit makes models comparable. A string's total log-probability does not depend on how it is tokenized, so bits per character is comparable across tokenizers and per-token perplexity is not: a character 7-gram's perplexity of 1.665 looks more than ten times better than a word trigram's 19.23 and codes worse. Mapping words to an unknown-word symbol can only lower the reported loss. On public EDGAR filings, Kneser–Ney does worse than a uniform guess, most of its honest code is spelling, and lzma beats every model on the page.
Representations
Representations
Track completeMeaning as geometry: co-occurrence statistics and PPMI, word2vec as implicit matrix factorization, GloVe, and contextual embeddings from masked language modeling.
Need the ML foundations? formalml.com →
Distributional Semantics: PPMI and Truncated SVD
Know a word by the company it keeps: pointwise mutual information subtracts the background every count row shares, a truncated SVD keeps its best low-rank picture, and the context, not the method, decides whether that picture shows topic or polarity
A word can be described by the company it keeps: the row of a matrix that counts how often it appears near every other word. We prove what each step of the classical recipe does to that row. Raw counts all share the corpus's background, so near the background every word looks alike; pointwise mutual information subtracts the background in log coordinates and leaves what is specific to the word. Plug-in PMI has a ceiling set by frequency, and on a pair seen once it is a function of frequency alone, which is most of the matrix: under independence PPMI is positive on rare pairs with large values, and a shuffled corpus has more PPMI energy than the real one. A truncated SVD keeps the best rank-d approximation, but the eigenvalue weight decides what word vectors' dot products approximate; for a symmetric window the unsmoothed PMI matrix is symmetric and has large negative eigenvalues, which half-power vectors misread and adding context vectors removes. A latent-class model makes the ratio matrix, not the PMI matrix, low rank; the two hypotheses agree only to first order. On a corpus with planted sectors, PPMI beats counts at every window and truncation lifts the ranking AUC from 0.81 to 0.97, but only in a narrow band of rank. Finance words show the limits: antonyms are each other's nearest neighbors in a window, and polarity is recoverable only from whole documents.
GloVe and the Geometry of Word Embeddings
GloVe fits log co-occurrence counts by a tempered Poisson least squares that never sees the pairs that did not occur; its biases widen the gauge to affine maps, so an exact analogy is a fact about count ratios while the answer a nearest-neighbor search returns depends on the factorization
GloVe fits word vectors to the logarithms of co-occurrence counts by weighted least squares, with a bias per word and per context. We prove that its loss is the second-order approximation of a Poisson log-bilinear likelihood, with the Poisson weight, the count itself, tempered to GloVe's capped weight, and with every pair that never occurred dropped; negative sampling, near its optimum, is the same least squares with the biases frozen and a third weight. With every pair observed and flat weights, GloVe is exactly a truncated SVD of the double-centered PMI matrix. With only the observed pairs, it cannot see which pairs occur: if the observed log counts are additive, the biases fit everything and the vectors learn nothing. On the finance corpus, where most observed pairs are seen once, GloVe's vectors group nouns by their planted sector at chance, while negative sampling and the SVD of the same counts do not; ten times the text, or counting every unseen pair as seen half a time, brings the sectors back. GloVe's biases widen the freedom of the factorization to affine maps. An exact analogy parallelogram is a statement about count ratios and survives every such map, while the answer vector arithmetic returns does not: one change of basis, at identical loss, cuts word2vec's success on city-to-region analogies from 0.633 to 0.253. Relations from a filings knowledge base preview the knowledge-graph embeddings that make such offsets their objective.
word2vec as Implicit Matrix Factorization
Skip-gram's three objectives share one optimum up to offsets, shifted PMI; negative sampling fits it by a count-weighted logistic PCA; and since the loss sees only the product of word and context vectors, the optimizer's balance, not the objective, decides which similarities we get
Word2vec learns word vectors by stochastic gradient descent on a classification loss and never forms a matrix, yet it fits one. We prove that skip-gram's three training objectives, the full softmax, noise-contrastive estimation, and negative sampling, all have optima equal to pointwise mutual information up to row and column offsets, and that negative sampling's is exactly PMI minus log k on observed pairs, with no optimum at all on unobserved ones. At word2vec's rank the objective is exactly a weighted logistic PCA whose local weight on a pair is at most the smaller of its count and its expected negative count, so frequent pairs are fitted closely and rare ones loosely, unlike a truncated SVD of shifted PMI, which weights every entry alike. The loss sees only the product of word and context vectors, so it fixes word similarity only up to an invertible change of basis: on the finance corpus, five factorizations of one trained product, all with the same loss, score from 0.74 to 0.90 at grouping nouns by sector. Gradient flow conserves the difference of the two Gram matrices, which pins balanced factors to the square-root weighting and makes word plus context vectors the positive part of a symmetric product. Word2vec's minibatch SGD keeps that balance only approximately, and on the same matrix it groups the planted sectors less well than the SVD, because the sector evidence lives in rare pairs. Finance words show what no factorization of window counts can add: antonyms stay nearest neighbors, and no polarity direction beats chance.
Contextual Embeddings and the Masked-LM Objective
Hide a token and ask a bidirectional encoder to put it back: the objective's optimum is the posterior of the hidden token given its corrupted context, with a closed form under BERT's 80/10/10 corruption; its conditionals belong to no joint distribution; and the vectors it learns give each occurrence its own point, which reads what the neighbors determine and nothing more
Word2vec gives each word type one vector, so a word with two senses gets one point for both. We prove that the point sees a mixture: a polyseme's PMI is the log of its senses' exponentiated PMIs, and any probe on a static vector is capped at the majority sense. BERT gives each occurrence its own vector, trained by hiding tokens and asking a bidirectional encoder to put them back. We prove that the masked-LM risk is entropy plus divergence, so its optimum is the posterior of the hidden token given the corrupted sentence, and that under BERT's 80/10/10 corruption that posterior has a closed form: the context's prediction at a mask, shrunk toward the visible token elsewhere, with the selection rate dropping out. With one mask per sequence the objective is Besag's pseudo-likelihood. Brook's lemma rebuilds a joint distribution from its conditionals, and a four-cycle test that every joint passes shows the trained encoder's conditionals belong to no joint: BERT is not a language model in the generative sense. We train two encoders from scratch on Topic 11's block, one on a hidden Markov language whose exact posteriors check the closed form, one on the shared filings. On the filings, context separates a company from its product, and the news a direction word reports, where static vectors cannot; it does not recover a word's dictionary polarity, which the sentences barely carry.
Attention & Transformers
Track completeThe transformer as mathematics: attention as kernel smoothing, positional encodings as rotations, the residual stream, and empirical scaling laws.
Need the ML foundations? formalml.com →
Attention as Kernel Smoothing
Softmax attention is the Nadaraya–Watson estimator with an exponential kernel: the temperature is a bandwidth whose trade-off is exact on the circle, the 1/√d scale keeps the kernel from collapsing into a lookup, and a weighted lookup over every earlier position reaches in one step what a recurrent state forgets
A recurrent language model compresses its whole prefix into one state vector; attention instead looks up every earlier position, weighting each by the exponential of a query–key dot product. We prove that this lookup is the Nadaraya–Watson kernel smoother: the exponential kernel is a Gaussian in the distance between query and key, tilted by the keys' lengths, and its weights are the posterior responsibilities of a Gaussian mixture centered on the keys. The temperature is the bandwidth. On equispaced keys around a circle the smoother is diagonalized by the Fourier basis, with gains that are ratios of modified Bessel functions, so its bias–variance trade-off is a closed form whose optimal concentration grows like the two-fifths power of the number of keys; a one-layer attention model trained to denoise random functions finds that optimum to within 0.53%. The scale 1/√d keeps random dot products from saturating the softmax, whose Jacobian vanishes as it saturates. A lookup's gradient reaches an earlier position in one step, diluted by the number of competitors rather than decayed through a product of Jacobians, so one attention layer trained on the recurrent topic's long-lag task reads the bit at ten times the training lag. On text, a kernel cache over a document's own earlier words, mixed with Kneser–Ney, helps a little on synthetic filings and a lot on real ones, by copying words the training text never contained. Factoring the kernel with positive random features turns the lookup back into a fixed-size recurrent state, at a cost that grows exponentially with the kernel's sharpness.
Positional Encodings: Sinusoids, Rotations, and RoPE
A positional encoding is a kernel over offsets: sinusoids are a fixed stationary kernel that rules distance logarithmically, RoPE rotates queries and keys so every learned score sees only the offset, and ALiBi multiplies in an exponential kernel that extrapolates in length because it cannot look far
Attention averages over a set, so at the last word, one layer cannot tell the dog bit the man today from the man bit the dog today; a positional encoding supplies the order. We read every way of supplying it as a kernel over offsets. The sinusoidal code turns the hands of d/2 clocks, a shift of position is one rotation, and the inner product of two positions is a stationary kernel — the characteristic function of the code's frequencies — that converges to a cosine-integral ruler logarithmic in distance. Added to the content, though, a stationary code does not make attention relative: only a d-dimensional family of score matrices, the commutant of the rotations, sees offsets alone, and trained heads sit as far from it as random matrices. ALiBi's weights are the attention topic's Nadaraya–Watson smoother with a product of a content kernel and an exponential kernel in time, and one inequality both bounds its reach at every length and puts a horizon on what it can retrieve. RoPE rotates queries and keys so every score depends only on the offset, and up to the learned projections it is the only linear encoding that does. Its offset score is a trigonometric polynomial: single heads trained at 32 tokens learn to fall across the window and then rebound past it, in the planes that never completed a turn, so RoPE degrades with length where ALiBi holds. Interpolating positions keeps every angle inside the trained arc, and on whole synthetic filings, asked which company was named last, it reads at 0.986 past the training window, where ALiBi reads 0.580. Along the way we correct a published interpolation bound.
Neural Scaling Laws and Compute-Optimal Training
Loss falls as a power of parameters, data, and compute: in two solvable models the exponents are theorems — Zipf's tail for data, a spectrum's tail for size — and a Lagrange multiplier turns the fitted law into the compute-optimal split, whose exponent depends on what you count as a parameter
Test loss against parameters, data, or compute falls on near-straight lines on log–log axes, which means power laws. We ask where the exponents come from and how a compute budget should be split. In two models the exponents are theorems. A memorizer on Pitman–Yor text, the generator behind our finance corpus, must err when the next word is new, and that probability is an exact ratio of Gamma functions that falls as a power of the data. A linear model whose spectrum and target decay as powers of the coordinate index loses the tail it cannot represent as a power of its size, and the tail it has not yet learned as a power of its training time; for Gaussian data we compute one-pass SGD's expected loss exactly. We count a transformer's training FLOPs by instrumenting its arrays, so 6ND is proved with its missing attention term. Minimizing the fitted law on a compute budget gives the compute-optimal size in closed form, which the solvable model predicts and Chinchilla's three fitting approaches recover. We show that Kaplan's exponent and Chinchilla's are one law read in different units, train toy transformers on the shared filings, and ask what a finite domain corpus does to the optimum: a few repeated epochs are nearly free, and a cap binds only far below the data the law wants.
The Transformer Block and the Residual Stream
Attention alone averages tokens into one another; the residual connection keeps each token itself, and turns the block into a shared channel that heads read with linear maps and write into by addition, which makes the logits an exact sum of paths, keeps an identity path open for the gradient, and lets two heads compose into an induction head
The attention topic read attention as a kernel smoother, and smoothing averages: stack averaging operators and every token drifts toward the mean. We prove it. A row-stochastic matrix multiplies the diameter of the token cloud by at most its Dobrushin coefficient, which a softmax with bounded scores keeps below 1, so a stack of wide-kernel attention layers drives every token to one point, while a narrow kernel slows the collapse, and the cloud passes through clusters on its way there. The residual connection adds each token back, and then, for value maps of norm at most 1, each layer keeps at least one minus that coefficient of the diameter. LayerNorm is a projection onto a hyperplane followed by a radial projection onto a sphere, and its Jacobian forgets the radial direction. With the skip, the block becomes a shared stream: the Jacobian of a stack is a sum over all subsets of layers that always contains the identity, and the logits of an attention-only model are exactly a sum over chains of heads. We train such a model from scratch with rotary positions and watch two heads compose through the stream: a previous-token head, and an induction head whose keys read it. One layer cannot do this, because its keys cannot see the previous token. Then the finance case: on filings about companies never seen in training, a model trained on the filings completes none of their names, and the induction model, which never saw a filing, completes 0.768 of them.
Generation
Decoding & Generation
Track completeTurning a distribution into text: decoding as search, temperature and truncation sampling, and whether a language model knows what it does not know.
Need the ML foundations? formalml.com →
Decoding as Search: Greedy and Beam Search
The most probable string is a shortest path: uniform-cost search finds it, greedy and beam search prune the search with no guarantee, one pruned score certifies success, and the string they converge to is shorter than the text the model learned from
A language model gives every string a probability, and decoding picks one. The usual target is the mode, the single most probable string. Write each next-token probability as an edge of cost −log p in the tree of prefixes, and the mode becomes a shortest path. Uniform-cost search finds it exactly, with a work bound set by the mode's probability and the expected length. A Markov model's prefixes merge on their states, so Dijkstra's algorithm decodes it in polynomial time, while for recurrent networks the problem is undecidable in general. Greedy decoding and beam search are the same search with most of the frontier discarded. Neither carries any approximation guarantee, and a wider beam can return a worse string. One more number, the best score the beam ever pruned, certifies when beam search did find the mode. The mode itself is biased toward stopping. Once the end mark is the likeliest next token no continuation can win, length normalization breaks the monotonicity that exact search relies on, and without a length cap it can have no optimum at all. Greedy decoding on a finite-state model is eventually periodic, and a model that always ends with positive probability can still make beam search run forever. On a trigram and a small transformer trained on the shared filings, wider beams cut search error to zero, and the trigram's strings shrink to a mode a third the length of the text. Normalization lengthens them by drawing them onto the corpus's template sentences.
Sampling: Temperature, Top-k, and Nucleus
Ancestral sampling draws exactly from the model; temperature tilts it toward a Gibbs distribution and truncation cuts its tail, each at a cost we can compute exactly, and neither can add knowledge the model lacks
Searching for the most probable string returns text that is short, bland, and nothing like what the model learned from. Sampling asks for a string drawn from the model instead, and ancestral sampling, one token at a time, is exact: every string comes up exactly as often as the model says, and the sampler halts precisely when the model is consistent. The Gumbel-max trick makes the link to greedy decoding literal, since a draw is the argmax of noisy log-probabilities. In practice the draw is reshaped. Temperature divides the logits by τ, which picks the Gibbs distribution that trades expected score against entropy, raises the entropy monotonically, and divides the slope of the log–log rank–frequency line by τ. Applied at every step, though, temperature is not the tempered model: it reweights strings by the accumulated Rényi entropy of their prefixes, its cold limit is greedy decoding rather than the mode, and above τ = 1 the tempered sequence distribution need not exist. Truncation rules (top-k, nucleus, min-p, η, and locally typical sampling) keep part of each distribution, and the cost is exact: a cut that keeps mass m costs total variation 1 − m and KL −log m per step, and over a sentence the KL is the expected sum of −log m. A Kneser–Ney model knows exactly which words it saw after each context, yet no setting of any rule recovers that set on every prompt. On a trigram and a small transformer trained on the shared filings, every rule lands on one frontier between predictability and quality, and none of the settings run reaches the held-out text. On an order-5 Kneser–Ney fact model, lowering the temperature removes fabrication on contexts seen twice or more, makes it certain on a third of the contexts seen once, and raises it on contexts never seen.
Calibration of Language Models
When a model says 0.9, is it right nine times in ten? For a next-token distribution over thousands of words the question splits into a strong form no corpus can test and a weak form any corpus can; every log loss separates into what the text cannot tell us, what the model's context discards, and what a recalibration could recover
A forecast is calibrated when the events it calls 70% likely happen 70% of the time. For a language model's next-token distribution over thousands of words the notion splits. Canonical calibration asks the whole distribution to be right among the contexts that receive it; when every context has its own distribution, that is the same as being the true conditional, so no corpus can check it. Top-label calibration asks only that the top token's probability match how often it occurs, and any held-out text can check that. Every log loss splits exactly into three parts: what the text cannot tell any forecaster with the same context, what the forecast discards of that context, and a reliability term that recalibration removes, and no recalibration can recover more. Measuring the reliability term is itself biased: coarse bins hide miscalibration and finite samples invent it, and a calibrated twin of the model, the same confidences with outcomes redrawn from them, measures the invented part. Temperature scaling is a convex one-parameter fit that matches one moment, keeps every argmax, and can reorder confidences across rows. When a model family can scale its own logits, maximum likelihood leaves the training-data temperature at exactly 1, and an L2 penalty leaves the model underconfident on its own training data. A calibrated model's nucleus covers the next token exactly as often as its kept mass says, and sampling at any other temperature miscalibrates it on purpose. On text from their own generator, a Kneser–Ney trigram and a small transformer are nearly calibrated. On real filings both read as badly overconfident and the trigram's nucleus covers a third of the text, but for the trigram most of that is one token: the unknown-word token is last in every row, so no temperature can reach it, and a one-parameter map that moves mass onto it recovers six times what a temperature does. A fact model is underconfident about what it has seen and overconfident about what it has not, so no single temperature fixes it.
Tasks
Classification & Sentiment
Track completeLabeling documents: generative versus discriminative linear classifiers, lexicon-based financial sentiment, aspect-level sentiment, and low-rank fine-tuning.
Aspect-Based Sentiment Analysis
"Revenue rose, but costs climbed" holds two opinions, and no reader of the sentence alone gets both. One attention layer whose query is the aspect is a signed vote over the words: it can stretch a word's vote but never flip it, which decides what it can read and what it cannot. Its attention weights are not an explanation, and read aspect by aspect the corpus gives back the document tone that every document-level reader missed
A sentence can praise one thing and fault another. Aspect-based sentiment reads the polarity of each aspect a sentence evaluates, relative to that aspect's valence: a rise is good news for revenue and bad news for costs. On the synthetic financial filings of the previous two topics, every two-aspect sentence pairs opposite polarities, so any reader that sees only the sentence gets at most one aspect of each pair right, which caps it at 0.911. The standard fix lets the aspect query an attention layer. We show that such a layer is a signed vote: the sign of every logit difference is the sign of a sum of word votes, each rescaled by a positive attention factor. Three consequences follow. When the aspect enters only through the query, a word's vote has the same sign for every aspect, so the layer can read valence only if rise and fall words vote the same way, and the trained model does exactly that for one family of direction words and fails the rest. Replacing one word by another moves the vote in a direction no context can reverse, so no such layer reads negation at all, on any data; pairs of words as items repair it. And attention weights are not identified by decisions: shifting scores and rescaling values keeps every decision while moving 0.998 of the attention onto the words of aspect terms, which the trained model barely looks at (0.798 onto the asked-about aspect's own words). Read aspect by aspect, the tone-bearing sentences give back document tone at 0.883, against a Bayes ceiling of 0.890 that the document-level readers of the previous topics, at 0.637 to 0.739, never approached.
Lexicon Sentiment and the Loughran–McDonald Dictionary
A sentiment dictionary is a linear classifier whose weights were fixed before any label was seen. Net tone projects a filing onto one direction of the category simplex and discards the other; length, contamination, and the sentences no word list can read set its ceiling; and as a prior for logistic regression the dictionary is worth about a thousand labeled filings
A sentiment dictionary scores a document by counting its positive and negative words. Seen through the previous topic, that is a linear classifier whose weights are fixed at plus one, minus one, and zero. This page asks what the fixing costs. Geometrically, a dictionary places a document at its rates of positive and negative hits, and the usual net-tone score keeps only their difference. When each tone draws word categories independently, the Bayes rule cuts the rate plane with lines, and those lines are parallel to the net-tone direction exactly when the tones' log-rate ratios are antisymmetric; on synthetic financial filings they are not, and the two counts beat their difference 0.724 to 0.637, because neutral filings have few polar words of either kind. The sign of the raw score is wrong with probability at most (1 − (√a − √b)²) to the power of the length, an exponent that is exact, and at the hit rates of a finance dictionary a 210-token filing of independent tokens is still misread a quarter of the time. Counting finance words such as tax and liability as negative, as a general-purpose list does, leaves the gap between tones unchanged and inflates every variance, which is Loughran and McDonald's case for a finance-specific dictionary. No word list reads a bare direction ("costs rose") or a cost that "exceeded expectations". Used as the center of a ridge penalty instead of as a classifier, the dictionary becomes a prior: the fit is ridge on the part of the weights orthogonal to the dictionary's directions, it converges to the dictionary at rate 1/λ, and it is never more than 0.01 below the better of the dictionary and plain logistic regression, which needs 300 to 400 labeled filings to catch net tone and about 1,000 to 1,500 to catch the dictionary's two counts. Propagating polarity from four seed words with a harmonic function recovers most polar words' signs, but cannot tell polar words from topical ones.
Linear Text Classifiers: Generative vs. Discriminative
Naive Bayes and logistic regression draw the same kind of boundary through word-count space and disagree only about where to put it. They share a hypothesis class and match different moments; Zipf's long tail breaks the generative shortcut; on separable text the logistic MLE runs off to infinity while its direction settles on the maximum-margin line; and no unigram model of financial tone can see that "costs rose" is bad news
A bag-of-words classifier turns a document into a vector of word counts and draws a hyperplane through that space. Naive Bayes and logistic regression draw the same family of hyperplanes: naive Bayes's posterior is exactly a softmax of a linear function of the counts, and logistic regression's family is naive Bayes's with one extra feature, the document's length. They differ in how they place the boundary. Logistic regression's optimum reproduces every class-by-word co-count of the training data and every class total; naive Bayes reproduces each class's word totals instead, and misses its own class totals. How fast each learns is Ng and Jordan's question. With binary features whose rates are bounded away from zero, naive Bayes needs only logarithmically many documents in the vocabulary size to get every parameter right, though with explicit constants the bound is vacuous at realistic sizes. A multinomial vocabulary cannot meet that hypothesis: some word has probability at most 1/V, so getting every rate right to a relative tolerance costs order V tokens, and V to a power under Zipf's law. In worlds where naive Bayes is the true model it leads logistic regression through 4,096 documents; the textbook crossover appears when words come in bursts, which keep logistic regression well specified. Bursts that fall unevenly on the informative words lower naive Bayes's asymptote, and bursts on noise words produce a crossover with no gap between the asymptotes at all. With more words than documents the training set is separable, the logistic MLE does not exist, and the direction of the ridge-penalized solution converges to the maximum-margin separator as the penalty vanishes. Naive Bayes features act as a per-word prior. In synthetic financial filings, tone is in part an exclusive-or of aspect and direction, which no unigram model represents and bigrams do; logistic regression beats naive Bayes from about 60 documents up, naive Bayes is badly overconfident, and at 1,500 documents nothing comes within 0.11 of the full-text Bayes accuracy of 0.890, though bigrams reach 0.847 at 50,000.
Fine-Tuning and Low-Rank Adaptation
A pretrained encoder can be read by a probe, rewritten by full fine-tuning, or nudged inside a low-rank subspace. The factored update's first step is a random sketch of the full one, its two factors conserve a hyperbola under gradient flow, and what fine-tuning never touches is what it forgets least. On a toy encoder, the full update turns out to be nearly low-rank, and LoRA forgets more than full fine-tuning at ten times its rate but less at the same rate
A pretrained encoder can be adapted to a task in three ways: read it with a classifier on frozen features, rewrite every weight, or move each weight matrix by a product of two thin factors, the low-rank adaptation called LoRA. We fine-tune the masked-LM encoder of Topic 12 on Topic 19's aspect sentiment, where the frozen probe reads 0.761 of the test aspects, an encoder that was never pretrained reads 0.951 after fine-tuning, and the pretrained one reads 0.970 to 0.989, with the largest gap, construction by construction, on negated adjectives. The mathematics of the factored update is exact. Its gradient is the full gradient multiplied by the other factor, so from the standard zero start the first step is the full step pushed through a random rank-r map, and its size stays put as the rank grows only when the scale falls like one over the square root of the rank, under SGD in expectation and under Adam for every draw. Gradient flow conserves the difference of the factors' Gram matrices, a hyperbola in one dimension, and Adam does not. Full fine-tuning's update is nearly low-rank: truncating every matrix to rank 4 keeps 0.977 of its 0.989 on one seed, and 0.935 to 0.977 over three, though the update is spread over all twelve block matrices, which LoRA's class cannot hold. Gradient descent moves a linear layer only along its training inputs, so the 419 tokens never seen in fine-tuning keep their embeddings exactly, and starting from the probe's head distorts the encoder less: LP-FT keeps the masked-LM loss near 1.9 where full fine-tuning reaches 2.7. LoRA, constrained to two matrix types and run at ten times the rate, makes updates 17 times larger and forgets most of all; at a shared rate it learns less and forgets less than full fine-tuning, as Biderman and colleagues found. A random subspace step is unbiased, and the pretrained encoder learns more in every subspace tried; linearized fine-tuning is kernel gradient descent, which gradient descent from the probe's head follows and Adam leaves. Finally, the fine-tuned encoder reads negation but learns a shortcut that counterfactual rows remove, and read aspect by aspect, the filings give back their tone at 0.862 to 0.880 over three seeds against a ceiling of 0.890, where document-level fine-tuning reaches 0.559.
Information Extraction & Knowledge Graphs
Track completeFrom text to structure: sequence labeling with HMMs and CRFs, named entities, relation extraction under distant supervision, and knowledge-graph embeddings — closing with the earnings-call capstone.
Knowledge-Graph Embeddings: TransE and RotatE
Relations as translations or rotations of entity space: which patterns each geometry satisfies exactly, what neither abelian group can represent, and what training on an incomplete knowledge base actually learns.
A knowledge-graph embedding places every entity at a point and makes every relation a map, so that a true triple is one whose head the map carries to its tail. TransE makes the map a translation, RotatE a rotation of complex coordinates. We prove what each geometry can satisfy exactly: translations invert and compose but represent a symmetric relation only by collapsing its pairs, while rotations represent symmetry with phases of zero or pi, inversion with negated phases, and composition with added ones. Both are actions of abelian groups by isometries, and that alone has consequences: facts that hold within epsilon pull the heads of a many-to-one relation within two epsilon of each other and collapse every odd cycle of a symmetric one, and on the Cayley graph of the symmetric group S3 no abelian model can keep a false triple more than five epsilon from its query, while the symmetries of a triangle fit it exactly. DistMult, by contrast, scores every relation symmetrically, and RotatE turns out to be ComplEx restricted to unit-modulus relations. On the finance knowledge base the trained rotations solve the phase equations to three decimals, until facts are hidden: a hidden fact is a negative during training, and the model that can represent antisymmetry learns the false antisymmetry the incomplete knowledge base teaches. Finally we embed the facts the relation-extraction topic extracted about six held-out companies and ask what the geometry adds.
Named Entity Recognition
A tagger that learns names by their spelling cannot find a name it never saw. What generalizes is context, provided training removes the name the way test-time novelty does.
A named entity recognizer predicts a set of typed spans, and tag schemes such as BIO and BIOES are encodings of that set. We prove that each encoding is a bijection onto its valid tag sequences, that validity is a first-order constraint, and that decoding under the constraint is exactly conditioning a chain model on validity. BIOES is at least as expressive as BIO and strictly more so only for features that see one token at a time. A CRF on a contextual encoder backpropagates gold indicators minus marginals into the encoder, and on frozen states its objective stays strictly concave. We prove that micro F1 is a prediction-weighted mixture of per-type F1s, compute every span's marginal with one forward-backward pass, and decode the span set that maximizes F1 of expected counts by Dinkelbach's method over a weighted-interval program, which thresholds the marginals at half the optimal F1. Then we hold names out by design: taggers train on filings about 18 companies and are tested on filings about 6 others. A word-feature CRF finds no unseen organization at all. Dropout that rewrites names as an unknown-word symbol teaches a shortcut, which we prove: a feature seen only on entities argues against the outside tag. Dropout that deletes the name's features avoids it, and a tagger whose encoder reads unseen words as [UNK] at test, as its dropout did in training, reaches span F1 0.979 on the held-out companies.
Relation Extraction and Distant Supervision
Labeling sentences by what a knowledge base already knows teaches a classifier the knowledge base's opinion of each entity pair. Bags, aggregation, and positive-unlabeled corrections recover facts, not sentences.
Relation extraction reads which relation, if any, a sentence states between two recognized entities, and aggregates those readings into facts for a knowledge base. Distant supervision labels every sentence that mentions two entities with whatever the knowledge base already holds between them. We prove what a classifier trained that way learns: the expected distant label given its features, which on a sentence that states nothing is the share of such sentences whose pair the knowledge base relates. That bias does not shrink with data, and on the finance corpus the fitted classifier reproduces each co-mention template's rate within 0.05. Multi-instance learning labels bags of mentions instead. We show its noisy-OR gradient is an E-step whose weight on every other mention vanishes once one mention is a confident witness, and we characterize which mentions the bag labels determine at all; that explains where its sentence recall goes, while it wins at the fact level. Noisy-OR aggregation helps a model trained under it and overcounts for one trained sentence by sentence. An incomplete knowledge base turns distant supervision into positive-unlabeled learning: we prove the optimum shrinks by the keep rate c, so argmax keeps a template only while c exceeds one half, and held-out evaluation reports c times the true precision. Finally, on six companies held out from training, the named entity recognition topic's tagger feeds the extractor, which loses exactly the triples whose arguments the tagger missed.
Sequence Labeling: From HMMs to Linear-Chain CRFs
One recursion in two semirings tags a sequence. From the hidden Markov model to the conditional random field, the family of conditionals does not change; what changes is what the estimator optimizes and what the features can see.
A sequence labeler assigns one tag to every token, and the possible taggings of a sentence are the paths through a trellis. We prove that one forward recursion, run in the sum-product semiring, gives the partition function and every marginal, and run in the max-product semiring, gives the best path; formalStatistics' forward-backward and Viterbi algorithms are its probability case. The hidden Markov model tagger is counting, a tag n-gram times naive Bayes emissions. The linear-chain conditional random field models the tags given the words; its conditional likelihood is concave with gradient observed minus expected features. With indicator features the two models define exactly the same conditional distributions: the HMM-to-CRF direction is a reparametrization, and the converse is an h-transform we prove in full. So on those features they differ only in the estimator, and in a misspecified world the CRF wins on conditional log-loss even where it loses on accuracy. Locally normalized taggers (MEMMs) are different: their marginals cannot depend on words beyond their lookahead, which caps their posterior-decoded accuracy by a lagged Bayes rule, while Viterbi decoding escapes the cap. We prove that Viterbi minimizes sequence error and posterior decoding minimizes token error, prove the structured perceptron's mistake bound, and measure everything against an exact Bayes tagger in simulated worlds and on BIO-tagged entity spans in synthetic filings.
Capstone: An Earnings-Call NLP Pipeline
Four published stages read earnings calls into a knowledge graph. Which stage costs what is a question about a lattice of oracle swaps, what one stage should hand the next is a question about garbling, and when the knowledge base can repair its own output is a question about total unimodularity.
Four published stages read a synthetic earnings call: Topic 22's entity tagger with Topic 23's linker, Topic 23's relation reader, a tone reader (Topic 18's dictionary, Topic 17's logistic regression, or Topic 19's aspect reader), and an aspect stage. Their output is a call graph: the facts the calls state, and for every call its subject, tone, and aspect polarities. Replacing one stage's output by the gold one is that stage's oracle, and the sixteen ways of doing so form a Boolean lattice. A ceiling analysis climbs it along one path. We prove that each path credits a stage with the Harsanyi dividends of the coalitions it completes, so paths disagree exactly when stages interact, and that the Shapley value, the average over paths, is the only attribution that is efficient, symmetric, null on null players, and additive. On 126 calls tone takes the largest share and aspects exactly none, and the one sizable interaction is that entities and relations are complements. A stage that hands on its posterior instead of its argmax can only help a Bayes-optimal successor, because the argmax is a garbling; a fixed successor can lose, and on the calls span marginals raise fact AP from 0.744 to 0.851 but lower F1 at the hard pipeline's threshold, until the threshold is refit. With inverse and symmetric facts merged, a knowledge base's functional constraints form a totally unimodular system, so the LP relaxation computes the constrained MAP exactly; composition rules break total unimodularity, and on these calls they impose the closed world.
Text Mining
Track completeUnsupervised structure in corpora: latent Dirichlet allocation as a Bayesian model, and collocations and keywords by likelihood ratios and random walks.
Need the ML foundations? formalml.com →
Collocations and Keyword Extraction
A word pair is a 2×2 table, a point in a tetrahedron. PMI, X², and Dunning's G² measure its distance from independence in different ways, only an exact test survives rare pairs, and testing only the pairs you saw makes every rare sighting significant. Then keywords: keyness compares corpora, and TextRank, which compares nothing, ranks by frequency.
A word pair splits a corpus's bigram positions into a 2×2 table, which divided by its size is a point of a tetrahedron, and the independent tables form a saddle surface inside it. The product of a table's margins is its KL projection onto the saddle, so Dunning's log-likelihood ratio G² is 2N times the empirical mutual information, while Pearson's X² is N times the squared correlation. Two words seen once each, and only together, get the largest PMI and the largest X² of any table, though the exact probability of that sighting is 1/N; among pairs that always occur together, PMI falls as the evidence rises. Conditioning on the margins makes the joint count hypergeometric, which gives Fisher's exact test; X² and G² converge to χ² with one degree of freedom, but at small expected counts their exact sizes are far from nominal, each in its own regime. Testing only the pairs one has seen is a selection: when a pair's chance of being seen at all is below the level, every sighting rejects. Keyness is the same test with the corpus as the second variable. TextRank is PageRank on a co-occurrence graph: we prove that a positive stochastic matrix has one stationary distribution, that power iteration contracts by the damping factor, and that as the damping factor tends to one TextRank ranks words by their edge weight, which is twice their frequency up to boundary terms. On the finance corpus, whose generator records where every word came from, G² and Fisher's test rank fixed phrases first while PMI's top is noise seen once; Benjamini–Hochberg over the observed pairs rejects most of the truly null ones and over all candidate pairs rejects none; keyness finds the planted sector vocabulary; and TextRank tracks frequency.
Topic Models: Latent Dirichlet Allocation
Documents as points in a polytope of topics: when the topics can be recovered, the two classical ways to infer them, and what a topic model finds in filings whose generator plants seven vocabularies but only six identifiable topics.
Latent Dirichlet allocation models a document as a mixture of topics, each a distribution over words. Geometrically, every document's expected word distribution is a point of the topic polytope, the convex hull of the topics inside the word simplex, and fitting a document's topic proportions is a KL projection onto it. We prove when the polytope can be recovered: never up to relabeling, not in general, and not at all for a topic that every document carries at the same weight; but under separability every row of the normalized word co-occurrence matrix is a convex combination of the anchor words' rows, and when the topic–topic matrix is nonsingular the anchors are exactly the vertices. We derive the two classical inference engines, the collapsed Gibbs sampler from Dirichlet–multinomial conjugacy and mean-field coordinate ascent from the digamma identity, and show that the harmonic-mean estimator of held-out likelihood overestimates in expectation and has infinite variance as soon as a document holds enough anchor tokens. On the finance corpus, whose generator draws every document's content nouns half from its sector's vocabulary and half from a shared one, the shared vocabulary is exactly the unidentifiable topic: the anchor search finds one anchor per sector and none for it. Collapsed Gibbs finds the sectors, while mean-field VI and EM for the correctly specified single-membership model both stall at optima their own objectives rank far below the sectors.
Evaluation
Evaluation
Track completeMeasuring language systems honestly: n-gram overlap metrics and their limits, and significance testing for paired NLP comparisons.
Overlap Metrics: BLEU and ROUGE
A candidate and a reference make a grid of matches. Clipping is a matching on it and ROUGE-L is the longest chain through it, so a one-word edit moves either by a few n-grams at most: overlap cannot tell a paraphrase from a reversed fact. Read as a decision rule, BLEU's brevity penalty is why minimum Bayes risk beats the mode.
Lay a candidate along one axis and a reference along the other, and mark every cell where the two hold the same word: every overlap metric is a function of that grid. BLEU's clipped count is the size of a maximum matching on it, so repeating a matched word earns nothing extra, and ROUGE-L's longest common subsequence is the longest chain of marks rising in both directions, a longest path the dynamic program finds. With one reference per segment BLEU lies under an envelope set by the length ratio alone, which peaks only at the reference length; corpus precisions are mediants, so a system can win every sentence and lose the corpus; a sentence with no matched 4-gram scores zero whatever else it gets right, and add-one smoothing is the n-gram topic's shrinkage with a prior at perfect precision; adding a reference can lower BLEU. The central fact is a Lipschitz bound: k substituted words move a clipped count by at most nk and the subsequence by at most k. On synthetic filings whose generator records every fact, a summary that states each fact once and gets every one backward outscores the document's own fact-bearing sentences on BLEU, ROUGE, chrF, and an embedding match, all six; word vectors make it worse, since 'fell' is the nearest neighbor of 'rose'. Read as a decision rule, exact match picks the mode and BLEU does not: on the decoding topic's prompts the trigram's mode is short, the brevity penalty punishes it, and minimum Bayes risk over 64 samples scores best. Averaged over documents, system scores converge to their bias, not their quality: Kendall's τ against fact accuracy climbs to a plateau near one half.
Significance Testing for NLP
Resampled corpora are points on a simplex, and swapped outputs are vertices of a cube. A corpus metric is a surface over both. Randomization is exact on any surface, the bootstrap only where the surface is smooth, and neither sees the variance that the resampled unit hides.
A paired test set gives two pictures of another test set like this one. The paired bootstrap draws a random point of the simplex of segment weights; approximate randomization draws a random vertex of the cube of swaps between the two systems' outputs. A corpus metric such as corpus BLEU is a surface over both, and it is not an average, so the tests built for means do not apply as they stand. We prove that randomization's Monte Carlo p-value is valid for every statistic and every number of draws, and that on right-or-wrong outcomes it is exactly McNemar's test. The bootstrap gives two p-values in common use, Koehn's win count and the recentered rule of Berg-Kirkpatrick and colleagues, which invert the percentile and the basic interval. Away from the brevity penalty's kink, corpus BLEU is asymptotically a mean of per-segment influences and the bootstrap is consistent; at the kink the limit is not Gaussian and the bootstrap fails, the basic interval worst. An exact null built from i.i.d. samples of a trigram checks every claim: randomization holds its size from 10 prompts up, while the recentered bootstrap does not at 10. Clustered segments multiply a variance by the design effect, which pairing mostly cancels. A test set cannot test a training procedure: when two procedures are equal on average, test-set McNemar still rejects in 62.5% of training draws on 20,000 documents. On the finance testbed, 1,500 filings cannot resolve a half-point accuracy gap, and corpus BLEU and fact F1 disagree on 17 of 45 system pairs.
Building your foundations?
Many topics here build directly on machine-learning theory — representation learning, information theory, conformal prediction — covered on formalml.com . The deeper foundations live on formalcalculus.com (linear algebra, optimization, analysis) and formalstatistics.com (estimation, testing, calibration) — all with the same geometric-first approach.