formalNLP
The mathematics of language
Theorem-bearing explainers on the statistics of text, language modeling, word and contextual representations, attention and transformers, decoding, classification and sentiment, information extraction and knowledge graphs, text mining, and evaluation — each with rigorous mathematics, interactive visualizations, and working code.
Latest Topics
Capstone: An Earnings-Call NLP Pipeline
Four published stages read earnings calls into a knowledge graph. Which stage costs what is a question about a lattice of oracle swaps, what one stage should hand the next is a question about garbling, and when the knowledge base can repair its own output is a question about total unimodularity.
Four published stages read a synthetic earnings call: Topic 22's entity tagger with Topic 23's linker, Topic 23's relation reader, a tone reader (Topic 18's dictionary, Topic 17's logistic regression, or Topic 19's aspect reader), and an aspect stage. Their output is a call graph: the facts the calls state, and for every call its subject, tone, and aspect polarities. Replacing one stage's output by the gold one is that stage's oracle, and the sixteen ways of doing so form a Boolean lattice. A ceiling analysis climbs it along one path. We prove that each path credits a stage with the Harsanyi dividends of the coalitions it completes, so paths disagree exactly when stages interact, and that the Shapley value, the average over paths, is the only attribution that is efficient, symmetric, null on null players, and additive. On 126 calls tone takes the largest share and aspects exactly none, and the one sizable interaction is that entities and relations are complements. A stage that hands on its posterior instead of its argmax can only help a Bayes-optimal successor, because the argmax is a garbling; a fixed successor can lose, and on the calls span marginals raise fact AP from 0.744 to 0.851 but lower F1 at the hard pipeline's threshold, until the threshold is refit. With inverse and symmetric facts merged, a knowledge base's functional constraints form a totally unimodular system, so the LP relaxation computes the constrained MAP exactly; composition rules break total unimodularity, and on these calls they impose the closed world.
Overlap Metrics: BLEU and ROUGE
A candidate and a reference make a grid of matches. Clipping is a matching on it and ROUGE-L is the longest chain through it, so a one-word edit moves either by a few n-grams at most: overlap cannot tell a paraphrase from a reversed fact. Read as a decision rule, BLEU's brevity penalty is why minimum Bayes risk beats the mode.
Lay a candidate along one axis and a reference along the other, and mark every cell where the two hold the same word: every overlap metric is a function of that grid. BLEU's clipped count is the size of a maximum matching on it, so repeating a matched word earns nothing extra, and ROUGE-L's longest common subsequence is the longest chain of marks rising in both directions, a longest path the dynamic program finds. With one reference per segment BLEU lies under an envelope set by the length ratio alone, which peaks only at the reference length; corpus precisions are mediants, so a system can win every sentence and lose the corpus; a sentence with no matched 4-gram scores zero whatever else it gets right, and add-one smoothing is the n-gram topic's shrinkage with a prior at perfect precision; adding a reference can lower BLEU. The central fact is a Lipschitz bound: k substituted words move a clipped count by at most nk and the subsequence by at most k. On synthetic filings whose generator records every fact, a summary that states each fact once and gets every one backward outscores the document's own fact-bearing sentences on BLEU, ROUGE, chrF, and an embedding match, all six; word vectors make it worse, since 'fell' is the nearest neighbor of 'rose'. Read as a decision rule, exact match picks the mode and BLEU does not: on the decoding topic's prompts the trigram's mode is short, the brevity penalty punishes it, and minimum Bayes risk over 64 samples scores best. Averaged over documents, system scores converge to their bias, not their quality: Kendall's τ against fact accuracy climbs to a plateau near one half.
Significance Testing for NLP
Resampled corpora are points on a simplex, and swapped outputs are vertices of a cube. A corpus metric is a surface over both. Randomization is exact on any surface, the bootstrap only where the surface is smooth, and neither sees the variance that the resampled unit hides.
A paired test set gives two pictures of another test set like this one. The paired bootstrap draws a random point of the simplex of segment weights; approximate randomization draws a random vertex of the cube of swaps between the two systems' outputs. A corpus metric such as corpus BLEU is a surface over both, and it is not an average, so the tests built for means do not apply as they stand. We prove that randomization's Monte Carlo p-value is valid for every statistic and every number of draws, and that on right-or-wrong outcomes it is exactly McNemar's test. The bootstrap gives two p-values in common use, Koehn's win count and the recentered rule of Berg-Kirkpatrick and colleagues, which invert the percentile and the basic interval. Away from the brevity penalty's kink, corpus BLEU is asymptotically a mean of per-segment influences and the bootstrap is consistent; at the kink the limit is not Gaussian and the bootstrap fails, the basic interval worst. An exact null built from i.i.d. samples of a trigram checks every claim: randomization holds its size from 10 prompts up, while the recentered bootstrap does not at 10. Clustered segments multiply a variance by the design effect, which pairing mostly cancels. A test set cannot test a training procedure: when two procedures are equal on average, test-set McNemar still rejects in 62.5% of training draws on 20,000 documents. On the finance testbed, 1,500 filings cannot resolve a half-point accuracy gap, and corpus BLEU and fact F1 disagree on 17 of 45 system pairs.