5.1 What is in it🔗ℹ

handpress/lexicon ships with 318,722 spellings attested in 5,287 books printed between 1580 and 1640, drawn from EEBO-TCP (Partnership 2015). They are grouped into 45,719 sets of variants of one another, and 18,562 are mapped to the form still current.

It answers three questions, and they are genuinely different:

This is the more useful test, and a large corpus is what makes the difference matter. theere occurs seventeen times in 5,287 books against 145,517 for here, and wheere six. Those are not spellings anybody chose; they are the sweepings of a very large floor — foreign words, slips, mis-keyings. Set beside them manne at 1,147 occurrences and somme at 467 are real usage. The threshold is plausible-share, one in two hundred, and it is a judgement stated in the open rather than buried.This distinction cost the author some embarrassment. Working from the 24,000 words of the two Much Ado texts, I asserted in code, comments and three commit messages that nine forms the old rules could produce were “not early modern spellings at all”. Against 5,287 books, six of the nine occur. The claim was true of my sample and false of the language, which is the characteristic failure of a corpus too small for the question asked of it.

5.2 How the variants are grouped🔗ℹ

Without a modern wordlist to anchor it, her becomes a spelling of here. The reduction that correctly joins heere to here joins here to her by exactly the same steps, and no rule about letters distinguishes the two cases. What does distinguish them is that her is itself a current word, and a current word is not a misspelling of another current word.This is the design of Baron and Rayson (2008), arrived at here independently and after making the error it exists to prevent — which is at least a good way to understand why a tool is built as it is.

So the grouping runs in three steps:

1. Reduce each form to a skeleton that collapses the period’s orthographic alternations — the shared letters u/v and i/j, doubled letters, a terminal -e. Forms sharing a skeleton are candidates for one another.

2. Split each group against a modern wordlist, so that every current word keeps its own variants and takes none of its neighbour’s.

3. Assign each old form to the single nearest current word by edit distance, so heere goes to here — one letter away — and not also to her, which is two.

The reduction is deliberately conservative and guarded by length. Without the guard it merges as with asse and at with ate: different words, run together because they happen to differ by a doubled letter and a terminal e. A false merge is worse than a missed one, because it puts a wrong reading into the compositor’s hand rather than merely a wrong spelling.

It still errs. runne is assigned to rune rather than run, the first being nearer by edit distance. Nothing about the letters separates that case from heere/here, where distance gives the right answer. This is why VARD and its relatives keep a human in the loop, and why –modern-spelling should be read as approximate.

5.3 Rebuilding it🔗ℹ

The shipped lexicon covers 1580–1640. For another period, another language of book, or a narrower window, rebuild it. Two commands, and an hour:

A run picks its lexicon in this order: the file named by the HANDPRESS_LEXICON environment variable, then the shipped "lexicon/eebo-1580-1640.rktd", then the small "samples/ado-lexicon.rktd" built from the two Much Ado texts. The last is kept because it is small enough to read, and because the difference between it and the real one is instructive: on the same copy the proportion of words altered to fit the measure rises from 8.70 per thousand to 29.38 when a corpus is behind it. The compositor then has genuine variants to choose among instead of the handful a rule could invent.

The modern wordlist is a build input and is not redistributed. Only its verdict on public-domain forms is, which is a judgement about the corpus rather than a copy of the dictionary.