Estimating Vocabulary Size with a Simple Bayesian Model
vocabulary-estimation, beta-binomial, posterior-predictive, stratified-sampling
I am building Lexibench, a vocabulary-testing product. Much of its user interface is already in place, but its scorer is being replaced. The model in this post is a deliberately small first pass: it is not deployed in Lexibench.
I reached it through an unusually useful learning loop. Codex acts as a tutor and makes visual explanations; I annotate those explanations in Lavish; then we repeat. Each pass exposes a mistaken assumption or a word I was using too casually.
This article records where that loop has taken me so far. I will distinguish verified mathematics from provisional modelling choices throughout. The distinction matters: correct algebra cannot make a poorly defined target meaningful.
First define what is being estimated
The estimand here is:
Receptive knowledge of lemma–surface-form pairs in a fixed, versioned pool.
A lemma such as run may have several surface forms. The frequency data I have for a pair does not distinguish senses. Each pair therefore gets one canonical item in context. The context and translation select an intended meaning for that item, but they remain measurement metadata; this model does not pretend it has estimated sense-specific knowledge.
This is narrower than “How many words do you know?” That narrowing is useful: it gives the count a denominator, a pool version, and a repeatable item.
A simplified non-adaptive test
For exposition, imagine a cumulative pool split by frequency rank into eight strata with the same number of pairs. A round samples one item uniformly at random from each stratum. Later answers do not change which item is selected: selection remains non-adaptive.
An adaptive test does change later item selection in response to earlier answers. In principle, it can choose the next question expected to be most informative about the learner’s current knowledge, avoiding many questions that look much too easy or much too hard. That can shorten a test, but only if the item-difficulty model is trustworthy.
I therefore want to keep selection non-adaptive while collecting response data from real learners. Those data should reveal the relative difficulty of the actual questions—including context effects and departures from frequency rank—before I implement an adaptive test. Adapting sooner would risk steering the test with assumptions that have not yet been checked against learners.
most frequent] P --> S2[Stratum 2] P --> S3[Stratum 3] P --> S4[Stratum 4] P --> S5[Stratum 5] P --> S6[Stratum 6] P --> S7[Stratum 7] P --> S8[Stratum 8
least frequent] S1 --> R[One random item from each
8-item round] S2 --> R S3 --> R S4 --> R S5 --> R S6 --> R S7 --> R S8 --> R
Frequency rank partitions the illustrative pool; this diagram does not claim that rank is a calibrated item-difficulty scale.
The interface keeps three raw response values: :correct, :wrong, and :dont-know. Stage one maps a correct response to “known” and both other responses to “not known.” Keeping the raw outcomes distinct lets a later model treat guessing, slips, and explicit uncertainty differently without corrupting the original data.
(defn collapse-response
"Map a raw response to the binary stage-one likelihood outcome."
[response]
(case response
:correct 1
:wrong 0
:dont-know 0
(throw (ex-info "Unknown raw response"
{:response response
:allowed #{:correct :wrong :dont-know}}))))(defn collapse-responses
"Return the sufficient statistics {:k correct :n attempted}."
[responses]
{:k (reduce + (map collapse-response responses))
:n (count responses)})(into {} (map (juxt identity collapse-response)
[:correct :wrong :dont-know])){:correct 1, :wrong 0, :dont-know 0}The output confirms that :wrong and :dont-know remain different raw keys even though their stage-one likelihood contribution is the same.
One stratum: Beta in, Beta out
Let \(p_s\) be the knowing rate in stratum \(s\). This first pass gives every stratum the same prior:
\[p_s \sim \operatorname{Beta}(1,1).\]
That density is uniform over rates from zero to one. I am not claiming that it is universally “uninformative”; parameterization and context matter. If \(k\) of \(n\) sampled pairs are correct, the Bernoulli/binomial likelihood is proportional to \(p_s^k(1-p_s)^{n-k}\), so:
\[p_s \mid k,n \sim \operatorname{Beta}(1+k,1+n-k).\]
(defn posterior-parameters
"Beta posterior parameters for a Beta(alpha,beta) prior and k of n correct."
([k n] (posterior-parameters 1.0 1.0 k n))
([alpha beta k n]
{:pre [(<= 0 k n) (pos? alpha) (pos? beta)]}
{:alpha (+ alpha k)
:beta (+ beta (- n k))}))(posterior-parameters 3 4){:alpha 4.0, :beta 2.0}{:alpha 4.0, :beta 2.0} is the entire update. “Posterior” names the updated density for \(p_s\). The “likelihood” is the information supplied by the last response as a function of \(p_s\); it is not itself a posterior.
Try the update
Press Correct or Don’t know. The accessible SVG compares the uniform prior, the previous posterior, the last-response likelihood, and the current posterior. Curve height is density, not the probability of one exact value of \(p\).
Loading the Bayesian update simulator…
From a knowing rate to a vocabulary total
For a stratum containing \(N_s\) pairs, of which \(n_s\) were tested and \(k_s\) were correct:
- draw \(p_s\) from the posterior;
- draw the number known among the untested pairs from \(\operatorname{Binomial}(N_s-n_s,p_s)\);
- add the \(k_s\) observed correct pairs;
- sum the eight stratum totals.
The following functions are the executable version. Fastmath supplies the Beta and binomial distributions; a seeded Mersenne Twister makes the rendered result reproducible.
(defn posterior-predictive-stratum
"Draw known pairs in one stratum, including observed correct pairs."
[rng {:keys [pool-size k n]}]
(let [{:keys [alpha beta]} (posterior-parameters k n)
posterior (random/distribution :beta
{:alpha alpha :beta beta :rng rng})
p (random/sample posterior)
untested (- pool-size n)
predicted (random/distribution :binomial
{:trials untested :p p :rng rng})]
(+ k (random/sample predicted))))(defn posterior-predictive-total
"Draw and sum known-pair counts across strata."
([strata] (posterior-predictive-total 20260712 strata))
([seed strata]
(let [rng (random/rng :mersenne seed)]
(reduce + (mapv #(posterior-predictive-stratum rng %) strata)))))(defn quantile
"Nearest-rank-style quantile from a finite numeric sample."
[xs probability]
{:pre [(seq xs) (<= 0.0 probability 1.0)]}
(let [ordered (vec (sort xs))
i (long (Math/floor (* probability (dec (count ordered)))))]
(nth ordered i)))(defn equal-tail-quantiles
"Return the central equal-tail interval from a finite sample."
([xs] (equal-tail-quantiles xs 0.95))
([xs mass]
(let [tail (/ (- 1.0 mass) 2.0)]
{:lower (quantile xs tail)
:upper (quantile xs (- 1.0 tail))})))Worked example: a pedagogical 8,000-pair pool
8,000 is an illustration, not a current CEFR or Lexibench pool size. It consists of eight 1,000-pair strata. After four complete rounds, suppose the correct counts are [4 4 3 3 2 1 1 0].
(def worked-correct-counts [4 4 3 3 2 1 1 0])(def worked-strata
(mapv (fn [k] {:pool-size 1000 :k k :n 4})
worked-correct-counts))(defn analytic-stratum-mean
"Posterior-predictive mean for one stratum."
[{:keys [pool-size k n]}]
(let [{:keys [alpha beta]} (posterior-parameters k n)]
(+ k (* (- pool-size n) (/ alpha (+ alpha beta))))))(def worked-analytic-mean
(long (reduce + (map analytic-stratum-mean worked-strata))))worked-analytic-mean4334The posterior-predictive mean is 4,334 pairs. Row by row:
| Stratum | Pool | Correct / tested | Posterior | Mean known |
|---|---|---|---|---|
| 1 | 1000 | 4 / 4 | Beta(5, 1) | 834 |
| 2 | 1000 | 4 / 4 | Beta(5, 1) | 834 |
| 3 | 1000 | 3 / 4 | Beta(4, 2) | 667 |
| 4 | 1000 | 3 / 4 | Beta(4, 2) | 667 |
| 5 | 1000 | 2 / 4 | Beta(3, 3) | 500 |
| 6 | 1000 | 1 / 4 | Beta(2, 4) | 333 |
| 7 | 1000 | 1 / 4 | Beta(2, 4) | 333 |
| 8 | 1000 | 0 / 4 | Beta(1, 5) | 166 |
| Total | 8,000 | 18 / 32 | — | 4,334 |
The exact discrete posterior-predictive distribution gives a 95% equal-tail credible interval of 3,404–5,249. Reader-facing, I would report:
About 4,330 known pairs, with a 95% credible interval of roughly 3,400–5,250, conditional on this model and fixed pool.
“95% credible interval” is not “95% certainty that every modelling choice is right.” It describes posterior uncertainty conditional on the specified model, prior, observed responses, and pool.
A seeded numerical check
The Monte Carlo calculation below is an independent numerical check on the analytic mean and exact reference interval, not the source of those values.
(def worked-simulation-summary
{:draws simulation-draws
:mean (double (/ (reduce + worked-simulation) simulation-draws))
:equal-tail-95 (equal-tail-quantiles worked-simulation)})worked-simulation-summary{:draws 100000,
:mean 4331.49906,
:equal-tail-95 {:lower 3398, :upper 5244}}Small differences are Monte Carlo error. The verification tolerance used for this draft is ±20 pairs for the mean and ±40 pairs for each endpoint.
When should the test stop?
The stopping rule is also provisional:
- ask at least 32 items (four complete rounds);
- reassess only after another complete eight-item round;
- target a 95% interval half-width near 10% of the pool;
- use 96 items as a soft maximum;
- always allow the learner to stop voluntarily.
(defn stopping-check
"Evaluate the provisional rule at an eight-item round boundary."
[{:keys [items-tested interval pool-size voluntary?]
:or {voluntary? false}}]
(let [[lower upper] interval
complete-round? (zero? (mod items-tested 8))
assess? (and complete-round? (>= items-tested 32))
half-width (/ (- upper lower) 2.0)
target? (and assess? (<= half-width (* 0.10 pool-size)))
soft-max? (and assess? (>= items-tested 96))]
{:complete-round? complete-round?
:assess? assess?
:half-width half-width
:target? target?
:soft-max? soft-max?
:stop? (or voluntary? target? soft-max?)}))| Items | Complete round? | Assess? | Half-width | Stop? |
|---|---|---|---|---|
| 24 | Yes | No | 1,100 | No |
| 32 | Yes | Yes | 923 | No |
| 36 | No | No | 800 | No |
| 40 | Yes | Yes | 750 | Yes |
| 96 | Yes | Yes | 900 | Yes |
At 36 items the interval happens to be narrow enough, but the function does not assess it: 36 is not the end of an eight-item round. At 96, the soft maximum recommends stopping even when the width target is missed.
What this model postpones
A small coherent model is a useful place to begin, not a useful place to end. The next steps are:
use continuous pair frequency as a difficulty proxy rather than treating eight bins as calibrated difficulty — that is the next post;
define and version the conversion among CEFR descriptors, lemmas, and pair inventories;
aggregate pair probabilities hierarchically into latent lemma knowledge;
use all three response outcomes in a guessing/slip model;
investigate multiple contexts, item calibration, item-response theory, and adaptive selection.
In particular, I am not using a naive independent-form formula as a final lemma estimator. Forms of the same lemma are related, contexts vary in informativeness, and a latent-variable model should express that structure.
Sources and further reading
- Gelman et al., Bayesian Data Analysis, for posterior and posterior-predictive reasoning.
- Fastmath random-distribution documentation, for the executable Beta, binomial, sampling, and inverse-CDF interfaces used here.
- Apache Commons Math distribution API, the distribution implementation underneath this Fastmath path.
- Council of Europe, CEFR Companion Volume (2020), for the broader language-proficiency framework; it does not define this pair-count estimand.
- Brysbaert & New, SUBTLEX-US frequency norms, for evidence that contextual word frequency can be useful; this does not make frequency rank a calibrated item-difficulty scale.
This is a learning-in-public checkpoint. Follow-up posts will replace the roughest assumptions one at a time, beginning with continuous frequency.
:verified