Does Pair Frequency Predict Learner Responses?
vocabulary-estimation, logistic-regression, pair-frequency, model-validation
The result first: useful, but not promoted
The first post estimated a separate knowing rate in each of eight frequency strata. This follow-up removes those artificial difficulty steps. It asks whether one continuous pair-frequency curve can support better inference while item selection remains balanced, non-adaptive, and unchanged.
Three decision terms matter from the start:
- Gate
- A set of acceptance checks declared before seeing the results. Every check must pass.
- Promotion
- Replacing the current target scorer, v1, with the candidate scorer, v2.
- Non-promotion
- Keeping v1 because v2 failed the gate—without weakening the checks after seeing the result.
The outcome is deliberately asymmetric: a candidate may teach us something and still fail to earn operational trust. Here the historical gate used logical AND: passing four checks would still have meant non-promotion. The numbered map previews the seven stages explained below; each card links to its corresponding section.
- Construct
Turn corpus counts into one predictor per pair. - Curve
Connect frequency to probability through odds and log odds. - Infer
Use priors and a deterministic grid while separating latent and observed outcomes. - Select
Keep the balanced response-independent schedule unchanged. - Simulate
Separate replicates, phases, metrics, and seeds. - Gate
Apply all five precommitted checks. - Decide
Diagnose the failure and retain v1.
External evidence makes frequency a reasonable provisional predictor, but does not calibrate Lexibench’s Polish lemma–surface-form pairs:
Mandera et al. showed that Polish corpus-frequency measures predict lexical-decision performance, and that subtitle and written-corpus measures contribute differently.
Hashimoto found only a moderate relationship (
r = .50,r² = .25) between frequency and Rasch word difficulty for 403 English learners.Culligan found that log frequency from large corpora outperformed simpler corpus proxies considered in that study, while direct testing explained difficulty better.
Ha, Nguyen, and Stoeckel found age of exposure and contextual distinctiveness ahead of frequency in their random-forest analysis of meaning-recall difficulty.
Hoshino showed that distractor type and usable context alter multiple-choice item difficulty.
The defensible claim is therefore modest: frequency is a plausible proxy, not a calibrated item-difficulty scale.
1. Construct: real frequency values, not a real vocabulary pool
Here, fixture means a frozen input dataset that makes every simulation reproducible. “Real” describes only its source ranks 1–8,000, pair IDs, and observed pair_frequency_sn_sum values. The simulation does not use the pairs’ words, meanings, senses, contexts, distractors, or answer choices.
Nor were these 8,000 pairs curated as a CEFR or Lexibench vocabulary pool: they were taken by frequency rank, without selecting for learner relevance, lexical coverage, or item quality. Their highly skewed scores are retained instead of replaced by invented, evenly spaced frequencies, so the model sees a realistic numerical predictor range, clustering, and long tail. The resulting simulations test behaviour over that frequency distribution—not performance on real Lexibench test items.
From corpus word-form counts to one lemma–form-pair score
This section explains the raw frequency quantity used by the model below: where its counts come from, why an ambiguous surface form must be divided among lemmas, and what pair_frequency_sn_sum means after that division.
- Combine two counts for the surface form. SUBTLEX-PL provides an upstream field called
freq.sn.sum: the surface form’s count in SUBTLEX-PL’s subtitle corpus plus its count in the balanced-subcorpus National Corpus of Polish (BS–NCP). For a hypothetical surface form seen 700 times in the first corpus and 300 times in the second, this combined surface-form count is 1,000. At this point it belongs to the surface form, not to a particular lemma. - Split an ambiguous surface form using tagged uses. SUBTLEX also records how often that surface form was tagged with each lemma and part of speech. Suppose 90 of every 100 tagged uses belong to lemma A and 10 belong to lemma B. The importer uses those shares to divide the combined count: 900 for the
(surface form, lemma A)pair and 100 for the(surface form, lemma B)pair. - Store one score per pair. Those allocated values are
pair_frequency_sn_sum. An unambiguous surface form receives its entire combined count; when several part-of-speech links point to the same lemma, their shares are added before storing the pair score.
Giving both ambiguous pairs the full 1,000 would duplicate the same corpus evidence and falsely make both look common. The split is performed on linear counts because 900 + 100 still equals the original 1,000; logarithms and Zipf values do not have that additive property. Only after the pair score is constructed do we take its logarithm and z-score it to obtain \(x_i\) below.
This is an allocated frequency proxy, not a direct count observed separately for each pair and not calibrated learner difficulty. It distinguishes lemmas, not senses. It also assumes that SUBTLEX’s lemma–part-of-speech shares are a reasonable way to divide the combined total, even though that total includes BS–NCP counts.
Why take the logarithm and then z-score?
Raw pair-frequency scores span orders of magnitude and are strongly skewed toward a few very frequent pairs. The base-10 logarithm makes multiplicative differences easier to compare: 10 to 100 and 100 to 1,000 are both tenfold increases, so each becomes a step of 1 on the log scale. This stops the largest raw counts from dominating distance merely because their numbers are large.
Z-scoring then describes each log frequency relative to this fixed 8,000-pair fixture. It subtracts the fixture-wide mean log frequency and divides by its population standard deviation. The result uses standard-deviation units: 0 means average log frequency in this fixture, +1 means one standard deviation above it, and -1 means one below it. This gives the later threshold \(t\) and width \(w\) an interpretable common scale instead of raw corpus-count units.
Versioned fixture
ID: subtlex-pl-pair-frequency-source-ranks-1-8000-v1
SHA-256: 72f8a84b6e98fb15c868e046afa209035d405b1cdcb3906c25949ca99bdd579a
Transform
mean(log₁₀ frequency): 3.973669
population SD: 0.431078
Bounds after z-scoring
-1.052 to 6.976
8,000 lemma–form-pair scores
For pair \(i\), let \(f_i\) be its pair_frequency_sn_sum value. The prose above and the equation below describe the same two-step transformation:
\[x_i = \frac{\log_{10}(f_i)-\mu_{\log f}}{\sigma_{\log f}}.\]
x_i
The resulting standardised log-frequency predictor for pair i, measured in standard-deviation units.
i
The index identifying one lemma–surface-form pair.
f_i
The pair_frequency_sn_sum value for pair i: its allocated linear corpus-frequency proxy.
log₁₀(f_i)
The base-10 logarithm of f_i. It compresses the large, skewed differences between raw pair frequencies.
μ_log f
The fixture-wide mean of the 8,000 log₁₀ pair-frequency values.
σ_log f
The fixture-wide population standard deviation of those log₁₀ pair-frequency values.
About this equation
Subtracting μ_log f centres the logged frequencies; dividing by σ_log f expresses them in fixture-relative standard-deviation units. Higher x_i still means higher pair frequency, not calibrated item difficulty.
Here \(\mu_{\log f}\) and \(\sigma_{\log f}\) are the mean and population SD shown in the Transform card. Higher \(x_i\) means a more frequent pair within this fixture. The transformation preserves the pair ordering; it does not turn frequency into calibrated item difficulty or make scores independent of the versioned pool used to calculate the mean and SD.
Code detail: Constructing the standardised pair-frequency predictor
Source: pair_frequency_logistic_v2.cljc — frequency-transform
log-frequency
The code name for log10(f_i), one pair's logged frequency.
location
The implementation name for μ_log f, the fixture-wide mean log frequency.
scale
The implementation name for σ_log f, the fixture-wide population SD.
:x
The record key storing x_i, the resulting standardised predictor.
:pair-frequency-sn-sum
The fixture field containing f_i.
The transform logs every positive pair score, calculates the fixture-wide mean and population SD, then attaches x to each immutable pair record.
(defn frequency-transform [pairs]
(let [logs (mapv #(Math/log10 (:pair-frequency-sn-sum %)) pairs)
location (mean logs)
scale (population-sd logs)]
{:log10-mean location
:log10-population-sd scale
:pairs (mapv (fn [pair log-frequency]
(assoc pair :x (/ (- log-frequency location) scale)))
pairs logs)}))The fixture validator separately rejects non-positive frequencies, duplicate pair IDs, and rank drift before this function runs.
2. Curve: from probability to a logistic curve
A probability \(p\) describes expected successes out of all comparable cases. Odds compare successes with failures: \(p/(1-p)\). A probability of 0.9 means odds of 9 to 1; 0.1 means 1 to 9. Log odds take the natural logarithm of the odds. They are negative below 50%, zero at 50%, and positive above 50%. A logistic curve turns any real-valued log odds back into a probability between 0 and 1.
\[p_i = \operatorname{logistic}\!\left( 2\log(9)\frac{x_i-t}{w}\right).\]
p_i
The modelled probability that the learner knows pair i under the v2 binary response model.
logistic(a)
The function 1 / (1 + exp(−a)), which maps any real number to a probability between 0 and 1.
x_i
Pair i’s standardised log-frequency predictor.
t
The learner-specific threshold. When x_i = t, the model gives p_i = 0.5.
w
A positive width controlling how gradual the transition is; it spans the predictor distance from p = 0.1 to p = 0.9.
x_i − t
Pair i’s predictor position relative to the learner threshold.
log(9)
The natural logarithm of 9. Together with the factor 2, it makes t − w/2 map to 0.1 and t + w/2 map to 0.9.
About this equation
Larger w produces a gentler curve. Higher x_i relative to t produces a larger modelled knowing probability.
The threshold \(t\) is a continuous position on the standardised-frequency scale, not necessarily one observed pair or rank. If a pair has \(x_i=t\), the model assigns it \(p_i=0.5\); otherwise \(t\) lies between neighbouring pairs. Because higher \(x_i\) means greater frequency, pairs above the threshold get probabilities above 50% and pairs below it get probabilities below 50% for a fixed \(t\) and \(w\). The posterior keeps \(t\) uncertain rather than identifying one definitive “50% pair.” The positive width \(w\) is the predictor distance from 10% to 90%; larger widths mean a gentler transition.
v2/knowledge-probability takes arguments in the order (x, t, w) and returns the modelled probability, between 0 and 1, that the learner knows a pair at predictor position \(x\). It does not draw a simulated known/unknown response. The executable check below holds \(t=0.7\) and \(w=2.0\) fixed, then evaluates the three annotated positions.
The returned map therefore contains probabilities 0.5, 0.1, and 0.9. This checks the curve’s parameterisation; it is not fitted learner data.
Code detail: Turning predictor position into knowing probability
Source: pair_frequency_logistic_v2.cljc — logistic and knowledge-probability
x
The implementation name for x_i, pair i's standardised predictor.
threshold
The implementation name for t, the learner's 50% point.
width
The implementation name for w, the 10%-to-90% transition width.
transition-logit-span
The code constant equal to 2 log(9).
logistic
Maps the linear predictor to p_i between zero and one.
The stable logistic implementation avoids overflow for very large positive or negative log odds. Width must be positive.
(defn logistic [z]
(if (neg? z)
(let [e (Math/exp z)] (/ e (+ 1.0 e)))
(/ 1.0 (+ 1.0 (Math/exp (- z))))))
(defn knowledge-probability [x threshold width]
(when-not (pos? width)
(throw (ex-info "Transition width must be positive" {:width width})))
(logistic (/ (* transition-logit-span (- x threshold)) width)))The factor 2 log(9) makes the two half-width points exactly 0.1 and 0.9.
Explore the curve
Read the horizontal axis as pair frequency relative to this fixture and the vertical axis as modelled knowing probability. Before moving a slider, predict what will happen: threshold should slide the 50% point left or right; width should change the distance between the 10% and 90% points.
Loading the curve explorer…
3. Infer: priors and deterministic grid
The versioned defaults are:
\[t \sim \operatorname{Normal}(0, 2.5)\]
t
The standardised-frequency threshold where the learner’s modelled knowing probability is 0.5.
∼
“Is distributed as.” The threshold is uncertain before responses are observed.
Normal(0, 2.5)
A Gaussian prior with mean 0 and standard deviation 2.5 on the standardised-frequency scale.
0
The centre of the prior, equal to the pool’s mean log frequency after z-standardisation.
2.5
The prior standard deviation; its large size allows thresholds well beyond the observed predictor range.
About this equation
This is a provisional modelling choice, not a threshold distribution learned from Lexibench users.
Code detail: Encoding the threshold prior
Source: pair_frequency_logistic_v2.cljc — default-prior threshold fields
:threshold-mean
The implementation field for the prior mean of t; its value is 0.0.
:threshold-sd
The implementation field for the prior standard deviation of t; its value is 2.5.
default-prior
The versioned map containing both prior distributions used by v2.
(def default-prior
{:threshold-mean 0.0
:threshold-sd 2.5
:log-width-mean (Math/log 2.0)
:log-width-sd 0.6})The two threshold fields are the executable parameters of Normal(0, 2.5).
\[\log(w) \sim \operatorname{Normal}(\log(2), 0.6).\]
w
The positive predictor distance over which knowing probability rises from 0.1 to 0.9.
log(w)
The natural logarithm of w. Modelling it makes every implied value of w positive.
∼
“Is distributed as.” It expresses prior uncertainty about the width.
Normal(log(2), 0.6)
A Gaussian prior on log width, with mean log(2) and standard deviation 0.6.
log(2)
The prior centre on the log scale, corresponding to median width w = 2 on the original scale.
0.6
The prior standard deviation in log-width units.
About this equation
On the original scale this is a log-normal prior: widths are asymmetric around 2 and can never be zero or negative.
Code detail: Encoding the log-width prior
Source: pair_frequency_logistic_v2.cljc — default-prior log-width fields
:log-width-mean
The implementation field for the prior mean of log(w); its value is log(2).
:log-width-sd
The implementation field for the prior standard deviation of log(w); its value is 0.6.
Math/log
The natural logarithm used for log(2).
default-prior
The versioned map containing the prior parameters.
(def default-prior
{:threshold-mean 0.0
:threshold-sd 2.5
:log-width-mean (Math/log 2.0)
:log-width-sd 0.6})The implementation models width on the log scale, so every width returned to the original scale is positive.
These are provisional. The posterior is evaluated on 161 threshold points from min(x)-2 to max(x)+2, crossed with 81 log-spaced widths from 0.25 to 8. A prior states relative plausibility before this learner’s responses. A deterministic parameter grid lists the same 13,041 \((t,w)\) candidates on every run. It replaces a continuous search with a fixed approximation that can be replayed exactly.
For a binary outcome \(y_i\)—1 for correct, 0 otherwise—the likelihood at one grid point is:
\[L(t,w)=\prod_i p_i^{y_i}(1-p_i)^{1-y_i}.\]
L(t,w)
How well one threshold-and-width candidate predicts all observed binary outcomes.
∏ᵢ
Multiply one response contribution for every tested pair i.
pᵢ
The logistic knowing probability for tested pair i at this grid point.
yᵢ
The observed binary outcome: 1 for correct and 0 for wrong or don’t-know in v2.
pᵢʸⁱ(1−pᵢ)¹⁻ʸⁱ
Selects pᵢ after a correct response and 1−pᵢ otherwise.
About this equation
In plain English: candidates that assign higher probability to the responses actually seen receive more likelihood weight.
Code detail: Accumulating the response likelihood
Source: pair_frequency_logistic_v2.cljc — log-likelihood, log-bernoulli-eta, and linear-predictor
observations
The implementation collection indexed by i in the product.
x
The stored x_i predictor for one observation.
response
The raw response collapsed to y_i for this v2 likelihood.
threshold
The implementation name for t.
width
The implementation name for w.
log-bernoulli-eta
Returns log(p_i) when y_i = 1 and log(1-p_i) otherwise.
(defn log-likelihood [observations threshold width]
(reduce + 0.0
(map (fn [{:keys [x response]}]
(log-bernoulli-eta
(collapse-response response)
(linear-predictor x threshold width)))
observations)))The equation multiplies probabilities; the implementation adds their logarithms for numerical stability. These operations are algebraically equivalent.
The implementation adds log likelihoods to log prior weights, then uses log-sum-exp to normalise them safely into posterior probabilities that sum to 1. A wider grid check later repeats the calculation with 321×161 points to test whether the default numerical resolution changes the answer.
| Prior | Mean | 2.5% | Median | 97.5% |
|---|---|---|---|---|
| Default | 3,480 | 28 | 2,816 | 7,993 |
| 2× wider SDs | 2,673 | 0 | 1,136 | 8,000 |
The prior is broad on the count scale; doubling both prior standard deviations makes it more extreme. Neither prior was learned from Lexibench responses.
Code detail: Weighting the deterministic parameter grid by response likelihood
parameters
Every threshold-and-width candidate together with its log prior.
log-prior
The prior log density assigned to one grid candidate before learner responses are used.
log-likelihood
The summed response evidence for one threshold-and-width candidate.
log-weights
Unnormalised log posterior weights: log prior plus log likelihood.
normalize-log-weights
Exponentiates the stabilised log weights and makes them sum to one.
:weight
The normalized posterior probability mass stored on one grid point.
Each grid candidate receives its log prior plus the sum of Bernoulli log likelihoods. Normalisation is performed only after all candidates have been scored.
(defn posterior-grid [xs observations grid prior]
(let [parameters (parameter-grid xs grid prior)
log-weights
(mapv (fn [{:keys [threshold width log-prior]}]
(+ log-prior
(log-likelihood observations threshold width)))
parameters)
weights (normalize-log-weights log-weights)]
(mapv #(assoc %1 :weight %2) parameters weights)))The browser demonstration deliberately uses 41×21 points; the authoritative CLJ path keeps 161×81.
What is observed and what is predicted
Latent knowledge is the unobserved known-or-unknown state the model tries to infer for each pair. An observed outcome is what the quiz recorded. Those are not interchangeable: a learner can guess, slip, misunderstand a context, or meet a poor distractor. A prediction is what the fitted model says about an untested pair or future response; it is neither an observation nor direct access to the latent state.
Raw response events remain :correct, :wrong, or :dont-know. V2 still maps correct to 1 and both other values to 0 for inference. Tested pair outcomes are fixed; posterior-predictive simulation draws outcomes only for untested pairs. The point estimate is the mean of the resulting whole-pool totals. The reported 95% equal-tail credible interval runs from their 2.5th to 97.5th percentiles: conditional on this model and the observed responses, it is the central range of totals the model considers plausible. A fixed random seed makes the endpoints reproducible. The interval is not a guarantee that the learner’s true total lies inside it.
What observations contains—and where it comes from
In the likelihood, observations is the ordered collection of tested-pair response records indexed by \(i\). One record connects a selected pair to the predictor and outcome used by the scorer:
:pair-index- The selected pair’s zero-based position in the fixed 8,000-pair fixture.
:x- That pair’s log frequency, standardised using the complete fixture.
:response:correct,:wrong, or:dont-know.y- The v2 likelihood’s derived binary value: correct = 1; both other responses = 0.
In a deployed quiz, the response field would come from a learner’s recorded answer event. No Lexibench learner responses were used in this article. Here the two halves of an observation have different provenance:
The grid-resolution example immediately below uses one small, replayable stream rather than the later validation runs. It selects six complete eight-stratum rounds—48 pairs—with schedule seed 20260713. The generator uses a width of 1.5, chooses the threshold whose expected total is 4,000, and draws responses with seed 2026071303. The frozen grid-check resource stores each pair index and raw response; replay reconstructs \(x\) from the hashed fixture. Seed 2026071304 then drives 20,000 posterior-predictive draws on both the default and doubled grids. These 48 observations exist to check numerical grid resolution. They are not the tuning or held-out data.
The small generator emits :correct for a binary 1 and :dont-know for a binary 0. The browser lab alternates some displayed zero labels between :wrong and :dont-know only to demonstrate that v2 collapses both. The large gate runner stores the same measured distinction as bytes 1 and 0. None of these mechanisms claims to reproduce separate guessing, wrong-answer, and don’t-know processes.
Residual variation and misspecification
A residual is the difference left after the frequency curve has made its prediction. Two pairs with the same \(x\) can still differ because of the word, context, answer choices, or learner history. The supported simulation added independent mean-zero residual variation to log odds. That tests noisy local departures, not persistent calibrated item effects.
Model misspecification means the assumed probability model does not match the process generating outcomes. The fitted v2 interval includes uncertainty about \((t,w)\) and untested binary outcomes. It does not automatically widen for a wrong curve shape, systematic residuals, guessing, slips, context, distractors, or sense distinctions. Stress simulations introduce some of those mismatches explicitly later.
Default grid
Mean 4,971.5
95% ETI 4,062–5,858
Doubled grid
Mean 4,971.5
95% ETI 4,064–5,868
Difference
Mean 0.0004 pair
Endpoints 2 and 10 pairs
Passes <10 / <25 tolerances
4. Select: selection remains v1
Every attempt still creates eight response-independent queues from equal-count rank strata. Together, these queues determine the response-independent selection schedule. Each complete round selects one unseen pair per stratum and records its inclusion probability. Responses update inference only; they do not alter item order. Adaptive selection remains a non-goal.
Try the posterior update
Press Correct, Wrong, or Don’t know for the next prequeued item. The heatmap shows the joint threshold-and-width posterior; the two charts project that same distribution onto one parameter at a time, comparing the prior, the posterior before the latest answer, and the current posterior. A correct response contributes \(p_i\); wrong and don’t-know remain distinct stored events but both contribute \(1-p_i\) to this v2 binary likelihood.
Loading the posterior-update simulator…
Code detail: Creating a balanced schedule before any responses exist
selection-queues
Builds one seeded, shuffled queue for each frequency stratum.
rounds
The number of complete eight-stratum rounds available from every queue.
round-index
The zero-based round currently being assembled.
stratum-index
Identifies which of the eight equal-count frequency strata supplied an item.
:selection-probability
The recorded chance of selecting this item from its remaining stratum queue.
shuffle-vector
Deterministically randomises presentation order within a complete round.
Each stratum is shuffled from the explicit seed. A round takes the next item from every queue, records its selection probability, then shuffles those eight items for presentation.
(defn selection-schedule [pairs strata-count seed]
(let [queues (selection-queues pairs strata-count seed)
rounds (apply min (map count queues))]
(loop [round-index 0
state (normalize-seed (+ seed 104729))
result []]
(if (= round-index rounds)
result
(let [selected
(mapv (fn [stratum-index queue]
(assoc (nth queue round-index)
:stratum-index stratum-index
:round-index round-index
:selection-probability
(/ 1.0 (- (count queue) round-index))))
(range strata-count) queues)
[ordered next-state] (shuffle-vector selected state)]
(recur (inc round-index) next-state
(conj result ordered)))))))Tests assert one item per stratum, no repeats, deterministic replay, and equality when only response data change.
Try one seeded v1/v2 quiz
Loading the seeded simulation lab…
The browser uses a bounded 41×21 grid and 300 predictive draws so the mechanism stays interactive. After a completed quiz, the same joint heatmap and marginal charts show the fitted posterior. The authoritative checks and large simulation run in CLJ.
5. Simulate: a scorer before trusting it
Simulation makes the hidden truth available because the program creates it. A scenario states how knowledge and measurement are generated. A cell is one exact combination of scenario settings. A replicate is one newly drawn learner-pool under that cell. The nominal target is the cell’s intended expected known total before random draws; realised truth is the actual count of the 8,000 latent binary outcomes in one replicate. They are close on average but need not be equal.
V1 and v2 receive the same schedule and response prefix in each replicate. This is a paired comparison: differences between scorers are not mixed with different simulated learners or questions.
What one simulation replicate did
One replicate represents one possible learner-specific pattern of knowledge across the complete 8,000-pair fixture. It is more than a simulated sequence of quiz answers: because all 8,000 latent outcomes are generated first, the simulation knows the learner’s realised total and can check an estimate against that otherwise hidden truth.
- Start with the fixture’s 8,000 real standardised log-frequency values.
- Choose one scenario cell: a target total, curve width, and residual or measurement condition. Numerically move the threshold until the scenario’s no-random-residual expected total matches the target.
- For every pair, calculate its knowing probability and make one Bernoulli draw. The sum of those 8,000 binary outcomes is the realised truth for that replicate; it varies around the nominal target.
- Read the outcomes of the pairs in the phase’s balanced, non-adaptive schedule. In stress scenarios, optionally flip some latent outcomes to create measured responses with false positives or false negatives.
- Feed exactly the same response prefix to v1 and v2 after each complete eight-item round. Each model produces a count estimate, a 95% interval, and a response log score.
- Apply a candidate stopping rule at those checkpoints. Record whether the final interval contains the realised 8,000-pair truth, the signed and absolute count error, interval width, response log score, and quiz length.
- Fixture + cellChoose target, width, and residual or error condition.
- Latent poolDraw all 8,000 known-or-unknown outcomes; save their sum as realised truth.
- Scheduled observationsReveal seeded balanced pairs; apply only the declared measurement error.
- Paired scoringGive identical response prefixes to v1 and v2.
- StoppingApply the same candidate rule at complete rounds.
- MetricsCompare intervals, errors, log scores, and lengths with realised truth.
Code detail: Generating one complete latent learner-pool and its measured responses
known
The byte array containing one simulated latent known-or-unknown outcome per pair.
true-total
The sum of all 8,000 latent outcomes before any quiz responses are measured.
latent-probability
Computes the scenario's knowing probability for one pair.
responses
The byte array of measured outcomes for scheduled quiz items only.
measured-outcome
Applies the declared false-positive and false-negative measurement mechanism.
false-positive
The probability that a latent unknown pair is recorded as correct.
false-negative
The probability that a latent known pair is recorded as not correct.
rank-error-maximum
The largest additional measurement-error rate at the rarest rank.
The runner first draws every latent pair outcome and saves their sum. Only then does it look up the scheduled pair indexes and optionally flip measured outcomes.
(defn simulate-replicate
[xs selected
{:keys [threshold residual-sd false-positive false-negative
rank-error-maximum]
:or {false-positive 0.0 false-negative 0.0
rank-error-maximum 0.0}
:as scenario}
seed]
(let [rng (Random. (long seed))
known (byte-array pool-size)
true-total
(loop [index 0 total 0]
(if (= index pool-size)
total
(let [residual (if (zero? residual-sd)
0.0
(* residual-sd (.nextGaussian rng)))
probability
(latent-probability (nth xs index) scenario
threshold residual)
outcome (if (< (.nextDouble rng) probability) 1 0)]
(aset-byte known index (byte outcome))
(recur (inc index) (+ total outcome)))))
responses (byte-array maximum-items)]
(dotimes [index maximum-items]
(let [pair-index (:pair-index (nth selected index))
known? (= 1 (aget known pair-index))
rank-error (* rank-error-maximum (/ pair-index 7999.0))]
(aset-byte responses index
(byte (measured-outcome
rng known?
(+ false-positive rank-error)
(+ false-negative rank-error))))))
{:truth true-total :responses responses}))The first loop materialises all 8,000 latent outcomes. The second observes only the fixed schedule and applies any declared measurement-error mechanism.
Within a phase, one seeded 160-item schedule was reused for every cell and replicate. This paired the model comparisons and made the run deterministic. Tuning, held-out, and stress phases used different seeds and therefore different schedules. A limitation is that each phase is still conditional on one item schedule; these runs did not average over many independently drawn selection schedules.
What happened after a response stream existed
The large validation runner represents the 160 scheduled measured outcomes as a byte array of 1 and 0. That is an efficient representation of a binary simulation—not a production learner-event schema. The ordinary scorer accepts the preserved raw response keywords described above; the gate runner can use bytes because its declared scenarios generate only binary measured outcomes.
The runner did not generate a new learner for every stopping rule. It generated one complete latent pool and one 160-response stream for a replicate, then repeatedly exposed prefixes of that same stream. At 24, 32, 40, …, 160 responses, depending on the candidate rules being evaluated, it computed paired v1 and v2 results. Every checkpoint ended after a complete eight-item round.
For v2, observed-statistics reduces a prefix to two quantities: the number correct and the sum of \(x\) over correct responses. The scorer had already cached, for every parameter-grid point and checkpoint, the contributions from the selected \(x\) values and the moments of the untested pairs. It then:
- combines those cached terms with the observed correct count and correct-\(x\) sum to obtain one log posterior weight per grid point;
- normalises the weights and calculates the weighted expected whole-pool total;
- constructs a 95% interval by mixing grid uncertainty with uncertainty in the untested-pair total; and
- records the average response log score at that checkpoint.
The large-run interval used 512 deterministic systematic samples from the weighted grid and a moment-matched normal approximation for each grid point’s untested Poisson-binomial total. The smaller grid check above used pair-level posterior-predictive Bernoulli draws. The later numerical-shortcut callout records why these are different.
- Expose prefixesUse 24, 32, 40, … responses as required; never look beyond a checkpoint when scoring it.
- Score both modelsProduce paired estimates, 95% intervals, interval widths, and response log scores.
- Apply one ruleStarting at its minimum, select the first checkpoint meeting the precision target or soft cap.
- Recover hidden truthUse the saved sum of all 8,000 latent outcomes only for evaluation—not as scorer input.
- Measure one replicateRecord coverage, signed error, absolute error, interval width, log score, length, and stopping reason.
- Aggregate one cellAcross independent learners, calculate coverage, bias, MAE, mean width, mean log score, and quiz-length summaries.
- Judge the gateCompare every rule with v1 overall and in every supported cell; all five frozen checks must pass.
Code detail: Turning one simulated response stream into checkpoint scores and replicate metrics
xs
The complete vector of standardised pair-frequency predictors.
selected
The response-independent balanced item schedule shared by both scorers.
responses
The preserved simulated response stream consumed in complete-round prefixes.
item-counts
The complete-round checkpoints scored before a stopping rule chooses one.
truth
The simulated learner's latent known-pair total, retained only for validation.
result
One selected scorer result containing its estimate, interval, log score, and quiz length.
result-metrics
Converts a selected result and latent truth into validation metrics.
Every required prefix is scored once for both models. A rule then chooses one of those already-computed checkpoints, and only that selected result is compared with latent truth.
(defn checkpoint-scores [xs selected cache responses seed item-counts]
(into {}
(for [items-tested item-counts
:let [{:keys [correct sum-correct-x]}
(observed-statistics
xs selected responses items-tested)]]
[items-tested
{:v1 (score-v1 selected responses items-tested)
:v2 (score-v2 cache items-tested correct sum-correct-x
(+ seed items-tested))}])))
(defn result-metrics [truth result]
{:covered? (<= (:lower result) truth (:upper result))
:error (- (:estimate result) truth)
:absolute-error
(Math/abs (double (- (:estimate result) truth)))
:interval-width (:interval-width result)
:log-score (:log-score result)
:items-tested (:items-tested result)})The implementation calculates coverage and error only after a stopping rule has selected one checkpoint result.
View checkpoint scoring, stopping, replicate metrics, and cell aggregation
The result changes level as it moves through the validation. A checkpoint is evidence about one prefix; a replicate is evidence about one simulated learner; a cell summary is evidence about one declared condition; and the gate is the decision across all supported conditions.
| Level | Input | Output | What remains hidden |
|---|---|---|---|
| Checkpoint | One complete-round response prefix | v1 and v2 estimate, interval, width, and log score | Future responses and the 8,000-pair truth |
| Stopping rule | Ordered checkpoint results | First qualifying result and stopping reason | Later checkpoints |
| Replicate | Stopped result + saved latent truth | Coverage and error metrics for one simulated learner | Nothing needed for that replicate’s evaluation |
| Scenario cell | Metrics from 500 or 2,000 independent learners | Coverage, bias, MAE, width, log score, and length summaries | Other cells |
| Promotion gate | Aggregate and every-cell summaries for v1 and each v2 rule | Pass/fail for all five precommitted requirements | Held-out and stress diagnostics, until the rule is frozen |
What varied in the 45 supported cells
Tuning crossed all values in the following table. The target is exact for the baseline curve before random pair residuals and Bernoulli draws, hence “nominal”: the realised total in any replicate is not forced to equal it.
| Dimension | Values | Question asked |
|---|---|---|
| Nominal known fraction | 10%, 30%, 50%, 70%, 90% | Does performance hold for small, middle, and large vocabularies? |
| 10%–90% curve width | 0.75, 1.5, 3.0 SD | Does performance hold for steep and gradual frequency transitions? |
| Independent pair residual SD | 0, 0.5, 1.0 log-odds | What if otherwise similar-frequency pairs differ for unmodelled reasons? |
A residual was drawn independently for every pair in every replicate and added to its log odds before the latent outcome was drawn. An SD of 0 is the fitted v2 family exactly. At SD 1, a one-SD residual multiplies the pair’s odds by about e¹ = 2.72, so this is substantial unmodelled heterogeneity. It is not a persistent item calibration effect shared across simulated learners.
Three separated phases
Tuning searches choices using one dataset. A held-out evaluation uses fresh data and a different seed after choices are frozen. A stress test deliberately generates conditions outside the candidate model. Reusing the tuning seed or feeding diagnostic results back into the rule would erase that separation.
Evaluate 100 rules against all five checks.
| Phase | Cells | Replicates / cell | Simulated learner-pools | Rules scored |
|---|---|---|---|---|
| Rule tuning | 45 | 500 | 22,500 | 100 |
| Held-out supported diagnostic | 45 | 2,000 | 90,000 | 1 |
| Untuned stress diagnostic | 60 | 2,000 | 120,000 | 1 |
| Total | 150 cell-phases | — | 232,500 | — |
The 500-replicate tuning run generated 22,500 independent learner-pool realisations—not 2.25 million. The same 22,500 response streams were rescored under all 100 rules. Across the three phases the immutable artifacts record 232,500 simulated learner-pools.
Code detail: Reading immutable simulation-result artifacts at render time
tuning-result
The immutable map containing the historical tuning evidence used by the article.
io/resource
Resolves a versioned classpath resource without depending on the current directory.
slurp
Reads the frozen EDN resource as text.
edn/read-string
Parses the resource text as Clojure data rather than executing it as code.
The article does not rerun 232,500 learner-pools during publication. It reads frozen EDN data produced by the separately seeded validation runs.
(def tuning-result
(edn/read-string
(slurp (io/resource
"language_learning/vocabulary_estimation/
pair_frequency_logistic_v2_tuning.edn"))))The rendered tables, gate inspector payload, assertions, fixture ID, hash, seeds, and numerical-method record all come from those versioned resources.
Metrics: what counted as better?
Coverage is the fraction of repeated 95% intervals containing the replicate’s realised truth. Calibration means those intervals achieve their stated long-run coverage under the tested scenario; it does not mean every individual interval contains truth. Bias is mean signed error: estimate minus truth. Mean absolute error (MAE) ignores direction and averages the size of the count error.
Response log score rewards probability assigned to the response that occurred; larger (less negative) is better. It tests response prediction, not count calibration by itself. Quiz length is the number of administered items at stopping. Monte Carlo standard error (MCSE) describes noise from a finite number of simulation replicates. For estimated coverage \(\hat p\) from \(n\) independent replicates:
\[\operatorname{MCSE}(\hat p)\approx \sqrt{\frac{\hat p(1-\hat p)}{n}}.\]
p̂
The observed fraction of simulated intervals containing realised truth.
n
The number of independent replicates in the cell.
p̂(1−p̂)
The Bernoulli variance implied by the observed coverage.
square root
Converts the variance of the estimated fraction into a standard error.
About this equation
MCSE measures simulation noise, not uncertainty about whether the simulated scenarios resemble real learners.
Code detail: Computing coverage Monte Carlo standard error
Source: pair_frequency_logistic_v2_article.clj — coverage-mcse
coverage
The implementation name for p-hat, the observed coverage fraction.
replicates
The implementation name for n, the number of independent simulation replicates.
Math/sqrt
The square-root operation shown in the equation.
(defn coverage-mcse [coverage replicates]
(Math/sqrt
(/ (* coverage (- 1.0 coverage))
replicates)))The descriptive code names map directly to p-hat and n while leaving the equation's calculation unchanged.
6. Gate: precommitment before results
Precommitment means declaring candidate rules, metrics, thresholds, and the all-checks-must-pass logic before seeing their results. It prevents a near miss from becoming a pass by moving the finish line afterwards.
How stopping rules were tuned
The search crossed minimum lengths 24, 32, 40, and 48; interval half-width targets 5%, 7.5%, 10%, 12.5%, and 15% of the pool; and soft caps 64, 80, 96, 128, and 160. This produced 100 valid rules. A rule stopped at the first complete-round checkpoint at which its width target was met, or at its cap.
Code detail: Applying one candidate stopping rule at complete rounds
precision?
True when the current interval half-width meets the rule's requested pool-relative target.
soft-maximum?
True when the rule has reached its declared item cap.
target-half-width-ratio
The largest accepted interval half-width as a fraction of the 8,000-pair pool.
items-tested
The number of responses included in the current inference update.
pool-size
The fixed number of lemma–surface-form pairs in the versioned fixture.
:stopping-reason
Records whether precision or the soft maximum ended the quiz.
Starting at the rule's minimum, the runner checks eight-item boundaries until precision is reached or the soft cap is reached.
(let [precision?
(<= (/ (- upper lower) 2.0)
(* target-half-width-ratio pool-size))
soft-maximum?
(>= items-tested soft-maximum-items)]
(if (or precision? soft-maximum?)
{:items-tested items-tested
:stopping-reason
(if precision? :precision-target :soft-maximum)}
(recur (+ items-tested strata-count))))The 100 rules change only minimum, target half-width, and cap; the balanced schedule and scorers remain fixed.
Each rule had to pass all five checks below. Aggregate checks prevent a noisy single cell from dominating; cellwise checks prevent good averages from hiding a subgroup in which the new scorer is poorly calibrated or less accurate than v1.
| Check | Requirement | Purpose |
|---|---|---|
| Aggregate interval coverage | ≥94.5% | Keep the nominal 95% interval honest overall |
| Every-cell interval coverage | ≥94% | Protect each ability × width × residual condition |
| Aggregate MAE | Lower than v1 | Improve average count accuracy |
| Every-cell MAE | ≤105% of v1 | Do not buy average gains with a large local regression |
| Median quiz length | No longer than v1 | Do not require more answers for the gain |
Code detail: Requiring all five promotion checks with logical AND
and
Returns true only when every promotion requirement is satisfied.
aggregate
Metrics pooled across all supported tuning cells.
cells
The separate ability-by-width-by-residual tuning results protected by cellwise checks.
v1-aggregate
The pooled baseline metrics used for the aggregate comparison.
v1-cells
Baseline metrics aligned cell by cell with the v2 candidate results.
every?
Requires the stated comparison to hold in every individual cell.
The aggregate and cellwise checks are evaluated together. Clojure's and returns true only when every clause is true.
(and (>= (:coverage aggregate) 0.945)
(every? #(>= (:coverage %) 0.94) cells)
(< (:mae aggregate) (:mae v1-aggregate))
(every? true?
(map #(<= (:mae %1) (* 1.05 (:mae %2)))
cells v1-cells))
(<= (:median-items aggregate)
(:median-items v1-aggregate)))There is no score, majority vote, or discretionary override. One false clause prevents promotion.
Gate result: no rule passed
No rule satisfied all five checks. The rule with the best worst-cell coverage started at 48 items, targeted a 7.5%-of-pool half-width, and capped at 64. In tuning it achieved 95.48% aggregate coverage and MAE 253.0, but its worst cell covered only 92.4%, its worst cell’s MAE was 22.0% above v1, and its median length was 64 rather than v1’s 40. It was therefore not eligible for promotion.
Code detail: Selecting only among rules that already passed
choose-rule
Returns the preferred eligible stopping rule, or nil when no rule passed the gate.
:passes?
The precomputed all-requirements gate result for one candidate rule.
sort-by
Orders only eligible rules using the declared preference keys.
juxt
Builds the ordered tuple of median length, mean length, and deterministic tie-breakers.
tuning-result
The immutable historical tuning artifact containing every evaluated rule.
Selection first filters to passes? rules, then prefers the shortest by median and mean length with deterministic tie-breakers. With no passes, it returns nil.
(defn choose-rule [tuning]
(first
(sort-by
(juxt #(get-in % [:aggregate :median-items])
#(get-in % [:aggregate :mean-items])
#(get-in % [:rule :minimum-items])
#(get-in % [:rule :soft-maximum-items])
#(get-in % [:rule :target-half-width-ratio]))
(filter :passes? (:rules tuning)))))
;; Historical result:
(choose-rule tuning-result) ;=> nilThe best-worst-coverage rule was chosen only for diagnosis after the promotion gate had failed; it was never relabelled as a winner.
Inspect the historical gate
The five actual values below come from the immutable tuning artifact. Moving a slider asks a counterfactual question only: “What if the threshold had been different?” It never edits the EDN file, reruns the simulation, or changes the recorded non-promotion decision. Reset restores the precommitted gate.
Loading the gate inspector…
At the historical defaults, only aggregate coverage and aggregate MAE pass. Worst-cell coverage, worst-cell MAE ratio, and median length fail. The overall decision is therefore not promoted: two passes out of five are not enough, and even four would not have been enough.
7. Decide: diagnose the failure and retain v1
Because there was no eligible rule, the 2,000-replicate-per-cell run below is a held-out diagnostic, not a promotion gate. Following the declared priority—coverage, then MAE, then length—it examines that least-bad coverage rule: minimum 48, 7.5% target half-width, cap 64.
“Coverage” is the percentage of replicates whose reported 95% interval contains the realised 8,000-pair truth. Bias is mean estimate minus truth; MAE is the mean absolute size of that error. The response log score measures probability assigned to the observed answers; larger (less negative) is better. It evaluates response prediction, not count calibration by itself.
| Measure | v1 | v2 diagnostic | Requirement |
|---|---|---|---|
| Aggregate coverage | 68.72% | 95.11% | ≥94.5% |
| Worst-cell coverage | — | 91.95% | ≥94% |
| MAE | 553.2 | 252.6 | v2 aggregate lower |
| Worst-cell v2/v1 MAE | — | 121.1% | ≤105% |
| Mean interval width | 1,523 | 1,248 | reported |
| Mean response log score | -0.517 | -0.369 | higher is better |
| Median items | 40 | 64 | v2 ≤ v1 |
Code detail: Recording the large-run approximation and refusing to excuse a near miss
:interval-method
Names the complete numerical method recorded with the simulation result.
:parameter-mixture
States how uncertainty over threshold-and-width grid points was sampled.
:untested-pair-total
States the approximation used for each grid point's untested-pair total.
:draws
The fixed number of deterministic systematic grid samples used per interval.
Every result artifact names the approximation and draw count, so a future rerun can reproduce or replace it without pretending the computation was exact.
:interval-method
{:parameter-mixture :deterministic-systematic-grid-sampling
:untested-pair-total :moment-matched-poisson-binomial-normal
:draws 512}The shortcut was declared and recorded, but it was not validated tightly enough to rescue a failed cell. A candidate near a threshold needs stronger numerical evidence, not a post-hoc waiver.
View the deterministic mixture interval and artifact metadata
Aggregate v2 coverage and MAE improved substantially, but the gate protects against hiding weak cells in an average. Worst-cell coverage was 91.95%, worst-cell MAE was 21.1% worse than v1, and median length was 64 rather than 40. The decision is therefore unambiguous: retain v1.
Concretely, the weakest cell covered 1,839 of 2,000 realised truths and missed 161. Its Monte Carlo standard error is about 0.61 percentage points, so the 2.05-point miss against the 94% threshold is not a one- or two-draw wobble. Aggregate MAE can still improve while one cell regresses: that is exactly why both aggregate and cellwise checks were declared. A median of 64 means at least half of v2 diagnostic runs used 64 answers, versus a v1 median of 40.
V1’s poor aggregate coverage here is not evidence that v1 has passed a realistic validation study. These supported scenarios deliberately generate smooth frequency curves, so they are favourable to v2 and unfavourable to v1’s eight separate, sparsely observed rates—especially near 10% and 90%. The question was narrower: even under favourable synthetic conditions, did v2 meet absolute coverage requirements, avoid materially worse cells, and shorten or preserve test length? It did not.
Untuned stress diagnostics
I froze the same diagnostic rule, then ran 2,000 replicates per cell without retuning. Five ability targets were crossed with a non-logistic mixture, positive and negative frequency-related residuals, separate false-positive and false-negative rates at 2%, 5%, and 10%, and measurement error increasing with rank. The stopping reason in every run remained either :precision-target or :soft-maximum under the frozen 48 / 7.5% / 64 rule.
| Stress | How responses were generated | What it probes |
|---|---|---|
| Non-logistic mixture | Average of a steep curve (width 0.75, threshold t−0.6) and a gentle curve (width 3.0, threshold t+0.6). | A monotone frequency relationship that one logistic curve cannot exactly represent. |
| Frequency-related residual | Add either −0.75x or +0.75x to the baseline log odds. | An omitted systematic frequency component that weakens or strengthens the fitted slope. |
| False positive | Flip a latent unknown pair to a correct response with probability 2%, 5%, or 10%. | Guessing-like responses; truth remains the latent known count. |
| False negative | Flip a latent known pair to an incorrect response with probability 2%, 5%, or 10%. | Slip or measurement failure; truth remains the latent known count. |
| Rank-increasing error | Increase both flip rates linearly from 0 at the most frequent end to 2%, 5%, or 10% at rank 8,000. | Measurement quality that worsens systematically toward rarer pairs. |
Each mechanism was crossed with all five nominal ability targets. The mixture therefore contributed 5 cells, the two residual slopes 10, and each three-rate error family 15: 60 cells and 120,000 new learner-pool realisations in total.
| Stress | Cells | Coverage | Worst cell | Bias | MAE | Interval width | Log score | Median items |
|---|---|---|---|---|---|---|---|---|
| False negatives (2/5/10%) | 15 | 85.8% | 37.6% | -250.4 | 364.3 | 1,324 | -0.400 | 64 |
| False positives (2/5/10%) | 15 | 86.3% | 32.8% | 218.8 | 352.3 | 1,327 | -0.397 | 64 |
| Frequency-related residual | 10 | 95.5% | 94.3% | 1.0 | 252.7 | 1,254 | -0.349 | 64 |
| Non-logistic mixture | 5 | 94.5% | 94.3% | -25.5 | 276.8 | 1,337 | -0.408 | 64 |
| Error increasing with rank | 15 | 91.5% | 73.2% | 53.3 | 305.6 | 1,328 | -0.397 | 64 |
The frequency-related residual and mixture groups retained good aggregate coverage, but their worst-cell MAE still exceeded v1 by more than 5%. Measurement error produced severe worst-cell undercoverage. Stress results therefore reinforce the non-promotion decision; they were not used to revise the rule.
Model validation is not a software build
This article’s automated CLJ/CLJS checks establish parity, deterministic replay, and the declared implementation behaviour. Its render and browser checks establish that readers can inspect the evidence. Neither answers the model question: did v2 satisfy every frozen coverage, error, and test-length threshold? It did not. Perfect software and publication checks cannot turn that failed model gate into a pass.
Article 0 defines the full theory-to-algorithm cycle and the three validation lanes. This article keeps the local consequence visible because it determines the result: v2 remains a non-promoted checkpoint.
A later promoted scorer would still retain its versioned predecessor so a rollback could restore the earlier target without rewriting old events or articles. Here no rollback is needed: v2 never replaced v1.
What was learned—and what comes next
- Replacing eight independent rates with one continuous curve greatly improves aggregate MAE and response log score under related simulations.
- Mean-zero pair residuals are not harmless at every ability/width cell. Aggregate coverage concealed poor cells.
- A useful difficulty proxy is not automatically a safe scorer.
- The next change must be a new version, not a quiet retuning of v2 after seeing held-out results.
continuous-pair-frequency-logistic-v2 remains an experimental, replayable checkpoint. The current target stays stratified-beta-binomial-v1.
The next article will define and version how self-reported CEFR chooses a lemma–surface-form-pair pool. Later articles will consider correlated form pairs through latent lemma knowledge; model correct, wrong, and don’t-know separately with guessing and slips; calibrate complete item versions before evaluating IRT and adaptive selection; and investigate multiple contexts or senses only if stable sense identifiers and repeated observations make them identifiable. Those are previews, not features silently added to v2.
Pair frequency was useful enough to study further, but this continuous v2 scorer did not earn promotion. The gate stayed fixed, the negative result stayed visible, and v1 remained the target.