Does Pair Frequency Predict Learner Responses?
vocabulary-estimation, logistic-regression, pair-frequency, model-validation
A useful model that did not earn promotion
The first post estimated a separate knowing rate in each of eight frequency strata. This follow-up removes those artificial difficulty steps. It asks whether one continuous pair-frequency curve can support better inference while item selection remains balanced, non-adaptive, and unchanged.
External evidence makes frequency a reasonable provisional predictor, but does not calibrate Lexibench’s Polish lemma–surface-form pairs:
Mandera et al. showed that Polish corpus-frequency measures predict lexical-decision performance, and that subtitle and written-corpus measures contribute differently.
Hashimoto found only a moderate relationship (
r = .50,r² = .25) between frequency and Rasch word difficulty for 403 English learners.Culligan found that log frequency from large corpora outperformed simpler corpus proxies considered in that study, while direct testing explained difficulty better.
Ha, Nguyen, and Stoeckel found age of exposure and contextual distinctiveness ahead of frequency in their random-forest analysis of meaning-recall difficulty.
Hoshino showed that distractor type and usable context alter multiple-choice item difficulty.
The defensible claim is therefore modest: frequency is a plausible proxy, not a calibrated item-difficulty scale.
A real but deliberately non-lexical fixture
The fixture contains source ranks 1–8,000 and their real pair_frequency_sn_sum values. It is not a CEFR pool, a Lexibench pool, or a lexically curated inventory. It exists only to preserve the actual shape of the frequency predictor during simulation.
Versioned fixture
ID: subtlex-pl-pair-frequency-source-ranks-1-8000-v1
SHA-256: 72f8a84b6e98fb15c868e046afa209035d405b1cdcb3906c25949ca99bdd579a
Transform
mean(log₁₀ frequency): 3.973669
population SD: 0.431078
Bounds after z-scoring
-1.052 to 6.976
8 × 1,000 rank-stratified pairs
For pair \(i\):
\[x_i = z\!\left(\log_{10}(\text{pair-frequency}_{i})\right)\]
i
The index identifying one lemma–surface-form pair.
pair-frequency_i
The corpus frequency recorded for pair i; it distinguishes lemma–form pairs, not word senses.
log₁₀
Base-10 logarithm. It compresses the large, skewed differences between raw frequencies.
z(a)
Z-standardisation: subtract the fixture mean of a and divide by its population standard deviation.
x_i
The resulting standardised log-frequency predictor for pair i, measured in standard-deviation units.
About this equation
Higher x_i means higher corpus pair frequency within this fixed, versioned pool. It is still a proxy, not calibrated item difficulty.
\[p_i = \operatorname{logistic}\!\left( 2\log(9)\frac{x_i-t}{w}\right).\]
p_i
The modelled probability that the learner knows pair i under the v2 binary response model.
logistic(a)
The function 1 / (1 + exp(−a)), which maps any real number to a probability between 0 and 1.
x_i
Pair i’s standardised log-frequency predictor.
t
The learner-specific threshold. When x_i = t, the model gives p_i = 0.5.
w
A positive width controlling how gradual the transition is; it spans the predictor distance from p = 0.1 to p = 0.9.
x_i − t
Pair i’s predictor position relative to the learner threshold.
log(9)
The natural logarithm of 9. Together with the factor 2, it makes t − w/2 map to 0.1 and t + w/2 map to 0.9.
About this equation
Larger w produces a gentler curve. Higher x_i relative to t produces a larger modelled knowing probability.
The threshold \(t\) is the predictor value where \(p_i=0.5\). The positive width \(w\) is the predictor distance from 10% to 90%. Larger widths mean a gentler transition.
(def curve-check
{:at-threshold (v2/knowledge-probability 0.7 0.7 2.0)
:one-half-width-below (v2/knowledge-probability -0.3 0.7 2.0)
:one-half-width-above (v2/knowledge-probability 1.7 0.7 2.0)})The executable check returns probabilities 0.5, 0.1, and 0.9.
Priors and deterministic grid
The versioned defaults are:
\[t \sim \operatorname{Normal}(0, 2.5)\]
t
The standardised-frequency threshold where the learner’s modelled knowing probability is 0.5.
∼
“Is distributed as.” The threshold is uncertain before responses are observed.
Normal(0, 2.5)
A Gaussian prior with mean 0 and standard deviation 2.5 on the standardised-frequency scale.
0
The centre of the prior, equal to the pool’s mean log frequency after z-standardisation.
2.5
The prior standard deviation; its large size allows thresholds well beyond the observed predictor range.
About this equation
This is a provisional modelling choice, not a threshold distribution learned from Lexibench users.
\[\log(w) \sim \operatorname{Normal}(\log(2), 0.6).\]
w
The positive predictor distance over which knowing probability rises from 0.1 to 0.9.
log(w)
The natural logarithm of w. Modelling it makes every implied value of w positive.
∼
“Is distributed as.” It expresses prior uncertainty about the width.
Normal(log(2), 0.6)
A Gaussian prior on log width, with mean log(2) and standard deviation 0.6.
log(2)
The prior centre on the log scale, corresponding to median width w = 2 on the original scale.
0.6
The prior standard deviation in log-width units.
About this equation
On the original scale this is a log-normal prior: widths are asymmetric around 2 and can never be zero or negative.
These are provisional. The posterior is evaluated on 161 threshold points from min(x)-2 to max(x)+2, crossed with 81 log-spaced widths from 0.25 to 8. Log weights are normalized with log-sum-exp.
| Prior | Mean | 2.5% | Median | 97.5% |
|---|---|---|---|---|
| Default | 3,480 | 28 | 2,816 | 7,993 |
| 2× wider SDs | 2,673 | 0 | 1,136 | 8,000 |
The prior is broad on the count scale; doubling both prior standard deviations makes it more extreme. Neither prior was learned from Lexibench responses.
Explore the curve
Loading the curve explorer…
What is observed and what is predicted
Raw response events remain :correct, :wrong, or :dont-know. V2 still maps correct to 1 and both other values to 0 for inference. Tested pair outcomes are fixed; posterior-predictive simulation draws outcomes only for untested pairs. The point estimate is the posterior-predictive mean and the interval is a deterministic seeded 95% equal-tail interval.
The fitted model assumes that unmodelled pair and complete-item effects have conditional mean zero. Its credible interval does not include uncertainty from violating that assumption. Context, distractors, guessing, slips, and sense distinctions remain outside this model.
Default grid
Mean 4,971.5
95% ETI 4,062–5,858
Doubled grid
Mean 4,971.5
95% ETI 4,064–5,868
Difference
Mean 0.0004 pair
Endpoints 2 and 10 pairs
Passes <10 / <25 tolerances
Selection remains v1
Every attempt still creates eight response-independent queues from equal-count rank strata. Each complete round selects one unseen pair per stratum and records its inclusion probability. Responses update inference only; they do not alter item order. Adaptive selection remains a non-goal.
Try one seeded v1/v2 quiz
Loading the seeded simulation lab…
The browser uses a bounded 41×21 grid and 300 predictive draws so the mechanism stays interactive. The authoritative checks and large simulation run in CLJ.
The precommitted promotion gate
Tuning covered 45 supported cells: expected totals near 10%, 30%, 50%, 70%, and 90%; widths 0.75, 1.5, and 3.0; and zero-mean pair residual SDs 0, 0.5, and 1.0. Every cell used 500 replicates. The search crossed 100 complete-round rules.
No rule satisfied all four requirements. Because there was no eligible rule, the 2,000-replicate-per-cell run below is a held-out diagnostic, not a promotion gate. Following the declared priority—coverage, then MAE, then length—it examines the least-bad coverage rule: minimum 48, 7.5% target half-width, cap 64.
| Measure | v1 | v2 diagnostic | Requirement |
|---|---|---|---|
| Aggregate coverage | 68.72% | 95.11% | ≥94.5% |
| Worst-cell coverage | — | 91.95% | ≥94% |
| MAE | 553.2 | 252.6 | v2 aggregate lower |
| Worst-cell v2/v1 MAE | — | 121.1% | ≤105% |
| Mean interval width | 1,523 | 1,248 | reported |
| Mean response log score | -0.517 | -0.369 | higher is better |
| Median items | 40 | 64 | v2 ≤ v1 |
Aggregate v2 coverage and MAE improved substantially, but the gate protects against hiding weak cells in an average. Worst-cell coverage was 91.95%, worst-cell MAE was 21.1% worse than v1, and median length was 64 rather than 40. The decision is therefore unambiguous: retain v1.
Untuned stress diagnostics
I froze the same diagnostic rule, then ran 2,000 replicates per cell without retuning. Five ability targets were crossed with a non-logistic mixture, positive and negative frequency-related residuals, separate false-positive and false-negative rates at 2%, 5%, and 10%, and measurement error increasing with rank. The stopping reason in every run remained either :precision-target or :soft-maximum under the frozen 48 / 7.5% / 64 rule.
| Stress | Cells | Coverage | Worst cell | Bias | MAE | Interval width | Log score | Median items |
|---|---|---|---|---|---|---|---|---|
| False negatives (2/5/10%) | 15 | 85.8% | 37.6% | -250.4 | 364.3 | 1,324 | -0.400 | 64 |
| False positives (2/5/10%) | 15 | 86.3% | 32.8% | 218.8 | 352.3 | 1,327 | -0.397 | 64 |
| Frequency-related residual | 10 | 95.5% | 94.3% | 1.0 | 252.7 | 1,254 | -0.349 | 64 |
| Non-logistic mixture | 5 | 94.5% | 94.3% | -25.5 | 276.8 | 1,337 | -0.408 | 64 |
| Error increasing with rank | 15 | 91.5% | 73.2% | 53.3 | 305.6 | 1,328 | -0.397 | 64 |
The frequency-related residual and mixture groups retained good aggregate coverage, but their worst-cell MAE still exceeded v1 by more than 5%. Measurement error produced severe worst-cell undercoverage. Stress results therefore reinforce the non-promotion decision; they were not used to revise the rule.
What was learned
- Replacing eight independent rates with one continuous curve greatly improves aggregate MAE and response log score under related simulations.
- Mean-zero pair residuals are not harmless at every ability/width cell. Aggregate coverage concealed poor cells.
- A useful difficulty proxy is not automatically a safe scorer.
- The next change must be a new version, not a quiet retuning of v2 after seeing held-out results.
continuous-pair-frequency-logistic-v2 remains an experimental, replayable checkpoint. The current target stays stratified-beta-binomial-v1.
:verified-negative-result