Model Calibration & Predictive Diagnostics

Webmaster Model Assessment & Empirical Findings

Our multi-engine empirical analysis evaluates rating systems on their sole physical purpose: accurately predicting unseen match outcomes.

Through extensive walk-forward validation, diagnostic ablation, and testing across more than 380,000 tournament encounters, we uncovered critical structural failures in standard models and refined the foundational pillars that define the premier GlickoD* engine:

1. The Investigation: The "Tail Overconfidence" Illusion & Its True Root Cause

In naive Glicko-2, extreme probability forecasts ($>90\%$) appear to suffer from systematic overconfidence: when the model predicts $93\%$, actual tournament win rates hovered around $\approx 89\%$.

We initially hypothesized that this was caused by an entropic ceiling on Through the Ages skill expression: that cards, military draws, and multiplayer friction meant "a 99% win probability cannot physically exist." To bend predictions back toward reality, we formulated an adaptive temperature scaling function $T(P, N)$ to compress logits toward 50%.

However, deeper diagnostic ablation across top-3% elite tournament players revealed an eye-opening empirical truth:

• The Information Defect: The apparent overconfidence was NOT caused by game entropy. It was an uncalibrated newcomer artifact. When strong newcomers (future 1750+ players) enter tournaments like the WCS or Grand Slams with blank $1500$ priors and face Grandmasters ($1950+$), standard Glicko forecasts a $93\%+$ win chance for the GM. When the GM wins only $75\%$ against this hidden shark, the raw data falsely looks like "favorite overconfidence."
• Skill Expression in TTA is Real: For well-calibrated elite players, top players genuinely win $\mathbf{\ge 95\%}$ and even $\mathbf{98\%}$ of matches against lower-tier competition (e.g. an established 2000 GM facing an average 1475 player).
• The Verdict on Temperature Scaling: Once the 15-Game Recalibration Breakthrough discovers emergent player baselines, the overconfidence artifact vanishes. Applying temperature scaling on top of calibrated data actually worsened predictions by artificially suppressing legitimate elite win probabilities (+2% to +3% underconfidence). Consequently, temperature scaling was retired from GlickoD* ($T \equiv 1.0$), establishing near-perfect tail calibration ($|\Delta| < 0.2\%$) and an all-time record ECE of 0.24%.

2. The Proven Pillars of GlickoD*

• Pillar 1: 15-Game Recalibration Breakthrough (The Primary Anchor):

  • When new players enter the tournament pool with blank stats ($1500, \mathrm{RD}=350$), they inject severe statistical noise. Active tournament veterans average $\approx 1521$, while the overall pool mean is $\approx 1471$. Nominal "1521 vs. 1500" contests are actually mismatches against uncalibrated skill, producing both a 50% underconfidence droop (from below-average newcomers) and extreme tail overconfidence (from above-average newcomers).
  • The Fix: Once a player completes 15 games, their emergent baseline is discovered. All affected historical matches are recomputed using these calibrated priors, while the newcomer's active rating remains frozen during matches $1 \dots 15$ to eliminate self-amplifying feedback. This single architectural shift eliminates the root cause of noise across the entire graph.

• Pillar 2: Golden Ratio ($\phi$) Multiplayer Information Decomposition:

  • Standard Glicko naively decomposes $N$-player matches into independent 1v1 duels, treating a 4-player victory as 3 full 1v1 wins ($W(N) = N-1$). This artificially inflates certainty and deflates RD, allowing low-game-count players with short winning streaks to unrealistically dominate leaderboards.
  • Naive fractional weighting ($w = \frac{1}{N-1}$, yielding $1.0$ match equivalent) over-penalizes multiplayer play, falsely assuming a 4-player win yields no more evidence than a 1v1 duel.
  • GlickoD* deploys Golden Ratio information scaling $W(N) = \phi^{N-2}$ ($w_N = \frac{\phi^{N-2}}{N-1}$), giving $\approx 1.618$ effective match equivalents for 3-player and $\approx 2.618$ for 4-player games, perfectly capturing multilateral information without runaway certainty.

• Pillar 3: Bilateral Composite Variance ($c = \sqrt{\phi_A^2 + \phi_B^2}$):

  • In match prediction, performance difference variance is $\text{Var}(\Theta_A - \Theta_B) = \phi_A^2 + \phi_B^2$. GlickoD* applies full bilateral composite variance shrinkage $g(c)$, preventing overconfidence whenever either competitor has elevated rating uncertainty.

3. Seasonal Resets & Veteran Mobility

Applying a gentle annual soft season reset (expanding RD and regressing ratings toward competitive tier anchors via Gaussian Golden Ratio scaling) further improves out-of-sample forward predictive accuracy (record 0.24% ECE). It prevents inactive or historical records ("graveyard players") from permanently anchoring the leaderboard, prioritizing recent form while respecting career skill.