How We Measure Bot Strength
Every site with a game AI eventually says something like "the new bot is stronger." Most of the time that sentence stands on a handful of games somebody watched, or on the warm feeling of having just rewritten the search. This post is about what the sentence stands on here: the harness that runs before anything on this site claims one opponent is stronger than another, what that harness can honestly conclude, and — just as important — the questions it cannot answer.
Why "it beats the old bot" is not a measurement
When the playing engine was replaced last month (the story), the tempting shortcut was to play the new engine against the old one a few times, watch it win, and call the replacement validated. The problem is arithmetic. A bot that wins six games out of ten has produced a result entirely compatible with the two bots being exactly equal — flip a fair coin ten times and six heads is unremarkable. A short match does not measure strength; it samples it, once, noisily.
So the harness treats a match the way you would treat any other noisy sample. Games are played in pairs with the colors alternated, so that whatever advantage the first move carries cancels out instead of contaminating the score. Runs are driven by a fixed random seed, so a result can be reproduced exactly rather than argued about. And the output is not a win count but a 95% confidence interval around the win rate — the range of true strengths the observed score is consistent with.
The verdicts are deliberately asymmetric. A change is called a regression only when the entire interval sits below even — when the data cannot be explained by bad luck. It is called an improvement only when the entire interval sits above even. Everything in between gets the honest label: cannot tell. And a run with too few decided games refuses to conclude anything at all, loudly, rather than letting two lucky wins green-light a change. The asymmetry is on purpose: the harness exists to block regressions and flag gains, because proving superiority to high confidence takes far more games than proving non-breakage, and a gate that demanded proof of superiority for every change would simply never pass.
Beneath all of this sit two floor opponents that need no statistics: a pawn that walks the shortest path and never places a wall, and one that wanders at random. They exist because some failures are too large for confidence intervals — an engine that cannot beat a pawn that just walks forward is broken in a way no interval is needed to see.
Equal wall-clock, never equal simulations
Comparing two different engines raises a question that sounds pedantic and decides everything: what does "the same budget" mean? There are two candidate answers: the same number of simulated games per move, or the same amount of time per move. The first sounds more scientific. It lies.
A simulation is not a unit of cost. It is a unit of whatever work that particular engine chose to do in one — and different engines pack wildly different amounts of computation into a single simulated game. Cap both sides at the same simulation count and you have quietly handed the engine with expensive simulations a subsidy: it gets to do more total thinking per move and is never charged for it. You end up measuring strength per simulation, a quantity that no opponent at the board ever experiences. The only budget a real opponent experiences is time.
So every cross-engine comparison here runs at equal wall-clock: both sides are paced to the same fixed thinking time per move, with the simulation ceiling lifted so far that it never binds, so that time is the only thing that stops the search. The harness records each side's actual milliseconds per move, so a finished run carries the proof that it kept this promise — and it records simulations per move too, precisely so that an accidental equal-simulations confound would be visible in the record instead of buried in it.
The harness does keep a second, faster profile that caps both sides at approximately equal simulation counts, and it is fine for the question it was built for: same engine on both sides, one experimental flag on versus off, did anything break. There the subsidy cancels, because both sides' simulations cost the same. It is the moment the two sides are different programs that the cheaper comparison starts lying, which is why the rule is absolute: cross-engine strength claims run on the clock.
The trap suite: a test of behavior, not strength
Win rate has a blind spot, and it is a structural one. Self-play win rate is measured against an opponent that shares the engine's own blindnesses — so a weakness the engine cannot see is also a weakness its sparring partner cannot exploit, and it never shows up in the score. The specific blindness that matters in Quoridor is the wall pocket: positions where a bot's path can be catastrophically lengthened by walls it has not accounted for. Two engines that are both blind to cages will happily play thousands of games without either one ever punishing the other for it.
The answer is a test built from opponents who were not blind. Two recorded games in which a human beat the two strongest bots on the roster were replayed move by move, and every position where the bot was to move while already deep inside a wall pocket became a fixed test case, pinned with the move the bot actually played in the lost game. "Deep inside a pocket" is not a judgment call — it is measured by a metric deliberately independent of the engine's own evaluation: how much can the opponent lengthen this bot's path with one legal wall, and with two placed greedily? The greedy version is knowingly a lower bound — a coordinated pair of walls can do worse than two individually best ones — and the suite says so rather than pretending precision it does not have.
One position from those games has become the reference point. The bot stood five squares from its goal. One human wall could add four squares to that path; a second could seal a twelve-square detour. The bot stepped deeper into the pocket, and lost the game from a winning race. The suite exists so that this move, and every move like it in those two games, is permanently on the record as a question the engine gets asked again after every change. It is not a suite the engine passes cleanly today — Nemesis's page covers what that means for how to play the top of the roster — and keeping a failing test is the point. A regression suite you already pass measures nothing.
The ladder, and why difficulty is a simulation count
The six difficulty tiers are one engine, and that engine has an unusual property: there is no separate position evaluator to detune. It judges a position by playing it out — the simulations are the judgment (how a move gets chosen). That closes off the traditional way of making an engine weaker, which is degrading its evaluation, and leaves one honest dial: how much searching it is allowed before it must commit. At the time of writing, each rung of the upper ladder roughly triples the simulation budget of the rung below it. The exact counts are tuning values and will drift; the shape — multiplicative steps on a single dial — is the design.
The bottom of the ladder is the exception, and it is disclosed rather than hidden: the two gentlest tiers are weakened by a wrapper that runs outside the search and sometimes overrides its chosen move, leaving the search itself untouched (the details). From the third tier up, what the search picks is what gets played.
A ladder built this way still has to be checked, because "more searching" does not automatically mean "wins more" once real per-move time limits enter the picture. Each rung is played against the rung below it, at each tier's real playing budget, on more than one seed, and the ordering has to hold. The launch notes tell the full story of the first such run — a top rung that came out backwards, traced to a pacing table inherited from the deleted engine — and the honest residue of the re-run is still there to see: two adjacent rungs, reported on Oracle's page and Magnus's page, that a match of that length cannot tell apart. That is not the same as the tiers being identical. It is the measurement admitting the limit of its own resolution.
What the ratings mean, and what they do not
Each opponent on the bots page carries a rating. Here is exactly what those numbers are: fixed labels, assigned to mark where each tier sits on the ladder, spaced to suggest the size of the gaps. They do not move when a bot wins or loses, the way yours does — a ladder that shifted while you climbed it would be useless for measuring the climb.
What the measurements above actually support is narrower than the numbers imply, and it is worth stating plainly. The head-to-head runs support the ordering — that each rung beats the rung below it, with the two exceptions just admitted. The gap sizes are asserted, not measured: nobody has run the thousands of games per pair it would take to estimate real rating differences between tiers. And no bot's rating has been calibrated against any human rating pool — the numbers are not comparable to a chess Elo, and a bot's rating and your rating on this site are not on the same measured scale. The one placement that did come directly from measurement is the humblest: the experimental neural engine sits at the bottom of the roster because, at equal thinking time, it currently loses to everything it has been tested against.
So when this site says one opponent is stronger than another, what stands behind the sentence is: paired games at equal wall-clock, a confidence interval that is allowed to say "cannot tell," a behavioral suite built out of real defeats, and a ladder whose ordering has been checked and whose two soft spots are published rather than smoothed over. What does not stand behind it is any claim about you. Machine-versus-machine results say nothing direct about how a bot will fare against your particular habits — which is what the opponent pages are for, and why each one describes what that bot reliably sees and reliably misses rather than just repeating its number.