The Bug Our Win Rate Couldn't See

In early July an internal engineering note went into the repository with an unflattering title: "Why the bot walks into wall cages." Players had been saying for a while that the strongest bots felt stupid in one specific way, and one player went past saying it, beating the two strongest bots — including Nemesis — with the same kind of plan game after game. Meanwhile, every number used to track engine strength said things were fine. Self-play results were healthy. Nothing flickered.

Both things were true at once, and that is the real subject of this post: a metric that stays green while the behaviour it vouches for is broken, and the work of proving that your own dashboard cannot see what your players can.

What a cage is

If you have not met the term: a cage is a set of walls, placed over several turns, that turns the region in front of a pawn into a pocket — escapable, but only by a long detour. The rules forbid ever sealing a player in completely, so a cage is entirely legal. It does not remove your opponent's path; it stretches it, sometimes by eight or ten moves, which in a racing game is the whole game. The strategy article on traps and cages covers how to build one and how to escape one; the glossary has the short version.

One property matters here: a cage is a plan. No single wall makes a cage. Two or three walls, placed in the right order, each pointless without the others, make a cage — and "plan" turns out to be exactly the thing the engine of that era could not represent.

The position that settled the argument

The recorded game in which Nemesis lost gave us the decisive position. The bot's pawn stood five squares from its goal, with three reasonable moves available. For each, ask two questions: how far from the goal does it leave me, and how much distance can the opponent's best two-wall seal then add?

The retreat looks worst and is the only safe move: it gives up ground now but caps what walls can do afterwards. The forward step looks best and is the trap — it walks into a pocket where two walls turn a four-square finish into a twelve-square one. In the recorded game, the human set exactly this trap and won with it.

Replaying that position, the top-tier engine chose the forward step on all five random seeds — five fully independent re-runs of its search. And it was not a coin flip it kept losing: the retreat was not even inside the band of moves the search considered roughly equivalent. The engine did not overlook the safe move. It examined it, and actively preferred the trap.

The playouts were the judge

The engine of that era was a Monte Carlo tree search. To value a move, it plays thousands of fast, semi-random games — playouts — from the position after the move, and takes the fraction of those imagined games it wins as the move's worth. As a backstop it also carried a hand-written formula for judging playouts that ran out of depth: the kind of positional arithmetic you would expect to be doing the real work.

Measured, it did no work at all: across thirty-six recorded searches at three difficulty tiers, every playout ran to an actual finish and the backstop never fired once. The thing genuinely judging every position was the playout policy — the rule for how the imagined games get played. Ours was simple: about eighty-five percent of the time, step along your own shortest path; most of the rest of the time, place a plausible-looking wall. Both imaginary players race.

Now put the halves together. A cage takes two or three specific walls in a specific order. A playout places walls independently at random, so the chance that one stumbles onto the exact two-wall seal and then follows it up coherently is close enough to zero not to matter. In all the futures the engine imagined, nobody ever built a cage — so the danger of standing in a cageable pocket was not represented anywhere in its judgment. At the decisive position, the estimated values of the safe retreat and the trap differed by between 0.002 and 0.03. Noise.

The confidence numbers make the same point more vividly. Across fifty-four genuinely contested midgame positions, the engine's estimate of its own winning chances reached 90.5 percent in dead-level positions — and 93.7 percent in positions where it was actually two steps behind in the race. A saturated win rate did not mean "this game is decided." It meant "all of my imagined futures assumed a race."

Four fixes that did not fix it

The obvious responses were tried in order, and each was measured rather than eyeballed.

Teach the playouts to block. The imaginary players were given a preference for walls that cut the opponent's next step. The engine still walked into the trap five times out of five, and the overall trap-suite score got slightly worse — mean cage vulnerability moved from 9.33 to 9.44. It shipped switched off, as documented evidence of a negative result.

Make the playouts wall-happy. Dropping the step-along-your-path probability from 0.85 to 0.30 did flip the decisive move — by 0.107 against 0.109, a rounding error rather than a preference — and it flipped back at a different playout count. The suite got worse here too.

Break ties toward safety. Among moves the search genuinely could not tell apart, prefer the one with lower cage vulnerability. This fixed one of nine suite positions — and not the decisive one, because there the search was not tied. It preferred the trap.

Try someone else's engine. The same suite was run against the open-source gorisanson/quoridor-ai project (MIT licensed), another engine of the same family. Better on average — but it still left eight or more moves of cage vulnerability in five of the nine positions, and at the decisive position it chose a wall that left it worse off than our engine's losing move did. It also exposed a trap inside the metric itself: that engine answers six of the nine positions with a wall rather than a pawn move, and a wall move trivially cannot walk into a pocket — so part of its apparent edge was an artifact of what was being counted.

Four attempts, four measured failures: you cannot patch a playout policy into fearing a plan it cannot generate. The blindness was structural.

What a trap suite is, and why win rate cannot replace it

The lasting output of the investigation was not a fix but an instrument. The two recorded games in which a human beat the two strongest bots were replayed move by move, and every position where the bot stood inside a developing cage became a fixed test case, scored by cage vulnerability: after the move the engine chooses, how many extra moves can the best available wall seal inflict on it? At the time, the engine failed all nine positions.

Here is the claim in this post's title, stated plainly. Self-play win rate — the number that ordinarily decides whether an engine change ships — is provably blind to this entire class of behaviour. Not weakly correlated. Blind. Both players in a self-play match run the same playout policy, so neither ever builds a cage, so cage vulnerability is never punished, so a change that doubles it or halves it moves the win rate not at all. The same week demonstrated this from both directions: a tie-breaking change that was exactly strength-neutral in head-to-head play — eleven wins to nine across twenty games — cut the engine's aimless shuffling moves from 32 to 10, while the trap-aware playout change moved the cage metric, in the wrong direction, without the strength numbers registering anything. The two metrics are not two views of one quantity. They measure different things, and a green one cannot vouch for the other.

What actually changed

Three things shipped directly from the investigation. A fast path for checking wall legality, long disabled behind a comment declaring it unsound, turned out to be sound all along — a coordinate bug elsewhere had made it look broken — and re-enabling it made playouts about one and a half times faster end to end: honestly worth perhaps two or three percentage points of strength, not the six-fold speedup a microbenchmark implied. The tie-break described above shipped as well. And the trap suite became the metric of record for this behaviour.

The written findings ended with a recommendation: do not migrate to the other engine; start building one with a learned sense of position instead. The honest postscript is that the migration happened anyway — the next day. Not because the recommendation was wrong about cages, but because the same measurements had condemned the old engine on separate grounds: its position evaluator was, provably, never consulted, and a design organised around a formula that never runs is not worth defending. The replacement — a faithful port of that same open-source project — is the engine you play against today (how it scores a move). On the trap suite it fails five positions out of nine instead of nine out of nine, and at the decisive position from the recorded game it never plays the losing forward step, on any seed. Some of that improvement is real, some is the wall-move artifact flagged above, and five out of nine appears to be the ceiling for any engine of this family — the reference implementation scores the same.

Which leaves the conclusion the findings actually reached. The value signal itself has to change: an engine that learns what positions are worth from its own games, instead of assuming everyone races. That is the project's stated long-term direction — Grant Slatton, who solved small Quoridor boards exactly, described the game as one that "would be really amenable to an AlphaZero type approach" — and the trap suite is already its acceptance test. A learned engine that cannot beat five of nine will not have earned its keep.

Until then, the suite sits in the repository doing the one thing a metric built from your own losses is guaranteed to do: looking exactly where you were weakest. The win rate will stay green the whole time. That is precisely why it no longer gets the last word.