Why We Replaced the Engine

In July the program that plays every bot on the ladder was deleted and replaced. Not refactored, not put behind a flag — twenty modules and 5,826 lines of the old engine came out, and 3,117 lines of a different design went in. The launch changelog covered this in a few paragraphs; this is the long version, with the numbers, the two bugs the migration turned up, and the first strength test we had to throw out.

The engine we deleted

The old engine was a Monte Carlo tree search with a great deal of machinery on top. At its core it judged candidate moves by playing simulated games forward from them — rollouts — and it carried a static evaluator for positions the simulations did not finish: a three-term linear function of the difference in path lengths, the difference in walls remaining, and an urgency term. Above the search sat a plan layer, which cached up to three candidate plans and had rules for committing to one and margins for abandoning it; an opponent model that tried to predict the human's next move; and a search tree that persisted between moves and kept growing during the opponent's thinking time. Around all of that, patches had accumulated: an anti-dithering tie-break, a trap-aware rollout mode, a veto on backtracking moves.

Every piece of that had a plausible justification. The whole was still wrong, and the way we found out is the most useful part of this story.

The measurement that condemned it

We instrumented the value the search reported at its root, across 36 searches at every difficulty tier. It was exactly a whole number every single time. That one observation unravels the whole design: it means every simulated game was running to an actual finish, so the depth-cap fallback — the only place the linear evaluator was ever consulted — never fired. Not rarely. Never. The evaluator we had tuned and patched was dead code in practice.

The thing actually judging every position was the rollout policy: the cheap, mostly random move-picker that plays the simulations out. And that policy raced. In its playouts, neither side ever built a wall cage, so the engine's entire model of the game was a footrace. An engine whose imagined futures contain no traps evaluates a trap as a perfectly good place to stand.

It did exactly that. We had a recorded game in which a human beat Nemesis by caging it. Replaying the position at the ply where the game was actually lost, the old engine chose f4 — the exact losing move the recorded game contains — on five out of five random seeds, with only one move inside its near-best margin. It did not stumble into the trap for lack of attention. It preferred it.

The fixes that failed first

Replacement was not the first instinct. Three targeted repairs were tried against those trap positions before anything was deleted: rollouts that were taught to sometimes build blocking walls, a re-tuning of the rollout policy, and a cage-aware tie-break at the move-selection step. Each was measured, and each failed. The conclusion written into the migration record is one sentence: random playouts cannot represent a plan. The signal was broken, not the thresholds, and patching a broken signal had already produced the pile of patches we were standing on.

So the decision was a strict replacement — no hybrid, no compensating layers carried over, the old code deleted rather than disabled, so nothing could quietly lean on it again.

What replaced it

The new engine is a faithful port of the open-source gorisanson/quoridor-ai project by Kyutae Lee (MIT licensed). The counterintuitive part is that it doubles down on the very thing that had failed: it has no static evaluator at all. The value of a position is nothing more than the record of simulated games played through it, each one run all the way to the end of the game and scored 1 for a win and 0 for a loss. The rollout policy is no longer accidentally the evaluator — it is the evaluator, on purpose, and the design makes that honest rather than broken:

Four differences carry most of the weight. First, the playouts contain real blocking: the moves the search considers include walls placed to interfere with the opponent's path, filtered so they can never illegally seal a pawn in, so the simulated futures now have cages in them. Second, the search is far more exploitative — its exploration constant sits inside the square root of the selection formula and is small, so instead of spreading effort across alternatives it commits hard to lines that are winning playouts. Third, the final move is the robust child: the move the search visited most, not the one with the prettiest score. Fourth, everything the old engine layered on top is simply gone. No plan cache, no opponent model, no persistent tree, no pondering; the tree is rebuilt from scratch every move.

Difficulty across the ladder is now nothing but simulation count — from 2,500 simulated games per move at the low end to 60,000 at the top — and the gentlest tiers are softened further by a wrapper outside the search that corrupts moves after the fact, never by knobs inside it. Scout's page explains what that handicap looks like from across the board.

What survived the deletion is the code that never depended on the bad signal: the opening book, the exact endgame solver that takes over when the walls are gone and the race becomes arithmetic, the time-budget ladder for blitz, the trap test suite — and one frozen copy of the old engine, kept for the sole purpose of being measured against.

What was measured before it shipped

The swap was gated on measurements. The highlights:

• The trap suite. Nine positions from real games in which a human caged the strongest bots. The old engine was trappable in nine of nine. The new one is trappable in five of nine — which is exactly what the reference implementation scores on the same suite.
• The Nemesis ply. The new engine never plays f4 there, on any seed. It plays a defensive wall instead, every time.
• Head-to-head at equal wall-clock, against the frozen old engine: at the advanced tier the new engine won 75% of one match set and 100% of the other; at the extreme tier it won all twelve games.
• Speed. Roughly 1,700 playouts per second became 11,000–15,000 simulations per second, and the top tier's average think time dropped to about 3.7 seconds under the same 8-second cap.
• The ladder still works at both ends. Tier strength came out monotone — after a fix to the per-tier simulation budgets — and the beginner tier stayed beatable: a deliberately weak test opponent that does nothing but race beat it 14 games to 6.
• The opening book earned its place. The candidate books came out within noise of each other, and the canned-replies arm was rejected because it measurably increased exposure to cages, so we shipped our own.

The result we threw out

The first head-to-head run said the new engine lost to the old one — winning only 12–28% of games. Briefly, the whole project looked like a mistake. The run was wrong, and the way it was wrong is worth writing down: it gave both engines an equal number of iterations, 600 each. But 600 simulations cost the new engine about 46 milliseconds, while the old engine spent around 350 milliseconds plus its entire plan stack on the same allowance — and in production the new engine runs on the order of 50,000 simulations a move. Comparing engines at equal iteration counts compares nothing; equal wall-clock time is the only fair fight. The run is preserved in the gate record, marked invalid-as-run, as a warning to whoever benchmarks the next engine.

Two bugs the port surfaced

Porting someone else's engine faithfully means checking your own assumptions against theirs, and two of ours failed the check.

The first was a false accusation in a comment. Our wall-legality code carried a note declaring a cheap contact-point shortcut "proven unsound," forcing a full path search for every candidate wall. The shortcut was never unsound — the code that computed wall junction points had put horizontal and vertical walls in two different coordinate frames, so crossing walls never appeared to touch. Fixed, the shortcut disagreed with the full search zero times in 57,855 checks and now skips the expensive path search for 91% of walls.

The second was in the port itself: the original filters its interference walls so a candidate can never wall a pawn off entirely, and our first pass used the unfiltered list. Real games crashed a defensive guard within two games of play. That one was caught fast precisely because the guard existed — an argument for guards.

The ceiling, honestly

Five of the nine trap positions are still trappable, and that is not a bug backlog — the reference implementation scores the same. It is the ceiling of any engine that judges positions by random playouts: a random playout can stumble into a cage, but it cannot plan one, and it cannot plan its way out of one. Getting past that ceiling requires a value signal that can represent a plan — a model that has learned to judge positions rather than sample them. The trap suite already exists as the acceptance test for whatever attempts that.

In the meantime, the practical reading is on the pages that own it: what the top of the ladder still cannot see is on Nemesis's page, and the structures that exploit it are in traps and cages. The honest summary of the swap is this: we replaced an engine that was accidentally judged by its playouts with one that is deliberately judged by them, measured the difference at every rung, and shipped it knowing exactly where it still loses.