Two dead-end runs proved the YINSH AI was too small, not undertrained — Wintersalmon

The thing I kept tuning was never the bottleneck

YINSH is one of the board games in this workspace, and its AI opponent — a champion called gen-077 — had not improved in weeks. I spent three 12-hour GPU runs trying to beat it: first by feeding each training round more data, then by letting rounds build on each other. Both stalled just short. The fix turned out to be the one variable I had left fixed — the size of the neural network. Doubling it beat the champion on the first promotion, from scratch, in two generations.

TL;DR

Run 1 (more data per round): got closer — from −143 to −85 Elo vs the champion — then capped. Never won. (PRs #1278/#1281, doc 19.)
Run 2 (rounds build on each other): no better, −115 Elo, no upward climb. (PRs #1282/#1283, doc 21.)
The clue: the AI kept getting better against a random player but not against the champion — the signature of a network too small to represent stronger play.
Run 3 (network 2× bigger, 128 channels × 10 blocks, ~3.36M weights): beat the champion by +170 Elo, promoted twice, then plateaued. New champion gen-202. (PRs #1284/#1285/#1286, doc 23.)
A from-scratch network reached the old champion’s level in one training round — clear proof the old size, not the training, was the wall.

How the AI learns to play, in one loop

The training is an AlphaZero-style loop, and it helps to picture it as three steps repeating. Self-play: the current champion plays hundreds of games against itself, using tree search to pick strong moves. Train: a fresh neural network learns to imitate those searched moves — it is being taught the champion’s best instincts. Gate: the new network plays a match against the champion, and a strict statistical referee decides if it is genuinely stronger.

That referee matters more than it sounds. It is an SPRT gate — it keeps playing games until it is confident, then re-tests any winner on a fresh batch of 800 games and only crowns a new champion if the win is real, not luck. Across every run in this story the gate was never touched, so a “promotion” always means a true, measured improvement.

Two dead-ends: more data and momentum both stalled

My first hypothesis was the obvious one: the network was undertrained. So Run 1 doubled the self-play games (256 → 512), the training passes (3 → 5), and the memory of past rounds (a 6 → 8 round window). It worked — partly. The new networks climbed from −143 Elo up to −85 against gen-077, the best the old size ever managed. But −85 is still negative: closer, never ahead. (Elo is a relative score; below zero means weaker than the champion.)

Run 2 tried momentum — let each round start from the previous round’s network instead of resetting to the champion, so improvements could stack. It produced an almost identical curve, peaking at −115. The lesson surprised me: because every round trained on the same champion’s games, the starting point washed out after a few training passes. Same teacher, same ceiling.

The clue was which opponent it was beating

The honest signal was hiding in two numbers. Across both runs the network’s win rate against a random player kept rising (0.67 → 0.73), while its win rate against the champion stayed flat. A network that gets generally more competent but cannot pull ahead of a specific strong opponent is telling you something precise: it has enough capacity to represent champion-level play, but not enough to represent anything better. That is a capacity ceiling, not a training problem. The two failed runs were the evidence.

A bigger network won in two generations — from scratch

Run 3 changed one thing: the network went from 96 channels × 8 blocks (1.73M weights) to 128 × 10 (~3.36M), about twice the size. I expected a slow start, because a bigger net cannot inherit the old champion’s weights (the shapes do not match), so it began from random. I framed it as a “smoke test” to prove the bigger-network pipeline even ran.

It over-delivered. The first generation, from random, already matched the old champion’s best. The second generation beat gen-077 by +113 Elo in the gate and +170 on the fresh-seed re-test — a 65.8% win rate. That was the first champion change in the whole effort. A third generation promoted again (+43), a fourth came in dead-even, and the climb plateaued — so I stopped early rather than burn GPU on tied matches. New champion: gen-202, image 2026.06.28.030243-86a60ce.

The mistakes that cost the most time

Three of them, all worth a junior engineer’s attention. First, a wrong assumption about saved state: the training data on disk had drifted from what I expected, and launching blind would have trained on stale, mixed-quality games — I caught it by reading the actual files before launch, not trusting the handoff note. Second, the launch script had --warm-start hardcoded, so a different-sized network would have crash-loaded the old champion’s weights; the fix was a one-flag gate (PR #1284). Third, my own monitoring script cried “promotion!” on every later generation because it checked “did any generation ever promote” instead of “did this one” — a reminder that your dashboards lie in their own small ways. None of these were the science; all of them ate real time.

What this changes going forward

The headline is small and humbling: for weeks I tuned data, epochs, and momentum, and the answer was make the model bigger. YINSH now has a genuinely stronger champion for the first time since gen-077, and the diagnosis — read which opponent the metric moves against — is the part I will reuse. Next is pushing capacity further (an even bigger net, or seeding it from gen-202) and finally shipping gen-202 to players, who are still facing the old brain.

Keywords

Neural network — a program that learns patterns from examples instead of being hand-coded; here it predicts good YINSH moves and who is winning.
Channels / blocks — two dials for a network’s size; more of either means more capacity to represent complex play (and more compute to train).
Self-play — the AI plays games against itself to generate its own training examples, with no human games needed.
Elo — a relative skill score; here it is measured against the champion, so a negative number means weaker, positive means stronger.
SPRT gate — a statistical referee that plays only as many games as needed to be confident a challenger is truly stronger before crowning it.
From scratch — starting training from random weights rather than inheriting an existing model.
Capacity ceiling — the point where a model is too small to represent any better solution, no matter how much it trains.

References

docs/task-log/20260614-yinsh-ai-impl-plan/19-capacity-run-result.md — Run 1, the “more data” attempt that capped at −85.
docs/task-log/20260614-yinsh-ai-impl-plan/21-compounding-run-result.md — Run 2, the momentum attempt that matched it without climbing.
docs/task-log/20260614-yinsh-ai-impl-plan/23-bigger-net-128x10-result.md — Run 3, the bigger network and the two promotions.
#1284 — the one-flag fix that let a different-sized network train from scratch without crashing.
#1285 / #1286 — launch and stop of the winning 128×10 run.
AlphaZero (DeepMind) — the self-play + search + train loop this pipeline copies.

AI workflow note

This whole campaign was run by a Claude Code agent, not by hand: it edited the GitOps config, opened and merged the launch/stop PRs, and watched the 12-hour runs through a loopback network relay because the laptop’s Wi-Fi blocks the cluster directly. The pattern that worked was treating every run as a single experiment with a written hypothesis and a pre-committed success bar (promote, partial, or null), so the agent could read each result and decide the next move. The pattern that failed was trusting a handoff note about on-disk state — the agent only caught the drift by inspecting the files itself, which is now a standing pre-launch step.