Thinking smarter, not harder — a faster self-play loop that can't get less creative
The AI’s practice is the slow part, and it’s doing it the brute-force way
The board-game AIs I train learn by playing millions of games against themselves, and I’ve been chipping away at making that loop faster without making the player worse. Two earlier fixes were “bigger brain” and “think harder” (deeper search). This post is about the next one — think smarter, not harder — plus two supporting changes, and the single rule that governs all three: the AI is allowed to get faster, but not less creative. These are planned, not yet shipped; this is the design and the reasoning, written for an engineer who’s never touched an AI training pipeline.
TL;DR
- The “practice” phase (the AI playing itself to make its own lessons) is ~65% of each training round and is the bottleneck — measured at ~78 minutes of a ~120-minute round.
- Change 1 — measure everything first: the loop currently logs almost nothing per phase, and nothing about creativity. Add per-phase timing plus a “creativity meter” so we can prove the other changes help and catch trouble early.
- Change 2 — move the referee off the critical path: the promotion “gate” runs hundreds of games on the GPU machine while everything else waits. Run it on an idle 16-core PC instead, so the GPU starts the next batch immediately.
- Change 3 — think smarter, not harder: replace ~800 brute-force look-aheads per move with a technique (Gumbel search) that gets an equal-or-better lesson from ~50 — roughly 16× fewer GPU computations on the slow phase.
- The rule: every change is judged against an untouched strength referee and the new creativity meter. Faster is only worth it if creativity stays flat or rises.
How the loop works today, and why practice is the bottleneck
One “generation” of training is three phases in a line: the AI plays games against itself (practice), a fresh network is trained to copy the best moves from those games, and a strict statistical referee decides whether the new network is genuinely stronger. Only the practice output becomes lessons, so the quality of practice sets the ceiling on everything.
The catch is how the AI picks each practice move. It doesn’t trust its instinct; before every move it runs a search — it imagines many continuations, looks ahead, and plays what holds up best. Today that’s about 800 look-aheads per move. Each look-ahead is one computation on the GPU (the chip that does the math), so 800-deep search on hundreds of games is a firehose of tiny GPU jobs. That’s the whole reason practice takes ~78 minutes and the GPU is the limiting resource. Training itself is trivial by comparison — about 15 minutes. So the target is clear: make practice cheaper without making the lessons worse.
Change 1 — You can’t improve what you can’t see
Right now the loop records two numbers per round: total time and practice time. It records nothing about how varied the AI’s play is. That’s dangerous, because the classic failure of self-play is quietly collapsing into one repetitive style — and today that collapse would be invisible.
So the first change is pure instrumentation: log the time for each phase separately, and compute a creativity meter every round. The subtle part — which an early review caught — is what to measure: not just how varied the openings look, but how many genuinely different games and strategic lines the AI actually produces. A naive “spread” number can stay high while every game quietly funnels through the same three midgame plans, so the meter counts distinct positions and lines, not only their spread. This is the software-engineering reflex of adding tracing before you optimize a service: cheap, and it turns every later claim (“this was faster,” “this stayed creative”) into a number instead of a hope.
| Today | Planned | |
|---|---|---|
| Timing | total + practice only | every phase (practice / train / referee) |
| Creativity | not measured | distinct-games-and-lines meter, logged each round |
Change 2 — Move the referee onto the idle machine
After training, the referee — the “gate” — plays hundreds of full games between the challenger and the current champion to decide promotion. It runs on the GPU machine, and while it runs, everything else waits. But those games barely use the GPU; the machine is mostly idle during the very phase that blocks the next round.
Meanwhile a separate 16-core PC (the always-on server that runs the cluster) sits unused. The two players in the gate are frozen — their networks don’t change during the match — which makes the gate a clean, self-contained thing to relocate. So the plan is to run the gate’s game-playing on that idle PC, freeing the GPU machine to start the next batch of practice immediately. It’s the same instinct as moving your test suite onto a separate CI runner instead of blocking your dev laptop: two things that don’t depend on each other should run at the same time.
Change 3 — 800 brute-force look-aheads, or 50 clever ones
Here’s the big one. Why does the AI need ~800 look-aheads to produce one good lesson? Because the standard method turns each move into a lesson by counting how often search visited each candidate — and that count is only trustworthy after many visits. With few look-aheads, most candidate moves get visited once or not at all, and the “lesson” is mostly noise. Brute force buys quality.
Gumbel search (from a 2022 DeepMind paper) gets the same quality from far fewer look-aheads by spending them cleverly and building the lesson differently. Two ideas do the work. First, instead of spreading attention thinly, it picks a handful of promising candidate moves and runs a mini single-elimination tournament among them — give each a few look-aheads, drop the worst half, repeat, until one winner remains (the technique is called sequential halving). Second, and more important, it builds the lesson from the AI’s value estimates of moves rather than raw visit counts, with a math guarantee that the resulting lesson is a genuine improvement over the AI’s instinct even at a tiny number of look-aheads. That guarantee is exactly what standard search lacks, and it’s what lets us drop from ~800 to ~50.
| Today (standard search) | Planned (Gumbel search) | |
|---|---|---|
| Look-aheads per move | ~800 | ~50 |
| Lesson built from | how often each move was visited | value estimates + a mini-tournament |
| Quality at low counts | unreliable (noise) | guaranteed improvement |
| GPU jobs per move | ~800 | ~50 (~16× fewer) |
A worked example: on a ~60-move opening, standard search fans 800 look-aheads across all 60. Gumbel picks the 16 most promising, runs 16 → 8 → 4 → 2 → 1 over ~50 look-aheads, plays the winner, and records a lesson that provably beats the AI’s snap judgment. Same move quality, a fraction of the GPU work. It only touches practice; the referee keeps its rigorous full search, so the yardstick never moves.
The one rule: it can’t get less creative
Gumbel changes how the AI explores — it swaps the old randomness for the tournament’s built-in variety — so the honest risk is that it narrows the AI’s style. That’s why Change 1 comes first. The bar for shipping any of this is explicit: measured against the untouched strength referee, the new champion must be no weaker, and against the creativity meter, variety must be flat or higher — tracked across many generations, not judged on a single round. That last part matters because the classic way these systems go stale is a slow drift that’s invisible in any one round but obvious over twenty. Faster-but-more-repetitive is a reject, not a trade-off I’m willing to make. Even-or-better creativity is the whole point of measuring it.
What this changes
Three plateaus, three lessons: first a bigger network, then deeper search, now cheaper search. If this lands, a training round gets meaningfully shorter without the AI losing a step — and, just as importantly, I’ll be able to prove it didn’t lose its creativity, because for the first time the loop will be watching for exactly that. The next post will have the numbers.
Keywords
- Self-play — the AI plays games against itself to generate its own training examples, with no human games needed.
- Search / look-ahead (MCTS) — before each move the AI imagines many continuations and plays what looks best; more look-aheads mean stronger but slower play.
- GPU — the chip that does the heavy math; keeping it busy (or not overloading it) is most of the performance battle.
- Gate (referee) — a strict statistical test that only crowns a challenger as champion if it provably wins more games.
- Gumbel search — a way to get a good training lesson from very few look-aheads, using value estimates plus a mini-tournament, with a guarantee of improvement even at low counts.
- Sequential halving — the mini-tournament rule: give candidates equal tries, drop the worst half, repeat until one remains.
- Entropy (as a creativity meter) — a number for how varied a set of choices is; high opening entropy means the AI isn’t always playing the same lines.
- Pipeline overlap — running independent phases at the same time on different machines instead of one after another.
References
docs/task-log/20260703-yinsh-ai-session-improve/01-improvement-plan.md— the full plan these three changes come from, with code anchors.docs/task-log/20260614-yinsh-ai-impl-plan/30-followup-designs.md— the measured phase breakdown (~78m practice / ~15m train / ~13–26m referee) and the deferred speed backlog.docs/task-log/20260630-yinsh-missed-mate-diagnosis/01-diagnosis.md— why “more data” doesn’t fix every weakness; the separation of strength from other behaviors.- [[2026-06-29-thinking-harder-not-bigger]] — the previous fix (deeper search), and the half-idle-GPU trap this loop keeps running into.
- Policy improvement by planning with Gumbel (Danihelka et al., 2022) — the paper behind Change 3.
- mctx (DeepMind) — a reference implementation of Gumbel search to cross-check the math against.
- AlphaZero (DeepMind) — the self-play + search + train loop this pipeline copies.
AI workflow note
This is a design post, not a results post — it documents a plan before implementation, which departs from the guide’s “cite real data” bar in one way: the three changes are proposed, while the numbers I cite (phase timings, sim counts, the current champion) are all measured. A Claude Code agent did the research across the pipeline’s code and a dozen task-log docs, then drafted the plan and this explainer; the pattern that worked was forcing an explicit “what must not regress” bar (creativity) before any code, so the review has a yardstick. The next step is an independent review of the plan against that exact bar.
