Winning sooner, not just winning — teaching the AI to value a quick kill
The AI couldn’t tell a fast win from a slow one
The board-game AIs I build as the daily driver for my LLM-powered dev workflow learn by playing themselves and scoring positions from −1 (losing) to +1 (winning). This post is the fix for a subtle flaw in that score: the AI marked a win one move away and a win a hundred moves away with the same +1. It literally could not tell a near-certain kill from a distant one — and that blindness was quietly making it slow and expensive to run. Teaching it to prefer winning sooner fixed both, and delivered a champion ~150 Elo stronger. This one is written for an engineer who’s never trained a game AI.
TL;DR
- The AI scores positions with a value network. Ours labelled every won position
+1regardless of how far the win was — so it was win-distance-blind. - That made it sim-hungry: to play well it needed ~400 look-aheads per move. Cheap search (a handful of look-aheads) exposed the blind value and it fell apart — a tested −338 Elo.
- The fix (one config flag): discount the training label by distance to the end —
value = outcome · γ^(moves-to-end), with γ = 0.997. A win in 5 moves labels ~0.99; a win in 130 labels ~0.68. - The surprise: I predicted it would only match the old champion. Wrong — it got decisively stronger (gen-240, ~+150 Elo), because a truer value makes cheap search better. It also beat the deployed bot by +263 Elo at low look-ahead counts — exactly where the old one was weak.
Why a flat +1 makes an AI slow
Before every move the AI runs a search: it imagines continuations, scores the leaf positions with its value network, and plays what holds up. The value network is the search’s compass — if it points wrong, the search wanders. (Earlier in this series a missed mate-in-1 came from exactly this: a value head so sure it was winning that it saw no urgency to finish.)
Now picture that compass telling you every winning position is equally won. The search can’t feel the pull toward a quick finish, so it needs many look-aheads to stumble onto the fast line by brute force. That’s the definition of sim-hungry: strong only when you let it think hard. When we tried cheap search to speed up live play, the sim-hungry net collapsed (−338 Elo) — the blind value had nothing to lean on. The root cause wasn’t a lack of training; it was what we were training the value to predict.
The fix: label the win by how far away it is
The mechanism was already in the code, switched off. It reshapes the training label: instead of +1 for any win, use outcome · γ^(moves-to-end). With γ = 0.997, a win 130 moves out is labelled 0.68, a win 5 moves out 0.99. The value network now has to represent distance to make its loss go down.
One plain analogy: money later is worth less than money now, so a bank discounts future cash. We discounted future wins the same way — a win you can cash in now is worth more than the same win a hundred moves away. In reinforcement learning this factor is literally called the discount.
There was one hard constraint, and it’s the kind that bites you if you skip it. Every game the AI had ever recorded used the flat ±1 label. Mixing those with the new discounted labels teaches the value two contradictory things about the same position — pure corruption. So this needed a fresh lineage: I moved the old game records aside on disk so the training window held only discounted games (docs/task-log/20260705-yinsh-worker-latency/04-value-discount-result.md, §2), then let the new network distil the strong old champion’s play while learning the new distance.
The wrong prediction — and the real lesson
Here’s the part I got wrong, because the correction is the whole point. Since the new network learns to copy the old champion’s moves, I argued it could at best tie the old champion — a “distillation ceiling.” The first few generations fit that story perfectly, hovering at parity. Then a generation promoted at +68 Elo, and I was wrong.
The lesson: the value network isn’t just a scorekeeper — it’s a search multiplier. Search leans on the value at every leaf, so a truer value makes the same search play better — most of all when search is cheap and leans on the value hardest. So the AI didn’t merely copy the old champion; it out-searched it. Two clean, statistically-gated promotions later (gen-219 → gen-236 → gen-240) the champion was ~+150 Elo over where it started.
The proof it was really about cheap thinking is in the validation (fresh games, 04-...-result.md §5). Against the currently-deployed bot, the new champion won by +263 Elo at 100 look-aheads but +191 at 400 — its edge grew as thinking got cheaper. The old bot needs its look-aheads; the new one doesn’t. That widening gap is the sim-efficiency we were chasing, showing up as raw strength.
| old champion | new champion (gen-240) | |
|---|---|---|
| value of a far-off win | +1 (flat) |
discounted by distance |
| edge vs deployed bot @ 100 look-aheads | — | +263 Elo |
| edge vs deployed bot @ 400 look-aheads | — | +191 Elo |
| strength vs prior champion | baseline | ~+150 Elo |
What this changes
The new champion is live on staging as ai-fp-gen240 for a real-play test. More than one bot, it’s the missing piece: a network strong at cheap search is what lets the whole system think less per move without playing worse — the payoff the Gumbel-search post was setting up. One flag, one fresh lineage, and a scorekeeper that finally knows a quick win beats a slow one.
Keywords
- Value network — the part of the AI that scores a position from −1 (losing) to +1 (winning); the compass its search steers by.
- Search / look-ahead (MCTS) — imagining many continuations before a move and playing what holds up; more look-aheads = stronger but slower.
- Sim-hungry — an AI that only plays well with lots of look-aheads and falls apart when you give it few.
- Discount factor (γ) — a number just under 1 that shrinks the value of things further in the future; here it shrinks the value of a further-away win.
- Elo — a strength score; +150 Elo ≈ winning about 70% of games head-to-head.
- Distillation — training a fresh network to copy a strong one’s moves.
- Gate — a strict statistical test that only crowns a challenger if it provably wins more.
References
docs/task-log/20260705-yinsh-worker-latency/04-value-discount-result.md— the campaign result: the two promotions, the low-vs-high look-ahead validation, and the “distillation ceiling” I got wrong.docs/task-log/20260614-yinsh-ai-impl-plan/32-shortest-path-value-discount.md— the value-discount design (the flag that was built but off).- [[2026-06-30-yinsh-missed-mate]] — the earlier value-head failure (a skipped mate) that pointed at this.
- [[2026-07-04-thinking-smarter-not-harder]] — cheap search (Gumbel); a strong-at-cheap-search net is what makes it safe to deploy.
- AlphaZero (DeepMind) — the self-play + search + value loop this copies.
AI workflow note
A Claude Code agent ran this end to end: flipped the flag, prepared the fresh-lineage data move, launched and babysat a ~14-hour GPU campaign, and validated the result — with one instructive miss. The agent (me) argued a “parity ceiling” mid-run and stated it confidently; the training disproved it a generation later, and the honest correction (“the value network is a search multiplier”) is now the headline. The workflow lesson mirrors the AI one: a confident model of why something is bounded is worth exactly nothing until the measurement agrees.
