Can Jev three-crown its way to Ultimate Champion?

I got early access to Jev, TypeSafe AI's new System One model, and decided to play around with it.

Jev is a decision model. TypeSafe describes System One models as "built to make fast, structured decisions that software can use directly". It does not generate text.

I built clash-jev, a bot that uses Jev to make near real-time decisions on live game data.

How it works

• An Android tablet streams its screen to my laptop as H.264 video over adb.
• OpenCV reads the hand, the elixir and the towers from each frame. A small network that I trained on pictures I labelled by hand names the troops.
• Once a second the readings become one JSON game state.
• Jev answers three questions about that state. It picks a strategy, then a card, then a square.
• The bot checks that the play is physically possible and taps it on the tablet.
• The bot logs every decision next to a synced video. Any decision can be replayed and asked again on the replay site.

Nothing is trained to play. The bot uses no reinforcement learning and no simulator.

Part 1Architecture

The diagram shows every component, the boundary it runs in, and what crosses each boundary.

• The live loop is one Python process on my laptop, with the tablet on USB.
• Jev is the only external service in the loop. The policy calls it over HTTPS.
• Training runs offline and hands the runtime one ONNX file.
• The replay site is static. Its one function forwards a re-ask request to the TypeSafe API with the viewer's own key.
• The bot does not use an emulator. The game's anti-tamper check closes the game on emulators, and I did not try to get around that check.

Part 2Design decisions

I wanted to measure Jev's decisions and not my own. These are the decisions I made about how to ask, and why.

Part 3Capture

Capture gets the tablet's screen onto the laptop as frames the vision code can read. Every reader gets the newest frame, about 350 ms after the screen showed it.

• The tablet records its own screen with screenrecord and writes H.264 to standard output.
• adb exec-out carries the stream over USB.
• One thread decodes it with PyAV and keeps only the newest frame, as a NumPy array.
• The troop watcher, the match loop, the recorder and the page feed each take a copy.

adb exec-out screenrecord --output-format=h264 --size 720x1152 --bit-rate 8M --time-limit 180 -

Screen layout

Screen layout makes every reading independent of the device's resolution.

Part 4Reading the board

This stage turns a frame into readings. It reads the hand, the elixir and each tower's health, and it finds every troop with its owner, position and health. Colour thresholds and template matching do most of the work, and one small network names the troops.

The hand reader

The hand reader names the four cards in hand and the next card. It has to give the same answer when the game greys a card or washes it out.

• It shrinks the card art to 28×32 greyscale.
• It subtracts a blurred copy from each point and divides by the local spread. That removes broad light and dark areas and keeps the lines of the drawing.
• It compares the result with every known card by dot product. A match needs a score of 0.45 and a margin of 0.15 over the second-best card.
• A 14×16 brightness thumbnail is the fallback for a card that is enlarged while it is dragged.

Level badges

The badge finder locates every troop and reads its owner and health. It hands the troop network one crop per troop.

Part 5The troop model

The troop model names the card under each badge. I collected the pictures, labelled them by hand and trained the network myself.

Collecting and labelling

This step turns matches I play into labelled training pictures, with as few keystrokes as possible.

The network

The network maps a 96×96 crop to one of twelve cards.

Evaluation

The evaluation measures how often the network names a troop it has never seen.

The weak pair is Goblins and Spear Goblins, which differ by a spear a few pixels long. Bomber has six test crops.

Tracking across frames

Tracking turns single-frame reads into stable troops, so one wrong read or one hidden badge does not reach the state.

• A watcher thread reads the troops four times a second.
• It matches each badge to the nearest troop of the same owner from the previous read, within a tenth of the frame.
• Every read votes on the troop's name, weighted by confidence, so one wrong read loses the vote.
• A hidden troop keeps its name and position for 1.5 seconds.

Part 6The game state

This stage writes the readings down as the one JSON document Jev sees, once a second. Anything the vision code knows and does not write here does not exist for Jev.

{ "troop": "bomber", "class": "ranged support", "archetypes": ["splash", "ranged support"],
  "health": "low", "damage": "medium", "targets": "ground only", "flies": false,
  "lane": "left", "where": "my side", "tiles_from_my_tower": 10,
  "seen_for_seconds": 4.2, "health_remaining": 1.0 }

Part 7Asking Jev

This stage gets one play decision out of Jev. A decision takes up to three requests, because the card question depends on the strategy and the square question depends on the card.

"defend_left": {
  "what":    "Enemy units in the left lane are close to your left princess tower, or to
              your king tower once that princess tower has fallen: on your side of the
              river or about to cross it, walking toward the tower or already hitting
              it. Play a card against them, in the left lane.",
  "not_for": "When no enemy unit in the left lane is close to your tower
              (`tiles_from_my_tower` is the distance of each): units that are still far
              away on the enemy's side are not attacking it yet. Also when you mean to
              draw the attackers to the middle of your side: that is defend_centre."
}

Jev matches the wording of each option against the state, so the wording decides a great deal. I found each choice below by replaying logged moments under a changed wording and comparing the probabilities.

When to ask

The match loop decides when a question is worth a request.

• Jev is asked about every snapshot, one a second.
• After save_elixir or hold_elixir_for_threat the next question waits until elixir has gone up by one. A new enemy troop or damage to one of Jev's towers ends that wait at once.
• After a play the next question waits until the screen shows the play, because a stale frame would get the same card played twice.
• The bot does not wait for a card to become affordable before it asks. That would time every question to the cheapest card.
• A three-minute match comes to roughly 200 requests.

Part 8Acting

Acting turns Jev's pick into two taps on the tablet, or into a log line when the pick cannot be done.

Part 9Record, replay, re-ask

This stage records a match so that every decision can be inspected later, and publishes it without exposing the opponent.

Asking Jev again

Re-ask Jev sends a recorded moment to Jev again with an edited question, to show what a change of wording does.

• Pause the replay site on any decision and press Re-ask Jev.
• The request sent at that moment opens in three parts. The state and the briefing are locked. The question can be edited as a form or as JSON.
• Paste your own TypeSafe API key and press the button. The new probabilities appear next to the recorded ones.

Part 10Technical limits

• The troop classifier is 84% accurate. It confuses Goblins with Spear Goblins most often, and Bomber has only six test crops. The tracker's votes hide some wrong reads, not all of them.
• It knows twelve troops and nine hand cards. Every new card needs collected crops, hand labels and a retrain.
• Your own troops are invisible until they are hurt. The game draws a badge on your troop only once it takes damage. Finding troops without badges needs a detector trained on full frames.
• Touching badges merge. In a tight crowd two badges become one shape and one troop goes missing.
• Tower health is sometimes wrong. In one recorded win an enemy tower showed 501 of 1750 while the state said 0.97, because the reader counted the drained part of the bar as filled. The bot does not read the digits yet.
• The state is old by the time the card lands. A frame is about 350 ms old when it is read, and three requests and a tap add roughly 560 ms. A fast troop moves several tiles in that time.
• The opponent's elixir is an estimate. It pays for each enemy troop the bot sees, so it drifts whenever a troop is missed or misnamed.
• Each screen shape needs calibrating. The two affine transforms are measured once per aspect ratio, by hand.
• The bot does not detect the start or the end of a match. I press Start and Stop, and the results screen can read as a hand for a few seconds.

So can it three-crown its way to Ultimate Champion? Not yet. It currently has a 98% win rate and sits at 400 trophies. The recorded matches are on the replay site, each decision with its state, its questions and every probability.