What this is

TokenTown is an isometric city where every district is one stage of a transformer language model. A convoy carries a hidden state along the roads: it is cut into tokens at the docks, cast into a vector at the foundry, stamped with its position, then driven around the layer ring (attention, residual, feed-forward, residual) once per layer, before the stadium turns it into a probability distribution and the sampler picks one token. That token drives back up the feedback highway and the whole city runs again.

How much of it is real

Genuinely computed, live, in your browser: the tokenizer split; the embedding lookup; sinusoidal positional encoding; LayerNorm; multi-head scaled dot-product attention with causal masking over a real growing KV cache; the residual adds; a GELU feed-forward; and temperature / top-p sampling. The bars on the truck are the actual vector. The beams over the warehouse are the actual softmax weights. Prefill really does process every prompt token at once while decode really does process only one.

Scaled down: 12 dimensions instead of thousands, 2 attention heads instead of dozens, 2–12 layers instead of 80, and a vocabulary of a few hundred words instead of 100k+.

Deliberately faked: the weights are random, and nothing here was trained, so a random-weight model would emit noise. To keep the output readable, the final logits blend the real hidden-state projection with a bigram prior built from a small fixed corpus. The attention scores are also sharpened, and given a small first-token ("sink") and recency bias, so the map looks like the patterns trained models actually produce. Treat the text this city writes as scenery; treat the mechanism as the lesson.

Pacing

The first time the convoy reaches a district it stops long enough to read that district's explanation, between 9 and 26 seconds depending on how much there is to say. A progress bar under the panel text shows how long the stop has left. Once every district has been explained the city runs at a watchable pace instead of a readable one, and the repeated layers fast-forward because they are the same road with different weights. Space holds any stop indefinitely, S steps one stage at a time, and the Speed slider scales everything, including the reading stops, from 0.4× to 8×. Reset (⟲) replays the slow tour from the beginning; Run keeps what you have already read.

Controls

• Space play / pause · S advance one stage · R reset and replay the tour · F follow camera · L labels
• Drag to pan, scroll to zoom, click any district for its explanation.
• The view starts close on the convoy and rides along with it. The ⤢ button on the left (or a double-click on the map) pulls back to the whole city; turning off Follow lets you wander on your own.

Inspired by the idea behind PGSimCity (a city-shaped model of PostgreSQL); all code, art and copy here are original.