Challenge: Build the smallest transformer that can add two 10-digit numbers with >= 99% accuracy on a held-out 10K test set.

This started with Addition Under Pressure, where I gave Claude Code and Codex the same prompt: train the smallest possible transformer that can do 10-digit addition with at least 99% accuracy. Claude Code came back with 6,080 parameters and Codex came back with 1,644. The community has since pushed this dramatically lower.

Maintained by Dimitris Papailiopoulos (@dimitrispapail).

We track two categories:

• Trained — weights learned from data by any training algorithm (SGD, Adam, evolutionary search, etc.). The algorithm must be generic — it should work with any model and dataset, not just this specific problem. This encourages creative ideas around data format, tokenization, curriculum learning, and architecture search.
• Hand-coded — weights set analytically. This is a constructive proof that the architecture can represent addition, regardless of whether SGD would find it.

Both are valid. Both are interesting.

* Passed 8,192 random tests; not independently verified on our 10K test suite yet.

The model must operate as a genuine autoregressive transformer. This means:

1. Self-attention is required. The model must contain at least one self-attention layer. This is the defining feature of a transformer — without it, you have an MLP or RNN, not a transformer.

2. The model must be autoregressive. It receives a token sequence as input and predicts the next token. Output digits are generated one at a time, with each new token fed back as input for predicting the next. The carry propagation must emerge from this autoregressive process — not from explicit state variables passed between steps in Python.

3. Standard forward pass. The model's forward() method must be a standard tensor-in, logits-out computation. No problem-specific control flow (for-loops over digits, explicit carry variables, string manipulation) inside forward(). The autoregressive generation loop lives outside the model, exactly as it would for any language model.

4. The model does the work, not the code. The inference code should be generic autoregressive decoding that would work with any transformer checkpoint. If your generation loop contains addition-specific logic — manually pairing digits, threading carry state, indexing into specific positions — then the Python code is solving the problem, not the model.

In short: if you can swap in a different set of weights and use the exact same inference code for a different task, your setup is legitimate. If the inference code is inseparable from the algorithm, it's not.

• Architectural variations: rank-1/low-rank projections, factorized embeddings, custom positional encodings, alternative norms
• Hand-coded weights (constructive proofs are valid — they show the architecture can represent addition)
• Trained weights via any generic learning algorithm (shows the solution is learnable — encourages creative ideas on data format, tokenization, and curriculum)
• Input formatting choices (reversed digits, delimiters, etc.) as long as the format is fixed and doesn't encode the answer

• Must achieve >= 99% accuracy on 10,000 random test pairs (held-out, fixed seed)
• Inputs: two integers in [0, 9,999,999,999]
• Output: their sum as an integer
• Verified using verify.py with --seed 2025

• Count unique parameters (after weight tying/deduplication)
• Fixed/sinusoidal positional encodings are not counted (following the original Transformer paper convention)
• Learned positional encodings are counted

Option A: Open an Issue (easiest)

1. Click New Issue and fill in the template
2. Include a link to your code (GitHub repo, gist, etc.)
3. Include test results (accuracy on random pairs)
4. We'll verify and add you to the leaderboard

Option B: Open a Pull Request

1. Fork this repo
2. Update the leaderboard in README.md with your entry
3. Include verification results
4. We'll review and merge

Updates to the leaderboard are welcome via pull request.

python verify.py submissions/your_submission.py

This runs:

• 10 edge cases (boundary values, max carry chains)
• 10,000 random pairs (seed=2025)
• Reports accuracy, pass/fail, and timing

This challenge explores a fundamental question: what is the minimal transformer that can represent integer addition?

Addition requires three capabilities:

1. Digit alignment — pairing corresponding digits from two numbers
2. Per-digit arithmetic — computing sum and carry for each pair
3. Carry propagation — threading carry information across positions

Transformers solve these using attention (for alignment), MLPs (for arithmetic), and autoregressive generation (for carry propagation). The question is how small the architecture can be while still implementing all three.

• Parameter cliff at ~800: Sharp accuracy transition observed by multiple researchers
• Single layers beat two layers at equivalent parameter budgets (for trained models)
• d=7 was the sweet spot for early trained models — multiple independent teams converged on this
• d=4 now works with rank-3 factorization + grokking (311 params trained)
• Hand-coded models can go much smaller (36 vs 311 trained) since they don't need to be discoverable by SGD
• Rank-3 factorization is the key trick for trained models
• ALiBi enables extreme compression: the 36-param leader uses ALiBi with slope log(10) for base-10 positional weighting, achieving 100% accuracy with a 2-layer decoder (d=5) in float64

MIT