The first rubric had nine separate signals. A compilation gate. A check that the code actually used p5.brush rather than native p5. A code length ramp targeting around 3,000 tokens. HPSv3, a human preference model. Prompt adherence, judged by a council of GPT-5.4 and Gemini. And four more quality judges: recognisability, aesthetics, technique, depth.
The model plateaued around 0.65 reward and stayed there. Every rollout looked the same. A flat, clip-art flower with five rounded petals. The reward kept going up but the capabilities didn't seem to improve.
The diagnosis came from looking at the sub-rewards in isolation. The four quality judges plus prompt adherence were correlated with each other at 0.85 to 0.95. They were measuring the same thing five times. Code length, contributing roughly a third of the total reward, had saturated by step thirty and was producing zero gradient afterward. HPSv3, the one signal showing real variance, was weighted at 0.10. The rubric we made was telling the model the same thing over and over again.
The fix had two halves.
Replace absolute scoring with pairwise judgment. The original rubric asked the judge to score each rollout from zero to ten. The scores came back compressed near zero. Pairwise scoring asks a different question. The judge is shown the rollout, two references from the pool, and a single prompt: which of these is the better hibiscus watercolour? The reward is the fraction of comparisons it wins. The dynamic range opens up. The judge model handles a relative question more reliably than an abstract scale.
Build a reference pool of hand-rated examples.1,664 images, rated one at a time into love, okay, and nope. The 117 love-tier examples seeded the comparison pool. Every rollout from that point onward was being judged against the things I had decided were good. The next step, which we did not get to, would have been training a small reward model on the ratings themselves, (proper RLHF) so the model's sense of good could be applied without needing to compare against the pool every time.
The new rubric collapsed all of it into four components: a binary compile-and-uses-brush gate (0.05), a binary length check (0.05), HPSv3 (0.30), and the pairwise judge against the reference pool (0.60). Same base model, same training data. The next run reached the previous plateau three times faster, kept climbing past it, and produced code that compressed from 13,500 tokens to under 2,000. The model learned that winning compositions did not need verbose code.